Mathematical Foundations

TensorFlow - Mathematical Foundations

A solid understanding of certain mathematical concepts is crucial before diving deep into TensorFlow and machine learning. Mathematics forms the bedrock of machine learning algorithms, providing the tools to define problems, formulate solutions, and evaluate performance. TensorFlow itself is built around manipulating multi-dimensional arrays called tensors, which are a generalization of these mathematical objects.

Vector

A vector is an ordered list of numbers, which can be thought of as a point in space or a quantity with both magnitude and direction. In machine learning, inputs (features) and parameters are often represented as vectors. Vectors are 1D Tensors. For example, [1.0, 2.5, -0.7] is a 3-dimensional vector. In TensorFlow, this can be represented as: tf.constant([1.0, 2.5, -0.7]).

Machine learning algorithms frequently deal with high-dimensional data, making vectors a fundamental data structure.

Scalar

A scalar is a single number, as opposed to a vector or matrix. It has magnitude but no direction in a geometric sense. Scalars are 0D Tensors. Examples include the learning rate in an optimization algorithm or a single pixel intensity. In TensorFlow: tf.constant(42.0).

Matrix

A matrix is a two-dimensional array of numbers, arranged in rows and columns. The size of a matrix is defined by its number of rows and columns (e.g., an m x n matrix has ‘m’ rows and ‘n’ columns). Matrices are 2D Tensors. They are extensively used to represent datasets (samples as rows, features as columns), weights in neural network layers, and covariance matrices. Example: [[1, 2, 3], [4, 5, 6]] is a 2x3 matrix. In TensorFlow: tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]).

Mathematical Computations with Tensors

TensorFlow provides a rich library for performing mathematical operations on tensors, mirroring standard linear algebra operations.

Addition of matrices (or tensors)

Addition of two matrices (or tensors of the same shape) is performed element-wise. If A and B are matrices of the same dimensions, then (A + B)_ij = A_ij + B_ij.

$$Example:A=\begin{bmatrix}1 & 2 \3 & 4 \end{bmatrix}B=\begin{bmatrix}5 & 6 \7 & 8 \end{bmatrix}:then:A+B=\begin{bmatrix}1+5 & 2+6 \3+7 & 4+8 \end{bmatrix}=\begin{bmatrix}6 & 8 \10 & 12 \end{bmatrix}$$

In TensorFlow: C = tf.add(A, B) or C = A + B

Subtraction of matrices (or tensors)

Subtraction also operates element-wise, similar to addition, for matrices of the same dimensions. (A - B)_ij = A_ij - B_ij.

$$Example:A=\begin{bmatrix}1 & 2 \3 & 4 \end{bmatrix}B=\begin{bmatrix}5 & 6 \7 & 8 \end{bmatrix}:then:A-B=\begin{bmatrix}1-5 & 2-6 \3-7 & 4-8 \end{bmatrix}=\begin{bmatrix}-4 & -4 \-4 & -4 \end{bmatrix}$$

In TensorFlow: C = tf.subtract(A, B) or C = A - B

Matrix Multiplication

For two matrices A (m x n) and B (p x q) to be multipliable in the order A * B, the number of columns in A must equal the number of rows in B (i.e., n = p). The resulting matrix C will have dimensions m x q. The element C_ij is the dot product of the i-th row of A and the j-th column of B.

$$A=\begin{bmatrix}1 & 2 \3 & 4 \end{bmatrix}B=\begin{bmatrix}5 & 6 \7 & 8 \end{bmatrix} \text{ (both are 2x2, so n=2, p=2)}$$

$$c_{11}=(1 \times 5) + (2 \times 7)=19$$ $$c_{12}=(1 \times 6) + (2 \times 8)=22$$

$$c_{21}=(3 \times 5) + (4 \times 7)=43$$ $$c_{22}=(3 \times 6) + (4 \times 8)=50$$

$$C=\begin{bmatrix}c_{11} & c_{12} \c_{21} & c_{22} \end{bmatrix}=\begin{bmatrix}19 & 22 \43 & 50 \end{bmatrix}$$

In TensorFlow: C = tf.matmul(A, B) or C = A @ B (Python 3.5+ operator for matrix multiplication). Note: A * B in TensorFlow performs element-wise multiplication.

Transpose of a matrix

The transpose of an m x n matrix A, denoted A^T, is an n x m matrix obtained by interchanging its rows and columns. (A^T)_ij = A_ji.

$$Example:A=\begin{bmatrix}1 & 2 \3 & 4 \end{bmatrix}:then:A^{T}=\begin{bmatrix}1 & 3 \2 & 4 \end{bmatrix}$$

In TensorFlow: At = tf.transpose(A)

Dot product of vectors

The dot product (or scalar product) of two n-dimensional vectors v₁ and v₂ is the sum of the products of their corresponding components. If vectors are represented as column matrices (n x 1), the dot product can be expressed as v₁^Tv₂.

$$v_{1}=\begin{bmatrix}v_{11} \v_{12} \\vdots\v_{1n}\end{bmatrix}v_{2}=\begin{bmatrix}v_{21} \v_{22} \\vdots\v_{2n}\end{bmatrix}$$

The dot product is: $$v_{1}\cdot v_{2}=v_1^Tv_{2}=v_{11}v_{21}+v_{12}v_{22}+\dots+v_{1n}v_{2n}=\sum\limits_{k=1}^n v_{1k}v_{2k}$$

The result is a scalar value.

$$Example:v_{1}=\begin{bmatrix}1 \2 \3\end{bmatrix}v_{2}=\begin{bmatrix}3 \5 \-1\end{bmatrix} \implies v_{1}\cdot v_{2}=(1\times3)+(2\times5)+(3\times-1)=3+10-3=10$$

In TensorFlow (for 1D tensors/vectors): dot_product = tf.tensordot(v1_1d, v2_1d, axes=1) or dot_product = tf.reduce_sum(v1_1d * v2_1d). If v1 and v2 are column vectors (shape [n,1]): dot_product_matrix = tf.matmul(tf.transpose(v1), v2) which results in a 1x1 matrix, then tf.squeeze(dot_product_matrix) to get scalar.