PyTorch - Neural Network Basics

PyTorch - Neural Network Basics

At its core, a neural network is a collection of interconnected processing units called artificial neurons or perceptrons. Each neuron receives several inputs (x1, x2, …, xn), computes a weighted sum of these inputs, adds a bias, and then passes this sum through an activation function to produce an output.

Conceptually, a single neuron can be represented as follows:

Inputs: x_1, x_2, ..., x_n

Weights: w_1, w_2, ..., w_n (associated with each input)

Bias: b

The weighted sum is calculated as: z = (w_1*x_1 + w_2*x_2 + ... + w_n*x_n) + b

This sum z is then passed through an activation function f to produce the neuron’s output: output = f(z).

In mathematical notation, this is often expressed as:

$$Output = f(\sum_{j} w_j x_j + Bias)$$ Where f is the activation function.

Activation Functions

Activation functions introduce non-linearity into the network, allowing it to learn complex patterns. Common activation functions include:

• Sigmoid: Maps input to a range between 0 and 1. Often used in output layers for binary classification. f(z) = 1 / (1 + e^-z)
• ReLU (Rectified Linear Unit): Outputs the input directly if it’s positive, otherwise, it outputs zero. Widely used in hidden layers due to its efficiency. f(z) = max(0, z)
• Tanh (Hyperbolic Tangent): Maps input to a range between -1 and 1. f(z) = (e^z - e^-z) / (e^z + e^-z)
• Softmax: Used in output layers for multi-class classification, as it converts a vector of raw scores into a probability distribution.

A typical neural network architecture consists of multiple layers of these neurons:

1. Input Layer: Receives the initial data (features). The number of neurons in this layer corresponds to the number of features in the input data.

2. Hidden Layers: These are layers between the input and output layers. They perform intermediate computations and feature extraction. A network can have zero or more hidden layers. The ‘deep’ in ‘deep learning’ refers to having multiple hidden layers.

3. Output Layer: Produces the final result of the network (e.g., a class prediction, a continuous value). The number of neurons and the activation function in this layer depend on the specific task.

The connections between neurons in adjacent layers have associated weights. Learning in a neural network involves adjusting these weights (and biases) to minimize the difference between the network’s predictions and the actual target values.

Connections can be dense (fully connected), where every neuron in one layer is connected to every neuron in the next, or sparse (e.g., in Convolutional Neural Networks, neurons are connected only to a local region in the input).

Types of Neural Networks

Feedforward Neural Networks (FNNs)

Feedforward Neural Networks are the most basic type. Data flows in one direction: from the input layer, through any hidden layers, to the output layer. There are no loops or cycles in the network. Each layer processes the output of the previous layer. Simple FNNs are also known as Multi-Layer Perceptrons (MLPs).

Imagine data passing through a series of processing stages, where each stage transforms the data based on its learned weights, and then passes it to the next stage without looking back.

Recurrent Neural Networks (RNNs)

Recurrent Neural Networks are designed to handle sequential data, where the order of information matters (e.g., time series, text, speech). RNNs have connections that form directed cycles, allowing them to maintain an internal state or ‘memory’ of past inputs. This memory influences the processing of subsequent inputs.

Think of an RNN as reading a sentence word by word, and its understanding of the current word is influenced by the words it has already seen.

Other important types include Convolutional Neural Networks (CNNs), primarily used for image and spatial data, and Transformers, which have become state-of-the-art for many sequence processing tasks.

Building Neural Networks in PyTorch

PyTorch provides the torch.nn package, which contains all the building blocks for creating neural networks. This includes various layer types (e.g., nn.Linear, nn.Conv2d), activation functions (e.g., nn.ReLU, nn.Sigmoid), loss functions (e.g., nn.MSELoss, nn.CrossEntropyLoss), and containers like nn.Sequential.

A typical training procedure for a neural network in PyTorch involves the following steps:

1. Define the Neural Network: Create a class that inherits from nn.Module. Define the layers and their connections in the __init__ method, and specify how data flows through the network in the forward method. The network will have learnable parameters (weights and biases).
2. Prepare Data: Load and preprocess your data, typically using PyTorch’s Dataset and DataLoader classes to iterate over the data in batches.
3. Iterate and Process Input: Loop through your dataset. In each iteration (or step):
5. Backpropagate Gradients: Calculate the gradients of the loss with respect to the network’s parameters. This is done using loss.backward().
6. Update Weights: Adjust the network’s parameters (weights and biases) in the direction that minimizes the loss. This is typically done using an optimizer (e.g., SGD, Adam). optimizer.step()
7. Zero Gradients: Before the next iteration, clear the accumulated gradients using optimizer.zero_grad().

The core idea behind updating weights is often simplified by the rule: new_weight = old_weight - learning_rate * gradient. Optimizers implement this rule, often with more sophisticated enhancements to improve convergence and stability.

This iterative process of forward pass, loss computation, backward pass, and weight update is repeated for multiple epochs (passes over the entire dataset) until the model’s performance is satisfactory.