PyTorch - Terminologies

PyTorch - Core Concepts

This chapter introduces fundamental terminologies and concepts crucial for understanding and working with PyTorch.

PyTorch Tensors and NumPy

A PyTorch Tensor is a multi-dimensional array, similar in concept to NumPy’s ndarray. However, Tensors have key advantages: they can be moved to GPUs for accelerated computation, and they are the backbone of PyTorch’s automatic differentiation engine, Autograd.

While Tensors and NumPy arrays can often be converted between each other (e.g., torch.from_numpy(), tensor.numpy()), PyTorch Tensors are essential for building and training neural networks within the PyTorch ecosystem. They support a rich set of operations, much like NumPy.

Tensors and Autograd

Autograd is PyTorch’s automatic differentiation engine. When a Tensor’s requires_grad attribute is set to True, PyTorch starts tracking all operations performed on it. This tracking builds a computational graph dynamically.

After a forward pass (computations from input to output), calling .backward() on a scalar output (e.g., loss) automatically computes the gradients of that scalar with respect to all Tensors that had requires_grad=True and contributed to it. These gradients are accumulated in their .grad attribute. This mechanism is fundamental for training neural networks via backpropagation.

Previously, PyTorch used a Variable wrapper around Tensors for autograd. This is now deprecated; Tensors themselves handle all requires_grad and gradient tracking functionalities directly.

Dynamic Computational Graphs

PyTorch is known for its use of dynamic computational graphs (also referred to as “define-by-run”). This means the graph representing the model’s computations is built on-the-fly as the code executes. This provides great flexibility, especially for models with structures that change based on input data (e.g., RNNs handling variable-length sequences, or conditional computations).

This contrasts with older versions of frameworks like TensorFlow, which primarily used static graphs (define-then-run), where the entire graph was defined upfront before execution. While TensorFlow 2.x now also defaults to eager execution (dynamic graphs), PyTorch’s design philosophy has always centered around this dynamic nature, making it very intuitive for Python programmers and researchers who value flexibility and ease of debugging.

The torch.optim Package

The torch.optim package provides implementations of various optimization algorithms used for training neural networks. Once gradients are computed using .backward(), an optimizer is used to update the model’s parameters (weights and biases).

Commonly used optimizers include Stochastic Gradient Descent (SGD), Adam, RMSprop, etc. You initialize an optimizer with the model’s parameters and a learning rate, then call optimizer.step() after loss.backward() to apply the updates. It’s crucial to call optimizer.zero_grad() before each loss.backward() call to clear old gradients from the previous iteration.

Parallelism and torch.multiprocessing

PyTorch supports various forms of parallelism. torch.multiprocessing is PyTorch’s wrapper around Python’s multiprocessing module. It’s often used to speed up data loading by using multiple CPU cores to prepare data batches in parallel while the GPU is busy with model computations (e.g., via DataLoader’s num_workers argument).

For model training, PyTorch offers torch.nn.DataParallel for simple multi-GPU training on a single machine by splitting batches across GPUs, and torch.nn.parallel.DistributedDataParallel for more advanced, efficient multi-GPU and distributed training across multiple machines. These are higher-level utilities built on top of PyTorch’s core tensor and autograd functionalities.