GAN - Architecture

Generative Adversarial Network: Architecture and Key Variants

A Generative Adversarial Network (GAN) employs a unique architecture typically composed of two neural networks: a Generator and a Discriminator. These networks are trained in an adversarial setting, where the generator strives to create realistic data, and the discriminator aims to distinguish between real and generated data. For tasks like image generation, Convolutional Neural Networks (CNNs) are commonly used within both components. This chapter delves into the architecture of GANs, the roles of their core components, their objective functions, and introduces several important types of GANs that have advanced the field.

The Generator Network in GAN Architecture

The first primary component of a GAN’s architecture is the Generator (G). Its function is to learn the underlying distribution of the training data and synthesize new data samples from that learned distribution.

Generator: Function and Structure

The primary goal of the generator is to produce synthetic data samples that are intended to be indistinguishable from real data samples drawn from the target dataset. It typically starts with a random noise vector (often called a latent vector, ‘z’), usually sampled from a simple distribution like a Gaussian or uniform distribution. The generator then transforms this low-dimensional latent vector through a series of neural network layers into a higher-dimensional data sample (e.g., an image).

Generator: Common Layers and Components

The specific layers within a generator depend on the type of data being generated. For image generation, common components include:

• Input Layer: Receives the random noise vector ‘z’ as input.
• Fully Connected (Dense) Layers: Often used in the initial stages to project and reshape the noise vector into a higher-dimensional representation suitable for subsequent convolutional layers.
• Transposed Convolutional Layers (Deconvolutional Layers): These are key for upsampling. They increase the spatial dimensions of the feature maps, gradually transforming the compact latent representation into a full-sized image.
• Batch Normalization Layers: Used to stabilize training by normalizing the activations in intermediate layers, which helps with gradient flow and allows for higher learning rates.
• Activation Functions: Common choices include ReLU (Rectified Linear Unit) and LeakyReLU for hidden layers to introduce non-linearity. The output layer typically uses an activation function appropriate for the data range, such as Tanh (for data normalized to [-1, 1]) or Sigmoid (for data normalized to [0, 1]).

Generator: Objective Function

The generator’s objective is to produce data G(z) that the discriminator D classifies as real. In the original GAN formulation, this is often achieved by training G to maximize the probability that D makes a mistake on generated samples, i.e., maximize log(D(G(z))). A common alternative, non-saturating loss for the generator is to minimize -log(D(G(z))), which is equivalent to maximizing log(D(G(z))) but can have better gradient properties early in training. A simplified way to think of the generator’s loss L_G when it tries to make D(G(z)) close to 1 (real) is often related to:

L_G aims to minimize something like: E_{zp_z(z)} [log(1 - D(G(z)))] (original minimax formulation) or, more commonly in practice, E_{zp_z(z)} [-log(D(G(z)))] (non-saturating heuristic).

Here, G(z) is the data generated from noise z, and D(G(z)) is the discriminator’s probability estimate that G(z) is real.

The Discriminator Network in GAN Architecture

The second primary component of a GAN’s architecture is the Discriminator (D). Its function is to act as a binary classifier.

Discriminator: Function and Structure

The primary goal of the discriminator is to distinguish between real data samples (from the training dataset) and fake (synthetic) data samples produced by the generator. It takes a data sample (either real ‘x’ or fake G(z)) as input and outputs a scalar value representing the probability that the input sample is real.

Discriminator: Common Layers and Components

For image data, the discriminator is typically a standard Convolutional Neural Network (CNN) designed for classification:

• Input Layer: Receives a data sample (e.g., an image) from either the real dataset or the generator.
• Convolutional Layers: Used to extract features from the input image. These layers typically downsample the input by using strides or pooling, progressively reducing spatial dimensions while increasing feature depth.
• Activation Functions: LeakyReLU is commonly used in hidden layers to prevent the ‘dying ReLU’ problem and allow gradients to flow even for negative input values. It introduces non-linearity.
• Fully Connected (Dense) Layers: After several convolutional layers, the feature maps are often flattened and passed through one or more dense layers to perform the final classification.
• Output Layer: Produces a single scalar output. In the original GAN, this is often passed through a Sigmoid activation function to represent a probability (0 for fake, 1 for real). However, in some modern GAN variants (like WGAN), the final activation is omitted, and the raw logit is used.

Discriminator: Objective Function

The discriminator’s objective is to correctly classify real data as real and fake data as fake. It is trained to maximize the probability of assigning the correct label to both real examples and generated examples. Its loss function, L_D, encourages D(x) to be close to 1 (for real x) and D(G(z)) to be close to 0 (for fake G(z)). This is often expressed as maximizing:

L_D_maximize = E_{xp_data(x)} [log D(x)] + E_{zp_z(z)} [log(1 - D(G(z)))]

To use gradient descent, the discriminator minimizes the negative of this, i.e., -L_D_maximize. Here, x is a real data sample.

Prominent Types of Generative Adversarial Networks

Since the original GAN concept, numerous variations have been developed to improve training stability, sample quality, and control over the generation process. Here are some notable types:

Vanilla GAN

This refers to the original GAN architecture proposed by Ian Goodfellow et al. in 2014. It uses fully connected layers for both the generator and discriminator and establishes the fundamental adversarial training framework. While foundational, it can be challenging to train stably.

Deep Convolutional GANs (DCGANs)

Introduced by Radford et al., DCGANs provided key architectural guidelines that significantly stabilized GAN training, especially for image generation. These include: replacing pooling layers with strided convolutions (in discriminator) and fractional-strided convolutions (in generator), using batch normalization in both networks (except generator output and discriminator input), removing fully connected hidden layers for deeper architectures, using ReLU activation in generator for all layers except output (which uses Tanh), and LeakyReLU activation in the discriminator for all layers.

Conditional GANs (cGANs)

cGANs, proposed by Mirza and Osindero, extend the GAN framework by conditioning both the generator and discriminator on additional information ‘y’ (e.g., class labels, attributes, or even other images). This allows for control over the characteristics of the generated output. For example, a cGAN can be trained to generate an image of a specific digit if ‘y’ is the digit’s label.

CycleGANs

Developed by Zhu et al., CycleGANs are designed for unpaired image-to-image translation tasks, where there is no direct correspondence between images in the source and target domains (e.g., translating photos of horses to zebras). They use a cycle consistency loss, which ensures that translating an image from domain A to domain B and then back to domain A should result in an image similar to the original. This is achieved using two generators and two discriminators.

Progressive GANs (ProGANs)

Proposed by Karras et al. (NVIDIA), ProGANs achieve high-resolution image generation by progressively growing both the generator and discriminator networks during training. Training starts with low-resolution images, and new layers are gradually added to increase the resolution, allowing the model to learn coarse features first and then finer details. This approach improves training stability and speed for high-resolution synthesis.

StyleGANs (and StyleGAN2, StyleGAN3)

Also from NVIDIA (Karras et al.), StyleGANs build upon ProGANs and introduce novel techniques for improved image synthesis quality and unprecedented control over specific visual attributes (styles) at different scales (e.g., pose, hair, face shape). Key innovations include a mapping network for the latent code, adaptive instance normalization (AdaIN) for style modulation, and the separation of global features from stochastic variation. StyleGAN2 and StyleGAN3 further refined the architecture for even better quality and to address artifacts.

Laplacian Pyramid GAN (LAPGAN)

LAPGANs, introduced by Denton et al., use a multi-resolution approach based on the Laplacian pyramid framework to generate high-quality images. A cascade of GANs is trained, where each GAN in the pyramid is responsible for generating the detail (Laplacian residuals) at a specific scale, conditioned on a lower-resolution version of the image from the previous level. This was an earlier approach to generating higher-resolution images by breaking down the problem.

Conclusion

The Generative Adversarial Network architecture, with its competing generator and discriminator, provides a powerful paradigm for learning to generate realistic data. Understanding the roles of these components and their objective functions is key to grasping how GANs work. The evolution from Vanilla GANs to sophisticated variants like DCGANs, cGANs, StyleGANs, and others has dramatically improved the quality, stability, and controllability of generated samples, opening up a vast array of applications across various domains. Research in GANs continues to be vibrant, with new architectures and training techniques constantly emerging.