Maximum Likelihood Estimation

Generative AI Models: Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE) is a fundamental statistical method for estimating the parameters of a probability distribution that best describes a given dataset. The core idea of MLE is to find the parameter values that maximize the likelihood function, which essentially means finding the parameters that make the observed data most probable under the assumed statistical model. In simpler terms, MLE helps us guess the unknown characteristics (parameters) of a process that generated our data, by choosing the characteristics that are ‘most likely’ to have produced the data we see.

In the context of generative AI, particularly for models that explicitly define a likelihood function (like Variational Autoencoders - VAEs, or some types of energy-based models), MLE plays a crucial role in training. For instance, when training a VAE to generate images of handwritten digits (e.g., from the MNIST dataset), we want the model’s parameters to be such that the probability of observing our training digits, according to the model, is maximized. This is often expressed as maximizing the log-likelihood: log P(X | theta), where X is the data and theta represents the model parameters.

While models like Generative Adversarial Networks (GANs) don’t directly optimize a likelihood function in the same way, the underlying principle of learning a distribution that closely matches the data distribution is related. This chapter focuses on understanding the concept of Maximum Likelihood Estimation, its significance in generative modeling, its applications, and a simple Python implementation.

Understanding Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation is a powerful and widely used statistical method for inferring the parameters of a probability distribution from observed data. Let’s explore its mathematical foundations.

Mathematical Foundation of MLE

At the core of MLE is the likelihood function, denoted as L(theta | x). Here, ‘theta’ represents the set of parameters of the assumed probability distribution (e.g., mean and variance for a Gaussian distribution), and ‘x’ represents the observed data (typically a set of independent and identically distributed - i.i.d. - samples x_1, x_2, …, x_n).

The likelihood function L(theta | x) quantifies how probable the observed data ‘x’ is for a given set of parameter values ‘theta’. If the data samples are i.i.d., the likelihood function is the product of the probability density function (PDF) or probability mass function (PMF) evaluated at each data point:

This can be written as: L(theta | x) = product_{i=1 to n} f(x_i | theta), where f(x_i | theta) is the PDF/PMF.

For computational convenience and numerical stability, it’s often easier to work with the log-likelihood function, denoted as l(theta | x) or log L(theta | x). Since the logarithm is a monotonically increasing function, maximizing the log-likelihood is equivalent to maximizing the likelihood.

l(theta | x) = log(L(theta | x)) = sum_{i=1 to n} log(f(x_i | theta))

The goal of MLE is to find the parameter values, denoted as theta_hat (or theta_MLE), that maximize the likelihood function L(theta | x) or, equivalently, the log-likelihood function l(theta | x):

theta_hat = argmax_{theta} L(theta | x) = argmax_{theta} l(theta | x)

To find these optimal parameter values, we typically differentiate the log-likelihood function l(theta | x) with respect to each parameter in ‘theta’ and set these derivatives (the gradient) equal to zero. Solving these equations (known as the likelihood equations) yields the maximum likelihood estimates for the parameters:

d(l(theta | x)) / d(theta_j) = 0, for each parameter theta_j in theta.

It’s also necessary to check second-order conditions (e.g., using the Hessian matrix) to ensure that the solution corresponds to a maximum, not a minimum or saddle point, though for many standard distributions, this is guaranteed.

MLE in Generative Modeling

Generative modeling, as discussed, involves learning the underlying distribution of data to generate new, similar data samples. MLE is a key technique for training many types of generative models, especially those that are ‘prescribed’ or ‘explicit,’ meaning they define an explicit probability density function p(x | theta).

Here’s how MLE is generally applied in this context:

1. Model Selection

First, a probabilistic model (a family of distributions) is chosen that is believed to be appropriate for capturing the underlying structure of the data. Common choices include Gaussian distributions, mixture models (like Gaussian Mixture Models - GMMs), or more complex models defined by neural networks (e.g., in VAEs, normalizing flows).

2. Likelihood Function Definition

Next, the likelihood function L(theta | D) is defined based on the chosen model and the training dataset D = {x_1, x_2, …, x_N}. This function measures the probability (or probability density) of observing the entire dataset D given specific model parameters ‘theta’. As stated before, for i.i.d. data:

L(theta | D) = product_{i=1 to N} p(x_i | theta)

And the log-likelihood is: l(theta | D) = sum_{i=1 to N} log p(x_i | theta)

3. Maximization

The core step is to maximize the log-likelihood function with respect to the model parameters ‘theta’. This often involves numerical optimization algorithms like gradient ascent (or gradient descent on the negative log-likelihood) if an analytical solution is not feasible, which is common for complex models like neural networks.

4. Parameter Estimation

Once the (log-)likelihood function is maximized, the resulting parameter values (theta_hat) are the MLEs for the parameters of the generative model. This learned distribution, defined by p(x | theta_hat), can then be used to generate new data points by sampling from it.

Applications of MLE in Specific Generative Models

MLE finds applications across various generative modeling paradigms:

Gaussian Mixture Models (GMMs): MLE, often via the Expectation-Maximization (EM) algorithm, is used to estimate the parameters (means, covariances, and mixture weights) of the Gaussian components in GMMs. This enables modeling complex data distributions with multiple modes.
Variational Autoencoders (VAEs): VAEs are trained by maximizing a lower bound on the log-likelihood of the data, known as the Evidence Lower Bound (ELBO). While not direct MLE, maximizing the ELBO is closely related to maximizing the log-likelihood and involves learning parameters for both an encoder (inference network) and a decoder (generative network).
Normalizing Flows: These models transform a simple base distribution (e.g., Gaussian) into a complex data distribution through a series of invertible transformations. They are trained directly by maximizing the exact log-likelihood of the data, which can be computed efficiently due to the invertible nature of the transformations and the change of variables formula.
Autoregressive Models: Models like PixelCNN or WaveNet generate data sequentially, where each new element is conditioned on previous elements. They are trained by maximizing the likelihood of the observed sequences, factored as a product of conditional probabilities.
Generative Adversarial Networks (GANs): GANs do not typically optimize a likelihood function directly. Instead, they use an adversarial training process involving a generator and a discriminator. However, the goal of the generator is to learn a distribution that is indistinguishable from the true data distribution. Some GAN variants and theoretical analyses connect GAN objectives to minimizing divergences between distributions, which is conceptually related to finding a good fit for the data.

Implementing Maximum Likelihood Estimation using Python

We can implement a simple MLE example using Python to estimate the parameters of a Gaussian (Normal) distribution from a given dataset. For a Gaussian distribution, the MLE for the mean (mu) is the sample mean, and the MLE for the standard deviation (sigma) is the sample standard deviation (with a slight adjustment for unbiased variance if needed, but for MLE it’s the direct sample std).

Example: Estimating Parameters of a Gaussian Distribution

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Generate some sample data from a known Gaussian distribution
# True parameters
true_mu = 5.0
true_sigma = 2.0
np.random.seed(42) # for reproducibility
data = np.random.normal(loc=true_mu, scale=true_sigma, size=1000)

# Maximum Likelihood Estimation for a Gaussian distribution
def gaussian_mle(data):
    # MLE for mu is the sample mean
    estimated_mu = np.mean(data)
    # MLE for sigma is the sample standard deviation
    estimated_sigma = np.std(data)
    return estimated_mu, estimated_sigma

# Perform Maximum Likelihood Estimation
estimated_mu, estimated_sigma = gaussian_mle(data)

print(f"True mu: {true_mu}, True sigma: {true_sigma}")
print(f"Estimated mu (MLE): {estimated_mu:.4f}, Estimated sigma (MLE): {estimated_sigma:.4f}")

# Generate x values for plotting the PDFs
x_values = np.linspace(min(data), max(data), 200)

# Plot histogram of the data
plt.figure(figsize=(10, 6))
plt.hist(data, bins=30, density=True, alpha=0.6, color='skyblue', label='Data Histogram')

# Plot the true Gaussian PDF
true_pdf = norm.pdf(x_values, loc=true_mu, scale=true_sigma)
plt.plot(x_values, true_pdf, color='red', linestyle='--', linewidth=2, label=f'True Gaussian (mu={true_mu}, sigma={true_sigma})')

# Plot the estimated Gaussian PDF using MLE parameters
estimated_pdf = norm.pdf(x_values, loc=estimated_mu, scale=estimated_sigma)
plt.plot(x_values, estimated_pdf, color='green', linestyle='-', linewidth=2, label=f'Estimated Gaussian (MLE) (mu={estimated_mu:.2f}, sigma={estimated_sigma:.2f})')

plt.xlabel('Value')
plt.ylabel('Probability Density')
plt.title('Maximum Likelihood Estimation for Gaussian Distribution')
plt.legend()
plt.grid(True)
plt.show()

Expected Output Description

Running the above Python code will first print the true and estimated parameters (mean and standard deviation). Then, it will display a plot. The plot will show a histogram of the generated sample data. Overlaid on this histogram will be two curves: one representing the probability density function (PDF) of the true Gaussian distribution from which the data was sampled (typically a dashed red line), and another representing the PDF of the Gaussian distribution estimated using MLE (typically a solid green line). If MLE works well and the sample size is sufficient, the estimated PDF should closely match the true PDF and the data histogram.

Conclusion

Maximum Likelihood Estimation is a cornerstone of statistical inference and plays a vital role in training many generative AI models. It provides a principled way to estimate model parameters by finding values that make the observed data most probable.

The general process involves selecting a model, defining a likelihood function, maximizing this function (or its logarithm) to find optimal parameters, and then using these estimated parameters to define the learned generative model. While not all generative models (like GANs) directly optimize likelihood, the principles of MLE inform the broader goal of learning accurate data distributions.