Keras - Model Compilation

Keras - Model Compilation & Training

After defining the architecture of your model using the Sequential or Functional API, the next step is to compile it. Compilation configures the model for training by specifying the optimizer, loss function, and evaluation metrics. Once compiled, the model is ready to be trained using the fit method.

The Compilation Step (model.compile())

The compile method configures the learning process. It requires several key arguments:

model.compile(
    optimizer='rmsprop', # Optimization algorithm
    loss=None,           # Loss function to minimize
    metrics=None,        # List of metrics to monitor
    loss_weights=None,   # Optional weights for different outputs (multi-output models)
    weighted_metrics=None, # Metrics to compute weighted by sample_weight or class_weight
    run_eagerly=False,   # If True, model runs step-by-step (easier debugging, slower)
    steps_per_execution=1, # Number of batches to run in a single tf.function call
    **kwargs
)

Essential Arguments:

• optimizer: This dictates how the model’s weights are updated based on the gradients calculated from the loss function. It controls the learning process.
• • Common Choices: 'adam', 'sgd', 'rmsprop', or an instance like tf.keras.optimizers.Adam(learning_rate=0.001).
• • Why it matters: The optimizer significantly impacts training speed and convergence. Adam is often a good starting point.
• loss: The function that measures the discrepancy between the model’s predictions and the true target values during training. The optimizer’s goal is to minimize this loss.
• • Common Choices: 'binary_crossentropy', 'categorical_crossentropy', 'sparse_categorical_crossentropy' (for classification); 'mean_squared_error', 'mean_absolute_error' (for regression); or an instance like tf.keras.losses.BinaryCrossentropy().
• • Why it matters: The loss function must match the nature of the problem and the activation function of the output layer (e.g., use cross-entropy with softmax/sigmoid for classification).
• metrics: A list of metrics used to evaluate the model’s performance during training and testing. These are for monitoring and reporting purposes and do not affect the weight updates directly.
• • Common Choices: ['accuracy'] (for classification), ['mae', 'mse'] (for regression), or instances like [tf.keras.metrics.AUC(), tf.keras.metrics.Precision()].
• • Why it matters: Metrics provide human-understandable insights into how well the model is performing.

Example Compilation:

# For a multi-class classification model
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(), # Use if labels are integers
              # loss='sparse_categorical_crossentropy', # Equivalent string identifier
              metrics=['accuracy'])

# For a binary classification model
# model.compile(optimizer='adam',
#               loss='binary_crossentropy',
#               metrics=['accuracy', tf.keras.metrics.AUC()])

# For a regression model
# model.compile(optimizer=tf.keras.optimizers.RMSprop(),
#               loss='mean_squared_error',
#               metrics=['mae'])

The Training Step (model.fit())

The fit method trains the model for a fixed number of epochs (passes over the entire dataset). It takes the training data, validation data (optional), and training configuration parameters.

history = model.fit(
    x=None,                  # Input training data (NumPy array, tf.data.Dataset, etc.)
    y=None,                  # Target training data (labels)
    batch_size=None,         # Number of samples per gradient update
    epochs=1,                # Number of times to iterate over the entire dataset
    verbose='auto',          # Verbosity mode (0, 1, or 2)
    callbacks=None,          # List of tf.keras.callbacks.Callback instances
    validation_split=0.0,    # Fraction of training data to use as validation data
    validation_data=None,    # Data on which to evaluate loss and metrics at the end of each epoch (e.g., (x_val, y_val) or a Dataset)
    shuffle=True,            # Whether to shuffle training data before each epoch
    class_weight=None,       # Optional dict mapping class indices to a weight value
    sample_weight=None,      # Optional NumPy array of weights for training samples
    initial_epoch=0,         # Epoch at which to start training (useful for resuming)
    steps_per_epoch=None,    # Total number of steps (batches) to run per epoch (used with Datasets)
    validation_steps=None,   # Number of steps to run validation for (used with Datasets)
    validation_batch_size=None, # Batch size for validation data
    **kwargs
)

Key Arguments:

• x, y: The training data and corresponding labels/targets.
• batch_size: Controls how many samples are processed before the model’s weights are updated. Smaller batches can lead to noisier updates but might help escape local minima. Larger batches provide more stable gradients but require more memory.
• epochs: One epoch means the model has seen the entire training dataset once. Training usually requires multiple epochs.
• validation_data (or validation_split): Crucial for monitoring performance on data the model hasn’t trained on during the epoch. Helps detect overfitting (when training loss decreases but validation loss increases).
• callbacks: Allows automating tasks like saving the best model (ModelCheckpoint), stopping training early if performance plateaus (EarlyStopping), or logging for TensorBoard (TensorBoard).

The fit method returns a History object. This object contains a record of the loss and metric values during training, which is useful for plotting learning curves.

# Assuming x_train, y_train, x_val, y_val are prepared
# history = model.fit(x_train, y_train,
#                     batch_size=32,
#                     epochs=10,
#                     validation_data=(x_val, y_val),
#                     callbacks=[...])

# Access training history
# print(history.history.keys()) # dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
# loss_curve = history.history['loss']
# accuracy_curve = history.history['accuracy']

Example: Training an MLP on MNIST

Let’s put compilation and training together using the MNIST handwritten digit dataset.

Step 1: Import Modules

import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten # Flatten added for potential CNN use later
from tensorflow.keras.optimizers import Adam # Using Adam optimizer
import numpy as np

Step 2: Load and Preprocess Data

# Load data
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Preprocess images
# Reshape from (60000, 28, 28) to (60000, 784) for MLP
x_train = x_train.reshape(60000, 784).astype('float32') / 255.0
x_test = x_test.reshape(10000, 784).astype('float32') / 255.0

# No need to one-hot encode labels if using SparseCategoricalCrossentropy
num_classes = 10
# y_train = tf.keras.utils.to_categorical(y_train, num_classes)
# y_test = tf.keras.utils.to_categorical(y_test, num_classes)

Step 3: Build the Model

model = Sequential([
    tf.keras.Input(shape=(784,)), # Define input shape explicitly (alternative)
    Dense(512, activation='relu'),
    Dropout(0.2),
    Dense(512, activation='relu'),
    Dropout(0.2),
    Dense(num_classes, activation='softmax') # Output layer for 10 classes
])

model.summary()

Step 4: Compile the Model

model.compile(loss='sparse_categorical_crossentropy', # Use sparse version for integer labels
              optimizer=Adam(learning_rate=0.001),
              metrics=['accuracy'])

Step 5: Train the Model

batch_size = 128
epochs = 10 # Reduced for quicker demonstration

print("\nTraining Model...")
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test)) # Use test set as validation data here

print("\nTraining Complete.")

Step 6: Evaluate (Covered in next chapter)

# score = model.evaluate(x_test, y_test, verbose=0)
# print('\nTest loss:', score[0])
# print('Test accuracy:', score[1])

This complete example shows the workflow from data loading and preprocessing through model definition, compilation, and training using tf.keras.