AI with Python – Deep Learning

AI with Python – Foundations of Deep Learning

Artificial Neural Networks (ANNs) are computational systems inspired by the structure and function of biological neural networks in the human brain. They form the backbone of Deep Learning, a powerful subfield of machine learning. This chapter explores the fundamentals of Deep Learning and its implementation using Python.

Deep Learning has gained prominence due to advancements in computational power (especially GPUs), availability of large datasets, and algorithmic breakthroughs. It excels at learning intricate patterns from data, making it suitable for complex tasks like image recognition, natural language processing, and more.

Machine Learning vs. Deep Learning

Deep Learning is a specialized form of machine learning. Here’s a comparison highlighting key differences:

Data Dependency

Deep Learning algorithms generally require and perform significantly better with large amounts of data. Traditional Machine Learning (ML) algorithms can work well with smaller datasets, but their performance often plateaus as data size increases, unlike DL models which can continue to improve.

Hardware Dependency (Computational Power)

Deep Learning models, with their many layers and parameters, often demand high-end hardware, particularly GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units), for efficient training. Standard ML algorithms can often run effectively on CPUs.

Feature Extraction

In traditional ML, feature extraction (selecting and transforming raw data into features suitable for the model) is often a manual, domain-expert-driven process. Deep Learning algorithms, on the other hand, can automatically learn hierarchical features directly from the data. Lower layers might learn simple features (e.g., edges in an image), while deeper layers learn more complex, abstract features (e.g., object parts).

Execution Time (Training vs. Inference)

Training deep learning models can be very time-consuming, sometimes taking days or weeks, due to the large datasets and complex architectures. However, once trained, inference (making predictions on new data) can be relatively fast. Traditional ML algorithms generally have faster training times but might be slower at inference for certain complex models.

Approach to Problem Solving

Deep Learning often takes an end-to-end approach, learning directly from raw input to output. Traditional ML might break a problem into smaller, intermediate steps, with separate models or processes for each.

Convolutional Neural Networks (CNNs)

Convolutional Neural Networks (CNNs or ConvNets) are a class of deep neural networks particularly effective for analyzing visual imagery. While standard neural networks treat input as a flat vector, CNNs are designed to process data in a grid-like topology, such as an image (a 2D grid of pixels).

CNNs automatically and adaptively learn spatial hierarchies of features. The initial layers learn to detect simple features like edges and corners, subsequent layers combine these to detect more complex features like shapes or textures, and deeper layers can recognize object parts or even whole objects.

Key Architectural Components of CNNs

Unlike standard NNs where layers are fully connected, CNNs use specific layer types that exploit the spatial structure of images. Neurons in a CNN layer are typically arranged in 3 dimensions: width, height, and depth (depth here refers to the number of channels or feature maps).

Core Layers in CNNs

• Input Layer: Holds the raw pixel values of the image (e.g., height x width x channels, where channels could be 3 for RGB or 1 for grayscale).
• Convolutional Layer (Conv): The core building block. It applies a set of learnable filters (kernels) to the input. Each filter slides (convolves) across the input, computing dot products to create a feature map. This process detects specific patterns like edges, textures, etc. Key parameters include filter size, stride, and padding.
• Activation Layer (e.g., ReLU): Introduces non-linearity into the model, allowing it to learn more complex functions. The Rectified Linear Unit (ReLU) (f(x) = max(0,x)) is a common choice.
• Pooling Layer (e.g., MaxPooling): Reduces the spatial dimensions (width, height) of the feature maps, making the representations more robust to variations in object position and reducing computational load. MaxPooling selects the maximum value from a small region of the feature map.
• Fully Connected Layer (Dense): Typically found at the end of the network. Neurons in a fully connected layer have connections to all activations in the previous layer, similar to standard NNs. These layers perform high-level reasoning and produce the final classification scores or regression values.
• Output Layer: Produces the final output. For classification, it often uses a softmax activation function to output probabilities for each class.

Setting Up Your Python Environment for Deep Learning

For deep learning in Python, popular frameworks include TensorFlow and PyTorch. Keras is a high-level API that can run on top of TensorFlow (and previously other backends). We’ll focus on tensorflow.keras.

You can install TensorFlow (which includes Keras) using pip:

pip install tensorflow

For GPU support (highly recommended for significant DL tasks), you’ll need compatible NVIDIA drivers, CUDA Toolkit, and cuDNN. TensorFlow’s installation guide provides detailed instructions: tensorflow.org/install.

Building a Linear Regressor using an ANN

Neural networks can also be used for regression tasks. Here’s how to build a simple linear regressor using tensorflow.keras on the Boston housing price dataset (a classic dataset with 13 numerical features).

Import necessary packages:

import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.wrappers.scikit_learn import KerasRegressor # Compatibility wrapper
from sklearn.model_selection import cross_val_score, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# To load the Boston housing dataset (it was removed from sklearn 1.2, use an alternative source)
# For this example, we'll assume data is in a CSV file 'boston_housing.csv'
# You can find this dataset on Kaggle or other public dataset repositories.

Load and prepare the dataset. (Ensure boston_housing.csv is in your working directory or provide the correct path. The CSV should have 13 feature columns and 1 target column (MEDV)).

# Load dataset (replace with actual path or loading mechanism)
# Example: Assuming a CSV file where features and target are columns
# Make sure your CSV doesn't have a header row or skip it if it does.
# For example: dataframe = pd.read_csv('boston_housing.csv', delim_whitespace=True, header=None)
# This dataset has 506 samples, 13 features, 1 target.

# For reproducibility, we'll simulate loading it if not available:
# Create a dummy CSV for demonstration if you don't have the file
# This is NOT the actual Boston housing data, just for code flow.
import os
if not os.path.exists('boston_housing.csv'):
    dummy_data = np.random.rand(506, 14)
    pd.DataFrame(dummy_data).to_csv('boston_housing.csv', header=False, index=False)

dataframe = pd.read_csv('boston_housing.csv', header=None)
dataset = dataframe.values

X = dataset[:, 0:13].astype(float)
Y = dataset[:, 13].astype(float)

Define the Keras model using a function (required for KerasRegressor wrapper):

def baseline_model():
    model = Sequential()
    # Input layer: 13 inputs -> Dense layer with 13 neurons, ReLU activation
    # kernel_initializer='normal' was used in older examples, 'glorot_uniform' (default) or 'he_normal' are common for ReLU.
    model.add(Dense(13, input_shape=(13,), kernel_initializer='normal', activation='relu'))
    # Output layer: 1 neuron (for regression), no activation for linear output
    model.add(Dense(1, kernel_initializer='normal'))

    # Compile model: 'adam' optimizer, 'mean_squared_error' loss for regression
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

Set a random seed for reproducibility and evaluate the model using K-Fold cross-validation. We’ll also standardize the data.

seed = 7
np.random.seed(seed)

# Create a pipeline that standardizes the data then builds the Keras model
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)

# K-Fold Cross Validation
kfold = KFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold, scoring='neg_mean_squared_error')

print(f"Standardized Results: Mean MSE: {-results.mean():.2f} (Std Dev: {results.std():.2f})")

The output will show the mean squared error (MSE). Lower MSE is better. Note: cross_val_score returns negative MSE for minimization, so we negate it. input_dim in Dense is now input_shape for the first layer.

Image Classifier: A CNN Application (Cats vs. Dogs)

Let’s build a CNN to classify images of cats and dogs. We’ll use tensorflow.keras. You’ll need a dataset of cat and dog images, typically organized into train/cats, train/dogs, validation/cats, validation/dogs directories. A popular source is the Kaggle “Dogs vs Cats” competition: kaggle.com/c/dogs-vs-cats/data. For this example, assume you have a dataset_cats_dogs directory structured like this.

Import Keras layers and ImageDataGenerator for data augmentation:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt # For plotting training history

Define the CNN model architecture:

img_width, img_height = 128, 128 # Target image dimensions
input_shape_val = (img_width, img_height, 3) # 3 for RGB channels

classifier = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=input_shape_val),
    MaxPooling2D(pool_size=(2, 2)),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),

    Conv2D(128, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),

    Flatten(),
    Dense(512, activation='relu'),
    Dropout(0.5), # Dropout for regularization
    Dense(1, activation='sigmoid') # Sigmoid for binary classification (cat or dog)
])

# Compile the model
classifier.compile(optimizer='adam',
                   loss='binary_crossentropy',
                   metrics=['accuracy'])

Prepare data using ImageDataGenerator. This handles image loading, resizing, batching, and augmentation:

batch_size_val = 32
# Path to your dataset (adjust as needed)
# Assume: 'dataset_cats_dogs/train' and 'dataset_cats_dogs/validation'
# Each containing 'cats' and 'dogs' subdirectories.
# Create dummy directories and images if you don't have the dataset for code flow
if not os.path.exists('dataset_cats_dogs/train/cats'):
    os.makedirs('dataset_cats_dogs/train/cats', exist_ok=True)
    os.makedirs('dataset_cats_dogs/train/dogs', exist_ok=True)
    os.makedirs('dataset_cats_dogs/validation/cats', exist_ok=True)
    os.makedirs('dataset_cats_dogs/validation/dogs', exist_ok=True)
    # Create a few dummy images (e.g., by copying any small jpg)
    # from shutil import copyfile
    # for i in range(5): copyfile('any_image.jpg', f'dataset_cats_dogs/train/cats/cat{i}.jpg')
    # ... and so on for other folders.
    # This part is crucial for the code to run if real data is missing.
    print("Warning: Dummy dataset structure created. Use real data for actual training.")

train_datagen = ImageDataGenerator(
    rescale=1./255,      # Normalize pixel values to [0,1]
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

validation_datagen = ImageDataGenerator(rescale=1./255) # Only rescale for validation

training_set = train_datagen.flow_from_directory(
    'dataset_cats_dogs/train', # Path to training images
    target_size=(img_width, img_height),
    batch_size=batch_size_val,
    class_mode='binary') # 'binary' for 2 classes

validation_set = validation_datagen.flow_from_directory(
    'dataset_cats_dogs/validation', # Path to validation images
    target_size=(img_width, img_height),
    batch_size=batch_size_val,
    class_mode='binary')

Train the model. model.fit() is used instead of the deprecated fit_generator.

epochs_val = 10 # Reduce for quick demo; 25-50 might be needed for good results

# Ensure steps_per_epoch and validation_steps are reasonable for your dataset size
# If dataset is small and these are large, it can lead to issues.
# Typically: num_train_samples // batch_size and num_val_samples // batch_size
steps_per_epoch_val = training_set.samples // batch_size_val
validation_steps_val = validation_set.samples // batch_size_val

if steps_per_epoch_val == 0: steps_per_epoch_val = 1 # Avoid division by zero if dataset is too small
if validation_steps_val == 0: validation_steps_val = 1

history = classifier.fit(
    training_set,
    steps_per_epoch=steps_per_epoch_val,
    epochs=epochs_val,
    validation_data=validation_set,
    validation_steps=validation_steps_val)

# To save the model:
# classifier.save('cats_vs_dogs_cnn.h5')

After training, you can predict on new images. (Code for prediction and plotting history omitted for brevity but would involve loading an image, preprocessing it similarly to training data, and using classifier.predict()).

For further learning, explore concepts like transfer learning (using pre-trained models like VGG16, ResNet, or MobileNet), more advanced optimizers, and different regularization techniques. The official TensorFlow and Keras documentation are excellent resources: tensorflow.org/tutorials.