Scikit Learn - Modelling Process

Scikit-learn - Modeling Process

This chapter covers the typical modeling process in Scikit-learn. We’ll explore each step, starting with dataset loading and preparation, then moving to model training, evaluation, and persistence. Understanding this workflow is crucial for effectively applying machine learning techniques.

Dataset Loading

A dataset is a collection of data, typically organized into features and a target (or response). Key components include:

Features (X): These are the input variables or attributes used by the model to make predictions. Also known as predictors or independent variables.

• Feature Matrix: A 2D array-like structure (e.g., NumPy array or Pandas DataFrame) where rows represent samples and columns represent features. Shape: [n_samples, n_features].
• Feature Names: A list of names corresponding to each column in the feature matrix, useful for understanding the data.

Response (y): This is the output variable that the model aims to predict. Also known as the target, label, or dependent variable.

• Response Vector: A 1D array-like structure representing the target values for each sample. Shape: [n_samples].
• Target Names: For classification tasks, this represents the names of the different classes the response vector can take.

Scikit-learn includes several built-in datasets for practice, such as the Iris and Digits datasets for classification, and the California Housing dataset for regression. These are useful for quickly testing algorithms. You can find more under the sklearn.datasets module.

Example: Loading the Iris Dataset

Here’s how to load the classic Iris dataset. We use return_X_y=True for convenience to directly get the features (X) and target (y).

from sklearn.datasets import load_iris

# Load the dataset
iris = load_iris()
X = iris.data
y = iris.target

# Alternatively, load features and target directly
# X, y = load_iris(return_X_y=True)

feature_names = iris.feature_names
target_names = iris.target_names

print(f"Feature names: {feature_names}")
print(f"Target names: {target_names}")
print(f"\nFirst 5 rows of X (Features):\n{X[:5]}")
print(f"\nFirst 5 labels of y (Target):\n{y[:5]}")
print(f"\nShape of X: {X.shape}")
print(f"Shape of y: {y.shape}")

Output

Feature names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
Target names: ['setosa' 'versicolor' 'virginica']

First 5 rows of X (Features):
[[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]
 [4.6 3.1 1.5 0.2]
 [5.  3.6 1.4 0.2]]

First 5 labels of y (Target):
[0 0 0 0 0]

Shape of X: (150, 4)
Shape of y: (150,)

Splitting the Dataset

To evaluate a model’s performance on unseen data, it’s essential to split the dataset into a training set and a testing set. The model learns from the training set, and its performance is then assessed on the testing set. This helps prevent overfitting, where a model performs well on training data but poorly on new data.

Example: Train-Test Split

The train_test_split function from sklearn.model_selection is commonly used. Here, we split the Iris dataset, allocating 30% for testing and 70% for training.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# Load data
X, y = load_iris(return_X_y=True)

# Split data: 70% training, 30% testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Shape of X_train: {X_train.shape}")
print(f"Shape of X_test: {X_test.shape}")
print(f"Shape of y_train: {y_train.shape}")
print(f"Shape of y_test: {y_test.shape}")

Output

Shape of X_train: (105, 4)
Shape of X_test: (45, 4)
Shape of y_train: (105,)
Shape of y_test: (45,)

Key parameters for train_test_split:

• X, y: The feature matrix and response vector to be split.
• test_size: Proportional size of the test set (e.g., 0.3 for 30%). train_size can be used alternatively.
• random_state: A seed for the random number generator. Setting this ensures that the split is the same every time you run the code, which is crucial for reproducibility. Any integer value can be used.
• stratify=y: (Recommended for classification) Ensures that the proportion of target classes in the train and test sets is similar to the original dataset. This is important for imbalanced datasets.

Train the Model

After preparing the data, the next step is to train a machine learning model. Scikit-learn offers a wide array of algorithms, all following a consistent API: fit() to train, predict() to make predictions, and score() to evaluate.

Example: Training a K-Nearest Neighbors (KNN) Classifier

We’ll use the K-Nearest Neighbors (KNN) algorithm, a simple yet effective classifier. This example demonstrates the basic training and prediction workflow.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
import numpy as np

# Load and split data (as shown previously)
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

# Initialize the KNN classifier
# n_neighbors specifies the 'K' in KNN
knn_classifier = KNeighborsClassifier(n_neighbors=3)

# Train the model using the training data
knn_classifier.fit(X_train, y_train)

# Make predictions on the test data
y_pred = knn_classifier.predict(X_test)

# Evaluate the model's accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy on test set: {accuracy:.4f}")

# Example of predicting on new, unseen samples
# Note: New samples should have the same number of features as the training data (4 in this case)
sample_new_data = np.array([
    [5.0, 3.1, 1.6, 0.2],  # Expected: setosa (0)
    [6.5, 3.0, 5.2, 2.0]   # Expected: virginica (2)
])
predicted_species_indices = knn_classifier.predict(sample_new_data)
iris = load_iris() # Reload for target_names
predicted_species_names = iris.target_names[predicted_species_indices]
print(f"Predictions for new samples: {predicted_species_names}")

Output

Accuracy on test set: 0.9778
Predictions for new samples: ['setosa' 'virginica']

For more details on KNN, refer to its dedicated chapter. Choosing the right model and tuning its parameters (like n_neighbors for KNN) often involves techniques like cross-validation, discussed later.

Model Persistence

Training a model can be time-consuming. Once trained, you’ll often want to save (persist) it for later use without retraining. The joblib library (part of the SciPy ecosystem and recommended by Scikit-learn) is excellent for this, especially for objects containing large NumPy arrays.

Let’s save the knn_classifier we trained earlier:

import joblib

# Assume knn_classifier is our trained model from the previous step
# Save the model to a file
model_filename = 'iris_knn_classifier.joblib'
joblib.dump(knn_classifier, model_filename)
print(f"Model saved to {model_filename}")

# ... Later, in another script or session ...

# Load the model from the file
loaded_knn_model = joblib.load(model_filename)
print(f"Model loaded from {model_filename}")

# You can now use the loaded_knn_model to make predictions
# Example: y_pred_loaded = loaded_knn_model.predict(X_test)
# print(f"Accuracy of loaded model: {accuracy_score(y_test, y_pred_loaded):.4f}")

This saves the entire model object, including its learned parameters. Ensure the environment where you load the model has the same versions of relevant libraries (like Scikit-learn) for compatibility.

Data Preprocessing

Raw data is often not suitable for direct input into machine learning algorithms. Preprocessing transforms raw data into a more usable format. Scikit-learn’s sklearn.preprocessing module offers various tools. Common preprocessing steps include scaling, normalization, and encoding categorical features.

Why preprocess? Many algorithms (e.g., SVMs, KNN, PCA, neural networks) are sensitive to feature scales. Features with larger values might dominate the learning process. Preprocessing helps to standardize the data.

Binarization

Binarization converts numerical features into boolean values (0 or 1) based on a threshold. This can be useful for creating binary features from continuous ones.

import numpy as np
from sklearn.preprocessing import Binarizer

input_data = np.array([
    [2.1, -1.9, 5.5],
    [-1.5, 2.4, 3.5],
    [0.5, -7.9, 5.6],
    [5.9, 2.3, -5.8]
])

# Values > 0.5 become 1, others become 0
binarizer = Binarizer(threshold=0.5)
data_binarized = binarizer.fit_transform(input_data)
print(f"\nBinarized data (threshold=0.5):\n{data_binarized}")

Output

Binarized data (threshold=0.5):
[[1. 0. 1.]
 [0. 1. 1.]
 [0. 0. 1.]
 [1. 1. 0.]]

Mean Removal (Standardization)

Mean removal, more commonly known as standardization or Z-score normalization, transforms features to have zero mean and unit variance. This is achieved by subtracting the mean and dividing by the standard deviation for each feature.

import numpy as np
from sklearn.preprocessing import StandardScaler

input_data = np.array([
    [2.1, -1.9, 5.5],
    [-1.5, 2.4, 3.5],
    [0.5, -7.9, 5.6],
    [5.9, 2.3, -5.8]
], dtype=np.float64)

print(f"Original Mean: {input_data.mean(axis=0)}")
print(f"Original Std Dev: {input_data.std(axis=0)}")

scaler = StandardScaler()
data_scaled = scaler.fit_transform(input_data)

print(f"\nScaled Data (StandardScaler):\n{data_scaled}")
print(f"Mean of scaled data: {data_scaled.mean(axis=0)}")
print(f"Std Dev of scaled data: {data_scaled.std(axis=0)}")

Output

Original Mean: [ 1.75  -1.275  2.2  ]
Original Std Dev: [2.71431391 4.20022321 4.69414529]

Scaled Data (StandardScaler):
[[ 0.12896806  0.6318175   0.70299018]
 [-1.19734452  0.87470814  0.2769086 ]
 [-0.46049308 -1.57685687  0.72410327]
 [ 1.52886954  0.07033123 -1.7039976 ]]
Mean of scaled data: [2.22044605e-17 0.00000000e+00 0.00000000e+00]
Std Dev of scaled data: [1. 1. 1.]

Note: It’s crucial to fit the scaler on the training data only and then use transform on both training and testing data to prevent data leakage from the test set.

Scaling (Min-Max Scaling)

Min-Max scaling (or normalization) rescales features to a specific range, typically [0, 1] or [-1, 1]. This is done by subtracting the minimum value and dividing by the range (max - min) for each feature.

import numpy as np
from sklearn.preprocessing import MinMaxScaler

input_data = np.array([
    [2.1, -1.9, 5.5],
    [-1.5, 2.4, 3.5],
    [0.5, -7.9, 5.6],
    [5.9, 2.3, -5.8]
], dtype=np.float64)

# Scale features to the range [0, 1]
min_max_scaler = MinMaxScaler(feature_range=(0, 1))
data_scaled_minmax = min_max_scaler.fit_transform(input_data)
print(f"\nMin-Max Scaled Data (range [0, 1]):\n{data_scaled_minmax}")

Output

Min-Max Scaled Data (range [0, 1]):
[[0.48648649 0.58252427 0.99122807]
 [0.         1.         0.81578947]
 [0.27027027 0.         1.        ]
 [1.         0.99029126 0.        ]]

Normalization

Normalization in Scikit-learn (using sklearn.preprocessing.Normalizer) rescales each sample (row) independently to have unit norm (L1 or L2). This is different from scaling features (columns) like StandardScaler or MinMaxScaler do. Normalization is often used for text classification or clustering.

L1 Normalization (Least Absolute Deviations)

Rescales each sample so that the sum of the absolute values of its features is 1.

import numpy as np
from sklearn.preprocessing import Normalizer

input_data = np.array([
    [2.1, -1.9, 5.5],
    [-1.5, 2.4, 3.5],
    [0.5, -7.9, 5.6],
    [5.9, 2.3, -5.8]
], dtype=np.float64)

normalizer_l1 = Normalizer(norm='l1')
data_normalized_l1 = normalizer_l1.fit_transform(input_data)
print(f"\nL1 Normalized Data:\n{data_normalized_l1}")
print(f"Sum of absolute values per row (L1 norm): {np.sum(np.abs(data_normalized_l1), axis=1)}")

Output

L1 Normalized Data:
[[ 0.22105263 -0.2         0.57894737]
 [-0.2027027   0.32432432  0.47297297]
 [ 0.03571429 -0.56428571  0.4       ]
 [ 0.42142857  0.16428571 -0.41428571]]
Sum of absolute values per row (L1 norm): [1. 1. 1. 1.]

L2 Normalization (Least Squares)

Rescales each sample so that the sum of the squares of its features (Euclidean norm) is 1.

import numpy as np
from sklearn.preprocessing import Normalizer

input_data = np.array([
    [2.1, -1.9, 5.5],
    [-1.5, 2.4, 3.5],
    [0.5, -7.9, 5.6],
    [5.9, 2.3, -5.8]
], dtype=np.float64)

normalizer_l2 = Normalizer(norm='l2')
data_normalized_l2 = normalizer_l2.fit_transform(input_data)
print(f"\nL2 Normalized Data:\n{data_normalized_l2}")
print(f"Sum of squares per row (L2 norm squared): {np.sum(data_normalized_l2**2, axis=1)}")

Output

L2 Normalized Data:
[[ 0.33946114 -0.30713151  0.88906489]
 [-0.33325106  0.53320169  0.7775858 ]
 [ 0.05156558 -0.81473612  0.57753446]
 [ 0.68706914  0.26784051 -0.6754239 ]]
Sum of squares per row (L2 norm squared): [1. 1. 1. 1.]

Pipelines for Streamlining Workflows

Scikit-learn’s Pipeline object allows you to chain multiple processing steps (like scaling and PCA) with a final estimator (like a classifier). This is highly recommended for:

• Convenience: Combines multiple steps into one object.
• Preventing Data Leakage: Ensures transformations are fit only on the training data during cross-validation.
• Reproducibility: Encapsulates the entire preprocessing and modeling workflow.

Example of a simple pipeline:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load and split data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create a pipeline: StandardScaler then SVC
pipe = Pipeline([
    ('scaler', StandardScaler()),  # Step 1: Scale data
    ('svc', SVC())                # Step 2: Support Vector Classifier
])

# Fit the pipeline (applies scaler's fit_transform and then SVC's fit)
pipe.fit(X_train, y_train)

# Make predictions (applies scaler's transform and then SVC's predict)
y_pred_pipeline = pipe.predict(X_test)

accuracy_pipeline = accuracy_score(y_test, y_pred_pipeline)
print(f"Accuracy with Pipeline: {accuracy_pipeline:.4f}")

Output

Accuracy with Pipeline: 1.0000

Pipelines are a powerful tool for building robust and maintainable machine learning models. You can learn more in the Scikit-learn documentation on sklearn.pipeline.Pipeline.