Scikit Learn - Estimator API

Scikit-Learn - The Estimator API

This chapter introduces the core Application Programming Interface (API) in Scikit-Learn: the Estimator API. Understanding this API is fundamental to using Scikit-Learn effectively.

What is an Estimator?

In Scikit-Learn, an estimator is an object that learns from data. It can be any algorithm used for classification, regression, clustering, dimensionality reduction, or even a transformer for feature extraction or preprocessing. The Estimator API provides a consistent interface across this wide range of machine learning applications, making it easy to switch between different models.

All estimator objects expose a fit method for learning from data. This method typically takes the training data (and labels/targets for supervised learning) as input:

estimator.fit(data, targets)  # For supervised learning
estimator.fit(data)            # For unsupervised learning

Estimator parameters, often called hyperparameters, are set when an estimator instance is created. These parameters can be accessed and modified directly as attributes of the estimator object:

from sklearn.linear_model import LogisticRegression
# Instantiate the estimator with desired hyperparameters
estimator = LogisticRegression(C=1.0, solver='liblinear')
print(estimator.C)  # Accessing a hyperparameter

After the fit method is called and the estimator has learned from the data, the learned parameters (e.g., model coefficients) are stored as attributes on the estimator object. By convention, these learned attributes always end with an underscore (_):

# Assuming estimator has been fitted:
# print(estimator.coef_)  # Accessing a learned parameter (e.g., coefficients in LinearRegression)

Common Uses and Capabilities of Estimators

Estimators serve various purposes in a machine learning workflow:

Model Estimation and Prediction/Transformation

The primary role of an estimator is to learn a model from data. The characteristics of this learned model are determined by:

• The hyperparameters provided during its instantiation.
• The global random state (e.g., numpy.random), if the estimator’s random_state parameter is set to None and the algorithm involves randomness.
• The data passed to the most recent call to fit, fit_transform, or fit_predict.
• For estimators supporting incremental learning, the data passed in a sequence of calls to partial_fit.

Once fitted, supervised estimators typically have a predict() method for making predictions on new data. Unsupervised estimators might have predict() (for clustering), transform() (for dimensionality reduction or feature engineering), or fit_predict()/fit_transform() methods.

Data Transformation

Transformer estimators (e.g., StandardScaler, PCA, PolynomialFeatures) modify or reshape data. They implement fit() to learn necessary parameters from the data (like mean and standard deviation for StandardScaler) and transform() to apply the transformation. A convenience method fit_transform() first fits and then transforms the data.

Outlier Detection

Some estimators are designed to identify outliers or novelties. They typically use methods like fit() to learn the normal data distribution, and predict() or fit_predict() to classify samples as inliers or outliers.

Guiding Principles of the Estimator API

The Scikit-Learn API design is guided by several core principles to ensure usability and consistency:

Consistency

All objects (estimators, transformers, etc.) share a common interface composed of a limited set of methods (e.g., fit, predict, transform). Documentation and naming conventions are also consistent.

Limited Object Hierarchy & Standard Data Formats

Scikit-Learn follows these conventions:

• Algorithms are represented by Python classes.
• Datasets are primarily expected as NumPy arrays, Pandas DataFrames, or SciPy sparse matrices.
• Hyperparameters are passed as arguments to the constructor, and their names are standard Python strings.

Composition

Many complex machine learning workflows can be constructed by combining simpler building blocks. Scikit-Learn’s Pipeline object is a prime example, allowing chaining of multiple estimators.

Sensible Defaults

Estimators come with sensible default values for their hyperparameters, allowing users to get reasonable baseline results without extensive tuning.

Inspection

All hyperparameter values specified during instantiation and all learned parameters (ending with _) are exposed as public attributes, allowing for easy inspection of the estimator’s state.

Steps in Using an Estimator

The typical workflow for using a Scikit-Learn estimator involves these steps:

Step 1: Choose a Model Class

Import the appropriate estimator class from Scikit-Learn. For example, from sklearn.linear_model import LinearRegression.

Step 2: Instantiate the Estimator and Choose Hyperparameters

Create an instance of the chosen class, setting any desired hyperparameters. For example, model = LinearRegression(fit_intercept=True).

Step 3: Prepare the Data

Arrange your data into a features matrix (commonly X) and, for supervised learning, a target vector (y). X is typically a 2D array (samples, features), and y is a 1D array (samples).

Step 4: Fit the Model to Data

Call the fit() method of the model instance, passing your training data. For example, model.fit(X_train, y_train).

Step 5: Apply the Model

For supervised learning, use the predict() method to make predictions on new, unseen data. For example, predictions = model.predict(X_test). For unsupervised learning, you might use transform() or predict() depending on the task.

Supervised Learning Example: Linear Regression

Let’s illustrate these steps with a simple linear regression example, fitting a line to (x, y) data.

First, we’ll generate some sample data. We’ll also show how to load a dataset like Iris for context, though our linear regression will use synthetic data.

Data Preparation (Illustrative Iris Load)

import seaborn as sns
import pandas as pd
# Load Iris dataset using seaborn (returns a Pandas DataFrame)
# iris_df = sns.load_dataset('iris')
# X_iris = iris_df.drop('species', axis=1)
# y_iris = iris_df['species']
# print(f"Iris features shape: {X_iris.shape}")
# print(f"Iris target shape: {y_iris.shape}")

Output of Iris load (if uncommented):

Iris features shape: (150, 4)
Iris target shape: (150,)

Linear Regression Example

Now, let’s proceed with the linear regression steps using synthetic data:

# For plotting in Jupyter notebooks, you might use: %matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# Generate synthetic data
rng = np.random.RandomState(35)
x = 10 * rng.rand(40)
y = 2 * x - 1 + rng.randn(40) # y = 2x - 1 + noise

# Plot the raw data
# plt.scatter(x, y)
# plt.title('Synthetic Data for Linear Regression')
# plt.xlabel('x')
# plt.ylabel('y')
# plt.show()

The plot would show a scatter of points generally following an upward linear trend.

1. Choose a class of model

We’ll use LinearRegression:

from sklearn.linear_model import LinearRegression

2. Choose model hyperparameters

We’ll instantiate LinearRegression, keeping fit_intercept=True (which is the default). The normalize parameter is deprecated; feature scaling should be done with a preprocessor like StandardScaler if needed.

model = LinearRegression(fit_intercept=True)
print(model)

Output:

LinearRegression()
# (Older versions might show: LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False))

3. Arrange the data

Our target y is already a 1D array. Our feature x is also 1D, but Scikit-Learn expects X (features matrix) to be 2D [n_samples, n_features]. We reshape x:

X = x[:, np.newaxis] # Reshape x to be a 2D column vector
print(f"Shape of X: {X.shape}")

Output:

Shape of X: (40, 1)

4. Model fitting

Fit the model to our data (X, y):

model.fit(X, y)

After fitting, the model has learned parameters, which are stored with a trailing underscore. For LinearRegression:

print(f"Learned coefficient (slope): {model.coef_}")
print(f"Learned intercept: {model.intercept_}")

Output (values will be close to 2 and -1, respectively):

Learned coefficient (slope): [1.99839352]
Learned intercept: -0.9895459457775022

5. Applying the model to new data (Prediction)

We can now predict y values for new x values:

xfit = np.linspace(-1, 11, 50) # Generate new x values for prediction
Xfit = xfit[:, np.newaxis]      # Reshape for Scikit-Learn
yfit = model.predict(Xfit)      # Predict y values

# Plot the original data and the fitted line
# plt.scatter(x, y, label='Original data')
# plt.plot(xfit, yfit, color='red', label='Fitted line')
# plt.title('Linear Regression Fit')
# plt.xlabel('x')
# plt.ylabel('y')
# plt.legend()
# plt.show()

The plot would show the original scatter points along with a red line representing the linear model’s fit, passing through the data.

Complete Executable Example (Linear Regression)

import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression

# Generate synthetic data
rng = np.random.RandomState(35)
x = 10 * rng.rand(40)
y = 2 * x - 1 + rng.randn(40)

# 1. Choose model & 2. Instantiate with hyperparameters
model = LinearRegression(fit_intercept=True)

# 3. Arrange data
X = x[:, np.newaxis]

# 4. Fit model
model.fit(X, y)
print(f"Slope: {model.coef_}, Intercept: {model.intercept_}")

# 5. Predict on new data
xfit = np.linspace(-1, 11, 50)
Xfit = xfit[:, np.newaxis]
yfit = model.predict(Xfit)

# Visualization
# plt.scatter(x, y)
# plt.plot(xfit, yfit, color='red')
# plt.title('Linear Regression Example')
# plt.show()

Unsupervised Learning Example: Principal Component Analysis (PCA)

Let’s use PCA to reduce the dimensionality of the Iris dataset, which is an unsupervised learning task.

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris # Using scikit-learn's direct loader

# Load Iris dataset
iris = load_iris()
X_iris = iris.data
y_iris_labels = iris.target # Numerical labels
species_names = iris.target_names

print(f"Original Iris data shape: {X_iris.shape}")

Output:

Original Iris data shape: (150, 4)

1. Choose a class of model

# Already imported: from sklearn.decomposition import PCA

2. Choose model hyperparameters

We’ll reduce to 2 components for visualization:

model_pca = PCA(n_components=2, random_state=42) # random_state for SVD solver if applicable
print(model_pca)

Output (may vary slightly based on version):

PCA(n_components=2, random_state=42)

3. Arrange data

X_iris is already in the correct [n_samples, n_features] format. No target y is needed for unsupervised PCA fitting.

4. Model fitting

model_pca.fit(X_iris)

Learned parameters include components_ and explained_variance_ratio_.

print(f"Explained variance ratio by 2 components: {model_pca.explained_variance_ratio_}")

Output (example):

Explained variance ratio by 2 components: [0.92461872 0.05306648]

5. Transform the data to two dimensions

Use the transform method:

X_2D = model_pca.transform(X_iris)
print(f"Transformed data shape: {X_2D.shape}")

Output:

Transformed data shape: (150, 2)

Now, we can visualize the 2D data, colored by species:

# Create a DataFrame for easier plotting with Seaborn
iris_2D_df = pd.DataFrame(data=X_2D, columns=['PCA1', 'PCA2'])
iris_2D_df['species'] = [species_names[label] for label in y_iris_labels]

# Plot using Seaborn
# sns.lmplot(x="PCA1", y="PCA2", hue='species', data=iris_2D_df, fit_reg=False, height=6)
# plt.title('Iris Dataset after PCA (2 Components)')
# plt.show()

The plot would show the Iris samples projected onto two principal components, with different colors for each species, typically showing good separation between the species.

Complete Executable Example (PCA)

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris

# Load Iris dataset
iris = load_iris()
X_iris = iris.data
y_iris_labels = iris.target
species_names = iris.target_names

# 1. & 2. Choose model and instantiate
model_pca = PCA(n_components=2, random_state=42)

# 3. Data is X_iris (no y for unsupervised fit)
# 4. Fit model
model_pca.fit(X_iris)

# 5. Transform data
X_2D = model_pca.transform(X_iris)

# Visualization
# iris_2D_df = pd.DataFrame(data=X_2D, columns=['PCA1', 'PCA2'])
# iris_2D_df['species'] = [species_names[label] for label in y_iris_labels]
# sns.lmplot(x="PCA1", y="PCA2", hue='species', data=iris_2D_df, fit_reg=False, height=6)
# plt.title('PCA of Iris Dataset')
# plt.show()

For further details on Scikit-Learn’s API design and usage, refer to the Scikit-Learn API documentation.