Scikit Learn - Boosting Methods

Scikit-learn - Boosting Methods

Boosting methods are powerful ensemble learning techniques that build a strong model by sequentially training multiple weak learners (typically shallow decision trees). Each subsequent learner focuses more on the instances that previous learners misclassified, thus incrementally improving the overall model. The sklearn.ensemble module in Scikit-learn provides several popular boosting algorithms.

AdaBoost (Adaptive Boosting)

AdaBoost is one of the earliest and most successful boosting algorithms. It works by fitting a sequence of weak learners on repeatedly modified versions of the data. The predictions from all of them are then combined through a weighted majority vote (or sum) to produce the final prediction. The data modifications at each boosting iteration involve applying weights to each of the training samples. Initially, all weights are set equally, but on each step, the weights of incorrectly classified instances are increased so that subsequent weak learners focus more on these difficult cases.

AdaBoost for Classification

Scikit-learn provides sklearn.ensemble.AdaBoostClassifier. The key parameter is base_estimator (or estimator in newer versions), which specifies the weak learner. If None, it defaults to DecisionTreeClassifier(max_depth=1) (a decision stump). Another important parameter is n_estimators, the number of weak learners to train.

Implementation Example: AdaBoostClassifier

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate a synthetic classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5,
                           n_redundant=0, random_state=42, shuffle=False)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize AdaBoostClassifier with default base estimator (DecisionTreeClassifier(max_depth=1))
adb_clf = AdaBoostClassifier(n_estimators=100, random_state=42, algorithm='SAMME.R') # SAMME.R for real outputs
adb_clf.fit(X_train, y_train)

y_pred = adb_clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"AdaBoostClassifier Accuracy: {accuracy:.4f}")

# Example with a different base estimator
# custom_base_estimator = DecisionTreeClassifier(max_depth=2)
# adb_clf_custom = AdaBoostClassifier(estimator=custom_base_estimator, n_estimators=50, random_state=42)
# adb_clf_custom.fit(X_train, y_train)
# ...

Output

AdaBoostClassifier Accuracy: 0.8600

Example: AdaBoostClassifier on Breast Cancer Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import AdaBoostClassifier
import numpy as np

# Load dataset
data = load_breast_cancer()
X, y = data.data, data.target

# Initialize AdaBoostClassifier
adb_clf_cancer = AdaBoostClassifier(n_estimators=100, random_state=42)

# Evaluate using cross-validation
# (No explicit train/test split here, cross_val_score handles it)
# Note: For a robust evaluation, scaling might be considered if base estimator is sensitive
scores = cross_val_score(adb_clf_cancer, X, y, cv=5) # 5-fold cross-validation
print(f"AdaBoost on Breast Cancer - Cross-val scores: {scores}")
print(f"AdaBoost on Breast Cancer - Mean accuracy: {np.mean(scores):.4f}")

Output (may vary slightly)

AdaBoost on Breast Cancer - Cross-val scores: [0.97368421 0.97368421 0.94736842 0.99122807 0.92920354]
AdaBoost on Breast Cancer - Mean accuracy: 0.9630

AdaBoost for Regression

Scikit-learn provides sklearn.ensemble.AdaBoostRegressor. It works similarly to the classifier, but the weak learners predict continuous values, and their combination aims to minimize regression loss (e.g., linear, square, exponential).

Implementation Example: AdaBoostRegressor

from sklearn.ensemble import AdaBoostRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate a synthetic regression dataset
X, y = make_regression(n_samples=1000, n_features=10, n_informative=5,
                       random_state=42, shuffle=False)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize AdaBoostRegressor
adb_reg = AdaBoostRegressor(n_estimators=100, random_state=42, loss='square')
adb_reg.fit(X_train, y_train)

y_pred_reg = adb_reg.predict(X_test)
mse = mean_squared_error(y_test, y_pred_reg)
print(f"AdaBoostRegressor MSE: {mse:.4f}")
print(f"AdaBoostRegressor RMSE: {np.sqrt(mse):.4f}")

Output

AdaBoostRegressor MSE: 6234.1421
AdaBoostRegressor RMSE: 78.9566

Gradient Tree Boosting (GBT / GBRT)

Gradient Tree Boosting (also known as Gradient Boosted Regression Trees or GBRT) is a more general boosting algorithm. It builds an additive model in a stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage, a regression tree is fit on the negative gradient (residual errors) of the given loss function. GBT is highly effective and widely used for both classification and regression tasks.

Key parameters include n_estimators (number of boosting stages/trees), learning_rate (shrinks the contribution of each tree, helps prevent overfitting), max_depth (maximum depth of individual regression estimators), and loss (loss function to be optimized).

Gradient Boosting for Classification

Scikit-learn provides sklearn.ensemble.GradientBoostingClassifier. For binary classification, ‘deviance’ (logistic regression loss) or ‘exponential’ (similar to AdaBoost) are common loss functions. For multi-class, ‘deviance’ (multi-class logistic loss) is used.

Implementation Example: GradientBoostingClassifier

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_hastie_10_2 # A binary classification problem
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate dataset (make_hastie_10_2 is often used for boosting examples)
X, y = make_hastie_10_2(random_state=0)
# Convert y labels from {-1, 1} to {0, 1} if necessary for some metrics/loss functions
# y = (y + 1) // 2

X_train, X_test = X[:5000], X[5000:]
y_train, y_test = y[:5000], y[5000:]

gbc_clf = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
                                     max_depth=3, random_state=0)
gbc_clf.fit(X_train, y_train)

y_pred_gbc = gbc_clf.predict(X_test)
accuracy_gbc = accuracy_score(y_test, y_pred_gbc)
# For make_hastie_10_2, score() directly works as it expects {-1, 1}
# For other datasets, ensure y is 0/1 for binary 'deviance' or check predict_proba
# print(f"GradientBoostingClassifier Score (internal): {gbc_clf.score(X_test, y_test):.4f}")
print(f"GradientBoostingClassifier Accuracy: {accuracy_gbc:.4f}")

Output (make_hastie_10_2 target is -1, 1; accuracy_score handles this)

GradientBoostingClassifier Accuracy: 0.9143

Example: GradientBoostingClassifier on Breast Cancer Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

gbc_cancer = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
                                        max_depth=3, random_state=42)
scores_gbc = cross_val_score(gbc_cancer, X, y, cv=5)
print(f"GradientBoosting on Breast Cancer - Cross-val scores: {scores_gbc}")
print(f"GradientBoosting on Breast Cancer - Mean accuracy: {np.mean(scores_gbc):.4f}")

Output (may vary slightly)

GradientBoosting on Breast Cancer - Cross-val scores: [0.96491228 0.96491228 0.95614035 0.98245614 0.94690265]
GradientBoosting on Breast Cancer - Mean accuracy: 0.9631

Gradient Boosting for Regression

Scikit-learn provides sklearn.ensemble.GradientBoostingRegressor. Common loss functions include ‘ls’ (least squares), ‘lad’ (least absolute deviation), ‘huber’ (a combination of ls and lad, robust to outliers), and ‘quantile’.

Implementation Example: GradientBoostingRegressor

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_friedman1 # A regression problem with non-linear structure
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import numpy as np

X, y = make_friedman1(n_samples=2000, random_state=0, noise=1.0)
X_train, X_test = X[:1000], X[1000:]
y_train, y_test = y[:1000], y[1000:]

gbr_reg = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,
                                    max_depth=3, random_state=0, loss='squared') # 'squared' for squared error
gbr_reg.fit(X_train, y_train)

y_pred_gbr = gbr_reg.predict(X_test)
mse_gbr = mean_squared_error(y_test, y_pred_gbr)
print(f"GradientBoostingRegressor MSE: {mse_gbr:.4f}")
print(f"GradientBoostingRegressor RMSE: {np.sqrt(mse_gbr):.4f}")

Output

GradientBoostingRegressor MSE: 5.0120
GradientBoostingRegressor RMSE: 2.2387

Histogram-Based Gradient Boosting (HistGradientBoosting)

For larger datasets, Scikit-learn offers HistGradientBoostingClassifier and HistGradientBoostingRegressor. These are inspired by LightGBM and are significantly faster than traditional GradientBoostingClassifier/Regressor. They use a technique called histogram-based binning of continuous features, which speeds up tree construction and reduces memory usage. They also have built-in support for handling missing values.

Example: HistGradientBoostingClassifier

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=10000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

hgb_clf = HistGradientBoostingClassifier(max_iter=100, learning_rate=0.1, random_state=42)
hgb_clf.fit(X_train, y_train)

y_pred_hgb = hgb_clf.predict(X_test)
accuracy_hgb = accuracy_score(y_test, y_pred_hgb)
print(f"HistGradientBoostingClassifier Accuracy: {accuracy_hgb:.4f}")

Output (Accuracy may vary slightly depending on scikit-learn version)

HistGradientBoostingClassifier Accuracy: 0.8987

Boosting methods, especially GBT and its variants like XGBoost, LightGBM, and CatBoost (which are separate libraries but popular in the ecosystem), are often top performers in machine learning competitions and real-world applications. Scikit-learn’s implementations provide a solid foundation and are well-integrated within its API.

For further exploration, check the Scikit-learn User Guide: https://scikit-learn.org/stable/modules/ensemble.html#gradient-boosting