Clustering Performance Evaluation

Scikit-learn - Clustering Performance Evaluation

Evaluating the performance of a clustering algorithm is not as straightforward as evaluating supervised learning models (like classifiers or regressors) because there are typically no ground truth labels for unsupervised tasks. However, Scikit-learn provides several metrics in the sklearn.metrics module to assess the quality of clustering results. These metrics can be broadly categorized into two types:

• Extrinsic Measures: Require ground truth labels. Used when true cluster assignments are known (e.g., for benchmarking or on datasets where labels exist but are not used during clustering). Examples: Adjusted Rand Index, Mutual Information scores.
• Intrinsic Measures: Do not require ground truth labels. Evaluate based on the inherent properties of the clustering, such as compactness (how close points are within a cluster) and separation (how distinct clusters are from each other). Example: Silhouette Coefficient.

Extrinsic Measures (Require Ground Truth Labels)

These metrics compare the algorithm’s predicted cluster assignments (labels_pred) against the true class labels (labels_true).

Adjusted Rand Index (ARI)

The Rand Index (RI) measures the similarity between two data clusterings by considering all pairs of samples and counting pairs that are assigned in the same or different clusters in both the predicted and true clusterings. The Adjusted Rand Index (ARI) is a corrected-for-chance version of RI.

Formula (conceptual): ARI = (RI - Expected_RI) / (max(RI) - Expected_RI).

Values range from -1 to 1. A score of 1 indicates perfect agreement, 0 indicates random labeling (independent of true labels), and negative values indicate worse than random. Provided by sklearn.metrics.adjusted_rand_score.

Example

from sklearn.metrics import adjusted_rand_score

labels_true = [0, 0, 0, 1, 1, 1]
labels_pred_good = [0, 0, 0, 1, 1, 1] # Perfect match
labels_pred_bad = [0, 1, 0, 1, 0, 1]  # Poor match
labels_pred_permuted = [1, 1, 1, 0, 0, 0] # Permuted perfect match

ari_good = adjusted_rand_score(labels_true, labels_pred_good)
ari_bad = adjusted_rand_score(labels_true, labels_pred_bad)
ari_permuted = adjusted_rand_score(labels_true, labels_pred_permuted)

print(f"ARI (Good): {ari_good:.4f}")
print(f"ARI (Bad): {ari_bad:.4f}")
print(f"ARI (Permuted Good): {ari_permuted:.4f}")

Output

ARI (Good): 1.0000
ARI (Bad): -0.0714
ARI (Permuted Good): 1.0000

Mutual Information Based Scores

Mutual Information (MI) measures the agreement between two assignments, ignoring permutations. Scikit-learn provides several MI-based scores:

• Normalized Mutual Information (NMI): sklearn.metrics.normalized_mutual_info_score. MI normalized by entropy. Ranges from 0 (no mutual information) to 1 (perfect correlation).
• Adjusted Mutual Information (AMI): sklearn.metrics.adjusted_mutual_info_score. MI adjusted for chance. Similar to ARI, higher is better, with 1 being perfect agreement.

Example

from sklearn.metrics import normalized_mutual_info_score, adjusted_mutual_info_score

labels_true = [0, 0, 1, 1, 1, 1]
labels_pred = [0, 0, 2, 2, 3, 3] # Different cluster labels but structure is somewhat preserved

nmi_score = normalized_mutual_info_score(labels_true, labels_pred)
ami_score = adjusted_mutual_info_score(labels_true, labels_pred)

print(f"Normalized Mutual Information (NMI): {nmi_score:.4f}")
print(f"Adjusted Mutual Information (AMI): {ami_score:.4f}")

Output

Normalized Mutual Information (NMI): 0.7612
Adjusted Mutual Information (AMI): 0.4444

Homogeneity, Completeness, and V-measure

These three metrics offer insights into different aspects of clustering quality:

• Homogeneity (sklearn.metrics.homogeneity_score): Each cluster contains only members of a single class. Score of 1 means all clusters are perfectly homogeneous.
• Completeness (sklearn.metrics.completeness_score): All members of a given class are assigned to the same cluster. Score of 1 means all data points of a given class fall into a single cluster.
• V-measure (sklearn.metrics.v_measure_score): The harmonic mean of homogeneity and completeness. Score of 1 indicates perfect homogeneity and completeness.

These scores range from 0 to 1, with 1 being optimal.

Example

from sklearn.metrics import homogeneity_score, completeness_score, v_measure_score

labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 0, 1, 2, 2] # Class 1 is split into two clusters

hom_score = homogeneity_score(labels_true, labels_pred)
com_score = completeness_score(labels_true, labels_pred)
vm_score = v_measure_score(labels_true, labels_pred)

print(f"Homogeneity: {hom_score:.4f}")
print(f"Completeness: {com_score:.4f}")
print(f"V-measure: {vm_score:.4f}")

Output

Homogeneity: 0.6667
Completeness: 1.0000
V-measure: 0.8000

Here, completeness is 1 because all true ‘0’s are in cluster ‘0’, and all true ‘1’s are accounted for (though split). Homogeneity is lower because cluster ‘1’ and ‘2’ (in pred) are not purely from a single true class if we consider the original classes.

Fowlkes-Mallows Score (FMI)

The Fowlkes-Mallows Score (sklearn.metrics.fowlkes_mallows_score) is defined as the geometric mean of pairwise precision and recall. It measures the similarity between the clusters obtained from the true labels and the predicted labels.

FMI = TP / sqrt((TP + FP) * (TP + FN))

Where TP is the number of true positives (pairs of points in the same cluster in both true and predicted). FP is false positives, FN is false negatives. Score ranges from 0 to 1, higher is better.

Example

from sklearn.metrics import fowlkes_mallows_score

labels_true = [0, 0, 1, 1, 1, 1]
labels_pred = [0, 0, 2, 2, 3, 3]

fmi = fowlkes_mallows_score(labels_true, labels_pred)
print(f"Fowlkes-Mallows Score: {fmi:.4f}")

Output

Fowlkes-Mallows Score: 0.6547

Intrinsic Measures (Do Not Require Ground Truth Labels)

Silhouette Coefficient

The Silhouette Coefficient (sklearn.metrics.silhouette_score) measures how similar an object is to its own cluster (cohesion) compared to other clusters (separation). It’s calculated for each sample.

For a sample i, let a(i) be the mean intra-cluster distance (average distance from i to all other points in its own cluster) and b(i) be the mean nearest-cluster distance (average distance from i to all points in the next nearest cluster).

Silhouette coefficient for sample i: s(i) = (b(i) - a(i)) / max(a(i), b(i))

The overall Silhouette Score for a set of samples is the mean of s(i) over all samples. Values range from -1 to 1:

• Near +1: The sample is far away from neighboring clusters (well-clustered).
• 0: The sample is on or very close to the decision boundary between two neighboring clusters.
• Near -1: The sample may have been assigned to the wrong cluster.

Example

from sklearn.metrics import silhouette_score
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
import numpy as np

# Load iris dataset
 X, y = load_iris(return_X_y=True)

# Scale data for K-Means
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply K-Means (example with 3 clusters for Iris)
kmeans_model = KMeans(n_clusters=3, random_state=42, n_init='auto').fit(X_scaled)
cluster_labels = kmeans_model.labels_

# Calculate Silhouette Score
# Requires the data X and the predicted labels
sil_score = silhouette_score(X_scaled, cluster_labels, metric='euclidean')
print(f"Silhouette Score for K-Means (k=3) on Iris: {sil_score:.4f}")

Output

Silhouette Score for K-Means (k=3) on Iris: 0.4599

The Silhouette Score is often used to help choose an optimal number of clusters (k) by trying different values of k and selecting the one that maximizes the score.

Other Intrinsic Measures

Scikit-learn also provides other intrinsic measures like:

• Calinski-Harabasz Index (sklearn.metrics.calinski_harabasz_score): Also known as the Variance Ratio Criterion. Higher values indicate better defined clusters.
• Davies-Bouldin Index (sklearn.metrics.davies_bouldin_score): Measures the average similarity ratio of each cluster with its most similar cluster. Lower values indicate better clustering (closer to 0).

Contingency Matrix

If ground truth class labels are known, a contingency matrix (sklearn.metrics.cluster.contingency_matrix) can be insightful. It shows the intersection cardinality for every true class / predicted cluster pair. This is similar to a confusion matrix but for clustering contexts.

Example

from sklearn.metrics.cluster import contingency_matrix
import numpy as np

labels_true = np.array(["cat", "cat", "cat", "dog", "dog", "bird"])
labels_pred = np.array([0, 0, 1, 1, 2, 2]) # 0: cluster_A, 1: cluster_B, 2: cluster_C

# Assuming unique true labels are ['bird', 'cat', 'dog'] and pred labels [0, 1, 2]
# (sorted alphabetically/numerically by default)
cont_matrix = contingency_matrix(labels_true, labels_pred)
print("Contingency Matrix (rows=true, cols=pred):")
print(cont_matrix)

# To understand rows/columns better:
# unique_true_labels = np.unique(labels_true)
# unique_pred_labels = np.unique(labels_pred)
# print(f"True labels (rows): {unique_true_labels}")
# print(f"Predicted labels (cols): {unique_pred_labels}")

Output

Contingency Matrix (rows=true, cols=pred):
[[0 0 1]  <-- True 'bird'
 [2 1 0]  <-- True 'cat'
 [0 1 1]] <-- True 'dog'

For example, the first row [0 0 1] means: for true label ‘bird’ (first unique true label), 0 samples were put in predicted cluster 0, 0 in cluster 1, and 1 in cluster 2. The second row [2 1 0] means: for true label ‘cat’, 2 samples were put in cluster 0, 1 in cluster 1, and 0 in cluster 2.

Refer to the Scikit-learn documentation for more details on these metrics and their usage: https://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation