Model selection is an important part of machine learning, as it helps you pick the best model for your problem. Scikit-learn, for example, has built-in metrics like accuracy and precision. But these may not always be enough. Sometimes, you need a custom way to evaluate your model.
Custom scorers let you define your own way to measure model performance, helping you focus on the aspects that matter most for your problem.
In this article, we will show you how to create and use custom scorers. We will also explain how to use them with tools like GridSearchCV.
Defining a Custom Scorer
A custom scorer function must accept three arguments:
- estimator: The model (classifier or regressor) being evaluated
- X: The feature set (input data)
- y: The target variable (true labels)
The function should return a scalar value that reflects the model’s performance based on the custom metric.
For example, we can create a custom scorer using the F2-score. The F2-score focuses more on recall than precision.
from sklearn.metrics import fbeta_score
from sklearn.base import BaseEstimator
import numpy as np
def custom_f2_scorer(estimator, X, y):
# Predict the class labels
y_pred = estimator.predict(X)
# Compute F2-score with beta=2 (more weight on recall)
return fbeta_score(y, y_pred, beta=2)
Using the Custom Scorer with Model Selection
Now we have a custom scorer. We can use it with GridSearchCV or RandomizedSearchCV. These tools help tune model settings. They optimize the model using the custom scorer.
For this example, we’ll use GridSearchCV to search for the best hyperparameters of a RandomForestClassifier.
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# Step 1: Create a toy dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)
# Step 2: Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Step 3: Define a Random Forest model
rf_model = RandomForestClassifier(random_state=42)
# Step 4: Define the hyperparameters for GridSearchCV
param_grid = {'n_estimators': [100, 200], 'max_depth': [10, 20]}
# Step 5: Initialize GridSearchCV with the custom scorer
grid_search = GridSearchCV(estimator=rf_model,
param_grid=param_grid,
scoring=custom_f2_scorer,
cv=5)
# Step 6: Fit the model and perform the search
grid_search.fit(X_train, y_train)
# Step 7: Output the best hyperparameters and corresponding score
print(f"Best Parameters: {grid_search.best_params_}")
print(f"Best F2-Score: {grid_search.best_score_}")
Best Parameters: {'max_depth': 20, 'n_estimators': 100}
Best F2-Score: 0.8965661871921711
The result shows the best combination of hyperparameters for the RandomForestClassifier based on the F2-score.
Evaluating the Final Model
After performing the grid search, we can retrieve the best model and evaluate it on the test set using the custom scorer.
# Step 1: Get the best model from GridSearchCV
best_model = grid_search.best_estimator_
# Step 2: Evaluate the best model on the test set using the custom F2-score
final_f2_score = custom_f2_scorer(best_model, X_test, y_test)
print(f"Final F2-Score on Test Data: {final_f2_score}")
Final F2-Score on Test Data: 0.837696335078534
The output shows the performance of the best model based on the F2-score.
Advanced Custom Scorers
You can create more advanced custom scoring functions depending on your needs. Below are a few examples:
Cost-Sensitive Scoring
If your problem involves different misclassification costs (e.g., false positives are more costly than false negatives), you can create a custom scorer to penalize errors differently.
from sklearn.metrics import confusion_matrix
def cost_sensitive_scorer(estimator, X, y):
# Get predictions
y_pred = estimator.predict(X)
# Confusion matrix
tn, fp, fn, tp = confusion_matrix(y, y_pred).ravel()
# Penalize false positives more
cost = tn + 3 * fp + fn
return cost
Multi-Objective Scoring
Sometimes, you need to optimize multiple objectives. For example, you might balance precision and recall. You can also combine accuracy and speed in a custom metric.
from sklearn.metrics import precision_score, recall_score
def multi_objective_scorer(estimator, X, y):
# Get predictions
y_pred = estimator.predict(X)
# Calculate precision and recall
precision = precision_score(y, y_pred)
recall = recall_score(y, y_pred)
# Weighted average
return 0.7 * precision + 0.3 * recall
Conclusion
Custom scorers allow you to measure models based on what’s important for your problem. They help you focus on key things like recall or false positives. By using custom scorers with tools like GridSearchCV, you can easily find the best model. This makes sure the model fits your needs.
