PyCaret is a Python library that automates the development of a machine learning (ML) model. One of PyCaret’s most interesting features is the ability to define custom metrics for evaluating ML models like classifiers, regressors, ensembles, etc. This is useful for instance to measure business goals or requirements that common metrics like accuracy, F1-score, or RMSE cannot finely reflect.

This article illustrates defining and using custom metrics through a code example.

Setting Up The Environment

We first import the pandas library to handle our data in a DataFrame structure, and some PyCaret functions we will use, concretely from its classification package.

import pandas as pd
from pycaret.classification import setup, add_metric, compare_models, create_model

Some explanation of the imported modules:

• setup: initializes a PyCaret experimental environment for data preparation and ML pipeline configuration
• add_metric: adds a custom evaluation metric to the set of default metrics used in an ML model evaluation
• compare_models: compares and ranks multiple ML models based on performance
• create_model: creates and automatically trains an ML model

Next, we create a very simple dataset consisting of two numerical predictor features plus a target column representing class labels for binary classification, i.e. a supervised task where the aim is to predict whether an instance belongs (1) or not (0) to a class.

data = pd.DataFrame({
    'feature1': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    'feature2': [2, 4, 6, 8, 10, 1, 3, 5, 7, 9],
    'target_column': [0, 0, 0, 1, 1, 1, 0, 1, 1, 0]
})

Creating the Custom Metric Function

The following function defines a custom business-oriented metric that penalizes wrong predictions differently, depending on real business costs. Concretely, we assume that false negative (FN) costs are higher than false positive (FP) costs. A custom score that reflects the overall model’s accuracy in line with business needs, is then computed.

def business_cost_metric(y_true, y_pred, **kwargs):
    from sklearn.metrics import confusion_matrix
    
    cm = confusion_matrix(y_true, y_pred)
    
    # Business-specific misclassification costs
    fp_cost = 10  # Cost of a false positive
    fn_cost = 20  # Cost of a false negative
    
    fp = cm[0, 1]  # Location of false positives in confusion matrix
    fn = cm[1, 0]  # Location of false negatives in confusion matrix
    
    # Calculate custom score
    total_cost = fp * fp_cost + fn * fn_cost
    custom_score = 1 / (1 + total_cost)  # Higher is better
    
    return custom_score

Higher scores are better since the resulting custom score is inversely proportional to the cost of misclassifications.

Registering the Custom Metric Function

We now have the necessary ingredients to prepare our PyCaret experimental environment to train and evaluate models using the custom metric: the data, the target variable, and a number of folds for partitioning the dataset during the underlying cross-validation process that will take place.

exp = setup(
    data=data, 
    target='target_column',
    fold=3
)

Just one last thing to add; our newly created custom metric, of course! This is done via the add_metric method, as follows:

add_metric(
    id="business_cost_metric",
    name="Business Cost Metric",
    score_func=business_cost_metric,
    greater_is_better=True
)

Remember that business_cost_metric is the name of the function we defined above for implementing the custom metric.

Output:

Registered custom metric in PyCaret

One last thing: launching a comparison experiment with PyCaret, which will select, train, and evaluate an assortment of task-specific ML models — namely binary classifiers — for us. The registered custom metric will appear in the evaluation results after being added.

best_model = compare_models()

The returned table shows a comparison of several trained classifiers based on diverse ML techniques (rows) and a set of metrics including ours (columns), highlighting in yellow the best-performing model(s) under each technique.

Comparison of ML models using objective and custom metrics

Recall we used an extremely small and simple dataset merely for illustrative purposes, hence the evaluation results are not brilliant: an ML model will normally require more data than this to become a decent classifier.

Alternatively, we can just train a specific model of the desired type, in which case results are also broken down for each cross-validation fold we set before:

model = create_model('rf')

Output:

Single ML model evaluation using objective and custom metrics