Image by Editor

What are Statology Sprints?

Statology Sprints are carefully curated learning paths that distill complex concepts into practical tools you can apply immediately. We fuse foundational statistical material with expert insights to enhance your skills while skipping unnecessary filler. Each Sprint includes:

• Core concepts explained clearly
• Practical code examples you can adapt
• Connections between related ideas
• Links to deeper coverage when you need it

This Sprint focuses on Yellowbrick, a Python visualization library that extends the scikit-learn API to incorporate visual diagnostics in your machine learning workflow. By combining scikit-learn with matplotlib, Yellowbrick creates a suite of visual diagnostic tools called “visualizers” that help develop better machine learning models and gain deeper insights into your data.

Getting Started with Yellowbrick

Yellowbrick works as a diagnostic tool, helping data scientists understand model selection, hyperparameter tuning, and model evaluation. It integrates seamlessly with:

• scikit-learn for machine learning algorithms
• matplotlib for visualization base
• numpy for numerical computations
• pandas for data manipulation

Let’s begin by setting up Yellowbrick in your Python environment.

# Install using pip
pip install yellowbrick

# Or using conda (recommended)
conda install -c conda-forge yellowbrick

For our examples, we’ll be working with the Healthcare Provider Fraud Detection dataset, which contains information about healthcare providers, including:

• Beneficiary details (demographics, chronic conditions, etc.)
• Inpatient claims
• Outpatient claims
• Labels indicating potentially fraudulent providers

The dataset consists of several files, but we’ll focus on these four main components:

1. Train dataset (5,410 rows) – Contains provider IDs and fraud labels
2. Train Beneficiary dataset (138,556 rows) – Patient information
3. Train Inpatient dataset (40,474 rows) – Hospital admission claims
4. Train Outpatient dataset (517,737 rows) – Outpatient visit claims

Understanding Class Imbalance with Yellowbrick

Let’s start with a common challenge in machine learning: class imbalance. In our healthcare fraud dataset, we have a classic imbalanced classification problem where fraudulent providers represent a small minority of cases.

Yellowbrick’s ClassBalance visualizer helps us understand the distribution of our target variable:

# Import necessary libraries
import pandas as pd
import numpy as np
from yellowbrick.target import ClassBalance

# Load the training dataset
df_train = pd.read_csv('Train-1542865627584.csv')

# Examine the first few rows
print(df_train.head())

# Check the class distribution
print("\nClass Distribution:")
print(df_train["PotentialFraud"].value_counts())

   Provider PotentialFraud
0  PRV51001              No
1  PRV51003             Yes
2  PRV51004              No
3  PRV51005             Yes
4  PRV51007              No

Class Distribution:
No     4904
Yes     506

Now, let’s visualize this class imbalance using Yellowbrick:

# Visualize class balance
visualizer = ClassBalance()
visualizer.fit(df_train['PotentialFraud'])
visualizer.show()

Image By Author

This visualization reveals a significant class imbalance – only 9.4% of providers are flagged as potentially fraudulent, while 90.6% are non-fraudulent. This imbalance presents a challenge for model training, as classifiers tend to favor the majority class.

Addressing Class Imbalance

With only 9.4% of providers labeled as fraudulent, we need strategies to address this class imbalance before model training. Several resampling techniques can help, including random oversampling, random undersampling, and SMOTE (Synthetic Minority Over-sampling Technique). Yellowbrick helps us visualize how each technique transforms our class distribution.

➡️ Learn more about visualizing class imbalance with Yellowbrick

Feature Analysis with Yellowbrick

Once we’ve addressed class imbalance, our next step is to analyze features that might help distinguish between fraudulent and non-fraudulent providers. However, we face a data integration challenge – our target variable exists at the provider level, but most useful information is in individual claims spread across separate tables.

Let’s aggregate claims data to create provider-level features:

import pandas as pd

# Load datasets
train = pd.read_csv('Train-1542865627584.csv')  # Provider fraud labels
inpat = pd.read_csv('Train_Inpatientdata-1542865627584.csv')  # Hospital stays
outpat = pd.read_csv('Train_Outpatientdata-1542865627584.csv')  # Outpatient visits

# Aggregate inpatient claims totals by provider
ip_provider_amounts = inpat.groupby('Provider')['InscClaimAmtReimbursed'].sum().reset_index()
ip_provider_amounts = ip_provider_amounts.rename(columns={'InscClaimAmtReimbursed': 'IP_Claims_Total'})

# Aggregate outpatient claims totals by provider
op_provider_amounts = outpat.groupby('Provider')['InscClaimAmtReimbursed'].sum().reset_index()
op_provider_amounts = op_provider_amounts.rename(columns={'InscClaimAmtReimbursed': 'OP_Claims_Total'})

# Merge inpatient and outpatient totals
provider_claims = pd.merge(
    ip_provider_amounts,  # Left: Provider, IP_Claims_Total
    op_provider_amounts,  # Right: Provider, OP_Claims_Total
    on='Provider',
    how='outer'
).fillna(0)  # Fill NaN with 0 for providers with only one type of claims

# Merge with fraud labels
final_df = pd.merge(
    provider_claims,
    train[['Provider', 'PotentialFraud']],
    on='Provider',
    how='outer'
)

print(final_df.head())

  Provider  IP_Claims_Total  OP_Claims_Total PotentialFraud
0 PRV51001          97000.0           7640.0             No
1 PRV51003         573000.0          32670.0            Yes
2 PRV51007          19000.0          14710.0             No
3 PRV51008          25000.0          10630.0             No
4 PRV51011           5000.0          11630.0             No

Now we have a provider-level dataset that includes both inpatient and outpatient claims totals. This integrated dataset allows us to explore patterns that might distinguish fraudulent from legitimate providers.

Visualizing Feature Patterns with Parallel Coordinates

Yellowbrick’s Parallel Coordinates plot is an excellent tool for visualizing how feature values relate to our target variable:

# Parallel Coordinates: Visualize how individual feature values vary across classes
from yellowbrick.features import ParallelCoordinates
from sklearn.preprocessing import StandardScaler

# Prepare features and target
X = final_df[['IP_Claims_Total', 'OP_Claims_Total']]
y = (final_df['PotentialFraud'])

# Scale the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_scaled = pd.DataFrame(X_scaled, columns=X.columns)

# Create the parallel coordinates visualizer
visualizer_parallel = ParallelCoordinates()
visualizer_parallel.fit_transform(X_scaled, y)
visualizer_parallel.show()

Image By Author

The parallel coordinates plot reveals several interesting patterns:

• Fraudulent providers (shown in green) tend to have more extreme claiming patterns, with unusually high totals in both inpatient and outpatient categories.
• Many fraudulent providers show steep slopes between IP_Claims_Total and OP_Claims_Total, indicating significantly different activity levels between service types.
• Non-fraudulent providers (shown in blue) cluster more densely in the moderate range, showing consistent claiming patterns.

This visualization suggests that unusual claim amounts across different service types might be indicators of potential fraud.

➡️ Learn more about analyzing features with Yellowbrick

Model Evaluation with Yellowbrick Visualizers

Once we’ve engineered provider-level features and addressed class imbalance, our next step is to evaluate how well our model can distinguish between fraudulent and legitimate providers. While accuracy is a common metric, it often masks important performance aspects, especially with imbalanced datasets. Yellowbrick offers several powerful visualizers that provide deeper insights into model performance.

Let’s start by preparing our data and training a simple logistic regression model:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Encode the PotentialFraud column as a binary variable
le = LabelEncoder()
final_df["PotentialFraud"] = le.fit_transform(final_df["PotentialFraud"])  # 1 = Fraud, 0 = Non-Fraud

# Define features and target variable
X = final_df[["IP_Claims_Total", "OP_Claims_Total"]]
y = final_df["PotentialFraud"]

# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split the dataset into training and testing sets (80-20 split)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

# Train a logistic regression model
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X_train, y_train)

# Predictions
y_pred = log_reg.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")

Accuracy: 0.9325

Our model achieves over 93% accuracy. However, this number alone can be misleading with imbalanced datasets. Let’s explore Yellowbrick’s visualization tools to gain deeper insights into our model’s performance.

Understanding Model Performance with Confusion Matrix

Yellowbrick’s ConfusionMatrix visualizer helps us understand how well our model classifies each category by breaking down predictions into four components – true positives, true negatives, false positives, and false negatives:

from yellowbrick.classifier import ConfusionMatrix

# Create the confusion matrix visualizer
cm = ConfusionMatrix(log_reg, classes=['No Fraud', 'Fraud'])
cm.fit(X_train, y_train)
cm.score(X_test, y_test)
cm.finalize()

Image By Author

The confusion matrix reveals that while our model accurately identifies legitimate providers (969 true negatives), it struggles with fraudulent cases – correctly identifying only 40 out of 104 fraud cases, resulting in a low recall of just 38.5% for the fraud class. This insight is critical in fraud detection, where the cost of missing fraudulent providers (false negatives) is typically much higher than incorrectly flagging legitimate ones (false positives).

➡️ Learn more about using confusion matrices for model evaluation

Enhanced Performance Assessment with Classification Report

For a more comprehensive view of model performance, we can use Yellowbrick’s ClassificationReport visualizer, which provides precision, recall, and F1 scores for each class:

from yellowbrick.classifier import ClassificationReport
import matplotlib.pyplot as plt

# Create and display the classification report
cr = ClassificationReport(log_reg, classes=['No Fraud', 'Fraud'])
cr.fit(X_train, y_train)
cr.score(X_test, y_test)
cr.finalize()
plt.tight_layout()
plt.show()

Image By Author

This visualization provides a detailed breakdown of our model’s performance metrics:

For the Fraud class:
– Precision (0.816): When our model predicts fraud, it’s correct 81.6% of the time
– Recall (0.385): Our model identifies only 38.5% of all actual fraud cases
– F1-Score (0.523): The harmonic mean of precision and recall

For the Non-Fraud class:
– Precision (0.938): When our model predicts non-fraud, it’s correct 93.8% of the time
– Recall (0.991): Our model identifies 99.1% of all legitimate providers
– F1-Score (0.964): The harmonic mean of precision and recall

The report confirms our confusion matrix observations: our model excels at identifying legitimate providers but struggles significantly with fraud detection.

➡️ Learn more about classification reports for comprehensive model evaluation

Fine-tuning with ROC Curves and Threshold Analysis

Our poor fraud detection performance may stem from our model’s default decision threshold. Yellowbrick’s ROCAUC visualizer helps us understand how our model performs across different threshold values:

from yellowbrick.classifier import ROCAUC
import matplotlib.pyplot as plt

# Create the ROC-AUC visualizer
visualizer = ROCAUC(log_reg, 
                   classes=['No Fraud', 'Fraud'],
                   title='ROC Curves by Class',
                   per_class=True,
                   micro=False,
                   macro=True)

# Fit and score the visualizer
visualizer.fit(X_train, y_train)
visualizer.score(X_test, y_test)
visualizer.finalize()
plt.show()

Image By Author

The ROC curve shows our model has good discriminative ability with an AUC score of 0.94. To find the optimal threshold, we can use Yellowbrick’s DiscriminationThreshold visualizer:

from yellowbrick.classifier import DiscriminationThreshold

# Create the threshold visualizer
visualizer = DiscriminationThreshold(log_reg)
visualizer.fit(X_train, y_train)
visualizer.show()

Image By Author

This visualization suggests that a threshold of around 0.13 (rather than the default 0.5) maximizes the F1 score. Let’s see how this change affects our fraud detection rate:

from sklearn.metrics import recall_score

# Get predicted probabilities for the test set
y_pred_proba = log_reg.predict_proba(X_test)[:, 1]

# Calculate predictions using 0.5 threshold
y_pred_50 = (y_pred_proba >= 0.5).astype(int)
recall_50 = recall_score(y_test, y_pred_50)

# Calculate predictions using 0.13 threshold
y_pred_13 = (y_pred_proba >= 0.13).astype(int)
recall_13 = recall_score(y_test, y_pred_13)

print(f"Recall at 0.50 threshold: {recall_50:.3f}")
print(f"Recall at 0.13 threshold: {recall_13:.3f}")
print(f"Difference in recall: {recall_13 - recall_50:.3f}")

Recall at 0.50 threshold: 0.385
Recall at 0.13 threshold: 0.683
Difference in recall: 0.298

By lowering the threshold from 0.5 to 0.13, we dramatically improve our fraud detection rate from 38.5% to 68.3% – a 30 percentage point increase! This showcases how visual threshold analysis can significantly enhance model performance, especially with imbalanced datasets like our healthcare fraud example.

➡️ Learn more about threshold optimization with ROC curves

Bringing It All Together: Room Occupancy Detection

Let’s bring together the concepts we’ve learned by applying Yellowbrick’s visualization tools to a different binary classification problem: room occupancy detection. This problem involves determining whether a room is occupied based on environmental sensor data – a common task in smart building management systems.

First, let’s load Yellowbrick’s built-in occupancy dataset:

from yellowbrick.datasets import load_occupancy
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Load the occupancy dataset
X, y = load_occupancy()

# Get the actual column names from the dataset
actual_columns = ['temperature', 'relative humidity', 'light', 'CO2', 'humidity']

# Create a DataFrame with the correct column names
df = pd.DataFrame(X, columns=actual_columns)
df['occupied'] = y

# Rename columns to remove spaces for easier handling
df = df.rename(columns={
    'relative humidity': 'relative_humidity',
    'humidity': 'humidity_ratio'
})

# Examine the dataset structure
print(f"Dataset shape: {df.shape}")
print("\nFeature overview:")
print(df.describe().round(2))

Dataset shape: (20560, 6)

Feature overview:
         temperature relative_humidity light      CO2        humidity_ratio   occupied
count    20560.000          20560.000  20560.000  20560.000       20560.000  20560.000
mean        20.906             27.656    130.757    690.553           0.004      0.231
std          1.055              4.982    210.431    311.201           0.001      0.422
min         19.000             16.745      0.000    412.750           0.003      0.000
25%         20.200             24.500      0.000    460.000           0.004      0.000
50%         20.700             27.290      0.000    565.417           0.004      0.000
75%         21.525             31.290    301.000    804.667           0.005      0.000
max         24.408             39.500   1697.250   2076.500           0.006      1.000

Step 1: Explore Class Balance

Let’s first understand our class distribution using Yellowbrick’s ClassBalance visualizer:

from yellowbrick.target import ClassBalance

# Create the class balance visualizer
visualizer = ClassBalance(labels=['Unoccupied', 'Occupied'])
visualizer.fit(df['occupied'])
visualizer.show()

Image By Author

The visualization reveals an imbalance: only 23% of readings indicate the room is occupied, while 77% show it’s unoccupied. This imbalance is less severe than our healthcare fraud example, but still important to consider during model evaluation.

Step 2: Analyze Feature Relationships

Let’s examine how different environmental features relate to room occupancy using Yellowbrick’s Parallel Coordinates plot:

from yellowbrick.features import ParallelCoordinates
from sklearn.preprocessing import StandardScaler

# Get features for analysis
features = ['temperature', 'relative_humidity', 'light', 'CO2', 'humidity_ratio']
X_features = df[features].values

# Standardize features for better visualization
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_features)

# Create parallel coordinates visualization
visualizer = ParallelCoordinates(
    classes=['Unoccupied', 'Occupied'],
    features=features,
    normalize='standard'
)
visualizer.fit_transform(X_scaled, y)
visualizer.show()

Image By Author

The parallel coordinates plot reveals interesting patterns in how the environmental features relate to room occupancy:

• Light levels show a strong differentiation, with occupied rooms (green) having noticeably higher readings
• CO₂ levels are significantly elevated when the room is occupied, likely due to human respiration
• Temperature patterns show some distinction between occupied and unoccupied states
• Relative humidity and humidity ratio features display subtle variations but still contribute to the classification pattern

These clear separations in the feature space, particularly for light and CO₂ measurements, suggest our environmental sensor data contains robust signals for accurate occupancy detection.

Step 3: Train Multiple Models and Compare Performance

Now let’s train several classification models and compare their performance using Yellowbrick’s visualizers:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from yellowbrick.classifier import ClassificationReport

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.25, random_state=42)

# Initialize models
models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "K-Nearest Neighbors": KNeighborsClassifier(n_neighbors=5)
}

# Train and evaluate each model
for name, model in models.items():
    # Train the model
    model.fit(X_train, y_train)
    
    # Create classification report visualizer
    plt.figure(figsize=(10, 6))
    visualizer = ClassificationReport(
        model, 
        classes=['Unoccupied', 'Occupied'],
        support=True,
        title=f"{name} - Classification Report"
    )
    visualizer.fit(X_train, y_train)
    visualizer.score(X_test, y_test)
    visualizer.show()

Image By Author

Image By Author

Image By Author

The classification reports show that all three models perform exceptionally well on this dataset, with metrics consistently above 0.96 for both classes. While the Random Forest classifier achieves marginally higher scores for the “Unoccupied” class (0.997 precision and 0.996 F1), all models demonstrate remarkable performance with minimal differences between them. The high precision and recall values across all algorithms indicate that the environmental sensor data provides clear signals for occupancy detection, allowing effective classification despite the class imbalance of 1182 occupied vs 3958 unoccupied instances in the test set.

Let’s examine the Random Forest model’s confusion matrix for more detail:

from yellowbrick.classifier import ConfusionMatrix

# Get the Random Forest model
rf_model = models["Random Forest"]

# Create and display confusion matrix
cm = ConfusionMatrix(rf_model, classes=['Unoccupied', 'Occupied'])
cm.fit(X_train, y_train)
cm.score(X_test, y_test)
cm.show()

Image By Author

The confusion matrix confirms the Random Forest model’s exceptional performance on the room occupancy detection task. Out of 5,140 test instances, only 32 are misclassified: 21 unoccupied rooms incorrectly predicted as occupied (false positives) and 11 occupied rooms incorrectly identified as unoccupied (false negatives). With 99.4% of instances correctly classified (3,937 true negatives and 1,171 true positives), the model demonstrates that the environmental sensor data provides remarkably clear signals for accurately determining room occupancy status, even with the class imbalance present in the dataset.

Step 4: Feature Importance Analysis

Let’s identify which features contribute most to the Random Forest model’s predictions:

from yellowbrick.model_selection import FeatureImportances

# Create feature importances visualizer
viz = FeatureImportances(rf_model, labels=features)
viz.fit(X_train, y_train)
viz.show()

Image By Author

The feature importance visualization reveals a clear hierarchy in predictive power: light levels dominate with a relative importance of nearly 100%, making it by far the strongest indicator of room occupancy. Temperature follows as the second most important feature, contributing about 25% relative importance, while CO₂ levels rank third at approximately 18%. The humidity-related features (humidity_ratio and relative_humidity) contribute minimally, with relative importance values below 5%. This ranking aligns with common sense – when people enter a room, they typically turn on lights, and their presence affects temperature, while their respiration gradually increases CO₂ concentration. The overwhelming importance of light measurements suggests that even a simplified model focused primarily on light levels could perform reasonably well for basic occupancy detection applications.

Step 5: Threshold Optimization

Finally, let’s use Yellowbrick’s ROCAUC and DiscriminationThreshold visualizers to analyze and optimize our model’s decision threshold:

from yellowbrick.classifier import ROCAUC, DiscriminationThreshold

# Create ROC curve visualization
plt.figure(figsize=(10, 6))
roc_viz = ROCAUC(rf_model, classes=['Unoccupied', 'Occupied'])
roc_viz.fit(X_train, y_train)
roc_viz.score(X_test, y_test)
roc_viz.show()

# Create threshold visualization
plt.figure(figsize=(10, 6))
dt_viz = DiscriminationThreshold(rf_model)
dt_viz.fit(X_train, y_train)
dt_viz.show()

Image By Author

Image By Author

The ROC curve shows exceptional performance with an AUC of 1.0 for both classes, indicating the Random Forest model achieves near-perfect separation between occupied and unoccupied states. This extremely high classification performance demonstrates how well-suited the environmental sensor data is for occupancy detection, though the small number of misclassifications we observed in the confusion matrix (32 out of 5,140 instances) reminds us that no real-world model is truly flawless.

The threshold plot offers additional insights into the model’s robustness, showing that precision, recall, and F1 scores remain above 0.95 across a wide range of decision thresholds (approximately 0.05 to 0.90). This remarkable stability means:

• The default threshold (0.5) is already optimal, with the visualization suggesting a slight adjustment to 0.55
• Even when moving away from the default threshold, performance remains exceptional
• Systems deployed using this model could be fine-tuned based on specific needs without significant performance degradation, where (i) Smart lighting systems that prioritize user experience might use a lower threshold (0.2-0.4) to ensure lights are always on when needed and (ii) Energy conservation applications might use a higher threshold (0.6-0.8) to avoid unnecessary HVAC or lighting operation in unoccupied spaces

The queue rate line (purple) indicates that approximately 23% of instances would be flagged for occupancy at most thresholds, correctly matching our known class distribution.

Next Steps

This Sprint has equipped you with powerful tools for visual model evaluation using Yellowbrick. Here are some ways to extend what you’ve learned:

1. Apply to multiclass problems – Yellowbrick’s visualizers also support multiclass classification tasks. Try applying these techniques to problems with more than two categories, such as image classification or text categorization.
2. Explore regression visualizers – Yellowbrick offers specialized visualizers for regression tasks, including residuals plots, prediction error plots, and alpha selection. These can help you evaluate and refine models that predict continuous values.
3. Integrate with model selection workflows – Use Yellowbrick’s validation curve and learning curve visualizers to optimize hyperparameters and determine the right model complexity for your data.
4. Create custom visualizers – Extend Yellowbrick by creating domain-specific visualizers that address your particular modeling needs. The library is designed to be easily extended with custom visualization classes.
5. Build interactive model evaluation dashboards – Combine Yellowbrick with dashboard tools like Streamlit or Dash to create interactive applications that allow stakeholders to explore model performance visually.

The visual model evaluation approach we’ve demonstrated helps bridge the gap between technical metrics and practical understanding. By incorporating these visualization techniques into your machine learning workflow, you can make more informed modeling decisions, communicate findings more effectively to stakeholders, and ultimately build more reliable and useful models.

For more advanced applications and techniques, explore Yellowbrick’s comprehensive documentation and contribute to its growing open-source community.