Image by Editor | Midjourney

In classification problems, model evaluation goes beyond simple accuracy scores. A model might correctly classify cases with 93% accuracy, yet still miss critical positive cases or generate too many false positives. The key challenge lies in finding the right balance: How cautious should our model be when making classifications?

Two Yellowbrick visualizations can help us answer this question. The ROC curve shows how well our model distinguishes between classes, while the discrimination threshold plot helps us fine-tune exactly where we draw the line between them. Together, these tools help us optimize our model’s classification performance while managing the tradeoff between different types of errors.

Getting Started with Our Dataset

We will use the healthcare provider dataset, which combines inpatient and outpatient claims data to identify potentially fraudulent behavior. If you’d like to see how we constructed this dataset, check out our article on How to Analyze Features Using Yellowbrick. Once you have obtained the ‘final_df’ as illustrated in that post, the below blocks of code should run smoothly. 

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

# Encode the PotentialFraud column as a binary variable
le = LabelEncoder()
final_df["PotentialFraud"] = le.fit_transform(final_df["PotentialFraud"])  # 1 = Fraud, 0 = Non-Fraud

# Define features and target variable
X = final_df[["IP_Claims_Total", "OP_Claims_Total"]]
y = final_df["PotentialFraud"]

# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split the dataset into training and testing sets (80-20 split)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

# Train a logistic regression model
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X_train, y_train)

With our model trained, we can now explore how different decision thresholds affect its fraud detection performance.

Understanding ROC Curves

The Receiver Operating Characteristic (ROC) curve helps us understand how our model performs at all possible classification thresholds. By default, our model uses a 0.5 threshold – meaning any provider with a fraud probability above 50% gets flagged. The ROC curve shows what happens as we adjust this threshold up or down.

The curve plots two important rates:

• True Positive Rate (TPR): Out of all actual fraud cases, what percentage do we catch? 
• False Positive Rate (FPR): Out of all legitimate providers, what percentage do we incorrectly flag? 

First, let’s look at how our model performs on the training data through the ROC curve visualization: 

# Yellowbrick's ROC-AUC Curve
from yellowbrick.classifier import ROCAUC
import matplotlib.pyplot as plt

# Create figure with specified size
plt.figure(figsize=(8, 6))

# Create the ROC-AUC visualizer with class labels
visualizer = ROCAUC(log_reg, 
                   classes=['No Fraud', 'Fraud'],
                   title='ROC Curves by Class',
                   per_class=True,
                   micro=False,
                   macro=True)

# Fit and score the visualizer
visualizer.fit(X_train, y_train)
visualizer.score(X_test, y_test)

# Add a legend
visualizer.ax.legend(loc='lower right')

# Finalize and show the plot
visualizer.finalize()
plt.show()

Image By Author

 
The curve shows us:

• Strong discriminative ability with an AUC score of 0.94
• At the default 0.5 threshold (leftmost point of the curve), we achieve a high TPR while maintaining a very low FPR
• As we lower the threshold from 0.5, we move right along the curve, trading increased TPR for higher FPR
• The sharp initial rise in the curve (red arrow) shows high true positive rates even at low false positive rates
• The curves for both fraud and non-fraud classes follow similar patterns, suggesting balanced performance across classes

Exploring Classification Thresholds

While ROC curves show us overall model performance, we often need to choose a specific probability threshold for making predictions. By default, most models use 0.5 as the decision boundary, but this may not be optimal for imbalanced datasets like ours.

Yellowbrick’s Discrimination Threshold plot helps us visualize how different threshold values affect:
– Precision: The proportion of correct fraud predictions
– Recall: The proportion of actual fraud cases we catch
– F1 Score: The harmonic mean of precision and recall
– Queue Rate: The proportion of cases flagged for review

# Yellowbrick's Discrimination Threshold Plot
from yellowbrick.classifier import DiscriminationThreshold

# Create the visualizer with default parameters
visualizer = DiscriminationThreshold(log_reg)

# Fit the visualizer
visualizer.fit(X_train, y_train)

# Show the plot
visualizer.show()

Image By Author

Interpreting the Threshold Plot

The discrimination threshold plot reveals several patterns in our fraud detection model:

• The F1 score (red line) peaks around 0.13, suggesting this might be a better threshold than the default 0.5
• Precision (blue line) increases with higher thresholds, indicating more confidence in fraud predictions
• Recall (green line) decreases with higher thresholds, meaning we miss more actual fraud cases
• The queue rate (purple line) shows what percentage of providers would be flagged for investigation

The vertical line at 0.13 marks the threshold that maximizes the F1 score. This lower threshold means we cast a wider net, flagging more providers for review but potentially catching more fraud cases.

Let’s compare how different thresholds affect our model’s ability to detect fraud:

from sklearn.metrics import recall_score

# Get predicted probabilities for the test set
y_pred_proba = log_reg.predict_proba(X_test)[:, 1]

# Calculate predictions using 0.5 threshold
y_pred_50 = (y_pred_proba >= 0.5).astype(int)
recall_50 = recall_score(y_test, y_pred_50)

# Calculate predictions using 0.13 threshold
y_pred_13 = (y_pred_proba >= 0.13).astype(int)
recall_13 = recall_score(y_test, y_pred_13)

print(f"Recall at 0.50 threshold: {recall_50:.3f}")
print(f"Recall at 0.13 threshold: {recall_13:.3f}")
print(f"Difference in recall: {recall_13 - recall_50:.3f}")

Recall at 0.50 threshold: 0.385
Recall at 0.13 threshold: 0.683
Difference in recall: 0.298

Conclusion

Yellowbrick’s ROC and threshold visualizations reveal that our default 0.5 threshold might be too conservative for healthcare fraud detection. By lowering the threshold to 0.13, we can:

• Increase our fraud detection rate from 38.5% to 68.3%
• Maintain reasonable precision in our predictions
• Better balance the cost of missed fraud against false alarms

This analysis shows how visualization tools can help us make informed decisions about model deployment, moving beyond simple accuracy metrics to find the right balance for our specific use case.