Image by Editor | Midjourney
Understanding the Dataset
Healthcare fraud detection presents unique modeling challenges due to the inherent class imbalance in claims data. To address these challenges, we previously explored how Yellowbrick’s ClassBalance visualizer could help us understand and handle the imbalanced distribution between fraudulent and non-fraudulent providers. Through this analysis, we examined resampling techniques like SMOTE that could improve our model’s ability to detect potential fraud cases.
We then focused on feature engineering, demonstrating how aggregating individual claims to the provider level could unveil distinctive patterns. Using Yellowbrick’s Parallel Coordinates visualizer, we identified how combinations of inpatient and outpatient claims might signal fraudulent behavior, transforming granular transaction data into meaningful provider-level insights.
Now that we’ve preprocessed our data and engineered informative features, let’s evaluate how well our model can distinguish between fraudulent and legitimate providers. We’ll use Yellowbrick’s ConfusionMatrix visualizer to gain a deeper understanding of our model’s performance beyond simple accuracy metrics.
Our dataset represents 5,410 healthcare providers, with features aggregated from over 550,000 individual claims records. For a detailed breakdown of how we obtained the provider-level dataset from the original data source, please kindly refer to our following posts:
Here is a snapshot of the dataset (previously entitled ‘final_df’) where we will begin our exploration:
Provider IP_Claims_Total OP_Claims_Total PotentialFraud 0 PRV51001 97000.0 7640.0 No 1 PRV51003 573000.0 32670.0 Yes 2 PRV51007 19000.0 14710.0 No 3 PRV51008 25000.0 10630.0 No 4 PRV51011 5000.0 11630.0 No ... ... ... ... ... 5405 PRV57759 0.0 10640.0 No 5406 PRV57760 0.0 4770.0 No 5407 PRV57761 0.0 18470.0 No 5408 PRV57762 0.0 1900.0 No 5409 PRV57763 0.0 43610.0 No [5410 rows x 4 columns]
Our dataset contains:
- Provider IDs
- Total inpatient claims (IP_Claims_Total)
- Total outpatient claims (OP_Claims_Total)
- Fraud labels (PotentialFraud)
Data Preprocessing and Target Encoding
Before we can train our model, we need to encode our target variable. The LabelEncoder transforms our categorical ‘Yes’/’No’ fraud labels into binary values (1 for fraud, 0 for non-fraud):
from sklearn.preprocessing import LabelEncoder # Encode the PotentialFraud column as a binary variable le = LabelEncoder() final_df["PotentialFraud"] = le.fit_transform(final_df["PotentialFraud"])
Let’s view our encoded dataset:
final_df
Provider IP_Claims_Total OP_Claims_Total PotentialFraud 0 PRV51001 97000.0 7640.0 0 1 PRV51003 573000.0 32670.0 1 2 PRV51007 19000.0 14710.0 0 3 PRV51008 25000.0 10630.0 0 4 PRV51011 5000.0 11630.0 0 ... ... ... ... ... 5405 PRV57759 0.0 10640.0 0 5406 PRV57760 0.0 4770.0 0 5407 PRV57761 0.0 18470.0 0 5408 PRV57762 0.0 1900.0 0 5409 PRV57763 0.0 43610.0 0 [5410 rows x 4 columns]
Building Our Initial Model
With our data prepared, we can now train a logistic regression model using our claims totals as features:
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
# Define features and target variable
X = final_df[["IP_Claims_Total", "OP_Claims_Total"]]
y = final_df["PotentialFraud"]
# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Split the dataset into training and testing sets (80-20 split)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
# Train a logistic regression model
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X_train, y_train)
# Predictions
y_pred = log_reg.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"**Accuracy:** {accuracy:.4f}\n")
**Accuracy:** 0.9325
Our initial model achieves an accuracy of 93.25%, but accuracy alone doesn’t tell the complete story, especially with imbalanced datasets like ours. This is where Yellowbrick’s ConfusionMatrix visualizer becomes invaluable.
Visualizing Model Performance with Confusion Matrix
Yellowbrick makes it remarkably simple to create an insightful confusion matrix visualization. With just a few lines of code, we can transform our model’s predictions into a clear, interpretable format:
from yellowbrick.classifier import ConfusionMatrix cm = ConfusionMatrix(log_reg, classes=['No Fraud', 'Fraud']) cm.fit(X_train, y_train) cm.score(X_test, y_test) cm.finalize()
Image By Author
A confusion matrix shows us how well our model classifies each category by breaking down its predictions into four key components:
- True Negatives (969): Correctly identified non-fraudulent providers
- True Positives (40): Correctly identified fraudulent providers
- False Negatives (64): Fraudulent providers misclassified as non-fraudulent
- False Positives (9): Non-fraudulent providers misclassified as fraudulent
This breakdown verifies our earlier accuracy score of 93.25% ((969 + 40) / (969 + 40 + 64 + 9)), but more importantly, it reveals critical insights about our model’s performance.
Of particular concern is our model’s recall for fraudulent cases. Out of 104 actual fraud cases (40 + 64), our model only correctly identified 40, missing 64 fraudulent providers. This means our recall for fraud detection is only 38.5% (40/104). In a healthcare fraud detection context, this low recall is problematic as it indicates our model fails to identify more than half of the fraudulent providers.
This performance metric is particularly important in fraud detection where the cost of missing fraudulent cases (false negatives) is typically much higher than incorrectly flagging legitimate providers (false positives). A missed fraudulent provider could continue to submit false claims, resulting in significant financial losses.
Conclusion
Yellowbrick’s ConfusionMatrix visualizer effectively translates model performance into clear, interpretable results. By seamlessly integrating with scikit-learn, it helps data scientists understand their models beyond basic accuracy scores. In our healthcare fraud detection example, the visualizer highlighted important gaps in fraud identification that accuracy metrics alone failed to capture, showing the value of comprehensive model evaluation tools.
