Image by Author

The most misleading moment in any project isn’t a bad result. It’s a suspiciously good one. When a model performs exceptionally well, the natural response is excitement. My response is a set of specific questions. In this guide, I’ll share the three most common ways students build models that look perfect and break in practice, and how to catch each one before it causes real damage.

Quick Context: What Model Evaluation Actually Measures

Model evaluation tells you how well your model generalizes to data it hasn’t seen. The word “generalizes” carries a lot of weight. A model that has memorized training data, been contaminated by future information, or been repeatedly tuned against the same test set hasn’t generalized. It has overfitted in ways that standard metrics won’t reveal. The number looks right. The model is not.

Common Mistake #1: Evaluating Performance on Training Data Rather Than Held-Out Test Data

I had a student come into a session recently, visibly excited. His classification model had achieved 97% accuracy. When I asked which dataset he had evaluated it on, the answer was the training set. That single detail explained everything.

A model evaluated on its own training data will almost always score well. It has already seen those examples, memorized the answers, in a real sense. The score tells you nothing about how the model will perform on data it hasn’t encountered before.

This mistake is more common than it should be, partly because many tutorials demonstrate evaluation without clearly distinguishing between the training and test split. Students see the pattern, fit, predict, score, and replicate it without checking what dataset predict is being called on. The fix is straightforward: always evaluate on a held-out test set the model has never seen during training. That number is the one that matters.

Common Mistake #2: Applying Preprocessing Before the Train/Test Split

This mistake is subtler and, in my experience, far more common among students who already know to use a test set. They split their data correctly, then scale all of it before the split, or encode categorical variables across the full dataset, and the protection the split was supposed to provide quietly disappears.

Preprocessing steps like scaling and encoding learn parameters from data. A standard scaler calculates the mean and standard deviation of a feature. If those values are calculated from the full dataset before the split, the test set has already influenced the preprocessing. The model hasn’t technically seen the test labels, but the test data has informed the transformation applied to the training data. That’s a form of leakage.

The correct sequence is: split first, fit your preprocessing on the training set only, then apply that fitted transformer to both sets. Pipelines in scikit-learn enforce this sequence automatically. Using them isn’t optional. It’s the right default.

Common Mistake #3: Repeatedly Evaluating Against the Same Test Set Until the Model Passes

The third mistake is the hardest to recognize because it doesn’t feel like a mistake. It feels like iteration. A student runs a model, checks the test score, adjusts a hyperparameter, checks again, adjusts something else, checks again. The score eventually reaches an acceptable level. The student reports that score as the model’s performance on unseen data.

It isn’t. The moment you make any decision based on a test set score, including whether to keep adjusting, that test set is no longer unseen. You’ve used it to guide development. Over enough iterations, a model can be inadvertently fitted to the test set as surely as to the training data, and the reported performance won’t hold in production.

This pattern collapses the most promising projects. The solution is a validation set for iterative decisions and a test set that’s evaluated exactly once, at the end. That single discipline changes how reliably your results transfer to the real world.

How to Avoid These Mistakes: Quick Checklist

• Before reporting any performance metric, confirm which dataset it was evaluated on
• Split your data before fitting any preprocessing step, without exception
• Use a pipeline to enforce the correct preprocessing and evaluation sequence automatically
• Designate a validation set for all iterative tuning and model selection decisions
• Reserve the test set for final evaluation only, and evaluate it exactly once

Conclusion

A model that looks perfect in development isn’t a success. It’s a question that needs answering. A high accuracy score that arrived too easily is one of the most reliable signals that something needs a closer look. These three mistakes are entirely correctable, but only once you know to look for them. What does your evaluation process look like before you call a model finished?