Image by Editor
Data cleaning is often a necessary process to follow before data analysis, and even more so in data science workflows such as training machine learning models. Although data cleaning is intended to improve data quality, it requires careful, methodical execution, because mistakes — often made unintentionally — can introduce bias, yield distorted results, or undermine the quality of downstream analyses.
This article lists five frequent data cleaning mistakes that may negatively impact further analyses.
1. Removing Outliers “No Matter What”
Outliers are not always something to eradicate from your dataset. Blindly deleting outliers without first analyzing and understanding them could lead to the removal of rare but legitimate and relevant observations, thereby distorting the underlying distribution of the data. Be mindful when performing outlier detection as part of exploratory data analysis (EDA).
How to avoid it: Take a deep dive into the root cause of outliers and use domain knowledge before deciding whether to keep, transform, or remove them.
2. Dropping Missing Values “No Matter What”
Replace “outliers” with “missing values” in the mistake above and you have another. Blindly removing data instances (rows) or attributes (columns) with missing values not only reduces sample size; it may also introduce bias if the missing-data pattern is not random, e.g., certain groups may have a blank value for a field such as “permanent residence address.”
How to avoid it: To properly address this issue related to data completeness, analyze possible patterns behind missing values and consider applying an imputation method instead of jumping straight to automatic deletion.
3. Mixing Inconsistent Data Formats and Units
Mixing data expressed in different units, measurement scales, or temporal variables with diverse date formats can lead to incorrect calculations and misleading outcomes. This is a common pitfall in data integration and standardization processes, especially in large or complex organizations.
How to avoid it: To prevent this mistake, standardize units, formats, and data types as early as possible during data collection, followed by rigorous validation before downstream analysis.
4. Wrong Encoding of Categorical Variables
A common mistake when encoding categorical variables — a necessary process for using most types of machine learning models, for instance — is to treat categories as numeric values by default. This induces a sense of ordering that does not exist in the original variable and can mislead statistical and machine learning models. It often occurs during feature engineering and variable transformation.
How to avoid it: Use appropriate encoding: apply one-hot encoding for nominal (unordered) categories, ordinal encoding when a true order exists (e.g., t-shirt sizes S, M, L), and consider target or frequency encoding when justified. Avoid arbitrary integer mappings that imply a nonexistent order.
5. Cleaning Data Without Documenting the Process
This overarching and common mistake is not something to take lightly. Failing to document the steps taken in data cleaning workflows hinders the reproducibility of analyses and increases the chances of hidden errors in downstream work.
How to avoid it: The solution is simple yet often overlooked: keep clear records of all cleaning steps and decisions with a mix of documentation, logging, and version control.
