Image by Author
Ensuring data quality in data science projects is not a concern only for data leads. Even when doing data projects for your portfolio and getting a job, maintaining data quality across all data science projects should be high on your list. Why?
Here are several reasons:
- Shows your credibility and professionalism
- It’s a practice for real-world scenarios (your job!)
- Your analytical outcomes will be better
- Demonstrates your attention to detail
In short, bad data quality leads to misleading insights, which leads to bad decision-making. Then, you lose a job or don’t get one, and the next thing you know, you’re explaining to your cat why you’re applying to be an “influencer.”
So, here are some data quality tips. I found they can significantly improve and maintain data quality throughout my projects. They are quite simple tips that anybody can follow, no matter the context in which you’re doing the data projects.
1. Establish Clear Data Requirements
I always define what a “good” dataset means. This means deciding on the levels of:
- Accuracy: Are the figures exact, and are there duplicates?
- Completeness: Establish a threshold of how much missing data you will tolerate (10%? 8%? 15%?)
- Consistency: Decide on the format data from different sources should follow, e.g., YYYY-MM-DD for dates
- Timeliness: Decide on how old your data can be to still consider it relevant
Tools: Use a simple spreadsheet (Google Sheets or Excel) to make a checklist of all the above criteria.
2. Automate Data Collection and Cleaning
Manually collecting and cleaning data can be a time sink and could be more error-prone. Write reusable scripts that will automatically collect data and deal with data cleaning tasks, such as handling missing values, text standardization, data type conversion, etc.
Tools: Use BeautifulSoup (web scraping) and Selenium (browser automation) to automate data collection and Python library pandas for data cleaning.
3. Validate Data Regularly
I often create simple validation scripts to check my data for anomalies. This is important to maintain data integrity as the project progresses. For example, I regularly:
- Run summary statistics.
- Check if the data types are correct.
- Check if the data contains positives/negatives where there shouldn’t be any.
- Check for NULLs.
- Check for duplicates.
Tools: Use Great Expectations to write and automate daily data validation scripts.
4. Document Everything
I religiously keep track of all essential transformations I made on my data. This helps me to:
- Traceback and pinpoint where the error in my analysis occurred.
- Share my analysis and make sure others can repeat it with the same result (if used on the same data).
- Maintain version control, allowing me to experiment with different approaches, compare them, and revert if necessary.
Tools: Use Jupyter Notebooks or Google Colab to document code and results in real time.
5. Stay Consistent Across Projects
I’ve found that being consistent across datasets and projects results in fewer errors and saves me time.
Be consistent with:
- Naming conventions: Chose one naming convention and stick to it, e.g., snake_case (e.g., first_name) or CamelCase (e.g., FirstName)
- Data formats: Standardize data formats, e.g., your dates are always in the YYYY-MM-DD format.
- Handling missing values: Standardize the approach for handling missing values, e.g., decide if you’ll fill missing values with a mean, median, or a placeholder such as “No data.”
Tools: Use simple solutions such as Google Docs, Microsoft OneNote, or Microsoft Word to document your rules. You can also use GitHub Wikis or GitLab Wikis or simply store a README file directly in your project repository. If you want to get fancy, maintain the rules documentation in Confluence or Notion.
Conclusion
With these five tips, you can’t go wrong. They’re simple and don’t require paying for some special tools.
Start applying the tips from your next data science project (if you’re not already!), and I’m sure the quality of your projects will improve significantly.
