Image by Editor | Midjourney

AI tools such as ChatGPT and Claude have helped us with our work tremendously. Using AI, we can outsource many tedious tasks that previously took so much time.

In data analysis work, AI has proven that it could assist our job. By allowing AI to analyze our data, AI can help us understand our massive information accurately and swiftly. Furthermore, the process can be streamlined in our working pipeline, which makes the process automated.

There are many use cases where AI helps the data analysis work, for example:

• Generate code for the analysis
• Explaining insight coming from the data
• Automate data extraction
• Help data cleaning process
• Create synthetic data
• And many more that we could come up with

As AI helps our data analysis work, there are some best practices we can apply to improve the process. What are some of these best practices? Let’s explore them together.

1. Clearly Define the Problem Statement

Before anything else, we must clearly understand the problem we want to solve. Data analysis is all about finding insight to solve business problems, so we need to define the situation we are trying to tackle with data.

Defining the problem is even more essential for AI; the clearer we make the issue as we try to solve it, the more AI can help our data analysis.

For example, let’s assume you are a data analyst in your company. One day, the sales team wants to get insights from the data to improve their sales. You are working with a dataset containing sales history over the years and passing it to AI with the input “Improve sales.” The result would be disastrous, as AI reasoning still needs more specific input.

Discuss with the business teams what they really want from the analysis, their limitations, scope, and anything else that could help clearly define the problem the business is trying to solve. You can determine the problem by aligning with their KPIs or using SMART criteria (Specific, Measurable, Achievable, Relevant, Time-bound).

It’s much more insightful if the business problem is “We want to improve the sales of product X within 30 days with budget $X and timeframe X months. The sales targeting new customers and retain them for minimum 30 days.” It’s difficult to argue this is not a far superior prompt than “Improve sales.”

These inputs allow you to design the prompt to guide the AI analysis, the results of which would better assist you.

2. Prepare and Preprocess the Data

There are already many ways for AI to read our data directly, no matter our dataset type or format.

However, the AI models’ results are only as good as the data passed into them. A poor quality dataset only leads to poor results, even with the best AI model.

We can ask the AI model to prepare and preprocess our dataset. But even with clear direction, the AI may need to catch up on the vital parts that our analysis requires.

That’s why it’s better to prepare our data ourselves before we feed them into the AI model. Try to perform standard data analysis tasks upfront that AI might miss. these include:

• Data cleansing includes removing duplicates, handling missing values, and ensuring data consistency. If you already have standard rules to follow, AI can help you automate the process here; otherwise, you could ask AI for recommendations on how to approach things if you need more clarification.
• Feature selection and engineering are essential in the data analysis to acquire important insight. The process often requires a manual approach as business input is needed here, but AI can help provide recommendations based on the data pattern or suggest techniques to use.
• Outlier detection based on the dataset often requires in-depth analysis. However, AI could help identify the data pattern and recommend a solution.

Combining human knowledge and AI tools is a stronger approach for performing our data analysis work.

3. Choose the Right AI Model

Data analysis can be performed on a wide variety of data formats: tabular, text, visual image, audio, or mixed modalities. Depending on your data analysis procedure and expected results, you might be required to choose different kinds of models. In order to make the best informed decision, we need to understand how each AI model works.

For example, LLM models like GPT-4o are suitable for text or text-like data, such as tabular data presented as text. A model with more complex a reasoning processes may be even more ideal for data analysis, as it could help solve more complex problems.

However, foundationally, we need different models for unstructured data, such as image or audio data. Either we need to transform the data into data that the model can accept, or we use a specialized model that could accept specific data input. Understanding various kinds of models becomes essential when we want to use AI for data analysis.

4. Validate and Post-Process Outputs

The data analysis goal is always all about gaining insights from our data. This means that we should be able to be confident in the results we get from the data analysis, or else what’s the point?

AI-assisted data analysis could help our analysis process and the output can be convincing. However, we need to remember that AI, especially generative AI, is inherently able to hallucinate or provide inaccurate facts. It’s sometimes even biased or harmful, depending on the model.

To ensure that the output is in line with our data analysis target, we need to validate the output and have a post-processing procedure in the workflow where we can safeguard against any potential issues if the AI provides a problematic result. Validation can involve cross-checking the facts gained from AI tools with the business teams and domain expertise. It also includes verifying the quantitative and statistical results against any benchmark given if you feel it’s a sensitive matter. Post-processing can involve improving the AI result text or insight to align with the company terms and editing any output that becomes problematic.

Collaboratively use AI tools for data analysis alongside human control measures to acquire the best output from the data analysis task.

Conclusion

AI has become a tool that certainly helps our data analysis process. Implementing the tool in our data analysis workflow can improve the time it takes for analysis and can help accurately identify patterns that humans might miss.

To ensure that our data analysis process assisted with AI is always on-point, follow best practices, including: clearly defining the problem statement, preparing and preprocessing the data, choosing the right AI model, validating and post-processing outputs.

I hope this has helped!