Image by Author

As a professional who wants to get more valuable work done to be productive, it is logical to explore several means that can help you achieve this end goal. Doing simple tasks repetitively is tiring and also chops off valuable time that could have been put into achieving some other meaningful work. It is in light of this that several tools and techniques exist that can be used to automate repetitive analysis tasks.

Why not be a smart professional? I’m talking about the person who carefully seeks possible ways to automate simple repetitive tasks that would have prevented them from achieving critical work, letting the computer do the boring work for them.

Automating analysis tasks can be achieved in many niches, including but not limited to:

• Data Analysis
• Statistical Analysis and Data Mining
• Automated Testing
• Process Automation
• Continuous Integration and Deployment
• Network and Security Analysis

In this article, we will focus on how automating repetitive analysis tasks can be achieved in the data analysis niche.

Prerequisite

Foundational knowledge of Python is required for easy understanding of this article, as code snippets are used herein to help you fully understand the article is written in Python. A Unix-like operating system such as Linus or Mac is required to run the task scheduler. If you are using a Windows operating system, you can use a Unix-supported virtual sandbox or Windows Sub-system for Linux (WSL) to achieve a similar result. Download the dataset used for this article here.

What is Task Automation?

According to IBM, “Task automation is the process of applying technology to complete a task or augment a workflow to improve productivity. It involves automating time-consuming or mundane actions, reducing or eliminating the need for human intervention, and freeing workers for higher-value tasks.”

In a nutshell, task automation is the process of applying technology (such as software products, frameworks, or libraries) to help you perform repetitive tasks automatically. This saves time, prevents errors, and improves overall efficiency.

Several important factors must be considered when carrying out analysis task automation. You will learn about them shortly.

Key Considerations When Automating Repetitive Analysis Tasks

When automating repetitive analytics tasks, it is imperative to think carefully about the approach and techniques you intend to adopt, keeping in mind crucial factors such as efficiency, accuracy, and scalability. These are some of the keystone parameters you have to note when automating analysis tasks. You don’t want to adopt an approach that is prone to errors, that is not robust enough to scale when faced with large amounts of data, or one that is inefficient and unreliable (sometimes produces correct results but sometimes fails at it). The key considerations are below:

1. Define Clear Objectives

Clearly state the main reasons for wanting to adopt automation into your workflow, such as improving accuracy or reducing manual effort. This will help you streamline your approach and solve the problem directly without making it complicated.

2. Assess the Task’s Suitability for Automation

Analysis tasks suitable for automation have to be repetitive, follow a predictable pattern, be rule-based, and not require human judgment. They must also be frequent and not just one-off tasks.

3. Data Quality and Consistency

Ensure that the input data fed to the automation system is always consistent, clean, and well-structured to avoid errors during the automation process.

4. Choose the Right Tools and Technologies

Choose tools that perfectly align with the task’s complexity, your team’s level of expertise, and the existing tech stack. Ensure that the tool scales when the volume of data increases and can conveniently integrate into existing software systems.

5. Design for Flexibility and Maintenance

Make room for changes in your automation system to accommodate possible modifications in requirements, business rules, or data formats. Also, ensure regular updates and maintenance to keep your automation running well.

6. Ensure Accuracy and Validation

Test your automation to ensure its output (result) is accurate. Implement error-handling mechanisms to prevent complete system failure when exceptions occur. Validate your automated output by comparing it with your manual output.

7. Documentation and Transparency

Document the entire workflow, set-up, and configurations of the automation system adopted clearly and easily. This will show openness and credibility and help other professionals easily understand the system’s inner workings.

8. Monitor and Optimize

Ensure that the automation system adopted is working as expected by regularly monitoring the performance. This can be achieved through:

• Real-time tracking (observing performance using visualization tools like Tableau, Power-Bi, etc.)
• Error logging (using Python Logging module or other suitable framework)
• Alert systems (via SMS, Slac,k, etc.)
• Infrastructure health check

9. Cost-Benefit Analysis

Analyze the cost of setting up the automation system against the benefit you intend to derive from it to determine whether it is worth the extra investment. Also, look beyond the immediate use and consider the long-term benefits.

We just learned about some key considerations that must be strictly followed when automating repetitive analysis tasks. Now, it’s time to get our hands dirty with a live coding example.

Automating a Repetitive Analysis Task: A Practical Example

We will automate the analysis of a company sales dataset (sales_data.csv) by writing a Python script to be run intermittently using a cron job (time-based task scheduler). The analysis tasks to be automated by the Python script are:

• Calculate the total revenue realized (quantity * price)
• Identifying the best-selling product (product with the highest total sales)
• Finding the most profitable product (the product generating the highest revenue)

Let’s dive right into it. Follow the steps below:

Step 1

Create a Python file and give it a suitable name, e.g., automation_analysis.py, and load the dataset.

import pandas as pd

# Load the CSV file
df = pd.read_csv("sales_data.csv")

NB: Make sure you have the dataset in the same directory as the automation.py_analysis.py file or provide the correct path to the dataset file.

Step 2

Compute the total revenue for each sale or data point. Using this formula total_revenue = quantity * price

# Calculate total revenue per sale
df["Total Revenue"] = df["Quantity"] * df["Price"]

Step 3

Calculate the overall total revenue; this is achieved by summing all the individual data points or sales revenues.

# Calculate overall total revenue
total_revenue = df["Total Revenue"].sum()

Step 4

Identify the best-selling product. This is achieved by grouping sales data by product and then summing the quantity sold for each product. The product with the highest total amount sold is identified as the best-selling product.

# Find the best-selling product (by quantity sold)
best_selling_product = df.groupby("Product")["Quantity"].sum().idxmax()

Step 5

Identify the most profitable product. To determine this, sum up the total revenue for each product and find the maximum.

# Find the most profitable product (by total revenue generated)
most_profitable_product = df.groupby("Product")["Total Revenue"].sum().idxmax()

Step 5

Printing the results: The lines of code below print the individual analysis results.

print("Total Revenue: $", total_revenue)
print("Best-Selling Product:", best_selling_product)
print("Most Profitable Product:", most_profitable_product)

Let’s assemble the code fragments to make our automation script:

import pandas as pd

def analyze_sales(csv_file):
    """ Automates the analysis of sales data."""
    # Load the CSV file
    df = pd.read_csv(csv_file)

    # Calculate total revenue per sale
    df["Total Revenue"] = df["Quantity"] * df["Price"]

    # Calculate overall total revenue
    total_revenue = df["Total Revenue"].sum()

    # Find the best-selling product (by quantity sold)
    best_selling_product = df.groupby("Product")["Quantity"].sum().idxmax()

    # Find the most profitable product (by total revenue generated)
    most_profitable_product = df.groupby("Product")["Total Revenue"].sum().idxmax()

    # Print results
    print(f"Total Revenue: ${total_revenue:,.2f}")
    print(f"Best-Selling Product: {best_selling_product}")
    print(f"Most Profitable Product: {most_profitable_product}")

# Run the automation script
if __name__ == "__main__":
    analyze_sales("sales_data.csv")

The automation script is a Python code with a function called analyse_sales; it takes in a CSV file (file representing the sales dataset) as an argument and analyses the following:

• Total revenue per sale
• Overall total revenue on all the products in the dataset
• Finds the best-selling products based on the quantity sold
• Finally, identifies the best-performing product

By running this script, you should get an output similar to the screenshot below:

We now have our automation script working perfectly. The next thing we need to do is to use a task scheduler to run the script at intervals. For that, we will be using the cron to run the script at 8 (eight) hour intervals. Do as instructed in the steps below:

Step 6

Open your terminal and type

crontab -e

Running this command opens up the crontab file for editing.

Step 7

Add the cron job for the task: At the bottom of the already opened contrab file, add this:

0 */8 * * * /usr/bin/python3 /home/austin/Desktop/tutorials/AUTOMATION/automation_analysis.py >> /home/austin/Desktop/tutorials/AUTOMATION/sales_log.txt 2>&1

Your crontab file should look like this:

Let’s quickly go through the cron job for a better understanding:

• 0 */8 * * * (schedules or runs the script every 8 hours)
• /usr/bin/python3 (path to the python interpreter)
• /home/Austin/Desktop/tutorials/AUTOMATION/automation_analysis.py (absolute path to the automation script)
• /home/Austin/Desktop/tutorials/AUTOMATION/automation_analysis.py/sales_log.txt

NB: Use the correct absolute paths to the automation script as is on your computer. Don’t copy mine and use it; it won’t work.

To get the absolute path, go to the working directory where your files are and run this command in your terminal pwd, copy it, and add the filename to it.

Step 8

Save and exit the contract file. If you are using the nano text editor, which is most times set as default, then do the following:

• Press Ctrl and X to exit
• Press Y to save the changes
• Lastly, press Enter to confirm

Step 9

Verify that the cron job was added successfully:

crontab -l

You should see something similar to what is shown in the image below:

Conclusion

Having a clear plan about what task to automate and the requisite tools and technologies that can robustly support the process at scale is very imperative, as these are the fundamental considerations that ensure a productive automation system. Automating repetitive analysis tasks, if done right, could be one of the best productivity decisions you will make as a professional.

When designing your automation system, always be open to future improvements, making it flexible to accommodate further configurations and upgrades for enhanced performance.

Apply what you have learned so far in this article to other similar tasks to gain more familiarity and mastery.