Introduction

When running large data processing or statistical analysis tasks, you often need to keep an eye on resource usage or job statuses. Rather than manually checking, you can automate monitoring with Bash scripts. This short guide covers a few strategies for monitoring CPU, memory usage, and job completion so you can focus on other tasks.

First, you’ll need these basic Bash utilities:

• ps: To view running processes
• grep: For filtering process information
• top or htop (optional): Interactive process viewers
• mail or a similar utility (optional): For sending automated email alerts

Monitoring Data Processing Jobs

Let’s start with a monitoring script. Below is a simple example script, which we will call monitor.sh, that checks for a specific process name and sends an alert if it’s not running:

#!/bin/bash

PROCESS_NAME="data_analysis.py"

if pgrep -f "$PROCESS_NAME" > /dev/null
then
    echo "$PROCESS_NAME is running."
else
    echo "$PROCESS_NAME is not running! Consider restarting it."
fi

• PROCESS_NAME=”data_analysis.py” sets a variable to the process name we are interested
• The command pgrep -f searches running processes for the specified process name

You can modify this script to, for example, send an email or Slack notification when a process stops.

Let’s monitor resource usage. To keep tabs on CPU and memory usage, you can parse the output of ps. For example:

#!/bin/bash

# Display CPU and memory usage for a specific process
PROCESS_NAME="data_analysis.py"

ps aux | grep "$PROCESS_NAME" | grep -v "grep" | awk '{
    print "CPU Usage: " $3 "%, Memory Usage: " $4 "%"
}'

The above filters out the grep process itself and uses awk to display CPU and memory usage fields (fields $3 and $4 in ps aux output, respectively).

To automate monitor.sh to run periodically, add it to your crontab. For instance, to run the script every five minutes:

crontab -e

# Inside the crontab file:
*/5 * * * * /path/to/monitor.sh >> /path/to/monitor.log 2>&1

This also logs output to monitor.log so you can review it later.

You can learn more about cron here.

On systems with advanced monitoring tools, you can also integrate with the following to take your monitoring script to the next level:

• mail: mail -s "Alert: Process Stopped" user@example.com < /dev/null
• notify-send (on Linux desktop environments): For desktop notifications
• APIs (Slack, Twilio, etc.) for messaging

Final Thoughts

A simple Bash monitoring script can keep track of essential processes and resource usage, providing timely alerts to ensure data processing runs smoothly. By leveraging cron and other system utilities, you can automate these checks, minimize downtime, and ensure your data jobs complete successfully.