Bash is a powerful and versatile shell used by countless developers and data scientists. For beginners and practitioners, knowing a few essential scripts can automate day-to-day tasks and streamline your projects. Below are 10 starter scripts, organized into a few concise sections. Use them as inspiration and adapt them to your own needs!
Essential Housekeeping Scripts
1. Backup Important Files
Regular backups protect your critical work—like notebooks, scripts, or configuration files. You can use this script to copy targeted files into a backup directory with timestamps.
#!/usr/bin/env bash SOURCE_DIR="$HOME/projects" BACKUP_DIR="$HOME/backups" TIMESTAMP=$(date +"%Y%m%d_%H%M%S") mkdir -p "$BACKUP_DIR" echo "Backing up from $SOURCE_DIR to $BACKUP_DIR at $TIMESTAMP..." cp -r "$SOURCE_DIR" "$BACKUP_DIR/projects_$TIMESTAMP" echo "Backup completed!"
🔑 Key Idea
- Using timestamps in your backup folder names keeps your versions organized and easy to restore.
2. Disk Usage Monitor
Run this script to see if you’re running out of disk space, which can be critical when working with large datasets. You can set alerts to avoid last-minute surprises.
#!/usr/bin/env bash
THRESHOLD=80
USAGE=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$USAGE" -gt "$THRESHOLD" ]; then
echo "WARNING: Disk usage ($USAGE%) has exceeded the threshold ($THRESHOLD%)."
else
echo "Disk usage is $USAGE%. Everything is okay."
fi
🔑 Key Idea
- Periodically running a disk usage script helps you proactively manage storage issues.
3. Environment Checker
Before starting a data science project, ensure your essential tools (like Python, pip, and conda) are available. This script checks your environment so you can fix any missing dependencies.
#!/usr/bin/env bash
REQUIRED_CMDS=("python" "pip" "conda")
for cmd in "${REQUIRED_CMDS[@]}"; do
if ! command -v "$cmd" > /dev/null; then
echo "Missing $cmd!"
else
echo "$cmd is installed."
fi
done
🔑 Key Idea
- Quickly verify crucial tooling to avoid troubleshooting mid-project.
Automation and Productivity
4. Clean Up Temporary Files
Large amounts of temporary files can clutter your system. This script searches specified folders and deletes files that match a pattern (e.g., .tmp).
#!/usr/bin/env bash
TARGET_DIR="$HOME/tmp"
find "$TARGET_DIR" -type f -name "*.tmp" -exec rm -v {} \;
echo "Temporary files removed from $TARGET_DIR."
🔑 Key Idea
- Automating cleanups keeps your environment tidy and prevents performance issues.
5. Bulk Rename
Renaming files—especially when you have dozens or hundreds—can be tedious. This script replaces certain text in filenames using simple string substitution.
#!/usr/bin/env bash
TARGET_DIR="$HOME/projects/data"
OLD_TEXT="old"
NEW_TEXT="new"
cd "$TARGET_DIR" || exit
for file in *"$OLD_TEXT"*; do
mv "$file" "${file//$OLD_TEXT/$NEW_TEXT}"
done
echo "Bulk rename complete!"
🔑 Key Idea
- Use pattern matching and parameter expansion to batch-rename files effectively.
6. Schedule Automated Tasks
While not a script by itself, combining a Bash script with cron is an essential tool. For instance, you might schedule your backup script (backup.sh) to run daily:
# Example cron entry (edit with crontab -e): # 0 3 * * * /path/to/backup.sh
🔑 Key Idea
- Offload repetitive jobs to cron and free up mental space.
Data Scientist-Focused Tools
7. Python Virtual Env Setup
A consistent project environment reduces “it works on my machine” problems. This script creates (or re-creates) a Python virtual environment and installs dependencies.
#!/usr/bin/env bash PROJECT_DIR="$HOME/projects/my_project" cd "$PROJECT_DIR" || exit if [ -d "venv" ]; then rm -rf "venv" fi python -m venv venv source venv/bin/activate pip install --upgrade pip pip install -r requirements.txt echo "Virtual environment setup complete!"
🔑 Key Idea
- Automate environment creation for reproducible data science workflows.
8. Automatic Data Sync
Data scientists often move files between servers or local/remote machines. This script uses rsync to synchronize directories, allowing incremental updates and minimal transfer times.
#!/usr/bin/env bash LOCAL_DIR="$HOME/projects/data" REMOTE_HOST="username@remote.server.com" REMOTE_DIR="/remote/path/data" rsync -avz --progress "$LOCAL_DIR/" "$REMOTE_HOST:$REMOTE_DIR" echo "Data sync completed!"
🔑 Key Idea
- Leverage rsync for efficient, incremental data transfers.
9. Quick Model Score
Sometimes, you just need to run a quick script to evaluate a model’s performance. This one executes a Python script that reads a test file and outputs performance metrics.
#!/usr/bin/env bash PYTHON_SCRIPT="$HOME/projects/my_model/evaluate_model.py" DATA_FILE="$HOME/projects/my_model/test_data.csv" python "$PYTHON_SCRIPT" --data "$DATA_FILE"
🔑 Key Idea
- Keep a simple, one-command entry point to quickly measure model changes.
10. Simple Log Parsing
When your data science pipeline logs metrics or steps, a quick parsing script can help you extract relevant information.
#!/usr/bin/env bash LOG_FILE="$HOME/projects/my_model/run.log" grep -E "Accuracy|Loss" "$LOG_FILE"
🔑 Key Idea
- Harness Bash’s text processing commands (like grep, awk, and sed) to derive insights from logs without diving into complex tools.
Conclusion
Mastering a handful of Bash scripts can significantly improve your productivity and reliability as a data scientist. Whether you’re backing up files, automating routine tasks, or quickly setting up your Python environment, these starter scripts give you a strong basis to build from. Adapt them to your own workflow, and keep experimenting to streamline your processes even further.
