Introduction

Data transformation involves cleaning, restructuring, or otherwise manipulating data into a more useful format. For statisticians and data scientists, the ability to quickly reshape data at the command line can be a major advantage. This short tutorial explores how to transform data using key command line tools like cut, paste, sort, and uniq.

For this guide, ensure you have the following tools available in your Bash shell (generally pre-installed on Linux systems). Alternatively, you can use the default Z shell on macOS systems. These tools should all be installed by default on either setup.

• cut: For slicing out sections of each line
• paste: For merging corresponding lines from multiple files
• sort: For sorting lines of text
• uniq: For filtering out repeated lines or counting unique occurrences
• awk: Useful for more advanced transformations

Performing Data Transformations

Let’s create a simple file, data.csv, containing mock user records in comma-separated format:

echo -e "Name,Age,Score\nAlice,30,85\nBob,25,90\nAlice,30,88\nCharlie,35,85" > data.csv

This file has four lines (plus a header), each representing a user record with a name, age, and score.

We can use cut to extract specific columns from a CSV (or any delimited) file:

# Extract just the Name and Score columns from data.csv
cut -d ',' -f 1,3 data.csv

The cut command line options used are explained as follows:

• -d ',' specifies the comma as the delimiter
• -f 1,3 selects the first and third fields

We can sort our data and identify unique entries. In the following example, we pipe the output from cut to sort, and then the output from sort to uniq, in order to see how many unique names appear.

cut -d ',' -f 1 data.csv | sort | uniq

If, instead, we wanted to count how many times each name appears:

cut -d ',' -f 1 data.csv | sort | uniq -c

The -c option counts the unique entires, instead of displaying them.

If you have multiple files and need to merge them column-wise, paste is ideal:

echo -e "Math\nScience\nEnglish" > subjects.txt
echo -e "90\n88\n92" > scores.txt

paste subjects.txt scores.txt

This outputs lines with columns side by side. You can further transform them as needed.

Below is an example of combining these tools in a single script to transform data from data.csv:

#!/bin/bash

# Extract the "Name" and "Score" columns
cut -d ',' -f 1,3 data.csv > temp.txt

# Sort by Score descending
sort -t ',' -k2,2nr temp.txt > sorted.txt

# Display unique Name-Score pairs
uniq sorted.txt

# Remove temp file
rm temp.txt

Here, we first isolate the relevant columns, then sort them by the second column (Score) in descending order, and finally remove duplicates with uniq.

Final Thoughts

Command line tools such as cut, paste, sort, and uniq deliver a straightforward, scriptable approach to data transformation. Whether trimming down columns, merging files, or generating unique records, these utilities provide a quick and reliable way to reshape data. They can be combined in creative ways, tailored to your project requirements, and integrated into larger Bash scripts for automated workflows.