Introduction

For data scientists and statisticians, text processing forms the backbone of pre-processing and analysis workflows. Two of the most powerful and flexible command line tools for this task are sed (stream editor) and awk (text pattern scanning and processing). This short tutorial will show you how to harness their capabilities for data extraction and manipulation.

Check that you have these tools installed (usually included by default on Linux and macOS systems):

• sed: A versatile stream editor to transform text in a pipeline or file
• awk: Specialized for pattern scanning and text processing

Let’s consider a tab-separated file for our examples, items.tsv:

echo -e "ID\tItem\tPrice\n1\tApple\t1.20\n2\tBanana\t0.80\n3\tOrange\t1.00" > items.tsv

Performing Text Processing and Data Extraction

sed shines at quick transformations. For example, converting tab-delimited data to comma-separated format:

sed 's/\t/,/g' items.tsv

s/\t/,/g means “substitute tabs with commas globally on each line.”

You can also use sed to edit a file in place:

sed -i 's/Apple/Gala Apple/g' items.tsv

The above command modifies items.tsv directly, replacing “Apple” with “Gala Apple.”

Let’s see how we can perform data extraction. awk is excellent at dissecting each line into fields:

awk -F "\t" '{ print $1, $3 }' items.tsv

Let’s investigate what’s going on in the above awk command:

• -F "\t" sets the field separator as a tab
• { print $1, $3 } outputs the first and third fields (ID and Price)

It may look intimidating at first, but with just a quick explanation you can see how awk is both straightforward and powerful.

You can also specify a condition to only act on certain rows:

awk -F "\t" '$3 > 1.00 { print $2, $3 }' items.tsv

This prints the item and price for rows where the price ($3) is greater than 1.00.

To give a little taste of combining these tools, consider the following short script to rename items containing “Banana” to “Yellow Banana” and then calculate how many such items exist:

#!/bin/bash

# Rename Bananas in the stream
sed 's/Banana/Yellow Banana/g' items.tsv | \

# Print only lines that contain the updated Banana name, count them
awk -F "\t" '/Yellow Banana/ { count++ } END { print "Count of Yellow Banana:", count }'

Final Thoughts

sed and awk are indispensable tools for text processing and data extraction. Their power lies in their ability to filter, manipulate, and analyze text files on the fly. Whether you’re cleaning messy data, converting file formats, or extracting insights, these utilities can fit seamlessly into your command-line workflow and automate tasks in your data pipeline.