Introduction

Looking to perform text analytics or do some other natural language processing tasks? You may want to check out spaCy, a free open source library for natural language processing in Python. This will be the first in a series of articles on using spaCy for text analytics.

In this first article, we’ll set the stage by introducing spaCy. You’ll learn how to install spaCy, load language models, and carry out basic text processing tasks such as tokenization, lemmatization, and stopword removal. We’ll also briefly compare spaCy with other popular NLP libraries like NLTK and TextBlob, so you can see how it fits into the wider ecosystem.

Let’s get started.

Why spaCy?

With so many NLP and text analytics frameworks available, what sets spaCy apart from some of the others?

• Performance and Efficiency: spaCy is optimized for production use, focusing on speed and memory efficiency
• Easy-to-Use API: Its Pythonic API is intuitive, making it accessible for both beginners and experienced NLP practitioners
• Industrial Strength Capabilities: spaCy shines with advanced features like dependency parsing and named entity recognition, all available out of the box

Other popular libraries — like NLTK, TextBlob, or StanfordNLP — are often more academic or come with different design philosophies. spaCy’s sweet spot is bridging research and production with a clean, efficient design.

Installation and Setting Up spaCy

Installing spaCy

Here’s how to get the library installed:

pip install spacy

Downloading Language Models

After installing spaCy, you need to download the language models (e.g., the English model):

python -m spacy download en_core_web_sm

Pro Tip: spaCy offers multiple models (small, medium, large) for different languages. The sm models are lightweight and great for beginners or quick prototypes.

Basic Usage

Here’s a basic Python script using spaCy:

import spacy

# Load the English language model
nlp = spacy.load("en_core_web_sm")

text = """
Deep in the Pacific Ocean, an octopus shifted its color from a dull brown to an electric blue,
blending seamlessly with the coral. It was a survival instinct honed over millennia. Nearby, 
a robotic submersible whirred softly as it recorded data, its cameras scanning for signs of 
bioluminescent creatures. ‘These depths hold more mysteries than we can imagine,’ Dr. Patel 
murmured into her headset, jotting down observations. Above, aboard the research vessel, a 
team of marine biologists analyzed real-time footage, debating whether they had just discovered 
a new species. The ocean, vast and unpredictable, whispered its secrets in waves, waiting for
those patient enough to listen.
"""

doc = nlp(text)

print([token.text for token in doc])

In this snippet, doc becomes a spaCy “Doc” object that holds tokenized text and linguistic annotations.

Here is an explanation of the code above:

• import spacy: Brings in the spaCy library so you can use its natural language processing features
• nlp = spacy.load(“en_core_web_sm”): Loads a small, pre-trained English language model, which includes rules and statistical algorithms for tokenizing text, tagging parts of speech, recognizing named entities, etc.
• text = “spaCy is a great tool for natural language processing.”: Defines a simple string of text that you want to process
• doc = nlp(text): Passes the text through the spaCy pipeline, which splits it into tokens and applies language analysis (e.g., POS tagging); the result is a Doc object, which is spaCy’s main container for the processed text and its annotations
• print([token.text for token in doc]): Iterates over each token in the Doc object and prints the text of each token in a list, which is how you can see the individual words or tokens that spaCy has identified in your sentence

If you run that code, you will get something like the following as output:

['\n', 'Deep', 'in', 'the', 'Pacific', 'Ocean', ',', 'an', 'octopus', 'shifted', 'its', 'color',
'from', 'a', 'dull', 'brown', 'to', 'an', 'electric', 'blue', ',', '\n', 'blending', 'seamlessly', 
'with', 'the', 'coral', '.', 'It', 'was', 'a', 'survival', 'instinct', 'honed', 'over', 'millennia', 
'.', 'Nearby', ',', '\n', 'a', 'robotic', 'submersible', 'whirred', 'softly', 'as', 'it', 'recorded', 
'data', ',', 'its', 'cameras', 'scanning', 'for', 'signs', 'of', '\n', 'bioluminescent', 'creatures', 
'.', '‘', 'These', 'depths', 'hold', 'more', 'mysteries', 'than', 'we', 'can', 'imagine', ',', '’', 
'Dr.', 'Patel', '\n', 'murmured', 'into', 'her', 'headset', ',', 'jotting', 'down', 'observations', 
'.', 'Above', ',', 'aboard', 'the', 'research', 'vessel', ',', 'a', '\n', 'team', 'of', 'marine', 
'biologists', 'analyzed', 'real', '-', 'time', 'footage', ',', 'debating', 'whether', 'they', 'had', 
'just', 'discovered', '\n', 'a', 'new', 'species', '.', 'The', 'ocean', ',', 'vast', 'and', 
'unpredictable', ',', 'whispered', 'its', 'secrets', 'in', 'waves', ',', 'waiting', 'for', 
'\n', 'those', 'patient', 'enough', 'to', 'listen', '.', '\n']

Core Concepts: Tokens, Sentences, and POS Tagging

Tokenization

Tokens are individual words or symbols in NLP. spaCy automatically splits text into tokens when you pass text through the nlp pipeline.

Here’s an example:

for token in doc:
    print(token.text)

This is another way of expressing the same thing as the last line in the script above, and the output would be the same. Our text chunk is getting tokenized into words and symbols, saved to a list in order of appearance in the original text chunk, and then output to screen.

Sentences

Interested in sentences as opposed to words (tokens)? spaCy identifies sentence boundaries by default. You can iterate over sentences in a Doc object. Let’s do so and output them to the console.

for sent in doc.sents:
    print(sent.text)

Deep in the Pacific Ocean, an octopus shifted its color from a dull brown to an electric blue,
blending seamlessly with the coral.
It was a survival instinct honed over millennia.
Nearby, 
a robotic submersible whirred softly as it recorded data, its cameras scanning for signs of 
bioluminescent creatures.
‘These depths hold more mysteries than we can imagine,’ Dr. Patel 
murmured into her headset, jotting down observations.
Above, aboard the research vessel, a 
team of marine biologists analyzed real-time footage, debating whether they had just discovered 
a new species.
The ocean, vast and unpredictable, whispered its secrets in waves, waiting for
those patient enough to listen.

Note that as we split up our text in our example across numerous lines for legibility (and did not do so in agreement with sentence lengths), some of the reported “sentences” are incomplete, and for that reason.

Part-of-Speech (POS) Tagging

POS tagging can be defined as the process of identifying the particular part of speech to which a given word belongs, based on context and definition. In spaCy, each token has attributes like pos_ (coarse-grained tag) and tag_ (fine-grained tag) based on the predictions that the library is able to make about a given word’s POS. A coarse-grained tag could be NOUN, while a fine-grained tag might be NN (singular noun), NNS (plural noun), NNP (proper noun), etc. You can find out more about POS coarse- and fine-grained tags here.

Here is an example with a sample of the output:

for token in doc:
    print(token.text, token.pos_, token.tag_)

Deep PROPN NNP
in ADP IN
the DET DT
Pacific PROPN NNP
Ocean PROPN NNP
, PUNCT ,

Text Processing Basics

Let’s explore how to perform a few other core text processing tasks in spaCy.

Lemmatization

Lemmatization reduces a word to its base form (lemma). This is useful in text analytics to group together different forms of the same word (e.g., “runs,” “running,” “ran” → “run”).

Example:

for token in doc:
    print(token.text, "->", token.lemma_)

Sample output:

a -> a
robotic -> robotic
submersible -> submersible
whirred -> whir
softly -> softly
as -> as
it -> it
recorded -> record
data -> datum

Stop Word Removal

Stop words are commonly used words (like “the,” “is,” “in”) that often don’t carry meaningful information. spaCy includes a default stop word list.

Example:

spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS
filtered_tokens = [token for token in doc if not token.is_stop]
print("Filtered tokens:", [t.text for t in filtered_tokens])

Output (filtered words are those words which are not stop words):

Filtered tokens: ['\n', 'Deep', 'Pacific', 'Ocean', ',', 'octopus', 'shifted', 'color', 'dull', 'brown', 'electric', 
'blue', ',', '\n', 'blending', 'seamlessly', 'coral', '.', 'survival', 'instinct', 'honed', 'millennia', '.', 'Nearby', 
',', '\n', 'robotic', 'submersible', 'whirred', 'softly', 'recorded', 'data', ',', 'cameras', 'scanning', 'signs', '\n', 
'bioluminescent', 'creatures', '.', '‘', 'depths', 'hold', 'mysteries', 'imagine', ',', '’', 'Dr.', 'Patel', '\n', 
'murmured', 'headset', ',', 'jotting', 'observations', '.', ',', 'aboard', 'research', 'vessel', ',', '\n', 'team', 
'marine', 'biologists', 'analyzed', 'real', '-', 'time', 'footage', ',', 'debating', 'discovered', '\n', 'new', 'species', 
'.', 'ocean', ',', 'vast', 'unpredictable', ',', 'whispered', 'secrets', 'waves', ',', 'waiting', '\n', 'patient', 
'listen', '.', '\n']

Next Steps

Congratulations! You’ve taken your first step into the world of spaCy. By installing the library, downloading models, and exploring how to tokenize text, identify sentences, and extract linguistic features like POS tags, you now have a solid foundation.

In the next article, we’ll dive deeper into spaCy’s linguistic features, focusing on dependency parsing and named entity recognition — two powerful tools for gaining deeper insights from text data.