Introduction

In previous articles, we were introduced to the Python natural language processing (NLP) library spaCy, and saw how to use it for exploring linguistic features.

In this article, we’ll focus on how to prepare text data for machine learning and statistical modeling. Preprocessing is often the most time-consuming yet crucial step in NLP workflows. Specifically we will be covering:

• advanced text preprocessing techniques (custom pipelines, phrase matching)
• feature engineering methods (bag-of-words, TF-IDF, word embeddings)
• integrating spaCy with machine learning libraries like Scikit-learn for tasks like sentiment analysis or topic modeling

Advanced Text Preprocessing Techniques

Let’s start by looking at some advanced text preprocessing techniques.

Custom Pipeline Components

spaCy’s NLP pipelines are modular systems that process text through a series of components, with each component performing a specific linguistic analysis task, including tokenization, part-of-speech tagging, dependency parsing, etc. The pipeline architecture allows users to customize which components are included and in what order they run, making it easy to optimize for specific use cases or performance requirements. Each component in the pipeline can access all annotations added by previous components, enabling rich linguistic analysis where later components build upon the foundation laid by earlier ones.

spaCy’s NLP pipeline | Image source: spaCy

spaCy provides pre-trained pipeline models for multiple languages, which can be loaded with a single line of code, though users can also train custom pipelines on their own data to better handle specific domains or tasks. If you want to perform a custom operation — say a specialized tokenizer or a filter — you can create a custom component:

import spacy
from spacy.tokens import Doc
from spacy.language import Language

# Register the custom extension attribute on Doc
if not Doc.has_extension("filtered_tokens"):
    Doc.set_extension("filtered_tokens", default=None)

nlp = spacy.load("en_core_web_sm")

@Language.component("custom_component")
def custom_component(doc):
    # Filter out tokens with length <= 1 (using token.text for clarity)
    filtered_tokens = [token for token in doc if len(token.text) > 1]
    
	# Assign to the registered extension attribute
    doc._.filtered_tokens = filtered_tokens
    return doc

# Add custom component to pipeline
nlp.add_pipe("custom_component", last=True)

doc = nlp("This is a sample text.")
print([t.text for t in doc._.filtered_tokens])

Output:

['This', 'sample', 'text']

This approach keeps all your preprocessing steps within spaCy’s pipeline, making it easy to maintain. It also provides for a very powerful and flexible design approach for your NLP projects.

Phrase Matching

spaCy’s Matcher and PhraseMatcher are versatile tools designed for rule-based pattern matching within text documents.

The Matcher allows you to define sophisticated rules that identify sequences of tokens based on various linguistic attributes such as part-of-speech tags, lemmas, and syntactic dependencies, making it ideal for capturing complex patterns in natural language. You could use the Matcher to extract technical terminology from scientific articles or identify key phrases in legal documents, helping automate data extraction and categorization tasks.

In contrast, the PhraseMatcher is optimized for quickly locating exact phrases or expressions, which is particularly useful when working with extensive lists of fixed terms. The PhraseMatcher is well-suited for detecting brand names in social media posts, flagging specific idiomatic expressions in customer feedback, or filtering out content based on predefined keywords.

import spacy
from spacy.matcher import PhraseMatcher

# Load the spaCy model to create the nlp object
nlp = spacy.load("en_core_web_sm")

# Create Matcher object, add terms to patterns, add pattern to matcher object
matcher = PhraseMatcher(nlp.vocab)
terms = ["machine learning", "natural language processing"]
patterns = [nlp.make_doc(text) for text in terms]
matcher.add("TECH_TERMS", patterns)

doc = nlp("I like studying machine learning and natural language processing.")

# Check for matches, print results
matches = matcher(doc)
for match_id, start, end in matches:
    span = doc[start:end]
    print("Matched span:", span.text)

Output:

Matched span: machine learning
Matched span: natural language processing

Feature Engineering for NLP

Now we turn our attention to using spaCy for feature engineering functionality.

Bag-of-Words (BoW) and TF-IDF

Traditional but still powerful feature engineering methods include bag-of-words (BoW) and TF-IDF. BoW represents documents as vectors of word counts, while TF-IDF weighs terms by their frequency and how common they are across documents.

To leverage spaCy’s tokenization while using Scikit-learn’s vectorizers, you can create a custom tokenizer that uses spaCy’s capabilities:

import spacy
from sklearn.feature_extraction.text import TfidfVectorizer

nlp = spacy.load("en_core_web_sm")

# Custom tokenizer
def spacy_tokenizer(text):
    doc = nlp(text)
    return [token.lemma_.lower() for token in doc if not token.is_stop and token.is_alpha]

text_data = [
    "Natural language processing is a field of AI.",
    "I love working with machine learning models."
]

tfidf = TfidfVectorizer(tokenizer=spacy_tokenizer)
X = tfidf.fit_transform(text_data)

print(tfidf.get_feature_names_out())
print(X.toarray())

Output:

['ai' 'field' 'language' 'learning' 'love' 'machine' 'model' 'natural'
 'processing' 'work']
[[0.4472136 0.4472136 0.4472136 0.        0.        0.        0.
  0.4472136 0.4472136 0.       ]
 [0.        0.        0.        0.4472136 0.4472136 0.4472136 0.4472136
  0.        0.        0.4472136]]

This way, you get the best of both worlds: spaCy’s robust tokenization + Scikit-learn’s proven feature extraction techniques.

Word Embeddings

spaCy provides word vectors if you load larger models, such as en_core_web_md or en_core_web_lg. Each token has a .vector attribute, which can be used as input to machine learning models. If you need domain-specific embeddings (e.g., legal or biomedical), consider training your own or using pretrained vectors tailored to your domain.

nlp_md = spacy.load("en_core_web_md")
doc_md = nlp_md("This is a sentence.")

for token in doc_md:
    print(token.text, token.vector[:5])  # Just show the first 5 dimensions

Integrating spaCy with Machine Learning

After you’ve transformed text into numerical features (via BoW, TF-IDF, embeddings), you can feed these into scikit-learn (or other libraries like TensorFlow or PyTorch) for tasks such as sentiment analysis, topic modeling, and text classification.

Let’s consider an example. Imagine you have a dataset of movie reviews or customer feedback. Here’s a quick outline:

1. Load your data and convert to a list of text reviews
2. Create a custom spaCy tokenizer that removes stopwords, punctuation, and applies lemmatization
3. Vectorize your text using TF-IDF or embeddings
4. Train a sentiment classifier using scikit-learn’s Logistic Regression or any other classifier
5. Evaluate performance with metrics like accuracy, precision, recall, or F1-score

This workflow demonstrates the power of combining spaCy’s linguistic preprocessing with machine learning libraries.

Now let’s look at some code, in which we build a simple sentiment classifier with TF-IDF + logistic regression using Scikit-learn and spaCy.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Example: 1 for positive, 0 for negative
labels = [1, 0]
X_train, X_test, y_train, y_test = train_test_split(
    X, labels, test_size=0.5, random_state=42
)

clf = LogisticRegression()
clf.fit(X_train, y_train)
preds = clf.predict(X_test)
print("Accuracy:", accuracy_score(y_test, preds))

Next Steps

You’ve now seen how to use spaCy for more advanced preprocessing and how to extract meaningful features from text. This stage is crucial before feeding the data into any NLP or ML pipeline. In the next articles, we’ll look at building custom NLP pipelines and eventually explore full end-to-end projects to showcase spaCy’s capabilities in real-world text analytics.

Key takeaways from today include:

• Customization in spaCy makes it easy to tailor preprocessing to your specific domain needs.
• Feature Engineering is critical. Traditional methods like TF-IDF still hold value, and embeddings can provide richer contextual information.
• Seamless Integration with Scikit-learn allows you to quickly transition from raw text to trainable features.