Text analysis doesn’t have to be complicated. Let’s build a practical toolkit for analyzing text using Python’s textstat library. We’ll create working code you can use right away.
What is textstat, you ask? Textstat is a Python library that calculates statistical measures from text. These statistics include basic metrics — such as word count and sentence length — along with various readability scores — including Flesch Reading Ease and Gunning Fog Index. Textstat is particularly useful for quantifying text complexity and readability. This makes the library valuable for content analysis and assessment, as well as document classification tasks.
In order to make use of textstat, we will need to install the library, along with a few requirements.
pip install textstat pandas matplotlib
Basic Text Analysis
Let’s first build a set of foundational functions for text analysis.
import textstat
import pandas as pd
import matplotlib.pyplot as plt
def get_text_stats(text):
"""Get all basic statistics for a text"""
return {
'words': textstat.lexicon_count(text, removepunct=True),
'sentences': textstat.sentence_count(text),
'syllables': textstat.syllable_count(text),
'chars': textstat.char_count(text),
'letters': textstat.letter_count(text),
'avg_words_sentence': textstat.avg_sentence_length(text),
'avg_syllables_word': textstat.avg_syllables_per_word(text)
}
Let’s create some sample texts so that we can analyze different types of writing:
texts = {
'academic': """The results indicate a statistically significant relationship
(p < 0.05) between variables X and Y. Further analysis revealed
additional correlations.""",
'news': """City council approved the budget yesterday. Local businesses
praised the decision. The mayor called it a historic moment.""",
'story': """The old cat stretched lazily in the sun. Her whiskers twitched
as a butterfly flew past. She yawned."""
}
Finally, let's use the function to analyze the sample texts, and output the results.
# Create a DataFrame of stats
df = pd.DataFrame({name: get_text_stats(text)
for name, text in texts.items()})
print("\nText Statistics:")
print(df)
Text Statistics:
academic news story
words 19.0 18.0 18.0
sentences 3.0 3.0 2.0
syllables 44.0 31.0 21.0
chars 131.0 104.0 83.0
letters 125.0 101.0 80.0
avg_words_sentence 6.3 6.0 9.0
avg_syllables_word 2.3 1.7 1.2
Analyzing Multiple Files
Let's see how this works with multiple files of our own, instead of our toy sample texts. Let's say you have a folder of documents you would like to analyze. Here's a helper function that could help us with that.
import glob
def analyze_text_files(pattern):
"""Analyze all text files matching the pattern"""
stats = {}
for file in glob.glob(pattern):
with open(file, 'r') as f:
text = f.read()
stats[file] = get_text_stats(text)
return pd.DataFrame(stats)
And here's how we would use it:
stats_df = analyze_text_files('documents/*.txt')
Comparing Texts
Let's say you want to see how 2 text differ. We can use the following function to help us accomplish this.
def compare_texts(text1, text2):
"""Compare statistics between two texts"""
stats1 = get_text_stats(text1)
stats2 = get_text_stats(text2)
return pd.DataFrame({
'text1': stats1,
'text2': stats2,
'difference': {k: stats1[k] - stats2[k]
for k in stats1.keys()}
})
And here is how we could use this function to compare our sample academic and news articles from earlier:
academic = texts['academic']
news = texts['news']
comparison = compare_texts(academic, news)
print("\nComparison:")
print(comparison)
And here's what we get:
Comparison:
text1 text2 difference
words 19.0 18.0 1.0
sentences 3.0 3.0 0.0
syllables 44.0 31.0 13.0
chars 131.0 104.0 27.0
letters 125.0 101.0 24.0
avg_words_sentence 6.3 6.0 0.3
avg_syllables_word 2.3 1.7 0.6
Visualizing Our Results
The following function can plot the results of our summary statistics.
def plot_stats(df, stat='words'):
"""Plot a specific statistic across texts."""
df.loc[stat].plot(kind='bar')
plt.title(f'{stat.title()} by Document')
plt.ylabel(f'Number of {stat.title()}')
plt.tight_layout()
plt.show()
Here is how to use it on the words of our text:
plot_stats(df, 'words')
And here's what we get:
Common Pitfalls and Solutions
We have a good start on building a toolkit to help with our future text statistical analysis. There are a few common issues you may encounter in your pursuits, however, and so here are a few of the more common problems and how to solve them:
Text Cleaning
Is your text unclean? Here is a sample cleaning function to get you started in cleaning up your text pre-analysis. Remember that each scenario is unique, however, and that this may or may not work for you as is. It should give you some ideas of typical data cleaning issues that you might run into.
def clean_text(text):
"""Basic text cleaning for statistical analysis"""
import re
# Convert to lowercase for consistency
text = text.lower()
# Replace common contractions
text = text.replace("'s", " is").replace("n't", " not")
# Replace newlines and excess spaces
text = text.replace('\n', ' ').replace('\r', ' ')
# Remove special characters but keep sentence-ending punctuation
text = re.sub(r'[^a-z0-9\s.,!?]', '', text)
# Standardize sentence-ending punctuation spacing
text = re.sub(r'([.,!?])', r'\1 ', text)
# Fix multiple spaces
text = re.sub(r'\s+', ' ', text)
return text.strip()
Sentence Issue Detection
Having problems with some of your sentences, and want to investigate? Here is a function for that. Running this should be able to identify and report back on issues that you might be having, including potential fragments, potential missed sentence tokenizing, and potential abbreviation issues.
def check_sentences(text):
"""Check for common sentence detection issues and return problematic sentences"""
sentences = textstat.sentence_count(text, return_sentences=True)
issues = []
for i, sent in enumerate(sentences, 1):
# Check for very short sentences (potential fragments)
if len(sent.split()) < 3:
issues.append(f"Short sentence #{i}: {sent.strip()}")
# Check for very long sentences (potential missed breaks)
if len(sent.split()) > 40:
issues.append(f"Long sentence #{i}: {sent.strip()}")
# Check for common abbreviations that might break sentence detection
if any(abbr in sent for abbr in ['Mr.', 'Dr.', 'Mrs.', 'Ms.', 'Ph.D.', 'i.e.', 'e.g.']):
issues.append(f"Potential abbreviation issue #{i}: {sent.strip()}")
return issues
File Encoding
Common issues with reading text files are those related to encoding. If experiencing any of these, try forcing your text into UTF-8 encoding while reading with the following function.
def safe_read(filename):
"""Safely read text files"""
try:
with open(filename, 'r', encoding='utf-8') as f:
return f.read()
except UnicodeDecodeError:
with open(filename, 'r', encoding='latin-1') as f:
return f.read()
Wrapping Up
This has been an introduction to using textstat for performing statistical analysis on text data, and is but the first in our series. Keep an eye out for more information on how to use this library for additional statistical analysis tasks in the near future.
