Textstat is a Python library that helps us understand how complex our text is through the us of statistical measures. In our previous articles, we introduced its basic functionality as well as using it to calculate and interpret readability metrics.

Textstat can also be used to perform statistical analysis of text properties, which are measures of how easy or difficult a text is to read and understand. It does this by analyzing factors including sentence length, word length, syllable count, word complexity, and structural patterns. These factors then combine into a standardized readability score which can be used to estimate the grade level needed to understand the text, the relative reading difficulty — such as from “easy” to “very difficult” — and the comprehension level for different audiences.

For example, a particular piece of text with long sentences and high word complexity (many multisyllabic words) will score as more difficult to read when compared with a short sentences and simple words. These measurements can then be used to help match text complexity to the intended audience.

Let’s use statistical analysis to understand text structure, complexity, and writing style patterns. If you have yet to read the two previous articles, I suggest you start there first.

Statistical Analysis of Text Properties

Let’s begin with a function that calculates and returns the statistical properties of sentences, along with our required imports.

import textstat
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

def get_sentence_stats(text):
    """Calculate statistical properties of sentences"""
    sentences = textstat.sentence_count(text, return_sentences=True)
    lengths = [len(sent.split()) for sent in sentences]
    
    return {
        'mean_length': np.mean(lengths),
        'median_length': np.median(lengths),
        'std_length': np.std(lengths),
        'min_length': min(lengths),
        'max_length': max(lengths),
        'total_sentences': len(lengths),
        'variance': np.var(lengths)
    }

This isn’t revolutionary; we’re just using numpy and scipy to get basic statistics on the sentences from the text passed to the function, after splitting the sentences with textstat.

Now, let’s analyze how sentence lengths are distributed in different types of writing:

def analyze_distribution(text):
    """Analyze sentence length patterns"""
    sentence_count = textstat.sentence_count(text)
    
    # Split text into sentences using basic period detection
    sentences = text.replace('!', '.').replace('?', '.').split('.')
    sentences = [s.strip() for s in sentences if s.strip()]
    
    # Calculate lengths
    lengths = [len(sent.split()) for sent in sentences]
    
    # Ensure we have enough data for statistical analysis
    if len(lengths) < 3:
        raise ValueError("Text too short for statistical analysis")

    # Statistical tests and measures
    shapiro_stat, shapiro_p = stats.shapiro(lengths)
    skew = stats.skew(lengths)
    kurt = stats.kurtosis(lengths)
    
    return {
        'summary': pd.Series(lengths).describe(),
        'normality_p': shapiro_p,
        'skewness': skew,
        'kurtosis': kurt,
        'lengths': lengths
    }

# Example usage with different text types
texts = {
    'academic': """Machine learning algorithms exhibit varying performance characteristics across different data distributions. 
                  The empirical analysis of gradient descent optimization demonstrates consistent convergence patterns under 
                  specific conditions. Recent research indicates that neural network architectures with multiple hidden layers 
                  show superior performance on complex pattern recognition tasks. Furthermore, the statistical significance of 
                  these results (p < 0.01) suggests robust generalization capabilities.""",
    
    'news': """Local tech startup DataMind unveiled their new artificial intelligence platform yesterday. The system helps 
              businesses automate customer service operations. Early tests show 40% improvement in response times. 
              CEO Sarah Chen says the platform will create jobs rather than replace workers. The company plans to 
              expand services next month.""",
    
    'fiction': """The old computer rattled away in the back of the lab. Sarah typed frantically, her fingers dancing across 
                 the keyboard as lines of code filled the screen. The algorithm wasn't working. Again. She sighed and 
                 reached for her cold coffee. The clock showed 3 AM. Maybe the neural network just needed a different 
                 approach, she thought."""
}

results = {name: analyze_distribution(text) 
          for name, text in texts.items()}

# Format results for output
summary_df = DataFrame({
    name: {
        'mean': res['summary']['mean'],
        'std': res['summary']['std'],
        'normality_p': res['normality_p'],
        'skewness': res['skewness']
    }
    for name, res in results.items()
}).round(4)

print(summary_df)

And here is the result:

             academic    news  fiction
mean          11.8000  9.6000   7.8571
std            5.1672  2.3022   5.0143
normality_p    0.9708  0.0476   0.9701
skewness       0.1279  0.6817   0.2537

This should all be more or less straightforward, with perhaps the exception of the lines referring to shapiro_stat and shapiro_p.

The variable shapiro_stat from the Shapiro-Wilk test is a test statistic that quantifies how closely your data follows a normal distribution. The variable normality_p will hold the p-value from the Shapiro-Wilk test for normality, which also helps determine if our sentence lengths follow a normal distribution.

• The Shapiro-Wilk test result (shapiro_stat) assesses whether a sample of data comes from a normally distributed population
• The p-value (shapiro_p) tells us the probability of observing our data if it truly came from a normal distribution
• If p < 0.05, we typically reject the assumption of normality; if p ≥ 0.05, we fail to reject normality

In the context of text analysis, normal distributions of sentence lengths are actually quite rare. Most texts have: a positive skew with more short sentences than long; non-normal distributions with multiple peaks; patterns specific to their genre or author. As such, we would expect academic writing to show a different sentence length distribution than fiction. This is what leads us to be able to characterize writing styles with the help of normality testing.

Note that we have used a simplified sentence splitting approach in the above, as opposed to using a more robust technique such as incorporating one of NLTK's sentence tokenizers.

Comparing Writing Styles

Let's move on from sentence length distribution to something a little more interesting: writing style comparison? Here's how we can do so statistically:

def compare_texts(text1, text2):
    """Statistically compare two texts"""
    def get_sentence_lengths(text):
        sentences = text.replace('!', '.').replace('?', '.').split('.')
        sentences = [s.strip() for s in sentences if s.strip()]
        return [len(sent.split()) for sent in sentences]
    
    sent1 = get_sentence_lengths(text1)
    sent2 = get_sentence_lengths(text2)
    
    # Run statistical tests
    t_stat, t_p = stats.ttest_ind(sent1, sent2)
    ks_stat, ks_p = stats.ks_2samp(sent1, sent2)
    
    # Calculate effect size (Cohen's d)
    effect_size = (np.mean(sent1) - np.mean(sent2)) / np.sqrt(
        (np.var(sent1) + np.var(sent2)) / 2)
    
    return {
        'mean_difference': np.mean(sent1) - np.mean(sent2),
        't_test_p': t_p,
        'ks_test_p': ks_p,
        'effect_size': effect_size
    }

academic_vs_fiction = compare_texts(texts['academic'], texts['fiction'])

print("\nSummary Statistics:")
print(pd.Series(academic_vs_fiction).round(4))

Here's a walkthrough of the above function:

• The function processes two texts (text1, text2) into numerical distributions of sentence lengths
• These distributions become the basis for comparative statistical analysis
• The resulting sent1 and sent2 are lists of integers representing word counts per sentence
• Two statistical tests are applied, providing p-values indicating the statistical significance of differences:
  1. Independent t-test (stats.ttest_ind): Tests if the mean sentence lengths differ significantly
  2. Kolmogorov-Smirnov test (stats.ks_2samp): Tests if the sentence length distributions differ in shape
• Cohen's d is computed as the standardized difference between means using pooled variance
• This metric quantifies the magnitude of difference independent of sample size
• The function returns a comprehensive statistical comparison through a dictionary containing:
  • mean differences
  • significance tests (p-values)
  • effect size

And the result:

Summary Statistics:
mean_difference    3.9429
t_test_p           0.2141
ks_test_p          0.7374
effect_size        0.8512
dtype: float64

Interpreting Your Results

Here are some general rules for interpreting he results of our tests:

Basic Statistics

• Mean sentence length: 15-20 words is typical
• High variance suggests diverse writing style
• Outliers may indicate structural issues

Distribution Analysis

• Normal distribution isn't common
• Positive skew is typical (more short sentences)
• Compare results to genre norms

Wrapping Up

Armed with your new knowledge about using textstat, you can now you can statistically analyze your text's properties, and utilize the results to understand writing patterns and compare styles.

A few parting tips:

• Always check sample size (30+ sentences minimum)
• Consider effect size, not just significance
• Compare against similar text types
• Watch for outliers skewing results

And with that, our exploration of performing statistical text analysis with textstat is complete.