Descriptive statistics summarize and describe the main characteristics of a dataset using numerical measures. These summaries are essential for understanding data patterns, identifying outliers, and communicating key findings effectively. SciPy’s stats module provides a comprehensive set of functions for computing these statistics efficiently, making it a powerful tool for data analysis in Python.
Getting Started with SciPy Stats
First, let’s import the necessary modules:
from scipy import stats import numpy as np # only for array creation
Basic Descriptive Statistics Using stats.describe()
The stats.describe() function is one of SciPy’s most useful tools for quick statistical analysis. Let’s see it in action:
# Create sample data data = np.array([24, 23, 27, 25, 32, 28, 29, 26, 30, 25]) # Calculate descriptive statistics description = stats.describe(data) print(description)
DescribeResult(nobs=10, minmax=(23, 32), mean=26.9, variance=8.100000000000001, skewness=0.38042981252857955, kurtosis=-0.8952000316121635)
While this output provides a comprehensive overview, you might want to access individual components for more detailed analysis. Here’s how to break down each component:
# Accessing individual components of the description
print("Number of observations:", description.nobs)
print("Minimum and Maximum:", description.minmax) # returns a tuple (min, max)
print("Mean:", description.mean)
print("Variance:", description.variance)
print("Skewness:", description.skewness)
print("Kurtosis:", description.kurtosis)
Number of observations: 10 Minimum and Maximum: (23, 32) Mean: 26.9 Variance: 8.100000000000001 Skewness: 0.38042981252857955 Kurtosis: -0.8952000316121635
Looking at these results, we can interpret that:
- Our dataset has 10 observations ranging from 23 to 32
- The mean of 26.9 represents the average value
- The positive skewness (0.380) indicates a slight right skew
- The negative kurtosis (-0.895) suggests lighter tails than a normal distribution
Alternative Measures of Central Tendency
When dealing with data that might contain outliers or when your data doesn’t follow a normal distribution, SciPy offers several alternative measures of central tendency:
# Sample data
data = np.array([22, 24, 25, 24, 25, 23, 24, 100])
# Calculate different types of means
geometric_mean = stats.gmean(data)
harmonic_mean = stats.hmean(data)
trimmed_mean = stats.trim_mean(data, proportiontocut=0.2)
mode = stats.mode(data, keepdims=True)
print(f"Geometric Mean: {geometric_mean:.2f}")
print(f"Harmonic Mean: {harmonic_mean:.2f}")
print(f"20% Trimmed Mean: {trimmed_mean:.2f}")
print(f"Mode: {mode.mode[0]}, Count: {mode.count[0]}")
Geometric Mean: 28.52 Harmonic Mean: 26.32 10% Trimmed Mean: 24.17 Mode: 24, Count: 3
Notice how these different means provide varying perspectives on our data:
- The geometric mean (28.52) is useful for data that follows geometric progression
- The harmonic mean (26.32) is lower than the geometric mean, which is typical for positive data
- The trimmed mean (24.17) excludes the most extreme values
- The mode shows that 24 appears most frequently (3 times)
Advanced Statistical Analysis: Moments and Percentiles
For more sophisticated analysis, SciPy provides tools to examine statistical moments and percentile-based statistics:
# Generate sample data
data = np.array([24, 25, 26, 26, 25, 24, 25, 26, 28, 29, 30, 32, 48])
# Calculate statistical moments
first_moment = stats.moment(data, moment=1) # Should be close to 0 (mean-centered)
second_moment = stats.moment(data, moment=2) # Related to variance
third_moment = stats.moment(data, moment=3) # Related to skewness
fourth_moment = stats.moment(data, moment=4) # Related to kurtosis
# Calculate percentile-based statistics
percentiles = stats.mstats.mquantiles(data, prob=[0.25, 0.5, 0.75])
iqr = stats.iqr(data) # Interquartile Range
median_deviation = stats.median_abs_deviation(data) # Robust measure of variability
print("Statistical Moments:")
print(f"First Moment: {first_moment:.3f}")
print(f"Second Moment: {second_moment:.3f}")
print(f"Third Moment: {third_moment:.3f}")
print(f"Fourth Moment: {fourth_moment:.3f}\n")
print("Percentile Statistics:")
print(f"25th Percentile: {percentiles[0]:.3f}")
print(f"Median (50th): {percentiles[1]:.3f}")
print(f"75th Percentile: {percentiles[2]:.3f}")
print(f"Interquartile Range: {iqr:.3f}")
print(f"Median Absolute Deviation: {median_deviation:.3f}")
Statistical Moments: First Moment: 0.000 Second Moment: 37.751 Third Moment: 568.200 Fourth Moment: 11669.685 Percentile Statistics: 25th Percentile: 25.000 Median (50th): 26.000 75th Percentile: 29.300 Interquartile Range: 4.000 Median Absolute Deviation: 2.000
The moments provide insights into the shape of our distribution:
- First moment near 0 confirms it’s mean-centered
- Second moment (37.751) relates to the spread of the data
- The large third moment (568.200) indicates right skewness
- The fourth moment (11669.685) suggests heavy tails
The percentile statistics offer robust measures of central tendency and spread:
- The median (26.000) is resistant to outliers
- The interquartile range (4.000) shows the spread of the middle 50% of the data
- The median absolute deviation (2.000) provides a robust measure of variability
Conclusion
Descriptive statistics is a vast field encompassing many different techniques and approaches for summarizing data. While this tutorial covered some fundamental aspects, SciPy’s stats module offers a much more comprehensive suite of statistical tools. A good place to start your exploration using this package is with this list of methods that focus on summary statistics. Once you have experimented with the basics, you can move onto advanced tools that cover frequency statistics and masked statistics. Whether you’re performing basic exploratory data analysis or conducting advanced statistical research, SciPy’s statistical functions provide reliable, efficient, and well-documented tools for your data analysis needs.
