Image by Editor | Midjourney

Blood pressure is a fundamental vital sign that exhibits natural variation across populations, making it an excellent case study for statistical distribution analysis. In this article, we’ll explore how to systematically analyze standardized blood pressure measurements from the scikit-learn diabetes dataset to determine the most appropriate statistical distribution. This example will demonstrate key techniques that you can apply to your own datasets, whether you’re working with medical measurements, manufacturing tolerances, or customer satisfaction scores.

Understanding Your Dataset

Before diving into distribution analysis, it’s helpful to understand your data’s context and characteristics. The dataset of our choice contains standardized values (mean = 0, standard deviation = 1) from a medical study. This standardization allows us to focus on the shape and patterns of the distribution without worrying about the original units of measurement.

from sklearn.datasets import load_diabetes
import numpy as np
import matplotlib.pyplot as plt

# Load the diabetes dataset
diabetes = load_diabetes()
bp_data = diabetes.data[:, 3]  # Blood pressure column

print("First few blood pressure measurements (standardized):")
print(bp_data[:5])
print(f"\nTotal observations: {len(bp_data)}")

This gives us:

First few blood pressure measurements (standardized):
[ 0.02187239 -0.02632753 -0.00567042 -0.03665608  0.02187239]

Total observations: 442

Systematic Data Exploration

When choosing a statistical distribution, we need to examine several key characteristics of our data. The following function helps us systematically analyze these properties:

def explore_data_characteristics(data):
    """Analyze key characteristics to help choose between continuous and discrete distributions"""
    # Basic statistics
    print("Data Characteristics:")
    print(f"Sample size: {len(data)}")
    print(f"Unique values: {len(np.unique(data))}")
    print(f"Min value: {np.min(data):.2f}")
    print(f"Max value: {np.max(data):.2f}")

    # Check if values are mostly whole numbers
    whole_numbers = np.sum(np.abs(data - np.round(data)) < 1e-10)
    print(f"\nDiscrete vs Continuous indicators:")
    print(f"Proportion of whole numbers: {whole_numbers/len(data):.2%}")

    # Look at decimal precision
    decimal_places = [str(x)[::-1].find('.') for x in data if '.' in str(x)]
    if decimal_places:
        print(f"Maximum decimal precision: {max(decimal_places)} places")

    # Show value spacing
    sorted_unique = np.sort(np.unique(data))
    gaps = np.diff(sorted_unique)
    print(f"Minimum gap between values: {np.min(gaps):.3f}")
    print(f"Median gap between values: {np.median(gaps):.3f}")

# Run the exploration
explore_data_characteristics(bp_data)

The analysis reveals:

Data Characteristics:
Sample size: 442
Unique values: 100
Min value: -0.11
Max value: 0.13

Discrete vs Continuous indicators:
Proportion of whole numbers: 0.00%
Maximum decimal precision: 19 places
Minimum gap between values: 0.001
Median gap between values: 0.002

Let’s break down what these metrics tell us:

1. Sample Size and Coverage: With 442 observations and 100 unique values, we have a robust dataset that captures the natural variation in blood pressure measurements.
2. Value Range: The standardized measurements span from -0.11 to 0.13, which is typical for normalized medical data and suggests effective standardization.
3. Continuity Assessment:
   • Zero whole numbers in the dataset
   • High decimal precision (19 places)
   • Small, consistent gaps between values (minimum 0.001)

These characteristics strongly indicate that we’re working with continuous rather than discrete data, which will influence our choice of distribution families.

Visualizing Distribution Patterns

While numerical summaries are valuable, visualization often reveals patterns that numbers alone might miss. Let’s create a histogram to better understand our data’s distribution:

plt.figure(figsize=(10, 6))
plt.hist(bp_data, bins='auto', density=True, edgecolor='black')
plt.title('Distribution of Blood Pressure Measurements')
plt.xlabel('Blood Pressure (standardized)')
plt.ylabel('Density')
plt.grid(True, alpha=0.3)
plt.show()

Image By Author

Initial Distribution Fitting

Based on our analysis showing continuous characteristics, let’s use Phitter to identify potential distribution matches:

# Fit with Phitter
import phitter

# Initialize Phitter with our data
phi = phitter.PHITTER(
    data=bp_data,
    fit_type="continuous",  # Based on our analysis
    confidence_level=0.95
)

# Fit distributions
phi.fit()

# Look at distribution ranking by SSE (Sum of Squared Errors)
print("\nTop distributions by fit:")
print(phi.df_sorted_distributions_sse[['distribution', 'sse']].head(3))

This gives us:

Top distributions by fit:
   distribution      sse
0    triangular  7.346556
1      maxwell   7.928763
2    johnson_sb  8.458665

Our analysis using Phitter reveals several key insights:

1. Distribution Rankings: The triangular, Maxwell, and Johnson SB distributions show promising fits, with the triangular distribution having the lowest Sum of Squared Errors (SSE).
2. Practical Implications: While the triangular distribution provides the best mathematical fit, the slight asymmetry in our histogram suggests that the Johnson SB distribution might be more appropriate for modeling real-world blood pressure variations.
3. Next Steps: To make a final selection, we should consider:
   • The theoretical basis for blood pressure variation in medical literature
   • The specific requirements of our analysis (e.g., tail behavior importance)
   • The trade-off between fit accuracy and model simplicity

To better understand how these distributions fit our data, let’s visualize them alongside our histogram using Phitter’s built-in plotting functionality.

# Plot the fitted distributions
phi.plot_histogram_distributions()

The resulting plot shows our histogram overlaid with the probability density functions of several fitted distributions, allowing us to visually assess the quality of each fit.

Image By Author

Key Takeaways

This analysis demonstrates a systematic approach to distribution selection:

1. Start with thorough data exploration to understand basic characteristics
2. Use visualization to identify patterns and potential distribution families
3. Apply quantitative fitting techniques to rank potential distributions
4. Consider both statistical fit and practical implications when making the final selection

Remember that choosing the right distribution is both an art and a science – while mathematical fits are important, your domain knowledge and analysis requirements should guide the final decision.