One of the most well-tested and trustworthy collection of tools in our data kit is descriptive statistics. When we want to understand and summarize data, descriptive statistics cover nearly everything we need to get a quick snapshot of our dataset. From mean to range to IQR to skewness and beyond, descriptive stats are fundamental in getting familiarized with data.

Here are five custom Python functions that can help you master key descriptive statistics concepts.

1. Comprehensive Summary Statistics

The first function on our list is a comprehensive overview of key descriptive statistics for a given dataset. This function takes as input a list of integers, computes the various descriptive statistics, and returns them all as a dictionary.

import numpy as np
import pandas as pd

def comprehensive_summary(data):
    """
    Calculate comprehensive summary statistics for a given dataset.
    
    :param data: list of integers, non-zero length
    :return: dict containing descriptive statistics summary
    """
    summary = {
        "Mean": np.mean(data),
        "Median": np.median(data),
        "Mode": pd.Series(data).mode().values[0],
        "Standard Deviation": np.std(data),
        "Variance": np.var(data),
        "Minimum": np.min(data),
        "Maximum": np.max(data),
        "Range": np.ptp(data),
        "Q1": np.percentile(data, 25),
        "Q3": np.percentile(data, 75),
        "IQR": np.percentile(data, 75) - np.percentile(data, 25),
        "Skewness": pd.Series(data).skew(),
        "Kurtosis": pd.Series(data).kurtosis()
    }
    return summary

Now let’s test out the function on some data.

data = [1, 2, 2, 3, 4, 5, 5, 6, 7, 8, 9]
print(comprehensive_summary(data))

Output:

{'Mean': 4.7272727272727275, 
 'Median': 5.0, 
 'Mode': 2, 
 'Standard Deviation': 2.4896479886598457, 
 'Variance': 6.198347107438016, 
 'Minimum': 1, 
 'Maximum': 9, 
 'Range': 8, 
 'Q1': 2.5, 
 'Q3': 6.5, 
 'IQR': 4.0, 
 'Skewness': 0.19914483841331795, 
 'Kurtosis': -1.0505244444444441}

2. Frequency Distribution

The next function calculates the frequency distribution of a dataset, either using unique values or specified bins. If no number of bins is specified, unique values are used.

import numpy as np

def frequency_distribution(data, bins=None):
    """
    Calculate the frequency distribution of a dataset.
    If bins is None, use unique values. Otherwise, create bins.
    
    :param data: list of integers, non-zero length
    :param bins: number of bins to use for frequency binning
    :return: dict containing frequency distribution
    """
    if bins is None:
        unique, counts = np.unique(data, return_counts=True)
        return dict(zip(unique, counts))
    else:
        hist, bin_edges = np.histogram(data, bins=bins)
        return dict(zip(bin_edges[:-1], hist))

Let’s see what happens when we call the function with some data.

data = [1, 2, 2, 3, 3, 3, 4, 4, 5]
print(frequency_distribution(data))
print(frequency_distribution(data, bins=3))

Output:

{1: 1, 2: 2, 3: 3, 4: 2, 5: 1}
{1.0: 3, 2.333333333333333: 3, 3.6666666666666665: 3}

3. Correlation Matrix

This function calculates the correlation matrix for a given dataset, showing the relationships between variables. It leans on the Pandas corr() functionality.

import pandas as pd

def correlation_matrix(data):
    """
    Calculate the correlation matrix for a given dataset.
	
    :param data: dict of datasets, of the form dataset name (key), dataset list (value)
    :return: DataFrame containing correlation matrix
    """
    if not isinstance(data, pd.DataFrame):
        data = pd.DataFrame(data)
    return data.corr()

Let’s test out the new function.

data = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [2, 4, 5, 4, 5],
    'C': [1, 2, 3, 4, 5]
})
print(correlation_matrix(data))

Output:

          A         B         C
A  1.000000  0.774597  1.000000
B  0.774597  1.000000  0.774597
C  1.000000  0.774597  1.000000

4. Outlier Detection

This function detects outliers in a dataset. It uses either the IQR (Interquartile Range) or Z-score method, with IQR being the default.

import numpy as np

def detect_outliers(data, method='iqr', threshold=1.5):
    """
    Detect outliers in a dataset using IQR or Z-score method.

    :param data: list of integers, non-zero length
    :param method: string representing method to use (either 'iqr' or 'zscore')
    :param threshold: list of integers, non-zero length
    :return: DataFrame containing correlation matrix
    """
    if method == 'iqr':
        Q1 = np.percentile(data, 25)
        Q3 = np.percentile(data, 75)
        IQR = Q3 - Q1
        lower_bound = Q1 - (threshold * IQR)
        upper_bound = Q3 + (threshold * IQR)
        outliers = [x for x in data if x < lower_bound or x > upper_bound]
    elif method == 'zscore':
        z_scores = (data - np.mean(data)) / np.std(data)
        outliers = [data[i] for i, z in enumerate(z_scores) if abs(z) > threshold]
    else:
        raise ValueError("Method must be either 'iqr' or 'zscore'")
    
    return outliers

For the IQR method, the threshold multiplied by the IQR would be used in the calculations for lower and upper bounds. The threshold represents how many standard deviations away from the mean a data point needs to be to be considered an outlier when using the Z-score method.

Let’s see how the function works.

data = [1, 2, 3, 4, 5, 100]
print("IQR method:", detect_outliers(data, method='iqr'))
print("Z-score method:", detect_outliers(data, method='zscore', threshold=2))

Output:

IQR method: [100]
Z-score method: [100]

5. Descriptive Statistics by Group

Our final DIY function calculates descriptive statistics for a value column, grouped by another column. This can help provide insights into how statistics vary across different categories.

import pandas as pd

def group_statistics(data, group_column, value_column):
    """
    Calculate descriptive statistics for a value column, grouped by another column.
	
    :param data: DataFrame dataset
    :param group_column: DataFrame column to be used for grouping
    :param value_column: DataFrame column for descriptive statistics
    """
    if not isinstance(data, pd.DataFrame):
        data = pd.DataFrame(data)
    
    grouped = data.groupby(group_column)[value_column].agg([
        'count', 'mean', 'median', 'std', 'min', 'max'
    ])
    grouped['range'] = grouped['max'] - grouped['min']
    return grouped

Let’s give our function a test drive.

data = pd.DataFrame({
    'Group': ['A', 'A', 'B', 'B', 'B', 'C'],
    'Value': [1, 2, 3, 4, 5, 6]
})
print(group_statistics(data, 'Group', 'Value'))

Output:

       count  mean  median       std  min  max  range
Group                                                
A          2   1.5     1.5  0.707107    1    2      1
B          3   4.0     4.0  1.000000    3    5      2
C          1   6.0     6.0       NaN    6    6      0

Final Thoughts

You will undoubtedly enhance your descriptive statistics analysis abilities while armed with these DIY Python functions. While some edits may be needed for your specific situation, we have implemented functionality to provide a deeper understanding of your data’s characteristics, relationships, and distributions. Remember that creating and reusing functions such as these can help streamline your workflow, so make sure you tuck these away for later use before you forget.