Discovering hidden, multi-dimensional patterns in large datasets doesn’t necessarily demand costly enterprise analytics software if you get acquainted enough with the right open-source tools.

This article presents three Pandas-based methods to take EDA reports to the next level of depth and insight:

• pd.crosstab()
• pd.pivot_table()
• ProfileReport

1. Instant Mapping of Categorical Overlaps: Crosstab

A crosstab (short for “cross tabulation”) is a contingency table among two or more categorical variables in a dataset. It maps frequencies across such variables, namely, as probabilities of categories across the second variable, given a specific category for the first variable.

Does it sound like conditional probabilities? Indeed, crosstabs are a powerful tool to instantly reveal conditional probabilities involving categorical features.

For instance, if we take the California Housing Dataset — containing socio-demographic and housing attributes for districts across California — and discretize the house price variable into a 3-tier categorical variable, we can build a crosstab between the dataset’s signature categorical variables, ocean proximity, and the newly created “price tier” feature, as follows:

import pandas as pd

# Loading the open California Housing dataset
df_housing = pd.read_csv("https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv")

# Creating a categorical 'value tier' from the continuous house values
df_housing['value_tier'] = pd.qcut(df_housing['median_house_value'], q=3, labels=['Low', 'Medium', 'High'])

# Cross-tabulate location against value tier, mapping to percentages (conditional probabilities)
crosstab_pct = pd.crosstab(df_housing['ocean_proximity'], df_housing['value_tier'], normalize='index') * 100

For even more insight, let’s display the resulting crosstab as a heatmap:

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10, 6))
sns.heatmap(crosstab_pct, annot=True, fmt=".2f", cmap="YlGnBu")
plt.title('Ocean Proximity vs. House Value Tier (Percentage)')
plt.xlabel('Value Tier')
plt.ylabel('Ocean Proximity')
plt.show()

Thus, roughly speaking, while cheap houses dominate inland districts, expensive ones are the norm across the remaining geographical categories based on ocean proximity.

2. Multidimensional Aggregation: Pivot Table

Pivot tables are used for simultaneously aggregating numerical data across several categorical dimensions.

Consider it like a matrix containing descriptive statistics of your choice for every possible combination of categories, such as “old inland homes”, “new inland homes”, “old bay homes”, and so on. To make this example work, we quickly build a new categorical feature, age_tier, based on the original attribute describing the house age:

import pandas as pd

df_housing = pd.read_csv("https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv")

# Creating a second category by binning the house age
df_housing['age_tier'] = pd.qcut(df_housing['housing_median_age'], q=2, labels=['Newer Homes', 'Older Homes'])

# Creating a multi-dimensional pivot table
pivot = pd.pivot_table(
    df_housing, 
    values='median_house_value', 
    index='ocean_proximity',   # Dimension 1: rows
    columns='age_tier',        # Dimension 2: columns
    aggfunc='median', 
    margins=True               
)

print(pivot.round())

Output:

age_tier         Newer Homes  Older Homes       All
ocean_proximity                                    
<1H OCEAN           229350.0     197000.0  214850.0
INLAND              116000.0      92600.0  108500.0
ISLAND              368750.0     414700.0  414700.0
NEAR BAY            225000.0     239050.0  233800.0
NEAR OCEAN          219400.0     239300.0  229450.0
All                 175000.0     184200.0  179700.0

Notice that we chose the median as the aggregation to perform, and the house price (value) as the numerical variable to aggregate. You can display more than one aggregation at once — for example, by using aggfunc=['mean','median'] — but the results will be displayed in two separate matrices: one per aggregation function.

3. Comprehensive Auditing One-Liner: Profile Report

Lastly, a solution relying on an extra library that extends Pandas: ydata-profiling. This library delivers automated, enterprise-grade analytics for free, ingesting a DataFrame and building an interactive document that features missing values, correlation heatmaps, and histograms showing per-column distributions. It is particularly handy for quickly surfacing “alerts” and flagging issues like strong multicollinearity, skewness, and so on.

import pandas as pd
from ydata_profiling import ProfileReport

df_housing = pd.read_csv("https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv")

# Generating an exhaustive EDA report in one line
# Using 'minimal=True' speeds up computation on massive datasets
profile = ProfileReport(df_housing, title="Housing Data Automated Profiling", minimal=True)

# Exporting the interactive report into HTML file
profile.to_file("housing_eda_report.html")

While the report is exported into an HTML file, you can see an excerpt of it here: