Image by Author 

When analyzing multiple variables, you need to understand how their combined uncertainty compares to their individual uncertainties. Joint entropy and marginal entropy give you this comparison, showing whether variables are independent or contain overlapping information.

These concepts guide feature selection in machine learning, help identify redundant measurements in experiments, and inform data collection strategies. Once you understand the distinction between joint and marginal entropy, you’ll think differently about relationships in your datasets.

What Marginal Entropy Measures

Marginal entropy quantifies uncertainty in a single variable, ignoring all other variables in your dataset. The entropy formula treats each variable independently:

 

This measures how unpredictable outcomes of X are on their own. A fair coin has marginal entropy of 1 bit. A biased coin landing heads 90% of the time has lower marginal entropy of around 0.47 bits.

Here’s an example with weather conditions in two cities. City A’s temperature (hot/cold) might have entropy of 0.92 bits. City B’s temperature has its own marginal entropy, calculated separately. These individual measurements tell you nothing about how the temperatures relate to each other.

Understanding Joint Entropy

Joint entropy measures uncertainty in multiple variables at the same time. It accounts for their combined probability distribution:

 

You sum over all possible combinations of X and Y outcomes, weighting each by its joint probability. This captures the total uncertainty when considering both variables together.

The Independence Test

When variables are completely independent, their joint entropy equals the sum of their marginal entropies:

 

This makes sense. If knowing X tells you nothing about Y, the combined uncertainty equals the individual uncertainties added together.

When variables are dependent, joint entropy becomes less than the sum of marginals:

 

The difference shows how much information the variables share. In fact, this gap is the mutual information: I(X;Y) = H(X) + H(Y) − H(X,Y). Strongly dependent variables have joint entropy much lower than the sum of their marginals.

Worked Example: Weather Patterns

Let’s look at temperature (hot/cold) and rain (yes/no) over 100 days. Here’s the joint probability table:

	Hot	Cold
Rain	0.10	0.40
No Rain	0.40	0.10

First, we’ll calculate marginal probabilities by summing across rows and columns.

Step 1: Find marginal probability of rain

Add the probabilities across the Rain row:

• P(Rain) = 0.10 + 0.40 = 0.50

Add the probabilities across the No Rain row:

• P(No Rain) = 0.40 + 0.10 = 0.50

Step 2: Calculate marginal entropy of rain

Apply the entropy formula to these probabilities:

• H(Rain) = -(0.50 × log₂(0.50) + 0.50 × log₂(0.50))
• H(Rain) = -(0.50 × -1.000 + 0.50 × -1.000)
• H(Rain) = -(-0.500 + -0.500)
• H(Rain) = 1.0 bits

This equals 1.0 bits, which represents maximum uncertainty for a binary variable.

Step 3: Find marginal probability of temperature

Add the probabilities down the Hot column:

• P(Hot) = 0.10 + 0.40 = 0.50

Add the probabilities down the Cold column:

• P(Cold) = 0.40 + 0.10 = 0.50

Step 4: Calculate marginal entropy of temperature

Apply the entropy formula:

• H(Temperature) = -(0.50 × log₂(0.50) + 0.50 × log₂(0.50))
• H(Temperature) = -(0.50 × -1.000 + 0.50 × -1.000)
• H(Temperature) = 1.0 bits

Temperature also has 1.0 bits of marginal entropy.

Step 5: Calculate joint entropy

Now use all four joint probabilities from the table:

• H(Rain, Temperature) = -(0.10 × log₂(0.10) + 0.40 × log₂(0.40) + 0.40 × log₂(0.40) + 0.10 × log₂(0.10))
• H(Rain, Temperature) = -(0.10 × -3.322 + 0.40 × -1.322 + 0.40 × -1.322 + 0.10 × -3.322)
• H(Rain, Temperature) = -(-0.332 + -0.529 + -0.529 + -0.332)
• H(Rain, Temperature) = -(-1.722)
• H(Rain, Temperature) = 1.722 bits

Joint entropy equals 1.722 bits, which is less than 2.0 bits (the sum of marginals). This difference of 0.278 bits is the mutual information, showing that temperature and rain patterns aren’t independent. Hot days tend to be dry, while cold days tend to be rainy.

Detecting Relationships Through Entropy

The gap between joint entropy and the sum of marginals quantifies how strongly variables depend on each other. Calculating this difference helps you:

Identify Redundant Features: When H(X,Y) ≈ max(H(X), H(Y)), one variable provides little additional information beyond the other, indicating redundancy. In such cases, you can often remove the less informative variable without losing predictive power.

Discover Hidden Dependencies: Variables that appear independent through correlation analysis might still show entropy-based dependence, especially for nonlinear relationships.

Optimize Data Collection: When planning experiments, prioritize measuring variables whose joint entropy suggests they capture distinct information.

Practical Applications

Database designers use these concepts to normalize schemas, separating independent attributes from dependent ones. Feature selection algorithms compare joint entropy of feature pairs to identify which combinations provide the most information.

In time series analysis, comparing joint entropy across different time lags reveals temporal dependencies. Low joint entropy at specific lags indicates predictable patterns you can exploit for forecasting.

Key Differences Summary

Marginal entropy measures one variable at a time, treating others as if they don’t exist. Joint entropy considers variables together, revealing their relationships through probability patterns.

Independent variables have joint entropy equal to summed marginals. Dependent variables have lower joint entropy, with the size of the gap indicating how strongly they depend on each other. This mathematical relationship gives you an objective measure of how variables interact in your data.

Understanding these concepts builds directly on the foundations of information theory and extends your ability to quantify information content beyond single variables.