Introduction

Statistics often revolves around quantifying uncertainty. Two main frameworks — Frequentist and Bayesian — approach this task differently. Frequentist methods typically focus on the long-run frequency of events. Bayesian methods, however, incorporate prior beliefs and update them with observed data to form a new, “posterior” understanding of uncertainty.

This article walks through the basics of Bayesian thinking, showing how it differs from frequentist reasoning, and why the concept of updating your beliefs is at the heart of Bayesian statistics.

We being with a coin flip example, as it is an excellent way to see this difference in action.

A Simple Coin Flip

Frequentist Setup

Imagine flipping a coin 10 times and counting 7 heads. A frequentist approach might compute a point estimate for the probability of heads (p). To capture uncertainty, you could form a confidence interval based on this estimate — like a 95% confidence interval using a standard formula.

Bayesian View

Bayesian thinking adds the notion of a prior distribution. Suppose you start with a “flat” prior belief that probability could be anything between 0 and 1 (i.e., a Beta(1,1) prior). Once you observe your 7 heads and 3 tails, you update your distribution to Beta((1 + 7), (1 + 3)) = Beta(8,4). This new distribution, called the posterior, represents your updated belief about (p) after seeing the 10 flips.

The posterior distribution tells you not just a point estimate but how likely (p) is to be near any value from 0 to 1. The usual Bayesian point estimate (the mean of Beta(8,4)) would be ( 8 / (8 + 4) = 0.67 ).

Below is a short Python code snippet showing how you might simulate and visualize this process:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Observations: 7 heads, 3 tails
alpha_prior, beta_prior = 1, 1
alpha_post = alpha_prior + 7
beta_post = beta_prior + 3

# Define a grid to evaluate the Beta pdf
x = np.linspace(0, 1, 200)
prior_pdf = beta.pdf(x, alpha_prior, beta_prior)
posterior_pdf = beta.pdf(x, alpha_post, beta_post)

# Plot the prior and posterior
plt.plot(x, prior_pdf, label='Prior: Beta(1,1)', color='blue')
plt.plot(x, posterior_pdf, label=f'Posterior: Beta({alpha_post},{beta_post})', color='red')
plt.title("Bayesian Updating with a Coin Flip")
plt.xlabel("Probability of Heads (p)")
plt.ylabel("Density")
plt.legend()
plt.show()

This simple setup highlights the essence of Bayesian updating: begin with a prior, see the data, then obtain a posterior.

Prior vs. Posterior & Credible vs. Confidence Intervals

The shift from prior to posterior visually captures how the data modifies your beliefs. When a new observation is made, you simply recalculate and shift your distribution accordingly.

In practice, these two intervals can look numerically similar. Conceptually, however, they answer different questions. A 95% Bayesian credible interval means we are 95% confident the parameter is in that range—reflecting our updated uncertainty.

Live Updating of Beliefs with Data

One hallmark of Bayesian reasoning is the ability to continually update beliefs as new data arrives. Suppose you flip the coin 10 more times and see 5 heads, 5 tails. With a frequentist method, you might combine the total heads and tails and form a new estimate or confidence interval. In Bayesian terms, you already have a posterior from the first 10 flips (Beta(8,4)). You treat that posterior as your new “prior” and combine it with these 10 additional flips (5 heads, 5 tails).

The new posterior after 20 total flips would be Beta((8 + 5), (4 + 5)) = Beta(13,9). Notice how straightforward the updating formula is.

This live updating process is particularly powerful in real-world scenarios, such as online recommendation systems. As more user interactions occur, Bayesian models seamlessly update to reflect the most recent data.

Conclusion

Shifting from frequentist to Bayesian thinking means adopting a mindset of belief updating. You begin with a prior assumption, observe data, and incorporate that evidence into a posterior distribution. This process is repeated as new data arrives, giving a dynamic and intuitive way to handle uncertainty. Concepts like credible intervals are more direct in interpretation, while continuous updating fits real-life scenarios requiring frequent refinements of knowledge.

If you’re seeking more flexible and interpretable models, Bayesian methods offer a powerful alternative. By learning to code these approaches in Python and visualize how prior beliefs evolve into posterior beliefs, you can gain deeper insight into the parameters that govern your data—and confidently adapt your analyses to new evidence as it arises.

In the coming weeks, we will be embarking on a new journey of Bayesian discovery, leveraging Python libraries PyMC and ArviZ in our pursuit of understanding, as we progress from fundamentals to advanced concepts. We hope you will join us.