Simulate 10 coin flips

The core of Bayesian learning lies in synthesizing prior beliefs with new evidence while explicitly representing uncertainty. As you read, structure your understanding around the sequence: Theoretical Foundations → Bayesian Estimation → Frequentist Estimation → Comparative Case Study, then return to the code, examples, or metrics in the main text for verification.

After reading, validate your understanding using a small real-world task: identify what the inputs are, where the processing steps occur, and whether the outputs are verifiable and acceptable. If something fails, first revisit the Theoretical Foundations, then check the Bayesian Estimation section.

In previous discussions, we introduced Maximum A Posteriori (MAP) estimation—an important method in parameter estimation. Today, we delve deeper into comparing Bayesian and frequentist estimation, highlighting their fundamental differences in parameter estimation as well as their respective strengths and limitations.

Theoretical Foundations

Bayesian Estimation

When comparing Bayesian and frequentist estimation, consider four key dimensions:

• How each treats parameters,
• How each uses sample data,
• Whether and how each incorporates prior information, and
• How each interprets results.

Bayesian estimation is a parameter estimation framework grounded in Bayes’ theorem. By integrating prior knowledge with observed data, it yields a posterior distribution over the parameter. Given observed data $x$, the posterior distribution of parameter $\theta$ is:

$$p(\theta \mid x) = \frac{p(x \mid \theta)\, p(\theta)}{p(x)}$$

where $p(x \mid \theta)$ is the likelihood function, $p(\theta)$ is the prior distribution, and $p(x)$ is the marginal likelihood (or evidence). A common point estimate under Bayesian estimation is the posterior mean:

$$\hat{\theta}_{\text{Bayes}} = \mathbb{E}[\theta \mid x] = \int \theta\, p(\theta \mid x)\, d\theta$$

Frequentist Estimation

Frequentist estimation relies solely on the observed data—no prior information is incorporated. Within frequentist statistics, the most widely used method is Maximum Likelihood Estimation (MLE), which selects the parameter value that maximizes the likelihood function:

$$\hat{\theta}_{\text{MLE}} = \arg\max_{\theta}\, p(x \mid \theta)$$

This approach depends exclusively on the data, thereby avoiding any subjective influence from prior assumptions.

Comparative Case Study

To better understand the distinction between Bayesian and frequentist estimation, consider a simple example: estimating the probability $\theta$ of heads for a biased coin.

Don’t stop at “I understood” after reading Comparing Bayesian and Frequentist Estimation. Go back, pick one step, and implement it yourself—then note where you get stuck. This practice makes subsequent learning more robust.

Data Generation

Suppose we flip the coin 10 times and observe 7 heads:

import numpy as np

# Simulate 10 coin flips
np.random.seed(42)
n_flips = 10
heads = 7  # number of observed heads

Bayesian Estimation

We choose a Beta distribution as our prior $p(\theta)$—e.g., $\text{Beta}(1, 1)$, representing a uniform (non-informative) belief that the coin is fair. We then update this prior using the observed data.

The resulting posterior distribution is:

$$p(\theta \mid x) \sim \text{Beta}(1 + \text{heads},\; 1 + \text{tails}) = \text{Beta}(8,\; 4)$$

We compute the posterior mean using Python:

from scipy.stats import beta

# Prior parameters
a_prior = 1
b_prior = 1

# Updated posterior parameters
a_post = a_prior + heads
b_post = b_prior + (n_flips - heads)

# Posterior mean
posterior_mean = beta.mean(a_post, b_post)
posterior_mean

Frequentist Estimation

For frequentist estimation, we apply Maximum Likelihood Estimation:

$$\hat{\theta}_{\text{MLE}} = \frac{\text{heads}}{n\_flips} = \frac{7}{10} = 0.7$$

This result follows directly from the observed data. Its Python implementation is straightforward:

# Maximum likelihood estimate
mle_estimate = heads / n_flips
mle_estimate

At this point, organize Comparing Bayesian and Frequentist Estimation into a retrospective table: first clarify the central narrative, then test it with a small concrete task.

After finishing Comparing Bayesian and Frequentist Estimation, try walking through a small example end-to-end, then assess which steps you can now execute independently.

Summary of Key Differences

• Sources of Information:

  • Bayesian estimation combines prior knowledge with data—especially valuable when data is scarce.
  • Frequentist estimation relies entirely on observed data—most effective when sample size is large.

• Nature of Output:

  • Bayesian estimation yields a full posterior distribution, enabling explicit quantification of uncertainty (e.g., credible intervals).
  • Frequentist estimation delivers a single point estimate (e.g., MLE), with uncertainty typically assessed separately (e.g., via standard errors or confidence intervals).

• Practical Applicability:

  • Bayesian methods flexibly incorporate domain expertise and are especially suited to high-uncertainty or low-data scenarios.
  • Frequentist methods tend to perform well with large samples and are often simpler to implement and interpret.

In the next article, we will explore parameter selection and evaluation, diving deeper into how to choose appropriate models and methods based on estimation results.