Define objective function (negative because we minimize)

Bayesian learning centers on integrating prior beliefs with new evidence while explicitly quantifying uncertainty. As you read, structure your understanding as follows:
Bayes’ Theorem and the Posterior Distribution → Definition of Maximum A Posteriori Estimation (MAP) → MAP Application Example: Coin Tossing → Choosing a Prior Distribution,
then return to the code, examples, or metrics in the main text for verification.

After reading, test your understanding using a small real-world task:

• What are the inputs?
• Where does processing occur?
• Is the output verifiable and acceptable?
  If the task fails, first revisit “Bayes’ Theorem and the Posterior Distribution”, then check “Definition of Maximum A Posteriori Estimation (MAP)”.

In this tutorial, we delve into Maximum A Posteriori Estimation (MAP). In the previous part, we covered the fundamentals of Bayes’ theorem and its updating rule. Now, we apply Bayes’ theorem to parameter estimation—specifically, using MAP to infer unknown parameters.

Bayes’ Theorem and the Posterior Distribution

The core idea behind Bayes’ theorem is to update our belief about a parameter based on observed data. The posterior distribution represents our updated belief about the parameter after observing the data, expressed mathematically as:

When learning MAP, first compare what information the likelihood and the prior each contribute. When sample size is small or noise is high, the prior strongly influences the final estimate.

$$p(\theta \mid D) = \frac{p(D \mid \theta)\, p(\theta)}{p(D)}$$

where:

• $p(\theta \mid D)$ is the posterior distribution of parameter $\theta$ given data $D$;
• $p(D \mid \theta)$ is the likelihood function—the probability of observing data $D$ under parameter $\theta$;
• $p(\theta)$ is the prior distribution—our belief about $\theta$ before seeing the data;
• $p(D)$ is the marginal likelihood (or evidence), which is constant w.r.t. $\theta$ and thus plays no role in optimization.

Definition of Maximum A Posteriori Estimation (MAP)

Maximum A Posteriori Estimation (MAP) estimates the parameter value by maximizing the posterior distribution. Specifically, we seek:

You don’t need to absorb all details of “Maximum A Posteriori Estimation (MAP)” at once. Start with a small, hands-on problem you can verify yourself, then use the diagrams and main text to fill in conceptual gaps.

$$\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \, p(\theta \mid D)$$

Using Bayes’ theorem, this is equivalent to:

$$\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \, p(D \mid \theta)\, p(\theta)$$

since $p(D)$ is independent of $\theta$ and therefore omitted during maximization.

MAP Application Example: Coin Tossing

Suppose we have a biased coin and wish to estimate the probability $\theta$ that it lands heads up. We toss it 10 times and observe 7 heads (H) and 3 tails (T). We’ll use MAP to estimate $\theta$.

1. Choosing a Prior Distribution

We adopt a Beta distribution as the prior:

$$p(\theta) = \text{Beta}(\alpha, \beta) = \frac{\theta^{\alpha - 1} (1 - \theta)^{\beta - 1}}{B(\alpha, \beta)}$$

We select $\alpha = 2$ and $\beta = 2$, reflecting a prior belief that heads and tails are equally likely before any tosses.

2. Likelihood Function

Given 7 heads and 3 tails in 10 tosses, the likelihood is:

$$p(D \mid \theta) = \theta^7 (1 - \theta)^3$$

3. Computing the Posterior Distribution

Maximizing the posterior is equivalent to maximizing the unnormalized posterior:

$$p(\theta \mid D) \propto p(D \mid \theta)\, p(\theta) \propto \theta^7 (1 - \theta)^3 \cdot \theta^{1} (1 - \theta)^{1} = \theta^{8} (1 - \theta)^{4}$$

4. Solving for the MAP Estimate

To find the value of $\theta$ that maximizes $\theta^{8}(1 - \theta)^{4}$, we differentiate and solve for critical points:

$$\frac{d}{d\theta} \left( \theta^{8} (1 - \theta)^{4} \right) = 0$$

Alternatively, we can use numerical optimization:

import numpy as np
from scipy.optimize import minimize_scalar

# Define objective function (negative because we minimize)
def objective(theta):
    return - (theta**8 * (1 - theta)**4)

result = minimize_scalar(objective, bounds=(0, 1), method='bounded')
theta_map = result.x
print("MAP estimate:", theta_map)

This code outputs our estimated probability $\theta$ of heads.

If “Maximum A Posteriori Estimation (MAP)” hasn’t fully clicked yet, walk through these four actions again using this card.

When reviewing “Maximum A Posteriori Estimation (MAP)”, avoid jumping straight into large projects. Instead, first validate the core logic using a simple, concrete example.

Summary

This tutorial thoroughly introduced the concept and application of Maximum A Posteriori Estimation (MAP). Using the coin-tossing example, we demonstrated how to estimate parameters by maximizing the posterior distribution. In the next tutorial, we will compare Bayesian estimation with frequentist estimation to deepen our understanding of statistical inference.

Mastering MAP lays a solid foundation for further study in Bayesian learning. If you have questions or would like deeper discussion, feel free to ask!