6. Bayesian Theorem Fundamentals: Updating Rules and Examples

The core of Bayesian learning lies in integrating prior beliefs with new evidence, while explicitly representing uncertainty. As you read, structure your understanding as follows: “Bayesian Theorem Recap → Update Rule → Process of Updating Probabilities → Assigning Prior Probabilities”, then return to the code, examples, or metrics in the main text to verify your comprehension.

After reading, perform a quick reality check using a small, concrete task: Identify what the inputs are, where the processing steps occur, and whether the outputs are verifiable. If something goes wrong, first revisit the “Bayesian Theorem Recap”, then check the “Update Rule”.

In the previous article, we introduced the foundational concepts of Bayes’ theorem—including prior and posterior distributions. Now, we will delve deeper into the update rule embedded in Bayes’ theorem: how to revise our beliefs (or model parameters) in light of observed data.

Bayesian Theorem Recap

First, let’s briefly recall the formal expression of Bayes’ theorem:

When learning the update rule, begin by writing down your initial belief, then examine how that belief changes upon observing new evidence. The more concrete the example, the clearer your Bayesian intuition becomes.

$$P(H|D) = \frac{P(D|H) \cdot P(H)}{P(D)}$$

where

• $P(H|D)$ is the posterior probability: our updated belief about hypothesis $H$ after observing data $D$;
• $P(D|H)$ is the likelihood: the probability of observing data $D$ assuming hypothesis $H$ is true;
• $P(H)$ is the prior probability: our belief about $H$ before seeing any data;
• $P(D)$ is the marginal likelihood (or evidence): a normalizing constant ensuring the total probability across all possible hypotheses sums to 1.

The Update Rule

From the formula above, we see precisely how the posterior probability depends on both the prior and the observed data. Unlike classical statistical methods, Bayesian learning explicitly incorporates prior knowledge. Once new data arrives, we use Bayes’ rule to systematically update our degree of belief in a given hypothesis.

Before reading “Bayesian Theorem Fundamentals — Update Rules and Examples”, preview the diagram showing the path from problem to result. After reading, cross-check against the main text to confirm whether you can reproduce the reasoning step-by-step.

Process of Updating Probabilities

In practice, suppose we conduct an experiment to assess whether a coin is fair. Our hypothesis space could be:

• $H_1$: The coin is fair.
• $H_2$: The coin is biased.

Assigning Prior Probabilities

Before collecting any data, we might assign equal prior probabilities to the two hypotheses:

• $P(H_1) = 0.5$
• $P(H_2) = 0.5$

Collecting Data

Suppose we flip the coin 10 times and observe 7 heads and 3 tails. We now want to compute the updated probabilities of $H_1$ and $H_2$ given this outcome.

Computing Likelihoods

Next, we compute the likelihood of the observed data under each hypothesis:

• Under $H_1$ (fair coin, $p = 0.5$), the likelihood of 7 heads and 3 tails is:

$$P(D|H_1) = \binom{10}{7} \cdot (0.5)^7 \cdot (0.5)^3 = \frac{10!}{7!\,3!} \cdot (0.5)^{10}$$

Result: $P(D|H_1) = 0.1172$.

• Under $H_2$ (biased coin, assume $p = 0.8$ for heads), the likelihood is:

$$P(D|H_2) = \binom{10}{7} \cdot (0.8)^7 \cdot (0.2)^3 = \frac{10!}{7!\,3!} \cdot (0.8)^7 \cdot (0.2)^3$$

Result: $P(D|H_2) = 0.2013$.

Updating Posterior Probabilities

Now apply Bayes’ theorem to compute the posterior probabilities:

1. First compute the marginal likelihood $P(D)$:

$$P(D) = P(D|H_1) \cdot P(H_1) + P(D|H_2) \cdot P(H_2) = 0.1172 \cdot 0.5 + 0.2013 \cdot 0.5 = 0.15825$$

2. Then compute the posteriors:

• For $H_1$:

$$P(H_1|D) = \frac{P(D|H_1) \cdot P(H_1)}{P(D)} = \frac{0.1172 \cdot 0.5}{0.15825} \approx 0.3704$$

• For $H_2$:

$$P(H_2|D) = \frac{P(D|H_2) \cdot P(H_2)}{P(D)} = \frac{0.2013 \cdot 0.5}{0.15825} \approx 0.6296$$

Final results:

• $P(H_1|D) \approx 0.3704$
• $P(H_2|D) \approx 0.6296$

These calculations show that, after observing the data, our belief in the biased-coin hypothesis has increased.

After finishing “Bayesian Theorem Fundamentals — Update Rules and Examples”, try applying it to a scenario of your own—pay close attention to whether the inputs, processing steps, and outputs align coherently.

To adapt “Bayesian Theorem Fundamentals — Update Rules and Examples” to your own task, start small: isolate and validate just one critical decision point.

Conclusion

Through the above example, we demonstrated how Bayes’ theorem enables probabilistic updating—integrating new data to dynamically refine our beliefs about hypotheses. In practice, the power of Bayesian learning lies in its ability to formally incorporate prior knowledge and support self-correction as new evidence accumulates in changing environments.

In the next article, we will explore Maximum A Posteriori (MAP) Estimation, advancing further into the world of Bayesian statistical inference—and introducing a practical method for parameter estimation. Stay tuned!