Understanding Bayes' Theorem

Bayes’ Theorem updates beliefs in light of new evidence. It combines three key elements: what we believed before seeing the evidence (prior), how strongly the evidence supports the hypothesis (likelihood), and how common the evidence is overall (marginal probability).

I distinguish clearly between prior and likelihood. Confusing $P(A|B)$ with $P(B|A)$ is the most common mistake in Bayes-related problems.

In the previous article, we explored applications of the Central Limit Theorem and learned about the behavior of sums of many independent and identically distributed random variables. Now, we turn to a foundational concept in probability theory—Bayes’ Theorem. Bayes’ Theorem is a crucial tool for understanding probability and reasoning under uncertainty, and it finds widespread application across artificial intelligence and machine learning.

Core Concepts of Bayes’ Theorem

Bayes’ Theorem expresses the relationship among conditional probabilities. First, let’s define several essential terms:

When applying Bayes’ Theorem, begin by writing down the original (prior) probability, the probability of the evidence, and the relevant conditional probabilities—then examine how the evidence shifts the credibility of the event.

• Prior Probability $P(A)$: Our estimated probability that event $A$ occurs before observing any evidence.
• Likelihood $P(B|A)$: The probability that evidence $B$ occurs given that event $A$ has occurred.
• Marginal Probability of Evidence $P(B)$: The total probability that evidence $B$ occurs, computed using the Law of Total Probability: $$P(B) = P(B|A)P(A) + P(B|A')P(A'),$$ where $A'$ denotes the complement of $A$ (i.e., $A$ does not occur).
• Posterior Probability $P(A|B)$: The updated probability that event $A$ occurs after observing evidence $B$.

The mathematical statement of Bayes’ Theorem is:

$$P(A|B) = \frac{P(B|A)\,P(A)}{P(B)}$$

Understanding Bayes’ Theorem

To build intuition, let’s walk through a simple illustrative example.

After reading Understanding Bayes’ Theorem, take one minute to reflect: Are the core concepts clearly distinguished? Can you reproduce the solution steps? Can you restate the conclusions in your own words?

Example: Medical Testing

Suppose a rare disease affects 1% of the population (i.e., $P(\text{Disease}) = 0.01$). A medical test exists to detect it. If a person truly has the disease, the test returns positive with 90% sensitivity:
$P(\text{Positive} \mid \text{Disease}) = 0.9$.
However, if a person does not have the disease, the test still yields a false positive 10% of the time:
$P(\text{Positive} \mid \text{No Disease}) = 0.1$.

We want to compute the probability that a person actually has the disease given a positive test result—that is, $P(\text{Disease} \mid \text{Positive})$.

Applying Bayes’ Theorem:

1. Prior Probabilities:

   • $P(\text{Disease}) = 0.01$
   • $P(\text{No Disease}) = 1 - P(\text{Disease}) = 0.99$

2. Likelihoods:

   • $P(\text{Positive} \mid \text{Disease}) = 0.9$
   • $P(\text{Positive} \mid \text{No Disease}) = 0.1$

3. Marginal Probability of Evidence — compute $P(\text{Positive})$:

   $$P(\text{Positive}) = P(\text{Positive} \mid \text{Disease})\,P(\text{Disease}) + P(\text{Positive} \mid \text{No Disease})\,P(\text{No Disease})$$ $$= 0.9 \times 0.01 + 0.1 \times 0.99 = 0.009 + 0.099 = 0.108$$

4. Posterior Probability:

   $$P(\text{Disease} \mid \text{Positive}) = \frac{P(\text{Positive} \mid \text{Disease})\,P(\text{Disease})}{P(\text{Positive})}$$ $$= \frac{0.9 \times 0.01}{0.108} \approx 0.0833$$

Thus, even with a positive test result, the probability that the person actually has the disease is only about 8.33%. This illustrates how—even with relatively high test sensitivity—the rarity of the disease significantly reduces the posterior probability.

At this point, summarize Understanding Bayes’ Theorem into a concise review table: first articulate the main conceptual thread, then verify it using a small concrete task.

After finishing Understanding Bayes’ Theorem, try working through a small example end-to-end. Then assess which steps you can now perform independently.

Summary

Through this example, we see how Bayes’ Theorem enables us to rationally update our beliefs when new evidence arrives. Rather than discarding prior knowledge, we integrate it with observed data to refine our understanding of reality. This mode of probabilistic reasoning plays a central role in machine learning and data science—especially in model selection, hyperparameter tuning, and uncertainty quantification.

In the next article, we will delve deeper into Bayesian updating, exploring the roles of priors and posteriors—and how to flexibly revise our knowledge as new data streams in over time.