Generate synthetic data

Bayesian learning centers on synthesizing prior beliefs with new evidence while explicitly quantifying uncertainty. While reading, structure your understanding around the sequence: “Bayes Factor → Computing Bayes Factors → Example: Normal Distribution Models → Python Code Example”, then verify concepts, code, case studies, or metrics presented in the main text.

After reading, validate your comprehension using a small real-world task: identify what the inputs are, where the processing steps occur, and whether outputs are verifiable and acceptable. If something fails, first check “Bayes Factor”, then “Computing Bayes Factors”.

In the previous chapter, we explored an important aspect of model selection—model complexity. We examined how complexity affects model performance and discussed using information criteria to evaluate competing models. However, the real challenge lies in choosing among multiple candidate models—and Bayes factors provide a principled, effective tool for this purpose.

Bayes Factors

A Bayes factor is a quantitative measure used to compare two competing models. Suppose we have two models, $M_1$ and $M_2$. The Bayes factor $\text{BF}_{12}$ is defined as the ratio of their marginal likelihoods (i.e., the probability of the observed data under each model):

When learning about Bayes factors, first examine how each model explains the observed data individually; then compare the relative strength of evidence and explanatory power.

$$\text{BF}_{12} = \frac{P(\text{data} \mid M_1)}{P(\text{data} \mid M_2)}$$

Here, $P(\text{data} \mid M)$ denotes the marginal likelihood—the probability of observing the data under model $M$, obtained by integrating over all possible parameter values.

Interpretation: $\text{BF}_{12}$ quantifies how much the observed data support $M_1$ relative to $M_2$. If $\text{BF}_{12} > 1$, the data favor $M_1$; if $\text{BF}_{12} < 1$, they favor $M_2$.

Computing Bayes Factors

Although the definition of Bayes factors appears simple, computing them is often nontrivial. Evaluating $P(\text{data} \mid M)$ typically requires integrating over the entire parameter space—a computationally demanding operation. Analytic solutions exist only for simple models; for most realistic, complex models, numerical approximation methods are required.

Read “Bayes Factors and Model Comparison” through the lens of “Scenario → Concept → Action → Outcome.” First align these four elements; then revisit parameters, code, or workflows in the main text.

We illustrate Bayes factor computation via a concrete example.

Example: Normal Distribution Models

Suppose we observe a dataset drawn from a single normal distribution, and wish to compare two models:

• Model $M_1$: Mean $\mu$ is known (fixed at $\mu_0$); variance $\sigma^2$ is unknown.
• Model $M_2$: Both mean $\mu$ and variance $\sigma^2$ are unknown.

Under $M_1$, the marginal likelihood simplifies to:

$$P(\text{data} \mid M_1) \propto \sigma^{-n} \exp\left(-\frac{(x - \mu_0)^2}{2\sigma^2}\right)$$

(ignoring constants irrelevant to comparison).

Under $M_2$, where both $\mu$ and $\sigma$ are unknown and assigned priors, the marginal likelihood becomes:

$$P(\text{data} \mid M_2) = \int P(\text{data} \mid \mu, \sigma)\, P(\mu)\, P(\sigma)\, d\mu\, d\sigma$$

This integral generally has no closed-form solution and must be approximated—commonly using Markov Chain Monte Carlo (MCMC) methods.

Python Code Example

Below is a minimal Python example demonstrating Bayes factor computation using the PyMC3 library.

import numpy as np
import pymc3 as pm

# Generate synthetic data
data = np.random.normal(loc=5.0, scale=2.0, size=100)

# Model M1: relatively informative prior on mu
with pm.Model() as model1:
    mu = pm.Normal('mu', mu=5, sigma=1)
    sigma = pm.HalfNormal('sigma', sigma=1)
    likelihood = pm.Normal('likelihood', mu=mu, sigma=sigma, observed=data)
    trace1 = pm.sample(1000, return_inferencedata=False)

# Model M2: more diffuse priors
with pm.Model() as model2:
    mu = pm.Normal('mu', mu=0, sigma=10)
    sigma = pm.HalfNormal('sigma', sigma=10)
    likelihood = pm.Normal('likelihood', mu=mu, sigma=sigma, observed=data)
    trace2 = pm.sample(1000, return_inferencedata=False)

# Approximate marginal likelihoods (simplified illustration)
# Note: In practice, use dedicated methods like thermodynamic integration or bridge sampling.
marginal_likelihood1 = pm.sample_posterior_predictive(trace1)['likelihood'].mean()
marginal_likelihood2 = pm.sample_posterior_predictive(trace2)['likelihood'].mean()

# Compute Bayes factor BF₁₂
bayes_factor = marginal_likelihood1 / marginal_likelihood2
print(f"Bayes Factor BF_12: {bayes_factor}")

In this example, we generate normally distributed data and fit two distinct PyMC3 models. We then approximate their marginal likelihoods and compute the Bayes factor.

If “Bayes Factors and Model Comparison” hasn’t yet fully clicked, walk through the four actions on this card again.

When reviewing “Bayes Factors and Model Comparison,” avoid launching large-scale projects upfront. Instead, test your grasp using one simple, self-contained example to confirm whether the core logic is clear.

Conclusion

Bayes factors constitute a foundational tool for principled model selection. Compared to classical hypothesis testing, they offer a more intuitive, probabilistically coherent framework for comparing models. Though computation can be challenging, modern probabilistic programming tools make practical implementation feasible. A solid understanding of Bayes factors—both conceptually and computationally—lays essential groundwork for the next topic: overfitting and regularization. In the following chapter, we will explore how regularization techniques improve model generalization and effectively mitigate overfitting.