Generate synthetic data

Bayesian learning centers on integrating prior beliefs with new evidence while explicitly quantifying uncertainty. While reading, structure your understanding as follows: “Introduction to MCMC → Core Principles of MCMC → Case Study: Applying MCMC in Bayesian Regression → Implementing MCMC in Python”, then return to the code snippets, examples, or evaluation metrics in the main text for verification.

After reading, validate your understanding using a small real-world task: identify what the inputs are, where the processing steps occur, and whether the outputs are verifiable and acceptable. If something fails, first revisit the “Introduction to MCMC”, then consult the “Core Principles of MCMC”.

In the previous article, we discussed model evaluation and improvement in Bayesian classification—emphasizing how effective model selection and validation can enhance classifier performance. This time, we introduce the foundational concepts of Markov Chain Monte Carlo (MCMC), laying the groundwork for the next article on Gibbs sampling.

Introduction to MCMC

Markov Chain Monte Carlo (MCMC) is a statistical sampling technique that leverages Markov chains to draw samples from complex probability distributions—especially those from which direct sampling is intractable. In Bayesian statistics, MCMC is widely used to approximate posterior distributions.

When learning MCMC, begin by identifying: the target distribution, the sampling chain, the acceptance probability, burn-in period, convergence diagnostics, and sample autocorrelation.

Core Principles of MCMC

In MCMC, we construct a Markov chain whose stationary (equilibrium) distribution matches our desired target distribution. The chain performs a random walk across the state space via successive transitions. After sufficient iterations, the chain converges to the target distribution.

A widely used MCMC algorithm is the Metropolis–Hastings method, whose core steps are:

1. Initialization: Choose an initial state $x_0$.
2. Iteration:
   • Propose a candidate state $x'$ drawn from a proposal distribution $q(x' \mid x_t)$.
   • Compute the acceptance ratio: $\alpha = \min\left(1,\; \frac{\pi(x')\, q(x_t \mid x')}{\pi(x_t)\, q(x' \mid x_t)}\right)$
   • Accept $x'$ with probability $\alpha$; otherwise retain the current state $x_t$.

Repeat until the desired number of samples is obtained.

Case Study: Applying MCMC in Bayesian Regression

We illustrate MCMC through a simple example: Bayesian linear regression. Assume the model:

When reading “Bayesian Learning and Statistical Inference Tutorial: Foundations of Markov Chain Monte Carlo (MCMC)”, first clarify the problem context you aim to solve, then connect key concepts with hands-on practice steps. This approach helps prevent getting lost in isolated terminology when diving into technical details.

$y = \beta_0 + \beta_1 x + \epsilon$

where $\epsilon \sim \mathcal{N}(0, \sigma^2)$. Our goal is to draw samples from the posterior distribution:

$p(\beta \mid y, x) \propto p(y \mid x, \beta)\, p(\beta)$

We adopt a simple prior—for instance, $\beta \sim \mathcal{N}(0, 10^2)$.

Implementing MCMC in Python

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Generate synthetic data
np.random.seed(42)
x = np.random.rand(100)
true_beta = [2, 3]
y = true_beta[0] + true_beta[1] * x + np.random.normal(0, 0.5, x.shape)

# MCMC implementation
def mcmc_bayesian_linear_regression(x, y, iterations=10000):
    n = len(y)
    beta_samples = np.zeros((iterations, 2))
    beta = np.random.randn(2)  # Initial parameter values

    for i in range(iterations):
        # Propose new parameters
        beta_new = beta + np.random.normal(0, 0.5, 2)

        # Compute log-likelihoods (simplified for clarity)
        residuals_current = y - (beta[0] + beta[1] * x)
        residuals_new = y - (beta_new[0] + beta_new[1] * x)
        likelihood_current = np.sum(-0.5 * (residuals_current / 0.5)**2)
        likelihood_new = np.sum(-0.5 * (residuals_new / 0.5)**2)
        
        # Prior densities (Gaussian with variance 10²)
        prior_current = -0.5 * (beta[0]**2 / 10**2 + beta[1]**2 / 10**2)
        prior_new = -0.5 * (beta_new[0]**2 / 10**2 + beta_new[1]**2 / 10**2)
        
        # Acceptance ratio (in log-space for numerical stability)
        log_acceptance_ratio = (likelihood_new + prior_new) - (likelihood_current + prior_current)
        if np.log(np.random.rand()) < log_acceptance_ratio:
            beta = beta_new
        
        beta_samples[i] = beta
    
    return beta_samples

# Run MCMC
beta_samples = mcmc_bayesian_linear_regression(x, y)

# Plot trace plots
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.plot(beta_samples[:, 0], label=r'$\beta_0$ samples')
plt.title('Trace plot of $\beta_0$')
plt.subplot(1, 2, 2)
plt.plot(beta_samples[:, 1], label=r'$\beta_1$ samples', color='orange')
plt.title('Trace plot of $\beta_1$')
plt.legend()
plt.show()

# Visualize posterior distributions
sns.kdeplot(beta_samples[:, 0], label='Posterior of $\\beta_0$', fill=True)
sns.kdeplot(beta_samples[:, 1], label='Posterior of $\\beta_1$', fill=True, color='orange')
plt.title('Posterior distributions')
plt.legend()
plt.show()

In the code above, we implement a basic MCMC sampler for Bayesian linear regression. Through iterative sampling, we generate posterior draws for $\beta_0$ and $\beta_1$, then visualize their trace plots and posterior density estimates. This provides intuitive insight into how MCMC extracts representative samples from complex, analytically intractable posterior distributions.

If you haven’t yet fully internalized “Bayesian Learning and Statistical Inference Tutorial: Foundations of Markov Chain Monte Carlo (MCMC)”, use the four actions on this card to revisit and reinforce the material.

When reviewing “Bayesian Learning and Statistical Inference Tutorial: Foundations of Markov Chain Monte Carlo (MCMC)”, avoid attempting large-scale projects upfront. Instead, start with a single, simple example to confirm whether the core workflow is clear.

Conclusion

MCMC provides a powerful and flexible tool for parameter inference in complex Bayesian models. In the next tutorial, we will delve deeper into a specific MCMC variant—Gibbs sampling—a more efficient sampling strategy under certain conditional independence assumptions. A solid grasp of both the theoretical principles and practical implementation of MCMC is essential, forming a robust foundation for your journey through Bayesian statistics.