Compute variances

Covariance measures whether two variables change together; correlation removes the influence of scale. A high correlation does not imply causation.

I always start with a scatter plot. A single correlation coefficient cannot reveal outliers, nonlinear relationships, or subgroup structures.

In the previous article, we explored the properties of variance, learning how to quantify the dispersion of a single random variable. This article continues our discussion of key concepts in probability theory: covariance and correlation—essential tools for analyzing relationships between random variables, widely applied in machine learning and data analysis.

Definition of Covariance

Covariance is a measure describing the linear relationship between two random variables. Given random variables $X$ and $Y$, their covariance is defined as:

When learning covariance and correlation, first assess whether the two variables move in the same direction; then examine the sign and magnitude of the standardized correlation coefficient.

$$\text{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]$$

This formula helps us intuitively understand covariance: it quantifies how deviations of one variable from its mean relate to deviations of the other variable from its mean.

Properties of Covariance

1. Interpretation of Sign:

   • If $\text{Cov}(X, Y) > 0$, $X$ and $Y$ are positively associated: when one tends to increase, the other tends to increase as well.
   • If $\text{Cov}(X, Y) < 0$, $X$ and $Y$ are negatively associated.
   • If $\text{Cov}(X, Y) = 0$, there is no linear relationship between them.

2. Unit Sensitivity:

   • Covariance carries units equal to the product of the units of $X$ and $Y$, making interpretation less intuitive.

Example

Suppose $X$ and $Y$ represent a student’s study time (in hours) and exam score (in points), respectively. Observed data are shown below:

Study Time ($X$)	Exam Score ($Y$)
1	50
2	55
3	60
4	70
5	75

First, compute the expectations:

$$\mathbb{E}[X] = \frac{1 + 2 + 3 + 4 + 5}{5} = 3$$ $$\mathbb{E}[Y] = \frac{50 + 55 + 60 + 70 + 75}{5} = 62$$

Then apply the covariance formula:

$$\text{Cov}(X, Y) = \frac{1}{5} \sum_{i=1}^{5} (X_i - \mathbb{E}[X])(Y_i - \mathbb{E}[Y])$$

import numpy as np

X = np.array([1, 2, 3, 4, 5])
Y = np.array([50, 55, 60, 70, 75])

cov_xy = np.cov(X, Y)[0][1]  # Extract covariance between X and Y
cov_xy

The computed covariance $\text{Cov}(X, Y)$ is positive, indicating a positive linear association between study time and exam score.

Definition and Computation of Correlation

Correlation is the standardized version of covariance—designed specifically to eliminate unit dependence. It is typically quantified using the Pearson correlation coefficient, defined as:

When reading “Covariance and Correlation,” begin by reviewing the tasks, core concepts, exercises, and decision points illustrated in the accompanying figures—then return to the main text to fill in details. This approach helps you quickly identify which real-world scenarios this content applies to.

$$r_{XY} = \frac{\text{Cov}(X, Y)}{\sqrt{\text{Var}(X) \cdot \text{Var}(Y)}}$$

where $\text{Var}(X)$ and $\text{Var}(Y)$ denote the variances of $X$ and $Y$, respectively.

Properties of Correlation

1. Range:
   • The correlation coefficient $r_{XY}$ always lies in the interval $[-1, 1]$.
   • $r_{XY} = 1$ indicates perfect positive linear correlation; $r_{XY} = -1$, perfect negative linear correlation; $r_{XY} = 0$, no linear correlation.

Example

Continuing with the earlier example, we compute the Pearson correlation coefficient between study time and exam score.

# Compute variances
var_x = np.var(X)
var_y = np.var(Y)

# Compute correlation coefficient
correlation = cov_xy / (np.sqrt(var_x) * np.sqrt(var_y))
correlation

Running this code yields the correlation coefficient $r$. Suppose the result is $r = 0.95$: this indicates a strong positive linear relationship between study time and exam performance.

When reviewing “Covariance and Correlation,” place key concepts, procedural steps, and observable outcomes on the same page for efficient revision.

When practicing “Covariance and Correlation,” write input conditions, processing steps, and observable outcomes together—making future review straightforward.

Summary

In this article, we introduced covariance and correlation, fundamental tools for investigating relationships between two random variables. By computing covariance and correlation coefficients, we gain deeper insight into underlying data structure and dependencies—laying essential groundwork for the next topic: the Law of Large Numbers.

In the following article, we will delve into the Law of Large Numbers, exploring how sample means converge toward the population mean as sample size increases. We hope you’ll apply these concepts confidently to analyze real-world problems!