PGDDSA Study · Semester 1

Core Titles

Key headlines and terms for quick recall

Joint PMF/PDF $p(x, y), f(x, y)$
Marginal distributions — sum / integrate out
Conditional distributions $f(x | y) = f(x, y) / f_Y(y)$
Independence $f(x, y) = f_X(x) f_Y(y)$
Covariance, Correlation
Multivariate Normal $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$
Box–Muller / Cholesky for generation

Basic Idea

What it is, why it matters, how it works

Joint distributions

For two RVs $X, Y$ :

Discrete: joint PMF $p(x, y) = P(X = x, Y = y)$ , with $\sum_{x, y} p(x,y) = 1$ .
Continuous: joint PDF $f(x, y) \ge 0$ , with $\iint f(x, y) \, dx \, dy = 1$ .

Marginals

$f_X(x) = \int f(x, y) \, dy, \quad f_Y(y) = \int f(x, y) \, dx.$ Sum out the other variable.

Conditionals

$f_{X|Y}(x | y) = \frac{f(x, y)}{f_Y(y)} \quad (f_Y(y) > 0).$

Independence

$X$ and $Y$ are independent iff $f(x, y) = f_X(x) f_Y(y) \quad \text{for all } x, y.$ Equivalently $f_{X|Y}(x|y) = f_X(x)$ .

Covariance and correlation

$\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - E[X] E[Y].$ $\text{Corr}(X, Y) = \rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}, \quad -1 \le \rho \le 1.$

Independence ⇒ $\text{Cov} = 0$ . The converse is false in general (true for jointly normal).

Multivariate normal

A random vector $\mathbf{X} = (X_1, \dots, X_n)$ is multivariate normal with mean $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$ : $f(\mathbf{x}) = \frac{1}{(2\pi)^{n/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\!\left( -\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}) \right).$

Generating multivariate samples

Box–Muller: two independent uniforms → two independent standard normals.
Cholesky for multivariate normal: if $\boldsymbol{\Sigma} = L L^T$ and $Z \sim \mathcal{N}(0, I)$ , then $X = \boldsymbol{\mu} + L Z \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ .

Why this matters in Data Science

Most real datasets are multivariate. Covariance / correlation underlies PCA, linear regression, Gaussian mixture models, and many classifiers.

Mind Map

Visual structure of the concept

MULTIVARIATE RVs
├── Joint f(x, y) or p(x, y)
├── Marginal: integrate / sum out
├── Conditional: f(x|y) = f(x,y)/f_Y(y)
├── Independence: f = f_X · f_Y
├── Covariance
│   ├── Cov(X,Y) = E[XY] − E[X]E[Y]
│   └── Indep ⇒ Cov=0  (converse not always)
├── Correlation ρ ∈ [−1, 1]
├── Multivariate Normal 𝒩(μ, Σ)
└── Generation
    ├── Box-Muller → standard normals
    └── Cholesky Σ = LLᵀ ⇒ X = μ + LZ

Exam Q&A

Part A (2 marks) and Part B (20 marks) style questions

Part A (2 marks each)

Q1. Define joint PDF. A function $f(x, y) \ge 0$ with $\iint f(x, y) \, dx \, dy = 1$ , giving probabilities of regions via integration.

Q2. How is the marginal density $f_X$ obtained? By integrating out the other variable: $f_X(x) = \int f(x, y) \, dy$ .

Q3. State the independence condition for two continuous RVs. $f(x, y) = f_X(x) f_Y(y)$ for all $x, y$ .

Q4. Define covariance. $\text{Cov}(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y]$ .

Part B (20 marks)

Q. Define joint, marginal and conditional distributions for two continuous random variables. State and discuss independence and covariance. For the joint PDF $f(x, y) = 6xy$ on $0 \le x \le 1, 0 \le y \le 1, x + y \le 1$ , find the marginals and check whether $X, Y$ are independent.

Definitions.

Joint PDF: $f(x, y) \ge 0$ with $\iint f = 1$ . For a region $A$ : $P((X, Y) \in A) = \iint_A f(x, y) \, dx \, dy$ .

Marginals: $f_X(x) = \int f(x, y) dy$ , $f_Y(y) = \int f(x, y) dx$ .

Conditional: $f_{X|Y}(x|y) = \dfrac{f(x, y)}{f_Y(y)}$ when $f_Y(y) > 0$ .

Independence. $X \perp Y$ iff $f(x, y) = f_X(x) f_Y(y)$ for all $x, y$ . Equivalently $f_{X|Y}(x|y) = f_X(x)$ — conditioning provides no extra information.

Covariance. Measures linear co-movement: $\text{Cov}(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y].$

$\text{Cov} > 0$ : tend to increase together.
$\text{Cov} < 0$ : one tends to increase when the other decreases.
Independence ⇒ Cov = 0 (converse false in general).

Standardised version is correlation: $\rho = \dfrac{\text{Cov}}{\sigma_X \sigma_Y} \in [-1, 1]$ .

Worked problem. Verify integration domain $D = \{(x, y) : 0 \le x, 0 \le y, x + y \le 1\}$ .

Marginals. $f_X(x) = \int_0^{1-x} 6xy \, dy = 6x \cdot \frac{(1-x)^2}{2} = 3x(1-x)^2, \quad 0 \le x \le 1.$ By symmetry: $f_Y(y) = 3y(1-y)^2$ .

Independence check. $f_X(x) \cdot f_Y(y) = 9 x y (1-x)^2 (1-y)^2 \ne 6xy$ in general.

So $f(x, y) \ne f_X(x) f_Y(y)$ ⇒ $X$ and $Y$ are not independent.

(This is also evident from the support: the constraint $x + y \le 1$ couples $X$ and $Y$ — independence would require a rectangular support.)

Multivariate Random Variables and Joint Distributions