probability

the language of statistics. See distributions.

convergence §

3 types §

from strongest to weakest:

• almost sure: convergence happens with probability 1
  • $\mathrm{Pr}\left({\lim }_{n\to \infty }{X}_n=X\right)=1$
• in probability: the probability of being close to the limit increases
  • $\forall ϵ>0,\mathrm{Pr}\left(|X_n-X|\ge ϵ\right)\to 0$
• in distribution: the distribution of the variable approximates the distribution of the limit
  • $F_{X_n}(x)\to F_X(x)$ for all $x$ where $F_X$ is continuous

sequences §

If $\{X_n\}$ and $\{Y_n\}$ are sequences of random variables (RVs) such that $X_n\stackrel{P}{\to }X$ and $Y_n\stackrel{P}{\to }Y$ in probability, then:

• $X_n+Y_n\stackrel{P}{\to }X+Y$
• $X_n{Y}_n\stackrel{P}{\to }XY$

continuous mapping theorem §

If $g$ is a continuous function, then:

$$g(X_n)\stackrel{P}{\to }g(X)$$

LLN §

$\frac{1}{n}{\sum }_{i=1}^n{X}_i\to \mathbb{E}[X]$

strong: almost sure convergence

weak: convergence in probability

CLT §

$\frac{\sqrt{n}({\bar{X}}_n-\mu )}{\sigma }\stackrel{d}{\to }N(0,1)$

Law of total variance §

$\mathrm{Var}(X)=\mathbb{E}(\mathrm{Var}(X|Y))+\mathrm{Var}(\mathbb{E}(X|Y))$

This is useful for separating uncertainty into systematic variation (2nd term: variability explained by Y) and random noise (1st term: variability unexplained by Y).

Intuition:

• 1st term: average variance within groups created by Y
  • given Y, what is the residual variance of X?
• 2nd term: variance between group means
  • how much does the expected value of X change as Y varies?

Law of total expectation §

$$\mathbb{E}(X)=\mathbb{E}(\mathbb{E}(X|Y))$$

For continuous Y:

$$\mathbb{E}[X]={\int }_{-\infty }^{\infty }\mathbb{E}[X\mid Y=y]f_Y(y)dy$$

For discrete Y:

$$\mathbb{E}[X]=\sum _{y_i}\mathbb{E}\left[X\mid Y=y_i\right]P\left(Y=y_i\right)$$

Inequalities §

Cauchy-Schwarz §

Recall the most common form: $|\mathbf{u}\cdot \mathbf{v}|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert$ And the integral form:

$${\left({\int }_{D}f(x)g(x)d\mu (x)\right)}^2\le \left({\int }_{D}f(x{)}^{2}d\mu (x)\right)\left({\int }_{D}g(x{)}^{2}d\mu (x)\right)$$

The probabilistic form is:

$$\begin{array}{rl}\mathbb{E}[XY{]}^2& \le \mathbb{E}[X^2]\mathbb{E}[Y^2]\\ |\mathbb{E}(XY)|& \le \sqrt{\mathbb{E}\left(X^2\right)\mathbb{E}\left(Y^2\right)}\end{array}$$

Upper bound: this becomes an inequality f there is perfect correlation, i.e. $Y=cX$.

Markov §

for $a>0$

$$P(X\ge a)\le \frac{E|X|}{a}$$

Use case: when you only know the mean and need an upper bound on how often a random variable gets large

Chebyshev §

for $E(X)=\mu ,V(X)={\sigma }^2$,

$$P(|X-\mu |\ge a)\le \frac{{\sigma }^2}{a^2}$$

Use case: when you know the mean and variance and need an upper bound on the spread of the distribution

Jensen §

$E(g(X))\ge g(E(X))$ for $g$ convex; reverse if $g$ is concave