distributions

Discrete Distributions §

Bernoulli Distribution §

• Models: Single binary outcome (success/failure).
• Example: Flipping a coin.
• PMF:
  $$P(X=x)=p^x(1-p{)}^{1-x},x\in \{0,1\}$$

$$\mathbb{E}[X]=\sum _{x=0}^{1}x\cdot P(X=x)=0\cdot (1-p)+1\cdot p=p$$

$$\text{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2$$

$$\mathbb{E}[X^2]=\sum _{x=0}^1{x}^2\cdot P(X=x)=0^2\cdot (1-p)+1^2\cdot p=p$$

$$\text{Var}(X)=p-p^2=p(1-p)$$

Binomial Distribution §

X~Bin(n, p) = sum of i.i.d. Bern(p) RVs

• Models: Number of successes in $n$ independent Bernoulli trials.
• Example: Number of heads in $n$ coin flips.
• PMF:
  $$P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k},k=0,1,\dots ,n$$

$$\mathbb{E}[X]=np$$

$$\text{Var}(X)=np(1-p)$$

Multinomial Distribution §

• Models: Generalization of the binomial distribution for more than two outcomes.
• Example: Rolling a die 10 times and counting the occurrences of each face.
• PMF:
  $$P(X_1=k_1,\dots ,X_m=k_m)=\frac{n!}{k_1!\cdots k_m!}{p}_1^{k_1}\cdots p_m^{k_m}$$
  where $k_1+\cdots +k_m=n$ and $p_1+\cdots +p_m=1$.
  $$\mathbb{E}[X_i]=n{p}_i$$
  $$\text{Var}(X_i)=n{p}_i(1-p_i)$$
  $$\text{Cov}(X_i,X_j)=-n{p}_i{p}_j$$

Hypergeometric Distribution §

• Models: Number of successes in a n draws (without replacement)
• Example: Drawing 5 cards from a deck and counting the number of aces.
• PMF:
  $$P(X=k)=\frac{\left(\genfrac{}{}{0ex}{}{K}{k}\right)\left(\genfrac{}{}{0ex}{}{N-K}{n-k}\right)}{\left(\genfrac{}{}{0ex}{}{N}{n}\right)},\max (0,n+K-N)\le k\le \min (K,n)$$
  where $N$ is the population size, $K$ is the number of successes in the population, and $n$ is the sample size.
  $$\mathbb{E}[X]=n\frac{K}{N}$$
  $$\text{Var}(X)=n\frac{K}{N}\frac{N-K}{N}\frac{N-n}{N-1}$$

Negative hypergeometric distribution §

• Models: Number of draws (without replacement) to achieve r successes
• Example: Number of cards that must be drawn to collect 4 aces.
• PMF:

$$P(X=k)=\frac{\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)\left(\genfrac{}{}{0ex}{}{N-k}{K-r}\right)}{\left(\genfrac{}{}{0ex}{}{N}{K}\right)}$$

Geometric Distribution §

• Models: Number of trials until the first success.
• Example: Number of flips until first heads.
• PMF:
  $$P(X=k)=(1-p{)}^{k-1}p,k=1,2,\dots$$

$$\mathbb{E}[X]=\frac{1}{p}$$

$$\text{Var}(X)=\frac{1-p}{p^2}$$

Negative Binomial Distribution §

X~NBin(r, p) = sum of i.i.d. Geom(p) RVs

• Models: Number of trials needed to achieve k successes (inclusive of the k-th trial).
• Example: Number of coin flips required to get 3 heads.
• PMF:
  $$P(X=k)=\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)p^r(1-p{)}^{k-r},k=r,r+1,\dots$$
  where $r$ is the number of successes.
  $$\mathbb{E}[X]=\frac{r}{p}$$
  $$\text{Var}(X)=\frac{r(1-p)}{p^2}$$

Poisson Distribution §

$X\sim Pois(\lambda )$ models the #(events that occur in a unit of space or time). $\lambda$ is the expected number of events.

$$P(X=k)=\frac{{\lambda }^k{e}^{-\lambda }}{k!},k=0,1,2,\dots$$

$$\mathbb{E}[X]=\lambda$$

$$\text{Var}(X)=\lambda$$

Derivation §

Suppose you have n trials, each with $p=\frac{m}{n}$ of success. Then the probability of r successes can be modelled by the binomial distribution:

$$P(r\text{ successes})=\left(\genfrac{}{}{0ex}{}{n}{r}\right){\left(\frac{m}{n}\right)}^r{\left(1-\frac{m}{n}\right)}^{n-r}$$

As n tends to infinity, we have

$$\frac{1}{r!}\underset{{\to }_1}{\underbrace{\frac{n(n-1)\cdots (n-r+1)}{n^r}}}\cdot m^r\underset{\to e^{-m}}{\underbrace{{\left(1-\frac{m}{n}\right)}^n}}\underset{{\to }_1}{\underbrace{{\left(1-\frac{m}{n}\right)}^{-r}}}$$

This gives rise to the Poisson pmf:

$$P(r\text{ successes})\approx \frac{e^{-m}{m}^r}{r!}$$

Discrete Uniform Distribution §

• Models: All outcomes in a finite set are equally likely.
• Example: Rolling a fair die.
• PMF:
  $$P(X=k)=\frac{1}{n},k=1,2,\dots ,n$$
  $$\mathbb{E}[X]=\frac{n+1}{2}$$
  $$\text{Var}(X)=\frac{n^2-1}{12}$$

Continuous Distributions §

Continuous Uniform Distribution §

$$f(x)=\frac{1}{b-a},a\le x\le b$$

$$\mathbb{E}[X]=\frac{a+b}{2}$$

$$\text{Var}(X)=\frac{(b-a{)}^2}{12}$$

Exponential Distribution §

• Models: Time between events in a Poisson process.
• Example: Time between incoming calls.
• PDF:
  $$f(x)=\lambda e^{-\lambda x},x\ge 0$$

$$\mathbb{E}[X]=\frac{1}{\lambda }$$

$$\text{Var}(X)=\frac{1}{{\lambda }^2}$$

Gamma Distribution §

X~Gamma($\alpha ,\lambda$) models the amount of time until n events. E.g. Time until the $k^{th}$ earthquake.

• Gamma(n, λ) = sum of i.i.d. Expo(λ)
• Gamma(1, λ) ∼ Expo(λ)

Shape-Rate Parameterization: the preferred parameterization for Bayesian stats

$$f(x;\alpha ,\lambda )=\frac{{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}}{\Gamma (\alpha )},x>0$$

where $\Gamma (\alpha )$ is the Gamma function $\Gamma (\alpha )={\int }_0^{\infty }{t}^{\alpha -1}{e}^{-t}dt$

$$E(X)=\frac{\alpha }{\lambda }$$

$$Var(X)=\frac{\alpha }{{\lambda }^2}$$

Shape-Scale Parameterization: models the waiting time until the $\alpha$th event when each event occurs on average every $\theta$ units of time.

$$f(x;\alpha ,\theta )=\frac{1}{{\theta }^{\alpha }\Gamma (\alpha )}{x}^{\alpha -1}{e}^{-x/\theta },x>0$$

$$E(X)=\alpha \theta$$

$$Var(X)=\alpha {\theta }^2$$

Weibull Distribution §

• Models: Lifetimes of objects.
• Example: Time to failure of a machine.
• PDF:
  $$f(x)=\frac{k}{\lambda }{\left(\frac{x}{\lambda }\right)}^{k-1}{e}^{-(x/\lambda {)}^k},x\ge 0$$

$$\mathbb{E}[X]=\lambda \Gamma \left(1+\frac{1}{k}\right)$$

$$\text{Var}(X)={\lambda }^2\left[\Gamma \left(1+\frac{2}{k}\right)-{\left(\Gamma \left(1+\frac{1}{k}\right)\right)}^2\right]$$

Pareto distribution §

• Models: Heavy-tailed distributions, often used to model situations where a small number of occurrences account for the majority of the effect.
• Example: The distribution of wealth in a population, where a small percentage of people hold most of the wealth.
  $$f(x;x_m,\alpha )=\{\begin{array}{ll}\frac{\alpha x_m^{\alpha }}{x^{\alpha +1}}& \text{for }x\ge x_m,\\ 0& \text{for }x<x_m,\end{array}$$
  where $x_m$ is the scale parameter (minimum value) and $\alpha$ is the shape parameter.

$$\mathbb{E}[X]=\frac{\alpha x_m}{\alpha -1}$$

$$\text{Var}(X)=\{\begin{array}{ll}\frac{x_m^2\alpha }{(\alpha -1{)}^2(\alpha -2)}& \text{for }\alpha >2,\\ \text{undefined}& \text{for }\alpha \le 2\end{array}$$

Normal (Gaussian) Distribution §

$$f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$$

$$\mathbb{E}[X]=\mu$$

$$\text{Var}(X)={\sigma }^2$$

Bivariate Normal Distribution §

$$f(x,y)=\frac{1}{2\pi {\sigma }_x{\sigma }_y\sqrt{1-{\rho }^2}}\exp \left(-\frac{1}{2(1-{\rho }^2)}\left[\frac{(x-{\mu }_x{)}^2}{{\sigma }_x^2}-2\rho \frac{(x-{\mu }_x)(y-{\mu }_y)}{{\sigma }_x{\sigma }_y}+\frac{(y-{\mu }_y{)}^2}{{\sigma }_y^2}\right]\right)$$

$$\mathbb{E}[X]={\mu }_x,\mathbb{E}[Y]={\mu }_y$$

$$\text{Var}(X)={\sigma }_x^2,\text{Var}(Y)={\sigma }_y^2$$

$$\text{Cov}(X,Y)=\rho {\sigma }_x{\sigma }_y$$

Multivariate Normal Distribution §

$$f(\mathbf{x})=\frac{1}{\sqrt{(2\pi {)}^k|\Sigma |}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mu {)}^{\top }{\Sigma }^{-1}(\mathbf{x}-\mu )\right)$$

where $\mathbf{x}$ is a k-dimensional vector, $\mu$ is the mean vector, and $\Sigma$ is the covariance matrix.

$$\mathbb{E}[\mathbf{X}]=\mu$$

$$\text{Cov}(\mathbf{X})=\Sigma$$

Log-Normal Distribution §

• Models: Multiplicative processes.
• Example: Stock prices.
• PDF:
  $$f(x)=\frac{1}{x\sigma \sqrt{2\pi }}{e}^{-\frac{(\ln x-\mu {)}^2}{2{\sigma }^2}},x>0$$

$$\mathbb{E}[X]=e^{\mu +\frac{{\sigma }^2}{2}}$$

$$\text{Var}(X)=\left(e^{{\sigma }^2}-1\right)e^{2\mu +{\sigma }^2}$$

Chi-Square Distribution §

• Models: Sum of squares of normal variables.
• Example: Goodness-of-fit tests.
• PDF:
  $$f(x)=\frac{1}{2^{k/2}\Gamma (k/2)}{x}^{k/2-1}{e}^{-x/2},x\ge 0$$

$$\mathbb{E}[X]=k$$

$$\text{Var}(X)=2k$$

F-Distribution §

• Models: Ratio of two scaled chi-square distributions.
• Example: ANOVA testing.
• PDF:
  $$f(x)=\frac{\sqrt{{\left(\frac{d_{1}x}{d_{1}x+d_2}\right)}^{d_1}{\left(\frac{d_2}{d_{1}x+d_2}\right)}^{d_2}}}{xB\left(\frac{d_1}{2},\frac{d_2}{2}\right)},x\ge 0$$
  where $d_1$ and $d_2$ are the degrees of freedom.

For $d_2>2$:

$$\mathbb{E}[X]=\frac{d_2}{d_2-2}$$

For $d_2>4$:

$$\text{Var}(X)=\frac{2d_2^2(d_1+d_2-2)}{d_1(d_2-2{)}^2(d_2-4)}$$

The variance is undefined for $d_2\le 4$.

Beta Distribution §

• Models: Distribution of probabilities.
• Example: Distribution of success rates.
• PDF:
  $$f(x)=\frac{x^{\alpha -1}(1-x{)}^{\beta -1}}{B(\alpha ,\beta )},0\le x\le 1$$

where $B(\alpha ,\beta )={\int }_0^1{t}^{\alpha -1}(1-t{)}^{\beta -1}dt$ or $B(\alpha ,\beta )=\frac{\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}$

$$\mathbb{E}[X]=\frac{\alpha }{\alpha +\beta }$$

$$\text{Var}(X)=\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}$$

Dirichlet Distribution §

• Models: Probabilities of outcomes in a multinomial distribution.
• Example: Proportion of time spent on different activities during a day.
• PDF:
  $$f(x_1,\dots ,x_k)=\frac{1}{B(\alpha )}\prod _{i=1}^k{x}_i^{{\alpha }_i-1}$$
  where $B(\alpha )$ is the multinomial Beta function.
  $$\mathbb{E}[X_i]=\frac{{\alpha }_i}{{\sum }_{j=1}^k{\alpha }_j}$$
  $$\text{Var}(X_i)=\frac{{\alpha }_i({\alpha }_0-{\alpha }_i)}{{\alpha }_0^2({\alpha }_0+1)}$$
  where ${\alpha }_0={\sum }_{j=1}^k{\alpha }_j$.

t-Distribution §

• Models: Distribution of sample means when population variance is unknown.
• Example: Testing hypotheses about means.
• PDF:
  $$f(t)=\frac{\Gamma \left(\frac{\nu +1}{2}\right)}{\sqrt{\nu \pi }\Gamma \left(\frac{\nu }{2}\right)}{\left(1+\frac{t^2}{\nu }\right)}^{-\frac{\nu +1}{2}}$$
  where $\nu$ is the degrees of freedom.

For $\nu >1$:

$$\mathbb{E}[X]=0$$

For $\nu >2$:

$$\text{Var}(X)=\frac{\nu }{\nu -2}$$

The variance is undefined for $\nu \le 2$.

Cauchy Distribution §

• Models: Distributions with heavy tails.
• Example: Resonance behavior.
• PDF:
  $$f(x)=\frac{1}{\pi \left[1+(x-x_0{)}^2\right]}$$

statistical distance §

measures how different 2 probability distributions P and Q are from each other.

• asymmetric measure:
  • Kullback-Leibler Divergence: $D_{KL}(P\parallel Q)={\sum }_{x}P(x)\log \frac{P(x)}{Q(x)}$
    • MLE can be seen as minimizing the KL divergence
• symmetric measures:
  • total variation difference: $D_{TV}(P,Q)=\frac{1}{2}{\sum }_x|P(x)-Q(x)|$
  • Hellinger distance: $H^2(P,Q)=\frac{1}{2}{\sum }_x(\sqrt{P(x)}-\sqrt{Q(x)}{)}^2$