statistics

Statistics is the study of data collection, analysis, interpretation, and presentation to draw meaningful conclusions.

Etymology: Statistik (German) - description of a state/country; used to be called political arithmetic

Methods:

• descriptive statistics (location, scale, shape, count, dependence)
• inferential statistics

branches §

• probability
• applied statistics:
  • sampling
  • regression
  • frequentist inference
    • estimation
    • hypothesis testing
  • bayesian statistics
  • statistical learning
  • non-parametric methods
  • time series analysis
  • multivariate analysis

criteria §

sufficiency §

A statistic $T(X_1,\dots ,X_n)$ is sufficient for the parameter $\theta$ if:

• $f_{\theta }(X_1,\dots ,X_n\mid T=t)$ does not depend on $\theta$.

• OR if the density can be written as:

  $$f_{\theta }(s)=h(s)g_{\theta }(T(s))$$

  where $h$ is a function of the data only, $g$ is a function of $T$ and $\theta$. Both $h$ and $g$ are non-negative functions.

With a sufficient statistic, we no longer need the original data to make inferences about the parameter.

E.g. the sample mean $\bar{X}$ is a sufficient statistic for the population mean $\mu$.

Properties §

• Sufficient statistics provide data reduction.
• Sufficient statistics are not unique.
• Maximizing $f_{\theta }(s)$ is equivalent to maximizing $g_{\theta }(T(s))$.
  • If the MLE of $\theta$ is unique, then it is a function of $T(s)$.
• One-to-one functions of a sufficient statistic are also sufficient.

Minimal Sufficient §

A statistic $T$ is minimal sufficient if any of the following is true:

• The value of $T(s)$ can be calculated from the likelihood function $L(\cdot \mid s)$.
• OR $T$ is a function of a sufficient statistic.
• OR $T$ is a sufficient statistic that is also a function of the MLE.

completeness §

A statistic is complete if there is no non-trivial (i.e., not always zero) function of the statistic that has an expected value of zero for all values of the parameter. Completeness ensures that the statistic captures all the information about the parameter.

A statistic $T(X)$ is complete if for every measurable function $g$, the condition:

$$\mathbb{E}[g(T(X))]=0\text{for all }\theta$$

implies that $g(T(X))=0$ almost surely.

If a statistic is both sufficient and complete, it’s typically the best (most informative) statistic you can use for estimation. MVUEs (Minimum Variance Unbiased Estimators) are derived from complete, sufficient statistics.