likelihood

The likelihood function is defined as the probability of the observed data $x$ given parameters $\theta$:

$$L(\theta \mid x)=P(x\mid \theta )$$

Note: we can think of probabilities and likelihoods as duals. They reflect the relationship between data and parameters from opposite perspectives:

• Probability is used for prediction and inference about data.
• Likelihood is used for estimation and inference about parameters.

log likelihood §

If the data are i.i.d., the likelihood function is often the product of the probabilities for each data point.

It’s often easier to work with the log likelihood function as it simplifies the multiplication into addition:

$$\ell (\theta \mid x)=\log L(\theta \mid x)=\sum _i\log P(x_i\mid \theta )$$

score §

The score function is the derivative of the log-likelihood w.r.t. $\theta$

$$S(\theta |x)=\frac{\partial }{\partial \theta }\ell (\theta |x)$$

Solving $S(\theta |x){|}_{\theta =\hat{\theta }=0}$ gives the maximum likelihood estimator of $\theta$.

At the true value of the parameter, the expected value of the score is 0:

$$E[S(\theta )]{\mid }_{\theta ={\theta }_0}=0$$

fisher information §

$$\begin{array}{rl}I(\theta )& =\text{Var}(S(\theta \mid X))\\ & =E({\text{score}}^2)-E(\text{score}{)}^2\\ & =E({\text{score}}^2)\end{array}$$