least squares

OLS (ordinary least squares) §

Least squares: method of fitting the linear model $\hat{Y}=X^T\hat{\beta }$, or

$$\hat{Y}=\widehat{{\beta }_0}+\sum _{j=1}^p{X}_j\widehat{{\beta }_j}$$

to a set of training data.

If $f(X)=X^{\top }\beta$, then we can pick coefficients $\beta =f^{\prime }(X)$ to minimize the residual sum of squares (RSS)

$$\mathrm{RSS}(\beta )=\sum _{i=1}^N{\left(y_i-x_i^T\beta \right)}^2$$

In matrix notation:

$$\mathrm{RSS}(\beta )=(\mathbf{y}-\mathbf{X}\beta {)}^T(\mathbf{y}-\mathbf{X}\beta )$$

Differentiating w.r.t. $\beta$ we get the normal equations

$${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\beta )=0$$

If $X^{\top }X$ is no-singular, then the unique solution is given by

$$\hat{\beta }=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$$

Regularized LS estimate §

$$\hat{\beta }=({\mathbf{X}}^{\top }\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$$

The regularized least squares estimate minimizes the following objective:

$$J(\theta )=\sum _{i=1}^n(y_i-X_i\theta {)}^2+\lambda \sum _{j=1}^{p}R({\theta }_j)$$

Where:

• $y_i$: observed values.
• $X_i$: input features.
• $\theta$: coefficients/parameters.
• $\lambda$: regularization parameter (controls the trade-off between fit and penalty).
• $R({\theta }_j)$: regularization term applied to the coefficients.

Ridge Regression (L2 Regularization):

• Regularization term: $R(\theta )=\Vert \theta {\Vert }_2^2$.
• Objective:

$$J(\theta )=\sum _{i=1}^n(y_i-X_i\theta {)}^2+\lambda \sum _{j=1}^p{\theta }_j^2$$

• Tends to shrink coefficients evenly, but does not force any coefficients to zero.
• Use when you suspect many small/medium-sized effects.

Lasso Regression (L1 Regularization):

• Regularization term: $R(\theta )=\Vert \theta {\Vert }_1$.
• Objective:

$$J(\theta )=\sum _{i=1}^n(y_i-X_i\theta {)}^2+\lambda \sum _{j=1}^p|{\theta }_j|$$

• Can drive some coefficients exactly to zero, effectively performing feature selection.
• Use when you expect only a few variables to have significant effects.

WLS (weighted least squares) §

A generalization of OLS, used when error variances are not equal (hetereoskedastic).

The difficulty is estimating error variances - they are rarely known exactly. Weights are based on the estimated variances:

Let the weights be

$$w_i=\frac{1}{{\hat{\sigma }}_i^2}$$

Then the estimator is given by

$${\hat{\beta }}_{\text{WLS}}={\left(X^{\top }WX\right)}^{-1}{X}^{\top }Wy$$