linear regression

see also: least squares

assumptions §

• Linearity: $Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+\cdots +{\beta }_p{X}_p+ϵ$
• Independence: $Cov({ϵ}_i,{ϵ}_j)=0,i\ne j$
• Normality of error terms: $ϵ\sim N(0,{\sigma }^2)$
• Equal variance (homoscedasticity): $Var({ϵ}_i)={\sigma }^2$
• No perfect multicollinearity: $\rho (x_i,x_j)\ne \pm 1,i\ne j$

types §

• simple regression: 1 predictor, 1 response variable
• multiple regression: multiple predictors, 1 response variable
• multivariate regression: multiple predictors, multiple response variables

simple regression §

$$\begin{array}{rl}{Y}_i& ={\beta }_0+{\beta }_1{X}_i+{ϵ}_i\\ E(Y_i)& ={\beta }_0+{\beta }_1{X}_i\\ Var(Y_i)& ={\sigma }^2\end{array}$$

Residuals are predicted values of error ${ϵ}_i=Y_i-E(Y_i|X_i)$

$$e_i=Y_i-\widehat{Y_i}$$

The least squares estimators are BLUE: Best Linear Unbiased Estimators.

$$\begin{array}{rl}\widehat{{\beta }_0}& =\bar{Y}-\widehat{{\beta }_1}\bar{X}\\ \widehat{{\beta }_1}& =\frac{\hat{Y}-\bar{Y}}{X-\bar{X}}\end{array}$$

The fitted regression line is given by

$$\begin{array}{rl}\hat{Y}& =\underset{\bar{Y}-\widehat{{\beta }_1}\bar{X}}{\underbrace{\widehat{{\beta }_0}}}+\widehat{{\beta }_1}X\\ & =\bar{Y}+(X-\bar{X})\widehat{{\beta }_1}\end{array}$$

It always passes through $(\bar{X},\bar{Y})$.

ANOVA (analysis of variance) §

• sum of squares
  • SST (total): total variability in the response.
    • $\text{SST}={\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2=S_{YY}$
  • SSR (regression): variability explained by the model.
    • $\text{SSR}={\sum }_{i=1}^n({\hat{Y}}_i-\bar{Y}{)}^2$
  • SSE (error): variability unexplained by the model.
    • $\text{SSE}={\sum }_{i=1}^n(Y_i-{\hat{Y}}_i{)}^2$
    • quantity to minimize!
• mean squares
  • MSR (regression): average variability explained by the model.
    • $MSR=\frac{SSR}{p}$
  • MSE (error): average variability unexplained by the model.
    • $MSE=\frac{SSE}{n-p-1}$
• F-statistic: tests overall significance of the model.
  • $F=\frac{\text{MSR}}{\text{MSE}}$

evaluation §

• coefficient of determination, or R-squared: proportion of variance in $Y$ explained by $X$. $R^2=1-\frac{\text{SSE}}{\text{SST}}=\frac{SSR}{SST}$
  • in simple regression, $R^2=r^2$
• adjusted R-squared: adjusts $R^2$ for the number of predictors.

$$\begin{array}{rl}{R}_{adj}^2& =1-\frac{\frac{SSE}{n-p^{\prime }}}{\frac{SST}{n-1}}=1-\left(\frac{n-1}{n-p^{\prime }}\right)\frac{SSE}{SST}\\ & =1-\frac{(1-R^2)(n-1)}{n-p^{\prime }}\end{array}$$

diagnostics §

• Residual Analysis: Check residual plots for patterns (indicating violations of assumptions).
• Multicollinearity: High correlation among predictors.
  • Detected via Variance Inflation Factor (VIF).
• Heteroscedasticity: Non-constant variance of residuals.
  • Detected via residual plots; corrected by transformation or weighted least squares.
• Autocorrelation: Correlation of residuals with themselves over time.
  • Detected via Durbin-Watson test.

extension §

Polynomial Regression: Model non-linear relationships. $Y={\beta }_0+{\beta }_{1}X+{\beta }_2{X}^2+\cdots +ϵ$ Interaction Terms: Capture combined effects of predictors. $Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3(X_1\times X_2)+ϵ$

Ridge Regression: Adds $\lambda {\sum }_{j=1}^p{\beta }_j^2$ penalty.

Lasso Regression: Adds $\lambda \sum_{j=1}^p |\beta_j| penalty.