2. Ordinary Least Squares

In this chapter, unless otherwise noted, we will always assume the WCLM and will specify if additional assumptions are made.

2.1 OLS estimator

2.2 OLS under the SLM

Under the SLM $Y_i = \beta x_i + \epsilon_i$ we have the normal equation $\dvectr x \dvec x \betacheck = \dvectr x \vy$ and we get the following solution.

Under the SLMI $Y_i = \beta_1 + \beta_2 x_i + \epsilon_i$ we have the following normal equations $\dmattr X \dmat X \vbetacheck = \dmattr X \vy$: $$\begin{align*} &n\betacheck_1 + \betacheck_2 \sum_{i=1}^n x_i = \sum_{i=1}^n y_i \qand \\ &\betacheck_1 \sum_{i=1}^n x_i + \betacheck_2 \sum_{i=1}^n x_i^2 = \sum_{i=1}^n x_i y_i \end{align*}$$ We use the arithmetic mean $\mean{x} = \frac{1}{n} \sum_{i=1}^n x_i$ and the variables transforms $\check{\alpha} = \betacheck_1 + \betacheck_2 \mean{x}$ and $\check{\beta} = \betacheck_2$ and note the following algebraic sum idententities.

With that we get the solutions $\check{\alpha} = \mean{y}$ and $\check{\beta} = \frac{s_{xy}}{s_{xx}}$.

At the point $x=\mean{x}$ we have $\betacheck_1 + \betacheck_2 \mean{x} = \mean{x}$, hence the point $(\mean{x}, \mean{y})$ always lies on the fitted plane. A similar argument can be made for multiple linear regression with an intercept, i.e. $Y_i = \beta_1 + \sum_{i=2}^{p} \beta_i x_i + \epsilon_i$, where one can show that we have $\betacheck_1 = \mean{y} - \sum_{i=2}^{p} \betacheck_i \mean{x}_i$ and thus $(\mean{x}_2, \ldots, \mean{x}_p, \mean{y})$ lies on the fitted plane.

2.3 Geometric Interpretation

While we can interpret the model geometrically by looking at the rows, which shows that a model with intercept fits a $p{-}1$-dimensional hyperplane in a $p$-dimensional space through the $n$ points ${(x_{i,2} \ldots, x_{i,p}, y_i)}_{i=1}^{n}$, more mileage is to be had by interpreting the column vectors in the model.

The random vector $\vY$ of observations is a single point in $\R^n$. If we wary the value of the parameter $\vbeta$, the set $\Range{\dmat X} = \set{\dvec z \in \R^n \mid \dvec z = \dmat X \dvec b}$ describes the range of $\dmat X$, more specifically a $p$-dimensional hyperplane through the origin spanned by the columns of $\dmat X$. Then the OLS estimator $\vbetahat = \argmin_{\vbeta \in \R^p} \normbig{\vY - \dmat X \gvec \beta}^2$ is equivalent to $\vbetahat$ whose respective fitted value $\vYhat = \dmat X \vbetahat$ is the orthogonal projection of $\vY$ onto the hyperplane.

The set $\Null{\dmattr X} = \set{\dvec z \in \R^n \mid \dmattr X \dvec z = \rvec 0}$ is the null space of $\dmattr X$, more specifically the $n{-}p$-dimensional hyperplane through the origin orthogonal to $\Range{\dmat X}$.

2.4 Link to MLE under the SCLM

TODO

2.5 An intuition for the OLS parameters

The OLS coefficient $\betahat_j$ can be derived via the following three step procedure:

1. Perform OLS regression of $\dvec{x}_{:,j}$ against the other covariate observations $\set{ \dvec{x}_{:,k} \mid k \in \set{1,\ldots,n}\setminus \set{j}}$ to receive the residuals $\dvec r_j$.
2. Perform OLS regression of $\vY$ against the covariate observations $\set{ \dvec{x}_{:,k} \mid k \in \set{1,\ldots,n}\setminus \set{j}}$ to receive the residuals $\vepsilonhat_{\neg j}$.
3. Perform OLS regression of $\vepsilonhat_{\neg j}$ against $\dvec r_j$ to receive $\betahat_j$.

This is known as the Frisch-Waugh-Lovell Theorem. What this theorem hints at is that $\betahat_j$ measures the linear effect of $\dvec{x}_{:,j}$ on $\vY$ which is not explained by the linear effects of all other covariate observations $\set{ \dvec{x}_{:,k} \mid k \in \set{1,\ldots,n}\setminus \set{j}}$.

In general, this means that we cannot combine the parameters we get for the $n$ separate univariate OLS regression of $\vY$ agains $\dvec{x}_{:,j}$ to receive $\vbetahat$ as we have to factor in the linear effects of all other covariate observations. There is one notable exception where this is possible.