OLS

Summary

Setup

The linear regression model with random sampling,

\[y_i=β_1+β_2x_{i,2}+β_3x_{i,3}+…+β_kx_{i,k}+ε_i, i=1,…,n\]

The least squares principle and the OLS estimator

• If \(b_1,b_2,…,b_k\) are arbitrary constants, we define fitted values as

\[{\hat{y}}_i=b_1+b_2x_{i,2}+b_3x_{i,3}+…+b_kx_{i,3} i=1,…,n\]

• We define the residuals as

\[e_i=y_i-{\hat{y}}_i=y_i-b_1-b_2x_{i,2}-b_3x_{i,3}-…-b_kx_{i,k} i=1,…,n\]

• RSS ( Residual Sum of Squares ) is

\[RSS=\sum_{i=1}^{n}{ e_i^2 }\]

• Least squares principle : Select \(b_1,b_2,…b_k\) such that \(RSS\) is minimized .
• This problem can be solved analytically giving us a formula for \(b_1,b_2,…b_k\) which we call the OLS formula or the OLS estimator.
• For \(k>2\) you need matrix notation to express them.

OLS fit

• Using the OLS estimates for \(b_1,b_2,…b_k\) in the formula for fitted values and residuals, gives us the OLS fitted values and OLS residuals.
• The OLS results from the model with one explanatory variable will continue to hold.
• \(TSS\) , \(ESS\) , \(RSS\) are defined as before and \(TSS=ESS+RSS\)
• \(R^2\) is defined as before
• We define the adjusted \(R^2\) , denoted by \({\bar{R}}^2\) as

\[{\bar{R}}^2=1-\left( 1-R^2 \right) \frac{n-1}{n-k}=R^2-\left( 1-R^2 \right) \frac{k-1}{n-k}\]