The least squares principle

Summary

Setup

• Given data \(x_1,x_2,…,x_n\) and \(y_1,y_2,…,y_n\) and a linear trendline

\[y=b_1+b_2x\]

• where \(b_1\) and \(b_2\) are arbitrary constants , the residuals \(e_i\) will depend on \(b_1\) and \(b_2\) .
• Therefore, \(RSS\) will depend on \(b_1\) and \(b_2\) .

Least squares principle:

Select \(b_1\) and \(b_2\) such that \(RSS\) is minimized

• The problem

\[\min_{b_1,b_2} RSS=\min_{b_1,b_2} \sum_{i=1}^{n}{ e_i^2 }\]

• is called the least squares problem . The solution is given by the OLS formulas:

\[b_2= \frac{\sum_{i=1}^{n}{ \left( x_i-\bar{x} \right)\left( y_i-\bar{y} \right) }}{\sum_{i=1}^{n}{ {\left( x_i-\bar{x} \right)}^2 }}\]

\[b_1=\bar{y}-b_2\bar{x}\]