The least squares principle

Summary

Setup

• Given data $x_1,x_2,…,x_n$ and $y_1,y_2,…,y_n$ and a linear trendline

$$y=b_1+b_2x$$

• where $b_1$ and $b_2$ are arbitrary constants , the residuals $e_i$ will depend on $b_1$ and $b_2$ .
• Therefore, $RSS$ will depend on $b_1$ and $b_2$ .

Least squares principle:

Select $b_1$ and $b_2$ such that $RSS$ is minimized

• The problem

$$\min_{b_1,b_2} RSS=\min_{b_1,b_2} \sum_{i=1}^{n}{ e_i^2 }$$

• is called the least squares problem . The solution is given by the OLS formulas:

$$b_2= \frac{\sum_{i=1}^{n}{ \left( x_i-\bar{x} \right)\left( y_i-\bar{y} \right) }}{\sum_{i=1}^{n}{ {\left( x_i-\bar{x} \right)}^2 }}$$

$$b_1=\bar{y}-b_2\bar{x}$$