Weighted least squares

Summary

• Setup
• Random sample \(\left( y_i,x_i \right)\) for \(i=1, \ldots ,n\)
• Correctly specified linear regression model, \(y_i=x'_iβ+ε_i\) where \(E\left( ε_i \mid x_i \right)=0\) .
• \(Var\left( ε_i \mid x_i \right)=σ^2σ_i^2\) where \(σ^2\) is unknown and \(σ_i^2\) is known or in matrix form, \(Var\left( ε \mid X \right)=σ^2Ψ\) where \(Ψ=diag\left( σ_1^2, \ldots ,σ_n^2 \right)\) .
• Transformed variables \(y_i^*=y_i/σ_i\) , \(x_i^*=x_i/σ_i\) and \(ε_i^*=ε_i/σ_i\)
• Transformed variables, vector form: \(y^*=Ψ^{-1/2}y\) , \(X^*=Ψ^{-1/2}X\) , \(ε^*=Ψ^{-1/2}ε\)
• Transformed model: \(y_i^*={x_i^*}'β+ε_i^*\) or \(y^*=X^*β+ε^*\)
• Definition: the OLS estimator of the transformed model is called the weighted least squares (WLS) estimator or \(β\) .

\[b_{WLS}={\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}\sum_{i=1}^{n}{ x_i^*y_i^* }\]

• or in matrix form

\[b_{WLS}={\left( {X^*}'X^* \right)}^{-1}{X^*}'y^*\]

• The WLS estimator written in terms of the original variables

\[b_{WLS}={\left( \sum_{i=1}^{n}{ \frac{x_ix'_i}{σ_i^2} } \right)}^{-1}\sum_{i=1}^{n}{ \frac{x_iy_i}{σ_i^2} }\]

• or

\[b_{WLS}={\left( X'Ψ^{-1}X \right)}^{-1}X'Ψ^{-1}y\]