An alternative derivation of the WLS estimator
Problem
- Setup
- Random sample \(\left( y_i,x_i \right)\) for \(i=1, \ldots ,n\)
- Correctly specified linear regression model, \(y=Xβ+ε\) where \(E\left( ε \mid X \right)=0\) .
- \(Var\left( ε_i \mid x_i \right)=σ^2σ_i^2\) where \(σ^2\) is unknown and \(σ_i^2\) is known or in matrix form, \(Var\left( ε \mid X \right)=σ^2Ψ\) where \(Ψ=diag\left( σ_1^2, \ldots ,σ_n^2 \right)\) .
The OLS estimator can be derived by minimizing the objective function \(f\left( b \right)={\left( y-Xb \right)}'\left( y-Xb \right)\) . Show that the WLS estimator can be derived by minimizing the objective function
\[g\left( b \right)={\left( y-Xb \right)}'Ψ^{-1}\left( y-Xb \right)\]
Solution
\[g\left( b \right)={\left( y-Xb \right)}'Ψ^{-1}\left( y-Xb \right)=y'Ψ^{-1}y-b'X'Ψ^{-1}y-y'Ψ^{-1}Xb+b'X'Ψ^{-1}Xb=\]
\[y'Ψ^{-1}y-2y'Ψ^{-1}Xb+b'X'Ψ^{-1}Xb\]
since \(b'X'Ψ^{-1}y=y'Ψ^{-1}Xb\) (scalar, take transpose)
Differentiate with respect to \(b\) :
\[ \frac{dg}{db}=-2X'Ψ^{-1}y+2X'Ψ^{-1}Xb=0\]
or
\[X'Ψ^{-1}Xb=X'Ψ^{-1}y\]
Multiply both sides by \({\left( X'Ψ^{-1}X \right)}^{-1}\) :
\[b={\left( X'Ψ^{-1}X \right)}^{-1}X'Ψ^{-1}y\]