An alternative derivation of the WLS estimator

Problem

  • Setup
    • Random sample \(\left( y_i,x_i \right)\) for \(i=1, \ldots ,n\)
    • Correctly specified linear regression model, \(y=Xβ+ε\) where \(E\left( ε \mid X \right)=0\) .
    • \(Var\left( ε_i \mid x_i \right)=σ^2σ_i^2\) where \(σ^2\) is unknown and \(σ_i^2\) is known or in matrix form, \(Var\left( ε \mid X \right)=σ^2Ψ\) where \(Ψ=diag\left( σ_1^2, \ldots ,σ_n^2 \right)\) .

The OLS estimator can be derived by minimizing the objective function \(f\left( b \right)={\left( y-Xb \right)}'\left( y-Xb \right)\) . Show that the WLS estimator can be derived by minimizing the objective function

\[g\left( b \right)={\left( y-Xb \right)}'Ψ^{-1}\left( y-Xb \right)\]

Solution

\[g\left( b \right)={\left( y-Xb \right)}'Ψ^{-1}\left( y-Xb \right)=y'Ψ^{-1}y-b'X'Ψ^{-1}y-y'Ψ^{-1}Xb+b'X'Ψ^{-1}Xb=\]

\[y'Ψ^{-1}y-2y'Ψ^{-1}Xb+b'X'Ψ^{-1}Xb\]

since \(b'X'Ψ^{-1}y=y'Ψ^{-1}Xb\) (scalar, take transpose)

Differentiate with respect to \(b\) :

\[ \frac{dg}{db}=-2X'Ψ^{-1}y+2X'Ψ^{-1}Xb=0\]

or

\[X'Ψ^{-1}Xb=X'Ψ^{-1}y\]

Multiply both sides by \({\left( X'Ψ^{-1}X \right)}^{-1}\) :

\[b={\left( X'Ψ^{-1}X \right)}^{-1}X'Ψ^{-1}y\]