The variance of the WLS estimator

Summary

  • Setup
    • Random sample \(\left( y_i,x_i \right)\) for \(i=1, \ldots ,n\)
    • Correctly specified linear regression model, \(y_i=x'_iβ+ε_i\) where \(E\left( ε_i \mid x_i \right)=0\) .
    • \(Var\left( ε_i \mid x_i \right)=σ^2σ_i^2\) where \(σ^2\) is unknown and \(σ_i^2\) is known or in matrix form, \(Var\left( ε \mid X \right)=σ^2Ψ\) where \(Ψ=diag\left( σ_1^2, \ldots ,σ_n^2 \right)\) .
    • Transformed variables \(y_i^*=y_i/σ_i\) , \(x_i^*=x_i/σ_i\) and \(ε_i^*=ε_i/σ_i\)
    • Transformed variables, vector form: \(y^*=Ψ^{-1/2}y\) , \(X^*=Ψ^{-1/2}X\) , \(ε^*=Ψ^{-1/2}ε\)
    • Transformed model: \(y_i^*={x_i^*}'β+ε_i^*\) or \(y^*=X^*β+ε^*\)
    • WLS estimator \(b_{WLS}={\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}\sum_{i=1}^{n}{ x_i^*y_i^* }\) or
    • In matrix form: \(b_{WLS}={\left( {X^*}'X^* \right)}^{-1}{X^*}'y^*={\left( X'Ψ^{-1}X \right)}^{-1}X'Ψ^{-1}y\)
  • Result: \(Var\left( b_{WLS} \right|X^*)\)

\[Var\left( b_{WLS}|X^* \right)=σ^2{\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}=σ^2{\left( \sum_{i=1}^{n}{ \frac{x_ix'_i}{σ_i^2} } \right)}^{-1}\]

  • or

\[Var\left( b_{WLS}|X^* \right)=σ^2{\left( X'Ψ^{-1}X \right)}^{-1}\]

  • Definition: The transformed weighted least squares residuals \(e_i^*\) for \(1=1, \ldots ,n\) :

\[e_i^*=y_i^*-{x_i^*}'b_{WLS}= \frac{y_i-x'_ib_{WLS}}{σ_i}\]

  • or

\[e^*=y^*-X^*b_{WLS}=Ψ^{-1/2}(y-Xb_{WLS})\]

  • where \(e^*={\left( e_1^*, \ldots ,e_n^* \right)}'\) .
  • Definition:

\[{s^*}^2= \frac{1}{n-k}\sum_{i=1}^{n}{ {\left( e_i^* \right)}^2 }= \frac{1}{n-k}\sum_{i=1}^{n}{ {\left( y_i^*-{x_i^*}'b_{WLS} \right)}^2 }= \frac{1}{n-k}\sum_{i=1}^{n}{ \frac{{\left( y_i-x'_ib_{WLS} \right)}^2}{σ_i^2} }\]

  • or

\[{s^*}^2= \frac{1}{n-k}{e^*}'e^*= \frac{1}{n-k}{\left( y^*-X^*b_{WLS} \right)}'\left( y^*-X^*b_{WLS} \right)= \frac{1}{n-k}{\left( y-Xb_{WLS} \right)}'Ψ^{-1}(y-Xb_{WLS})\]

  • Result \({s^*}^2\) is an unbiased estimator of \(σ^2\) .
  • Result: the estimated variance of the WLS estimator

\[\widehat{Var }\left( b_{WLS}|X^* \right)={s^*}^2{\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}={s^*}^2{\left( \sum_{i=1}^{n}{ \frac{x_ix'_i}{σ_i^2} } \right)}^{-1}\]

  • or

\[\widehat{Var }\left( b_{WLS}|X^* \right)={s^*}^2{\left( {X^*}'X^* \right)}^{-1}={s^*}^2{\left( X'Ψ^{-1}X \right)}^{-1}\]

  • is an unbiased estimator of \(Var\left( b_{WLS}|X^* \right)\)