The variance of the WLS estimator
Summary
- Setup
- Random sample \(\left( y_i,x_i \right)\) for \(i=1, \ldots ,n\)
- Correctly specified linear regression model, \(y_i=x'_iβ+ε_i\) where \(E\left( ε_i \mid x_i \right)=0\) .
- \(Var\left( ε_i \mid x_i \right)=σ^2σ_i^2\) where \(σ^2\) is unknown and \(σ_i^2\) is known or in matrix form, \(Var\left( ε \mid X \right)=σ^2Ψ\) where \(Ψ=diag\left( σ_1^2, \ldots ,σ_n^2 \right)\) .
- Transformed variables \(y_i^*=y_i/σ_i\) , \(x_i^*=x_i/σ_i\) and \(ε_i^*=ε_i/σ_i\)
- Transformed variables, vector form: \(y^*=Ψ^{-1/2}y\) , \(X^*=Ψ^{-1/2}X\) , \(ε^*=Ψ^{-1/2}ε\)
- Transformed model: \(y_i^*={x_i^*}'β+ε_i^*\) or \(y^*=X^*β+ε^*\)
- WLS estimator \(b_{WLS}={\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}\sum_{i=1}^{n}{ x_i^*y_i^* }\) or
- In matrix form: \(b_{WLS}={\left( {X^*}'X^* \right)}^{-1}{X^*}'y^*={\left( X'Ψ^{-1}X \right)}^{-1}X'Ψ^{-1}y\)
- Result: \(Var\left( b_{WLS} \right|X^*)\)
\[Var\left( b_{WLS}|X^* \right)=σ^2{\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}=σ^2{\left( \sum_{i=1}^{n}{ \frac{x_ix'_i}{σ_i^2} } \right)}^{-1}\]
- or
\[Var\left( b_{WLS}|X^* \right)=σ^2{\left( X'Ψ^{-1}X \right)}^{-1}\]
- Definition: The transformed weighted least squares residuals \(e_i^*\) for \(1=1, \ldots ,n\) :
\[e_i^*=y_i^*-{x_i^*}'b_{WLS}= \frac{y_i-x'_ib_{WLS}}{σ_i}\]
- or
\[e^*=y^*-X^*b_{WLS}=Ψ^{-1/2}(y-Xb_{WLS})\]
- where \(e^*={\left( e_1^*, \ldots ,e_n^* \right)}'\) .
- Definition:
\[{s^*}^2= \frac{1}{n-k}\sum_{i=1}^{n}{ {\left( e_i^* \right)}^2 }= \frac{1}{n-k}\sum_{i=1}^{n}{ {\left( y_i^*-{x_i^*}'b_{WLS} \right)}^2 }= \frac{1}{n-k}\sum_{i=1}^{n}{ \frac{{\left( y_i-x'_ib_{WLS} \right)}^2}{σ_i^2} }\]
- or
\[{s^*}^2= \frac{1}{n-k}{e^*}'e^*= \frac{1}{n-k}{\left( y^*-X^*b_{WLS} \right)}'\left( y^*-X^*b_{WLS} \right)= \frac{1}{n-k}{\left( y-Xb_{WLS} \right)}'Ψ^{-1}(y-Xb_{WLS})\]
- Result \({s^*}^2\) is an unbiased estimator of \(σ^2\) .
- Result: the estimated variance of the WLS estimator
\[\widehat{Var }\left( b_{WLS}|X^* \right)={s^*}^2{\left( \sum_{i=1}^{n}{ x_i^*{x_i^*}' } \right)}^{-1}={s^*}^2{\left( \sum_{i=1}^{n}{ \frac{x_ix'_i}{σ_i^2} } \right)}^{-1}\]
- or
\[\widehat{Var }\left( b_{WLS}|X^* \right)={s^*}^2{\left( {X^*}'X^* \right)}^{-1}={s^*}^2{\left( X'Ψ^{-1}X \right)}^{-1}\]
- is an unbiased estimator of \(Var\left( b_{WLS}|X^* \right)\)