Proof of the Gauss Markov theorem

Problem

• Setup:
• a linear regression model \(y=Xβ+ε\) with a random sample
• the Gauss-Markov assumptions, \(E\left( ε \right|X)=0\) and \(Var\left( ε \right|X)=σ^2I\)
• \(b={\left( X'X \right)}^{-1}X'y\) is the OLS estimator of \(β\)

Let \(\tilde{b}=Ay\) be an arbitrary unbiased linear estimator of \(β\) where \(A\) is an arbitrary \(k×n\) matrix that may depend on \(X\) . ( \(A={\left( X'X \right)}^{-1}X'\) for the OLS estimator). Show that

\[Var\left( \tilde{b}|X \right)-Var\left( b \right|X)\]

is a positive semidefinite matrix.

Hints

a) Evaluate \(E\left( \tilde{b} \mid X \right)\) and show that unbiasedness of \(\tilde{b}\) implies that \(AX=I\)

b) Use the following neat idea (conditionally on \(X\) )

\[Var\left( \tilde{b} \right)=Var\left( b+\left( \tilde{b}-b \right) \right)\]

Now show that

\[Var\left( b+\left( \tilde{b}-b \right) \right)=\]

\[=E\left( \left( b-β \right)\left( b-β \right)' \right)+E\left( \left( b-β \right)\left( \tilde{b}-b \right)' \right)+E\left( \left( \tilde{b}-b \right)\left( b-β \right)' \right)+E\left( \left( \tilde{b}-b \right)\left( \tilde{b}-b \right)' \right)\]

c) Let’s focus on the second term, \(E\left( \left( b-β \right)\left( \tilde{b}-b \right)' \right)\) .

Before attacking this term, we have

\[\tilde{b}-b=Ay-{\left( X'X \right)}^{-1}X'y=\left( A-{\left( X'X \right)}^{-1}X' \right)y\]

Define \(C= A-{\left( X'X \right)}^{-1}X'\) such that

\[\tilde{b}-b=Cy\]

Show that \(AX=I\) implies that \(CX=0\) and \(Cy=Cε\)

d) Now show that

\[E\left( \left( b-β \right)\left( \tilde{b}-b \right)' \right)=0\]

as well as

\[E\left( \left( \tilde{b}-b \right)\left( b-β \right)' \right)=0\]

(this is a \(k×k\) matrix of zeroes). You are now actually done! You have showed that (conditionally on \(X\) )

\[Var\left( \tilde{b} \right)=Var\left( b+\left( \tilde{b}-b \right) \right)=Var\left( b \right)+Var\left( \left( \tilde{b}-b \right) \right)\]

or

\[Var\left( \tilde{b} \right)-Var\left( b \right)=Var\left( \left( \tilde{b}-b \right) \right)\]

The right-hand side, being a variance matrix, must be positive semi definite.