Maximum likelihood, one variable with explanatory variables
Summary
Setup and statistical model
- Given: random sample \(\left( y_i,x_{i,1}, \ldots ,x_{i,k} \right)\) for \(i=1, \ldots ,n\) .
- Result: The density/distribution of \(\left( y_i,x_i \right)\) , where \(x_i=\left( x_{i,1}, \ldots ,x_{i,k} \right)\) , can be written as a product of a conditional density/distribution \(f_C\left( y_i \mid x_i \right)\) of \(y_i\) given \(x_i\) and the density/distribution of \(x_i\) , denoted by \(f_x\left( x_i \right)\) :
\[f\left( y_i,x_i \right)=f_C\left( y_i \mid x_i \right)f_x\left( x_i \right)\]
- We postulate a statistical model by assuming that
- the joint conditional density/distribution of \(y_i\) given \(x_i\) , \(f_C\left( y_i \mid x_i \right)\) , is known up to an unknown \(p×1\) vector of parameters \(θ\) . We denote it by \(f_C\left( y_i \mid x_i;θ \right)\)
- the density/distribution of \(x_i\) , \(f_x\left( x_i \right)\) , does not depend on \(θ\) .
Likelihood
- The likelihood contribution:
\[L_i\left( θ \right)=f_C\left( y_i \mid x_i;θ \right)f_x\left( x_i \right)\]
- The likelihood function
\[L\left( θ \right)=\_{i=1}^{n}{ f_C\left( y_i \mid x_i;θ \right)f_x\left( x_i \right) }=\_{i=1}^{n}{ f_C\left( y_i \mid x_i;θ \right) }\_{i=1}^{n}{ f_x\left( x_i \right) }\]
- Result: the argument that optimizes the complete likelihood function will be the same as the argument that optimizes
\[\_{i=1}^{n}{ f_C\left( y_i \mid x_i;θ \right) }\]
- The second part does not depend on \(θ\) and it is simply a constant in the optimization.
- Therefore, we drop the second part and redefine
\[L_i\left( θ \right)=f_C\left( y_i \mid x_i;θ \right)\]
- and
\[L\left( θ \right)=\_{i=1}^{n}{ f_C\left( y_i \mid x_i;θ \right) }\]
- The distribution of the x-data will be irrelevant . All we care about is the conditional densities \(f_C\left( y_i \mid x_i;θ \right)\) .
- From this point on, everything is the same as ML estimation for one variable. Instead of specifying a family for the density of \(y_i\) , we specify a family for the conditional density of \(y_i\) given \(x_i\) .