Chapter 5
Generalized and Weighted Least Squares Estimation
The usual linear regression model assumes that all the random error components are identically and
independently distributed with constant variance. When this assumption is violated, then ordinary least
squares estimator of the regression coefficient loses its property of minimum variance in the class of linear
and unbiased estimators. The violation of such assumption can arise in anyone of the following situations:
1. The variance of random error components is not constant.
2. The random error components are not independent.
3. The random error components do not have constant variance as well as they are not independent.
In such cases, the covariance matrix of random error components does not remain in the form of an identity
matrix but can be considered as any positive definite matrix. Under such assumption, the OLSE does not
remain efficient as in the case of an identity covariance matrix. The generalized or weighted least squares
method is used in such situations to estimate the parameters of the model.
In this method, the deviation between the observed and expected values of yi is multiplied by a weight i
where i is chosen to be inversely proportional to the variance of yi .
For a simple linear regression model, the weighted least squares function is
n
S ( 0 , 1 ) i yi 0 1 xi .
2
The least-squares normal equations are obtained by differentiating S ( 0 , 1 ) with respect to 0 and 1 and
equating them to zero as
n n n
ˆ0 i ˆ1 i xi i yi
i 1 i 1 i 1
n n n
ˆ0 i xi ˆ1 i xi2 i xi yi .
i 1 i 1 i 1
The solution of these two normal equations gives the weighted least squares estimate of 0 and 1 .
Econometrics | Chapter 5 | Generalized and Weighted Least Squares Estimation | Shalabh, IIT Kanpur
1
, Generalized least squares estimation
Suppose in usual multiple regression model
y X with E ( ) 0, V ( ) 2 I ,
the assumption V ( ) 2 I is violated and become
V ( ) 2
where is a known n n nonsingular, positive definite and symmetric matrix.
This structure of incorporates both the cases.
- when is diagonal but with unequal variances and
- when is not necessarily diagonal depending on the presence of correlated errors, some of the
diagonal elements are nonzero.
The OLSE of is
b ( X ' X ) 1 X ' y
In such cases, OLSE gives unbiased estimate but has more variability as
E (b) ( X ' X ) 1 X ' E ( y ) ( X ' X ) 1 X ' X
V (b) ( X ' X ) 1 X 'V ( y ) X ( X ' X ) 1 2 ( X ' X ) 1 X ' X ( X ' X ) 1.
Now we attempt to find better estimator as follows:
Since is positive definite, symmetric, so there exists a nonsingular matrix K such that.
KK ' .
Then in the model
y X ,
premutliply by K 1 , this gives
K 1 y K 1 X K 1
or z B g
where z K 1 y, B K 1 X , g K 1 . Now observe that
E ( g ) K 1) E ( ) 0
and
Econometrics | Chapter 5 | Generalized and Weighted Least Squares Estimation | Shalabh, IIT Kanpur
2
Generalized and Weighted Least Squares Estimation
The usual linear regression model assumes that all the random error components are identically and
independently distributed with constant variance. When this assumption is violated, then ordinary least
squares estimator of the regression coefficient loses its property of minimum variance in the class of linear
and unbiased estimators. The violation of such assumption can arise in anyone of the following situations:
1. The variance of random error components is not constant.
2. The random error components are not independent.
3. The random error components do not have constant variance as well as they are not independent.
In such cases, the covariance matrix of random error components does not remain in the form of an identity
matrix but can be considered as any positive definite matrix. Under such assumption, the OLSE does not
remain efficient as in the case of an identity covariance matrix. The generalized or weighted least squares
method is used in such situations to estimate the parameters of the model.
In this method, the deviation between the observed and expected values of yi is multiplied by a weight i
where i is chosen to be inversely proportional to the variance of yi .
For a simple linear regression model, the weighted least squares function is
n
S ( 0 , 1 ) i yi 0 1 xi .
2
The least-squares normal equations are obtained by differentiating S ( 0 , 1 ) with respect to 0 and 1 and
equating them to zero as
n n n
ˆ0 i ˆ1 i xi i yi
i 1 i 1 i 1
n n n
ˆ0 i xi ˆ1 i xi2 i xi yi .
i 1 i 1 i 1
The solution of these two normal equations gives the weighted least squares estimate of 0 and 1 .
Econometrics | Chapter 5 | Generalized and Weighted Least Squares Estimation | Shalabh, IIT Kanpur
1
, Generalized least squares estimation
Suppose in usual multiple regression model
y X with E ( ) 0, V ( ) 2 I ,
the assumption V ( ) 2 I is violated and become
V ( ) 2
where is a known n n nonsingular, positive definite and symmetric matrix.
This structure of incorporates both the cases.
- when is diagonal but with unequal variances and
- when is not necessarily diagonal depending on the presence of correlated errors, some of the
diagonal elements are nonzero.
The OLSE of is
b ( X ' X ) 1 X ' y
In such cases, OLSE gives unbiased estimate but has more variability as
E (b) ( X ' X ) 1 X ' E ( y ) ( X ' X ) 1 X ' X
V (b) ( X ' X ) 1 X 'V ( y ) X ( X ' X ) 1 2 ( X ' X ) 1 X ' X ( X ' X ) 1.
Now we attempt to find better estimator as follows:
Since is positive definite, symmetric, so there exists a nonsingular matrix K such that.
KK ' .
Then in the model
y X ,
premutliply by K 1 , this gives
K 1 y K 1 X K 1
or z B g
where z K 1 y, B K 1 X , g K 1 . Now observe that
E ( g ) K 1) E ( ) 0
and
Econometrics | Chapter 5 | Generalized and Weighted Least Squares Estimation | Shalabh, IIT Kanpur
2
