Comparison of MINQUE and Simple Estimate of the Error Variance in the General Linear Models

Transcription

1 Acta Mathematicae Applicatae Sinica, English Series Vol. 19, No. 1 (003) Comparison of MINQUE and Simple Estimate of the Error Variance in the General Linear Models Song-gui Wang 1,Mi-xiaWu,Wei-qingMa 3 1, Department of Applied Mathematics, Beijing Polytechnic University, Beijing 1000, China ( 1 wangsg88@yahoo.com.cn) 3 Department of Probability and Statistics, Peing University, Beijing , China Abstract Comparison is made between the MINQUE and simple estimate of the error variance in the normal linear model under the mean square errors criterion, where the model matrix need not have full ran and the dispersion matrix can be singular. Our results show that any one of both estimates cannot be always superior to the other. Some sufficient criteria for any one of them to be better than the other are established. Some interesting relations between these two estimates are also given. Keywords General linear model, MINQUE, mean square error 000 MR Subject Classification 6J05 1 Introduction We consider the general linear model y = Xβ + e, E(e) =0, Cov (e) =σ V, (1) where y is an n 1 observable random vector, an n p matrix X and n n nonnegative definite matrix V is nown, while β is a p 1 vector of unnown parameter, the positive scalar σ is also unnown. The error vector e has the normal distribution N(0,σ V ). The matrices X and V are both allowed to be of arbitrary ran. Throughout the paper, it is assumed that the model is consistent [5], i.e., y M(X..V ), where M(A) stands for the range of a matrix A and (A..B) denotes the partitioned matrix with A and B placed adjacent to each other. In the literature, there are two important estimates of σ. One of them is the MINQUE (Minimum Norm Quadratic Unbiased Estimate) σ m = y M(MVM) + My/, () suggested by Rao [6],whereM = I X(X X) + X, A + stands for the Moore-Penrose inverse of amatrixa, =ran(x.v ) ran (X). According to [6, Theorem 3.4], the MINQUE can be represented in several different forms. In fact, σ m is the estimate of σ based on the generalized least squares residuals, that is σ m =(y Xβ ) T (y Xβ )/, wheret = V + XX, A denotes a generalized inverse, and β =(X T X) X T y. Another estimate of σ is given by σ s = y My/, (3) Manuscript received September 18, 000. Revised April 11, 00. Partially supported by the National Natural Science Foundation of China (No ), the Natural Science Foundation of Beijing and a Project of Science and Technology of Beijing Education Committee.

2 14 S.G. Wang, M.X. Wu, W.Q. Ma which is obtained simply by replacing V by I in (), and is called simple estimate or the ordinary least squares estimate. Some authors studied statistical properties of σ s when V has some special structures, see, for example, [,4]. Groß [3] established some necessary and sufficient conditions for the equality σ m = σ s when X and V can be deficient in ran, without the normality assumption of error distribution. The object of the present note is to mae further comparison of these two estimates. Obviously in the general case σ s need not even be unbiased. Thus the mean square error (MSE) criterion is adopted, where the mean square error of an estimate θ of a scalar parameter θ is defined by MSE( θ) =E( θ θ). Some sufficient conditions are obtained for the inequality MSE ( σ m) MSE ( σ s). (4) The reverse of (4), however, also can hold in some cases. Some interesting relations between these two estimates are also obtained. To illustrate theoretical results, two examples are given. Comparison of the Estimates The following lemmas are necessary for the proof of our main theorem. Lemma 1. Let Σ be n n nonnegative definite matrix with ran r. A random vector X N p (µ, Σ) if and only if X = µ + AU, wherea is p r matrix with ran r and AA =Σ, U N r (0,I r ). A proof can be found in [5]. Lemma. Let X be an n p matrix and V n n nonnegative definite matrix. Then ran (VM)=ran (V..X) ran (X), wherem = I X(X X) + X. Proof. Denote by dim (S) the dimension of a linear space S. Wehave ran (VM)=dim(VM)=dim{VMt, for any t n 1 } =dim {Vu,X u =0} =ran(v..x) ran (X). The last equality follows from Theorem.1.4 of [11]. Lemma 3. σ m = σ s = u i /, (5) λ i u i /, (6) where u i N(0,σ ), i =1,, are independent and λ 1... λ > 0 are the positive eigenvalues of MV. Proof. Since MX =0,thus σ m and σ s can be rewritten as σ m = e M(MVM) + Me/, σ s = e Me/. InviewofLemma1ande N(0,σ V ), r=ran(v ), we note that there is an n r matrix A such that e = Aε, ε N(0,σ I r ),V= AA,thus σ m = ε Q 1 ε/, (7) σ s = ε Q ε/, (8) where Q 1 = A M(MAA M) + MA, Q = A MA. It is easy to verify that Q 1 Q = Q Q 1,which implies (see for example [5]) that there is an r r orthogonal matrix T such that both T Q 1 T

3 Comparison of MINQUE and Simple Estimate of the Error Variance in the General Linear Models 15 and T Q T are diagonal. By using Lemma, it can be shown that ran (Q 1 )=ran(a M)=ran(A MA)=ran(VM)=ran(V.X) ran (X) =. (9) We note that Q 1 is a projection matrix, thus where Λ =(λ 1,,λ ). Denote u = T ε,then T Q 1 T =diag(i, 0), (10) T Q T =diag(λ, 0), (11) u N r (0,σ I r ). (1) Substituting (10), (11) and (1) in (7) and (8) yields (5) and (6). The proof of Lemma 3 is completed. Denote r 0 =ran(x), which implies ran (M) =n r 0.Thus min { n r 0, ran (V ) }. In particular, when V > 0, that is, V is a positive definite matrix, we have = n r 0, which follows from Lemma. By using Poincare theorem (see, for example, [11]), we obtain α r0+i λ i α i i =1,...,. (13) where α 1 α α n are the eigenvalue of V. From Lemma 3, it is easy to show the following fact. Theorem 1. α 1 σ m λ 1 σ m σ s λ σ m α r0+ σ m. (14) From (14) we have α r0+ σ s/ σ m α 1. The results above show that if the eigenvalues α 1 and α r0+ are very close, then so are the estimates σ s and σ m. Denote tr (MV) [tr (MV)] f(mv,)= tr (MV)+ +, (15) where is defined in Lemma 3 as the number of the nonzero eigenvalues of MV,tr(A) denotes the trace of matrix A. Theorem. (a) If f(mv,) > 1, thenmse( σ m) < MSE ( σ s); (b) If f(mv,)=1,thenmse( σ m)=mse ( σ s); (c) If f(mv,) < 1, thenmse( σ m) > MSE ( σ s). Proof. It follows form (5) that ( ) MSE ( σ m)=var( σ m)=var u i / = σ4. On the other hand, from (6) we have thus ( ) ( ) Var ( σ s)=var λ i u i / = λ i σ 4 /, E( σ s)= σ λ i, MSE ( σ s)=e( σ s σ ) = E( σ s) σ E( σ s)+σ 4 =Var( σ s)+(e σ s) σ E( σ s)+σ 4 = σ4 ( λ i ) + σ4 ( λi ) σ4 λi + σ 4 [ = σ4 λ i + ( λ i ) λ i + ].

4 16 S.G. Wang, M.X. Wu, W.Q. Ma Note that tr (MV)= λ i and tr (MV) = λ i, the proof of Theorem is completed. Theorem involves the design matrix X whichisexpressedintermsofm, and this is not convenient for applications. However, it follows from (13) that ( ) ( ) ( ), α r0+i tr(mv) α i, tr(mv) α i Thus Denote αr 0+i ( tr(mv) ) 1 αr 0+i f(mv,) 1 l = 1 αi. α r0+i α i + 1 ( αi αr 0+i α r0+i α r0+i + 1 α i + 1 ( ) + ( α r0+i α i ) +. ) +, u = 1 αi α r0+i + 1 ( ) α i +, according to Theorem, we easily obtain the following corollary. Corollary 1. (a) If l>1, thenmse( σ m) < MSE ( σ s); (b) If 0 <u<1, thenmse( σ m) > MSE ( σ s). It is clear that l and u depend only on V and ran (MV), therefore Corollary 1 is more convenient than Theorem in applications. For example we consider the model (1) with ran (X) =1andV =diag(λ, λ, λ, αλ), where λ>0andα>0. It is easy to see =3. When we tae λ =andα =, then l =3.5 > 1, according to (a), σ m is the better estimate of σ. When we tae λ = 1 and α =1.1, then u 0.67 < 1, according to (b), we now that σ s is better. However, Corollary 1 does not always wor. For example, when we tae λ = 4 5 and α =, then l = 0.1 < 1, and u.1 > 1, we cannot mae any decision by Corollary 1, so we must return to Theorem again. We note that in many situations, such as sample surveys, animal genetic selection, economic panel data and longitudinal data, X and V may satisfy the condition MVM = tp MV 1/ for some t>0, where P A = A(A A) A. The condition implies that the nonzero eigenvalues of MVM: λ i = t, i =1,,. By using the special information about X and V,weobtain another result. Theorem 3. Suppose that MVM = tp MV 1/ for some t>0 and, (a) when + <t<1, MSE ( σ m) > MSE ( σ s); (b) when t = + or 1, MSE( σ m)=mse ( σ s); (c) if (a) and (b) are not cases, MSE ( σ m) < MSE ( σ s). Proof. Note that MV and MVM have the same nonzero eigenvalues. If MVM = tp MV 1/, then the nonzero eigenvalues of MV are λ i = t, i =1,,. hence f(mv,)=t + t t +.

5 Comparison of MINQUE and Simple Estimate of the Error Variance in the General Linear Models 17 The conclusions follow from straightforward discussion. Theorem 4. If MVM = tp MV 1/ or V > 0 and MVM = tm, then σ s = t σ m with unit Probability. Proof. It is easy to see that the hypothesis MVM = tp MV 1/ implies VMVMV = tv MV. Let V 0 = V/t,thenwehaveV 0 MV 0 MV 0 = V 0 MV 0, in view of [3, Proposition 1], Theorem 4 is proved. Remar 1. Under the condition of Theorem 3, obviously when t =1, we have σ s = σ m; when t =( )/( +), σ s < σ m, but their MSE s are equal. Further, when 0 <t<1, σ s is a shining estimate of σ m, but when t>1, we have σ s > σ m and MSE ( σ s) > MSE ( σ m), so if t>1, we should choose σ m as the estimate of σ. 3 Examples The estimate of σ are often used in the estimation of variances of estimable functions. In what follows we will give two simple examples to illustrate applications of the results obtained in this paper. Example 1. Consider the following linear model y = µ1 n + e, E(e) =0, Cov(e) =σ V. (16) This model has been found useful in certain statistical inference problems on the mean µ of a population when the observations y 1,,y n are not independent. For some examples of applications in medical data and animal genetic selection, the reader is referred to [7 9]. For the model (16), if the matrix V has following form 1 ρ ρ ρ 1 ρ......, (17) ρ ρ 1 where ρ is nown and satisfies 0 <ρ<1, then MVM =(1 ρ)m, and = n 1, which is clear by noting the fact V =(1 ρ)i + ρ11. According to Theorems 3 and 4, we have the following statements (a) if 0 <ρ< 4 n+1, then MSE ( σ s) < MSE ( σ m); 4 (b) if n+1 <ρ<1, then MSE ( σ s) > MSE ( σ m); (c) if ρ = 4 n+1, then MSE ( σ s)=mse( σ m), thus σ m and σ s cannot be distinguished by the mean square error criterion; (d) σ s =(1 ρ) σ m < σ m. In practice, ρ is usually unnown, we can use any estimate ρ as its true value. we can easily choose better estimate from σ m and σ s according to the above statement based on ρ and the sample size n. Although the least squares estimate (LSE) µ = y coincides with the best linear unbiased estimate (BLUE) of µ under model (16) (see [11]), however, its variance depends on V. For general matrix V, Tong [10] established the following lower and upper bounds on the variance of the generalized least squares estimate µ =(1 V 1 1) 1 1 V 1 y for all V with eigenvalues α 1 α n > 0, α n σ Var ( µ) α 1σ n n. (18) To obtain better estimated bounds of Var ( µ) in (18), we can replace by σ s or σ m by using Corallary 1.

6 18 S.G. Wang, M.X. Wu, W.Q. Ma Example. Consider the following linear model for longitudinal data y ij = x ijβ + α i + e ij, i =1,,m, j =1,,n, (19) where y ij denotes the ith observation of the response variable on the jth individual, x ij is a p 1 vector of nown explanatory variables. β is a p 1 vector of fixed effects, the α i are random individual effects, and the e ij are random errors. Assume that the α i are mutually independent N(0,σ α), the e ij are mutually independent N(0,σ e)andα i and e ij are independent of one another (see, for example, [1]). After introducing the following matrix notations y =(y 1,,y m), y i =(y i1,,y in ), X =(X 1,,X m), X i =(x i1,,x in ), α =(α 1,...,α m ), e =(e 1,...,e m), e i =(e i1,,e in ), the model (19) can be rewritten as y = Xβ +(I m 1 n )α + e, where α N(0, σ αi m ),e N(0,σ ei mn ), and denotes the Kronecer product of matrices. Cov (y) =σ e[ Imn +(I m θ1 n 1 n) ], where θ = σ α/σ e > 0. Denoting V (θ) =I mn +(I m θ1 n 1 n), then V (θ) > 0 and the eigenvalues of V (θ) are1+nθ and 1 with multiplicity m and m(n 1) respectively. For a special case m =, n =5, therefore, eigenvalues are 1 + 5θ (with multiplicity ) and 1 (with multiplicity 8). Let ran (X) =. Then = mn ran (x) =8. Because of l =1 10θ <1, (a) of Corollary 1 fails to wor, but u =(5θ +15θ )/, it is easy to see if 1.45 <θ<1.6, then 0 <u<1. According to Corollary 1, we now that σ s as the estimate of σ e is better. However, let ran (X) =1,then =9, and l =1+(65/9)θ + (100/9)θ > 1for any θ>0, which shows that σ m is better than σ s. References [1] Diggle, P.J., Liang, K., Zeger, S.L. Analysis of longitudinal dada. Oxford, New Yor, 1994 [] Dufour, J. Bias of S in linear regressions with dependent errors. The Amer. Stati., 40: (1996) [3] Groß, J. A note on equality of MINQUE and simple estimator in the general Gauss-Marov model. Statistics Probability Letters. 35: (1997) [4] Neudecer, H. Bounds for the bias of the least squares estimator of σ inthecaseofafirst-orderautoregressive process. Econometrica, 45: (1977) [5] Rao, C.R. Linear statistical inference and its applications. Wiley, New Yor, 1973 [6] Rao, C.R. Projectors, generalized inverses and the BLUE s. J. Roy. Statist. Soc. (Series B), 36: (1974) [7] Rawlings, J.O. Order statistics for a special case of unequally correlated multinormal variables. Biometrics, 3: (1976) [8] Shaed, M., Tong, Y.L. Comparison of experiments via dependence of normal variables with a common marginal distribution. Ann. Statist., 0: (199) [9] Shouri, M.M., Lathrop, G.M. Statistical testing of genetic linage under heterogeneity. Biometrics., 49: (1993) [10] Tong, Y.L. The role of the covariance matrix in the least squares estimation for a common mean. Linear Algebra and Its Applications. 64: (1997) [11] Wang, S.G., Chow, S.C. Advanced linear models. Marcel Deer Inc., New Yor, 1994