Maximum Likelihood Estimation of Isotonic Normal Means with Unknown Variances*

Transcription

1 Journal of Multvarate Analyss 64, (1998) Artcle No. MV Maxmum Lelhood Estmaton of Isotonc Normal Means wth Unnown Varances* Nng-Zhong Sh and Hua Jang Northeast Normal Unversty, Changchun,Chna Receved November 5, 1996; revsed June 3, 1997 To analyze the sotonc regresson problem for normal means, t s usual to assume that all varances are nown or unnown but equal. Ths paper then studes ths problem n the case that there are no condtons mposed on the varances. Suppose that we have data drawn from ndependent normal populatons wth unnown means + 's and unnown varances _ 's, n whch the means are restrcted by a gven partal orderng. Ths paper dscusses some propertes of the maxmum lelhood estmates of + 's and _ 's under the restrcton and proposes an algorthm for obtanng the estmates Academc Press AMS subject classfcatons 6H1, 6F10. Key words and phrases sotonc regresson, partal order, restrcted maxmum lelhood estmaton. 1. INTRODUCTION For ndependent normal populatons wth unnown means + and unnown varances _, =1,...,, ths paper studes the maxmum lelhood estmaton(mle) of +=(+ 1,..., + ) and _ =(_,..., 1 _) subject to the condton that the means + 's are restrcted by a gven partal orderng. Many nterestng partal orderngs may be consdered () a smple order on the means + 1 }}}+ ; () n the study of doseresponse relatonshps the means exhbt an unmodal trend + 1 }}}+ h }}}+, whch s called the umbrella order and ncludes the smple order wth h=; () a smple tree order on the means s of the form for =,...,, whch mples that some experments are desgned such that several treatments are sgnfcantly more effectve than a control. * Project supported by the Natonal Natural Scence Foundaton of Chna X Copyrght 1998 by Academc Press All rghts of reproducton n any form reserved.

2 184 SHI AND JIANG Let x j, j=1,..., n, be observatons from the th normal populaton. The log-lelhood functon s gven by l(+, _ )= =1{ &n ln _ & 1 _ n j=1 (x j &+ ) +c, (1.1) = where c s a constant whch does not depend on the parameters. By P denote a partal order defned on a fnte set 3=(% 1,..., % ). A -dmensonal vector + s sad to be an sotonc functon f % t, % s # 3, % t P% s mples + t + s.byd denote the set of all sotonc functons. The MLE of (+, _ ) s the maxmum soluton of (1.1) for + # D and _ # R +.If all varances are nown or unnown but equal, the MLE of + subject to the order restrcton s the maxmum soluton of (1.1) for + # D and equvalently s the soluton of mn =1 (x &+ ) w (1.) for + # D, where x = j x j n and w =n _ when varances are nown; w =n when all varances are unnown but equal. The soluton now s called the sotonc regresson of (x, w) wth x =(x 1,..., x ) and w= (w 1,..., w ). There are a number of elegant algorthms for obtanng the sotonc regresson, see for example, Barlow et al. (197) and Robertson et al. (1988). In the study of the sotonc regresson problem, the assumpton about varances must be needed. In practce, sometmes we cannot obtan much nformaton about the varances and ths paper deals wth the problem n the case that there are no condtons mposed on the varances. Sh (1994) consdered a smlar problem of estmatng the MLE of (+, _ ), n whch varances are also assumed to be restrcted by a gven partal order. The paper proposed an algorthm to compute the MLE and showed the convergence of the algorthm under the followng Condton A. For =1,...,, _ >(b&a), where _ denotes the sample varance of the th normal populatons, and a and b denote the mnmal and maxmal sample means respectvely. Secton 3 of ths paper proposes an algorthm of obtanng the MLE of (+, _ ) for our problem and shows that the convergence of the algorthm does not need any mposed condtons. However, we do not now f the algorthm converges to the true MLE and hence a condton as the Condton A s also necessary to show that the algorthm converges to the MLE, whch s dscussed n Secton 5. A numercal example usng the algorthm s gven n Secton 4. It s nown that the MLE s not unque for

3 ISOTONIC NORMAL MEANS 185 our problem and t s nterestng to study some propertes of the MLE, whch s gven n the next secton.. EXISTENCE OF THE MLE In ths secton, for convenence, we assume that the normal means are restrcted by the smple order, that s, + 1 }}}+. Note that smlar results gven n ths secton may be obtaned for any partal order restrctons. Let (+^, _^ ) be the MLE of (+, _ ) subject to the order restrcton. Then +^ # D and _^ # R +, whch satsfy l(+^, _^ )=sup[l(+, _ ); + # D, _ # R + ], (.1) where l(+, _ ) s gven n (1.1). For any fxed _ # R +, by the dscusson n Secton 1, the soluton of sup[l(+, _ ); + # D] must be the sotonc regresson of (x, w). From Theorem 1.6 of Barlow et al. (197) we have a+^ b (.) for =1,...,, where a and b are defned n Condton A. On the other hand, t s easy to chec that l(+, _ )l(+, _ (+)) for any fxed + # D, where _ (+)= j (x j &+ ) n and _ (+)=(_ 1(+),..., _ (+)). Then l(+^, _^ )=l(+^, _^ (+^ ))=sup[l(+, _ (+)); + # D 0 ], (.3) where D 0 =[+ # D; a+ b, =1,..., ]. Let L(+)= =1 &n ln[_ +(x &+ ) ], (.4) where the _ 's are defned n Condton A. Thus +^ s the soluton of (.3) f and only f +^ # D 0 and satsfes L(+^ )=sup[l(+); + # D 0 ]. (.5) Snce L(+) s a contnuous functon of + and D 0 s a compact set, the soluton of (.5) and then of (.1) exsts. It means that the MLE of (+, _ ) under the order restrcton exsts.

4 186 SHI AND JIANG As D s a polyhedral convex cone, for any gven + # D, there unquely exsts a subscrpt set [ 1,..., t ] wth 1 1 <}}}< t < such that + may be wrtten as + 1 =}}}=+ 1 <+ 1 +1=}}}=+ <}}}<+ t +1=}}}=+. (.6) Defnton. A vector + # D s sad to be a favorable pont f there s a subscrpt set [ 1,..., t ] such that + satsfes (.6) and s+1 = s +1 (x &+ ) w (+)=0 (.7) for s=0, 1,..., t, where w (+)=n _ (+), 0=0, and t+1 =. Theorem.1. If +^ s the soluton of (.5), there s a subscrpt set [ 1,..., t ] such that +^ satsfes (.6) and (.7), namely t s a favorable pont. The above theorem shows that the MLE of + must be a favorable pont and ts proof s gven n the Appendx. Because L( +) s not a concave functon, n general, the soluton of (.5) wll not be unque. However, we have the followng result. Theorem.. There are fntely many favorable ponts. Proof. For a fxed subscrpt set, the components + of a vector + wll be a constant for # [ s +1,..., s+1 ], s=0, 1,..., t, f the vector satsfes (.6). If % s+1 denotes the constant, then (.7) may be wrtten as s+1 = s +1 n (x &% s+1 ) g (% s+1 )=0, (.8) where g (% s+1 )=1 j (x j &% s+1 ) and s=0, 1,..., t. Snce the left-hand sde of (.8) s a polynomal of % s+1, the number of the soluton of (.8) s fnte. As there are fntely many subscrpt sets, the theorem follows. _ Condton B. For =1,...,, _ >max[(x &a),(x &b) ],, a, and b are as defned n Condton A. where Theorem.3. Proof. If Condton B holds, the favorable pont unquely exsts. Condton B mples L(+) =&_ + +(x &+ ) <0

5 ISOTONIC NORMAL MEANS 187 for all + satsfyng a+ b and =1,...,. Then L(+) s a concave functon on [a, b]. By (.), the soluton of (.5) unquely exsts and, by Theorem.1, the favorable pont unquely exsts. Let (+^, _^ ) be a MLE subject to the gven order restrcton. By Theorem.1, +^ must be a favorable pont and, from (.3), _^ =_ (+^ ) for =1,...,. Then Theorem. mples that the number of the MLE s fnte. If Condton B holds, by Theorem.3 the MLE s unque. 3. THE PROPOSED ALGORITHM For a gven subscrpt set, one can obtan all favorable ponts accordng to the subscrpt set usng (.6) and (.8), n whch all real roots of some polynomals need to be found. Then the MLE of (+, _ ) may be obtaned by comparng all favorable ponts for all subscrpt sets. However, ths procedure s very hard to carry out. Ths secton proposes an teraton algorthm. As n the dscusson n Secton 1, f the varance vector _ s gven the MLE of + may be obtaned as n the sotonc regresson problem, and f the mean vector + s gven the MLE of _ s just _ (+) as defned n (.3). The followng algorthm s based on ths consderaton. Algorthm. Step(0, 0). Let + (0) =x and _ (0) =_. Step(n, 1). Fnd + (n), the sotonc regresson of (x, w (n&1) ) for w (n&1) n _ (n&1). Step(n, ). Let _ (n) =_ (+(n) ) and _ (n) =(_ (n) 1,..., _(n) ). The above algorthm shows that = l(+ (n), _ (n&1) )l(+ (n), _ (n) )l(+ (n+1), _ (n) ) (3.1) and L(+ (n) )L(+ (n+1) ) for n1. By the monotoncty, L(+ ( p) )=L(+ ( p+q) ) for all ntegers q1 f L(+ ( p) )=L(+ ( p+1) ). Thus we can gve a termnaton crteron for the algorthm. For example, we stop the teraton at Step(n,) f max 1 + (n&1) &+ (n) 10 &m for some ntegers m1. The proof of the followng theorem s gven n the Appendx.

6 188 SHI AND JIANG Theorem 3.1. The pont sequence [+ (n) ] gven n the above algorthm converges to a favorable pont as n. Corollary 3.1 If Condton B holds and the lmtng pont of [+ (n) ]s +*, then the MLE of (+, _ ) s (+*, _ (+*)). The above corollary may be shown by usng Theorem.3. It must be noted that n the above corollary Condton B s not always needed. For example, f the sample means satsfy x 1 }}}x, the MLE wll be (+ (1), _ (1) ), the one step result of the algorthm. However, n general, the condton cannot be omtted. A detaled dscusson s gven n Secton NUMERICAL EXAMPLE For llustraton, the proposed algorthm s used to treat the data shown n Sh (1994). There are fve dstrcts n Jln Provnce of Chna Laoyuan (Group 1), Qanfu (Group ), Changchu (Group 3), Tonghua (Group 4), and Jln (Group 5). The data gave the scores of 100 students per dstrct obtaned n the Natonal Matrculaton Examnaton held n 199. Past experence showed that the condtons of educaton of the dstrct +1 was lely better than dstrct for =1,..., 4. We proceed to estmate the examnaton scores of students of the fve dstrcts. By X we denote the examnaton score of dstrct, then X follows a normal dstrbuton wth unnown mean + and unnown varance _ for =1,..., 5. Pror nformaton tells us that the means exhbt an ncreasng trend + 1 }}}+ 5. We use the proposed algorthm to estmate the MLE of + 's and _ 's subject to the smple order restrcton. It s easy to chec that Condton B s satsfed, and by Corollary 3.1 we can obtan the MLE by the proposed algorthm. The computed results of estmatng the means and the varances are lsted n Table I, n whch Cran's (1980) program was used as a subroutne to compute sotonc TABLE I Computng Results of Examnaton Scores =1 = =3 =4 =5 +(0) _(0) (1) _(1) () _()

7 ISOTONIC NORMAL MEANS 189 regresson and the teraton s termnated when max + (n&1) &+ (n) 10 &3. The computed results show that the teraton s termnated at n=. In the table, for the termnated crteron 10 &3, + (1) =+ () means that the MLE of + s the same as the sotonc regresson of (x, w ), wth w =n _ of _ s dfferent from the sample varance, that s, _ (0) {_ ().. The MLE 5. DISCUSSION Recall that our problem s to fnd the soluton (+^, _^ ) whch maxmzes l(+, _ ) subject to + # D and _ # R +, (5.1) where l(+, _ ) s gven n (1.1). For estmatng +, one may replace _ by _ n (1.) to obtan the sotonc regresson +$, say, of (x, w ) wth w =n _ for =1,...,. The estmate of _ wll be _ or, by the lelhood prncple, _ (+$) as defned n (.3). However, the estmates (+$, _ ) and (+$, _ (+$)) are the results of Step (1,1) and Step (1,) n the proposed algorthm respectvely. Furthermore, the expresson of L( +) n (.4) may be wrtten as L(+)= =1 &+ ) &n ln _1+(x _ & +c, where c s a constant whch does not depend on the parameters. If Condton B holds, by Taylor expanson, L(+)= =1 j=1 (&1) j [(x &+ ) w ] j. j Then the estmate +$, the sotonc regresson of (x, w ), s the frst approxmaton, j=1, for the soluton of (5.1). On the other hand, by (.3), a reasonable estmate of (+, _ ) s of the form (&, _ (&)) and satsfes =1 E(x &+ ) w (x )> =1 E(& &+ ) w (&) (5.) for any + # D, where w (})=n _ ( } ) and =1,..., ; see Brun (1965), Lee (1981, 1988) and Hwang and Peddada (1994). The Eq. (5.) mples that the mean square error of the estmate (&, _ (&)) s strctly less than that of the usual estmate. Let +* be the lmtng pont of the proposed algorthm. For every step n>1 of the algorthm, by (3.1) we have

8 190 SHI AND JIANG =1 E(x &+ ) w (x )> =1 =1 E(+ (1) &+ ) w (x ) E(+ (n) &+ ) w (+ (n&1) ), and then the estmate (+*, _ (+*)) satsfes (5.) even f t s not the MLE. Note that + (1) =+$ n the second port of the above expresson. It wll be very mportant to now the rato at whch Condton B s satsfed for some regular cases and to now how many favorable ponts, from the proposed algorthm, are the MLE even f Condton B s not satsfed. Therefore, some smulaton studes are needed. Some smulaton results are lsted n Table II, n whch =5 and 7 respectvely. The normal means are consdered n two cases (1) equal means + 1 =}}}=+ ; () equal spacng means + +1 &+ =, where TABLE II The Smulaton Results for Condton B Mean Varance NUNB NUNC n=10 n=15 n=0 n=10 n=15 n=0 = &. & = Note. The smulatons were run tmes. In the tables NUNB denotes the number of tmes that Condton B s not satsfed and NUNC denotes the number of tmes that the algorthm does not converge to true MLE.

9 ISOTONIC NORMAL MEANS 191 =1,..., &1 and =0.1. The varances are consdered n four cases, equal, ncreasng, unmodal, and decreasng, as lsted n the table. We tae the smulatons for equal sample szes n =10, n =15, and n =0 respectvely. The smulatons are run tmes for each case. In the table, NUNB denotes the number of tmes that Condton B s not satsfed and NUNC denotes the number of tmes that the favorable pont obtaned by the proposed algorthm s not the MLE. The NUNB depends on the relatonshp of szes of means and varances, and also depends on the sample sze. NUNC=0, almost everywhere, tells us that the favorable pont obtaned by the proposed algorthm s the MLE even f Condton B s not satsfed for the consdered regular cases. However, we can fnd some abnormal cases such that the NUNC does not equal zero, as lsted n Table III. In the table, for =5, the means are 1., 0.9, 0.6, 0.3, and 0.0, and t s anttonc wth respect to the smple order; the varances are 1.0, 0.8, 0.6, 0.4, and 0. respectvely. The smulatons are run tmes for equal sample sze n =0. It may be seen that Condton B s not satsfed for any of the cases. The favorable pont from the proposed algorthm s not the MLE for 3 cases. For these cases, let +^ denote the true MLE, +* denote the favorable pont obtaned from the proposed algorthm, and +$ denote the sotonc regresson of TABLE III The Smulaton Results for Abnormal Cases =5, n=0, NUNB=10000, NUNC=3 L(u^ ) L(u*) L(u$) L(u^ )L(u*) L(u^ )L(u$) L(u^ ) L(u*) L(u$) L(u^ )L(u*) L(u^ )L(u$) Note. The smulatons were run tmes. The values of the log-lelhood functon are lsted, n whch u^ denotes the true MLE, u* denotes the estmate from the algorthm, and u$ denotes the sotonc regresson usng the sample varances.

10 19 SHI AND JIANG (x, w ) wth the estmate of varances _ (+$), the result of Step (1,) n the algorthm. The values of L(+^ ), L(+*), and L(+$) are lsted n Table III. The dfferences L( +^ )&L( +*) and L( +^ )&L( +$) are also lsted n Table III. The dfferences, L( +^ )&L( +*), are very small, correspondng to the values of L(+^ ) and L(+*) for all cases. The value of L(+*) s greater than that of L(+$). Consequently, we recommend the teraton algorthm for practcal use because ts computaton s smple and t has good propertes of convergence. APPENDIX The Proof of Theorem.1. We need to fnd a vector +^ whch belongs to D and maxmzes L(+), gven n (.4), subject to + # D. The Lagrangan functon now s gven by &1 8(+, *)= 1 L(+)+ =1 * (+ +1 &+ ), where *=(* 1,..., * &1 ) and * 's are the Lagrangan multplers. The KuhnTucer condtons are usually used to deal wth such problems. If +^ maxmzes L( +) subject to + # D, then +^ satsfes the followng condtons 1. +^ 1 }}}+^ ;. 8+ +^ =0 for =1,..., ; 3. * 0 for =1,..., &1; 4. * (+^ +1&+^ )=0 for =1,..., &1. The second of these condtons corresponds to the equaton (x &+^ ) w (+^ )+* &1 &* =0 (A1) for =1,...,, where w (+^ ) s gven n (.7) and * 0 =* =0. Let [ 1,..., t ] be the subscrpt set such that * j =0, j=1,..., t; * >0, otherwse. Then the frst, thrd and fourth condtons gven n the above mply that +^ satsfes (.6) and, from (A1), we have (.7). The KuhnTucer condtons are necessarly satsfed for our problem. Furthermore, f L( +) s a concave functon, by the dscusson n Mangasaran (1969, p. 94) these are also suffcent condtons and there exsts unquely a favorable pont; see also Theorem.3 n ths paper. To prove Theorem 3.1, we need the followng lemmas.

11 ISOTONIC NORMAL MEANS 193 Lemma A.1. Let [ y n ] be a unformly bounded sequence n R. If y n & y n&1 0, as n, and the sequence s not convergent, then there are nfntely many accumulaton pont of the sequence, where 0 denotes the -dmensonal zero vector. Proof. At frst, we consder the case of =1. Let =lm nf [y n ] and ;=lm sup [y n ]. By the condton, <;. For any z #(, ;), we wll show that z s an accumulaton pont of [y n ], namely, there s a subsequence of [y n ] whch converges to z. For any = 1 >0, by the assumpton there s a postve nteger N 1 such that y n+1 & y n <= 1 f n>n 1. On the other hand, <z<; mples that there s a postve nteger n 1 >N 1 such that y n1 <z and y n1 +1>z. Then we have y n1 &z <y n1 +1& y n1 <= 1. Smlarly, for any = >0 wth = <= 1, there s a term y n wth n >n 1 such that y n &z <=. If one contnues ths procedure, a subsequence [y nj ] may be obtaned and t converges to z. For >1, denote y n =(y 1n,..., y n ). Let =lm nf[y n ] and ; =lm sup [y n ], =1,...,. Wthout loss of generalty, assume that 1 <; 1. For any z 1 #( 1, ; 1 ), from the above dscusson there s a subsequence [y 1nj ] of [y 1n ] wth y 1nj z 1,asn j. As the sequence s unformly bounded, for =, there s a subsequence of [y nj ] whch converges to z for some z n the nterval [, ; ]. So we can obtan a subsequence of [y n ] and a pont z, nr, wth the frst component z 1 such that the subsequence converges to z. Because there are nfntely many ponts n ( 1, ; 1 ), the proof of ths theorem s completed. Lemma A.. Let [+ (n) ] be the sequence from the proposed algorthm and let [+ (n j ) ] be a subsequence. If the subsequence s convergent, then + (n j ) & + (n j &1) 0, as n j. Proof. As _ (n) s a contnuous functon of + (n), _ (n j ) s also convergent. Recallng the expresson (1.1), l(+ (n j ), _ (n j ) ) s convergent. From n j&1 n j &1 and (3.1), we have 0l(+ (n j &1), _ (n j &1) )&l(+ (n j ), _ (n j &1) ) l(+ (n j&1 ), _ (n j&1 ) )&l(+ (n j ), _ (n j ) ) 0 (A)

12 194 SHI AND JIANG as n j. For smplfyng the notaton, denote n j by m. Snce the dfference of l(+ (m&1), _ (m&1) ) and l(+ (m), _ (m&1) ) may be wrtten as =1 [(x &+ (m) ) &(x &+ (m&1) ) ] w (m&1), where w (m&1) s defned n the Algorthm, (A) mples that the dfference converges to 0 when m tends to nfnte. Because + (m) s the sotonc regresson of (x, w (m&1) ), followng the dscusson n Sh (1994, p. 91), we have + (m) &+ (m&1) 0,asm. Lemma A.3. Let [+ (n) ] be the sequence from the proposed algorthm and [+ (m) ] be a subsequence. If the subsequence s convergent, then t converges to a favorable pont. Proof. Assume that + (m) &, asm. Snce & belongs to D, there s a subscrpt set [ 1,..., t ] such that & satsfes (.6). Now we show that & satsfes (.7). Let and (m) s+1 =mn[+(m) ; s +1 s+1 ] ; (m) s+1 =max[+(m) ; s +1 s+1 ], where s=0, 1,..., t, 0 =0, and t+1 =. Because + (m) converges to &, for any $>0, there s an nteger M$ such that max[; (m) s+1 &(m) s+1 ; s=0, 1,..., t]<$, (A3) for m>m$. From the proposed algorthm, + (m) s the sotonc regresson of (x, w(+ (m&1) )). Therefore for any =>0, by the lemma n Barlow et al. (197, p. 34) and (A3), there s an nteger M wth M>M$ such that s+1 (x } &+ (m) ) w (+ (m&1) ) (A4) }<= = s +1 for m>m and s=0, 1,..., t. Asw(+) s a contnuous functon of +, we have s+1 = s +1 (x && ) w (&)= lm s+1 m = s +1 = lm m =0 s+1 = s +1 (x &+ (m) ) w (+ (m) ) (x &+ (m) ) w (+ (m&1) )

13 ISOTONIC NORMAL MEANS 195 for s=0, 1,..., t. The second equaton of the above follows Lemma A and the last follows (A4). Then & satsfes (.7) and s a favorable pont. The Proof of Theorem 3.1. Let l n&1 =l(+ (n), _ (n&1) ) and l n = l(+ (n), _ (n) ), where n>1 and + (n) and _ (n) are as gven n the algorthm. From l(+ (n, ), _ (n) )l(x, _ ) and (3.1), the real number sequence [l m ; m=1,,...] s monotone ncreasng and bounded. Then the sequence s convergent and l n &l n&1 0, as n. By usng a method smlar to that shown n the proof of Lemma A, we can prove that + (n) &+ (n&1) 0, as n. Snce [+ (n) ] s unformly bounded, f t s not convergent, by Lemma Al, there are nfntely many accumulaton ponts and, by Lemma A3, they are all favorable ponts. Ths contradcts Theorem.. Therefore [+ (n) ] s convergent and, by Lemma A3 agan, t converges to a favorable pont. ACKNOWLEDGMENTS We than the referee for helpful comments. We rewrote Secton 5 tang hs comments nto account. We are also grateful to the edtors for ther help and support. REFERENCES 1. Barlow, R. E., Bartholomew, D. J., Bremner, J. M., and Brun, H. D. (197). Statstcal Inference under Order Restrctons. New Yor Wley.. Brun, H. D. (1965). Condtonal expectaton gven a _-lattce and applcatons. Ann. Math. Statst Cran, G. M. (1980). Amalgamaton of means n case of smple orderng. Appl. Statst Hwang, J. T. G., and Peddada, S. D. (1994). Confdence nterval estmaton subject to order restrctons. Ann. Statst Lee, C. I. C. (1981). The quadratc loss of sotonc regresson under normalty. Ann. Statst Lee, C. I.C. (1988). The quadratc loss of order restrcted estmatons for several treatment means and a control mean. Ann. Statst Mangasaran, O. L. (1969). Nonlnear Programmng. New Yor Wley. 8. Robertson, T., Wrght, F. T., and Dystra, R. L. (1988). Order Restrcted Statstcal Inference. New Yor Wley. 9. Sh, N.-Z. (1994). Maxmum lelhood estmaton of means and varances from normal populatons under smultaneous order restrctons. J. Multvarate Anal