A profile likelihood method for normal mixture with unequal variance

Transcription

1 This is the author s fina, peer-reviewed manuscript as accepted for pubication. The pubisher-formatted version may be avaiabe through the pubisher s web site or your institution s ibrary. A profie ikeihood method for norma mixture with unequa variance Weixin Yao How to cite this manuscript If you make reference to this version of the manuscript, use the foowing information: Yao, W. (2010). A profie ikeihood method for norma mixture with unequa variance. Retrieved from Pubished Version Information Citation: Yao, W. (2010). A profie ikeihood method for norma mixture with unequa variance. Journa of Statistica Panning and Inference, 140(7), Copyright: 2010 Esevier B.V. Digita Object Identifier (DOI): doi: /j.jspi Pubisher s Link: This item was retrieved from the K-State Research Exchange (K-REx), the institutiona repository of Kansas State University. K-REx is avaiabe at

2 A Profie Likeihood Method for Norma Mixture with Unequa Variance Weixin Yao Department of Statistics, Kansas State University, Manhattan, Kansas 66506, U.S.A. Abstract It is we known that the norma mixture with unequa variance has unbounded ikeihood and thus the corresponding goba maximum ikeihood estimator (MLE) is undefined. One of the commony used soutions is to put a constraint on the parameter space so that the ikeihood is bounded and then one can run the EM agorithm on this constrained parameter space to find the constrained goba MLE. However, choosing the constraint parameter is a difficut issue and in many cases different choices may give different constrained goba MLE. In this artice, we propose a profie og ikeihood method and a graphica way to find the maximum interior mode. Based on our proposed method, we can aso see how the constraint parameter, used in the constrained EM agorithm, affects the constrained goba MLE. Using two simuation exampes and a rea data appication, we demonstrate the success of our new method in soving the unboundness of the mixture ikeihood and ocating the maximum interior mode. Key words: EM agorithm, Maximum ikeihood, Mixture modes, Profie Preprint submitted to Esevier February 2, 2010

3 ikeihood, Unbounded ikeihood 1. Introduction Let x = (x 1,..., x n ) be independent observations from a m-component norma mixture density f(x; θ) = π 1 φ(x; µ 1, σ 2 1) + π 2 φ(x; µ 2, σ 2 2) + + π m φ(x; µ m, σ 2 m), where θ = (π 1,..., π m 1, µ 1,..., µ m, σ 1,..., σ m ), φ( ; µ, σ 2 ) is the norma density with mean µ and σ 2, and π j is the proportion of jth component with m j=1 π j = 1. The og-ikeihood for x is ogl(θ; x) = n og{π 1 φ(x i ; µ 1, σ1)+π 2 2 φ(x i ; µ 2, σ2)+ 2 +π m φ(x i ; µ m, σm)}. 2 i=1 For a genera introduction to mixture modes, see Lindsay (1995), Bohning (1999), McLachan and Pee (2000), and Frühwirth-Schnatter (2006). It is we known that ogl(θ; x) in is unbounded without any restriction on the component variance, and so the goba maximum ikeihood estimator (MLE) of θ, by maximizing, does not exist. For exampe, if we set µ 1 = x 1 and et σ1 2 0, the ikeihood vaue goes to infinity. However, for mixtures of norma distributions, at east in the univariate case, there is a sequence of roots corresponding to oca maxima in the interior of the parameter space that are consistent and asymptoticay norma and efficient (Kiefer, 1978 and Peters and Waker, 1978). Note that if there are mutipe 2

4 oca maxima in the interior of the parameter space, there is aso a probem of identifying the consistent sequence, which is a very difficut probem itsef. In this artice, we do not focus on this issue. Instead, when the ikeihood is unbounded, we define the MLE as the maximum interior/oca mode. Hathaway (1985) provided some theoretica support of using the maximum interior/oca mode. One of the commony used methods to avoid the unboundness of the og ikeihood and to find the maximum interior mode is to run the EM agorithm (Dempster et a., 1977) over a constrained parameter space Ω C = {θ Ω : σ h /σ j C > 0, 1 h j m}, (2) where C (0, 1], Ω denotes the unconstrained parameter space. See Hathaway (1985, 1986) and Bezdak, Hathaway, and Huggins (1985) for more detai. However, a big chaenge for this method is to choose the appropriate cut point C. If C is too arge, it is possibe that the consistent oca maxima does not beong to the constrained parameter space Ω C and thus the found estimate wi be miseading. Even the consistent oca maxima is in Ω C, it is sti possibe that Ω C misses some interior modes worthy of consideration. On the other hand, if C is too sma, it is possibe that some boundary point, satisfying σ h /σ j = C for some h and j, maximizes the og ikeihood over the constrained parameter space Ω C. In this situation, the found estimate is on the boundary of Ω C and thus depends on the choice of C. 3

5 Another commony used method is to use maximum penaized ikeihood estimator that adds penaty term to the unequa variance. See Chen, Tan, and Zhang (2008) and Chen and Tan (2009). In this artice, we propose a profie og-ikeihood method and a graphica way to sove the unboundness issue of ikeihood and find the maximum interior mode for the norma mixture with unequa variance. Unike the constrained EM agorithm (Hathaway, 1985, 1986), our proposed method does not need to specify a cut point C. In addition, based on our proposed method, we can ceary check whether there are some other minor interior modes and see how the choice of C in (2) affects the constrained goba MLE. Using the simuation study and a rea data appication, we demonstrate the effectiveness of our proposed method and show how the seection of cut point C affects the constrained MLE (Hathaway, 1985, 1986). The rest of the paper is organized as foows. Section 2 proposes a profie og ikeihood method to sove the unboundness issue of the ikeihood function for the norma mixture with unequa variance. In Section 3, we use two simuation exampes and a rea data appication to demonstrate how our proposed method works. We summarize our proposed method and give the discussion in Section New method In this section, we wi first introduce our profie og ikeihood method for two component norma mixtures and provide a simpe EM agorithm. We 4

6 wi then extend the profie og ikeihood method to norma mixtures of more than two components Mixtures of two components Given a sampe x = (x 1,..., x n ) from the two-component norma mixture, the og-ikeihood for x is ogl(θ; x) = n og{π 1 φ(x i ; µ 1, σ1) 2 + π 2 φ(x i ; µ 2, σ2)}, 2 (3) i=1 where θ = (π 1, µ 1, µ 2, σ 1, σ 2 ) and φ(x; µ, σ 2 ) = 1 2πσ exp{ 1 2σ 2 (x µ)2 }, Note that without any restriction, the above og-ikeihood is unbounded and the goba MLE is undefined. In this section, we propose a profie ikeihood method to avoid the unboundness issue and to find the maximum interior mode of ogl(θ; x). Let σ 1 = kσ 2 kσ, where k (0, 1]. Then the og-ikeihood of (3), for each fixed k, is ogl(η; x, k) = n og{π 1 φ(x i ; µ 1, k 2 σ 2 ) + π 2 φ(x i ; µ 2, σ 2 )}. (4) i=1 where η = (π 1, µ 1, µ 2, σ). Note that for each fixed k, the og-ikeihood of (4) is bounded. Hence the goba MLE for (4) is we defined. In order to 5

7 estimate k, we define the profie og-ikeihood for k as p(k) = max ogl(η; x, k), (5) η where ogl(η; x, k) is defined in (4). Let Ω C = {θ Ω : min(σ 1, σ 2 )/ max(σ 1, σ 2 ) C > 0}, (6) where Ω is the unconstrained parameter space for θ. Theorem 2.1. We have the foowing properties about the profie ikeihood p(k) defined in (5). (a) The profie ikeihood p(k) is unbounded and goes to infinity when k goes to zero. (b) The ˆθ = (ˆπ 1, ˆµ 1, ˆµ 2, ˆσ 1, ˆσ 2 ) maximizes the og ikeihood ogl(θ; x) of (3) constrained in Ω C, where ˆσ 1 ˆσ 2, if and ony if ˆk = ˆσ 1 /ˆσ 2 maximizes the profie og-ikeihood p(k) of (5) in K C, where K C = {k (0, 1] : k C}. (c) Suppose k is a oca mode for the profie og-ikeihood p(k) with the corresponding η = ( π 1, µ 1, µ 2, σ). Let θ = ( π 1, µ 1, µ 2, k σ, σ). Then θ is a oca mode for the og ikeihood ogl(θ; x) of (3). The proof of Theorem 2.1 is given in the Appendix. From (a), one can know that p(k) is aso unbounded. Therefore, we cannot estimate k by maximizing p(k) directy. Based on (b), one can know that finding the maximum interior mode of ogl(θ; x) of (3) is equivaent to finding the maximum interior mode of p(k). Noting that k is a one-dimensiona parameter, hence our profie ikeihood method transfers the probem of ocating the maximum 6

8 interior mode for a high-dimensiona function ogl(θ; x) into ocating the maximum interior mode for a one-dimensiona function p(k). For one dimension function p(k), one can easiy use the pot of p(k) versus k to ocate the maximum interior mode of p(k) without choosing a cut point C in advance, which is one of the major advantages of our proposed method and wi be iustrated in more detai in Section 3. Let ˆk be the maximum interior mode of (5). Then fixing k at ˆk, we can find the MLE of (4), denoted by ˆη(ˆk), and the corresponding ˆθ(ˆk). The ˆθ(ˆk) is our proposed maximum interior mode of (3). Based on the pot of p(k) versus k, one can aso ceary see how the cut point C in (6) affects the constrained MLE (Hathaway 1985, 1986). We wi demonstrate this using exampes in Section 3. Note that the profie og-ikeihood p(k) does not have an expicit form. Therefore, we can ony numericay evauate p(k) for a set of grid points of k. The foowing is the EM agorithm to find p(k) for any fixed k. Agorithm 1: Starting with the initia parameter vaues {ˆπ (0) 1, ˆµ (0) 1, ˆµ (0) 2, ˆσ (0) 1 = kˆσ (0) 2 }, iterate the foowing two steps unti convergence. E Step: Compute the cassification probabiities: ˆp (t+1) ij = ˆπ(t) j 2 =1 ˆπ (t) φ(x i ; ˆµ (t) j φ(x i ; ˆµ (t), ˆσ2(t) j ), ˆσ 2(t) ), i = 1,..., n, j = 1, 2 7

9 M step: Update the component parameters: ˆµ (t+1) j = ˆσ 2(t+1) 1 = n i=1 ˆp(t+1) ij x i n i=1 ˆp(t+1) ij n [ i=1 ˆσ (t+1) 2 = ˆσ (t+1) 1 /k., ˆπ (t+1) j = ˆp (t+1) i1 (x i ˆµ (t+1) 1 ) 2 + k n n i=1 ˆp(t+1) ij, j = 1, 2. n (t+1) 2ˆp i2 ] (x i ˆµ (t+1) 2 ) 2, Simiar to the genera EM-agorithm, this agorithm is ony guaranteed to converge to a oca mode. In order to find the maxima mode (goba MLE) for each fixed k, we may run the agorithm from severa initia vaues and choose the converged mode which has the argest og-ikeihood (note that the maxima mode is we defined since the og ikeihood (4) is bounded for each fixed k) Mixtures of more than two components When there are more than two components, i.e., m > 2, et k = σ /σ, where σ σ (2)... σ are ordered sequence of (σ 1,..., σ m ). Let Θ k = {θ = (π 1,..., π m 1, µ 1,..., µ m, σ 1,..., σ m ) σ = kσ }. Then one can define the profie og ikeihood as p(k) = max θ Θ k n og f(x i ; θ, k), k (0, 1]. (7) i=1 8

10 It can be easiy seen that the above defined profie og ikeihood p(k) aso has the properties given in Theorem 2.1. In addition, simiar to the way proposed in Section 2.1, one can aso use p(k) in (7) to find the maximum interior mode and check how the constraint parameter affects the constrained MLE for the constrained EM agorithm. Due to the compicated nature of the constrained optimization, finding p(k) is not trivia for each fixed k. In (t + 1)th step of EM agorithm, E step finds the cassification probabiities ˆp (t+1) ij = ˆπ(t) j m =1 ˆπ (t) φ(x i ; ˆµ (t) j, ˆσ2(t) j ) φ(x i ; ˆµ (t), ˆσ 2(t) ), i = 1,..., n, j = 1,..., m. In M step, the component means and the mixing proportions are updated by ˆµ (t+1) j = n i=1 ˆp(t+1) ij x i n i=1 ˆp(t+1) ij, ˆπ (t+1) j = Let n j = n i=1 ˆp(t+1) ij and Sj 2 = n n i=1 ˆp(t+1) ij, j = 1,..., m. n i=1 ˆp(t+1) ij (x i µ (t+1) j ) 2. For simpicity of notation, we omit the dependence of n j and S j on t+1. For a fixed k (0, 1], based on the EM agorithm theory, ˆσ (t+1) = (ˆσ (t+1) 1,..., ˆσ m (t+1) ) are updated by minimizing subject to σ = kσ. m j=1 ( ) n j og σ j + S2 j, (8) 2σj 2 9

11 Note that due to the abe switching issue of mixture modes (Yao and Lindsay, 2009), the component index does not have rea meaning. Without oss of generaity, we wi assume that the component index satisfies S 2 1/n 1 S 2 2/n 2... S 2 m/n m. (If the component index does not satisfy the above constraint, we can aways permute the component index such that the above constraint hods.) Note that when k = 1, the component variance are a equa and thus the computation of p is straightforward. In the foowing, we wi mainy consider the situation when 0 < k < 1. Proposition 2.1. Let ˆσ (t+1) = (ˆσ (t+1) 1,..., ˆσ m (t+1) ) be the maximizer of (8), subject to σ = kσ, where k (0, 1). Let (ˆσ (t+1),..., ˆσ (t+1) ) be the corresponding ordered sequence. Then, we have the foowing resuts about ˆσ (t+1). (a) If S1/n 2 1 k 2 Sm/n 2 m, there exists 1 i < j m such that ˆσ (t+1) ˆσ (t+1) 2 =... = ˆσ (t+1) i S i+1 / n i+1, ˆσ (t+1) j S j 1 / n j 1, and ˆσ (t+1) = S / n, = i + 1,..., j 1. 1 = = ˆσ (t+1) j+1 =... = ˆσ m (t+1) (b) If S1/n 2 1 > k 2 Sm/n 2 m, there exists 1 i < j m such that ˆσ (t+1) i = ˆσ (t+1) S 1 / n 1, ˆσ (t+1) j = ˆσ (t+1) S m / n m, and ˆσ (t+1) = S / n, i and j. The proof of Proposition 2.1 is given in the Appendix. From the Proposition 2.1, we can see that the constrained maximizer of (8) depends on whether S 2 1/n 1 < k 2 S 2 m/n m hods. When S 2 1/n 1 k 2 S 2 m/n m, ˆσ (t+1) = ˆσ (t+1) (), = 1,..., m. (Note that we have assumed S 2 1/n 1 S 2 2/n 2... S 2 m/n m.) However, when S 2 1/n 1 > k 2 S 2 m/n m, ˆσ (t+1) is not necessary equa to ˆσ (t+1) 1 and ˆσ (t+1) is not necessary equa to ˆσ (t+1) m. 10

12 Proposition 2.2. (a) For any 1 i < j m, under the constraint that σ 1 = σ 2 =... = σ i = σ and σ j = σ j+1 =... = σ m = σ/k, the objective function (8), as a function of σ by fixing {σ i+1,..., σ j 1 }, is minimized at i ˆσ (i,j) 2 =1 = S2 + k 2 m =j S2 i =1 n + m =j n. (9) In addition, (8) is monotone decreasing when σ < ˆσ (i,j) and monotone increasing when σ > ˆσ (i,j). (b) For any 1 i < j m, under the constraint that σ i = σ = kσ j, the objective function (8), as a function of σ by fixing {σ, i and j}, is minimized at ˇσ 2 (i,j) = S2 i + k 2 S 2 j n i + n j. (10) In addition, (8) is monotone decreasing when σ < ˇσ (i,j) and monotone increasing when σ > ˇσ (i,j). The proof of Proposition 2.2 is given in the Appendix. Based on the Proposition 2.1 and 2.2, we propose to use the foowing two steps to find ˆσ (t+1) that minimizes (8) subject to σ = kσ. Step 1: If S 2 1/n 1 k 2 S 2 m/n m, for a pairs 1 i < j m, et σ (i,j) be the minimizer of (8) under the constraint σ 1 = σ 2 =... = σ i, σ j = σ j+1 =... = σ m = σ 1 /k, σ 2 1 S 2 i+1/n i+1, and σ 2 m S 2 j 1/n j 1, when { σ 2 = S 2 /n, = i + 1,..., j 1} are fixed, where σ 1 2 = ˆσ 2 (i,j), k2 S 2 j 1/n j 1 ˆσ 2 (i,j) S2 i+1/n i+1 ; S 2 i+1/n i+1, ˆσ 2 (i,j) > S2 i+1/n i+1 ; k 2 S 2 j 1/n j 1, ˆσ 2 (i,j) < k2 S 2 j 1/n j 1 ; where ˆσ (i,j) is defined in (9). 11

13 If S 2 1/n 1 > k 2 S 2 m/n m, for a pairs 1 i < j m, et σ (i,j) be the minimizer of (8) under the constraint σ i = k σ j and σ 2 i S 2 1/n and σ 2 j S 2 m/n m, when { σ 2 = S 2 /n, i and j} are fixed, where σ i 2 = ˇσ 2 (i,j), k2 S 2 m/n m ˇσ 2 (i,j) S2 1/n 1 ; S 2 1/n 1, ˇσ 2 (i,j) > S2 1/n 1 ; k 2 S 2 m/n m, ˇσ 2 (i,j) < k2 S 2 m/n m ; where ˇσ (i,j) is defined in (10). Step 2: Let (ĩ, j) be the index of (i, j) such that σ (ĩ, j) minimizes (8) among σ (i,j) s, 1 i < j m. Then ˆσ (t+1) = σ (ĩ, j) minimizes (8) subject to σ = kσ. By carefu anaysis of the properties of ˆσ (t+1), one might be abe to further shorten the computations of Step 1 by skipping the cacuation of σ (i,j) s for some (i, j). See the remarks after the proof of Proposition 2.1 in the Appendix for more detai. 3. Exampe In this section, we wi use two simuation exampes and a rea data appication to show how our proposed method works. For simpicity of reporting, we mainy consider the case when m = 2. When m > 2, the resuts are simiar. The Agorithm 1 is used to find the profie og-ikeihood p(k) in (5) over 200 equay spaced grid points of k from 10 4 to 1. Note that when 12

14 k is cose to zero, the smaer component variance, say σ1, 2 is aso cose to zero. Therefore, when k is sma, the initia vaue for µ 1 shoud be one of the observations, otherwise, it is possibe that there wi be no observations assigned to the first component. For agorithm 1, we used 30 initia vaues for each k. The initia vaues for mixing proportions π 1 and π 2 are both 1/2. The initia vaues for the arger component variance σ2 2 is haf of the sampe variance. The first 15 initia vaues for the component means are randomy samped from the observations (x 1,..., x n ). For each of the samped component means, say (x i, x j ) for some i j, we aso used its permuted vaues (x j, x i ) as the initia component means in order to avoid misspecifying the abes between component means and component variance. When k is not cose to zero, one might try some other methods to choose the initia vaues. See McLachan and Pee (2000, 2.12) and Karis and Xekaaki (2003) Simuation Studies Exampe 1: 100 observations are generated from 0.3N(0, )+0.7N(1, 1). Figure 1 is the profie og-ikeihood pot of p(k) vs. k. From the pot, we can see that p(k) goes to infinity when k goes to zero. To better ook at the structure of the profie og-ikeihood pot for the interior parameter space, in Figure 1 (b), we aso provide the pot excuding the area where k is very cose to zero and the corresponding og-ikeihood is reativey very arge. From Figure 1 (b), one can see that there are three interior modes. The information about these three modes are reported in Tabe 1 (they can be easiy ocated 13

15 based on the estimated profie og-ikeihood p(k)). By comparing the vaues of ogl, one can know that the maximum interior mode is at k = Tabe 1: Loca maximizers for Exampe 1 Loca maximizer og L π 1 µ 1 µ 2 σ 1 σ 2 k = k = k = Based on the profie og-ikeihood p(k) and Figure 1, one can aso see that when k < 0.07 the profie og ikeihood is greater than (the profie og ikeihood vaue of the maximum interior mode). The vaue 0.07 can be found based on the estimated p(k). Therefore, when the constrained EM agorithm (Hathaway 1985, 1986) is used to find the MLE, if C < 0.07 in Ω C of (6), the constrained MLE is on the boundary of the parameter space Ω C. In fact, in this case, the constrained MLE even depends on the cut point C, which is not reasonabe. If 0.07 < C < , the constrained EM agorithm can find the maximum interior mode and give the same resut as our profie ikeihood method. However, if C is too arge, it is possibe for the constrained EM agorithm to miss some interior modes. For exampe, if < C < , the constrained EM agorithm wi miss the first interior mode (k = ). Athough the missed one is not the maximum interior mode, in many cases the interior mode can aso provide usefu information, especiay for custering appication (McLachan and Pee, 2000, 8.3.2). 14

16 Exampe 2: 100 observations are generated from 0.3N(0, )+0.7N(1.5, 1). Figure 2 is the profie og-ikeihood pot. From the pot, we can see that there are about three interior modes. The corresponding information is reported in Tabe 2. The main controversy is on the first mode with k = , denoted by ˆθ 1. Athough ˆθ 1 has the argest og-ikeihood among a three modes, it is hard to say whether it is a rea interior mode or a spurious mode that is very cose to the boundary of the parameter space. If one thinks that the mode ˆθ 1 with k = is reasonabe, then one might use it since it has the argest ikeihood among a three modes. If one thinks that ˆθ 1 is not of practica interest since one of the component proportions is ony about 0.07 and the corresponding variance is aso very sma, then one might choose the mode with k = , which has the second argest ikeihood in Tabe 2. In addition, from Figure 2, one can aso see that the area around the mode with k = is much arger than the area around the mode ˆθ 1 with k = Therefore, when using the genera EM agorithm, one might expect that most of the initia vaues wi converge to the mode with k = Tabe 2: Loca maximizers for Exampe 2 Loca maximizer og L π 1 µ 1 µ 2 σ 1 σ 2 k = k = k =

17 Based on Figure 2 and the estimated p(k), one can aso get that when C > in Ω C of (6), the constrained EM agorithm (Hathaway 1985, 1986) wi miss the first mode. When C < 0.06, the constrained EM agorithm can aways find the estimate with arger og ikeihood than the mode with k = In this case, the constrained goba MLE aso depends on the cut point C. If C < 0.01, the constrained goba MLE occurs at the boundary of Ω C and has arger og-ikeihood than the first mode of k = Rea Data Appication The Crab Data: We consider the famous crab data set anayzed by Pearson (1894). The histogram of the data is shown in Figure 3. The data set consists of the measurements on the ratio of forehead to body ength of 1000 crabs samped from the bay of Napes. Foowing Pearson (1894), we use a two-component norma mixture mode to anayze this data set. Figure 4 is our proposed profie og-ikeihood pot. For this exampe, when k is from 10 4 to 10 2, the corresponding og-ikeihood is too arge, which wi affect the dispay of the pot. Therefore, we ony provide the profie og-ikeihood pot for k vaues from 10 2 to 1. From the pot, we can see that there are ony one interior mode (with k = ). When k = , the corresponding MLE of (π 1, µ 1, µ 2, σ 1, σ 2 ) is (0.5360, , , , ). If the constrained EM agorithm is used, based on Figure 4 and the estimated p(k), when the cut point C < 0.05 in Ω C of (6) the constrained goba MLE occurs on the boundary of Ω C and thus depends on the vaue C. When 16

18 C > 0.05, the constrained MLE is the same as the maximum interior mode found by our proposed profie og-ikeihood method. 4. Discussion In this paper, we proposed a profie og ikeihood method to sove the unboundness issue of the ikeihood function for the norma mixture with unequa variance. Unike the usua constrained EM agorithm (Hathaway 1985, 1986), our proposed method does not need to specify a cutting point C in advance. Based on the profie og-ikeihood pot and the estimated p(k), one can easiy identify the maximum interior mode. In addition, based on our proposed method, one can aso ceary see how the cutting point C in (6) affects the constrained goba MLE for the constrained EM agorithm (Hathaway 1985, 1986). The Matab programs for cacuating the profie ikeihood is avaiabe to downoad at wxyao/. For mutivariate norma mixture with unequa covariance matrix, Σ i (i = 1..., m), the ikeihood function is aso unbounded. Simiar to the univariate case, one can aso put some constraint on the covariance matrix. For exampe, et k be the minimum of a the eigenvaues of Σ h Σ 1 j (1 h j m) or et k be the minimum of Σ h / Σ j (1 h j m) (Hathaway 1985, Ingrassia, 2004). Then one can define the profie og ikeihood for k simiar to (7) and use it to find the maximum interior mode. The main difficuty ies on how to maximize the mixture ikeihood under the above constraints. These require further research. 17

19 5. Acknowedgements The authors are gratefu to the two referees for their insightfu comments, which greaty improved this artice. APPENDIX: PROOFS Proof of Theorem 2.1: (a) Let µ 1 = x 1. Then og L(η; x, k) in (4) goes into infinity when k goes to zero. Then the resut foows. (b) Given any k K C, et η(k) be the corresponding parameter maximizing ogl(η; x, k) and θ(k) be the parameter vaue corresponding to η(k). Noting that θ(k) Ω C and ˆθ maximizes ogl(θ; x) in Ω C, hence p(k) = ogl(η(k); x, k) = ogl(θ(k); x) ogl(ˆθ; x). Since ˆk = ˆσ 1 /ˆσ 2, one can easiy know that θ(ˆk) = ˆθ and p(ˆk) = L(ˆθ; x). Hence p(ˆk) p(k). Therefore, ˆk maximizes p(k) in K C. The reverse argument can be proved simiary. (c) Suppose θ is not a oca mode for the og ikeihood of ogl(θ; x) of (3). Then for any given sma ɛ > 0, then exists a θ satisfying θ θ ɛ and ogl( θ; x) > ogl( θ; x), where is the Eucidian norm. Let θ = ( π 1, µ 1, µ 2, σ 1, σ 2 ) and k = σ 1 / σ 2, where σ 1 σ 2. Then p( k) = ogl( η; x, k) = ogl( θ; x), where η = ( π 1, µ 1, µ 2, σ 2 ). Noting that p( k) = ogl( θ; x) < p( k), hence k k. Since θ θ ɛ, where θ = ( π 1, µ 1, µ 2, k σ, σ), 18

20 hence σ 1 k σ ɛ and σ 2 σ ɛ. Therefore k σ ɛ σ + ɛ k = σ 1 σ 2 k σ + ɛ σ ɛ. Let ɛ 0, then k k. Since p( k) < p( k) for a k, k can not be a oca mode, which contradicts the assumption. Hence θ is a oca mode for the og ikeihood of ogl(θ; x) of (3). Before we prove the Proposition 2.1, we first provide a usefu Lemma. Lemma 5.1. Let ˆσ (t+1) = (ˆσ (t+1) 1,..., ˆσ m (t+1) ) be the minimizer of (8), subject to σ = kσ, where k (0, 1). Let ˆσ (t+1) ˆσ (t+1) (2)... ˆσ (t+1) be the corresponding ordered minimizer. Then ˆσ (t+1) S 1 / n 1 and ˆσ (t+1) S m / n m or ˆσ (t+1) S 1 / n 1 and ˆσ (t+1) S m / n m. Proof: For simpicity of proof, we wi assume that S 1 /n 1 < S 1 /n 2 <... < S m /n m. Let Q(σ) = m j=1 ( ) n j og σ j + S2 j. 2σj 2 Note that Q(σ) σ 2 j = n j (σ 2 2σj 4 j Sj 2 /n j ). Hence Q(σ) is minimized when σ 2 j = S 2 j /n j. In addition, Q(σ) is monotone increasing when σ 2 j > S 2 j /n j and monotone decreasing when σ 2 j < S 2 j /n j. ˆσ (t+1) m If ˆσ (t+1) < S 1 / n 1 and ˆσ (t+1) = ˆσ (t+1) = σ (t+1) /k, if considering Q(σ) as a function σ m by fixing other arguments. S j+1 / n j+1. < S m / n m, one can easiy see that Suppose ˆσ (t+1) = ˆσ (t+1) i and S j / n j ˆσ (t+1) < It can be seen that ˆσ (t+1) 19 = S / n, i and j, and

21 ˆσ (t+1) = ˆσ m (t+1), j i and > j. However, under the above assumptions, when ˆσ (t+1) moves coser to S 1 / n 1 and ˆσ m (t+1) = ˆσ (t+1) /k moves coser to S m / n m, the Q(σ) wi decrease. Therefore, the contradiction occurs. Simiary, we can prove the contradiction if we assume ˆσ (t+1) > S 1 / n 1 and ˆσ (t+1) > S m / n m. Therefore, the resut foows. Proof of Proposition 2.1: (a) Based on Lemma 5.1, since S 2 1/n 1 k 2 S 2 m/n m, ˆσ (t+1) S 1 / n 1 and ˆσ (t+1) S m / n m. Suppose S i / n i ˆσ (t+1) < S i+1 / n i+1 and S j 1 / n j 1 < ˆσ (t+1) S j / n j. Based on the properties of Q(σ) as a function of σ j, one can easiy see that ˆσ (t+1) 1 = ˆσ (t+1) 2 =... = ˆσ (t+1) i = ˆσ (t+1) < S i+1 / n i+1, ˆσ (t+1) j = ˆσ (t+1) j+1 =... = ˆσ (t+1) m = ˆσ (t+1) > S j 1 / n j 1, and ˆσ (t+1) = S / n, = i + 1,..., j 1. (b) Based on the Lemma 5.1, since S 2 1/n 1 > k 2 S 2 m/n m, ˆσ (t+1) S 1 / n 1 and ˆσ (t+1) S m / n m. Suppose ˆσ (t+1) = ˆσ (t+1) i be easiy seen that ˆσ (t+1) and ˆσ (t+1) (2) = ˆσ (t+1) j. It can = S /n, i, j and i < j. In addition, if ˆσ (t+1) = S 1 / n 1, then ˆσ (t+1) = ˆσ (t+1) 1. Suppose ˆσ (t+1) = ˆσ (t+1) j = kˆσ (t+1) 1. If considering Q(σ) as a function of σ 1, we can easiy prove that the minimizer is not S 1 / n 1. The contradiction occurs. Hence, ˆσ (t+1) < S 1 / n 1. Simiary, we can aso prove ˆσ (t+1) Remarks: > S m / n m. 1. From the above proof, we can see that we have proved the stronger resuts than Proposition 2.1, i.e. the strict inequaity hods for ˆσ (t+1). Hence, in Step 1 of Section 2.2, we ony need to consider σ (i,j) s when the strict inequaity constraint hods. For exampe, if S 2 1/n 1 > k 2 S 2 m/n m, 20

22 we ony need to consider σ (i,j) s when k 2 /Sm/n 2 m < ˇσ (i,j) < S1/n 2 1, where ˇσ (i,j) is defined in (10). 2. In addition, when S1/n 2 1 k 2 Sm/n 2 m, it can be seen that Q( σ (i,j) ) < Q( σ (i,j )) when i > i, j < j, and the strict inequaity constraint hods for σ (i,j) and σ (i,j ), since σ (i,j) minimizes Q(σ) over arger parameter space than σ (i,j ). Let n(i) be the argest j vaues for fixed i such that the inequaity constraint hods for σ (i,j) and ñ(i) = max{n,..., n(i 1)}. Then, we ony need to consider i when n(i) > ñ(i), i.e. for i, we ony need to consider j = ñ(i) + 1,..., m. If ñ(i) = m for some i, then we can stop and need not cacuate σ (,j) for = i + 1,..., m 1. Proof of Proposition 2.2: (a) Under the constraint that σ 1 = σ 2 =... = σ i = σ and σ j = σ j+1 =... = σ m = σ/k, Q(σ) σ 2 = Therefore the resut foows. i =1 n + m =j n ( σ 2 2σ 4 (b) The proof is simiar to the proof of (a). i =1 S2 + k ) 2 m =j S2 i =1 n + m =j n. 21

23 profie og ikeihood k (a) profie og ikeihood k Figure 1: Profie og-ikeihood pot for Exampe 1: (a) for a k vaues from 10 4 to 1; (b) for k vaues from 0.15 to 1. (b) 22

24 profie og ikeihood k (a) profie og ikeihood k Figure 2: Profie og-ikeihood pot for Exampe 2: (a) for a k vaues from 10 4 to 1; (b) for k vaues from 0.03 to 1. (b) 23

25 Figure 3: Histogram of crab data. The number of bins used is

26 profie og ikeihood k (a) profie og ikeihood k Figure 4: Profie og-ikeihood pot for crab data: (a) for a k vaues from 10 2 to 1; (b) for k vaues from 0.15 to 1. (b) 25

27 References Bezdek, J.C., Hathaway, R.M., and Huggins, V.J. (1985). Parametric estimation for norma mixtures. Pattern Recognition, 3, Böhning, D. (1999). Computer-Assisted Anaysis of Mixtures and Appications, Boca Raton, FL: Chapman and Ha/CRC. Chen, J., Tan, X., and Zhang, R. (2008). Inference for norma mixture in mean and variance. Statistica Sincia, 18, Chen, J. and Tan, X. (2009). Inference for mutivariate norma mixtures. Journa of Mutivariate Anaysis, 100, Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum ikeihood from incompete data via the EM agorithm (with discussion). J. R. Statist. Soc. B, 39, Frühwirth-Schnatter, S. (2006). Finite Mixture and Markov Switching Modes, Springer, Hathaway, R.J. (1985). A constrained formuation of maximum-ikeihood estimation for nomra mixture distributions. Ann. Statist., 13, Hathaway, R.J. (1986). A constrained EM agorithm for univariate mixutres. Journa of Statistica Computation and Simuation, 23,

28 Ingrassia, S. (2004). A ikeihood-based constrained agorithm for mutivariate norma mixture modes. Statistica Methods & Appications, 13, Karis, D. and Xekaaki, E. (2003). Choosing initia vaues for the EM agorithm for finite mixtures. Computationa Statistics & Data Anaysis, 41, Kiefer, N.M. (1978). Discrete parameter variation: efficient estimation of a switching regression mode. Econometrica, 46, Lindsay, B. G., (1995). Mixture Modes: Theory, Geometry, and Appications. NSF-CBMS Regiona Conference Series in Probabiity and Statistics v 5, Hayward, CA: Institure of Mathematica Statistics. McLachan, G. J. and Pee, D. (2000). Finite Mixture Modes. New York: Wiey. Pearson, K. (1894). Contribution to the mathematica theory of evoution. Phiosophica Transactions of the Roya Society of London A, 185, Peters, B.C. and Waker, H.F. (1978). An iterative procedure for obtaining maximum ikeihood estimators of the parameters for a mixture of norma distributions. SIAM Journa on Appied Mathematics, 35, Yao, W. and Lindsay, B. G. (2009). Bayesian Mixture Labeing by Highest Posterior Density. Journa of the American Statistica Association, 104,