Maximum Empirical Likelihood Estimatio (MELE Natha Smooha Abstract Estimatio of Stadard Liear Model - Maximum Empirical Likelihood Estimator: Combiatio of the idea of imum likelihood method of momets, Maximum Etropy Empirical Likelihood Estimator, Uivariate Iformatioal Measure, Relative Etropy, Kullback Leibler Distace (Discrepacy, Kullback-Leibler Iformatio Cirterio, Cressie-Read Distace (Discrepacy, α-etropy, Geeralized Empirical Likelihood (GEL Maximum Empirical Likelihood Estimatio Let y t, t =,2,...,, be a radom sample from the populatio distributio f (y t ;. The parametric imum likelihood estimatio requires the kowledge of the desity fuctio, ad imizes the log likelihood fuctio l L = l f (y t ; The kowledge of the desity fuctio implies the kowledge about the relatioship betwee the parameters ad the desity fuctio. Cosider a discrete Beroulli distirbutio: f (y t ; = y t ( y t, y t or Sice f (y t ; = P(y t = or P(y t, we may cosider the ML estimatio as a costraied imizatio l( s.t. = y t ( y t The distribtuio fuctio cotais all statistical properties of a radom variable, icludig momets, shape, etc. Suppose that we do ot have complete iformaito cotaied i the distributio fuctio, but have oly iformatio about the momets. Ca we still apply the same optimizatio procedure, replacig the desity fuctio with momet coditios as the costraits, i.e., ubiased estimatig fuctio? For example, l ( s.t. > 0, E(y t Note that we require a strict positivity for each. Otherwise, the joit desity will be zero. I fact, this coditio is already embedded i the object fucito because the object fucito will be egative ifiity if oe of the is zero. Oce problem with this formulatio is that the objective fuctio is ot bouded, while the desity fuctio is bouded. We eed a ormatilizatio costrait o. We will use = as the ormalizatio. Aother problem is how to compute the momet coditio without kowig the distributio fucito. The MME computes it by the sample mea: E(y t = (y t
This is equivalet to usig ( as the probability for each observatio. The empirical likelihood estimator uses the empirical probability i the place of ( : E(y t = (y t Uder these two modificatios we ca pose the problem as l( s.t. > 0, ( =, (y t ( Whe we itroduce the momet iformatio we eed to cosider the followig Lagrage equatio ( L = l( κ + (y t First order coditios are oliear i the parameters ad Lagrage multiplies, ad we caot fid the closed form solutios. However, from the first order coditios, we ca still derive importat results κ + (y t (3 κ = (4 (y t (5 Cosider equatio (3: κ κ + (y t κ + (y t (multiplied both sides by + (y t (summig both sides over t κ (by equatios (4 ad (5 Usig the estimate of κ, (6, i equatio (3, we obtai a estimate for : = ˆκ = (6 (2 + (y t = ˆ = + ˆ(y t (7 Whe we plug this estimate for,(7, ito equatio (5, we obtai ˆ as the solutio too: ˆ arg (y t + ˆ(y t (8 This ca be writte as a (-th order polyomial equatio i. The estimate of ca the be foud from the imizatio of the cocetrated objective fuctio l( ˆ (9 Page 2 of 5
The example above is the case of oly oe empirical momet. The MELE of distributio parameter i a geeral form ca be summarized as follows. We wish to imize the empirical likelihood fuctio l(l = l( subject to the strict positivity, ormalizatio, ad ubaised estimatig fuctio > 0, =, E(h t ( E[h(y t ;] where h t ( h(y t ; is a m vector for each t. Sice the true distirbutio fuctio is ukow, this is computed by empirical probability E(h t ( = h t ( Thus, the empirical Lagrage equaito ca be writte as L = l( κ ( where is a m-dimesioal vector of Lagrage multipliers. Oce agai, from the first order coditios, we ca derive importat results: + h t ( (0 κ + h t ( ( κ = (2 h t ( (3 Cosider equatio (: κ + h t ( κ + h t ( (multiplied both sides by κ + h t ( (summig both sides over t κ (by equatios (2 ad (3 = ˆκ = (4 Usig the estimate of κ, (4, i equatio (, we ca derive the solutio for for a give ad : + h t ( = ˆ = [ h t (] (5 By pluggig i (5 ito equatio (3, the solutio for for a give is give implicity by: h t ( h t ( (6 Page 3 of 5
Substituig the solutio for ito the empirical log-likelihood fuctio, we ca write a cocetrated likelihood fuctio as l(l = l( = l ( [ h t ( ] = Therefore, the origial imizatio problem is equivalet to mi l ( h t ( s.t. l( l ( h t ( (7 h t ( h t ( (8 Note that the costrait equatio that defies the solutio for implicitly is the same as the first order coditio of the imizatio of the followig equatio with respect to : l ( h t ( Hece, we ca write the problem i compact form as ˆ MEL argmi (9 l ( h t ( (20 Maximum Etropy Empirical Likelihood Estimator Uivariate Iformatioal Measure Suppose we have a set of evets E i with a probability p i. If we receive a message that a certai evet has occured, what will be a reasoable measure of the iformatioal cotet of such a message? Shao measured it by l( p i = l(p i. If p i is low (e.g., p i.0, E i is ot expected to occur ofte. If E i did occure, the ews of such a occurece is a surprise ad cotais a great deal of iformatio. This measure is ot symmetric i the sese that the message of occurece ad the message of o-occurece give differet values l( p i ad l( p i. Whe p i is low, the message of the occurece of E i is a big surprise, but the message of o-occurece is ot much of a surprise. Before we receive a message, the expected iformatioal cotet of the message is called Shao s etropy. Whe we have evets with probabilities p i, Shao s etropy is expressed as SE(p = p i l( p i = p i l(p i (2 The imum etropy empirical likelihood (MEEL estimator imizes this objective fuctio subject to the ormalizatio ad empirical momets costraits: ( p i p i l(p i s.t. p i =, where h i ( is a m vector. Settig uhe Lagrage equatio L = p i l(p i κ ( p i From the first order coditios, we ca derive importat results: p i h i ( (22 + p i h i ( (23 (l(p i + κ + h i ( (24 κ p i = (25 p i h i ( (26 Page 4 of 5
From equatio (24, we ca obtai a expressio for p i : (l(p i + κ + h i ( l(p i = h i ( κ + = p i = exp [ h i ( (κ + ] Takig this expressio for p i, we ow substitute it i (25: e h i ( e κ+ = = e κ+ = e h i ( Thus, the explicity solutio for p i is give by: ˆp i (, = e h i ( e h i ( (27 Now, substitutig the expressio for p i, (27, ito equatio (26, we derive a solutio for : e h i ( e h i ( h i ( = ˆ ( = e h i ( h i ( (28 Therefore, the origial imizatio problem is equivalet to ( p i ˆp i l( ˆp i s.t. e h i ( h i ( e h i ( (29 Note that the costrait equatio that defies the solutio for implicitly is the same as the first order coditio of the miimizatio of the followig equatio with respect to : mi l( e h i ( (30 Hece, we ca write the problem i compact form as ˆ MEEL arg mi l( e h i ( (3 Page 5 of 5