The Likelihood Ratio Test

Size: px

Start display at page:

Download "The Likelihood Ratio Test"

Alicia Horton
5 years ago
Views:

1 LM 05 Likelihood Ratio Test 1 The Likelihood Ratio Test The likelihood ratio test is a geeral purpose test desiged evaluate ested statistical models i a way that is strictly aalogous to the F-test for reduced models (RM) ad full models (FM) commoly employed with liear models (see Biostatistics Worksheet 402). I both, failure to reject the ull hypothesis results i model simplificatio. The likelihood ratio test works ot oly with liear models, show here, but may be applied to a very wide array of problems ivolvig Geralized Liear Models (GLM), where Maximum Likelihood (ML) or Restricted Maximum Likelihood (REML) methods are utilized to estimate model parameters. The latter methods/models iclude, amog others, Logistic Regressio (see GLM 020), Poisso Regressio (see GLM 040), ad Liear Mixed Models (see LMM 060) described i Worksheets o the Biologist's Aalytic Toolkit Website uder Statisistical Models. For direct compariso of results, the data set aalyzed here is the same as for the geeral F test (Biostatisics Worksheet 402). As ca be see, F test ad likelihood ratio tests give similar but ot exactly the same results. Helpful discussio of this approach appears i Kuter et al. (KNNL) Applied Liear Statistical Models 5th Editio, ad umerous statistics websites. Example i R: #LOG LIKELIHOOD AND LIKELIHOOD RATIO TEST setwd("c:/data/models/") K=read.table("KNNLCh9SurgicalUit.txt") K a ach(k) Fittig Full ad Reduced liear models: Full Model: #FITTING THE FULL LINEAR MODEL FM=lm(Y~X1+X2+X3+X4+X5+factor(X6)) FMg=glm(Y~X1+X2+X3+X4+X5+factor(X6)) aova(fm) aova(fmg) Note: R's fuctio glm() is also employed here sice this fuctio produces a data class for which the geeral wrapper aova() assumes aova.glm() which produces likelihood ratio results as default. > K X1 X2 X3 X4 X5 X6 Y > aova(fm) X *** X e-08 *** X e-13 *** X X factor(x6) Residuals Sigif. codes: 0 *** ** 0.01 * σ FM = = ^ "stadard error" (stadard deviatio of the residuals) for the full model > aova(fmg) Model: gaussia, lik: idetity Terms added sequetially (first to last) Df Deviace Resid. Df Resid. Dev NULL X X X X X factor(x6)

2 LM 05 Likelihood Ratio Test 2 Reduced Model: FITTING A REDUCED LINEAR MODEL RM=lm(Y~X1+X2+X3+X5) RMg=glm(Y~X1+X2+X3+X5) aova(rm) aova(rmg) > aova(rm) X *** X e-08 *** X e-14 *** X Residuals Sigif. codes: 0 *** ** 0.01 * σ RM = = ^ "stadard error" (stadard deviatio of the residuals) for the reduced model > aova(rmg) Model: gaussia, lik: idetity Terms added sequetially (first to last) Df Deviace Resid. Df Resid. Dev NULL X X X X Estimatig Stadard Error usig Maximum Likelihood: differece i umber of parameters > betwee models FM & RM k MLσ FM := σ FM k + r MLσ RM := σ RM := 54 k := 7 r := 2 σ FM := σ RM := MLσ FM = MLσ RM = > #CALCULATING MAXIMUM LIKELIHOOD STANDARD DEVIATION > =legth(k[,1]) > #NUMBER OF CASES IN DATASET K [1] 54 > k=legth(k) > k #NUMBER OF VARIABLES IN FM [1] 7 > r=2 #DIFFERENCE IN NUMBER OF VARIABLES FM VS RM > r [1] 2 > #EXTRACTING STANDARD ERRORS: > #FOR FM: > FMsigma = summary(fm)$sigma > FMsigma > #FOR RM: > RMsigma = summary(rm)$sigma > RMsigma [1] > #MAXIMUM LIKELIHOOD STANDARD DEVIATIONS > #FOR FM: > FMsigma.ML = FMsigma*sqrt((-k)/) > FMsigma.ML [1] > #FOR RM: > RMsigma.ML = RMsigma*sqrt((-k+r)/) > RMsigma.ML [1]

3 LM 05 Likelihood Ratio Test 3 Calculatig Log Likelihoods for Each Model: l likelihood value for FM > l likelihood value for RM > ^ l ="atural logs" i base e > # LOG LIKELIHOOD OF MODELS > # FM: > sum(log(dorm(x = Y, mea = predict(fm), sd = FMsigma.ML))) [1] > loglik(fm) 'log Lik.' (df=8) > # RM: > sum(log(dorm(x = Y, mea = predict(rm), sd = RMsigma.ML))) [1] > loglik(rm) 'log Lik.' (df=6) Note: log likelihoods for each model are calculated here usig maximum likelihood estimates of stadard error for each model separately. This cotrasts with the use of stadard error usig oly the FM i the test below. Likelihood Ratio Test: Assumptios: - Stadard Liear Regressio depeds o specifyig i advace which variable is to be cosidered 'depedet' ad which 'idepedet'. This decisio matters as chagig roles for Y & X usually produces a differet result.\ - Y 1, Y 2, Y 3,..., Y (depedet variable) is a radom sample. Note: Although a Normal distributio is assumed here for Y i a liear model, i other istaces of the likelihood ratio test, this assumptio does't apply. - X 1, X 2, X 3,..., X (idepedet variable) with each value of X i matched to Y i Withi this setup, two models for the relatioship betwee X ad Y variables are explicitly compared: Full Model: Y i = β 0 + Σβ j X i + ε i Reduced Model: Y i = β 0 + Σβ k X i + ε i Hypotheses: where: Y i ad [X 1,X 2,... X i ] are matched depedet ad idepedet variables, ad β 0 is the y itercept of the regressio lie (traslatio) β j are slope coefficiets for the full set of idepedet variables X 1,X 2,... X j β k are slope coefficiets for a smaller set of idepedet variables withi X j ε i is the error factor i predictio of Y i ad a radom variable ~N(0,σ 2 ). H 0 : coefficiets i j but NOT INCLUDED i k = 0. Note: this is always the more parsimoious (i.e., smaller) model H 1 : at least some of these coeficiets ot 0

4 LM 05 Likelihood Ratio Test 4 Degrees of Freedom: = 54 k = 7 r = 2 Sum of Squares ad Stadard Error for FM: < = umber of matched observatios i dataset < k = umber of variables i FM < r = differeces i umber of variables betwee FM & RM > aova(fm) X X e-08 X e-13 X X factor(x6) Residuals > #LIKELIHOOD RATIO TEST: > #SUM OF SQUARES ERROR FOR MODELS: > SSE.FM = sum((y-predict(fm))^2) #SSE for FM > SSE.FM [1] > SSE.RM = sum((y-predict(rm))^2) #SSE for RM > SSE.RM [1] > aova(rm) X *** X e-08 *** X e-14 *** X Residuals s := SSE FM := SSE RM := > #STANDARD ERROR FOR FM: > s=summary(fm)$sigma > s > s=sqrt(summary(fmg)$dispersio) > s ^ Stadard errors are the square root of MSE, see above. Relative Likelihoods: 1 SSE FM 2 LFM := e 1 SSE RM 2 LRM := e C := Likelihoods: 1 ( 2 π 2 ) Λ FM := C LFM Λ RM := C LRM see eq 1.26 i KNNL LFM = LRM = C = Λ FM = Λ RM = > #RELATIVE LIKELIHOODS FOR THE MODELS: > LFM = exp(-(1/2)*(sse.fm/s^2)) #TIMES CONSTANT C > LFM [1] e-11 > LRM = exp(-(1/2)*(sse.rm/s^2)) #TIMES CONSTANT C > LRM [1] e-11 > #CONSTANT C: > #NUMBER OF CASES IN DATASET K [1] 54 > C=1/((2*pi*s^2)^(/2)) #CONSTANT IN EQ 1.26 IN KNNL > C [1] > LCFM=C*LFM > LCFM [1] > LCRM=C*LRM > LCRM [1]

5 LM 05 Likelihood Ratio Test 5 Likelihood Ratio Test Statistic: SSE FM := SSE RM := > #LOG LIKELIHOOD RATIO STATISTIC: > LRT=(SSE.RM - SSE.FM)/s^2 > LRT #LOG LIKELIHOOD RATIO STATISTIC [1] LRT := ( ) SSE RM SSE FM LRT = < differece here due to roudig... Critical Value of the Test: α := 0.05 CV := qchisq 1 α, r Decisio Rule: IF F > CV, THEN REJECT H 0 OTHERWISE ACCEPT H 0 LRT = Probability Value: P := 1 pchisq( LRT, r) IMPORTANT NOTE: Prototype i R: < Probability of Type I error must be explicitly set ( ) CV = CV = P = < ote degrees of freedom reflect differece betwee the models > #PROBABILITY OF NULL HYPOTHESIS RM > P=1-pchisq(LRT,2) > P #PROBABILITY [1] FALURE to reject H 0 i this test meas that the MORE PARSIMONIOUS model RM is PREFERRED! #LIKELIHOOD RATIO TEST: aova(rm,fm,test="lrt") aova(rmg,fmg,test="lrt") > #LIKELIHOOD RATIO TEST: > aova(rm,fm,test="lrt") Model 1: Y ~ X1 + X2 + X3 + X5 Model 2: Y ~ X1 + X2 + X3 + X4 + X5 + factor(x6) Res.Df RSS Df Sum of Sq Pr(>Chi) > aova(rmg,fmg,test="lrt") Model 1: Y ~ X1 + X2 + X3 + X5 Model 2: Y ~ X1 + X2 + X3 + X4 + X5 + factor(x6) Resid. Df Resid. Dev Df Deviace Pr(>Chi)

4.5 Generalized likelihood ratio test

4.5 Generalized likelihood ratio test 4.5 Geeralized likelihood ratio test A assumptio that is used i the Athlete Biological Passport is that haemoglobi varies equally i all athletes. We wish to test this assumptio o a sample of k athletes.