MM and ML for a sample of n = 30 from Gamma(3,2) ===============================================

Transcription

1 and for a sample of n = 30 from Gamma(3,2) =============================================== Generate the sample with shape parameter α = 3 and scale parameter λ = 2 > x=rgamma(30,3,2) > x [1] [9] [17] [25] > summary(x) Min. 1st Qu. Median Mean 3rd Qu. Max > xave=mean(x) > xave [1] ! the sample average, x > xsd=sqrt(var(x)) > xsd [1] ! the sample standard deviation, s > stem(x) The decimal point is at the > hist(x) 1

2 Estimation of the Unknown Parameters α and λ: =================================== Now pretend we don t know that this sample is from a Gamma(3,2) population. Treat it as a random sample of n = 30 data points from some population. We will use the Gamma(α,λ) distribution as the statistical model for this data set. Method of Moments estimates (Es) for this sample: > amm=xave^2/(29*xsd^2/30) > amm [1] ! αˆ > lmm=amm/xave > lmm [1] ! λˆ Method of Maximum Likelihood estimates (Es) for this sample: Evaluate log of average average of logs: > c=log(xave)-mean(log(x)) > c [1] First, let s plot f ( α ) = Γ'( / Γ( log( + c, the function of α for which we need to find the root to determine the Es. Note that Γ '( / Γ( can be calculated as digamma (α ) in R. a) We know α > 0. The E suggests α is around 3. Let s plot f (α ) for 0 < α < 10: > xx=seq(0.1,10,0.1) > y=digamma(xx)-log(xx)+c > plot(xx,y) > abline(0,0) 2

3 b) It looks like f (α ) is monotone increasing, with a root somewhere between 2 and 4. (You can check 2 monotonicity by plotting f '(. Note that Γ' '( / Γ( [ Γ'( / Γ( ] can be calculated as trigamma ( α ) in R.) Plot f (α ) on the interval [2, 4]: > xx=seq(2,4,0.01) > y=digamma(xx)-log(xx)+c > plot(xx,y) > abline(0,0) The root is close to 2.5, so αˆ is going to be a fair bit different from ˆ α From the plot, it is clear we can use Newton s method to find the root as precisely as desired. The R function N0gamma below determines the root (iteration stops once absolute change < 10-6 ) and prints out the details of the iterations (Note that f (α ) is used in N0gamma. No real reason for that.): > N0gamma function(a,x){ it=0 e= diff=1 dx=log(mean(x))-mean(log(x)) while(abs(diff)>e){ it=it+1 f=log(a)-digamma(a)-dx df=1/a-trigamma(a) diff=-f/df cat("it= ",it,"a= ",a," f= ",f," df= ",df," diff= ",diff,"\n") a=a+diff cat("e= ",a,"\n") return(a) Let s try an initial guess of 2 for the root: > N0gamma(2,x) It= 1 a= 2 f= df= diff= It= 2 a= f= df= diff= It= 3 a= f= df= diff=

4 It= 4 a= f= e-06 df= diff= e-05 It= 5 a= f= e-12 df= diff= e-11 E= [1] Note how the iteration converges to the root α : the first step is quite large (diff = 0.42) but the successive steps are smaller: 6x10-3, 1x10-5 and 9x10-11 at the 3 rd, 4 th and 5 th iterations, reflecting the quadratic convergence property of Newton s method. The value of the function at the successive iterations very quickly becomes very small. Also note that the derivative of the function changes very little once you get close to the root (as is also clear from the plot). For this function, Newton s method works very well. What if we start further away? Let s try an initial guess of 4: > N0gamma(4,x) It= 1 a= 4 f= df= diff= It= 2 a= f= df= diff= It= 3 a= f= df= diff= It= 4 a= f= df= diff= It= 5 a= f= e-05 df= diff= It= 6 a= f= e-08 df= diff= e-07 E= [1] Basically the same story: even though the first iteration overshoots the root, the successive iterations quickly locate the root. If you used a very large initial guess, the first step could lead to a negative updated value of α not a permissible value. N0gamma would then bomb when it tries to evaluate the function at that updated value. (Try an initial value of 6, for example). Depending on the nature of the function, Newton s method can require careful guidance until it gets into the vicinity of the root where the quadratic convergence property takes hold. > aml= ! αˆ > lml=aml/xave > lml [1] ! λˆ Evaluation of the Precision of these Estimates via Parametric Bootstrap Method: ========================================================== The R function bootgamma first calculates αˆ and for the sample x and then generates B samples of size n (the same size as the original sample) from the Gamma( αˆ, λˆ ) distribution. The Es are evaluated for each of these B bootstrap samples and are stored in a Bx2 matrix. > bootgamma function(x,b){ n=length(x) am=(n/(n-1))*mean(x)^2/var(x) lm=am/mean(x) ab=matrix(0,nrow=b,ncol=2) λˆ 4

5 for(i in 1:B){ xb=rgamma(n,shape=am,rate=lm) ab[i,1]=(n/(n-1))*mean(xb)^2/var(xb) ab[i,2]=ab[i,1]/mean(xb) return(ab) Let s do B = 1000 bootstrap samples: > ymm=bootgamma(x,1000) > summary(ymm) X1 X2 Min. :1.354 Min. : st Qu.: st Qu.: Median :3.429 Median : Mean :3.616 Mean : rd Qu.: rd Qu.: Max. :9.242 Max. : > plot(ymm[,1],ymm[,2]) Now do the same thing for the Es αˆ and λˆ. The R function bootgamma uses the R function Ngamma to evaluate the Es for the bootstrap samples; Ngamma is a version of the funtion N0gamma above with all the detailed output suppressed. > bootgamma function(x,b){ n=length(x) am=ngamma(mean(x)^2/var(x),x) lm=am/mean(x) ab=matrix(0,nrow=b,ncol=2) for(i in 1:B){ xb=rgamma(n,shape=am,rate=lm) ab[i,1]=ngamma(mean(xb)^2/var(xb),xb) ab[i,2]=ab[i,1]/mean(xb) return(ab) 5

6 Let s do B = 1000 bootstrap samples: > yml=bootgamma(x,1000) > summary(yml) X1 X2 Min. :1.357 Min. : st Qu.: st Qu.: Median :2.657 Median : Mean :2.797 Mean : rd Qu.: rd Qu.: Max. :6.892 Max. : > plot(yml[,1],ym[,2]) Note that the x and y scales are different in this plot than in the corresponding plot for the Es. Both the summaries and the plots seem to indicate less scatter (in both dimensions!) for the bootstrap Es than for the bootstrap Es. So it looks like, at least for this example, that the Es are less variable than the Es. Let s compare graphically: Bootstrap estimates of α : > boxplot(ymm[,1],yml[,1]) 6

7 Bootstrap estimates of λ : > boxplot(ymm[,2],yml[,2]) In both cases, the boxplots are centered at different values for the E and the E, but it is also clear that box part of the boxplot (the central portion of those datasets) is considerably less spread out for the E than for the E. In case you prefer histograms to boxplots: > par(mfrow=c(2,1)) > hist(ymm[,1],prob=t) > hist(yml[,1],prob=t) 7

8 > hist(ymm[,2],prob=t) > hist(yml[,2],prob=t) To quantify the variability of the bootstrap values of αˆ and λˆ, let s calculate the variancecovariance matrix: > cov(ymm) [,1] [,2] [1,] [2,] So, the bootstrap estimates of: " the SEs of αˆ and λˆ are = and = 0.690, respectively, " the correlation between αˆ and λˆ is / = Similarly for the bootstrap Es: > cov(yml) [,1] [,2] [1,] [2,] So, the bootstrap estimates of: " the SEs of αˆ and λˆ are = and = 0.495, respectively, " the correlation between αˆ and λˆ is / = The bootstrap estimates of the SEs are much smaller for the Es than for the Es, indicating that, at least for this example, the Es are much less variable than the Es. We will soon see this is a general phenomenon: in a sense to be described more precisely in class, Es are always better than Es. In fact, we will show that (in that same sense) no estimator can do better than the E. 8

9 You may be curious how much things change if you do more bootstrap samples: > ymm=bootgamma(x,10000) > summary(ymm) X1 X2 Min. : Min. : st Qu.: st Qu.: Median : Median : Mean : Mean : rd Qu.: rd Qu.: Max. : Max. : > cov(ymm) [,1] [,2] [1,] [2,] > yml=bootgamma(x,10000) > summary(yml) X1 X2 Min. :1.262 Min. : st Qu.: st Qu.: Median :2.666 Median : Mean :2.801 Mean : rd Qu.: rd Qu.: Max. :8.889 Max. : > cov(yml) [,1] [,2] [1,] [2,] Plotting the bootstrap Es and the bootstrap Es (note the differences in the scales) yields: 9

10 The resulting bootstrap estimates of the SEs: " of αˆ and λˆ are = and = 0.669, respectively, " of αˆ and λˆ are = and = 0.493, respectively, In summary, once the results are expressed as estimated SEs, they differ in only minor ways from the corresponding results obtained with B = If we assume that n = 30 is large enough to consider both the Es and the Es to be, for all practical purposes, unbiased (note that the results above suggest this may NOT be the case), then we would estimate the efficiency of the E relative to the E to be approximately: " (0.770/1.078) 2 = 0.51, or 51% for the estimation of α, " (0.493/0.669) 2 = 0.54, or 54% for the estimation of λ. That is, we would need almost twice the sample size to attain the same asymptotic variance with the Es as with the Es. Confidence Intervals for the Unknown Parameters α and λ: =========================================== We will see in class that ˆ θ ± 1.96 Sˆ E( ˆ) θ provides an approximate 95% confidence interval for θ whenever the estimator θˆ is asymptotically normally distributed, as is the case for Es and Es. Using the bootstrap estimates of the SEs (let s use the B=10000 results), we obtain approximate 95% confidence intervals: For α of: " ± 1.96 x $ (1.05, 5.27) based on the E, αˆ " ± 1.96 x $ (1.03, 4.05) based on the E, αˆ For λ of: " ± 1.96 x $ (0.50, 3.13) based on the E, λˆ " ± 1.96 x $ (0.49, 2.42) based on the E, λˆ Note that, in both cases, the lower endpoints of the two confidence intervals are basically the same so the main difference is in the upper endpoint. Because the estimated SEs for the Es are smaller than those for the Es, the confidence intervals based on the Es are quite a bit shorter that is, the Es lead to considerably tighter ranges of plausible values for the unknown parameters. Comparison to Results Based on Large Sample Theory: ======================================== As noted in class, the (bivariate) CLT for the first two sample moments together with the delta method yield the asymptotic (bivariate) distribution of the E and the general large sample theory results for Es presented in class yield the asymptotic distribution of the E. Specifically, for the case of sampling from a Gamma(α,λ) distribution, as n : and ˆ θ ˆ α = ˆ λ N α 1 2α ( α + 1), λ n 2( α + 1) λ 2( α + 1) λ 2 (2α + 3) λ / α 2, 10

11 ˆ θ ˆ α = ˆ λ N α 1 α, λ n[ αg( 1] λ λ 2 λ g( 2, where g( = Γ' '( / Γ( [ Γ' ( / Γ( ] 2 is the trigamma function. Plugging the values of the estimates into these expressions (need to use Es in the expressions for θˆ and Es in the expressions for θˆ as that is all we would have in practice) yields the estimated (asymptotic) SEs: " for αˆ and λˆ as [2(3.1609)(4.1609)/30] 1/2 = and [9.3219(1.8131) 2 /30(3.1609)] 1/2 = 0.568, respectively, " for αˆ and λˆ as [2.5376/30(0.2222)] 1/2 = and [(1.4555) 2 (0.4816)/30(0.2222)] 1/2 = 0.391, respectively. Compared to the bootstrap results, in both cases, it appears these asymptotic expressions underestimate the variability to some degree; that is, it looks like n = 30 may not be a large enough sample size for these asymptotic results to provide accurate approximations. Let s check this using simulation. As you have learned in the lab, it is very easy to use R to carry out simulation experiments and this is a powerful tool that you can use in many circumstances, for example, when you want to check the properties of some random phenomenon and the necessary probability calculations are too hard to do analytically. Simulate distributions of E and E for a sample of n = 30 from Gamma(3,2): ============================================================ The function simgamma evaluates Es for repeated samples of n = 30 from Gamma(3,2). You could easily make this function suitable for any Gamma distribution. We simulate 1000 samples: > simgamma function(n,s){ ab=matrix(0,nrow=s,ncol=2) for(i in 1:S){ xb=rgamma(n,shape=3,rate=2) ab[i,1]=(n/(n-1))*mean(xb)^2/var(xb) ab[i,2]=ab[i,1]/mean(xb) return(ab) > ymm=simgamma(30,1000) > summary(ymm) X1 X2 Min. :1.328 Min. : st Qu.: st Qu.: Median :3.293 Median : Mean :3.385 Mean : rd Qu.: rd Qu.:

12 Max. :8.597 Max. : > cov(ymm) [,1] [,2] [1,] [2,] Now do the same thing for the Es: > simgamma function(n,s){ ab=matrix(0,nrow=s,ncol=2) for(i in 1:S){ xb=rgamma(n,shape=3,rate=2) ab[i,1]=ngamma(mean(xb)^2/var(xb),xb) ab[i,2]=ab[i,1]/mean(xb) return(ab) > yml=simgamma(30,1000) > summary(yml) X1 X2 Min. :1.568 Min. : st Qu.: st Qu.:1.750 Median :3.122 Median :2.095 Mean :3.283 Mean : rd Qu.: rd Qu.:2.583 Max. :8.783 Max. :5.849 > cov(yml) [,1] [,2] [1,] [2,] First consider estimation of α. Plot the histograms of the simulated Es and Es of α and superimpose a plot of the normal density corresponding to the asymptotic approximations: > hist(ymm[,1],prob=t) > xp=seq(1.33,8.59,0.01) > s=sqrt(2*3*4/30) > s [1] > lines(xp,dnorm(xp,mean=3,sd=s)) > hist(yml[,1],prob=t) > xp=seq(1.57,8.78,0.01) > s=sqrt(3/(30*(3*trigamma(3)-1))) > s [1] > lines(xp,dnorm(xp,mean=3,sd=s)) 12

13 The histograms of the simulated Es and Es are skewed to the right (positively) and hence more spread out than the (symmetric) asymptotic normal approximation. Although not overly dramatic, this skewness is apparent in the qqplots for the E (top plot below) and E (bottom plot). > qqnorm(ymm[,1]) > qqnorm(yml[,1]) 13

14 Because of this skewness in the true distributions, the (symmetric) asymptotic normal approximations will underestimate the variability as we suspected was the case (now we know!). We conclude that a larger value of n is required for the asymptotic normal approximations to the distributions of αˆ and αˆ to be accurate. Now consider estimation of λ similarly. > hist(ymm[,2],prob=t) > xp=seq(0.89,5.74,0.01) > s=sqrt((2*3+3)*2*2/(3*30)) > s [1] > lines(xp,dnorm(xp,mean=2,sd=s)) > hist(yml[,2],prob=t) > xp=seq(1.03,5.84,0.01) > s=sqrt(2*2*trigamma(3)/(30*(3*trigamma(3)-1))) > s [1] > lines(xp,dnorm(xp,mean=2,sd=s)) The histograms of the simulated Es and Es are again skewed to the right (positively) and hence more spread out than the (symmetric) asymptotic normal approximation. This skewness can also be seen in the qqplots for the E (top plot below) and the E (bottom plot). > qqnorm(ymm[,2]) > qqnorm(yml[,2]) 14

15 Transformations to Improve Asymptotic Normal Approximations: =============================================== The asymptotic normal approximations to the distributions of the Es and the Es do not appear to be very accurate for this example. Might these approximations perform better on a different scale? For positive random variables that have (positively) skewed distributions, a natural transformation to consider is the log it pulls in the very large positive values so might eliminate the positive skewness. Of course, it could induce negative skewness! From the asymptotic results given above, the delta method easily yields that for the case of sampling from a Gamma(α,λ) distribution, as n : and ˆ ˆ log( α ) log( α ) 1 2( α + 1) / α 2( α + 1) / α log( θ = ) ˆ N 2,, log( λ ) log( λ) n 2( α + 1) / α (2α + 3) / α ˆ ˆ log( α ) log( 1 1/ α log( θ ) = ˆ N 2, log( λ ) log( λ) n[ αg( 1] 1/ α 1/ α g(. Let s compare the histograms of the logs of the simulated estimates of α to these asymptotic normal approximations: > hist(log(ymm[,1]),prob=t) > xp=seq(0.29,2.15,0.01) > s=sqrt(2*4/(3*30)) > lines(xp,dnorm(xp,mean=log(3),sd=s)) > hist(log(yml[,1]),prob=t) > xp=seq(0.46,2.17,0.01) > s=1/sqrt(30*3*(3*trigamma(3)-1)) > lines(xp,dnorm(xp,mean=log(3),sd=s)) 15

16 Note how much more symmetric than the earlier histograms these are. The asymptotic normal approximations to the distributions of log( ˆ α ) and log( ˆ α ) are still far from perfect but they now look reasonably accurate. The improvement is also apparent in the qqplots that look very much like straight lines: > qqnorm(log(ymm[,1])) > qqnorm(log(yml[,1])) 16

17 Similarly, for the simulated estimates of λ: > hist(log(ymm[,2]),prob=t) > xp=seq(-0.11,1.74,0.01) > s=sqrt((2*3+3)/(3*30)) > lines(xp,dnorm(xp,mean=log(2),sd=s)) > hist(log(yml[,2]),prob=t) > xp=seq(0.03,1.76,0.01) > s=sqrt(trigamma(3)/(30*(3*trigamma(3)-1))) > lines(xp,dnorm(xp,mean=log(2),sd=s)) These histograms are also much more symmetric than the earlier histograms of the estimates on the raw scale. The asymptotic normal approximations to the distributions of log( ˆ λ ) and log( ˆ λ ) now look reasonably accurate and this is also relected in the qqplots that look very much like straight lines: > qqnorm(log(ymm[,2])) > qqnorm(log(yml[,2])) 17

18 You may want to check how much different things look for this example when: " the values of α and λ being considered are different, " the value of n being considered is different. Of course, you can equally well carry out such an investigation for any other example! Your computer is a powerful tool for learning. You can use it to carry out: " exact calculations of probabilities that are not otherwise feasible, " the Monte Carlo method to approximate probabilities (and other integrals), " simulation studies to help understand when asymptotic approximations are accurate. THE END 18