Random Effects ANOVA Grant B. Morgan Baylor University This post contains code for conducting a random effects ANOVA. Make sure the following packages are installed: foreign, lme4, lsr, lattice. library(foreign) library(lsr) library(lme4) Loading required package: Matrix raneff<-read.spss("/users/grant_morgan/box Sync/Teaching/EDP 6360 - Experimental Design/Data/Ra Warning in read.spss("/users/grant morgan/box Sync/Teaching/EDP 6360 - Experimental Design/Data/Random Effect.sav", : /Users/grant morgan/box Sync/Teaching/EDP 6360 - Experimental Design/Data/Random Effect.sav: Unrecognized record type 7, subtype 18 encountered in system file re-encoding from CP1252 First, we should generate some descriptive statistics for the outcome variable, Score, for each of the randomly selected raters. library(psych) describeby(raneff$score, raneff$rater) Descriptive statistics by group group: 1 vars n mean sd median trimmed mad min max range skew kurtosis se X1 1 7 13.86 3.13 13 13.86 1.48 10 19 9 0.45-1.45 1.18 -------------------------------------------------------- group: 2 vars n mean sd median trimmed mad min max range skew kurtosis se X1 1 7 9.57 2.07 10 9.57 1.48 6 12 6-0.47-1.35 0.78 -------------------------------------------------------- group: 3 vars n mean sd median trimmed mad min max range skew kurtosis se X1 1 7 10 3.46 11 10 1.48 5 15 10-0.19-1.49 1.31 -------------------------------------------------------- group: 4 1
vars n mean sd median trimmed mad min max range skew kurtosis se X1 1 7 10.14 2.27 11 10.14 1.48 6 12 6-0.66-1.19 0.86 The null hypothesis in the random effects ANOVA states that the variance of the means is equal to zero. Let s generate some plots to see how the distribution of scores for each of the randomly selected raters overlaps. library(lattice) xyplot(raneff$score ~ raneff$rater, pch = 16, col = "blue") boxplot(raneff$score ~ raneff$rater, pch = 16, col = "blue") 2
6 8 10 12 14 16 18 1 2 3 4 Both of these plots indicates that the variability seems pretty similar for each rater, but the scores appear to be centered at different locations for each rater. We also see this from the descriptive statistics. Next, let s estimate the random effects ANOVA model. raneff$rater<-as.factor(raneff$rater) raneff.out<-aov(raneff$score ~ Error(raneff$Rater), data=raneff) summary(raneff.out) Error: raneff$rater Df Sum Sq Mean Sq F value Pr(>F) 3
Residuals 3 83.25 27.75 Error: Within Df Sum Sq Mean Sq F value Pr(>F) Residuals 24 187.4 7.81 raneff.out Call: aov(formula = raneff$score ~ Error(raneff$Rater), data = raneff) Grand Mean: 10.89286 Stratum 1: raneff$rater Terms: Residuals Sum of Squares 83.25 Deg. of Freedom 3 Residual standard error: 5.267827 Stratum 2: Within Terms: Residuals Sum of Squares 187.4286 Deg. of Freedom 24 Residual standard error: 2.794553 sum((raneff.out$"within"$residuals)**2) [1] 187.4286 sum((raneff.out$"raneff$rater"$residuals)**2) [1] 83.25 raneff.out$"within"$df.residual [1] 24 raneff.f<-(sum((raneff.out$"raneff$rater"$residuals)**2)/raneff.out$"raneff$rater"$df.residual) (sum((raneff.out$"within"$residuals)**2)/raneff.out$"within"$df.residual) raneff.f [1] 3.553354 4
raneff.p<-pf(raneff.f,raneff.out$"raneff$rater"$df.residual,raneff.out$"within"$df.residual,low raneff.p [1] 0.0293461 raneff.results<-matrix(c((sum((raneff.out$"raneff$rater"$residuals)**2)/raneff.out$"raneff$rate pf(raneff.f,raneff.out$"raneff$rater"$df.residual,raneff.out$"within"$df.residual,lower.tail= colnames(raneff.results)<-c("f","p-value") raneff.results F p-value [1,] 3.553354 0.0293461 For some reason, the aov function in R does provide not the p-value for random effects ANOVA so I have computed it. According to the results, there is sufficient evidence to conclude that the variance of the means of the raters is nonzero. There is no post hoc analysis because we are not dealing with specific (i.e., fixed) values of the raters. If we replicated the sudy, the raters would likely be different. We can get a better understanding of the random effect by computing the variance components and effect size. We can compute the intraclass correlation coefficient. This estimates the proportion of variance in the outcome variable that is explained by the random effect in the population. raneff.vc<-lmer(raneff$score ~ 1 + (1 raneff$rater), data = raneff) summary(raneff.vc) Linear mixed model fit by REML ['lmermod'] Formula: raneff$score ~ 1 + (1 raneff$rater) Data: raneff REML criterion at convergence: 139.3 Scaled residuals: Min 1Q Median 3Q Max -1.8791-0.4845 0.1257 0.5890 2.1388 Random effects: Groups Name Variance Std.Dev. raneff$rater (Intercept) 2.849 1.688 Residual 7.810 2.795 Number of obs: 28, groups: raneff$rater, 4 Fixed effects: Estimate Std. Error t value (Intercept) 10.8929 0.9955 10.94 raneff.varexp<-as.numeric(varcorr(raneff.vc)) / (as.numeric(varcorr(raneff.vc))+attr(varcorr(ra raneff.varexp [1] 0.267273 5
The variance components are 2.85 for the rater effect and 7.81 for the residual. The ICC is computed by dividing the rater effect by the sum of the two variance components: ICC = σ 2 Rater σ 2 Rater + σ2 Residual = 2.85 2.85 + 7.81 The ICC for this study is.27, which indicates that the raters explained 27% of the variability in the scores in the population. 6