A Variance Estimator for Cohen s Kappa under a Clustered Sampling Design THESIS

Transcription

1 A Variance Estimator for Cohen s Kaa under a Clustered Samling Design THESIS Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Mahmoud Hisham Abdel-Rasoul Graduate Program in Public Health The Ohio State University 0 Master's Examination Committee: Professor Rebecca Andridge, Advisor Professor Soledad Fernandez

2 Coyright by Mahmoud Abdel-Rasoul 0

3 Abstract Assessing agreement between raters is often required to evaluate the consistency of measurements from different raters. Several methods to assess agreement between raters have been reviously develoed for binary matched air data (Donner et al. 996; Donner et al. 000; Durkalski et al. 996; Eliasziw and Donner 99; Obuchowski 998. The most commonly used coefficient of agreement between raters is Cohen s kaa statistic (Donner et al With the excetion of Feder (006 who roosed a variance estimator for the survey weighted Cohen s kaa coefficient, revious research related to Cohen s kaa has focused on study designs where the matched airs are assumed to be indeendent. Studies that have develoed methods to analyze agreement between raters for clustered study designs have focused on other forms of the kaa coefficient or McNemar s test. We roosed a variance estimator for Cohen s kaa under a clustered samling design. Our study was motivated by the Detroit Middle School Asthma Proect (DMSAP which intended to assess the agreement between arents and students on several asthma related measures (Clark 00. A large samle variance estimator for Cohen s kaa under a simle random samling design has been reviously roosed by Fleiss (969. We used the variance estimator for a roortion under a clustered samling scheme roosed by Rao and Scott (99 and the covariance estimator which takes into account the correlation between the ii

4 roortions in addition to the intracluster correlation roosed by Obuchowski (998 along with the delta method to obtain a variance estimator for Cohen s kaa under a clustered samling design. We conducted an extensive simulation study to assess the erformance of the roosed variance estimator under different study conditions. The roosed variance estimator was comared to the variance estimator roosed by Fleiss (969 relative to the emirical variance. The erformance of the roosed variance estimator and the variance estimator under SRS were also comared using a real data examle from The Detroit Middle School Asthma Proect. The roosed variance estimator erformed at least as well as the variance estimator under a simle random samling design for all study conditions exlored. The roosed variance estimator erformed better than the variance estimator roosed by Fleiss (969 under several study conditions, articularly when the correlation between two first raters and the correlation between two second raters was 0., and when values of kaa were greater than 0.. The variance estimator under a simle random samling design underestimates the true variance for Cohen s kaa when a clustered samling design is emloyed and correlation between raters within a cluster is not negligible. iii

5 Dedication This thesis is dedicated to my family. iv

6 Vita May Brookfield High School, Brookfield, Ohio B.S. Microbiology, The Ohio State University M.P.H. Eidemiology, The Ohio State University 0... M.S. Biostatistics, The Ohio State University Graduate Teaching Associate, Deartments of Eidemiology and Biostatistics, The Ohio State University Graduate Research Associate, Center for Biostatistics, The Ohio State University Publications Arnold, L. E., R. A. DiSilvestro, D. Bozzolo, H. Bozzolo, L. Crowl, S. Fernandez, Y. Ramadan, S. Thomson, X. Mo, M. Abdel-Rasoul, and E. Joseh (0. Zinc for attention deficit/hyeractivity disorder: Placebo controlled double-blind ilot trial alone and combined with amhetamine. Journal of Child and Adolescent Psychoharmacology, -9. v

7 Perruzi, P., S. Bergese, A. Viloria, E. Puente, M. Abdel-Rasoul, and E. A. Chiocca (00. A retrosective cohort-matched comarison of conscious sedation versus general anesthesia for suratentorial glioma resection. Journal of Neurosurgery 4, Makary M, E.A. Chiocca, N. Erminy, M. Antor, S. Bergese, M. Abdel-Rasoul, S. Fernandez, and R. Dzwonczyk (0. Clinical outcomes of low-field intraoerative MRI-guided tumor resection neurosurgery. Acceted for ublication in Journal of Magnetic Resonance Imaging. Fields of Study Maor Field: Public Health vi

8 Table of Contents Abstract... ii Dedication... iv Vita...v Publications...v List of Tables... ix List of Figures...x Chater : Introduction... Cohen s Kaa Coefficient... Grou Randomized Trials...5 Literature Review...6 McNemar s Test...7 Kaa Statistics... 0 Chater : Variance of Cohen s Kaa under Simle Random Samling and Clustered Samling Designs... 4 Chater 3: Monte Carlo Simulation Study... 6 Data Generation... 7 vii

9 Simulation Results... 8 Chater 4: Real Data Examle... 5 Chater 5: Discussion... 8 References... 3 viii

10 List of Tables Table. x contingency table for classification [frequency (roortion] of binary scale ratings by raters... Table. Hyothetical examle...4 Table 4. Control grou data from DMSAP [count(roortion]... 5 Table 5. Treatment grou data from DMSAP [count (roortion]... 6 ix

11 List of Figures Figure. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination.... Figure. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0., r 4 = 0.005,. =. = 0.5, and r 3 varies to allow for different κ values. b Fixed r = r = 0.0, r 4 = 0.005,. =. = 0.5, and r 3 varies to allow for different κ values.... Figure 3. Ratio of roosed method and SRS variance estimates to emirical variance as a function of one marginal roortion (.. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ = 0.. b Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ = 0.. c Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ = Figure 4. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a x

12 Fixed r = r = 0.0, r 4 = 0.0,. =. = 0.5, r 3 varies for different κ values, and samle size of 5 for each of 0 clusters. b Fixed r = r = 0.0, r 4 = 0.0,. =. = 0.5, r 3 varies for different κ values, and samle size of 5 for each of 0 clusters Figure 5. Confidence intervals (95% for κ by treatment and variance estimation method for DMSAP examle xi

13 Chater : Introduction Methods to assess the agreement between raters on the resence or absence of a characteristic or outcome are frequently alied in studies from a variety of scientific fields. For examle, a researcher may be interested in determining the extent to which radiologists agree on the detection of a secific morbidity, the extent of agreement between reviewers to fund grants, or the extent of agreement between arents and teachers on behavioral traits of students. The most widely used coefficient of agreement in scientific research is Cohen s kaa coefficient (Donner et al The kaa coefficient is a measure of agreement between raters for binary or categorical outcomes. There are several forms of the kaa coefficient, and the choice between which version of the statistic to use deends on the roblem at hand. Binary data for which a kaa coefficient is estimated can be organized in a table similar to Table below. The cells in the table are frequencies (roortions of the ossible resonses for both raters. Cohen s Kaa Coefficient Rater Rater 0 Total n ( n ( n ( 0 n ( n ( n ( Total n ( n ( N Table. x contingency table for classification [frequency (roortion] of binary scale ratings by raters

14 Cohen introduced two roortions that are used to define an index of agreement between two raters on a binary scale. The observed roortion of agreement o is defined as: o n n N The exected roortion of agreement due to chance e is defined as: e n n N n n Cohen then defined the agreement index as the observed roortion of agreement after the roortion of chance agreement is removed from consideration. The result is then scaled to obtain a value of when agreement is erfect, a value of 0 when agreement is only due to chance, and negative values when observed agreement is lower than the agreement exected by chance. Cohen s kaa coefficient is defined in terms of these observed and exected roortions as: o e ( e ( ( Other commonly used kaa coefficients are the intraclass kaa (κ ICC introduced by Kraemer (979, and a weighted kaa statistic introduced by Cohen (968. Intraclass kaa assumes equal marginal distributions ( = and considers the raters interchangeable. The weighted kaa does not assume equal marginal robabilities and assigns a weight deending on the imortance of the level of rating (Cohen 968. Cohen s kaa is a secial form of the weighed kaa where the two ratings are considered equally imortant and each has a weight of 0.5 (Cohen 960. Other indices of

15 agreement have been roosed. Holley et al. (964 roosed using simly the roortion of overall agreement ( o. Dice (945 roosed an index S d which is the conditional robability that one (of two randomly chosen raters classifies an item as ositive given that the other rater classified the item as ositive, S d..5( 0 Goodman and Kurskall (954 roosed an index λ r which considers the frequency of correct redictions with and without knowledge of the oint ratings and is defined as ( ( r. Goodman and Kurskall s lambda is related to Dice s index since r = S d. Scott (955 also introduced an index of inter-rater agreement ( s which accounts for exected roortion of agreement due to chance. This index is known as Scott s i and defined as s o e e, where o is defined as before but e is defined as ( n n e. It is clear that Scott s i differs from Cohen s kaa in the way the exected roortion of agreement due to chance is estimated. Fleiss (97 roosed a generalization of Scott s i for the multi-rater case. Fleiss kaa is defined as 3 4N ( n n

16 f P P P e e. The mean of the extent to which the raters agree for the i th subect is defined as P Nn( n N k i n i Nn, for n ratings er subect, i=,, N total subects, and =,, k categories of assignment where n i reresents the number of raters assigning the i th subect to the k th category, and the sum of the squared roortions of all assignments to the th category of classification is k P e with N n i Nn i. The kaa coefficients and Scott s i differ from the other mentioned methods that assess agreement because they account for agreement between raters due to chance. For a detailed discussion of the different forms of kaa see Bloch and Kraemer (989. A hyothetical examle matched aired data set is resented in Table to comare the mentioned indices for the case of two raters. Rater Rater 0 Total 36 (0.45 ( ( (0.5 (0.5 3 (0.40 Total 56 (0.7 4 ( Table. Hyothetical examle 4

17 For the above table, the robability of observed agreement ( o is 0.6, Cohen s kaa is estimated to be 0.3, Dice s index (S d is 0.69, Goodman and Kurskal s lambda ( is r 0.38, and Scott s i ( s is 0.. Clearly, the values of the different indices differ greatly. Grou Randomized Trials Grou randomized trials are studies in which grous are assigned to treatments rather than individuals, but the units of observation are the members of those grous. The grous consist of members considered to have some social, geograhic, or other connections rather than being comosed of randomly selected individuals (Murray 998. Analysis of data arising from grou randomized trials imoses an inherent violation of the indeendence assumtion required by traditional statistical techniques. The grous are considered to be indeendent of each other, but the members within the grous may not be indeendent; observations on members within a grou are likely to be correlated. The correlation between the members for each grou need to be accounted for when analyzing grou randomized trial data. Additionally, since the units of assignment are the clusters rather than the subects within clusters, the degrees of freedom should be based on the number of clusters rather than the number of subects. We roose a variance estimator for Cohen s kaa under a clustered samling design. Most recent discussions on kaa coefficients either limit the samling design to a simle random samling (SRS scheme, or focus on the intraclass form of the kaa coefficient (Donner et al. 996; Donner et al This work was motivated by a roblem that arose from the Detroit Middle School Asthma Proect (DMSAP (Clark 00. This study was a grou randomized trial to assess the efficacy of in-school 5

18 rograms to enhance asthma management. One of the secondary outcomes of the study was to assess the agreement between students and arents on a number of measures related to asthma management. In searately conducted interviews, each of the arent and child were asked questions about the child s exeriences managing asthma at school. One question of interest was, Has the child exerienced roblems taking medication at school? The secondary obective of the study intended to determine if the arents are good substitutes for children in answering questions about what haened at school. Literature Review Matched aired dichotomous data can be analyzed in several ways deending on the hyothesis being exlored. McNemar (947 roosed a test of marginal homogeneity to determine if the robability of success is equal for two methods being comared and each method is alied to the same subect. In this case, traditional chisquared tests for contingency tables and Fisher s exact test are not aroriate since the question of interest is not to test the indeendence of two variables for dichotomous data. If a researcher was more interested in degree to which the two methods agree rather than testing equality, descritive statistics were develoed to indicate the degree of agreement between two ratings that adust for chance agreement in the form of kaa statistics. McNemar s test and kaa statistics are related in the tye of data they analyze and share similar roerties; articularly, with regards to the estimation of the marginal robabilities and variance estimates of the marginal robabilities. We first discuss methods develoed to extend McNemar s test to the case where matched airs may not 6

19 be indeendent, and then continue with a discussion of methods develoed to extend the use of kaa coefficients for correlated data. McNemar s Test McNemar s test, as originally introduced, required the assumtion that matched airs were indeendent of each other. However, methods to analyze matched aired data when airs may not be indeendent have been more recently roosed. Durkalski et al (003 roosed an adustment to McNemar s test with a variance estimator based on method of moments that is used for hyothesis testing for clustered matched aired data. The version of McNemar s test which assumes indeendent airs has the null hyothesis of equal marginal roortions, H 0 : =, and is defined as Mc ( ˆ vâr( ˆ ˆ ˆ ( n ( n n n where Mchas a chi-square distribution with degree of freedom under the null hyothesis. Durkalski et al. roosed the following test statistic to test for marginal homogeneity when airs may not be indeendent: V ( K ( n k k n k N k K ( nk nk k N k 7

20 where k=,, K clusters, N k is the total number of subects in cluster k, n k and n k are the number of discordant airs in cluster k, and degree of freedom under H 0 : =. V has a chi-square distribution with Eliasziw and Donner (99 also investigated comaring correlated roortions for clustered data. They roosed an adustment for McNemar s test that involves estimating the correlation among discordant airs within a cluster and using this correlation estimate to adust the test statistic. They adust McNemar s test statistic by an inflation factor (design effect of ( n c ˆ. Their roosed adustment to McNemar s test statistic D is given by D ( n Mc c ˆ where n c S 0 K d S ( S 0 K d is the number of clusters with one or more discordant airs and S is the mean number of discordant airs in the K clusters. S 0 is an adusted mean cluster size given by K ( S S K K S k k ( d S0 S, K ( K S where S k is the total number of discordant airs in cluster k, k=,, K, and the ICC is estimated by d d ˆ MSB MSB ( S 0 MSW. MSW 8

21 where MSB is the between subects mean squared error and MSW is the within subects mean square error. has a chi-square distribution with degree of freedom under H D 0: =. An alternative adustment to McNemar s test was roosed by Obuchowski (998. She estimates ˆ i and vâr( ˆ i using the estimators roosed by Rao and Scott (99. Obuchowski s roosed test statistic is O ( ˆ vâr( ˆ ˆ ˆ where ˆ, is the overall roortion of resonses to treatment i, i=,, i ˆ ˆ vâr( = ˆ ˆ vâr( vâr( côv(, and ˆ ˆ ˆ i K k x ik K k n k vâr( ˆ i K( K ( K k ( x ik n k ( K k n k côv( ˆ, ˆ i K K i' K( K ( ( xik nk i ( xi' k nk i' ( nk k k and ˆ i ˆ i / ; ( ' where we have a random samle of K clusters from a oulation, there are n k units in the k th cluster k, k=,, K, each unit receives i treatments, and x ik denotes the number of units in the k th cluster with a ositive resonse to treatment i, i=,, I. Obuchowski s test statistic has a chi-square distribution with degree of freedom under the null 9

22 hyothesis (H 0 : =. Obuchowski s method is based on samling techniques and does not require the assumtion of constant within cluster correlation. Kaa Statistics Donner et al. (996 roosed rocedures for testing homogeneity of intraclass kaa statistics in the case of two raters and a dichotomous outcome under the common correlation model (Donner et al 996. Two aroaches were examined to test the homogeneity of two or more intraclass kaa statistics which were assumed to be indeendent. One aroach was based on obtaining the ratio of the kaa statistic to its large samle standard error and comaring the ratio to the standard normal distribution. The second aroach alied the goodness-of-fit rocedure to the common correlation model. They used the maximum likelihood estimator of κ as derived by Bloch and Kraemer (989 defined as: ˆ N ˆ n ( ˆ where (n3 n / N, and the large samle variance of ˆ was defined as: ˆ var( ˆ N ( ( ( (, =,. The test statistic based on the first aroach mentioned above for testing the equality of two kaa statistics was given as ˆ ˆ Z V. vâr( ˆ vâr( ˆ 0

23 The test statistic based on the goodness-of-fit aroach testing the equality of two kaa statistics was given as G 3 [ n l l l NPˆ l ( ˆ] NPˆ ( ˆ which has an aroximate distribution with degree of freedom under H 0. Donner et al. (000 develoed rocedures for testing the equality of two deendent intraclass kaa statistics for a dichotomous outcome in the two rater case. They comared the intraclass kaa coefficient for agreement between ratings of two radiologists to the intraclass kaa coefficient between ratings of two non-radiologists from the same samle of 40 subects. Hence, the assumtion of indeendent subects was relaxed in their study. They develoed and comared model based rocedures to test the equality of two deendent intraclass kaa coefficients. The Donner et al. (000 aer extends revious work by Donner et. al (996 by accounting for the covariance between the kaa statistics being tested as derived from a generalized common correlation model. Donner et al. (000 develoed a Wald statistic Z VD to test the equality of the two deendent intraclass kaa statistics as Z VD [vâr( ˆ ˆ vâr( ˆ ˆ côv( ˆ, ˆ ] where the model and terms defining the statistics above are defined in Donner et al. (000, and Z VD is comared to the standard normal distribution.

24 The methods develoed by Donner et al. (996 and Donner et al. (000 cannot be alied to the roblem motivated by the Detroit Middle School Asthma Proect since they focus on the intraclass kaa coefficient which assumes that raters have equal marginal roortions. In the DMSAP, we assume arents and children may reort incidents at school with differing frequencies, so we do not want to assume equal marginal roortions. In addition, although the methods develoed by Donner et al. (000 do take the covariance between two deendent kaa statistics into account, this is not the same as the correlation between subects within a cluster arising from the grou randomized trial study design. Feder (006 roosed a variance estimator for survey-weighted Cohen s kaa. He resented a Taylor linearization derivation for the variance estimator of Cohen s kaa to analyze the agreement between resonses Y and Y of individuals at two time oints, T and T for comlex survey data. Feder s study was motivated by The National Survey on Drug Use and Health (NSDUH which uses a multistage area robability samling design. Feder shows that ˆ U where U = ( + /[( (( + ( + ]. Feder defined a variable x i for each individual i in a oulation U as x F I G I H I, i [ Y 0, Y 0] [ Y 0, Y ] [ Y, Y 0] where U F, G U, and H U, and I is an indicator variable. The oulation mean of x for oulation U with size N was defined as X N x N( F G H i U i.

25 Then, assuming F, G, and H are oulation values, the estimate of the oulation mean of x becomes X ˆ N w x i i F ˆ G ˆ ˆ H, where w i are the survey weights. Feder then estimated the variance of ˆ by calculating,, and then calculating F, G, and H. He calculated x i for every erson in the samle and calculated the variance of Xˆ using a standard software ackage that allows for comlex study designs. Feder shows that var( ˆ = var( Xˆ. Through a simulation study, Feder further showed that the large samle variance under SRS given by Fleiss et al. (969 results in a negatively biased variance estimate for the comlex study design used in NSDUH. The methods mentioned, however, focus on the intraclass kaa statistic rather than Cohen s kaa statistic. These methods assume equal marginal roortions for different raters and do not allow for different marginal roortions within and across clusters. With the excetion of Feder (006, methods that do investigate Cohen s kaa statistic do not consider the clustered samling design. 3

26 Chater : Variance of Cohen s Kaa under Simle Random Samling and Clustered Samling Designs (969 as The large samle variance of Cohen s kaa coefficient is defined by Fleiss vâr( ˆ ( o o ( [ N ( [ i ( i o e ( e 4 ( i [ o 4 o e ( ] e ] i 3 e ( ii i i ] This estimator assumes indeendent airs. In this section we derive an alternative variance estimator that allows correlation between airs. Rao and Scott (99 roose a method for comaring indeendent grous of clustered binary data and a variance estimator for a roortion under a clustered samling scheme. We slightly modify Rao and Scott s notation. Assume we have a random samle of m clusters from a oulation, with n units in the th cluster, =,.m. Each unit is rated by raters. Let x i denote the number of ositive ratings in the th cluster by rater I, I=,. Let i denote the robability that a randomly selected unit from the oulation will have a ositive rating by rater i. Then, E(X i = n i. where ˆ i is an estimator of i, ˆ i m x i 4 m n,

27 which is ust the overall roortion of ositive ratings by rater i. Rao and Scott give an estimator of the variance of i, taking clustering into consideration. The estimator of the variance for a large number of clusters is vâr( ˆ i m( m ( m ( x i n ˆ i ( m n Obuchowski roosed an estimator of the covariance between i and i as côv( ˆ, ˆ i m m ˆ ˆ i' m( m ( ( xi n i ( xi' n i' ( n We use Rao and Scott s estimator to obtain estimates for Var( ˆ, Var( ˆ, and Var( ˆ. We use Obuchowski s estimator to obtain estimates for Cov( ˆ, ˆ, Cov( ˆ, ˆ, and Cov( ˆ, ˆ. We use these estimators along with the delta method to obtain a variance estimator for ˆ. Taking ˆ ˆ ( ˆ ( ˆ ˆ ˆ ˆ ( ˆ, straightforward alication of the delta method yields Var( ˆ = A (Var( ˆ + B (Var( ˆ + C (Var( ˆ + (AB(Cov( ˆ, ˆ + AC(Cov( ˆ, ˆ + BC(Cov( ˆ, ˆ where A= ˆ, B= ˆ ˆ ˆ, and C= ˆ ˆ. Plugging in the estimators secified above gives the large samle variance estimate for ˆ in the case of clustered airs. For equal cluster sizes, the roosed variance estimator is identical to the survey weighted variance estimator roosed by Feder (006. 5

28 Chater 3: Monte Carlo Simulation Study We conducted a Monte Carlo simulation study to assess the erformance of our roosed variance estimator for Cohen s kaa statistic for multilevel clustered binary data. We assessed the erformance of the variance estimator by comaring it to both the variance estimator under a simle random samling design and the emirical variance estimate. Each exeriment was reeated 000 times. We considered several correlation structures and varied the marginal resonse robabilities, Cohen s kaa (oint resonse robability, the number of clusters, and the number of units er cluster. Due to the multile levels of clusters, there were four different correlation coefficients to consider. The correlation between two first raters in the same cluster (r, correlation between two second raters in the same cluster (r, correlation between the first rater and the second rater on the same subect (r 3, and correlation between a first rater and a second rater on different subects in the same cluster (r 4. Among the correlation coefficients to be considered, r 3 is likely to be the largest as it involves two ratings on the same subect. The smallest of the correlation coefficients is likely to be r 4 since it involves ratings by different raters on different subects. The ratings on different subects by the same rater (r and r for the different raters resectively are exected to be larger than r 4 but not as large as r 3 since r and r involve the same rater resectively but different subects. 6

29 Correlation Structures: r 4 < r = r in {0, 0.0, 0.} < r 3 r 4 < r < r (where r / r in {.0,.5,.5} < r 3 Marginal Probabilities: = = 0.5 with κ in {0., 0., 0.3, 0.4, 0.5, 0.6, 0.7} 0.5 = in { 0., 0.4, 0.5, 0.6, 0.8} with κ in {0., 0., 0.3} Samle Sizes: n=50 with m=0 n=5 with m=0 3 n=5 with m=0 Data Generation A latent variable mixed model was used to generate the outcome for each rater. The data were generated for each rater searately using the same strategy. The outcomes were initially obtained as Z scores from the latent variable model below and secifying the different model arameters and random effects for the different scenarios for the correlations detailed in the simulation strategy. The simulated Z score values from the latent variable model which were greater than zero were set to, and the simulated values which were less than zero were set to zero to obtain correlated binary data based on the different simulation scenarios secified in the simulation strategy. The latent variable model for rater ( = (, on subect i (i = (,, n k in cluster k (k = (,, K used to generate the data was defined as: Y i:k = µ + G k + M i:k + RG k + i: k ; 7

30 Where µ (=, is the mean for the marginal robability for rater =,, as secified by the inverse cumulative normal distribution function given a secific marginal robability ( or for each rater resectively. The random error is secified by i: k and is distributed N(0,. The random effect for cluster G k also has a normal distribution with a mean of zero and a variance of σ k. The random effect for rater within cluster RG k and the random effect for subect within cluster M i:k are also normally distributed, each with a mean of zero and variances σ rk, and σ m:k resectively. The random effects are assumed to be indeendent of each other. The sum of the variance comonents was fixed at 00 for each trial. A ratio of the roosed variance estimator to the emirical variance was comared to the ratio of the variance under SRS to the emirical variance. Confidence intervals (95% for the ratio of the variance estimator to the emirical variance were obtained using bootstra methods. Bootstraing methods allow us to obtain a variance estimate for the variance estimators by taking advantage of the emirical distribution of the observed data since we do not have the arametric distribution of the ratios of variance estimators to the emirical variance. It was exected that the 95% confidence interval for the ratio of the roosed variance estimator to the emirical variance would contain for the different simulation scenarios. Simulation Results Figure a dislays the results of the simulations where we fix r = r = 0., r 4 = 0.0,. =. = 0.5, and r 3 varies to allow for κ values from 0. to 0.7 by 0.. The results indicate that the variance estimator for kaa under simle random samling 8

31 underestimates the true variance for values of κ > 0.. The 95% confidence intervals for the ratio of the variance estimator under SRS to the emirical variance have an uer bound < for all values of κ > 0.. The variance estimator that takes clustering into account erforms well for all values of κ. All of the 95% confidence intervals for the ratio of the variance estimator that accounts for clustering to the emirical variance include. Figures b and c dislay results from simulations under the same conditions as Figure excet r = r = 0.0 for figure b and r = r = 0.0 for figure c. The variance estimator under SRS and the variance estimator that accounts for clustering erform well for all values of κ in figure b. Both variance estimators erform equally well in the case where r = r = 0.0 as dislayed in figure c. Figure a dislay results from simulations under the same conditions as Figure a excet r 4 =0.05. The variance estimator under SRS underestimates the variance for all values of κ under these circumstances but not to the extent it does when r 4 =0. The variance estimator that accounts for clustering, however, rovides estimates similar to the emirical variance as the confidence intervals for the ratio with the emirical variance includes for all values of κ. Figure b dislays results for simulation conditions similar to those for figure a excet r = r = 0.0. The variance estimates using both estimators are similar to the emirical variance estimates as all 95% confidence intervals dislayed include. As r 4 increases, the variance estimator under SRS erforms slightly better than it does for lower values of r 4. Figures 3a, 3b, and 3c dislay results where we fix r = r = 0., r 4 = 0.0, and. = 0.5. We fixed values of κ to be 0., 0., and 0.3 for figures 3a, 3b, and 3c resectively. 9

32 One of the marginal robabilities (. varies similarly in all 3 figures. The variance estimator that accounts for clustering erforms ust as well when the marginal robabilities are not equal as it did when the marginal robabilities were both fixed to 0.5. The variance estimator that assumes SRS erforms worse as κ increases, which can be deduced when comaring Figures 3a, 3b, and 3c. It also aears that the naïve variance erforms worse when. is further from.. Figures 4a and 4b dislay results from simulations under the same conditions as Figure a, excet the samle size for figure 4a was set to n=5 and m=0, and n=5 and m=0 for figure 4b. The variance estimator that accounts for the clustered samling design erforms ust as well in figures 4a and 4b as in figure a. The variance estimator that assumes SRS erforms worse for smaller samle sizes and smaller clusters. 0

33 Figure. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0., r 4 = 0.0,. =. = 0.5, and r 3 varies for different κ values. b Fixed r = r = 0.0, r 4 = 0.0,. =. = 0.5, and r 3 varies for different κ values. c Fixed r = r = 0.00, r 4 = 0.0,. =. = 0.5, and r 3 varies for different κ values.

34 Figure. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0., r 4 = 0.005,. =. = 0.5, and r 3 varies to allow for different κ values. b Fixed r = r = 0.0, r 4 = 0.005,. =. = 0.5, and r 3 varies to allow for different κ values.

35 Figure 3. Ratio of roosed method and SRS variance estimates to emirical variance as a function of one marginal roortion (.. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ = 0.. b Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ = 0.. c Fixed r = r = 0., r 4 = 0.00,. = 0.5, and κ =

36 Figure 4. Ratio of roosed method and SRS variance estimates to emirical variance as a function of kaa (κ. Lines are bootstraed 95% intervals ( th ercentiles to show simulation error. Results from 000 relicates for each arameter combination. a Fixed r = r = 0.0, r 4 = 0.0,. =. = 0.5, r 3 varies for different κ values, and samle size of 5 for each of 0 clusters. b Fixed r = r = 0.0, r 4 = 0.0,. =. = 0.5, r 3 varies for different κ values, and samle size of 5 for each of 0 clusters. 4

37 Chater 4: Real Data Examle We used art of the data from the Detroit Middle School Asthma Proect which was the motivation for this study to analyze the erformance of the new variance estimator for Cohen s kaa comared to the variance under an SRS design. One of the secondary outcomes of the study intended to assess the agreement between students and arents. A child and his/her arent were searately asked Has the child/have you exerienced roblems with too many [school] absences due to asthma? The goal was to determine if the arents are good substitutes for children in answering questions about what haened at school. Data for students and arents from one of the treatment grou (received a school based intervention and the control grou (no intervention were comared with resect to interrater reliability. There were 47 comlete airs from 6 clusters (4 treatment, control. The average cluster size was 8 airs with a minimum cluster size of 6 airs and a maximum of 30 airs. The data are resented in Tables 3 and 4 for the control grou and treatment grous resectively. Parent Child 0 Total 5 ( ( (0.5 0 ( ( (0.75 Total 6 (0. ( Table 3. Control grou data from DMSAP [count(roortion] 5

38 Parent Child 0 Total 9 ( ( ( ( ( (0.76 Total 7 (0. 06 ( Table 4. Treatment grou data from DMSAP [count (roortion] The kaa coefficient for the control grou was estimated to be 0.3 with a naïve standard error of and a standard error of when clustering was taken into account. The intervention grou had a kaa coefficient of 0.08 with a naïve standard error of and an adusted standard error of The intra-class correlation (r between students was estimated to be 0.08, and between arents (r was estimated to be The naïve standard errors were under estimated by % and 7% for the control and treatment grous resectively. The estimates for the kaa coefficient and the naïve and cluster adusted confidence intervals are dislayed in Figure 5. 6

39 Figure 5. Confidence intervals (95% for κ by treatment and variance estimation method for DMSAP examle. 7

40 Chater 5: Discussion Assessing agreement between raters is often required to evaluate the consistency of measurements from different raters. Several methods to assess agreement between raters have been reviously develoed for binary matched air data (Donner et al. 996; Donner et al. 000; Durkalski et al. 996; Eliasziw and Donner 99; Obuchowski 998. The most commonly used coefficient of agreement between raters is Cohen s kaa statistic (Donner et al With the excetion of Feder (006 who roosed a variance estimator for the survey weighted Cohen s kaa coefficient, revious research related to Cohen s kaa has focused on study designs where the matched airs are assumed to be indeendent. Studies that have develoed methods to analyze agreement between raters for clustered study designs have focused on other forms of the kaa coefficient or McNemar s test. We roosed a variance estimator for Cohen s kaa under a clustered samling design. Our study was motivated by the Detroit Middle School Asthma Proect (DMSAP which intended to assess the agreement between arents and students on several asthma related measures. A large samle variance estimator for Cohen s kaa under a simle random samling design has been reviously roosed by Fleiss (969. We used the variance estimator for a roortion under a clustered samling scheme roosed by Rao and Scott (99 and the covariance estimator which takes into account the correlation between the 8

41 roortions in addition to the intracluster correlation roosed by Obuchowski (998 along with the delta method to obtain a variance estimator for Cohen s kaa under a clustered samling design. We conducted an extensive simulation study to assess the erformance of the roosed variance estimator under different study conditions. The roosed variance estimator was comared to the variance estimator roosed by Fleiss (969 relative to the emirical variance. The erformance of the roosed variance estimator and the variance estimator under SRS were also comared using a real data examle from The Detroit Middle School Asthma Proect. The roosed variance estimator erformed at least as well as the variance estimator under a simle random samling design for all study conditions exlored. The roosed variance estimator erformed better than the variance estimator roosed by Fleiss (969 under several study conditions, articularly when the correlation between two first raters and the correlation between two second raters was 0., and when values of kaa were greater than 0.. The variance estimator under a simle random samling design underestimates the true variance for Cohen s kaa when a clustered samling design is emloyed and correlation between raters within a cluster is not negligible. One limitation of the roosed method is that it does not allow for incororation of covariates when assessing interrater reliability. Previous research has been conducted using modeling based methodology to incororate covariates in the analysis of interrater agreement. The maximum likelihood estimator of the intraclass kaa coefficient when the outcome deends on covariates has been derived by Shoukri et al. (995 and Shoukri 9

42 and Mian (996. Lisitz et al. (00, 003 roosed models that account for categorical and continuous covariates ermitting the comarison of indeendent agreement indexes between two raters. Lisitz et al. (00 roosed to use two logistic regressions and a linear regression for binary variables to model Cohen's kaa coefficient as a function of covariates. Lisitz et al. (003 roosed a modification to (Lisitz et al., 00 by estimating agreement robability as a function of covariates using two stage logistic regressions. Barnhart and Williamson (00 develoed methods for comaring several deendent agreement indexes between two raters using weighted least squares. Klar et al. (000 roosed the use of generalized estimating equations (GEE to model the intraclass kaa coefficient as a function of covariates. Covariates associated with the marginal robabilities of classification by each rater are identified by using generalized estimating equations (GEE with a logistic link function. Covariates associated with the intraclass kaa coefficient are then identified using a second set of generalized estimating equations (GEE with a link function based on Fisher's Z transformation. Although various modeling techniques that incororate covariates in the assessment of interrater reliability have been develoed, most of them are either only alicable to the intraclass kaa coefficient, or do not take the clustered samling design into consideration if they aly to Cohen s kaa. Future research in this area is required to incororate covariates in the assessment of Cohen s kaa coefficient under a clustered samling design. 30

43 References Barnhart, H. X. and J. M. Williamson (00. Weighted least-squares aroach for comaring correlated kaa. Biometrics 58, Bloch, D. A. and H. C. Kraemer (989. x kaa coefficients: measures of agreement or association. Biometrics 45, Clark, N. M., S. Shah, J. A. Dodge, L. J. Thomas, R. R. Andridge, D. Awad and R. J. A. Little (00. An evaluation of asthma interventions for reteen students. Journal of School Health 80, Cohen J. (960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 0, Cohen, J. (968. Weighted kaa: nominal scale agreement with rovision for scaled disagreement or artial credit. Psychological Bulletin 70, 3-0. Dice, L. R. (945. Measures of the amount of ecologic association between secies. Ecology 6, Donner, A., M. Eliasziw, and N. Klar (996. Testing the homogeneity of kaa statistics. Biometrics 5, Donner, A., M. M. Shoukri, N. Klar, and E. Bartfay (000. Testing the equality of two deendent kaa statistics. Statistics in Medicine 9, Durkalski, V. L., Y. Y. Palesch, S. R. Lisitz and P. F. Rust (003. Analysis of clustered matched-air data. Statistics in Medicine, Eliasziw M. and A. Donner (99. Alication of the McNemar test to non-indeendent matched air data. Statistics in Medicine 0, Feder, M. (006. Variance estimation of the survey-weighted kaa measure of agreement. American Statistical Association Proceedings of the Survey Research Methods Section, Fleiss, J.L., J. Cohen and B. S. Everitt (969. Large samle standard errors of kaa and weighted kaa. Psychological Bulletin 7, Fleiss, J. L. (97. Measuring nominal scale agreement among many raters. Psychological Bulletin 76,

44 Goodman, L. A. and W. H. Kruskal (954, 959, 963, 97. Measures of association for cross classifications I, II, III, IV. Journal of the American Statistical Association 49, 54, 58, 67, , 3-63, , Holley, J. W. and J. P. Guilford (964. A note on the G index of agreement. Educational and Psychological Measurement 3, Klar, N., S. R. Lisitz, and J. G. Ibrahim (000. An estimating equations aroach for modelling kaa. Biometrical Journal 4, Kraemer, H. C. (979. Ramifications of a oulation model for reliability. Psychometrika 44, as a coefficient of Lisitz, S. R., J. Williamson, N. Klar, J. Ibrahim, and M. Parzen (00. A simle method for estimating a regression model for between a air of raters. Journal of the Royal Statistical Society, Series A 64, Lisitz, S. R., M. Parzen, G. M. Fitzmaurice, and N. Klar (003. A two-stage logistic regression model for analyzing inter-rater agreement. Psychometrika 68, McNemar, Q. (947. Note on the samling error of the difference between correlated roortions or ercentages. Psychometrika, Murray DM. (998. Design and Analysis of Grou-Randomized Trials. New York: Oxford University Press. Obuchowski, N. (998. On the comarison of correlated roortions for clustered data. Statistics in Medicine 7, Rao, J. N. K. and A. J. Scott (99. A simle method for the analysis of clustered binary data. Biometrics 48, Scott, W. A. (955. Reliability of content analysis: the case of nominal scale coding. Public Oinion Quarterly 9, Shoukri, M. M., S. W. Martin, and I.U.H. Mian (995. Maximum Likelihood estimation of the kaa coefficient from models of matched binary resonses. Statistics in Medicine 4, Shoukri, M. M. and I. U. H. Mian (996. Maximum Likelihood estimation of the kaa coefficient from bivariate logistic regression. Statistics in Medicine 5,