Florida State University Libraries

Transcription

1 Florida State University Libraries Electronic Theses, Treatises and Dissertations The Graduate School 0 Effect-Size Index for Evaluation of Model- Data Fit in Structural Equation Modeling Mengyao Cui Follow this and additional works at the FSU Digital Library. For more information, please contact lib-ir@fsu.edu

2 THE FLORIDA STATE UNIVERSITY COLLEGE OF EDUCATION EFFECT-SIZE INDEX FOR EVALUATION OF MODEL-DATA FIT IN STRUCTURAL EQUATION MODELING By MENGYAO CUI A Thesis submitted to the Educational Psychology and Learning Systems in partial fulfillment of the requirements for the degree of Master of Science Degree Awarded: Spring Semester, 0

3 Mengyao Cui defended this thesis on March 8 th, 0. The members of the supervisory committee were: Yanyun Yang Professor Directing Thesis Betsy Becker Committee Member Russell Almond Committee Member The Graduate School has verified and approved the above-named committee members, and certifies that the thesis has been approved in accordance with university requirements. ii

4 I dedicate this to my grandparents, Hao Cui and Lanrong Han iii

5 ACKNOWLEDGEMENTS I want to begin my acknowledgements by thanking my academic adviser Dr. Yanyun Yang. She strongly recommended me to write this thesis. Thanks to her advice, I learned a lot during the process of writing this thesis. It was an important and helpful experience in my academic life. She has given me useful suggestions when I felt confused or lost in the research. This thesis could not have been completed without her help. She has also told me to put my feet on the ground and try my best to become a good researcher. It s such a pleasure that she is my adviser. I would like to say thank you to my committee members, Dr. Betsy Becker and Dr. Russell Almond. I really appreciated their valuable comments and advice for my thesis. They pointed out the issues and problems which I didn t even realize. I also want to express my gratitude to my family. I am grateful to my grandma Lanrong Han, parents, Baojian Cui, Tongkun Kang, uncle and aunt Baozhi Cui and Jing Wang. I would not have the chance to study in this master s program without their selfless love and support. Especially, I wish to extend my sincere love to my grandpa, Hao Cui, who would be so cheerful to see his elder granddaughter finish her master s thesis. iv

6 TABLE OF CONTENTS List of Tables... vi List of Figures... vii Abstract... viii. CHAPTER ONE: INTRODUCTION.... CHAPTER TWO: LITERATURE REVIEW Null Hypothesis and Alternative Hypothesis...3. Test Statistic to Assess Model-data Fit Central and Non-central Chi-square Distributions Type I, Type II Errors, and Power Power Analysis for Individual Parameters Power Analysis for Overall Model-data Fit....7 Effect Size Using RMSEA Effect Size Using Other Indices Summary CHAPTER THREE: METHODOLOGY Fit Function as Effect Size Index in SEM Simulation Study Study Design Data Analysis CHAPTER FOUR: RESULTS Performance of Fit Function in the Population Performance of Fit Function at the Sample Level Means, Standard Deviations, and Confidence Intervals Distributional Characteristics of Fit Functions Relation between Effect Size and Power CHAPTER FIVE: DISCUSSION... 5 APPENDIX A APPENDIX B REFERENCES BIOGRAPHICAL SKETCH... 6 v

7 LIST OF TABLES Table 4. Fit Function in the Population...3 Table 4. Number of Replications with Non-convergence and Improper Solutions for Each Condition...3 Table 4.3 Mean (Standard Deviation) and Confidence Intervals of Sample Fit Functions for the Correctly Specified Models...34 Table 4.4 Mean (Standard Deviation) and Confidence Intervals of Sample Fit Functions for the Mis-specified Models...36 Table 4.5 Rejection Rate under each Condition...50 vi

8 LIST OF FIGURES Figure. CFA Models for Demonstration...5 Figure. Central and Non-central Chi-square Distributions, Type I and Type II errors, and Power...9 Figure.3 Regression Model...0 Figure.4 Illustration of Power under Close-fit Test Hypothesis...5 Figure.5 Illustration of Power under the Not-close-fit Test Hypothesis...6 Figure.6 Plot of Power with Pairs of RMSEA holding the d=5, sample size=00, alpha= Figure 3. Factor Structures for Data Generation...4 Figure 4. Distribution of the Sample Fit Functions for Correctly Specified Models c-3c...38 Figure 4. Distribution of the Sample Fit Functions for Correctly Specified Models c-3c...39 Figure 4.3 Distribution of the Sample Fit Functions for Correctly Specified Models c-3c...40 Figure 4.4 Distribution of the Sample Fit Functions for Correctly Specified Models 4c-6c...4 Figure 4.5 Distribution of the Sample Fit Functions for Correctly Specified Models 4c-6c...4 Figure 4.6 Distribution of the Sample Fit Functions for Mis-specified Models m-3m...43 Figure 4.7 Distribution of the Sample Fit Functions for Mis-specified Models m-3m...44 Figure 4.8 Distribution of the Sample Fit Functions for Mis-specified Models m-3m...45 Figure 4.9 Distribution of the Sample Fit Functions for Mis-specified Models 4m-6m...46 Figure 4.0 Distribution of the Sample Fit Functions for Mis-specified Models 4m-6m...47 vii

9 ABSTRACT This study focused on developing and examining an effect-size index for evaluation of model-data fit in structural equation modeling. Based on MacCallum and his colleagues work (996, 006), the discrepancy function under maximum likelihood estimation was defined as an effect-size index. The formulas for computing the expected value and variance of this index were derived for both correctly specified and mis-specified models. A simulation study was conducted to examine the performance and distributional characteristics of the index under various degrees of freedom, sample sizes, and degrees of model misspecification. The results showed that the fit function in the sample is a function of sample sizes, degrees of freedom, and degree of model misspecification. The observed sample means tended to be biased when the sample size was small. The distribution of the fit function was very similar to the chi-square distribution. It was also demonstrated that power of detecting mis-specified models was a function of the sample sizes, degrees of freedom, and degree of model misspecification. In addition, it was suggested that meta-analysis across studies can be conducted using the fit function as an effect-size index for certain models. viii

10 CHAPTER ONE INTRODUCTION Structural equation modeling (SEM) has become an increasingly popular statistical method in various disciplines. SEM is referred to as a family of related statistical techniques. It provides a flexible framework for testing hypothesized interrelationships among observed and/or latent variables. Path analysis (PA) and confirmatory factor analysis (CFA) are two types of SEM that have been widely used in applied research. A path model examines hypothesized relations (e.g., direct effects) among a set of observed variables. It allows for inclusion of multiple predictors and multiple outcomes and estimates all model parameters in one analysis. A CFA model (a.k.a. measurement model) is concerned with the relationship between measured variables and latent factors, assuming measured variables are reflective indicators of latent factors. In a CFA model, researchers tend to have a priori hypotheses about the factor structure underlying a set of measured indicators. In other words, the number of factors and their corresponding indicators are clearly specified (Kline, 00). Many methodological studies in SEM, for example, those establishing cutoffs for fit indices (e.g., Hu & Bentler, 999) and examining the consequences of violations of assumptions in analyses (e.g., Flora & Curran, 004), have been conducted within CFA framework. One important aspect of SEM analysis is to examine whether the hypothesized relationships among variables are supported by the data, which involves assessment of goodness of model-data fit. The discrepancy between a model and data is quantified by a fit function (see details later in the next chapter). Numerous studies have focused on evaluation of model-data fit (e.g., Bentler & Bonett, 980; Browne & Cudeck, 993; Hu & Bentler, 999). In general, two approaches have been considered: the conventional likelihood ratio test (i.e., test) and descriptive measures of the model-data fit (MacCallum, Browne, & Sugawara, 996). Some of the descriptive measures of model-data fit are a function of the model chi-square, such as root mean square error approximation (RMSEA), but others are not (e.g., Root Mean Squared Residual, SRMR). The null hypothesis of the test is that the specified model holds exactly in the population. Failure to reject the null hypothesis (e.g., a non-significant test) suggests that the specified model fits the data well, which is preferred for most SEM applications. Generally, it

11 is desirable to have large sample sizes in SEM analyses. When the sample size is not large enough, it is likely that a non-significant test is not a result of correct model specification, but a consequence of lack of power for rejecting a mis-specified model (Kim, 005). In other words, the hypothesized model might still be mis-specified even when the chi-square test turns out to be non-significant. For this reason, Type II error rate and power are considered as important issues in SEM. Type II error rate (β) is the probability of failing to reject a null hypothesis when the alternative hypothesis is true; power is the probability of rejecting the null hypothesis when the alternative hypothesis is true (Everitt & Skrondal, 00). Specifically, power = - β. In SEM, power analysis can be conducted for either specific parameter(s) or the model as a whole (Hancock & Mueller, 006; Kim, 005; MacCallum, Browne, & Cai, 006; MacCallum & Hong, 997; MacCallum et al., 996; Satorra & Saris, 985). Power is a function of alpha level, sample size, degrees of freedom, and effect size. The former three factors that impact power are defined straightforwardly. Effect sizes have been defined in many ways, such as standardized mean differences (i.e., Cohen s d), correlations, standardized regression slopes, odds ratios, or relative risks in traditional statistical procedures, depending upon the nature of the data and research questions of interest. However, the definition of effect size is not clear in SEM. Previous studies (Kim, 005; MacCallum et al., 996; MacCallum, Browne, & Cai, 006; MacCallum & Hong, 997) have developed several frameworks for power analysis in SEM. But none of them focused on the definition and distributional characteristics of the effect-size index. My interest is to define an effect size for evaluation of overall model-data fit in structural equation models. If an effect size is clearly defined, power analysis can be carried out within a consistent framework. In the current literature, both the chi-square test and RMSEA have been used to conduct power analyses and sample size determinations. As I describe in the next chapter, both approaches come with limitations. In addition, once an effect size is defined, metaanalyses can be conducted to synthesize model results across studies. Built on MacCallum and his colleagues work (996, 006), the purpose of the current study is to define an effect size using the fit function, to derive its expected value and variance, and to examine its distributional properties for structural equation models. A simulation study is designed to investigate the influence of sample size, model complexity (i.e., degrees of freedom), and effect size on power in both population and samples.

12 CHAPTER TWO LITERATURE REVIEW Within SEM, researchers have developed power analysis methods for specific parameters in a model (Satorra & Saris, 985; Satorra, Saris, & Pijper, 99) and for the model as a whole (Kim, 005; MacCallum, Browne, & Cai, 006; MacCallum & Hong, 997; MacCallum et al., 996). In this chapter, I first introduce the key concepts associated with power analysis in SEM. I then review the existing methods for conducting power analysis for SEM. Finally, I discuss the advantages and disadvantages of each method. The basic idea of power analysis for SEM is the same as that for other statistical techniques, such as t test, multiple regression, and analysis of variance. Several key concepts are essential to conduct a power analysis. They are null hypothesis and alternative hypothesis, test statistic, central and non-central distributions, non-centrality parameter, Type I and Type II error rates, and effect size. All concepts have been well-studied for SEM except effect size, which is the focus of this study. Null Hypothesis and Alternative Hypothesis To conduct power analysis, null and alternative hypotheses, the test statistic to assess the null hypothesis, and central and non-central distributions should be clearly defined (Hancock, 006). SEM analyzes the covariance/correlation matrix of observed variables (and the mean structure if researchers are interested in examining means/intercepts of variables). In this study, I focus only on the covariance structure. Let s denote the population covariance matrix among p observed variables as, with variances in the diagonal and covariances in the off-diagonal. The dimension of this matrix is p p. Given the hypothesized relationships among the variables, can be expressed as a function of model parameters, ( ), where is the parameter vector including q parameters (see Bollen, 989). For example in a bivariate regression model, y x e, the observed covariance matrix is: 0 var( y) cov( xy) var( x) 3

13 Based on this regression model, the variance of y is decomposed into two components, one is explained by x and the other is not explained by x. This decomposition leads to an implied (or reproduced) covariance matrix. Specifically, ( ) var( x) var( e) var( x) var( x) where β, var(x), and var(e) are model parameters. One of the aspects of SEM analysis is to examine whether the reproduced covariance matrix, ( ), resulting from such a decomposition is consistent with the observed covariance matrix in the population. If it is, the model and data fit to each other. A multiple regression model always shows perfect model-data fit because it has zero degrees of freedom. However, most models are over-identified (i.e., have degrees of freedom greater than zero) and tend to show a certain degree of misfit in the population. In practice, a sample covariance matrix is obtained and fit to a model. Thus model-data misfit may come from both sampling fluctuations and model misspecification. A major purpose of SEM analysis is to evaluate the degree of misfit and determine whether the misfit is due only to the sampling fluctuations, or due to model misspecification. The null hypothesis in SEM is that the reproduced covariance matrix is identical to the covariance matrix in the population: H ( ). 0 : 0 If the null hypothesis is true, the model perfectly explains the interrelationships among the observed variables. The alternative hypothesis is that these two matrices are not the same in the population: H ( ). : 0 The above alternative hypothesis is expressed as a general case indicating that there is a discrepancy between reproduced and population covariance matrix. In power analysis, the probability of rejecting a specific alternative hypothesis when it is true is of interest. Therefore, a specific alternative model should be determined. Consequently, ) corresponds to a specific model that generates a different covariance matrix from. Theoretically, there are many alternative models. Below I use a CFA model to illustrate null and alternative hypotheses in SEM. ( 4

14 F F X X X3 X4 X5 X6 a. Correctly specified model (-factor model with cross-loadings) F F X X X3 X4 X5 X6 b. Mis-specified -factor model F X X X3 X4 X5 X6 c. Mis-specified -factor model Figure. CFA models for demonstration Suppose we have a true population CFA model underlying six measured variables as depicted in Figure.a, where the factor loadings are either.8 (indicated by solid lines) or. (indicated by dashed lines), the correlation between the two factors is.5, and the variance of 5

15 measurement error for each measured variable is one minus the sum of the square of the factor loadings. The population covariance matrix for this true model is: If the hypothesized model is consistent with the one depicted in Figure.a, then the reproduced covariance matrix ) is identical to. That is, the H 0 is true. Now if a researcher ( 0 wants to test a hypothesis that there are two factors underlying these six measured variables without the two cross-loadings. The model is shown in Figure.b. Fitting this model to the population covariance matrix would yield a reproduced covariance matrix as shown below: ( factor ) On the other hand, if a researcher wants to test a hypothesis that there is only one factor underlying these six measured variables. The model is shown in Figure.c. Fitting this model to the population covariance matrix would yield a reproduced covariance matrix as shown below:.5.63 ( factor ) As can be demonstrated through this example, the alternative hypothesis is model specific, although the general expression of H ( ) is commonly used to indicate an : 0 alternative hypothesis. As I will discuss in the next section, these two alternative models 6

16 correspond to different model degrees of freedom and non-centrality parameters, resulting in different power. Test Statistic to Assess Model-data Fit In SEM, the parameters are estimated so that the difference between the observed covariance matrix and the reproduced covariance matrix is minimized (Bollen, 989). This difference is quantified by the discrepancy function, also called the fit function, F (, ( )). The fit function is defined differently with different estimation methods. For the most popular estimation method, maximum likelihood (ML), the minimization of the fit function is defined as: F tr ) ln ( ) ln ( ( ) p, () where p is the number of observed variables in the model, ln denotes the natural logarithm of the determinant of a matrix, and tr( ) stands for the trace of a matrix. In the population, the fit function F equals zero if and only if ( ), suggesting that the model is correctly specified. If (), F is greater than zero, suggesting that there is some misfit between model and data. The greater the F is, the more seriously the model is mis-specified. Therefore, the fit function F reflects the degree to which model and data do not match in the population. In practice, the population covariance matrix is unknown. A covariance matrix S, presumably from a representative sample, is then used to estimate the model parameters. As such, a positive fit function between S and ( ) comes from two sources: model misspecification and sampling error. If the null hypothesis is true (i.e., the model is correctly specified), the sample size (n) is sufficiently large, and the assumptions underlying the estimation method are satisfied, the T statistic (shown below) will approximately follow a chi-square distribution with mean of df where df is the model degrees of freedom and Fˆ is the fit function based on the sample: T ( n ) Fˆ. () The degrees of freedom is determined as the difference between the number of model observations (i.e., the unique number of variances and covariances among observed variables) and model parameters, which is df p( p ) q, (3) 7

17 where p is the number of observed variables in the model, and q is the number of model parameters. Most SEM softwares print T statistic directly as chi-square and name the value as the chi-square test. Central and Non-central Chi-square Distributions In SEM, chi-square test is used to test for a difference between observed and reproduced covariance matrices. The central tendency and shape of a chi-square distribution are determined by the model degrees of freedom (df) and non-centrality parameter (Zelen & Severo, 965). When the H 0 is true (i.e., the model is correctly specified), the difference between observed and reproduced covariance matrices is due to sampling error. Then the T statistic follows an asymptotically central chi-square distribution with mean of df 0 and variance of df 0 where df 0 denotes degrees of freedom from the correctly specified model. For the example I shown above (see Figure.a), the model degrees of freedom for the correctly specified model is 6. The critical value can be obtained under the central chi-square distribution to conduct statistical testing. As an illustration, the chi-square distribution shown with a solid line in Figure. is a central chi-square distribution with df 0 =6. A critical value associated with. 05 is.59. However, when H 0 is false and a specific H is true, (e.g., the -factor model without crossloadings in the above example), the T statistic follows an asymptotically non-central chi-square distribution with a mean equal to its degrees of freedom plus a non-centrality parameter,, where ( N ) F. The expected value of the sample chi-square equals df + and the variance is (*df )+(4* ), where d denotes degrees of freedom associated with a specific mis-specified model. It should be noted that the non-centrality parameter is also model-specific. For the example I showed above, the degrees of freedom corresponding to the -factor model without cross-loadings (Figure.b) and the -factor model (Figure.c) are 8 and 9, respectively. Given N=00, the non-centrality parameter corresponding to the -factor model without crossloadings and the -factor model are 8.45 and 89.56, respectively. As an illustration, the chisquare distribution shown by a dashed line in Figure. is a non-central chi-square distribution with df =8 and =8.45. The magnitude of indicates the degree of model misspecification in the population. Although not shown in the Figure., the non-central chi-square distribution corresponding to the -factor model (df =9 and =89.56) is different from the one for -factor model without cross-loadings. 8

18 critical.59 dchisq(x, df = 6) α β Power x Figure. Central and non-central chi-square distributions, Type I and Type II errors, and power. Type I, Type II Errors, and Power There are multiple ways to assess the model-data fit for the overall model, but the chisquare test statistic is the only one that allows researchers to conduct a statistical test of the fit hypothesis. In general, each statistical decision comes with Type I and Type II errors. As shown in Figure, a Type I error, with probability, occurs when the true null hypothesis is rejected; the Type II error rate,, is the probability of failing to reject the false null hypothesis when the alternative hypothesis is true (Everitt & Skrondal, 00). Power is the probability of rejecting a false null hypothesis which equals ( ) (Everitt & Skrondal, 00). Within SEM, power is the probability of rejecting a model when it is mis-specified. As illustrated in Figure., power is the probability of obtaining the critical value or a greater value under the non-central distribution. As I mentioned earlier, a non-central distribution is characterized by the degrees of freedom and non-centrality parameter,. Therefore, power is closely related to. Other things held constant, power is greater when the non-centrality parameter increases (i.e., with greater degrees of model misspecification). Therefore, the key point in power analysis of SEM is to determine the magnitude of. In the following section, I summarize previous studies on 9

19 conducting power analysis for SEM, including power analysis for individual model parameter(s) and overall model fit. Power Analysis for Individual Parameters Satorra and Saris (985) developed a procedure to determine power for detecting nonzero values for individual SEM parameters. Hancock (006) provided a comprehensive introduction of this procedure. Below I use a regression model with two predictors (x and x ) to illustrate the procedure. x e y x Figure.3 Regression model The regression equation is y among these three variables is: var(y) cov(x,y) var(x ) cov(x, y) cov(x,x ) var(x ) 0 x x e. The population covariance matrix This model includes six parameters: two direct effects (β and β ), variances and covariance of exogenous variables (var(x ), var(x ), and cov(x, x )) and residual variance (var(e)). These six parameters as a set form a parameter vector, which is denoted as R. The covariance matrix implied by this model can be expressed as a function of the parameter vector F : var(x ) var(x ) cov(x,x ) var( e) ( F ) var(x ) var(x ) var(x ) cov(x,x ) var(x ). Assuming this regression model is correctly specified and we are interested in examining the power for detecting the direct effect from x to y with a magnitude of.5, the null and alternative hypotheses are: 0

20 H H 0 : 0, and : 0(.5). All five other parameters are referred to as peripheral parameters (Hancock, 006) and these five parameters needed to be specified a priori based on the previous studies or the best guess by the researcher. Based on the predefined parameters (and the value of.5 for the direct effect), the covariance matrix,, can be obtained. Then two models should be analyzed in order to examine the power for detecting a direct effect of.5 from x to y. The first model is called the full model, which is the model with all six parameters to be freely estimated. This model yields a fit function, F( F ). The second model is called the restricted model, which is identical to the full model except that the direct effect from x to y is fixed to zero. Specifically, the implied covariance matrix for the restricted model is: var( x) var( e) ( R ) 0 var( x) var( x ) cov( x, x ) var( x This model yields a fit function, F( R ). If the direct effect of.5 is not significantly different from 0 in the population, then the two models should yield identical fit functions. Otherwise, the restricted model will fit worse than the full model. In other words, the null and alternative hypotheses for β can be expressed as: H : F( ) F( ), 0 R F H : F( ) F( ) R F When the H 0 is true, F ) F( ). The sampling distributions of the T statistics ( R F corresponding to the full model and the restrictive model follow central chi-square distributions with expected values of central chi-square distribution: diff R ) df and df. Consequently, chi-square difference statistic also follows a F ( n )*[ Fˆ( ˆ ) Fˆ( ˆ )]. (4) It is equivalent to: R F diff (5) R F with degrees of freedom df diff df df. R F

21 However, when the H 0 is not true, and instead F ) F( ), then the chi-square ( R F difference statistic follows a non-central chi-square distribution with a non-centrality parameter. Recall that represents the difference between the expected values of the null and alternative distributions. In this case, the non-centrality parameter can be expressed as: n ) F( ) ( n ) F( ) ( n )[ F( ) F( )], (6) R F ( R F R F Once the chi-square distributions under the null and alternative hypotheses have been determined, power is the probability of rejecting the specified true alternative hypothesis (Everitt & Skrondal, 00). In this case, the power is the area under the alternative hypothesis distribution where the chi-square test value is equal to or larger than the null-hypothesis critical value: Pr{ df }. (7), critical This framework allows for conducting power analysis for one or more specific parameters in a model. This approach has been developed under the ML estimation method with an assumption of multivariate normality. It assumes that the researcher already has one or more alternative parameters in mind. It is especially useful when researchers have known values for most parameters in the model and intend to examine the contribution of a limited set of parameters. Therefore, this approach is particularly useful for post hoc power analysis for specific model parameter(s). However, it is difficult to specify the alternative parameter vector if less information is available, and this occurs more often than not. For example, there might be no previous studies about the structure of relations among variables, or the magnitude of parameters in a model. For this reason, a priori power analysis (for sample size determination) is extremely difficult within this framework. In addition, the definition of effect size is not clear. Power Analysis for Overall Model-data Fit MacCallum, Browne and Sugawara (996) described a framework for conducting power analysis for a model as a whole in terms of the root mean square error of approximation (RMSEA, ). They computed power based on the difference in RMSEAs between two models under a variety of conditions. RMSEA is one of the fit indices most commonly used to evaluate model-data fit. The definition of RMSEA is based on the property that the minimum value of the discrepancy function is equal to, or closely approximated by a sum of df squared discrepancy terms, where

22 the discrepancy terms represent systematic lack of fit of the model (MacCallum et al., 996 pp ). RMSEA is a function of the discrepancy function, F, and degrees of freedom, df, specifically: F. (8) df It indicates the degree of model-data misfit per degree of freedom. Holding all other conditions constant, RMSEA is smaller for models with more degrees of freedom. RMSEA is an index of badness-of-fit, where a value of zero suggests the best fit and greater values suggest worse fit (e.g., Kline, 00). Previous simulation and empirical studies provided cutoff points for making decisions on model-data fit in terms of RMSEA (Browne & Cudeck, 993; Browne & Mels, 990; Hu & Bentler, 999; Steiger, 989). Some studies (Browne & Cudeck, 993; Browne & Mels, 990; Steiger, 989) considered values of less than.05 as representing close fit, from.05 to.08 as average fit, and larger than.0 as poor fit. Hu and Bentler (999) recommended using the value of.06 as cutoff point for close fit, while MacCallum and his colleagues (996) chose.05. One major contribution of MacCallum et al. (996) is that they expanded significance testing in SEM from exact fit to close fit and not-close fit. The null and alternative hypotheses I mentioned at the beginning of this chapter are those for the exact-fit hypotheses. The null hypothesis, H : ( ), means that the model exactly fit the population data with F 0. This 0 test is equivalent to H : 0. The alternative hypothesis would be H : 0, meaning that the 0 model does not exactly fit the data in the population. When H 0 is true, the test statistic T ( N ) F follows an asymptotic central chi-square distribution. Otherwise, the distribution of T is a non-central chi-square distribution. Let 0 denotes the value of RMSEA under the null hypothesis, and be the value under the alternative hypothesis. In their framework, MacCallum et al. (996) determined the central and non-central chi-square distributions based on the choice of the values of RMSEA. In other words, a pair of RMSEAs, 0 and, is used to define the non-centrality parameter. When H 0 is true, 0 equals zero. From Equation (8), F can be written as df, thus can be re-expressed as a function of degrees of freedom and RMSEA: N ) F ( N ) df. (9) ( 3

23 Power is the probability of rejecting the H 0 given the true H, which is the area under the H distribution where the chi-square value equals to, or is larger than, the critical value (See Equation (7)). The exact fit hypothesis is very stringent. It is always false to some degree for overidentified models in empirical applications. Therefore, MacCallum et al. (996) suggested testing hypotheses of close fit and not-close fit. For testing close fit, the hypotheses are: H H 0 :.05 :.05 A true H 0 means the model closely fits the data in the population. The test statistic follows a non-central chi-square distribution under both H 0 and H with nonzero 0 and. Power is the probability of rejecting the close fit hypothesis when the model does not closely fit the data (see Figure.4 with a specific RMSEA of.08 for ). Compared to the test for exact fit, the procedure for computing power is the same except that the distribution under H 0 is no longer central. 4

24 Figure.4 Illustration of power under close-fit hypothesis test. (Figure from MacCallum et al., 996, p. 40.) For the not-close fit test, the hypotheses are: H H 0 :.05 :.05 Rejection of H 0 would support the conclusion that model and data are closely fit to each other, that is, would support H. Unlike the tests for exact fit and close fit, the non-central chisquare distribution under an alternative hypothesis of not-close fit is located on the left side of the null-hypothesis distribution (see Figure.5 with a specific RMSEA of.0 for ). That is because 0 is greater than, so are their associated 0 and values. Thus, power is the probability of a chi-square value under the alternative hypothesis that is smaller than the critical value: Pr{ df }. (0), critical 5

25 Figure.5 Illustration of power under the not-close fit hypothesis. (Figure from MacCallum et al., 996, p. 4) Effect Size Using RMSEA MacCallum et al. (996) defined the effect size as the difference between 0 and. Power is highly related to the separation of the chi-square distributions under null and alternative hypotheses. This separation is determined by the associated non-centrality parameters for these two distributions, 0 and. The difference between 0 and is a function of degrees of freedom, sample size, and 0 and. Holding the degrees of freedom, sample size, alpha level, and 0 as constants, power monotonically increases if increases. Compared to the framework provided by Satorra and Saris (985), conducting power analysis using a pair of RMSEAs doesn t require the users provide specific values for parameters for null and alternative hypotheses. Instead, users choose a pair of RMSEA values (e.g.,.05 and.0 for close fit and poor fit; Steiger, 989). Another benefit is that it is possible to compute the confidence intervals of RMSEA values. The confidence intervals represent the degree of confidence that the population value of the fit index locates between the upper and lower boundaries of the interval (MacCallum et al., 996). 6

26 However, it is problematic to use pairs of RMSEA values as an effect-size index. MacCallum and his colleagues (996) also emphasized that power is not a simple function of the difference between 0 and. As illustrated in Figure.6, the relationship between power and the difference between 0 and is not monotonic. The power level varies for different pairs of RMSEA values that yield the same difference value between 0 and. For example, let. 0 0,. 06for one pair of models, and. 0 0, 07. for the other pair of models. If effect size is defined as 0, these two pairs yield the same effect size of.05. But the estimated power is not the same for these two cases even if the degrees of freedom and sample sizes are the same. MacCallum et al. (996) stated that a similar phenomenon also occurs in power analysis for testing differences between correlation coefficients or proportions (Cohen, 998). The conventional method is to define the effect size as a function of a transformation of correlation coefficients or proportions. In SEM, MacCallum, Browne, and Cai (006) recently suggested using the fit function as a measure of effect size with the ML estimation method. According to Equation 8, RMSEA is a function of a transformed value of F. power es Figure.6 Plot of power with pairs of RMSEA values for the d=5, sample size=00, and alpha=.05. 7

27 In their recent study, MacCallum et al. (006) provided a procedure for testing differences of overall model-data fit between two nested structural equation models. Let Model A be nested in Model B, which means the freely estimated parameters in Model A are a subset of those in Model B. For example, model B is a CFA model involving all the loadings, errors, variances and covariances of latent variables. If the covariances among factors are all fixed as one, then Model A will be nested in Model B. The fit function (using the ML method) reflects the discrepancy between the implied covariance matrix and the observed covariance matrix (see Equation ()). In this case, the null hypothesis is that there is zero difference in the population discrepancy functions (F) between these two models, which can be expressed as H : F A F 0. In other words, these two nested models share the same degree of model-data fit in the population. The test statistic T follows a chi-square distribution with degrees of freedom df df df A B A B when B H is true and the multivariate normality assumption for ML estimation is met. When H 0 is false and H is true, T follows an approximately non-central chi-square distribution with df A B df A dfb with non-centrality parameter ( N )( FA FB ) ( N ). () FA FB Recall in Equation (8), the RMSEAs are defined for Model A and B as A, B, df df respectively. Based on Equation (), the effect-size index, denotes as, can be expressed as: ( F F ) ( df df ). () A B A A B B Hence, the non-centrality parameter can be determined by degrees of freedom and RMSEA ( N )( F F ) ( N )( df df ). (3) A B A A B So far, the effect size is clearly defined as the difference of fit functions between two nested models under the ML estimation method, which is also a transformation of the RMSEA index. In the present research, I derived the formula for the expected value and variance of the fit function and examined the performance and distributional characteristics of the fit function under various conditions. Effect Size Using Other Indices It is also feasible to conduct a power analysis for testing overall model-data fit using other fit indexes such as the goodness of fit index (GFI), GFI adjusted for degrees of freedom (AGFI), or comparative fit index (CFI) (Kim, 005; MacCallum & Hong, 997). The procedure B A B

28 for conducting power analysis using other fit indexes is similar to the one using a pair of RMSEA or discrepancy functions. The key difference is the way the non-centrality parameters for chi-square distributions associated with null and alternative hypotheses are defined. MacCallum and Hong (997) demonstrated how to define by using values of GFI or AGFI indexes. However, they also stated that it is questionable to set up appropriate values of these two fit indexes for null and alternative hypotheses. When using GFI to determine the, the power is decreased as the degrees of freedom become larger for specific models associating with null and alternative hypotheses. Power seems to be lower for models with greater degrees of freedom, which is an undesirable phenomenon. It indicates that the GFI-based indexes have more difficulty in detecting a true alternative hypothesis for a simpler model than for a more complex model, in contrast to power analyses based on the AGFI or RMSEA. With regard to the use of AGFI for power analysis, it is quite difficult to establish meaningful and appropriate cutoff point values of AGFI. For this reason, researchers don t recommend the use of GFI and AGFI for conducting power analysis (MacCallum & Hong, 997). Kim (005) offered equations for determining a non-centrality parameter based on RMSEA and three other fit indexes: CFI, McDonald s Fit Index (Mc), and Steiger s gamma. He defined the effect size as the specific fit index chosen to conduct the power analysis and determine the minimum sample size required for achieving a specific level of power. He found that power varies with the type of fit index, and is a function of number of variables, degrees of freedom, value of the fit index (i.e., the degree of misspecification in a model), and sample size. Monte Carlo studies by Hu and Bentler (999) showed that the fit indexes behave differently under different conditions. For example, the correctly specified population models tend to be overrjected for small sample sizes when using Mc and RMSEA. CFI was found to be sensitive to models with mis-specified factor loadings. In addition, CFI is called the comparative fit index. It evaluates the model-data fit of the researcher s hypothesized model by comparing its fit with that of a more restricted model: the null model. It is computed as one minus the ratio of noncentrality parameters of the researcher s model and the null model. However, the definition of the null model is different depending on the SEM software used (e.g., Mplus, or SAS). Hence, CFI might not be appropriate to be used to compare different models. In conclusion, Mc, Steiger s gamma, and RMSEA behave differently fordifferent estimation methods. Hu and Bentler (999) suggest using at least two classes of fit indexes to evaluate the overall model-data 9

29 fit for small sample sizes. Kim (005) notices that it is not clear how using different fit indexes impacts power analysis and sample size determination. Summary Researchers have proposed different methods of conducting power analysis for likelihood ratio tests (i.e., for overall structural equation models). To obtain the power for the test of overall model fit, these methods rely on the use of fit indices. In some approaches mentioned earlier, effect size is defined as the fit index (MacCallum et al., 996; MacCallum & Hong, 997; Kim, 005). However, a previous simulation study by Hu and Bentler (999) indicated instability of the behaviors of the fit indexes under different conditions. Some fit-index-based power analysis approaches have their own weaknesses. For instance, power results in different values even if the effect sizes are the same in the RMSEA-based method (MacCallum et al., 996). GFI has an undesirable negative relationship with the degrees of freedom, and it is difficult to choose meaningful cutoffs of AGFI. The CFI index is not appropriate for comparing the degree of goodness of fit across different models. Therefore, I would not recommend defining the effect size as a single fit index. The discrepancy function, F, however, directly reflects the difference between the model-implied covariance matrix and observed covariance matrix. The noncentrality parameters of the distributions associated with null and alternative hypotheses are also determined by the fit function. In addition, F under ML estimation is a scale free and scale invariant index. Scale free means if a variable is linearly transformed, then the parameter estimates of the variable can be algebraically converted back to the original one. Scale invariance means the fit function under ML estimation would not be affected by the scale of the observed variables in a model (Kline, 00). Hence, I prefer to adopt the suggestion by MacCallum and his colleagues (006) who defined effect size in terms of the fit function. In this study, I focus on how to compute the expected value and variance of the effect-size index for a structural equation model, and then investigate the distributional characteristics of the index under various conditions. 0

30 CHAPTER THREE METHODOLOGY Fit Function as Effect Size Index in SEM In the previous chapter, I discussed the key concepts for conducting power analysis in SEM including test hypotheses, test statistics, central and non-central distributions, Type I and Type II errors, and effect size. In covariance structure modeling, the null hypothesis (H 0 ) is that the covariance matrix reproduced by the hypothesized model matches exactly the observed covariance matrix in the population. For the ML estimation method, the discrepancy function, F, indicating the difference between the two matrices is: F tr ) ln ( ) ln ( ( ) p, where and ( ) are the observed and reproduced covariance matrices, respectively, among the p observed variables. In the population, when H 0 is true, F=0; when H 0 is false and the alternative hypothesis (H ) is true, F > 0. Therefore, F reflects the degree to which model and data misfit in the population and can be defined as effect size (MacCullum et al., 006): where denotes effect size. F (4) At the sample level with sample size n, when H 0 is true, T ( n ) Fˆ has an asymptotic central chi-square distribution with mean of df 0 and variance of df 0 where ˆF is fit function estimated based on the sample and d 0 is the model degrees of freedom associated with the correctly specified model: T ~ 0 ( df 0,df0). (5) A critical value can be obtained to conduct statistical testing. As mentioned earlier, in power analysis, a specific alternative hypothesis is of particular interest. When a specific alternative model is chosen, the corresponding non-centrality parameter can be calculated. If this specific model is mis-specified, then the alternative hypothesis is true and the model is associated with a specific degrees of freedom and non-centrality parameter. Then the test statistic T ( n ) Fˆ follows a non-central chi-square distribution characterized by the non-centrality parameter corresponding to the specific mis-specified model:

31 T ~ df,df ), (6) ( where and d denotes chi-square and degrees of freedom associated with the alternative hypothesis and ( n ) Fˆ and it is the chi-square value from the specific mis-specified model in the population (i.e., no sampling error is involved). is crucial for defining effect size and conducting power analysis because power is the probability of obtaining the critical value or a greater value of the chi-square under the distribution with non-centrality parameter. For sample data, can be estimated as sample model chi-square minus df, that is, ˆ df where ˆ is the estimated non-centrality parameter based on the sample. As mentioned before, the fit function F directly reflects the degree of model misspecification. In the population, F would not be affected by the sample size. For sample data, ˆF values can be obtained by dividing the observed chi-square value for a model by the sample size (SAS Proc Calis provides the sample fit function). According to Equation (), the effect-size index could be expressed using the observed chi-square value: ˆ F. n Thus, it is clearly shown that the effect size is determined by degrees of freedom, sample size, and the degree of model misspecification. In Equation (7), the sample size and degrees of freedom can be considered as constants once data are collected and the model is specified. The expected values and variances of chisquare distributions are known for both central and non-central distributions. Therefore, it is possible to derive the expected value and variance formulas for fit function Fˆ. When H 0 is true, the model chi-square follows a central distribution, and the expected value and variance of the fit function are: (7) E( ) df0 E( Fˆ) E( ), n n n (8) ˆ Var ( ) df0 Var ( F) Var ( ) n ( n ) ( n ). (9) Note that the expected value of the fit function is very close to the ratio of the degrees of freedom to sample size when the model is correctly specified. The variance is also determined by

32 the degrees of freedom and sample size in a model. Taking the square root of the variance in Equation (9) provides a standard deviation of the sample fit function. When H 0 is false and a specific H is true, the expected value and variance of the fit function are then: ˆ E( ) df E( F) E( ), (0) n n n ˆ Var ( ) df 4 Var ( F) Var ( ). () n ( n ) ( n ) The expected value and variance of the fit functions become more complex for a mis-specified model since a nonzero model-dependent non-centrality parameter needs to be taken into account. Since sample sizes can be considered as constants, the fit function shares the same skewness and kurtosis with the chi-square distribution based on Equation (7). Till now, the fit function has been clearly defined as an effect size index, and I have presented its expected value and variance for both correctly specified and mis-specified models. Next I examine distributional characteristics of the fit function under various conditions and show how the fit function is related to degrees of freedom, sample size, degree of model misspecification, and power using computer generated data. Data were simulated in the population and sample. The purpose of simulating data at the population level (i.e., for the population covariance matrix) is to investigate the relationship between effect size (i.e., the degree to which a model is mis-specified), sample size, and power. The purpose of simulating data at the sample level is to examine sampling fluctuation of the fit function. Simulation Study The simulation study was designed to examine the performance of the effect-size index under various manipulated conditions. All data generation and data analyses were completed using SAS 9.. Two factors were considered to derive population covariance matrix: factor structure (6 models) and model parameters (5 levels), resulting in 5 conditions. Three levels of sample size were considered for the sample level investigation. For each condition at the sample level, 000 datasets were generated. Two models were then analyzed for each dataset. One was correctly specified model and the other was a mis-specified model. Below I describe in detail the design factors. 3

33 Study Design Factor Structure Six factor structures were considered for data generation as presented in Figure 3.. Model (Figure 3.a) contained six items with the first three items measuring the first factor and the other three measuring the second factor. Model (Figure 3.b) contained eight items with the first four items measuring the first factor and the other four measuring the second factor. Model 3 (Figure 3.c) contained ten items with the first five items measuring the first factor and the other five measuring the second factor. Model 4 (Figure 3.d) had the same factor structure as Model. In addition, the item 3 cross loaded on the second factor and the item 4 cross loaded on the first factor as indicated by the dashed lines in Figure 3.d. Model 5 (Figure 3.e) had the same factor structure as Model. In addition, the item 4 cross loaded on the second factor and the item 5 cross loaded on the first factor as indicated by the dashed lines in Figure 3.e. Model 6 (Figure 3.f) had the same factor structure as Model 3. In addition, the item 5 cross loaded on the second factor and the item 6 cross loaded on the first factor as indicated by the dashed lines in Figure 3.f..5,.8,.95 F F X X X3 Mo X4 Skew X5 ness=. X6 a. Two-factor model with 6 observed variables 4