Analysis of cross-national surveys: Latent variable modelling for substantive comparisons and assessment of measurement

Transcription

1 Analysis of cross-national surveys: Latent variable modelling for substantive comparisons and assessment of measurement Jouni Kuha London School of Economics and Political Science ESRC Research Methods Festival Oxford, J. Kuha Analysis of cross-national surveys RMF, /24

2 The LCAT project Research project Latent variable modelling of categorical data: Tools of analysis for cross-national surveys, or LCAT for short Funded by ESRC grant RES , under the Methods for Comparative Cross-National Research initiative Investigators: Jouni Kuha, Irini Moustaki, Sally Stares, Jonathan Jackson, and George Gaskell Methodology Institute and/or Department of Statistics, London School of Economics and Political Science See for more J. Kuha Analysis of cross-national surveys RMF, /24

3 In this session 1 Jouni Kuha: Introduction and overview 2 Irini Moustaki: Methodological exploration sensitivity of comparative analyses 3 Sally Stares: Two applied examples J. Kuha Analysis of cross-national surveys RMF, /24

4 Key concepts Cross-national surveys Multi-item measures of constructs Latent variable modelling especially for categorical items: latent class and latent trait modelling Cross-national comparative research questions about constructs Measurement modelling and equivalence of measurement J. Kuha Analysis of cross-national surveys RMF, /24

5 Cross-national social surveys Surveys where the same questions (translated) are asked in several countries For example: Eurobarometers (and other regional barometer surveys, for Africa, Asia and Latin America) World Values Survey European Values Study International Social Survey Programme European Social Survey A key purpose is obviously answering comparative cross-national research questions. J. Kuha Analysis of cross-national surveys RMF, /24

6 Multiple-item scales and latent variables Theoretical constructs that are difficult or impossible to capture directly with one measure attitudes, opinions, abilities, motivations, etc. Measurement of such a construct is achieved through several observable indicators (questionnaire items). The contructs of interest are regarded as latent variables, and the indicators as measures of them. A statistical latent variable model is set up to represent this situation J. Kuha Analysis of cross-national surveys RMF, /24

7 Example Data from Round 3 of European Social Survey (2006), Personal and social well-being rotating module. optim: Im always optimistic about my future. positive: In general I feel very positive about myself. failure: At times I feel as if I am a failure. ideal: On the whole my life is close to how I would like it to be Response options for each: Agree strongly, Agree, Neither agree nor disagree, Disagree, Disagree strongly. Asked in 23 countries. J. Kuha Analysis of cross-national surveys RMF, /24

8 Multigroup latent variable models A simple latent variable model for cross-national analysis is of this form: η is a latent variable (or variables), representing the construct of interest y j are the observed items, regarded as measures of the construct Country g is an explanatory variable for η, so that distribution of the construct(s) may vary between countries Two key parts of the model: structural model for η given g measurement model for y j given η g η y 1 y 2 y 3 J. Kuha Analysis of cross-national surveys RMF, /24

9 Latent variable models Within this basic idea, different assumptions about the nature of the variables give different model families: Observed indicators Continuous Categorical Latent Continuous Factor analysis Latent trait models variables Categorical Latent profile analysis Latent class models Factor analysis models are most commonly used The focus of the LCAT project is on models for categorical indicators, i.e. latent trait and latent class models. Useful, because most items in surveys are categorical. J. Kuha Analysis of cross-national surveys RMF, /24

10 Cross-national comparisons of latent distributions A multigroup latent variable model can be used to estimate how distributions and associations of latent variable vary between countries In simple analysis of one construct: Latent trait and factor analysis models: Means and variances of latent traits (factors) Latent class models: Probabilities of the latent classes Example below: 1-trait ordinal latent trait model for the 4 well-being items above Estimates of country means of the trait ( happiness ) J. Kuha Analysis of cross-national surveys RMF, /24

11 Happiness example: Estimates of national averages Cyprus Austria Switzerland Denmark Germany Ireland Spain Slovenia Finland Sweden Norway Bulgaria UK Netherlands Portugal Poland Belgium Estonia Ukraine France Russia Slovakia Hungary Avarage of latent trait HAPPY J. Kuha Analysis of cross-national surveys RMF, /24

12 How about the measurement models? The analysis above specified that the measurement model for every item was the same in every country. i.e. all the items work the same way across countries This is the assumption of cross-national measurement equivalence (or invariance). If equivalence does not hold for all items, we have nonequivalence of measurement. Possible reasons for non-equivalence in survey items: translation differences different salience of the questions in different national contexts J. Kuha Analysis of cross-national surveys RMF, /24

13 Equivalent and non-equivalent measurement models (1) (2) Item 1 Item 2 Item 3 Item 1 Item 2 Item 3 (b) (a) Latent Variable Latent Variable Country Country J. Kuha Analysis of cross-national surveys RMF, /24

14 Modelling measurement non-equivalence Latent variable models can accommodate non-equivalence: Fit models which do not assume equivalence Use standard methods of model selection to compare these models with equivalence, to assess the level of nonequivalence This can be used to assess if all items behave qualitatively similarly ( contruct equivalence ) examine how measurement properties vary, and if some countries or items are particularly different compare latent distributions, allowing for measurement nonequivalence Examples of this in the talk by Sally Stares. J. Kuha Analysis of cross-national surveys RMF, /24

15 What to do about non-equivalence Allowing for non-equivalence is technically straightforward, although sometimes computationally challenging. The bigger challenge is what to do with the conclusions. Ideally, we would conclude that measurement equivalence holds for all items and countries or perhaps except for a few rogue items, which could be omitted... and then proceed with the comparative analysis with increased confidence However, this is unlikely to be the conclusion very often for cross-national surveys of many countries. J. Kuha Analysis of cross-national surveys RMF, /24

16 An examination of one cross-national survey Consider data from Round 5 of European Social Survey: 26 countries Arguably best-designed and methodologically most careful of the big cross-national surveys Consider 8 multi-item scales: institutional trust, political engagement, generalised trust, satisfaction with government, attitudes to immigration, fear of crime, freedom at work Latent-variable analysis (factor, latent trait or latent class models) allowing one item at a time to be non-equivalent: A total of 35 survey questions Of these, are declared non-equivalent (compared to full equivalence model) according to standard model selection criteria (AIC, BIC, LR test) J. Kuha Analysis of cross-national surveys RMF, /24

17 What if equivalence does not hold? If we conclude that measurement equivalence does not hold, there are two basic options: Use a well-fitting non-equivalence model (assuming one can be found) Use the equivalence model anyway Some comments on each of these... J. Kuha Analysis of cross-national surveys RMF, /24

18 Using a non-equivalence model This can be computationally hard There is no guarantee that any of the items are formally equivalent. If not, latent trait and factor analysis models are not even identifiable. Can be interpretationally uncomfortable, especially if many items are nonequivalent: Is it really the same latent variable any longer, if most of the items measure it in different ways across countries? A non-equivalence model inevitably implies that two respondents who give identical answers are regarded as having different level of the underlying construct, if they come from different countries. J. Kuha Analysis of cross-national surveys RMF, /24

19 Using an equivalence model This is the wrong model and conclusions from it may be misleading if we believe in the models and the reality of the latent variables. An alternative is not to believe in the model fully, but to use it just as an aid to derive (implicit or explicit) measures of the constructs In the spirit of pragmatic measurement, where we define concepts through measurements. An equivalence model defines the level of the contruct to be the same for everyone who gives the same responses. But this hides all problems of measurement, including obviously non-equivalent items Statistical analysis should be used at least to check for such items. J. Kuha Analysis of cross-national surveys RMF, /24

20 Sensitivity of conclusions There is one more possible way out of the dilemma: What if it does not matter, i.e. different equivalence and non-equivalence models give similar enough conclusions about the interesting cross-national comparisons. i.e. what if these conclusions are not very sensitive to our assumptions about the measurement model On the next slides, an assessment of this in a real example: Estimated country means of the happiness trait from the equivalence model, and models where one or two items at a time are non-equivalent Conclusions are clearly not identical which ones do we prefer? In Irini Moustaki s talk next: Examination of sensitivity the other way round, when we know the truth. J. Kuha Analysis of cross-national surveys RMF, /24

21 Sensitivity analysis in an example Cyprus Austria Switzerland Denmark Germany Ireland Spain Slovenia Finland Sweden Norway Bulgaria UK Netherlands Portugal Poland Belgium Estonia Ukraine France Russia Slovakia Hungary Mean of latent trait HAPPY J. Kuha Analysis of cross-national surveys RMF, /24

22 Sensitivity analysis in an example Cyprus Austria Switzerland Denmark Germany Ireland Spain Slovenia Finland Sweden Norway Bulgaria UK Netherlands Portugal Poland Belgium Estonia Ukraine France Russia Slovakia Hungary Mean of latent trait HAPPY J. Kuha Analysis of cross-national surveys RMF, /24

23 Sensitivity analysis in an example Cyprus Austria Switzerland Denmark Germany Ireland Spain Slovenia Finland Sweden Norway Bulgaria UK Netherlands Portugal Poland Belgium Estonia Ukraine France Russia Slovakia Hungary Mean of latent trait HAPPY J. Kuha Analysis of cross-national surveys RMF, /24

24 Conclusions In cross-national survey analysis we are interested in the substantive comparisons, not the measurement. However, different assumptions about measurement can change conclusions Available statistical tools should be used to look for and remove particularly problematic items However, most or all items may display some non-equivalence according to statistical criteria If this is the case for the very best like the ESS, it is unlikely that it will ever be otherwise This surely does not mean that cross-national surveys are not immensely valuable But we need to acknowledge such complications and interpret conclusions accordingly Arguably cross-national surveys are an exercise of pragmatic rather than strongly representational meeasurement J. Kuha Analysis of cross-national surveys RMF, /24