Comparison Group Selection with Rolling Entry in Health Services Research Rolling Entry Matching Allison Witman, Ph.D., Christopher Beadles, Ph.D., Thomas Hoerger, Ph.D., Yiyan Liu, Ph.D., Nilay Kafali, Ph.D., Sabina Gandhi, Ph.D., Peter Amico, Ph.D., Ann Larsen 1 RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org
Acknowledgement and Disclaimer This research was supported by the Centers for Medicare and Medicaid Services under the U.S. Department of Health and Human Services Contract number: HHSM-500-2010-00021I The contents of this presentation are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies. 2
Rolling Entry Interventions Dynamic Variable Rolling entry: participants start intervention at different times Time Rolling entry presents challenges for comparison group selection Small number of entrants in a period hinders propensity score models Precipitating events prior to entry may be hard to pin down, but generate changes in dynamic variables (e.g., utilization, spending) What is the counterfactual entry date for comparison? 3 Time
Rolling Entry Matching (REM) Selects Comparison Groups for Rolling Entry Interventions Small number of entrants in a period Pools all entrants into a single propensity score model Dynamic variables that predict treatment change prior to entry Comparisons with similar pattern during the same time period are the best match Entry date of comparison is chosen based on entry date of best-matched treatment observation REM: An example Synthetic intervention Implementation of REM 1. Quasi-panel dataset 2. Apply matching methodology 3. Matching algorithm Results 4
Rolling Entry Matching Synthetic Intervention Synthetic population that resembles Medicare claims data from actual intervention 600 participants and 10,000 potential comparisons 8 entry quarters Quarterly enrollment pattern: 40, 50, 60, 70, 80, 90, 100, 110 Follow participants for 8 baseline and 8 intervention quarters Spending spike Participants: quarter prior to entry Comparisons: Random quarter Treatment effect is -$50 slope change in intervention period 5
Implementing Rolling Entry Matching: Step 1 1. REM quasi-panel dataset 1 observation of each participant Potential comparisons appear 8 times, once for each entry quarter 6 ID QUARTER TREAT AGE PREVIOUS VISITS 1 3 1 72 3 2 8 1 73 5 3 1 0 71 3 3 2 0 71 4 3 3 0 72 0 3 4 0 72 2 3 5 0 72 3 3 6 0 72 4 3 7 0 73 1 3 8 0 73 5
Implementing Rolling Entry Matching: Step 2 2. Apply matching method Propensity score Mahalanobis distance Coarsened exact matching Using synthetic data, we estimate a propensity score model containing baseline: Demographics Chronic conditions Previous Health care utilization Previous Medicare payments 7
Implementing Rolling Entry Matching: Step 3 3. REM matching algorithm Match on propensity score 1:variable matching within a caliper up to 3 comparisons per participant Match within quarters Comparisons selected for the quarter in which they are the best match Comparisons used with replacement within a quarter 8 ID QUARTER TREAT 1 3 1 2 8 1 3 1 0 3 2 0 3 3 0 3 4 0 3 5 0 3 6 0 3 7 0 3 8 0 AGE PREVIOUS VISITS PROPENSITY SCORE 72 3 0.13 73 5 0.15 71 3 0.14 71 4 0.18 72 0 0.10 72 2 0.11 72 3 0.17 72 4 0.18 73 1 0.14 73 5 0.15
Results: Treatment and Comparison Balance Absolute Standardized Differences Before and After Matching Payments in Previous Quarter Payments in Previous Year Age Male White Hispanic Black Disabled ESRD Number of Chronic Conditions Number of Dual Eligible Months ED Visits in Previous Quarter ED Visits in Previous Year IP Admissions in Previous Quarter IP Admissions in Previous Year 0.1.2.3.4.5.6.7.8.9 1 Before Matching After Matching All variables reach a standardized difference 0.10 after matching 9
Results: Difference-in-Difference Treatment Effect Estimate change in slope using a difference-in-difference framework Synthetic treatment effect is -$50 Quarterly Spending Intervention Effect -50.10* (28.38) Observations 21,760 R-squared 0.704 Notes: Table presents difference-in-difference linear regression coefficients and standard errors. The regression included controls for age, gender race, disability status, dual status, chronic diseases, ED use, inpatient use, a linear function of time, and indicators for the treatment group and post-intervention period. *** p<0.01, ** p<0.05, * p<0.1 10
Conclusion Implications for policy and practice Post ACA, rolling entry interventions utilized to maximize enrollment and impact Evaluation methodology must advance to provide unbiased impact estimates Rolling Entry Matching (REM) is a new methodology for selecting comparison groups in the presence of rolling intervention entry Limitations Results illustrated in synthetic data only More research is needed to test methodology In synthetic intervention, performs similarly to cohort matching 11
Acknowledgements and Contact Acknowledgments Funded by the Centers for Medicare and Medicaid Services Contact information Allison Witman RTI International 3040 E. Cornwallis Road Research Triangle Park, NC 27709 www.rti.org awitman@rti.org 12
Results: REM versus Cohort Matching Propensity score matching can perform poorly when sample size is small Concato et al 1995, Peduzzi et al 1995, Peduzzi et al 1996, Zhao 2004 Vary treatment group size between 300 and 4,500 Treatment group cohorts range between 25 and 825 Comparison group fixed at 10,000 13