Predicting the FFY 2014 VBP Exchange Slope and Break-Even Score Figure 1: Data Used in Predictive Modeling for FFY 2014 2014 Baseline Period 2014 Performance Period 2q2009 3q2009 4q2009 1q2010 2q2010 3q2010 4q2010 1q2011 2q2011 3q2011 4q2011 1q2012 2q2012 3q2012 4q2012 1 2 3 4 5 6 7 Prediction Period 8 9 Press Ganey Data CMS Data 10 11 Periods 10 and 11 not used in forecast model Overview The following discussion reviews the process used to predict the slope of the Value-based Purchasing exchange function for the Federal Fiscal Year (FFY) 2014. Based on research completed for FFY 2013, a non-parametric model, Exponentially Weighted Moving Average
(EWMA) was used to determine the exchange slope and break-even score 1. Measure performance at the individual hospital level was separately modeled for the 12 clinical measures and 8 satisfaction domains included in FYY 2014 Value-based Purchasing Final Rule 2. When Press Ganey client specific data were not available, public data were used. As CMS only releases data in rolling 4-quarter increments, the 4-quarter increments was used in the correlation model. As illustrated in Figure 1, data were calculated for a total of 9 time periods, each period consisting of 3-calendar quarters. Step 1: Model Selection Using Press Ganey Data The Press Ganey product databases are supported by more than 600 core measures and 1,500 HCAHPS facilities. These databases provide a sufficiently large data set for predictive modeling. In addition to the size of the databases, data available via the Press Ganey products is more current than the most recently released public data from CMS. Further, these data can be aggregated to accurately reflect the performance period defined in the Final Rule. Correlation testing to validate the generalizability of Press Ganey data to public CMS Hospital Compare data was conducted for the time period July 1, 2009 to March 31, 2012 (See Appendix A) 3. Press Ganey data are highly correlated with the public data, as all p values of coefficients are less than 0.001. Based on this high degree of correlation, only Press Ganey data were used in our Value-based Purchasing predictive modeling selection analysis. Step 2: Predicting the Exchange Function Slope Exponentially Weighted Moving Average (EWMA) modeling was applied to the combined Press Ganey/CMS dataset to predict the measure-specific rates in the performance period for each hospital. The predicted individual measure performance scores were then used to calculate VBP scores for the FFY 2014 (See Appendix B). The predicted distribution of VBP Scores for FFY 2014 is shown in Figures 2. 1 EWMA is a commonly used statistical model used to smooth forecast time series data by recursive rolling averages over time. This method assigns progressively larger weighting to more recent data points. This model is particularly well suited for quality data that are typically tracked over time and in which more recent periods are seen as more indicative of performance.. 2 The defined FFY 2014 performance period consisted of three (3) calendar quarters running from April 1, 2012 through December 31, 2012. Starting in FFY 2015, both the baseline and performance periods will be four (4) calendar quarters, starting on January 1 st and running through December 31 st. 3 In our research for FFY 2013, three parametric and one non-parametric were examined using cross-validation. Data were partitioned into two subsets: Subset 1, composed of time periods 1 through 8, was used as the Training Set while subset 2, composed of time period 9, was used as a Validation Set. We performed the analysis on the Training Set and validated the analysis on the Validation Set. For each predictive model, training data were used to predict each VBP measure s performance rate for each individual hospital in period 9. The predicted results were then compared with the validation data in period 9. Exponentially Weighted Moving Average (EWMA) achieved the highest validation correlation R 2 and was selected as the best-fit for VBP predictive modeling (See Appendix B).
Figure 2: Predicted National Distribution of FFY 2014 VBP Scores 4 4 The predicted nationwide Mean VBP Score for FFY 2014 was 47.02. The Mean VBP Score should not be confused with the predicted break-even VBP Score which computed by dividing the desired earn-back (100) by the predicting exchange slope value.
In order to determine the slope of the exchange function, it was first necessary to estimate the total national holdback incentive that would not be earned back. MEDPAR data for each hospital were used to estimate the total base DRG payment that would be held back at the FFY 2014 holdback rate of 1.25%. Table 1 illustrates these calculations for two hypothetical hospitals. Table 1: Estimating Incentive Payments Withheld and Earned Back for Two Hospitals Total Base DRG Payment from MEDPAR FFY 2014 Holdback Rate Estimated FFY 2014 Incentive Payment Withheld Predicted FFY 2014 VBP Score Estimated FFY 2014 Incentive Payment Earned Back Estimated FFY 2014 Incentive Payment Not Earned Back Hospital A $10,000,000.00 1.25% $1,250,000.00 50.00 $625,000.00 $625,000.00 Hospital B $4,000,000 1.25% $500,000.00 33.33 $165,000.00 $483,500.00 $1,750,000.00 $790,000.00 $1,108,500.00 Calculations to determine both the estimated Total FFY 2014 Incentive Payment Withheld and the Total FFY 2014 Incentive Earned Back were performed for each individual hospital subject to Value-based Purchasing. These values are then used to determine the estimated exchange slope required to redistribute the incentive payment not earned back. Using the two hospitals in Table 1 simply for illustration, an exchange slope of 2.15 would be required to redistribute the $1,108,500.00 not earned back based by these two hospitals. Figure 3: Estimating an Exchange Slope for Hospitals A and B
Step 3: Predicting the Break-Even VBP Score The obvious goal of Value-based Purchasing is to improve hospital performance so that hospitals earn back all (100%) of the Medicare incentive being withheld. Calculations outlined in Step 2 previously based on the predicted Total Incentive Withheld and Total Incentive Earned Back amounts resulted in an estimated FFY 2014 exchange function slope of 2.10. Using this estimated exchange function slope, we can now predict the FFY 2014 break-even VBP score by dividing the desired earn-back (100%) by the predicted exchange function slope (2.10). This calculation yields the VBP score that would be needed to earn-back the full amount of the incentive payment being withheld. Our predicted FFY 2014 break-even VBP Score is approximately 48 as illustrated in Figure 3. Figure 3: FFY 2014 Predicted Break-Even VBP Score Calculation FFY 2014 ( ) ( )
Appendix A: VBP Measure Press Ganey and CMS Correlation Table for FFY 2014 Measure Correlation Coefficients Hospital Count Measure Correlation Coefficients Hospital Count AMI-7a 1.000 2 Communication with nurses 0.937 1505 AMI-8a 0.714 162 Communication with doctors 0.898 1505 HF-1 0.858 454 Responsiveness of staff 0.929 1,487 PN-3b 0.767 456 Pain Management 0.873 1,470 PN-6 0.745 457 New medicines explained 0.876 1,444 SCIP-CARD-2 0.801 424 Discharge information 0.928 1,489 SCIP-INF-1a 0.737 449 Cleanliness and quietness of hospital environment 0.963 1,505 SCIP-INF-2a 0.711 448 Overall rating of hospital 0.964 1,504 SCIP-INF-3a 0.777 448 SCIP-INF-4 0.810 123 SCIP-INF-9 0.927 324 SCIP-VTE-1 0.574 124 SCIP-VTE-2 0.816 456
Appendix B: Predicted FFY 2014 Measure-level Performance Using EWMA Measure Mean Standard Deviation Facility Count 10 th 25% 50 th 75 th 90 th AMI-7a 72.0405 34.4438 56 0.00 45.48 88.32 100.00 100.00 AMI-8a 93.3022 12.1178 1,495 80.89 91.5 98.39 100.00 100.00 HF-1 91.8267 14.1114 3,061 77.91 89.63 97.07 100.00 100.00 PN-3b 96.8163 6.4795 3,043 91.78 96.1 98.84 100.00 100.00 PN-6 94.8759 8.3343 3,054 87.65 93.1 97.47 100.00 100.00 SCIP-CARD-2 94.9459 11.19 3,036 87.01 94.91 98.81 100.00 100.00 SCIP-INF-1a 97.2601 8.4374 3,141 94.47 97.65 99.31 100.00 100.00 SCIP-INF-2a 97.2667 8.1662 3,143 94.63 97.67 99.18 100.00 100.00 SCIP-INF-3a 96.6556 7.6978 3,136 92.43 96.43 98.69 100.00 100.00 SCIP-INF-4 95.7088 5.7827 1,155 89.21 93.75 97.31 99.74 100.00 SCIP-INF-9 94.1957 10.9547 2,912 85.1 92.86 98.04 100.00 100.00 SCIP-VTE-1 97.5213 8.7043 2,829 95.09 98.32 100.00 100.00 100.00 SCIP-VTE-2 97.241 8.0849 3,120 93.52 97.56 99.67 100.00 100.00 Clean/Quiet 65.5741 7.8655 3,255 56.11 60.29 65.37 70.11 75.38 Discharge inst 84.3231 5.3225 3,255 77.87 81.6 84.78 87.72 90.03 Hosp Rate 69.3568 9.3624 3,255 58.26 63.91 69.67 74.96 80.26 MD Comm 80.864 5.5936 3,255 74.52 77.69 80.81 84.1 87.35 Meds Expl 62.7676 7.1965 3,255 54.42 58.69 62.64 66.61 71.16 Pain Mgmt 70.506 6.521 3,255 63.31 67.16 70.77 74.12 77.63 RN Comm 77.7719 5.900 3,255 70.80 74.84 78.09 81.17 84.34 Responsiveness 65.6016 9.0207 3,255 54.93 60.47 65.42 70.54 76.56 AMI Survival 84.9121 1.8837 2,484 82.56 83.73 84.95 86.22 87.26 HF Survival 88.243 1.9726 3,030 85.70 87.03 88.37 89.61 90.65 PN Survival 87.7839 2.2790 3,055 84.78 86.38 88.00 89.41 90.50
Figure 4: Scatterplot of Predicted Scores and Incentive Payment at Risk 5 5 Figure 4 illustrates the relation of predicted VBP Scores and incentive payments at risk across the nation. When hospitals perform poorly, less of the incentive payment held back is actually earned back and the slope of the exchange function is higher, reflecting the increased amount of incentive not earned back that must be redistributed. As you can see, the data do not appear to be particularly skewed.
Slope Figure 5: Exchange Function Slopes according to Data Timeframe 6 3.5 3 2.5 2 1.5 1 2011/Q1 2011/Q2 2011/Q3 2011/Q4 2012/Q1 2012/Q2 2012/Q3 2012/Q4 Performance Period (Ending Quarter) 6 Figure 5 examines nationwide performance and the exchange slope. Performance has continuously improved over the period of the graph and, as performance has improved, the exchange slope function has decreased.