Statistical Uncertainty in the Medicare Shared Savings Program Derek DeLia,¹ Donald Hoover,² Joel C. Cantor¹

Transcription

1 Medicare & Medicaid Research Review 2012: Volume 2, Number 4 A publication of the Centers for Medicare & Medicaid Services, Center for Strategic Planning Statistical Uncertainty in the Medicare Shared Savings Program Derek DeLia,¹ Donald Hoover,² Joel C. Cantor¹ ¹Rutgers University Center for State Health Policy ²Rutgers University Statistics/Institute for Health Objective: Analyze statistical risks facing CMS and Accountable Care Organizations (ACOs) under the Medicare Shared Savings Program (MSSP). Methods: We calculate the probability that shared savings formulas lead to inappropriate payment, payment denial, and/or financial penalties, assuming that ACOs generate real savings in Medicare spending ranging from 0 10%. We also calculate expected payments from CMS to ACOs under these scenarios. Results: The probability of an incorrect outcome is heavily dependent on ACO enrollment size. For example, in the MSSP two-sided model, an ACO with 5,000 enrollees that keeps spending constant faces a 0.24 probability of being inappropriately rewarded for savings and a 0.26 probability of paying an undeserved penalty for increased spending. For an ACO with 50,000 enrollees, both of these probabilities of incorrect outcomes are equal to The probability of inappropriate payment denial declines as real ACO savings increase. Still, for ACOs with 5,000 patients, the probability of denial is at least 0.15 even when true savings are 5 7%. Depending on ACO size and the real ACO savings rate, expected ACO payments vary from $115,000 to $35.3 million. Discussion: Our analysis indicates there may be greater statistical uncertainty in the MSSP than previously recognized. CMS and ACOs will have to consider this uncertainty in their financial, administrative, and care management planning. We also suggest analytic strategies that can be used to refine ACO payment formulas in the longer term to ensure that the MSSP (and other ACO initiatives that will be influenced by it) work as efficiently as possible. Keywords: Medicare, Econometrics, Health Care Organizations and Systems, Health Economics, Health Policy / Politics / Law / Regulation, Incentives in Health Care, Payment Systems: FFS / Capitation / RBRVS / DRGs / Risk Adjusted Payments etc., Health Care Costs, Health Care Financing / Insurance / Premiums doi: E1

2 Introduction The Patient Protection and Affordable Care Act (PPACA) authorizes the Center for Medicare & Medicaid Services (CMS) to develop initiatives aimed at controlling healthcare spending while simultaneously improving care coordination, health outcomes, and patient experiences (PPACA; P.L ). One of the most prominent of these initiatives governs how Medicare will interact with Accountable Care Organizations (ACOs) (Fleming, 2010). An important element of this interaction is the concept of shared savings, which is designed to align incentives between Medicare and providers. ACOs need the income derived from these savings to recover their relevant start-up and operational costs, which include investments in interoperable information technology and added personnel, such as physician extenders and patient navigators. To address these needs, CMS created the Medicare Shared Savings Program (MSSP), which provides a framework (within the statutory parameters set by the PPACA) for Medicare and ACOs to share any savings that are generated through improved care management, provided the ACO meets a variety of quality and other standards established by the program. On April 7, 2011, CMS released its proposed rules for the program (CMS, 2011a). After a 60-day comment period, CMS issued its response and final rules on October 20, 2011 (CMS, 2011b). Although many are enthusiastic about Medicare ACOs, public comments on the proposed rules revealed significant opposition to specific details of the MSSP (Evans, 2011). Opposition to these details has not been universal, however, as some have argued that hoped for reductions in the growth of Medicare spending will not occur unless Medicare sets high expectations and stringent rules (Ginsburg, 2011). In response to public comments, CMS ultimately changed some MSSP provisions. Although rule making for the MSSP has concluded, the final rules allude to the idea that Medicare policy toward ACOs is expected to evolve with the accumulation of experience. Thus, it remains important to assess how well the program is designed to meet its goals and how CMS might position itself to make future refinements. In light of Medicare s influence on reimbursement policies throughout the health sector, this assessment can also be beneficial to the design of ACO policies by private insurers and state Medicaid programs. One aspect of the MSSP that has not received much attention is the extent to which random factors beyond the control of an ACO can influence healthcare spending (e.g., positive or negative responses to treatment). The presence of such factors may falsely generate apparent savings in an ACO where spending rose, leading to inappropriate payment. For example, an inefficient ACO may have an unusual year where many patients who usually require an extensive regimen of treatment regain their health much more quickly (and with fewer medical resources) than expected. Alternatively, these factors may falsely generate apparent spending increases within an ACO that saved money, leading to inappropriate denial of payment. For E2

3 example, a number of patients in an efficient ACO may experience unforeseen and inevitable complications in their illnesses leading to greater use of medical resources. Payment formulas that minimize the chance of inappropriate payment tend to increase the chances of inappropriate payment denial and vice versa. Thus, it is important for ACO policy to incorporate optimized statistical formulas that balance these two competing risks. To that end, this paper provides a statistical analysis of the MSSP formulas that are designed to address random variation in healthcare spending. First, we summarize the relevant formulas and place them into a statistical framework where analytic details can be outlined explicitly. 1 Second, we adapt our framework so that it conforms to the decision rules for statistical inference in the MSSP. Third, we use these decision rules to analyze both the likelihood that an ACO would be rewarded and the expected size of financial reward under alternative scenarios regarding ACO expenditure performance. Finally, we discuss policy implications and how our more comprehensive modeling framework might be incorporated into future ACO reimbursement formulas. Summary of ACO payment rules under the MSSP The MSSP specifies two models from which ACOs could choose to determine how shared savings are administered (Exhibit 1). The first, called the one-sided model, would allow an ACO to retain part of the savings it produces within its population of assigned Medicare beneficiaries. The share of savings retained by the ACO could reach a maximum of 50% contingent on meeting quality standards and other provisions. Under this model, ACOs would not be held accountable for any Medicare spending increases. However, ACOs in the one-sided model would still face financial risk, since they would need to recover their start-up and operating costs, which could be substantial (Watteau, 2011). The second, two-sided, model would place the ACO at risk for financial penalties if per capita Medicare spending increases. In exchange, the ACO would be eligible for up to 60% of shared savings contingent on meeting quality and other standards. After a phase-in period, the ACO would have to pay a penalty equal to 100% minus the final shared savings rate (e.g., if final shared savings rate is 60%, the penalty would be 40% of any spending increases). ACOs opting for the one-sided model in the first agreement period would have to switch to the two-sided model in subsequent periods. Under the proposed rules, ACO savings would be measured by a sequence of steps. First, CMS determines the benchmark level of per capita spending within the ACO. This is done by taking a weighted average of the most recent three years of per capita spending among patients 1 Our summary of the MSSP formulas is based on the final MSSP rule (CMS, 2011b). If an earlier proposed rule remained intact and was not fully explained in the final rule, then we based our summary on the details stated in the proposed rule (CMS, 2011a). E3

4 who would be assigned to the ACO according to pre-existing healthcare utilization patterns. The most recent year is weighted at 0.6, the middle year at 0.3, and the earliest year at 0.1 (CMS, 2011a). To adjust for medical cost inflation, the middle and least recent years are trended forward using the national growth rate in Medicare Part A and B expenditure for fee-forservice beneficiaries nationally (CMS, 2011a). Second, the benchmark value is updated using the projected absolute amount of growth in national per capita expenditures for Parts A and B services under the original Medicare FFS program (CMS, 2011a). Exhibit 1. Key provisions in proposed shared savings rules. One-sided Model Two-sided Maximum shared savings rate for the ACO 50% 60% Minimum savings rate (MSR) 5,000 patients ,000 patients ,000 patients Minimum loss rate 5,000 patients N/A ,000 patients N/A ,000 patients N/A Application of shared savings First-dollar First-dollar Application of shared losses N/A First-dollar SOURCE: CMS, 2011a; CMS, 2011b. Average per capita expenditures within the ACO in the performance year are then compared to the updated benchmark. In both models, if performance year expenditures are less than the benchmark (by an amount described below), then the ACO would be eligible for a financial reward. In the two-sided model, if performance year expenditures are greater than the benchmark (by an amount described below), then the ACO would pay a penalty. Additionally, beneficiaries with extremely large expenditures will have their spending amounts truncated at the 99 th percentile level for each year to minimize variation from catastrophically large claims (CMS, 2011a). Also, all expenditure amounts are to be risk adjusted using CMS s Hierarchical Condition Categories (CMS-HCC) with additional demographic adjustments in the performance year [which are discussed further below] (CMS, 2011a). In response to the problem of normal variation (i.e., random fluctuations in healthcare spending not captured by HCCs), CMS requires ACOs to achieve a minimum savings rate (MSR) before expenditure savings are officially recognized (CMS, 2011a). For example, if the MSR is set at 2%, the ACO would have to achieve spending growth that is two percentage points lower than the benchmark for these savings to be credited. In the one-sided model, the MSR E4

5 threshold is set between 0.02 and 0.039, with higher thresholds for smaller A COs to account for the fact that savings performance is measured with less precision when A COs have a smaller number of patients (Exhibit 1). In the two-sided model, the MSR threshold is set at 0.02 regardless of ACO size. (This makes the MSR lower for smaller A COs, which compensates them for taking on the financial risks inherent in the two-sided model.) In the two-sided model, the threshold for establishing whether the ACO has generated losses instead of savings is also set at The final rules stipulate that A COs will be rewarded for credited savings on a "firstdollar" basis in both models (CMS, 20llb). The final rules also specify that in the two-sided model, A COs would be responsible for a share of any credited losses (i.e., spending increases) on a first-dollar basis (CMS, 20llb). Statistical assessment of ACO payment rules under the MSSP Our ultimate goal is to analyze the likelihood that an ACO would meet the relevant MSR threshold under alternative scenarios regarding its underlying true performance in controlling healthcare spending. We also wish to estimate the size of the expected ACO payment under these scenarios (described below). This analysis requires detailed knowledge about random variability in the ACO savings rate (ASR), which we specify mathematically as: ASR = [(Y 8 +A)- Yp ]j( Y 8 +A) (1) The variable Y 8 is the A CO's risk-adjusted baseline per capita spending level (i.e., 3-year weighted average), Y is the A CO's risk-adjusted performance year per capita spending level, p and A is the projected absolute amount of growth in per capita Medicare expenditures nationally as described above. But variability in the ASR in Equation 1 is driven by a complicated mixture of variability in the random variables Y 8, Y p, and A. Variability in A is driven by factors affecting the growth in Medicare spending nationally such as progression of illness among Medicare beneficiaries and changes in medical technology. Variability in Y and Y are driven by a number of 8 p additional random factors at the patient and ACO levels (including factors affecting the weighted components used to construct Y 8 ). To understand this variability, we use a variance components model of ACO spending (Diggle, Heagerty, Liang, & Zeger, 2002), which is described fully in the Technical Appendix. To understand how A COs would be rewarded or penalized under the MSSP, we need to calculate the probability that ASR crosses the relevant MSR threshold given a specified level of true ACO savings. The key elements of the calculation are illustrated in Exhibit 2, which shows the probability distribution for the measured ASR, under the assumption that the true underlying savings rate is zero. Although the distribution is centered at the true savings rate of zero, the measured ASR could be positive or negative due to normal variation. In the one-sided DeLia, D, Hoover, D, Cantor,! C ES

6 model, the ACO would not be rewarded for savings unless the measured ASR crosses the relevant MSR threshold, which we call T n (where n represents the number of patients assigned to the ACO). Exhibit 2. Probability that an ACO is inappropriately rewarded for measured savings due to normal variation in healthcare expenditures (One-side model). Probability 0 l ACO savings rate (ASR) 0 ACO savings rate (ASR) Tn T n SOURCE: Authors diagram. CMS sets the MSR threshold (T n ) to limit the probability that an ACO with true savings of zero would have a measured ASR above this threshold. This probability is represented by the area under the probability distribution to the right of T n in Exhibit 2. CMS developed the MSR thresholds shown in Exhbit 1 so that this probability would equal 0.1 for ACOs with 5,000 assignees (the minimum allowable under the PPACA), 0.05 for ACOs with 20,000 assignees, and 0.01 for ACOs with 50,000 beneficiaries. Other MSRs are determined by linear interpolation for all other ACO sizes less than 60,000. For ACOs with 60,000 patients or more, the MSR is set at In the two-sided model, the MSR, to establish savings, is set at 0.02 for ACOs of any size (i.e., number of assignees). To establish the presence of losses for which the ACO would pay penalties under the two-sided model, a similar threshold is set at negative E6

7 Scenario analysis Using this framework, we assess the statistical risks faced by CMS and ACOs in MSSP. First, we calculate the financial losses incurred by CMS when normal variation allows ACOs to be rewarded for savings that did not truly occur. Then we calculate financial losses to ACOs when they fail to be rewarded for savings that did occur. In our calculations, we consider true underlying ACO savings rates that vary from 0 to 0.1 (i.e., 10% savings).we also calculate the corresponding probabilities of failing to be rewarded and of paying a financial penalty under the two-sided model. In all of these calculations, we consider a variety of different ACO enrollment sizes. The formulas for our calculations are derived in the Technical Appendix. Exhibit 3 shows the expected financial risk to CMS due to normal variation. There is clearly less financial risk to CMS on average in the two-sided model, partly because ACOs could pay a penalty to CMS in this model. Under both models, an increase in ACO size initially increases CMS s financial liability due to the larger enrollment of patients to which inappropriate per capita payments would apply. But after 10,000 patients, increases in ACO size decrease the average financial risk to CMS, as the greater precision in savings measurement makes it increasingly unlikely that an ACO with no real underlying savings would receive a financial reward. Overall, the expected financial liability to CMS per ACO due to normal variation is less than $200,000 in the one-sided model and less than $150,000 in the two-sided model across all ACO sizes. Exhibit 3. Financial risks to CMS when measured savings are driven solely by normal variation. Expected loss for CMS (in $1,000's) $250 $200 $150 $100 $50 One-sided model Two-sided model $0 ACO enrollment SOURCE: Authors calculations based on assumptions specified in the text and Technical Appendix. E7

8 Exhibit 4 shows the risks faced by ACOs. Echoing the results in Exhibit 3, CMS is fairly well protected from inappropriate ACO payments in the case where the ACO produces no real savings or losses. Exhibit 4. Financial risks and expected income for ACOs with varying levels of true savings. ACO size True savings One-sided model Probability that ACO would not be rewarded for savings Expected ACO income Probability that ACO would not be rewarded for savings Two-sided model Expected ACO income Probability ACO would have to pay penalties for what appear to be losses 5, $156, $114, , $184, $120, , $82, $51, , $449, $705, , $1,548, $2,257, , $3,559, ,882,041 < , $679, $1,032, , $2,894, $3,841,500 < , $7,831, $9,794,901 < , $960, $1,377, , $4,381, $5,474,370 < , $11,597, $14,005,192 < , $1,276, $1,734, , $5,771, $7,008,362 < , $14,692,356 <0.001 $17,637,463 < , $1,613, $2,095, , $7,032, $8,459,470 < , <0.001 $17,642,774 <0.001 $21,171,500 < , $1,953, $2,456, , $8,229,663 <0.001 $9,878,945 < , <0.001 $20,583,498 <0.001 $24,700,199 < , $2,922, $3,526,993 < , <0.001 $11,761,999 <0.001 $14,114,400 < , <0.001 $29,405,000 <0.001 $35,286,000 <0.001 SOURCE: Authors calculations based on assumptions specified in the text and Technical Appendix. In the one-side model, the probability that an ACO would not be rewarded under this scenario matches exactly what CMS specified in setting MSR thresholds. In the two-sided model, CMS is somewhat less protected from paying an ACO inappropriately due to the lower MSR thresholds in this model. ACOs, however, face some risk of paying a penalty in the two-sided model even E8

9 under the assumption that no real losses are generated. This risk is especially high for ACOs with the smallest enrollment level considered (n = 5,000). ACOs that achieve modest savings (2 3%) face a substantial risk of not being rewarded in both the one-sided and two-sided models. This risk is much greater for smaller ACOs. In the two-sided model, ACOs that achieve modest savings face a non-zero risk of having to pay a penalty for losses, although this risk appears negligible for ACOs with 20,000 or 50,000 assignees. When the true underlying ACO savings rate increases to 0.04 and then again to 0.05, the risk to the ACO of being improperly denied a financial reward progressively declines. Also, the probability of inappropriately being charged a penalty becomes very small. Nevertheless, the probability of not being rewarded remains non-trivial except in the case of an ACO with 50,000 patients that achieves a true underlying savings rate of The negative risks facing ACOs diminish greatly when savings rates above 5% are achieved. Still, for an ACO with 5,000 enrollees, substantial risks of not being rewarded remain fairly high even if true savings are 7 10% percent. Not surprisingly, under both models expected ACO income is fairly small when the true savings rate is 0. Expected ACO income grows steadily with a greater savings rate and larger ACO size. In the scenarios where true savings are produced, expected ACO income is greater in the two-sided model relative to the one-sided model. This difference is due primarily to the lower MSR threshold in the two-sided model. This relationship reverses in the scenario where true savings are zero, since the two-sided model creates a fairly large probability that the ACO would have to inappropriately pay a penalty. Policy issues Discussion The PPACA requires CMS to develop a formula that will limit the extent to which Medicare will pay ACOs, inappropriately, for measured savings that are due solely to normal variation (PPACA; P.L ). Using statistical assumptions similar to those in the MSSP, our analysis suggests that the MSSP is fairly well designed for this purpose. We find as well that the risk of inappropriate payment to ACOs declines rapidly as ACOs grow beyond 20,000 enrollees. The MSSP does not appear to offer the same level of protection for ACOs that face the risk of being inappropriately denied payment when they do generate savings for Medicare. The risk of inappropriate payment denial is especially acute for smaller ACOs and for those that generate relatively modest Medicare savings. Moreover, in the two-sided model, some ACOs that save money for the Medicare program face a non-trivial risk of having to pay a penalty for apparent spending increases that are due to normal variation. E9

10 The statistical risks facing ACOs greatly reduce the expected income that an ACO can achieve within the 3-year agreement period between CMS and the ACO under the MSSP. In its rule making, CMS assumes that the initial investment and first year operating costs for an ACO are approximately $1.8 million (CMS, 2011a). This assumption is disputed, however, by the American Hospital Association (AHA), which estimates the relevant first year costs in the range of $11.6 to $26.1 million (Watteau, 2011). Using the figure assumed by CMS, our calculations suggest that expected ACO income over the 3-year agreement period would be insufficient to recover these costs in many cases, particularly when an ACO is small (5,000 assignees) and savings are modest (<5%). Expected ACO income is much more likely to be insufficient using AHA s lower figure and always insufficient in our analysis using AHA s higher figure (for example when ACO size is < 50,000 and savings < 10%). Although other factors such as present value discounting and tolerance for risk are important considerations that we have not assessed, our calculations suggest that the MSSP rules may create disincentives for ACOs anticipating real, but modest, savings to participate. These disincentives are especially strong for smaller ACOs. As a result, the proposed rules may limit interest in ACO participation in Medicare primarily to very large coalitions of providers. A program that becomes skewed towards large ACOs raises a number of important policy concerns. Large ACOs may accumulate a great degree of monopoly power leading to higher prices and inefficient resource allocation in private markets (Richman & Schulman, 2011). Even in Medicare where prices are set nationally, ACOs that become too large could limit choices and care options available to patients. Also, a very large ACO that covers a wide geographic area might overlook very local and specialized needs of specific communities in a way that is less likely to occur with a smaller more locally focused ACO. The MSSP does include provisions to monitor ACOs on the basis of healthcare quality, patient access, and anticompetitive behavior. It is also possible that contracting with many small ACOs could raise administrative costs for CMS, and smaller ACOs might operate at a scale that is too small to generate hoped for efficiencies in care management. Thus far, ACOs entering the MSSP include a fairly diverse mix of small and large hospital and physician led ACOs (Centers for Medicare & Medicaid Services, 2012a; Evans, 2012). ACOs selected during the first application wave in April 2012 vary in size from approximately 5,000 to 70,000 Medicare beneficiaries (Centers for Medicare & Medicaid Services, 2012b). This diversity will provide an opportunity to monitor the relationship between ACO size and efficiency, and the extent to which the MSSP formulas reward and penalize ACOs as intended. ACO-specific considerations Although our analysis considers a wide range of ACO performance scenarios, it does not include the specific details that would be required for a particular group of providers to decide whether it would be advantageous to form an ACO for participation in the MSSP. To assess whether it E10

11 should enter the MSSP, an ACO would need to consider its ability to improve care coordination and reduce avoidable and unneeded service utilization, as well as other factors that could affect its performance measurement under MSSP rules. It is well established that the level, year-to-year variability, and growth trend in per capita healthcare spending varies substantially across geographic areas within the United States (Fisher, Bynum, & Skinner, 2009; Fisher et al., 2003; Medicare Payment Advisory Commission, 2011). Providers in areas with a lower level of spending might find it easier to achieve a given MSR, since this would represent a smaller level of spending reduction (e.g., 3% savings from $8,000 is less than 3% savings from $12,000). However, it might also be the case that low spending areas are already providing care very efficiently, making it difficult to achieve continued savings under the MSSP. As shown in the Technical Appendix, an important feature in setting the MSR thresholds is the level of random, or normal, variation in per capita healthcare spending, which is measured by the coefficient of variation (CV). For an ACO that truly achieves a savings rate above its applicable MSR, the savings would be easier to detect in areas where the local CV is less than the national CV used to set the MSR. Similarly, in areas where the local trending factor and projected spending growth are less than the corresponding national figures, the target level of trended & projected baseline spending would be inflated relative to true local conditions. Thus, it would be easier for ACOs in these areas to achieve credited savings under the MSSP relative to other areas. Also, the MSSP rewards ACOs based on how well they reduce spending relative to the national rate of growth in per capita healthcare spending. Thus, an ACO that reduces spending relative to a local growth trend would find it harder to be rewarded for doing so if local growth in spending is substantially faster than the national growth rate. (For example, if the national rate is 5% and an ACO reduces local growth from 10% to 6%, it would not be rewarded for doing so.) More broadly, a group of providers considering the formation of an ACO might use a modified version of the analytic framework presented above and in the Technical Appendix to assess the specific financial risks that they would face if they participated in the MSSP. In so doing, they could use their own local data regarding recent trends and variability in healthcare spending and consider saving and loss scenarios that are more relevant to their specific care management strategies. Quality measurement In our analysis, we assume that ACOs meet all of the healthcare quality benchmarks established by CMS, making them eligible for the maximum share of credited savings and the minimum share of credited losses. In practice, many ACOs could fall short of these benchmarks thereby reducing financial gains to ACOs and financial liabilities to CMS. Just as there is normal E11

12 variation in per capita spending, there could be similar variation in healthcare quality. Thus, it would be useful in future analyses to determine whether a minimum ACO size is needed to reliably measure ACO quality performance. Simplifying assumptions The numerical calculations above depend on a number of assumptions that are similar to those made in the MSSP and prior research on similar issues. Nevertheless, these assumptions are highly simplified and in need of refinement as the MSSP evolves. First, all of our calculations are made conditional on baseline ACO-level Medicare spending and projected growth in national per capita Medicare spending. If these quantities are not known at the time the ACO agrees to participate in Medicare, or otherwise themselves subject to random error, then these quantities are more appropriately viewed as random variables adding to the variability and complexity in the statistical calculations outlined above (including, for example, regression to the mean among very high or low cost ACO patients). This would add another layer of uncertainty to expected financial transfers between CMS and ACOs. Similarly, healthcare spending quantities are risk-adjusted in the MSSP using the CMS- HCCs. Since the HCCs are based on other statistical models (Pope et al., 2004), risk adjustment adds another layer of random variation that can affect observed ACO savings. The final MSSP rules specify that the growth rate trending factor and benchmark update will be calculated separately within four categories of Medicare beneficiaries: end-stage renal disease, disabled, aged dual eligibles, and aged non-dual eligibles (CMS, 2011b). These quantities might also be unknown at the time of ACO startup and thus would have to be estimated as well. Second, to be consistent with the rules for statistical inference in the MSSP, our detailed calculations in the Technical Appendix include a number of simplifying assumptions that may not be met in practice. First, we did not account for the hierarchical structure of patients clustered within groups of ACO providers and, instead, assumed that all observations within the same group are statistically independent. If this is not the case, then our analysis underestimates the variance of the measured ACO savings rate (ASR), and thus, understates the level of statistical uncertainty in the MSSP formulas. We also assume that different patients appear in the baseline and performance year spending calculations. This assumption is valid to the extent that individuals die, move away, or move into the ACO s service area in the middle of the contracting period. Most patients, however, are likely to remain in the ACO service area for an extended period of time. Among this stable set of ACO patients, individuals with high spending in one year are likely to have high spending in subsequent years. If so, our analysis overestimates the variance of the measured ASR, and thus, overstates the level of statistical uncertainty in the MSSP formulas. Third, our model overlooks some additional technical details that might be important in the measurement of the ASR. Like CMS, we appealed to the Central Limit Theorem to use the normal distribution in our calculations. This might be problematic, however, given the large E12

13 skewness that tends to be common in healthcare spending data (Manning, 1998; Mullahy, 1998). Although the theorem technically still would apply, it is not clear whether the ACO sizes specified above would be sufficiently large with skewed data. Despite the potential problems associated with very large ACOs noted above, it might be the case that only large ACOs would have sufficient enrollment to measure their performance with the required precision. In addition, CMS s strategy of censoring individual-level healthcare expenditure at the 99 th percentile raises similar technical issues regarding the analysis of truncated probability distributions that have not been considered. This truncation could be especially problematic for ACOs with care coordination models that focus specifically on patients in the upper tail of the healthcare spending distribution (a potentially very valuable care management strategy in light of the share of total spending by the most expensive patients). Finally, we implicitly assume that the number of patients served by the ACO remains fixed over time. This assumption will surely be violated in practice as patients move in and out of the ACO service area, new patients become eligible for Medicare, and others die during the evaluation period. The number of patients used in the ASR calculations also depends on the method that CMS will use to assign patients to the ACO in the baseline and performance periods, a method that CMS acknowledges may be subject to change as the ACO policy evolves (CMS, 2011a). In theory, it is possible to address all of these modeling issues and improve the precision with which ACO savings are measured. Medicare claims files (which CMS will use to implement the MSSP) contain most of the information needed to test the validity of simplifying assumptions, estimate needed parameters, and determine the required sample sizes for reliable statistical inference. Other issues, particularly those regarding ACO and provider-level group effects will be difficult to address until after ACO experience accumulates. Moreover, variation in per patient spending eventually might be reduced as providers adopt best practices and as inefficient care processes are implemented less frequently. Ultimately, our analysis suggests there is greater statistical uncertainty in MSSP than previously recognized. In the short run, CMS and ACOs will have to consider this uncertainty in their financial, administrative, and care management planning. In the long run, it will be important to refine ACO payment formulas to ensure that the MSSP, and other ACO initiatives that will be influenced by it, work as efficiently as possible. Correspondence Derek DeLia, Associate Research Professor, Rutgers University, Center for State Health Policy, 112 Paterson Street, Room 540, New Brunswick, NJ, 08901, ddelia@ifh.rutgers.edu, Tel Acknowledgments We acknowledge research support from the Agency for Healthcare Research and Quality (Grant no. R24- HS019678). We also acknowledge helpful comments from Gregory Pope, Stuart Gutterman, and Sujoy Chakravarty. E13

14 References Centers for Medicare & Medicaid Services (2012a, April 10). New Affordable Care Act program to improve care, control Medicare costs, off to a strong start. Retrieved from heckdate=&checkkey=&srchtype=1&numdays=3500&srchopt=0&srchdata=&keywordt ype=all&chknewstype=1%2c+2%2c+3%2c+4%2c+5&intpage=&showall=&pyear=&ye ar=&desc=false&cboorder=date Centers for Medicare & Medicaid Services (2012b, April 10). Summaries of ACOs selected for April 1, 2012 start-up. Retrieved from CMS, Medicare Program, Medicare Shared Savings Program: Accountable Care Organizations, 73 Fed. Reg (proposed April 7, 2011a) (to be codified at 42 C.F.R. pt. 425). CMS, Medicare Program, Medicare Shared Savings Program: Accountable Care Organizations; Final Rule, 42 C.F.R. 425 (2011b). Diggle, P. J., Heagerty, P. J., Liang, K. Y., & Zeger, S. L. (2002). Analysis of longitudinal data (2nd ed.). New York: Oxford University Press. Evans, M. (2011). Hell no, we won t ACO. Providers, once eager to sign up for new program, say proposed rule on accountable care is a deal-breaker. Modern Healthcare, 41(23), 6 7. PubMed Evans, M. (2012). Doctors on deck. ACOs led by doctors seek to manage costs, quality and hospital relationships. Modern Healthcare, 42(16), 6 7. PubMed Fisher, E. S., Bynum, J. P., & Skinner, J. S. (2009). Slowing the growth of health care costs Lessons from regional variation. The New England Journal of Medicine, 360(9), doi: PubMed Fisher, E. S., Wennberg, D. E., Stukel, T. A., Gottlieb, D. J., Lucas, F. L., & Pinder, E. L. (2003). The implications of regional variations in Medicare spending. Part 1: The content, quality, and accessibility of care. Annals of Internal Medicine, 138(4), PubMed Fleming, C. (2010, July 27). Health policy brief: Accountable care organizations. Retrieved from Ginsburg, P. B. (2011). Spending to save ACOs and the Medicare Shared Savings Program. The New England Journal of Medicine, 364(22), doi: PubMed Manning, W. G. (1998). The logged dependent variable, heteroscedasticity, and the retransformation problem. Journal of Health Economics, 17(3), doi: PubMed E14

15 Medicare Payment Advisory Commission (2011). Report to the Congress: Regional variation in Medicare service use. Washington, DC: Medicare Payment Advisory Commission. Mullahy, J. (1998). Much ado about two: Reconsidering retransformation and the two-part model in health econometrics. Journal of Health Economics, 17(3), doi: PubMed Pope, G. C., Kautter, J., Ellis, R. P., Ash, A. S., Ayanian, J. Z., Lezzoni, L. I.,... Robst, J. (2004). Risk adjustment of Medicare capitation payments using the CMS-HCC Model. Health Care Financing Review, 25(4), PubMed Richman, B. D., & Schulman, K. A. (2011). A cautious path forward on accountable care organizations. Journal of the American Medical Association, 305(6), doi: PubMed Watteau, M. (2011, May 13). New study finds the start up costs of establishing an ACO to be significant. Retrieved from E15

16 MMRR 2012 Volume 2, Number 4 Medicare & Medicaid Research Review 2012 Volume 2, Number 4 Mission Statement Medicare & Medicaid Research Review is a peer-reviewed, online journal reporting data and research that informs current and future directions of the Medicare, Medicaid, and Children s Health Insurance programs. The journal seeks to examine and evaluate health care coverage, quality and access to care for beneficiaries, and payment for health services. U.S. Department of Health & Human Services Kathleen Sebelius Secretary Centers for Medicare & Medicaid Services Marilyn Tavenner Acting Administrator Editor-in-Chief David M. Bott, Ph.D. The complete list of Editorial Staff and Editorial Board members may be found on the MMRR Web site (click link): MMRR Editorial Staff Page. Contact: mmrr-editors@cms.hhs.gov Published by the Centers for Medicare & Medicaid Services All material in the Medicare & Medicaid Research Review is in the public domain and may be duplicated without permission. Citation to source is requested. SSN: doi: E16

17 Statistical Uncertainty in the Medicare Shared Savings Program Derek DeLia, 1 Donald Hoover, 2 Joel C. Cantor1 1Rutgers University-Center for State Health Policy 2Rutgers University-Statistics/Institute for Health Supplement doi: Technical Appendix: Statistical Model of ACO Payment Rules in the Medicare Shared Savings Program This technical appendix describes in detail the methodology we use to assess the statistical properties of the ACO payment mechanisms in the Medicare Shared Savings Program (MSSP). As described in the main text, the ACO savings rate (ASR) can be written as: ASR = [(Y8 +A)- Yp]j(Y8 +A) (Al) where Y8 is the ACO's risk-adjusted baseline per capita spending level (i.e., 3-year weighted average), Yp is the ACO's risk-adjusted performance year per capita spending level, and A is the projected absolute amount of growth in per capita Medicare expenditures nationally, introduced in the main text. In the one-sided model, the ACO is rewarded if ASR > Tn where Tn is the MSR threshold set by CMS for an ACO with n assignees. (The method for determining Tn is described below.) In the two-sided model, an ACO would be rewarded if ASR > 0.02 and would pay a penalty if ASR < Sources of random variability in the payment formula Our ultimate goal is to analyze the likelihood that an ACO would meet the relevant MSR threshold under alternative scenarios regarding its underlying true performance in controlling healthcare spending. We also wish to estimate the size of the expected ACO payment assuming the ACO has met the MSR threshold in each scenario. This analysis requires knowledge about random variability in ASR, which is a complicated function of the random variables Y8, Yp, and A. 1This approach follows the general framework originally proposed by Fisher et al. (2009). SSN doi hnp//dx.doi.org/ /mmrr s04 El

18 Variability in A is driven by factors affecting the growth in Medicare spending nationally, such as progression of illness among Medicare beneficiaries and changes in medical technology. Variability in Y8 and Yp are driven by a number of additional random factors at the patient and ACO levels (including factors affecting the weighted components used to construct Y8 ). To understand this variability, we use the following variance components model of ACO spending: (A2) where Y;t is risk and growth trend adjusted spending (as described in the main text) for ACO patient i at time t, and 11t is the mean ACO spending per patient at time t, where t=l, 2, 3 are the baseline years and t=4 is the first performance year. If the ACO is effective at reducing growth and 113. trend/risk-adjusted healthcare spending, then 114 would be substantially less than 11v 112, The term E;t is a random deviation, or error term, for patient i at time t. Deviations can be specific to the ACO, individual patients, or time. Thus, we model E;t as (A3) where Ut is the random deviation in mean growth trend/casemix-adjusted healthcare spending at time t (experienced by all ACO patients) and V;t is the corresponding deviation that is specific to patient i at time t. Estimation and inference regarding 11t requires assumptions about the components of E;t Following common practice in variance components and hierarchical linear modeling (Diggle, Heagerty, Liang, & Zeger, 2002), we assume that E(ut) = E(v;t) = 0, V(ut) = aj, and V(v;t) = aj. Although we will need to modify them later, a fully specified model would typically include the following additional assumptions: 1. The ACO-specific (ut) and patient-specific (v;t) error components are uncorrelated. 2. Spending deviation for one patient is unrelated to spending deviation by other patients-i.e., cov(v;t, v1t) = For any particular patient, spending may be correlated from one time period to the next-i.e., cov(v;5, V;t) = y. 4. Random deviations in mean ACO health spending may be correlated over time i.e., cov(u5, ut) = 15. Substituting equations 2 and 3 into equation 1 and accounting for the 3-year baseline calculation gives (A4) where w = and r = 0.1 I7=1 E;tfn I7=1 E;z/n + o.6 I7=1 E;Jfn. SSN doi: /dx_dolorg/1 0_5600/m mrr_002_04_s04 E2

19 To understand how ACOs would be rewarded or penalized under the proposed rules, we need to calculate the probability that ASR crosses the relevant MSR threshold given a specified level of ACO savings. To do so, we define s as the proportion by which the ACO reduces average healthcare expenditure relative to (ω + A), which is the weighted average of mean ACO spending per patient in the baseline period plus the absolute change in per capita Medicare spending i.e., μ 4 = (1 s)(ω + A). In the one-sided model, the probability that the ACO would be rewarded can be written as: p 1 = Prob[ASR > T n µ 4 = (1 s)(ω + A)] (A5) Using the payment rules described above, the expected ACO payment can be written as: p 1 E[ASR ASR > T n ] (ω + A) n θ (A6) where θ is the share of savings retained by the ACO (and 1 θ is the share retained by CMS). Similar formulas can be written for the two-sided model. The probability of reward can be written as: p 2 = Prob[ASR > 0.02 µ 4 = (1 s)(ω + A)] (A7) In the two-sided model, there is also a probability that the ACO would have to pay a penalty, which can be written as: q 2 = Prob[ASR < 0.02 µ 4 = (1 s)(ω + A)] (A8) The expected ACO payment in the two-sided model (which incorporates the possibility of both ACO rewards and penalties) can be written as: p 2 E[ASR ASR > 0.02] (ω + A) n θ S + q 2 E[ASR ASR < 0.02] (ω + A) n θ L (A9) where θ S is the share of savings retained by the ACO and θ L is the share of losses that would be paid by the ACO in the form of a financial penalty. Calculation of formulas A5 A9 is very complex. In contrast, the decision rules for statistical inference in the MSSP ACO payment formulas are much simpler. Simplicity is clearly an important element of the proposed formulas because it makes them transparent to all participants and allows CMS to calculate ACO payments in a timely way. In the next section of this Technical Appendix, we outline the decision rules for statistical inference that are embedded in the MSSP and show how our modeling framework can be made to conform to these rules with additional simplifying assumptions. SSN: doi: E3

20 Statistical inference in the MSSP Statistical inference in the MSSP is driven by the MSR threshold T n. In the one-sided model, CMS specifies a sliding scale confidence interval (CI) based on the number of assigned beneficiaries (CMS, 2011). (In other words, the MSR represents the limit of a one-sided CI.) CIs are set at 90% for ACOs with 5,000 patients (the minimum allowable), 95% for ACOs with 20,000 patients, and 99% for ACOs with 50,000 beneficiaries. These CIs translate into the MSRs established by CMS (see Exhibit 1 in the main text). After setting these anchor MSRs, other MSRs are determined by linear interpolation for all other ACO sizes less than 60,000. For ACOs with 60,000 patients or more, the MSR is set at 2%. Using our statistical model above, we are able to replicate exactly the three anchor MSRs set by CMS if we add some (admittedly very strong) simplifying assumptions: 1. Projected absolute amount of growth in per capita Medicare spending (A) is not subject to random variation. 2. u t = 0 (i.e., no ACO random effects) n 3. i=1 v it = 0 for t=1, 2, 3 (i.e., patient-level deviations from the mean cancel each other perfectly in the baseline years) 4. γ = 0 (i.e., deviation from expected spending is not correlated within the same patient over time). These assumptions are similar to those made by Pope and Kautter (2011) in their more general analysis of potential ACO shared savings formulas. 2 (It should be noted that assumption 3, in particular, is quite strong but could be formulated differently, to the same effect, if all analyses are conditioned on the baseline values of v it ) Under these assumptions, ASR = 1 (μ 4 +v 4)/(ω + A) where v 4 = i=1 v i,4 /n. Applying the Central Limit Theorem to v 4, ASR is normally distributed with mean 1 μ 4 /(ω + A) and variance V(ASR) = (ω + A) 2 n V( i=1 v i,4 /n) = σ 2 v /[n(ω + A) 2 ] (A10) With these assumptions we calculate the probability that an ACO, which produces no real savings, will be rewarded inappropriately. In this case, ASR is normally distributed with mean equal to 0 [since μ 4 = (ω + A)] and variance as given in Equation 10. In doing this calculation, it is useful to view the ACO reimbursement mechanism as a statistical hypothesis test where the n 2 More specifically, Pope and Kautter (2011) consider the general problem of a payer entering a shared savings contract with a group of providers whose cost performance cannot be monitored perfectly, an issue known in the economics literature as the principle-agent problem. Their model is similar to ours in that they use the Central Limit Theorem as the basis for constructing a hypothesis test that would allow the payer to distinguish true savings from apparent savings that are driven by normal variation. SSN: doi: E4

21 null hypothesis is that the ACO produces no savings. If the null hypothesis is true, there is a chance that the ACO will meet the MSR threshold due to random chance alone. This is known as the probability of Type I error and can be written as: Prob[ASR > T n µ 4 = (ω + A)] (A11) Exhibit 2 in the main text shows diagrammatically the relationship between T n, the probability of Type I error, and random variability in ASR under the null hypothesis. Under CMS s proposed rules, T n is set so that the probability of Type I error, which is the area to the right of T n in Exhibit 2, equals 0.1 when n=5,000. This probability equals 0.05 when n=20,000 and it equals 0.01 when n=50,000. To complete the calculation, we require an estimate of σ 2 v /(ω + A) 2. This quantity reduces to σ v 2 μ2 under the null hypothesis and can be viewed as the square of the coefficient of 4 variation (CV) in patient expenditures within the ACO. Thus, we write the variance of ASR as V(ASR) = 1 n CV2. Using basic principles of statistical hypothesis testing, the null hypothesis is rejected if /( 1 CV) > n Z, where Z is the critical value of the standard normal distribution. The rejection rule can be expressed alternatively as ASR > ( 1 n CV) Z = T n (A12) where T n is the MSR threshold defined in the MSSP. As described above, MSRs are based on one-sided confidence intervals corresponding to significance tests at the 10%, 5%, and 1% levels. These CIs are associated with the critical Z values in Exhibit A1. If CV is set at 2.15, then Inequality A12 produces the MSRs set by CMS as shown in Exhibit A1. (Or alternatively, one can use the parameters set by the MSSP and solve for CV in Inequality A12 as CV = n(z /T n ) to derive CV=2.15.) Exhibit A1. Derivation of minimum savings rates in proposed ACO rules. ACO enrollment (n) Confidence level for one-sided confidence interval Critical value from standard normal distribution (Z ) Coefficient of variation (CV) MSR threshold (T n ) 5,000 90% ,000 95% ,000 99% SOURCE: Authors calculations derived from MSR thresholds in the Medicare Program. We note as well that our estimate of the coefficient of variation is the same order of magnitude as that reported by Pope and Kautter (2011) and is in the middle of the range provided by other SSN: doi: E5