Updates in ACS NSQIP Modeling, July 24, Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons

Transcription

1 Updates in ACS NSQIP Modeling, July 24, 2012 Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons

2 Overview What is a statistical model and why is it needed? 1. The central task of ACS NSQIP is to provide fair comparisons of surgical quality across hospitals. 2. Since every patient, and every hospital s patient pool, is different, we need to compensate for those differences. 3. A statistical model is a mathematical recipe that gives direction on how to apply compensation. 4. There are many different ways to generate the recipe and none of them will be perfect in the way it compensates. 5. Nevertheless, there is no justification for not compensating and for not using the best recipe available.

3 Overview This is our third SAR cycle using hierarchical statistical methods. ACS NSQIP modeling incorporates: Risk Adjustment for procedure and patient Hospitals are given credit for doing higher risk procedures ( 350 CPT groups) on sicker patients (patient level risk indicators). (Logistic modeling, which we used previously, stopped here.) Shrinkage Adjustment When sample sizes are small we combine the limited information we have for the hospital with what we know about all hospitals. After shrinkage adjustment hospitals are no longer assigned extreme/unreasonable values (e.g., when there is either no event or there is 1 event for the 1 case submitted).

4 Overview For purposes of review, let s step through two thought experiments demonstrating the need for both risk adjustment and shrinkage adjustment. These are artificial examples, but do represent what actually happens, though to a lesser degree.

5 Overview The Need for Risk Adjustment Because patients and procedures are not randomly assigned to hospitals, imbalance in patient co morbidities and procedure complexity/risk are inevitable and requires statistical compensation. Simpson s paradox is an extreme example of improper conclusions which might be drawn when there isn t adjustment for a lurking risk variable in the presence of imbalance.

6 Overview The Need for Risk Adjustment Consider that we have two hospitals, Hospital A has a 1.0% mortality rate for 1000 patients, and Hospital B has a 1.6% mortality rate for 1000 patients. Which hospital is better? Hospital A Hospital B 10/1000=1.0% 16/1000=1.6% ASA Class 1,2 1/500=0.2% 0/200=0.0% ASA Class 3 3/300=1.0% 2/300=0.7% ASA Class 4,5 6/200=3.0% 14/500=2.8% At every level of ASA-associated risk, the bad hospital (B) outperforms the good hospital (A). This is an example of Simpson's Paradox - the relationship for a group as a whole (A better than B) is reversed for the all subgroups (B better than A). Risk-adjustment is needed to compensate for imbalance associated with lurking risk variables.

7 Overview The Need for Shrinkage Adjustment When sample sizes are small and/or event rates low, the logistic model quality metric (the Observed/Expected ratio) that is assigned to hospitals is often times too extreme. Consider the situation where: Each of 100 hospitals contributes 1 case The true situation is that all hospitals have equal quality The overall event rate is 5%. Under this scenario, the most likely outcome is that 95 hospitals will report 0 events, and 5 hospitals will report 1 event. But given that all hospitals have the same quality, are these estimates reasonable or useful?

8 Overview The Need for Shrinkage Adjustment 100 All hospitals have a true event rate of 5%. Each hospital submits 1 case. The expected distribution of rates is shown by the black circles. Hospitals with Assigned Event Rate Observed event rate There will be 95 hospitals with an O/E ratio of 0.0 and 5 hospitals with a very largeo/e ratio Observed Hospital Event Rate

9 Overview The Need for Shrinkage Adjustment 100 All hospitals have a true event rate of 5%. Each hospital submits 1 case. The expected distribution of rates is shown by the black circles. Hospitals with Assigned Event Rate Observed event rate Event rate after shrinkage adjustment There will be 95 hospitals with an O/E ratio of 0.0 and 5 hospitals with a very largeo/e ratio. Odds ratios from a hierarchical model with shrinkage adjustment, will be very close to Observed Hospital Event Rate

10 Overview ACS NSQIP hierarchical modeling, with shrinkage adjustment, represents the state of the art standard for provider profiling. Site summary reports and hospital specific bar plots, provide a comprehensive and consistent template for reporting statistical results. But we are always looking for ways to improve model validity, metric reliability, ease of use, and value. Tomorrow s methods and reports will be different. That s a good thing.

11 Special Bonus Section 10 Reasons why you really don t want to go back to O/E ratios and caterpillar plots 10. Logistic model O/E ratios don t have shrinkage and are unstable you re more likely to look better, or look worse, than you actually are. 9. Logistic model O/E ratios fail to account for lack of independence of patients within hospitals (minor issue). 8. The confidence interval we can estimate for the O/E ratio is less accurate than the confidence interval we can estimate for the hierarchical odds ratio. 7. Odds aren t hard to understand. The odds is the number cases with event/number of cases without event. If there are 5 deaths in 100 cases the odds is 5/95=0.053, which is very close to the rate which is 5/100=0.050.

12 Special Bonus Section 10 Reasons why you really don t want to go back to O/E ratios and caterpillar plots 6. Hierarchical odds ratios aren t hard to understand. The odds ratio is the odds at your hospital compared to the odds at the average hospital. Less than 1.0 is good; greater than 1.0 is bad. Just like the O/E ratio. 5. Conceptually, you can think of odds ratios as O/E ratios. Odds ratio = odds(h) / odds(avgh) = observed odds / expected odds O/E = (O/n) / (E/n) = observed rate / expected rate and we know that when rates are low, odds rate so the odds ratio rate ratio

13 Special Bonus Section 10 Reasons why you really don t want to go back to O/E ratios and caterpillar plots 4. Caterpillar plots are too cumbersome when there are 100 s of models and 100 s of hospitals. Even if the PowerPoints automatically locate your hospital, the plot is mostly wasted space. 3. Caterpillar plots can exaggerate inter hospital differences by giving equal weight to differences anywhere in the distribution (ignore density) the caterpillar x axis is rank, which makes differences in the center look larger than they are (very small differences in O/E ratios). 2. Bar plots present better information than caterpillars, in a much denser, accessible format.

14 Special Bonus Section 10 Reasons why you really don t want to go back to O/E ratios and caterpillar plots Recognizing that using the best available statistical technique is not just a matter of style, but a matter of substance 1. It s the right thing to do.

15 Special Bonus Section 10 Reasons why you really don t want to go back to O/E ratios and caterpillar plots There s always going to be tension between what is the theoretically best statistical analysis and what is possible to implement in actual practice resources and community understanding/acceptance are legitimate concerns. Right now that sweet spot for surgical quality profiling is not logistic model O/E ratios, but hierarchical model odds ratios.

16 Overview Behind the scenes, we are continuously improving the efficiency of our code and procedures and, with increased resources, we are just able to keep pace with needs for new models in new programs. Currently we build hundreds of models for: Essentials; Targeted; (DOD versions); Measures (Essential Parsimonious); Florida; Pediatric; Risk calculators; Real time risk adjustment.

17 Essentials We have added 5 new Essential models for return to operatingroom (ROR) for General/Vascular, General, Colorectal, Vascular, and All Cases. This brings the number of Essential models from 63 to 68. (But actually 67 in this SAR cycle, as we ve had to temporarily suspend Subspecialty Ortho Mortality a reliable hospital effect could not be detected). Until hospitals have had a chance to evaluate the ROR results, and determine whether they are under or over coding ROR, these results should be used cautiously.

18 Targeted We now have 12, rather than 6, months of Targeted data. This has allowed us to successfully run 18 new models which brings the number of Targeted models from 36 to 52. (2 previously run models were no longer successful). There were no outliers in 29 of these 52 models. Many targeted models are tenuous (1 model was lost when DOD data were added). In the next SAR, we will begin incorporating target specific predictors for vascular surgeries, as feasible. Over time we will expand the outcomes evaluated. With 11 standard outcomes, target specific outcomes, and 34 targets, there are many hundreds of potential new models.

19 FSCI/CMS/Measures Parsimonious modes are now Measures. Important to review the table of new SAR model naming conventions. Model names that were appropriate at the inception of ACS NSQIP are now either uninformative or confusing e.g. Overall Mortality (also referred to as, Overall (non Multispecialty) 30 day Mortality ) is actually mortality for all cases in ACS NSQIP done by General or Vascular Surgeons. A better model name would be GV Mortality. New, more informative, model names are used in this SAR. Florida and CMS (Hospital Compare) programs includes models that are subsets of Measures. If you ve registered for any of the three HC outcomes, your HC results (average, better than average, worse than average) will be based on ACS NSQIP outlier status in the corresponding measure.

20 Naming Conventions Essentials Overall (General/Vascular) Overall Mortality Overall Morbidity Overall Cardiac Overall Pneumonia Overall Unplanned Intubation Overall Ventilator > 48 Hours Overall DVT/PE Overall Renal Failure Overall UTI Overall SSI Essentials General/Vascular GV Mortality GV Morbidity GV Cardiac GV Pneumonia GV Unplanned intubation GV Ventilator > 48 hours GV DVT/PE GV Renal Failure GV UTI GV SSI GV ROR See the Important Changes to the Report sections in this and the last SAR for a discussion of model construction and naming conventions.

21 Pediatric We have just reported on the same 9 models for 2011 data as we did for 2010 data. However, we have included many new predictors. We may be reporting on additional models in the next several months. Clinical issues & case eligibility are still evolving.

22 Risk Calculators Risk calculators are built on ACS NSQIP data. ACS NSQIP participants have their patients predicted probabilities adjusted for the hospital the same patient will have different predicted probabilities at different hospitals. This is unique among all published surgical risk calculators. Risk calculator site is: Major update in the works. (1) will bring in 2 more years of recent data; (2) evaluating a new risk calculator approach that will cover every ACS NSQIP primary surgery. If #2 successful, this will also be unique among all risk calculators.

23 Risk Calculators

24 Risk Calculators

25 Real Time Risk Adjustment There are advantages to real time risk adjustment. Results are immediate (but may be less reliable) Interventions can be immediate Choose a time interval for study. Non hierarchical O/E ratios are estimated using prior SAR regression equation (excluding hospital effect). Process might be applied to small data sets (due to limited time intervals). Therefore, we have incorporated an O/E shrinkage transformation. All things being equal, the shrinkage transformed O/E ratio will be closer to the subsequent hierarchical odds ratio than the untransformed O/E ratio.

26 Real Time Risk Adjustment What is relationship would you expect between an O/E ratio constructed in real time and the hierarchical odds ratio that you would later see in the SAR? Small N, more shrinkage 3 Hierarchial Odds Ratio 2 Hierarchial Odds Ratio Large N, less shrinkage O/E Ratio Rescaled Shrinkage Transformed O/E Ratio

27 Statistical Core ACS Statistical Staff Mark Cohen, PhD Lynn Zhou, PhD Kristopher Huffman, MS Vivian Fei, PhD Yaoming Liu, PhD Principal Consultant Bruce Hall, MD, PhD, MBA

28 Questions