Overview of CQI modeling for risk adjustment: Motivation and central concepts

Transcription

1 Overview of CQI modeling for risk adjustment: Motivation and central concepts Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons

2 This presentation s motivation Provide an accessible explanation of the quantitative aspects of CQI risk adjustment Why should the findings be trusted?

3 CQI s motivation Earnest A. Codman, MD (A Study in Hospital Efficiency: As Demonstrated by the Case Report of the First Five Years of a Private Hospital, 1918) So I am called eccentric for saying in public: That hospitals, if they wish to be sure of improvement, Must find out what their results are. Must analyze their results, to find their strong and weak points. Must compare their results with those of other hospitals.... Must welcome publicity not only for their successes, but for their errors...

4 Sun-Tzu, General (The Art of War, 5 th century BC, Sonshi Group translation) [01.01] Warfare is important to a nation. It is a matter of life and death. It is the way to survival or to destruction. So study it. [01.02] Study the five factors of warfare: Way, Heaven, Ground, General, and Law. Calculate your strength in each and compare them to your enemy's strengths.

5 Simple approach to results and comparative results Our hospital does 1,000 operations and has 30 events Hospital Rate = = = X 100% = 3.0% Hospital Odds = = = hospitals do 100,000 operations and have 2,000 events Comparison Rate = = X 100% = 2.0% Comparison odds = = Rate ratio = #EEEEEEEEEEEE NN Odds ratio (OR)= 30 1,000 #EEEEEEEEEEEE # NNNNNNNNNNNNNNNNNN 2, ,000 2,000 98, = 1.5. Hospital has 50% greater rate = Hospital has 51% greater odds What could be simpler!?

6 But there is a problem Many of us believe that other hospitals outcomes are based on But our hospital s outcomes are based on the fundamental requirement to adjust for differences in risk across hospitals, to risk adjust

7 If these hospitals were to do your procedures on your patients Fair comparison Apples to apples

8 Raw results (Non-risk-adjusted) Complex Mathematics To Risk-adjusted results

9 Raw results (Non-risk-adjusted) Complex Mathematics Many Complex Mathematics To Risk-adjusted results Different complex methodologies: Assumptions, robustness, accuracy, speed, metrics reported None yield a perfect representation of reality

10 None yield a perfect representation of reality Predictive perfection requires that: Risks are accurately estimated for every possible covariate pattern Requires the correct model form, but this in unknown Methods yield only asymptotically unbiased estimates Impossible to include all nonlinear and non-additive effects There is measurement error in predictors Innumerable unknown/inaccessible covariates preclude a true model But very close to reality close enough to guide QI

11 Our methodology 1. Regression 2. Use all data, from all hospitals, to make case-level predictions 3. Compare hospital s risk-adjusted predictions to actual outcomes 4. Include smoothing 5. Evaluate statistical significance (confidence interval)

12 1. Regression Our methodology

13 9 y = x Score on Outcome Variable The equation defines the relationship between a predictor and an outcome Score on Predictor Variable We can fit a line to the data model the data with an equation. This model has 2 features (those features needed to define a line): Intercept and slope

14 Our models are more complex Many predictors multiple regression Binary outcomes - logistic regression Nested cases and interest in smoothing - hierarchical logistic regression Conceptually the same - Fit the data; estimate the parameters Just more computationally intensive

15 Our methodology 2. Use all data, from all hospitals, to make case-level predictions Once we have the equation, we can get a predicted probability for each patient

16 9 8 y = x The statistical model is the existential source of risk adjustment Score on Outcome Variable The predictors (all the Xs age, ASA, CPT code, etc.) yield the probability of outcome Y, dependent upon the values of those predictors Score on Predictor Variable The estimated probability is, therefore, risk-factor adjusted. For any value on the X (predictor) scale, there is a value on the Y (outcome) outcome scale. Insert value of X in the equation and solve for Y Y = X

17 Our methodology 3. Compare hospital s risk-adjusted predictions to actual outcomes

18 Many patients Many hospitals Many predictors # Actual Events # Predicted Events Model predicts outcome Applied to your Hospital s patients Profiling depends on comparing model-predicted events to actual events

19 # Predicted Events # Actual Events Profiling depends on comparing model-predicted events to actual events The risk adjustment paradigm Observed # of Events (O) for a hospital is known Build regression model, using data from all hospitals Use this to generate each patient s risk For the site, sum patients risk to get expected (E) O If the site is average : O=E, 1, and OR=1 E Model for pancreatectomy mortality; hospital submits 100 cases O = = 12 E = = 9.25 O = 12 = 1.30 E 9.25

20 There are 3 reasons why O E = 1.30 and not = 1 1. The model is imperfect 2. Sampling error 3. The site s quality differs from that of the average site

21 Rate and odds ratios We ve moved from rates, rate ratios, odds, and odds ratios to risk-adjusted rate ratios (O/E) or risk adjusted odd ratios Achieved by Your metric now describes your performance in comparison to the average ACS NSQIP hospital, if that hospital were to do the same procedures on the same patients as your hospital. The fair comparison

22 4. Include smoothing Our methodology

23 Smoothing also know as a shrinkage adjustment, pooling, adjusting for reliability When sample size is small, estimates will be unstable and potentially misleading. Smoothing uses information from the whole sample to improve the estimate for the hospital providing a small sample

24 A very reasonable approach to decision making The classic baseball batting average example Predict a MLB player s season batting average from the first day where of the season: every patient survives without any complication, 4 for 4: despite Batting average what a = small 1000 sample might indicate 0 for 4: Batting average = 0 Smoothing algorithm finds the best weighting of An often used estimate for the information estimates true rate is the sample rate might be 275 and 265 There is no hospital where every surgical patient dies or

25 Colorectal Morbidity for 61 Hospitals Providing <= 50 cases Performance metric O/E or OR These estimates will be closer to the truth than quality metric = Without Logistic smoothing O/E Modeling Method and Metric With Hierarchical smoothing OR

26 Your 5 submitted cases with 0 deaths: Will not yield an odds ratio of 0 Will not result in you being Exemplary Will not result in you being in the 1 st decile After smoothing, you will look very much like the average hospital: OR 0.99 and 5 th decile.

27 Our methodology 5. Evaluate statistical significance (confidence interval)

28 Confidence Intervals Risk adjusted O/E and OR estimates are based on samples Samples have sampling variability Is amount of difference from 1.0 meaningful, or can it be explained by chance? Calculate a range of reasonably true values for OR (95% CI) If that range does not include 1.0 (P<0.05), reject chance

29 If the 95% CI does not overlap the reference value, then the hospital is a statistical outlier (at P<0.05). Low outlier (good) Exemplary High outlier (bad) Needs Improvement W Not an outlier As Expected

30 CQI-Type Modeling 1. Reliance on high-quality clinical data 2. High confidence that statistical methods yield useful quality assessments 3. Risk adjustment allows a site to be compared to the average site, if the average site were to do the same procedures on the same patients* 4. Smoothing stabilizes small-sample estimates; estimates are shrunken towards the average so impossibly good or poor quality will not be assigned 5. Confidence intervals provide a range for reasonable estimates of quality; reject average status when P < Growing armamentarium of quantitative resources

31 if the average site were to do the same procedures on the same patients* In other words, How well do we do what we do Patient-centric perspective is: which hospital offers me the best outcomes for my operation. In other words, How well do we do what the patient needs done CQI targeted models can address both questions simultaneously

32 Review material in the main SAR documents Attend the breakout sessions Visit us at the Statistics table us

33 ACS CQI Statistical Team Mark Cohen, PhD Arielle Grieco, MPH Kristopher Huffman, MS Yaoming Liu, PhD Brian Matel, BA Xiangju Meng, MS Christine Sullivan, MBA, MS - csullivan@facs.org Vanessa Thompson, PhD vthompson@facs.org Lynn Zhou, PhD lzhou@facs.org