Deciphering the Statistical Reports

Transcription

1 Deciphering the Statistical Reports Kristopher Huffman, MS Division of Research and Optimal Patient Care Continuous Quality Improvement American College of Surgeons July 24 th, 2012

2 Overview Site Summary Report Site Specific Bar Plots SAR Summary Report

3 Site Summary Report The Site Summary Report provides details on your hospital s performance for every model in which your hospital submitted cases.

4 Site Summary Report The Site Summary Report provides details on your hospital s performance for every model in which your hospital submitted cases.

5 Site Specific Bar Plots The site specific bar plots provide a graphical description of your hospital s performance for every model in which your hospital submitted cases. At the same time, the site specific bar plots allow your site to see the general performance of all sites in a particular model. Much of the information contained in the site summary report is represented in the site specific bar plots.

6 Odds Ratio Estimate Numerical Value of Odds Ratio Estimate Reference Line Odds Ratio Estimate

7 The Background Bar The Background Bar gives the range of odds ratio estimates for ALL sites in the model. Among all hospitals in the Urology SSI model, the highest odds ratio estimate was Among all hospitals in the Urology SSI model, the lowest odds ratio estimate was 0.61.

8 The Background Bar An opening at the top of the background bar indicates that the highest odds ratio estimate in the model was greater than 4.5.

9 Decile Lines and Rank The Decile Lines tell you where one decile ends and where another decile begins. To construct Decile Ranks, all hospitals in a model are ordered from smallest odds ratio to largest odds ratio and then membership is assigned to ten sequential groups, each representing approximately 10% of the total number of hospitals. Any odds ratio in this range is in the 1 st Decile. If your odds ratio is in the 1 st decile, then your odds ratio was less (better) than at least 90% of the other odds ratios for NSQIP sites in the model. Any odds ratio in this range is in the 10th Decile. If your odds ratio is in the 10th decile, then your odds ratio was greater (worse) than at least 90% of the other odds ratios for NSQIP sites in the model. Said in another way, a hospital in the 1st decile will have an odds ratio that is in the lowest 10% of all odds ratios. Said in another way, a hospital in the 10th decile will have an odds ratio that is in the highest 10% of all odds ratios. Since the odds ratio estimate falls in the fourth decile, the decile status is given by a 4.

10 The Background Bar and Decile Lines Hospital A Hospital B For a specific model, the background bar and decile lines DO NOT change across individual site bar plots.

11 Site Specific Bar Plot Numerical Value of Odds Ratio Estimate Reference Line Background Bar Decile Lines Confidence Interval Odds Ratio Estimate Decile Rank

12 The Confidence Interval Every statistical estimate contains some amount of uncertainty. To quantify our uncertainty in the estimate we construct a confidence interval. The confidence interval allows us to specify a certain error tolerance and provide a margin of error for our estimate. The upper limit of the confidence interval is The odds ratio estimate for this site is This is our best guess of the true odds ratio for this site. The lower limit of the confidence interval is We are 95% confident that the true odds ratio for this site is somewhere between 0.37 and If your confidence interval is entirely above the reference line, we have good evidence that your true odds ratio is greater than 1 and you are labeled a high statistical outlier. Similarly, if your confidence interval is entirely below the reference line, we have good evidence that your true odds ratio is less than 1 and you are labeled a low statistical outlier.

13 The Confidence Interval An arrow at the top of the confidence interval indicates that the upper limit of your confidence interval extends beyond 4.5.

14 The Confidence Interval Hospital A Hospital B The confidence interval is a site specific attribute. The confidence interval DOES change across individual site bar plots.

15 Exemplary Status For a given model, Exemplary status is assigned to your hospital if either one of the following statements is true. (1) Your hospital is a low statistical outlier. (2) Your odds ratio estimate is in the 1 st decile. If your hospital is classified as Exemplary for a model, then the essential features on the bar plot will be color coded using the above color.

16 Exemplary Status This site is a low statistical outlier in the All Cases Pneumonia Model, since the confidence interval is entirely below the reference line of 1. This means that there is good, statistically significant evidence that the true odds ratio of this site is less than 1 and hence, that the site is performing better than the average NSQIP hospital. On this basis, the site is assigned Exemplary Status for this model. This same site is NOT a low statistical outlier for the All Cases DVT/PE model, since the confidence interval is NOT entirely below the reference line of 1. However, the odds ratio estimate is in the 1 st decile. This means that the site s odds ratio estimate for this model was better than at least 90% of the other odds ratio estimates for NSQIP sites in the model. Although this is not a statistically significant result, it does provide evidence that the site is performing better than the average NSQIP hospital. On this basis, the site is assigned Exemplary Status for this model. Note the L indicating that this site is a low statistical outlier for this model.

17 Exemplary Status You DO NOT have to be in the 1 st decile to be a low statistical outlier. This site is a low statistical outlier for the All Cases Morbidity Model, and the odds ratio estimate is in the 2 nd decile.

18 Needs Improvement Status For a given model, Needs Improvement status is assigned to your hospital if either one of the following statements is true. (1) Your hospital is a high statistical outlier. (2) Your odds ratio estimate is in the 10 th decile. If your hospital is classified as Needs Improvement for a model, then the essential features on the bar plot will be color coded using the above color.

19 Needs Improvement Status This site is a high statistical outlier in the All Cases Morbidity Model, since the confidence interval is entirely above the reference line of 1. This means that there is good, statistically significant evidence that the true odds ratio of this site is greater than 1 and hence, that the site is performing worse than the average NSQIP hospital. On this basis, the site is assigned Needs Improvement Status for this model. This same site is NOT a high statistical outlier in the All Cases UTI model, since the confidence interval is NOT entirely above the reference line of 1. However, the odds ratio estimate is in the 10 th decile. This means that the site s odds ratio estimate for this model was worse than at least 90% of the other odds ratio estimates for NSQIP sites in the model. Although this is not a statistically significant result, it does provide evidence that the site is performing worse than the average NSQIP hospital. On this basis, the site is assigned Needs Improvement Status for this model. Note the H indicating that this site is a high statistical outlier for this model.

20 Needs Improvement Status You DO NOT have to be in the 10 th decile to be a high statistical outlier. This site is a high statistical outlier for the All Cases Morbidity Model, and the odds ratio estimate is in the 9 th decile.

21 As Expected Status For a given model, As Expected status is assigned if your hospital is classified as neither Exemplary nor Needs Improvement. This will happen when BOTH of the following statements are true. (1) Your confidence interval crosses the reference line of 1. (2) Your odds ratio estimate is NOT in the 1 st or 10 th decile. If your hospital is assigned to As Expected for a model, then the essential features on the bar plot will be colored black.

22 As Expected Status Note that the odds ratio estimate is less than 1. However, the confidence interval goes from 0.67 to This means we are 95% confident that the true odds ratio is somewhere between 0.67 and Since 1 is in this interval, we cannot reasonably exclude the possibility that the true odds ratio for this site is 1. As such, there is not statistically significant evidence to suggest that the true odds ratio for this site is significantly different from 1. In addition, the odds ratio estimate is in neither the 1 st nor 10 th decile, so there is no further evidence that the site is performing any different than the average NSQIP hospital. On this basis, the site is assigned As Expected Status for this model.

23 Site Specific Bar Plots

24 Performance Status in Procedure Targeted Models In the Procedure Targeted models there were 29 models where no sites were statistical outliers (high or low). In models where this happened, we did not assign Exemplary or Needs Improvement based on decile rank alone. Given this, the color coding scheme for representing Exemplary or Needs Improvement is dropped in the bar plots for these models, with an * appearing next to the decile rank whenever a site s odds ratio estimate is in the 1 st or 10 th decile.

25 Performance Status in Procedure Targeted Models The odds ratio estimate for this model is in the 10 th decile. Note that the odds ratio estimate, confidence interval, and decile rank are NOT color coded. There is, however, an * next to the decile rank. The odds ratio estimate for this model is in the 1 st decile. Note that the odds ratio estimate, confidence interval, and decile rank are NOT color coded. There is, however, an * next to the decile rank.

26 SAR Summary Report Another invaluable tool for understanding the risk adjusted reports is the SAR Summary Report, which is found in the main SAR document that is distributed to all hospitals

27 Other Advice/Resources 1) Read the Main SAR Document 2) Read the Main SAR Document again 3) Repeat (1) and (2) as necessary 4) Glossary of Statistical Terms and Concepts 5) Statistical Modeling Description 6) Bibliography (accessible from workstation)