SHMI Quarterly Profiling and Forecasting Paul Hawgood 4 th April 2017
1. Introduction The Summary Hospital-level Mortality Indicator [SHMI] value published by NHS Digital [NHSD] is the one and only figure that should be quoted and, as far as interpretation goes, it is the Band that is paramount. Breaking the SHMI value down by age or CCS Group may help with investigation and understanding but breaking SHMI down by time-periods [months or quarters] can yield spurious results. This is due to two main reasons: 1. The modelling period comprises 36 months so is unaffected by seasonality whereas selected months or quarters will be influenced by the seasonal variation in hospital-level mortality. 2. The same quarter appears in four separate releases and its relative risk changes with each release (this characteristic is expanded upon and illustrated below). However, understanding how data are changing over time, including by quarter, is a useful aid to unravelling the underlying reasons for changes in reported SHMI values. This report explains how to interpret data for individual quarters and how this may be used to anticipate changes in future SHMI values. 2. Quarterly profiling By way of an example, Table 1 illustrates the time-line for the quarter January 2015 to March 2015. The black headings represent six of the SHMI time-periods. The second row lists the calendar quarters six quarters are listed to show the period before and after the relevant SHMI. The third row identifies which fiscal quarter the calendar quarter relates to. Due to the data-processing time-lags inherent with SMRs, the quarter January 2015 to March 2015 does not feature in the SHMI released July 2015 as that covers the period January 2014 to December 2014. It makes its first appearance in the SHMI released October 2015 as it replaces the previous Q4, January 2014 to March 2014. It features in the following three releases, representing the ever-older data period as time passes. By June 2016, it is no longer included in the SHMI data for a trust as the Q4 that is now included is January 2016 to March 2016. Page 1 of 7
January 2014 to December 2014 (pub. Jul 2015) Oct13 - Dec13 Jan14 - Mar14 Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Fiscal Q Q4 Q1 Q2 Q3 April 2014 to March 2015 (pub. Oct 2015) Jan14 - Mar14 Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Q1 Q2 Q3 Q4 July 2014 to June 2015 (pub. Jan 2016) Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Q2 Q3 Q4 Q1 October 2014 to September 2015 (pub. Mar 2016) Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 Q3 Q4 Q1 Q2 January 2015 December 2015 (pub. Jun 2016) Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 Jan16 - Mar16 Q4 Q1 Q2 Q3 April 2015 to March 2016 (pub. Sep 2016) Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 Jan16 - Mar16 Apr16 - Jun16 Q1 Q2 Q3 Q4 Table 1: rolling SHMI quarters Sometimes, a particular quarter is considered by a trust to be particularly poor or good. By this they usually mean that there have been an unusually high or unusually low number of deaths. This is a useful starting point although, as we will see, the expected number of deaths for that quarter is equally relevant for it is the ratio of the two figures that is used to calculate SHMI. However, the knowledge that an unusual quarter is about to leave the SHMI data period does tend to create a feeling of impending improvement or deterioration in the SHMI value that is next to be published. Later, we will consider what might be deemed unusual but, first, it is important to know that the number of Expected deaths per quarter changes for the same quarter over time. This is because the model changes with each release. Generally, mortality rates improve over time. This means that, based on a rolling 36-month model, the statistical risk of death for the data when it is the most recent quarter [e.g. Jan 15 Mar 15 in the April 2014 to March 2015 release] is usually at its highest and, by the time nine months of improvement has elapsed, the statistical risk of death for data when it is the oldest quarter [e.g. Jan 15 Mar 15 in the January 2015 to December 2015 release] is usually at its lowest. The extent of this impact changes from trust to trust due to differences in case-mix but, by way of illustration, one trust s SHMI is shown below in table 2 and table 3. Page 2 of 7
January 2014 to December 2014 Oct13 - Dec13 Jan14 - Mar14 Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 April 2014 to March 2015 Jan14 - Mar14 Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 SHMI for the Q 1.085 July 2014 to June 2015 Apr14 - Jun14 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 1.089 October 2014 to September 2015 Jul14 - Sep14 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 1.102 January 2015 December 2015 Oct14 - Dec14 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 Jan16 - Mar16 1.122 April 2015 to March 2016 Jan15 - Mar15 Apr15 - Jun15 Jul15 - Sep15 Oct15 - Dec15 Jan16 - Mar16 Apr16 - Jun16 Table 2: rolling SHMI quarters with illustrative value Calendar Quarter Fiscal Quarter Reporting Period Position Discharges Observed Expected SHMI Jan 15 - Mar 15 Q4 Apr14 - Mar15 4 15,716 550 506.77 1.085 Jan 15 - Mar 15 Q4 Jul14 - Jun15 3 15,716 550 505.09 1.089 Jan 15 - Mar 15 Q4 Oct14 - Sep15 2 15,716 550 499.00 1.102 Jan 15 - Mar 15 Q4 Jan15 - Dec15 1 15,716 550 490.10 1.122 Table 3: changing SHMI values for the same quarter This confirms that the number of discharges and the number of Observed deaths is unchanged i.e. it is the same data for the same data period and the numerator does not change. What does change, though, is the number of Expected deaths. In turn, this changes the SHMI for this quarter [Q4 being one of the higher quarters due to the seasonal effect of mortality, hence the values are well above 1.00; other quarters offset this by being below 1.00 an illustration of why non-annual SHMI analysis should not be undertaken]. We can see that the same data for the same patients has a different contribution to make to the full-year trust SHMI depending on whether it is the most recent quarter or oldest quarter. Of course, although three of the quarters are getting older [and, usually, worse] this is balanced by the worst quarter falling out of the cycle and a new quarter coming in at its best time. If this is happening to every trust in every SHMI release, why does this matter? It only matters when we start to attempt to forecast how normal the quarter about to be lost is and the impact of adding in a new normal quarter. Page 3 of 7
3. Forecasting the direction of movement of the next SHMI value The starting point for this is to establish how data for the quarter that is about to come out of the SHMI compares to the normal value for the same fiscal quarter in previous years. Chart 1 shows a trust s SHMI split by quarter [blue columns] alongside the historical average for the same fiscal quarter, in the same sequential position [grey columns]. July 2015 to September 2015 is about to be replaced by July 2016 to September 2016 and we need to assume that this quarter will be normal. The SHMI for July 2015 to September 2015 is higher than the historical average so the hypothesis is that the next SHMI of this trust will show a reduction. The degree of certainty of this forecast will depend on the difference between these two values. Chart 1: SHMI by quarter c.f. historic average and showing the quarter to be replaced The challenge that we have is that we have only a short history of the Expected number of deaths by quarter and so the construction of a normal value is weak. Despite this, we modelled the anticipated future impact of January 2015 to March 2015 being replaced by January 2016 to March 2016, the impact of April 2015 to June 2015 being replaced by April 2016 to June 2016 and the impact July 2015 to September 2015 being replaced by July 2016 to September 2016. We set different thresholds of difference between the two SHMI values and tested whether the new full-year SHMI increased or decreased as forecast. Tables 4a 4d below show our results with Table 5 providing an explanation of each column: Page 4 of 7
Lose:Gain True n False n True % >120% 10 0 100% >115% 20 5 83% >110% 34 8 81% >105% 44 29 60% <100% 20 7 74% <95% 8 2 80% <90% 4 0 100% Table 4a: SHMI forecast test results January to March Lose:Gain True n False n True % >120% 4 2 67% >115% 11 3 79% >110% 22 8 73% >105% 41 24 63% <100% 22 13 63% <95% 11 7 61% <90% 2 2 50% Table 4b: SHMI forecast test results April to June Lose:Gain True n False n True % >120% 1 0 100% >115% 6 1 86% >110% 13 5 72% >105% 28 12 70% <100% 39 19 67% <95% 20 10 67% <90% 7 4 64% Table 4c: SHMI forecast test results July to September Page 5 of 7
Column Lose:Gain True n False n Meaning This is the SHMI value of the quarter to be removed expressed as a percentage of the average of historical values for the same fiscal quarter at the same phase in its SHMI life i.e. in its last appearance (position 1). A high value would suggest that the quarter about to be removed was poor and that the next SHMI value would reduce (tests where Lose:Gain is >105%). A low value would suggest that the quarter about to be removed was good and that the next SHMI value would increase (tests where Lose:Gain is <100%). This is the number of trusts that met the Lose:Gain threshold and saw a reduction in their SHMI value. This is the number of trusts that met the Lose:Gain threshold but did not see a reduction in their SHMI value. True % True n expressed as a percentage of (True n + False n) Table 5: column explanations for Table 4 Findings at this stage are mixed. The January to March changes showed that the greater the relative difference in values, the greater the accuracy of the forecast. Even a threshold of >110% correctly forecasted 81% of improvements and a threshold of <95% correctly forecasted 80% of deteriorations. However, the data for April to June were less conclusive. Fewer trusts showed forecasted improvements at each threshold and the results of the forecasts turned out to be less reliable e.g. for a threshold of >120%, in January to March all 10 trusts which had a value >120% prior to the next SMI release had a reduction in their SHMI value whereas for April to June only four of the six trusts had a reduction in their SHMI value. The July to September changes showed similar results to January to March in that the greater the relative difference in values, the greater the accuracy of the forecast. Adding the results together from the three iterations of forecast gives an overall picture of the accuracy of forecasting see Table 4d. This shows that for potential decreases, the greater the relative difference in values, the greater the accuracy of the forecast. However, for potential increases, the accuracy of the forecast is the same regardless of the size of the relative difference. Lose:Gain True n False n True % >120% 15 2 88% >115% 39 9 80% >110% 69 21 77% >105% 113 65 63% <100% 81 39 68% <95% 39 19 67% <90% 13 6 68% Table 4d: SHMI forecast test results sum of 3 iterations Page 6 of 7
This forecast is confined to the direction of change of the SHMI value and does not attempt to quantify the size of this change. It follows, therefore, that it is not possible to attempt to forecast a change in Band. [In addition to not forecasting the size of change, it would not be possible to pre-calculate the Over-Dispersal [OD] limits; any attempt to forecast the imminent Band would also be reliant on assuming no change in these OD limits, an assumption that is not correct]. 4. Summary and conclusion So far, we have tested this methodology for three SHMI releases. Initial findings are encouraging but we will continue to test the methodology. We will then provide information to individual trusts on our findings in order to give them an informed view of the likely direction of change for the forthcoming SHMI value. In theory, this analysis could be undertaken by month, however, this would serve no purpose. SHMI is published quarterly, not monthly, and it is a full quarter s data that is changing. Calculations at a monthly level would be more volatile and less accurate and, for forecasting purposes, three months would need to be added together to return to the quarterly picture. Page 7 of 7
Produced by Advancing Quality Alliance (AQuA) 3rd Floor, Crossgate House, Cross Street, Sale M33 7FT www.aquanw.nhs.uk @AQuA_NHS AQuA 2017 All information contained in this document is, as far as we are aware, correct at time of going to press. No part of this document report may be reproduced in whole or in part without written permission of AQuA.