PROMs Risk Adjustment: a case-mix approach. James Coles, CHKS Ltd 22nd November 2012

Transcription

1 PROMs Risk Adjustment: a case-mix approach James Coles, CHKS Ltd 22nd November 2012

2 Structure of Talk Why Risk Adjust PROMs Development of Risk Adjustment Model Application of the Model to Trust data Some results of Risk Adjustment Possible future analyses

3 Why Risk Adjust PROMs?

4 Objective Our overall aim was to risk adjust the PROMs data to facilitate like for like comparisons across providers or commissioners, and across time periods In practice, this means standardising for the impact of patient characteristics and other factors so as to isolate the remaining effects of treatment efficacy and quality of care (which are within the control of the provider) In applying the risk adjustment model to routine monthly data extracts, only variation that is outside the control of the providers will be adjusted out.

5 A basis for comparisons!!

6 Provide a level playing field

7 Risk-Adjusting Patient Reported Outcomes Patient A Male, Hernia, Age 65, Baseline EQ5d Index score of 0.6 Patient B Female, Hernia, Age 65 Baseline EQ5d Index score of 0.6 Had symptoms for more than a year Reports suffering from Depression Patient A s Expected Improvement in Outcome (EQ5d Index) at 3 months is about twice that of Patient B ( ) cf ( )

8 EQ5d Index Score Risk adjustment in practice - Before 0.9 Effect of Risk Adjustment at two providers Trust B (Unadj) Trust A (Unadj) National average figure Baseline 3 months

9 EQ5d Index Score Risk adjustment in practice - After 0.9 Effect of Risk Adjustment at two providers Trust A (Adj) 0.88 Trust B (Adj) Trust B (Unadj) 0.86 Trust A (Unadj) National average figure Baseline 3 months

10 Development of the PROMs Risk Adjustment Model

11 Approach to model building Focus on evidence-based and supportable theoretical models Need to balance statistical refinement with transparency, simplicity and understandability Acceptable to both stakeholders in the NHS (incl. clinicians) and academic peers Model applied since mid 2010, with refined model from May 2012

12 For the technically minded the fixed effects model y ij = β 0 +β 1 x ij1 + β 2 x ij2 +β 3 x ij3. +u j +e ij i = 1,.n j, j = 1,..,J, u j ~ N(0,σ u2 ) e j ~ N(0,σ e2 ) where y ij is the Q2 follow-up scores of the various instruments e.g. EQ5D index, EQ5D visual analogue scale (VAS), Aberdeen Varicose vein score, for the ith patient from the jth provider units (NHS or independent hospital or treatment centre); x ij are independent predictor variables incl. relevant Q1 baseline score; u j are the provider unit level random effects and e ij are the patient level error terms The development of a Fixed Effects model became a possibility in 2011 as increased data became available from most provider units

13 Data: Variables included in establishing the models Baseline Q1 scores (including powers) Demographics (incl. age^2) Other patient characteristics, attitudes and perceptions Clinical risk factors (from HES) e.g. HRGs, procedure groups, Charlson Index and level of comorbidities Clinical risk factors (e.g. patient reported comorbidities) Most local area and provider related variables that had been included in the earlier model were excluded from the Fixed Effects model as the overall provider effect is identified by such models

14 Approach to model building (2) Fixed Effects model preferred to a Random Effects one as in testing the R.E. model coefficients were less consistent All selected variables (c. 50) were included in model development from the outset. Analysis of diagnostics led to some variables being excluded because of: Collinearity Non significant coefficient (t-stat) No variation across records

15 Approach to model building testing and further considerations The completed OLS models were examined for: Face validity (appropriate variables and direction) Stability, scale and direction of the coefficients Explanatory power of the model The eleven models were tested using completely new sets of data (approx ⅔ sample used in model creation) Examination of the residual sums of squares and differences in the relative significance of each coefficient between the original and test samples would indicate where there might be any concerns

16 Results of testing Models had face validity. Some comorbidity coeffts have +ve values which may seem counter-intuitive, but could be explained on inspection Differences in the coeffts between the two samples were generally not significant at the 1% level All the models predicted well out of sample actual v predicted correlations at provider level > 0.76 Final models include between 14 and 31 variables

17 Application of the models to PROMs data extracts

18 Application of the models to PROMs data extracts Production of case-mix adjusted outputs at provider level requires: the derivation of models to predict Q2 (follow-up scores at the individual patient level analyse the actual Q2 relative to the predicted value and use the resulting ratio at provider level to case-mix adjust the data so that performance can be measured across providers given a standardised case-mix

19 Risk-Adjusting PROMs at provider level For provider j, their relative performance is calculated from rj Q2 Q2 j( pred) The provider specific post-op PROMs score (standardised to the rj National case-mix ) is rj x j Q2( nat) Risk adjusted change in health status rj x Q2( nat) Q1nat ( )

20 Some results and potential developments

21 Results key variables Key driver variables across the models include: the Q1 baseline scores, time since symptom onset whether the patient considered themselves disabled, whether the patient had assistance in completing the questionnaire and whether they had had previous treatment patient reported comorbidities Ethnicity (particularly Asian) was an important factor in a number of the models

22 Impact of Risk Adjustment Groin Hernias Varicose Veins THR TKR n= No of providers where, when risk adjusted, the EQ5d Index (Q2, follow-up) increases by more than 5% increases by between 2.5 and 5% increases or decreases by less than 2.5% decreases by between 2.5 and 5% decreases by more than 5% Insufficient responses

23 Possible developments Risk Adjusted analyses at Sub-group level to investigate potential biases or differential outcomes EQ5d Profiles Prediction of Q2 Follow-up Profile and the identification of the contribution of individual variables, using Classification Trees and similar methodologies

24 References The Information Centre PROMs pages jsessionid=95c85067d449557b4aaafb6d51a50978?s iteid=1937&categoryid=1632 Risk Adjustment methodology report (available as a pdf)