Forecasting Ontario Provincial Drug Expenditures a Hybrid Approach to Improving Accuracy CADTH 2018 HALIFAX, APRIL 16, 2018

Transcription

1 Forecasting Ontario Provincial Drug Expenditures a Hybrid Approach to Improving Accuracy CADTH 2018 HALIFAX, APRIL 16, 2018

2 Outline 1. Introduction (Oncology Drug Funding at Cancer Care Ontario) 2. Forecasting Approach 3. Results & Findings 4. Forecasting Challenges 5. Managing Forecasting & Workflow 6. Limitations 7. Future Directions 8. Conclusions 2

3 Oncology Drug Funding Cancer drugs are costly how are they administered? The Provincial Drug Reimbursement Program (PDRP) at Cancer Care Ontario (CCO) administers outpatient injectable cancer drug funding to hospitals and drug centers through the New Drug Funding Program (NDFP), the Evidence Building Program (EBP), and the Case by Case Review Program (CBCRP) Provincial Drug Reimbursement Program (PDRP) In 2016/17, the NDFP and EBP together funded $315.1M in injectable cancer drug claims for New Drug Funding Program (NDFP) Evidence Building Program (EBP) Case by Case Review Program (CBCRP) 28,179 treated cases Due to the significant cost associated with cancer drugs, it is important for the PDRP to develop accurate budget forecasts 3

4 Drug Budget Forecasting Different approaches to expenditure forecasting what is the most effective? Automation vs supervision Top Down Data driven forecasts often use historical time series data, but are only effective when the time series are long and stationary Expert driven forecasts can account for new drugs or anticipated disruptive factors, but are arbitrary without any statistical input Allocation vs aggregation Data Driven Expert Driven Top down forecasts are effective in situations where the low level units belong to the same markets and display similar trends and behaviour Bottom up forecasts are better options when the low level units are less homogenous, and are especially effective at capturing short to medium term trends Bottom Up 4

5 Drug Budget Forecasting We took a hybrid approach to drug forecasting There are over 40 cancer drugs currently funded through the PDRP with approximately 140 policies 1 budget forecast Budget Each policy contains patient eligibility criteria and treatment regimen specifications describing the associated drug dosage and ~ 40 drug forecasts Drug 1 Drug 40 frequency Policy forecasts may be modified mid-year ~ 150 policy forecasts Policy 1 Policy 2 Policy 150 Lack of homogeneity amongst policies necessitates a bottom-up approach An automated analytical methodology incorporating time series data was developed to accommodate the large number of forecasts Time Series Data Expert Input Expert input is incorporated to adjust for policy changes 5

6 Percent of Expenditures Data Insights the 80 / 20 Split A large proportion of the total budget is composed of a small number of policies Proportion of policies contributing to total 2016/17 expenditures 97 policies 24 policies Small Budget Drivers (~20% of the budget) 121 policies in 2016/ policies Trastuzumab Adjuvant Treatment for Breast Cancer Big Budget Drivers (~80% of the budget) 19 policies in 2016/17 6

7 Forecasting Strategy Developed a strategy that implements established analytical forecasting methods as well as user judgement Forecasts for Big Budget Drivers (BBD) include both automated and user-defined forecasts Policies with sufficient data are forecasted using four models and the best forecast is selected by the user Policies with insufficient data are forecasted using naïve methods, however the user is encouraged to input their own forecast Forecasts for Small Budget Drivers (SBD) are completely automated Policies with sufficient data are forecasted using all four models and the best forecast is selected by comparing the Goodness of Fit results of the methodologies on the previous fiscal year Policies with insufficient data are forecasted using naïve methods 7

8 Visual Depiction of Forecasting Methodology Length of Time Series Big Budget Driver or Small Budget Driver Forecasting Model Output < 12 months BBD or SBD Naive Naive Forecast months BBD or SBD Exponential Smoothing Exponential Smoothing Forecast Exponential Smoothing Policy SBD ARIMA Neural Network "Best" Forecast 24 + months Hybrid (ARIMA + NNET) Exponential Smoothing Exponential Smoothing Forecast BBD ARIMA Neural Network Hybrid (ARIMA + NNET) ARIMA Forecast Neural Network Forecast Hybrid Forecast

9 Results Overview of Methods Data: Cost and Dosage for fiscal year 2016 (FY2016) and open year FY2017 (Apr 2017 to Jan 2018) Restrictions: Only time series with 6+ months of data were used for the current comparisons Results: Data are aggregated including both BBD and SBD policies; presented by fiscal year Measures: Total cost and Unit Doses (for reimbursed units of drugs); and Percent error (% Error) Forecasts: For each policy, dosage is modelled and multiplied by unit costs to derive total costs Notes: o Some policies have more than one drug included (e.g. combination therapies) o SBD and BBD budget driver classifications are made by fiscal year o As a result, the same policy could have different classifications in different data years (e.g. SBD in FY2016, then BBD in FY2017) 9

10 Cost Comparison with Forecasts Monthly Cost (FY2016, open FY2017) $35.0 M FY2016 open FY2017 $30.0 M $25.0 M $20.0 M $15.0 M $10.0 M $5.0 M $0.0 M Forecast Range Actual Forecast Actual costs are within forecasts (i.e. forecasted totals for all reimbursed policies) Total forecasted values generally in line with actual values month-on-month (i.e. within min, max forecasted ranges) General shape of forecast fluctuations similar for all BBD (range $15M to $20M); and SBD (range $3.5M to $5.0M) policies 10

11 Unit Doses Comparison with Forecasts Unit Doses (FY2016, open FY2017) 9.0 M 8.0 M 7.0 M 6.0 M 5.0 M 4.0 M 3.0 M 2.0 M 1.0 M 0.0 M FY2016 open FY2017 Forecast Range Actual Forecast Some months show total dose units (i.e. for all reimbursed units) higher than forecast range Total forecasted values generally contain the actual values month-on-month However, 5/12 months (FY2016) had actuals significantly higher than forecasts; SBD forecasts were generally worse than BBD 11

12 < < Forecast Performance % Error Histogram (FY2016, open FY2017) % error 0% error FY 2016/17 FY 2017/18 Frequency of policies Total cost Forecasts for high-cost policies generally have lower % Error However, % Error performance in FY2016 was worse than for open FY2017 policies (i.e. greater spread in % error values) Distribution of % error is symmetric (i.e. no specific forecasting bias) similar number of under- and over-forecasted policies 12

13 < < Forecast Performance % Error Histogram (BBD, SBD) 40% 0% error 0% error 35% 30% 25% 20% 15% 10% 5% 0% BBD SBD Frequency of policies Total cost SBD policies have the widest spread in forecast % Error compared to BBD policies SBDs have the largest variation in % Error; However, they represent the lowest budget costs compared to BBD The variation is higher (in absolute terms) than BBD policies with greater spread (i.e. larger range in % Error) Recall: individual SBD forecasts were automatically chosen based on GOF this may be one reason for the larger % Error variation 13

14 Forecasting Challenges Newly introduced or existing drug policies can be met with the following challenges for predicting drug expenditures: 1. Potential price changes (e.g. generics) 2. Market share expectations (e.g. new policy introduction) 3. Changing patient behaviors (e.g. uptake patterns) 4. Physician treatment preferences (i.e. prescribing patterns) 5. Timelines for drug implementations (e.g. existence of compassionate access programs) 14

15 Managing Forecasting Challenges Scenario Analyses (e.g. BIA) Additional Relevant Data Multiple Imputation Techniques Internal analysis & clinical input Internal analysis Multiple Forecast Scenarios Generated 15

16 NDFP Forecast Reporting Key deliverables per month to the MOHLTC: Monthly drug utilization by patients and costs incurred NDFP provides low*, mid +, and high α forecasts for all policies (including pipeline drugs) by month and projected for the fiscal year Drug policies which are newly introduced, subjected to price changes, or potentially affected by the introduction of new market space competitors, are given additional attention * Assuming lower utilization + Assuming average utilization α Assuming higher utilization 16

17 Tool Contribution to Forecasting Workflow CCO data retrieval and preparation Forecast algorithms executed Forecast discussion and validation (Clinical input, Forecast report drafted and revised Internal analysis) 17

18 Limitations Strengths Provides data-driven baseline forecast and enhances reporting time and workflow Accurately forecasts policies with temporally robust data Weaknesses Error is significantly higher on policies with shorter time series (N.B. new policies with scarce data) Difficult to model knowledge of external events (e.g. drug policy revisions leading to volume movement from older to newer policies; price changes; drug shortages; etc.) o o Tool does not integrate contextual information & requires manual validation at policy level Tool does not provide additional scenario analyses Forecasts for SBD policies are currently largely automated (chosen based on best GOF) there is opportunity to use a semi-supervised approach to SBD forecasts (similar to current BBD forecasts) 18

19 Future Directions Enhancing the Forecasting Tool (Version 2.0) Consider additional computational approaches (e.g. imputation, extrapolation using CAGR) Analysis of patient uptake patterns (i.e. predicting consecutive/intermittent drug uptake) Integrating additional metrics (e.g. new case enrolment) to generate potential forecast outcomes Scenario builder similar to budget impact analyses could provide further insight 19

20 Conclusions Unique approach to drug forecasting in Canada Providing an operational tool to forecast drug expenditures Using a semi-supervised approach (i.e. automation with expert review) Leveraging statistical models to provide a variety of forecasts Opportunity for improvement re: forecast accuracy and scenario analysis to assist with routine reporting 20

21 Thank you! Presenter Luciano Ieraci, MSc, PhD(c) Co-Presenter Thomas Shin, MA, MPH Jessica Arias, PMP Daphne Sniekers, PhD Paula Murray, PhD Senior Methodologist, Strategic Analytics Senior Analyst, New Drug Funding Program Program Manager, New Drug Funding Program Senior Methodologist, Strategic Analytics Methodologist, Strategic Analytics 21