Probabilistic Sensitivity Analysis Prof. Tony O Hagan

Transcription

1 Bayesian Methods in Health Economics Part : Probabilistic Sensitivity Analysis Course outline Part : Bayesian principles Part : Prior distributions Part 3: Uncertainty in health economic evaluation Part : Probabilistic bili sensitivity analysis Part : Formulating input uncertainty The first two parts are an introduction to Bayesian methods in general, while the last three parts are specific to their use in health economic evaluation Synopsis (part ) Economic modelling Probabilistic Sensitivity Analysis Computing output uncertainties Expressing output and decision uncertainty 3

2 Economic modelling A single clinical trial rarely provides all the evidence relevant to a cost-effectiveness study Often there are other trials, perhaps only looking at efficacy Trial conditions are rarely what we need for deciding on cost-effectiveness Limited follow-up Restricted enrolment Unrealistic compliance Wrong comparators Wrong outcomes etc Health economists rely more on modelling Vaccine example A simple economic model will illustrate the principles A decision is to be made regarding whether or not to vaccinate some subgroup of the population at risk of contracting a disease Treatment reduces risk, but does not guarantee immunity against disease Subjects showing symptoms of the disease require further treatment until cure Vaccine model inputs The following need to be specified in order to evaluate the mean incremental cost and efficacy of the proposed vaccination programme p : P(disease vaccine) p : P(disease no vaccine) V : Cost of vaccine D : Cost of visit to doctor q : Loss of health utility if having disease m : Average number of GP visits if having disease

3 The vaccine economic model Cost Efficacy p Disease V+mD q Subject Vaccine pp No Disease V No Vaccine p p Disease No Disease md q 7 Vaccine model outputs Once all the inputs are specified we can use the model to compute relevant outputs to assess cost-effectiveness ICER of vaccine: V ( p p ) md ρ = ( p p ) q Incremental Net Benefit (INB) of vaccine: β(k) = K ( p p ) q V + ( p p ) md Where K is the willingness-to-pay by the healthcare provider for a single unit gain of utility 8 Uncertainty in economic models Invariably there is uncertainty surrounding model inputs Treatment efficacies, resource use by patients Clearly, this implies uncertainty regarding the true ICER, or INB Three issues:. How to formulate uncertainty on model inputs?. How to compute resulting uncertainty in outputs? 3. How to express the uncertainty to guide decisions? In this talk, we will examine the second and third of these The first will be dealt with in the final talk of this course 9 3

4 PSA These three steps comprise what health economists call Probabilistic Sensitivity Analysis (PSA) PSA involves computing, expressing and analysing uncertainty about the outputs (e.g. incremental net benefit) of an economic model It is considered to be an important part of any health economic evaluation by several regulatory agencies In the UK, the National Institute for Health and Clinical Excellence (NICE) demands that PSA be a part of any cost-effectiveness evaluation Computing output uncertainty So how do we compute the uncertainty in outputs of the economic model? Assume we have already specified uncertainty in inputs in the form of probability distributions (see next talk) The usual method is Monte Carlo Sample input values from their distributions, compute outputs for each set of input values, to obtain a sample of outputs Typically, we need large samples, e.g., Tools: Treeage, Excel + Crystal Ball, WinBUGS Treeage and Crystal Ball Treeage is a commercial software package designed specifically for building and analysing economic models Easy to use but limited in the range of input distributions and output analyses Crystal ball is a commercial add-on to the Microsoft Excel spreadsheet that does Monte Carlo sampling Limited like Treeage, but the flexibility of the spreadsheet can be used to handle a wide range of input distributions and subsequent analyses Both are widely used in health economics Other similar packages exist It is also possible to program some spreadsheets directly, but this requires substantial expertise

5 WinBUGS WinBUGS is the Windows version of the BUGS package Widely used in Bayesian statistics Freely available from Can be used to generate samples from input distributions With great flexibility Particularly when these are obtained formally as posterior distributions (see next talk) Can also be programmed to produce a wide range of output analyses If the economic model is not too complex 3 Each sampled set of inputs produces a (Δ e, Δ c ) point from the model The sampled points provide a visual idea of uncertainty on the C-E plane N.B. Frequentist analysis using a bootstrapped sample looks similar but is fundamentally different Monte Carlo results delta-c delta-e The bootstrap points are sample means, not population means Basic PSA for the vaccine model Distributions were specified for the vaccine model inputs Input values were sampled The willingness to pay was set at K = Expected INB is.7 Histogram of INB 3 Computed by Monte Carlo 3 Positive, so should use vaccine But considerable uncertainty 9% interval ( 7.9, +.) Standard deviation.3 P(INB > ) =.3 y Frequency INB

6 When Monte Carlo is impractical Computing PSA by Monte Carlo can demand extensive computation Typically takes at least a few thousand runs to get accurate evaluation of outputs and uncertainty Impractical if it takes more than a few seconds for a single run, runs at a minute each takes 7 days of non-stop computing Micro-simulation models are a class of economic models that t are slow to run Also known as patient-level models Δ e, Δ c and cost-effectiveness measures computed by simulating many individual patients To get accurate values, typically need to simulate a very large number of patients for each sampled set of model inputs Monte Carlo adds another layer of sampling Two ways to gain efficiency Efficient nested Monte Carlo To get accurate PSA, we don t need to simulate such a large number of patients in each run Accept that the outputs of each run are then imprecise But we can now make more runs Optimal number of patients in each run is the ratio of within-run to between-run variances To a good approximation Under simple conditions Typically reduces computational load by a factor of 3 O'Hagan, A., Stevenson, M. and Madan, J. (); Monte Carlo probabilistic sensitivity analysis for patient level simulation models: Efficient estimation of mean and variance using ANOVA; Health Economics, Emulation When the computational load of Monte Carlo is still too high, there is another, more radical, alternative Build a meta-model for the economic model Called an emulator A fast approximation of the original model Build the emulator using a small number of model runs Run the emulator thousands of times 8

7 model runs Consider one input and one output Emulator estimate (solid line) interpolates data Emulator uncertainty (dotted lines) grows between data points 3 x 9 3 model runs Adding another point changes the estimate and reduces uncertainty 3 x model runs And so on x 7

8 Emulation Given enough model runs, the emulator accurately reproduces the relationship between inputs and outputs i.e. it can replace the economic model This can often be achieved with only a few hundred runs Even when there are many inputs The emulator becomes a very fast surrogate for the economic model See Stevenson M.D., Oakley, J. and Chilcott, J.B. (); Gaussian process modelling in conjunction with individual patient simulation modelling: A case study describing the calculation of cost-effectiveness ratios for the treatment of osteoporosis; Medical Decision Making, 89- Analysing the uncertainty Now consider how to present the uncertainty about cost-effectiveness In ways that are useful to decision makers We have already discussed EICER the Expected Incremental Cost-Effectiveness Ratio EINB Expected Incremental Net Benefit Adopt treatment if EINB > (or a more complicated condition on EICER) CEAC Cost-Effectiveness Acceptability Curve Plots probability treatment is more cost-effective, against K What more might we wish to do? 3 Uncertainty and sensitivity Sensitivity analysis can and should go further than just looking at the overall uncertainty in the key outputs It is often important to know how that overall uncertainty links to individual uncertain inputs or groups of inputs Which inputs are responsible for most of the output uncertainty? If we wanted to reduce output uncertainty, which inputs should we get more information about? Possibility of delaying adoption decision How to prioritise research agenda Even: Just how do individual inputs or groups of inputs influence the output? Particularly useful for checking face-validity of the economic model 8

9 Variance-based SA A measure of overall uncertainty in the cost-effectiveness decision is the variance of the INB Variance-based sensitivity analysis decomposes this into components due to each of the inputs And due to interactions between the inputs Closely related to analysis of variance in statistics A related idea is main effect plots Plot the expected INB as a function of each input, when expectation is taken with respect to all the others Show how the output responds to changes in that input It turns out that the variance component for a given input is the variance of its main effect SA for the vaccine example Three parameters were uncertain in the vaccine model m mean GP visits p infection rate with vaccine p infection rate without vaccine Sensitivity analysis shows most uncertainty in output (INB) is due to uncertainty in p 78% of total uncertainty In order to reduce uncertainty about net benefit, research should concentrate on p Parameter Variance m p.. p. Total 7. Vaccine example main effects INB IN B m p Effect of m in blue, p in red, p in green Solid line between quartiles of the input Dashed line out to 9% range of the input 7 9

10 C-E trajectories The main effects show the influence of each input on the expected INB A related plot traces the influence in the costeffectiveness plane The trajectory shows the path taken by (the expectation over other inputs of) (Δ e, Δ c ) Solid line between quartiles of the input Dashed line out to 9% range of the input 8 Trajectories for the vaccine model Black line is INB= (slope K) Blue trajectory is for m Red is for p Green is for p Δc Note: Solid parts of m and p trajectories 3 barely cross the black line Trajectories are straight in this very simple model Δe 9 Value of information analysis Another way to think of the importance of an input is to ask to what extent the uncertainty about its value would influence the cost-effectiveness decision If we learnt the value of a given input, the expected INB might change sign Then we would change decision and get an improved net benefit Trajectory crosses the line of slope K EVPI (expected value of perfect information) for a given input is the difference between (a) the expected incremental net benefit we could get if we first learnt the value of that input, and (b) the EINB if we made the decision now Often denoted as EVPPI (expected value of partial perfect information) Because we are talking about getting perfect information on just one input (or group of inputs) 3

11 VoI for the vaccine model Value of information analysis confirms that there is much greater value in learning about p If we learnt the value of p exactly, there would be little value in learning about the other parameters Parameter EVPPI m p p. All. 3 Computation SA and VoI calculations are even more computationally demanding than ordinary PSA With Monte Carlo methods we need a separate simulation for each input (or group of inputs) The emulation approach really bears dividends here We build the emulator just once, using just one set of training runs of the economic model Then we use the emulator to compute SA/VoI Oakley, J. E. and O'Hagan, A. (); Probabilistic sensitivity analysis of complex models: a Bayesian approach, Journal of the Royal Statistical Society B, Many treatments We have hitherto assumed we are comparing just two treatments what if we want to compare more than? Define INB for each treatment against a common comparator If the comparator is one of the treatments its INB is identically zero Select treatment with highest EINB This is the most cost-effective option Computations and plots can readily be adjusted to match this new definition e.g. the CEAC now has a line for each treatment showing its probability of being the most cost-effective, plotted against K 33

12 What s left? We have now finished the fourth talk in this series Thanks for listening! The final talk addresses the specification of uncertainty t about model inputs A crucial first step in PSA And one that is little understood 3 3