Known unknowns and unknown unknowns: uncertainty from the decision-makers perspective Neil Hawkins Oxford Outcomes
Outline Uncertainty Decision making under uncertainty Role of sensitivity analysis
Fundamental Problem For the patient about to be treated: Counterfactual treatment outcomes are, in general, unobservable Rely on results of historical studies Studies are expensive and take time Finite sample size Long term outcomes generally not observable Requirement to extrapolate from surrogate endpoints Bias, sample not exchangeable with future subject
In the Best of All Possible Worlds* for a patient with a given set of characteristics, we would know the ultimate outcome of each available treatment option with certainty. *The optimist proclaims that we live in the best of all possible worlds; and the pessimist fears this is true. James Branch Cabell
In the Real World Patients Vary Individuals are uncertain about their response to treatment
There is Uncertainty Regarding How They Vary OR OR Decision-makers are uncertain about population level effects
There is Ambiguity Regarding the Uncertainty OR OR OR OR OR Different interpretations of data Different prior beliefs May be difficult to choose parameter distributions Some parties may be considered partial
The solution expected utility* 1. First remove dominated options if there s a blue pill and a red pill, and the blue pill is half the price of the red pill and works just as well, why not pay half the price for the thing that is going to make you well? Obama 2007 2. Assign a subjective probability distribution to unknown parameters 3. Select treatment that maximises subjective expected benefit 4. Consider requirements for further evidence *utility is health benefit of a treatment minus of its cost in comparable units (net health benefit)
Alternative Decision Rules Maximin Choose option that maximises minimum utility Minimax regret Choose option that minimised maximum regret Regret is the difference between the chosen option and the optimum option (given perfect knowledge)
Health Benefits Expected Utility Treatment Scenario (Probability) Expectation x=1 (0.7) x=2 (0.1) x=3 (0.3) A 0 0 0 0 B 2-1 -4 1.2 C 7-2 -11 4.5 Treatment C has greatest expected health benefit Estimate dependant on the subjective distribution of X
Expected Value of Perfect Health Benefits Information (EVPI) Treatment Scenario (Probability) Expectation x=1 (0.7) x=2 (0.1) x=3 (0.3) A 0 0 0 0 B 2-1 -4 1.2 C 7-2 -11 4.5 Perfect Information 7 0 0 4.9 Current Information 7-2 -11 4.5 EVPI = 0.4 (4.9-4.5) Estimate dependant on the subjective distribution of X
Maximin Health Benefits Treatment Scenario Worst A B C A 0 0 0 0 B 2-1 -4-4 C 7-2 -11-11 Treatment A has greatest minimum health benefit Estimate does not require subjective distribution for X May be applied in case of ambiguity
Health Benefits Regret Minimax Regret Treatment Scenario A B C A 0 0 0 B 2-1 -4 C 7-2 -11 Best 7 0 0 Treatment Scenario Worst A B C A -7 0 0-7 B -5-1 -4-5 C 0-2 -11-11 Treatment B has minimum regret
Another Issue: Risk Preference Risk neutral Ambivalent between a certain 100 or a 50/50 gamble between 20 and 180 Risk averse Prefer 100 over 50/50 20 and 180 Risk seeking Prefer a 50/50 20 and 180 over a 100 Implies that 180 is more than 9 times as valuable as 20. Or decision-maker likes gambling
Healthcare Decision Makers May not be risk neutral May not maximise expected health benefit With respect to cost Large budget holders risk neutral small budget holders risk averse Large budget impact may shift acceptable CE threshold Requirement for budget impact modelling With respect to patient safety First do no harm
Individual Patient Preferences Probability of Outcome Treatment Option Outcomes Surgery Radiation Watchful Waiting Cancer Eradication 1 0.6 0 Surgical Risk 1 0 0 Impotence 0.7 0.7 0 Bladder Incontinence 0.2 0 0 Health Benefits depend on individual risks and individual preferences
Deterministic Threshold Analysis Nuitjen et al. Pharmacoeconomics 2007: 25(1) 55-71
Probabilistic Cost-effectiveness Acceptability Curve Nuitjen et al. Pharmacoeconomics 2007: 25(1) 55-71
Figure 7: Value of information estimates Expected value of perfect information (Millions) Probabilistic Value of Information Analysis 120 100 Patent price (λ= 20,000) Generic price (λ= 20,000) Patent price (λ= 30,000) Generic price (λ= 30,000) 80 60 40 20 0 Overall Longitudinal genotyping Crosssectional genotyping RCT Utilities Costs Woods et al, In submission
Deterministic Scenario Analysis
Sensitivity Analysis Determines optimal policy under set of alternative assumptions Does not identify optimum decision Decision not sensitive Unbridled joy Decision is sensitive Identifies which assumptions are important Allows decision-maker to choose assumptions Provides estimates of value of information Informs need for and feasibility of further research Allows decision-maker to use alternative decision rules Allows decision-maker to account for issues not explicitly considered in the model, such as individual preferences
Challenges of Sensitivity Analysis Probabilistic analysis can be difficult to interpret. Need for partial analysis. Comprehensive deterministic analysis can be difficult if there may be many uncertain parameters Difficult to prejudge decision-makers requirements Important role for interactive tools
References CHOOSING TREATMENT POLICIES UNDER AMBIGUITY Charles F. Manski Department of Economics and Institute for Policy Research, Northwestern University Forthcoming: Annual Review of Economics, Vol. 3.