{ } Sample Size and Power for the Comparison of Cost and Effect. Goal of Sample Size Calculation. Sample Size Formula, Common SDs.

Transcription

1 Sample Size and Power for the Comparison of Cost and Effect Henry Glick Applications of Statistical Considerations in Health Economic Evaluations ISPOR 13 th International Meeting May 4, 2008 Goal of Sample Size Calculation Prior to the literature that described confidence intervals for cost-effectiveness ratios, sample size was commonly based on the larger of the sample sizes needed for estimating pre-specified cost and effect differences i.e., what sample size was required to identify a $1000 difference in costs, and what was required to identify a 10% reduction in mortality Current sample size calculations are based on the number of study subjects needed to rule out that the therapy is unacceptable (equivalent to ruling out that the net monetary benefits of the intervention are less than 0) Sample Size Formula, Common SDs Sample size for NMB uses the following formula: ( α+ β) 2 sd c + 2 W sd q - 2W ρ (2sd c) (2sd q) n = 2 (WΔQ ΔC) { } where n = n/group; tα/2 and tβ = t-statistics for α and β errors; sd = standard deviation for cost (c) and effect (q); W = maximum willingness to pay one wishes to rule out; and ρ = correlation of the difference in cost and effect 1

2 Sample Size Formula, SDs Differ When the SDs differ, the formula becomes: { } (t + t ) (sd + sd ) + W (sd + sd ) - 2W (sd + sd ) (sd + sd ) n = (WQ C) α β ρ c0 c1 q0 q1 c0 c1 q0 q1 /2 2 where n = n/group; tα/2 and tβ = t-statistics for α and β errors; sd = standard deviation for cost (c) and effect (q); W = maximum willingness to pay one wishes to rule out; and ρ = correlation of the difference in cost and effect Correlation Between Costs and Effects $93,500 $46,200 $27,500 $22,670 Incremental Costs $11, Point Estimate CER: $27,500 Incremental QALYS Means and S.D.s identical Correlations: 0.95; Correlation Between Costs and Effects All else equal, the required sample size is less when the therapies have a Win/Lose (positive) correlation As the effectiveness increases, the cost increases (e.g., stroke care) All else equal, the required sample size is greater when the therapies have a Win/Win (negative) correlation As the effectiveness increases, the cost decreases (e.g., asthma care) Extreme values of correlation between costs and effects can have dramatic effects on the confidence interval for the cost effectiveness ratio/nmb and thus on the sample size required to demonstrate value for the cost 2

3 Where to Obtain the Necessary Data? When therapies are already in use: Expected differences in outcomes and standard deviations can be derived from feasibility studies or from records of patients Potential sources Medical charts of administrative data sets Patient logs of their health care resource use Asking patients and experts about the kinds of care received by those with the condition under study In addition, at least one study has suggested that the simple correlation between costs and effects observed in these data may be an adequate proxy for the measure of correlation used for estimating sample size Obtaining Data for Novel Therapies For novel therapies, information about the magnitude of the incremental costs and outcomes may not be available May need to be generated by assumption Data on the standard deviations for those who receive usual care/placebo may be obtained from feasibility studies or patient records One may assume that the standard deviation will apply equally to both treatment groups, or one may make alternative assumptions about their relative magnitudes The correlation also would be obtained from such data ssizeprg.do quietly do ssizeprg Contains 4 immediate form PROGRAMS related to sample size and power to detect NMB differences that are greater than 0 The command do ssizeprg simply loads these programs; it does not calculate anything Documentation program: ssizeprgdoc 3

4 ssizeprg.do (cont.) Programs for calculating sample size and power cess1i: Calculates sample size under the assumption that the standard deviations for cost and effect are common between the 2 treatment groups cess2i: Calculates sample size under the assumption that the standard deviations for cost and effect differ between the 2 treatment groups cepow1i: Calculates power to detect NMB greater than 0 under the assumption of common standard deviations cepow2i: Calculates power to detect NMB greater than 0 under the assumption that the standard deviations differ ssizeprg.do (cont.) All 4 programs presume two arm trials and a common sample size for both treatment groups These programs yield results that are identical to those derived from the NHB formula in: Willan AR. Analysis, sample size, and power for estimating incremental net health benefit from clinical trial data. Control Clin Trials 2001;22: ssizeprgdoc: cess1i * PROGRAM: CESS1I * cess1i is used to estimate sample size when one assumes * there are common standard deviations for cost and effect * between the 2 treatment groups (SDs, not SEs for the difference * in cost and effect). * COMMAND LINE: cess1i [diffc] [diffe] [sdc] [sde] [corr] [wtp] [alpha] [beta] * The 8 arguments are all numbers ** `1' Difference in costs ** `2' Difference in effects ** `3' Standard deviation, costs (assumed the same for both groups) ** `4' Standard deviation, effects (assumed the same for both groups) ** `5' Correlation, difference in costs and effects ** `6' Maximum willingness to pay ** `7' Two-tailed alpha level (e.g., 0.05) ** `8' One-tailed beta level (e.g., 0.80) 4

5 Saved results (scalars) * r(diffc) * r(diffq) * r(sd_c) * r(sd_e) * r(rho) * r(wtp) * r(alpha) * r(beta) * r(nmb) * r(sampsize) ssizeprgdoc: cess1i (cont.) Implementing cess1i Suppose the expected difference in cost is 25; the expected difference in QALYs is 0.05; the expected SDs for cost and QALYs are 1000 and 0.195, respectively; the expected correlation of the difference is -0.1; your maximum WTP is 75,000; and you want a 2-tailed alpha of.05 and a 1-tailed beta of 0.8: Point estimate = 25 / 0.5 = 500 Implementing cess1i (cont.). cess1i SAMPLE SIZE CALCULATION (Common SD Costs and Effects) Assumptions Difference in costs: 25 Difference in effects:.05 Standard deviation, costs: 1000 Standard deviation, effects:.195 Correlation, difference in costs and effects: -.1 Willingness to pay: Two-tailed alpha level:.05 One-tailed beta level:.8 Expected NMB: 3725 *** SAMPLE SIZE PER GROUP *** 246 5

6 Implementing cess1i (cont.). return list scalars: r(diffc) = 25 r(diffq) =.05 r(sd_c) = 1000 r(sd_e) =.195 r(rho) = -.1 r(wtp) = r(alpha) =.05 r(beta) =.8 r(nmb) = 3725 r(sampsize) = 246 Calculate Sample Sizes Compare the sample sizes required for the following expected results: What is happening? Sample Size Calculations Sample Size Parameters Sample Size

7 Sample Size Often More Sensitive to SD q than to SD c The sample size formula is symmetric for the SDs of cost and effect except for the following: (sd c0 + sd c1) + W (sd q0 + sd q1) Changes in the square of the QALY SD are weighted by the square of WTP; changes in the square of the cost SD are unweighted When WTP is substantially greater than SD for cost, percentage changes in the QALY SD will have a greater effect on sample size than will equivalent percentage changes in cost SD Calculate Sample Sizes (II) Compare the sample sizes required for the following expected results: What is happening? Sample Size Calculations Sample Size Parameters Sample Size Holding all else equal, when the correlation of the difference in cost and effect is negative, one needs a larger sample than when the correlation of the difference is positive 7

8 Calculate Sample Sizes (III) Compare the sample sizes required for the following expected results: What is happening? When WTP ~ PE, NMB ~ 0, Sample Size Sample Size Parameters Sample Size 41,831 39,412 At willingnesses to pay of 900 and 100, the expected value of NMB approaches 0 (when WTP = 900, NMB = 20; when WTP = 100, NMB = -20 Power to detect a difference is lowest as NMB appoaches 0 Checking Your Sample Size Calculation Based on your original design criteria, the sample size formula indicated you need 246 per group You decide to use ceapowersimulator to double check the sample size The program draws random samples of size 492 with the appropriate means, sds, and correlation quietly do ceapowersimulator ceapowersimulator When you look at the results, you find that 99.8% of the point estimates from your repeated samples -- which look very much like the cloud of points we plot on the CE plane -- are acceptable 8

9 300 Distribution of Point Estimates WTP: 75, Dif f er ence in Cost N = 246 / gr oup Dif f erence in Q ALYs Are we using the wrong formulas or are we looking at the wrong outcome of our simulation? Point Estimates Address the Wrong Question What are we trying to insure when we calculate sample size with an alpha of 0.05 and a 1-beta of 0.8? While 99.8% of the point estimates satisfy our willingness to pay of 75,000 per QALY, in only 79.9% of repeated experiments, do the 95% CI allow us to be 95% confident that the therapy is good value Implication: Sample size calculations are about CI in repeated experiments, they aren t about the distribution of point estimates from repeated experiments Experiments that Do and Do Not Yield Confidence 300 WTP: 75, Difference in Cost N = 246 / group Difference in QALYs 9

10 ssizeprgdoc: cess2i * PROGRAM: CESS2I * cess2i is used to assess sample size when one * assumes there are Rx-specific standard deviations * for the 2 treatment groups' costs and effects (SDs, not SEs for the difference in costs and effects) * COMMAND LINE: cess2i [diffc] [diffe] [sdc0] 9sdc1 [sde0] [sde1] [corr] [wtp] [alpha] [beta] * The 10 arguments are all numbers * `1' Difference in costs * `2' Difference in effects * `3' Standard deviation, costs, group 0 * `4' Standard deviation, costs, group 1 * `5' Standard deviation, effects, group 0 * `6' Standard deviation, effects, group 1 * `7' Correlation, difference in costs and effects * `8' Willingness to pay * `9' Two-tailed alpha level (e.g., 0.05) * `10' One-tailed beta level (e.g., 0.80) * Saved results (scalars) * r(diffc) * r(diffq) * r(sd_c0) * r(sd_c1) * r(sd_e0) * r(sd_e1) * r(rho) * r(wtp) * r(alpha) * r(beta) * r(nmb) * r(sampsize) ssizeprgdoc: cess2i (cont.) Implementing cess2i Suppose the expected difference in cost is 25; the expected difference in QALYs is 0.05; the expected SDs for cost are 800 and 1200; the expected SDs for QALYs are 0.19 and 0.20; the expected correlation of the difference is -0.1; your maximum WTP is 75,000; and you want a 2-tailed alpha of.05 and a 1-tailed beta of 0.8: 10

11 Implementing cess2i (cont.). cess2i SAMPLE SIZE CALCULATION (Different SD, Costs and Effects) Assumptions Difference in costs: 25 Difference in effects:.05 Standard deviation, costs, group 0: 800 Standard deviation, costs, group 1: 1200 Standard deviation, effects, group 0:.19 Standard deviation, effects, group 1:.2 Correlation, difference in costs and effects: -.1 Ceiling ratio: Two-tailed alpha level:.05 One-tailed beta level:.8 Expected NMB: 3725 *** SAMPLE SIZE PER GROUP *** 247 Implementing cess2i (cont.). return list scalars: r(diffc) = 25 r(diffq) =.05 r(sd_c0) = 800 r(sd_c1) = 1200 r(sd_e0) =.19 r(sd_e1) =.2 r(rho) = -.1 r(wtp) = r(alpha) =.05 r(beta) =.8 r(nmb) = 3725 r(sampsize) = 247 Calculate Sample Sizes Calculate the sample size for the case when the SDs for cost and QALYs were 500, 1500, and Calculate the sample size for common SDs of 1035 and (3.5% increases over the original example) 11

12 Separate SDs Tend to Increase Sample Size Sample Size Parameters Sample Size ssizeprgdoc: cepow1i PROGRAM: CEPOW1i * cepow1i is used to assess power when one assumes * that the 2 treatment groups have common standard * deviations for costs and effects (SDs, not SEs for the difference in cost and effect) * COMMAND LINE: cepow1i [diffc] [diffe] [sdc] [sde] [corr] [wtp] [alpha] [sampsize] * The 8 arguments are all numbers * `1' Difference in costs * `2' Difference in effects * `3' Standard deviation, costs (assumed the same for both groups) * `4' Standard deviation, effects (assumed the same for both groups) * `5' Correlation, difference in costs and effects * `6' Willingness to pay * `7' Two-tailed level (e.g., 0.05) * `8' Sample size per group Saved results (scalars) * r(diffc) * r(diffq) * r(sd_c) * r(sd_e) * r(rho) * r(wtp) * r(alpha) * r(sampsize) * r(nmb) * r(power) ssizeprgdoc: cepow1i 12

13 Implementing cepow1i Suppose the expected difference in cost is 25; the expected difference in QALYs is 0.05; the expected SDs for cost and QALYs are 1000 and 0.195, respectively; the expected correlation of the difference is -0.1; your maximum WTP is 75,000; you want a 2-tailed alpha of.05; and the current sample size plans are for 246 per group Implementing cepow1i (cont.). cepow1i POWER CALCULATION (Common SD Costs and Effects) Assumptions Difference in costs: 25 Difference in effects:.05 Standard deviation, costs: 1000 Standard deviation, effects:.195 Correlation, difference in costs and effects: -.1 Willingness to pay: Two-tailed alpha level:.05 Sample size per group 246 Expected NMB: 3725 *** POWER TO DETECT DIFFERENCE ***.799 Implementing cpow1i (cont.). return list scalars: r(diffc) = 25 r(diffq) =.05 r(sd_c) = 1000 r(sd_e) =.195 r(rho) = -.1 r(wtp) = r(alpha) =.05 r(sampsize) = 246 r(nmb) = 3725 r(power) =

14 Power Table (Example 1) Sample Size Power for WTP = 75, Power Graph: WTP Power WTP ssizeprgdoc: cepow2i PROGRAM: CEPOW2I * cepow2i is used to assess power when one assumes * there are Rx-specific standard deviations for for the * 2 treatment groups' costs and effects (SDs, not SEs * for the difference in costs and effects) * COMMAND LINE: cepow2i [diffc] [diffe] [sdc0] 9sdc1 [sde0] [sde1] [corr] [wtp] [alpha] [sampsize] * The 10 arguments are all numbers * `1' Difference in costs * `2' Difference in effects * `3' Standard deviation, costs, group 0 * `4' Standard deviation, costs, group 1 * `5' Standard deviation, effects, group 0 * `6' Standard deviation, effects, group 1 * `7' Correlation, difference in costs and effects * `8' Willingness to pay * `9' Two-tailed alpha level (e.g., 0.05) * `10 Sample size 14

15 Saved results (scalars) * r(diffc) * r(diffq) * r(sd_c0) * r(sd_c1) * r(sd_e0) * r(sd_e1) * r(rho) * r(wtp) * r(alpha) * r(sampsize) * r(nmb) * r(power) ssizeprgdoc: cepow2i Implementing cepow2i Suppose the expected difference in cost is 25; the expected difference in QALYs is 0.05; the expected SDs for cost are 800 and 1200; the expected SDs for QALYs are 0.19 and 0.20; the expected correlation of the difference is -0.1; your maximum WTP is 75,000; you want a 2-tailed alpha of.05; and the current sample size plans are for 246 per group Implementing cepow2i (cont.). cepow2i POWER CALCULATION (Different SD, Costs and Effects) Assumptions Difference in costs: 25 Difference in effects:.05 Standard deviation, costs, group 0: 800 Standard deviation, costs, group 1: 1200 Standard deviation, effects, group 0:.19 Standard deviation, effects, group 1:.2 Correlation, difference in costs and effects: -.1 Ceiling ratio: Two-tailed alpha level:.05 Sample Size: 246 Expected NMB: 3725 *** POWER TO DETECT DIFFERENCE ***

16 . return list Implementing cpow2i (cont.) scalars: r(diffc) = 25 r(diffq) =.05 r(sd_c0) = 800 r(sd_c1) = 1200 r(sd_e0) =.19 r(sd_e1) =.2 r(rho) = -.1 r(wtp) = r(alpha) =.05 r(sampsize) = 246 r(nmb) = 3725 r(power) =.799 Sample Size and Power Table (Example 1) WTP 50,000 75, , , ,000 Sample Size For 80% power Power for N = For this type of experiment, as one increases the WTP, sample size decreases and power increases Cost Minimization Suppose we are performing what we expect will be a cost-minimization study (cost savings but no difference in effect) We expect a cost savings of 1250 and difference in QALYs of 0.0; we expect an SD for cost of 5500 and for effect of 0.25; we expect the correlation of the difference to be 0.5; and we want an alpha and beta of 0.05 and 0.8, respectively 16

17 Calculate Sample Size and Power Sample size Calculate the required sample size for a WTP of 50,000 Calculate the required sample size for a WTP of 200,000 Power: Assuming a sample size of 1183 Calculate power for a WTP of 50,000 Calculate power for a WTP of 200,000 What happened? Sample Size Calculations Sample Size / Power Parameters Sample Size ,658 Power WTP 50,000 75, , , ,000 Sample Size and Power Table Sample Size for 80% power ,360 22,658 Power for N = For this kind of experiment, as WTP increases, the required sample size increases and power decreases Why? 17

18 C a Significantly Different from C b and Q a Not Significantly Different from Q b C = -1250; SE = 492; Q = 0; SE =.027; S.D. for effectiveness = 0.30 = 0.5; DOF = 498 Difference in Costs LL: 26,610 UL: -8, Difference in QALYs Dominance Similar kinds of issues can arise if you design your trial with the idea that you will document dominance In other words, you may be in a situation in which as you increase the WTP, the sample size decreases and power increases (e.g., if the lower limit remains in the lower right quadrant, and the upper limit is moving up into the upper right quadrant Or you may be in a situation where as the WTP increases, the sample size increases and power decreases (if the lower limit rotates into the lower left quadrant) Summary We ve provided you with programs to calculate sample size and power for the comparison of cost and effect In many cases, as one s WTP increases, the necessary sample size will be decreased and power will increase (a pattern 1 experiment) In other cases, as one s WTP increases, the necessary sample size will be increase and power will decrease (a pattern 2 experiment) 18