Cost-Effectiveness Analysis and Cost-Benefit Analysis. Dagmara Celik Katreniak HSE

Transcription

1 Cost-Effectiveness Analysis and Cost-Benefit Analysis Dagmara Celik Katreniak HSE

2 Proposal Presentations Work in a pair or alone? Pick a date: November 17 th, 2014 November 24 th, 2014 December 1 st, 2014 December 8 th, 2014

3 Overview So far: Randomization Experimental design Sample Size and Power Calculations Next: Cost-Effective Analysis, Cost-Benefit Analysis Data Analysis Topics

4 Next class Data analysis Read: Pitfalls of Participatory Programs (Banerjee et al) Worms: Identifying impacts on education and health in the present of treatment externalities (Miguel and Kremer)

5 Terminology Minimum Detectable Effect Type I error, Type II error Connection between Minimum Detectable Effect, sample size and the level of power? Statistic versus Scientific significance Level of randomization Clustering versus stratification Imperfect Compliance versus Sample size/power

6 Today Cost-Effectiveness Analysis and Cost-Benefit Analysis Data collection Outcome Variables Data analysis Questions Case study ***

7 Notes to covariates Adding sum_r1 into regression of sum_r5 on treatment Adding covariates (baseline survey!) decreases variance of estimated coefficient does not influence size of effect but its significance Be careful! controlling for covariate influenced by treatment biases the treatment effect cause it captures part of its impact use covariates from baseline, esp. pre-treatment value of outcome variable Covariates should be chosen in advance to avoid data mining

8 Notes to stratification After randomization treatment and control are similar in expectations After stratification treatment and control groups within stratas are similar in practice along the stratified variables Pairwise matched design When stratified what is the average treatment effect? Weighted average of difference between T-C across all stratas Blocking versus controlling for observables Blocking more efficient, reduces variance

9 Notes to design Cross-cutting design Solo A, Solo B, A+B, control A+B and control Solo A and control / solo B and control Policy relevant research vs. academic research Important due to cost-reduction

10 Budget How to plan your budget? Main sections Main travel cost (visa, flight, vaccination, per-diem) Technical equipment Salary for you and your team Accommodation, office Rewards for subjects Office support Transportation in the field Overhead Other expenses (research permit, ) Ideally 10-15% as safety fund Exchange rate differences, Inflation

11 Where to get funding? International funding USAID, UNHCR, UNICEF,.. 3IE, GDN, National funding University grants Governmental grants Firms Individuals / Foundations Bill and Melinda Gates, Karel Janecek,

12 Cost Effectiveness Ratio of costs to effects or effects to costs Costs for a given level of effectiveness How costly is additional year of schooling? Effectiveness level for a given cost Given extra $100, how will attendance change? Ratio can be compared to other studies Conditional on similar computation Helpful mainly for policymakers to be able to choose between multiple programs What are the most cost-effective ways to increase school attendance? J-PAL training course, 2011

13 CEA versus CBA Cost-Effectiveness Analysis (CEA) Outputs of the project that are non-monetary E.g. sick days, stress level, happiness level, etc. Cost-Benefit Analysis (CBA) Monetary comparison of the benefits and costs of a project E.g. savings in USD in reduced sick days, etc. Measure of economic worth of a project Question is how to assign value to different benefits (e.g. saved life, lowered stress level, etc.) CEA versus CBA Key difference is the unit of measurement Dollars (CBA) versus reduction in stress (CEA) J-PAL training course, 2011

14 CEA versus CBA CBA allows for the judgment whether a program is worth implementing or not Hard to measure CEA does not require measurement assumptions Relatively simple, easy to compare to other studies of the same outcome of interest Does not offer monetary value of a project Despite multiple impacts, CEA only compares costs to effect on one outcome J-PAL training course, 2011

15 Comparative Cost-Effectiveness in the field Rare to see within a country Even more rare across countries Why? Lack of data Different time of program implementation Different exchange rates, inflation, discount rates, etc. Hard to replicate studies (limited funding) No incentives to do CEA Examples: Kremer, Miguel and Thornton (2005) Banerjee, Cole, Duflo and Linden (2007) J-PAL training course, 2011

16 Comparative Cost-Effectiveness in the Reasons: field Time and budget limited Learning from past mistakes Better implementation of programs

17 C-E comparison of Deworming Program to other programs Source: J-PAL

18 Quantification of Impacts (1) Good measures the choice of outcome variables Multiple Impacts We can only compare costs of an intervention to effects on one outcome Imprecision in Estimating Impacts Imprecision coming from data collection Non-significant results are also results There is no effect or there is no power to confirm the effect Include even insignificant results in the analysis J-PAL training course, 2011

19 Quantification of Impacts (2) Spillover Effects Not easy to capture Deworming program Spillover effect type 1: effect on untreated Spillover effect type 2: effect on control group Which spillover effect to count? J-PAL training course, 2011

20 Partial versus General Equilibrium The impact of an intervention tested on a subsample is positive What would be the effect if the intervention was scaled up to the national level? Both higher and lower effect possible Declining marginal benefit of the intervention Increase attendance from 40-50% is easier compared to attendance from 85-90% Do not estimate general equilibrium effect J-PAL training course, 2011

21 Aggregate Impact Aggregate Impact = Impact per unit * Sample size * duration Time = to make sure it comes from the same time interval Proximal effect may need to be recalculated to the final effect (e.g. increase in water chlorination incidence of diarrhea) J-PAL training course, 2011

22 Quantification of Costs (1) Keep your budget up-to-date and detailed Check J-PAL for general worksheet Talk to researchers, people with local experience J-PAL training course, 2011

23 Quantification of Costs (2) What was in the field already and what is new? Computer learning program Usage of already existing computers Purchase of new computers Additional teacher (balshaki program) Impact of the additional teacher + impact of having smaller classes Goods and services provided for free is it cost? Think about larger scale; Always free, subsidized or just now? What is the market situation? J-PAL training course, 2011

24 Quantification of Costs (3) Costs of user time, cost to beneficiaries If time not voluntarily used In terms of local wages (usually low) Overlapping goods and services Hard to disentangle sometimes Anything paid before the implementation of the project should not be included Transfers E.g. cash transfer programs Administrative costs to be included, cash transfers themselves? Overhead (usually given by organization) J-PAL training course, 2011

25 Quantification of Costs (5) Inflation Inflate forward to today as a base Inflate backward to some base year Exchange rates Exchange all costs to common unit Disadvantage exchange rate reflect prices of tradables; Not real purchasing power differences documented E.g. skilled labor is much cheaper in India than in Mexico, so a highly labor-intensive program piloted in Mexico will appear less cost-effective when adjusted by PPP Standard versus PPP Exchange rates J-PAL training course, 2011

26 CEA/CBA to conclude Not popular and not widely implemented Hard to do follow up studies and compare CEA/CBA Funding Low academic and publication motivation Clear benefits of CEA/CBA J-PAL training course, 2011

27 CEA/CBA Literature Recommended source for cost-effectiveness analysis Henry Levin and Patrick McEwan (2001): Cost- Effectiveness Analysis, book, SAGE Publications, Inc; J-PAL website J-PAL training course, 2011