Taking a Critical Look at Cost-Benefit Analysis as Part of an Evaluation Catherine Mueller February 21, 2013
Overview 2 Directive of the Treasury Board of Canada Secretariat Policy on Evaluation: demonstration of economy and cost-benefit analysis Usefulness of cost-benefit analysis in evaluation Issues with cost-benefit analysis in evaluation Role of evaluators in a cost-benefit analysis List of key elements required for a critical reading of a costbenefit analysis
Directive of the Treasury Board of Canada Secretariat Policy on Evaluation Some definitions from the Treasury Board of Canada Secretariat Centre of Excellence for Evaluation: 3 Economy: minimizing the use of resources. Economy is achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes. Effectiveness: the extent to which a program is achieving expected outcomes. Efficiency: the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produced the same level of output. The level of input and output could be increases or decreases in quantity, quality, or both.
Directive of the Treasury Board of Canada Secretariat Policy on Evaluation (cont.) 4 Directive of the 2009 Policy on Evaluation: Core Issues Demonstration of economy Optimization of effectiveness and efficiency Core Issue 5: Assessment of resource utilization in relation to the production of outputs (costs) and progress toward expected outcomes (benefits) Cost-benefit analysis may be selected as a line of investigation in an evaluation
Usefulness of Cost-Benefit Analysis in an Evaluation 5 A decision aid that can be used to evaluate the benefit of a project, program or regulations to society The analysis can help with judging and evaluating various strategic choices in terms of their consequences for all groups of society affected by the decisions Provides a framework for analyzing available data in a structured format (costs and benefits)
Usefulness of Cost-Benefit Analysis in an Evaluation (cont.) 6 Facilitates collaboration between the various stakeholders by providing a structured framework Transparency of the decision-making process: identification and quantification of all elements by providing a structured framework for decision making Investigate the opportunity costs of the public activity: measurement of the benefits of one public activity that would be forgone if available resources were allocated to another alternative
Issues with Cost-Benefit Analysis in an Evaluation 7 Onerous research Analysis is complex to implement, particularly for obtaining data on specific types of effects Strategic biases Economic value is based on individual preferences and measured through surveys Respondents can answer strategically to obtain something from decision makers Limited accuracy Sometimes intangible factors are difficult to measure but useful for decision making
Issues with Cost-Benefit Analysis in an Evaluation (cont.) 8 Possibility of slanting analyses Assumptions may be slanted to favour one project or decision over others
Type of Questions for Cost-Benefit Analysis 9 Does the proposal provide a net benefit for society as a whole? Should the project, program or regulations under consideration be implemented? Should the project or program be continued? Of various alternative projects, which should be selected?
Role of Evaluators in Cost-Benefit Analysis 10 Establish the organizational prerequisites Justify the cost of a cost-benefit analysis Independence in the conduct of the study Recognize criticisms and limitations of the study and include them in the final evaluation report Specify the various scenarios Define the choices with the stakeholders and experts/consultants Improved status quo vs. other alternatives with public intervention
Role of Evaluators in Cost-Benefit Analysis (cont.) Define the scope of the analysis Task carried out in collaboration with stakeholders and experts/consultants Scope of the analysis: determine which groups will bear the consequences that will be identified Groups affected directly and indirectly by the decision 11 List the consequences and select their measurement indicators Task carried out by experts/consultants Measurement of impacts: identify all project impacts Negative impacts/costs and positive impacts/benefits Use of proxy, a variable close to that for which data is being sought, about which more information is available
Role of Evaluators in Cost-Benefit Analysis (cont.) 12 Make a quantitative prediction of the consequences over the life of the project Task carried out by experts/consultants Each project scenario will have effects for several years after implementation Scientific uncertainty: some predictions require scientific knowledge that may be uncertain at the time of project start-up Monetize the consequences Task carried out by experts/consultants Willingness to pay for market goods
Role of Evaluators in Cost-Benefit Analysis (cont.) 13 Discount future costs and benefits Task carried out by experts/consultants Discount rate Net present value Average annual net benefit Analyze the robustness of the results Task carried out by experts/consultants Uncertainty analysis Recommend a decision Task carried out by evaluators Select the scenario that offers the greatest net benefit Cost-benefit analysis recommends the implementation of the project that offers the greatest increase in societal welfare
List of Key Elements Required for a Critical Reading of a Cost-Benefit Analysis 14 The following elements must appear in the cost-benefit analysis technical report: Summary or executive summary outlining the critical assumptions, main results and recommendations Background of the study: why cost-benefit analysis was chosen as a line of investigation in the final evaluation report Objectives of the project(s), program(s) or activity/activities studied Main risks associated with possible decisions Description of scenarios studied Timeline of onset of costs, benefits and net benefit
List of Key Elements Required for a Critical Reading of a Cost-Benefit Analysis (cont.) 15 The following elements must appear in the cost-benefit analysis technical report: Main assumptions related to the analysis Discount rate used Calculation of net present value Uncertainty analysis Other important information, such as distributive effects, other quantified costs and benefits, and decision factors that were not quantified Comparison of the preferred option with its alternatives
References 16 Meunier, Valérie and Éric Marsden (2009). L analyse coût-bénéfices Guide méthodologique, Les cahiers de la sécurité industrielle, Institut pour une culture de sécurité industrielle, 47 p. Treasury Board of Canada Secretariat (2007). Canadian Cost-Benefit Analysis Guide: Regulatory Proposals, Government of Canada, 51 p. Lamari, Moktar (2012). Méthode de mesure de rendement des programmes publics Introduction Séance 1, École nationale d administration publique, September 7, 17 slides. Florio, Massimo and Silvia Maffii (2008). Guide to Cost Benefit Analysis of Investment Projects: Structural Funds, Cohesion Fund and Instrument for Pre- Accession, Directorate General Regional Policy, European Commission, July, 257 p.
17 For questions or comments: catherine.mueller@enap.ca crexe.enap.ca Thank you!