The Need for Standards: Lessons from Benefit Cost Analysis of Communities That Care

Size: px

Start display at page:

Download "The Need for Standards: Lessons from Benefit Cost Analysis of Communities That Care"

Calvin Hopkins
5 years ago
Views:

for Benefit Cost Analysis of Preventive Interventions

of Benefit Cost Analysis of Preventive Interventions

November 18 19, 2013 Margaret Kuklinski, PhD Social

1 The Need for Standards: Lessons from Benefit Cost Analysis of Communities That Care Workshop on Standards for Benefit Cost Analysis of Preventive Interventions for Children, Youth, and Families Session 1: Overview of Benefit Cost Analysis of Preventive Interventions for Children Institute of Medicine Washington, DC November 18 19, 2013 Margaret Kuklinski, PhD Social Development Research Group, School of Social Work University of Washington

Social Development Research Group: Improving Lives Through Research Develop, Implement, Test Preventive Interventions Promote Positive Youth

2 Social Development Research Group: Improving Lives Through Research Develop, Implement, Test Preventive Interventions Promote Positive Youth Development Reduce Risk & Enhance Protection Community, School, Family, Individual What are the economic implications of better youth outcomes?

3 As Studies & Applications Increase So Does the Need for Standards RANGE OF TOPICS & PROGRAM AREAS Benefit cost analysis of the Perry Preschool Program and its policy implications. Barnett, Substance Use Prevention Dollars and Cents. Miller & Hendrie, Benefits and costs of intensive foster care services: The Casey Family Programs compared to state services. Zerbe et al., Hands on guide to cost benefit analysis of crime prevention efforts. Jacobsen, Can we compare findings across studies? DETAILED SUMMARY INFORMATION Net Present Value Benefit Cost Ratio Internal Rate Of Return Cash Flows Investment Risk Discount Rate Benefits & Costs By Stakeholder: Participants, Taxpayers, Others What are the best reporting practices? INCREASING USE IN POLICY DECISIONS Washington State Institute for Public Policy Consumer Reports lists comparing BCAs Pew MacArthur Results First Initiative Increasing states use of BCA in policy WA E2SHB 2536 (2012) Greater funding and support for evidence based programs, defined partly on BCA results How do we ensure a level playing field?

Opportunity costs: teachers, volunteers, board members? Estimating Benefits Significant effects or all effects?

4 Where Are Standards Needed? Research Design Quality Well designed RCT Assessing Costs Cost estimate in multisite trial? Opportunity costs: teachers, volunteers, board members? Estimating Benefits Significant effects or all effects? What discount rate? Shadow Prices Projection Period Deadweight Cost of Taxation Uncertainty Potential for Double Counting Benefits Marginal v. Average Costs Monte Carlo Methods Outcomes to Measure Point Estimates & Confidence Intervals

5 CTC Reduced Delinquency & Substance Use Initiation Through Grade 12 Randomized controlled trial 24 communities, 7 states Longitudinal panel: 4,407 students 10 Year Study: Grade 5 to Age 19 Grade 12 Results: Lower rates of initiation community wide Significantly lower initiation: Delinquency, Alcohol use, Cigarette smoking Non significant but in expected direction: High school graduation, marijuana use, other drug use

6 Investments in CTC Promote Positive Youth Development Coalition of diverse stakeholders owns and operates CTC. Survey youth: Identify epidemiological levels of risk and protection. Target most widespread risks for intervention. Implement evidence based programs to address prioritized risk factors. Monitor fidelity, assess efficacy, make course corrections. Coordinator oversees everything.

7 Investments Were Made in 4 Major Cost Categories CTC Investments per Community: $745,000 * for 5 Years Thousands of Dollars $800 $600 $400 $200 $32 (4%) $174 (23%) $264 (35%) $276 (37%) Other Contributions Training, Technical Assistance, Monitoring Prevention Programs Coalition & Coordinator $ $0 1 * Discounted 2011 Dollars

8 How Do We Estimate Costs Per Youth When Costs Per Youth Vary Across Sites? $6,000 $5,730 / / Cost per Youth Discounted 2011 Dollars Youth Ages $2,000 $1,500 $1,000 $500 $0 $283 * Weighted Average = $ % per youth for 5 years * $390 $413 Simple Average = $1,159 * Median = $634 $468 $466 $567 $700 $745 $748 $1,621 $1, Largest Smallest CTC Communities * Discounted 2011 Dollars

How Do We Value Non budgetary Economic Resources?

offset by benefits volunteers & teachers receive.

9 How Do We Value Non budgetary Economic Resources? BOARD MEMBERS VOLUNTEERS TEACHERS Option 1: Time cost is fully offset by benefits volunteers & teachers receive. Option 2: Value time at appropriate wage plus fringe rate for the volunteer position. Option 3: Value time at volunteer s own wage plus fringe rate. $556 + $89 per youth $$$ INCREASING OPPORTUNITY COST $$$ Discounted 2011 Dollars

100% 80% 60% 58% Significant Effects: p <.05 Non significant Effects: p >.

10 Should We Monetize Significant Effects or All Effects? What Discount Rate? Through Grade 12: CTC youth reported lower initiation of delinquency and substance use, and more graduated high school. 100% 80% 60% 58% Significant Effects: p <.05 Non significant Effects: p >.05 75% 67% 67% 57% 50% 52% 47% 40% 43% 83% 79% 40% 20% 0% CTC Control Delinquency Alcohol Use Cigarette Smoking Marijuana Use Other Drug use High School Graduation Total Benefits * Significant Effects Only 3.5% Discount Rate: $4, % Discount Rate: $2,312 Total Benefits * All Effects 3.5% Discount Rate: $8, % Discount Rate: $4,070 * Discounted 2011 Dollars

11 Different Viable Assumptions Lead to a Range of BCA Conclusions Range of Estimates * Low High SDRG Analysis Assumptions Significant Effects, Opportunity Costs = $89, 7% Discount Rate All Effects, Opportunity Costs = $0, 3.5% Discount Rate Significant Effects, Opportunity Costs = $0, 3.5% Discount Rate Benefits $2,312 $8,123 Costs (635) (556) Net Present Value $1,667 $7,617 Benefit Cost Ratio $4,477 (556) $3, * Discounted 2011 Dollars

12 Summary Benefit cost analysis in prevention has come a long way More publications, diverse topics and programs Wealth of useful summary statistics and information Increasing policy application With this growth, the need for standards and methodological guidance has increased We need to be able to compare findings across studies Policy, funding, uptake implications of comparative BCAs underscore the need for standards

13 Summary Different viable assumptions lead to different bottom lines Today s focus: 3 dimensions related to cost and benefits estimates Many other decision points where standards would be helpful: Monte Carlo methods, shadow prices, which outcomes to measure, relevant projection period

14 Conclusion Challenge for the field and for a possible IOM panel: Identify areas and assumptions where there is common ground Where consensus is less clear, delineate the major alternatives Researchers, policy makers, and practitioners would benefit from guidance that increases methodological consistency and capacity to compare results across studies.

15 Thank you Please contact Margaret Kuklinski, PhD, with any questions or comments: Social Development Research Group School of Social Work University of Washington

NLPES Excellence in Evaluation Award Submission New Mexico Legislative Finance Committee Program Evaluation Unit Narrative

NLPES Excellence in Evaluation Award Submission New Mexico Legislative Finance Committee Program Evaluation Unit Narrative Introduction. The New Mexico Legislative Finance Committee s (LFC) Program Evaluation Unit is the accountability arm of the New Mexico Legislature. The LFC has effectively integrated key legislative functions,