Introduction to Evaluation

Similar documents
Nonprofit Budgeting Part 2: Building Better Budgets

Better Transparency and Accountability

This is a reminder regarding the requirement for annual commission work plans.

Nonprofit Budgeting Part 2: Building Better Budgets

Learning Objectives. Managing for Results 3/7/2016

2 nd INDEPENDENT EXTERNAL EVALUATION of the EUROPEAN UNION AGENCY FOR FUNDAMENTAL RIGHTS (FRA)

Weber State University Information Technology Division. Policy Guide

Nonprofit Budgeting Part 1: Budget Basics

SUMMARY OF BORROWER SURVEY DATA

Law Department Budgeting and Forecasting. How to Plan, Implement and Benefit From a Formal Budgeting Process

Government Pension Investment Fund

Al-Amal Microfinance Bank

FRAMEWORK AND WORK PROGRAM FOR GEF S MONITORING, EVALUATION AND DISSEMINATION ACTIVITIES

Managing operational tax risk through technology

True Program Costs: Program Budgets and Allocations

Pension Transfer Workshop eworkbook July 2018 version 3.3

Legal entity reduction: Savings on tap?

A guide to how we fund and deliver Flood Risk Management Schemes

Module 4. Table of Contents

3: marginally satisfactory Parliament prioritizes formation of committees Bring expertise expertise Advocacy and goodwill to form FS

Guidance for Outcome-Based Wellness Incentive Programs. Diane Andrea, RD,LD Wellness Consultant

Roundtable on Long-Term Investment Policy. Session Notes. 26 November 2014 L Hôtel du Collectionneur Paris

CHAPTER II-4 ROLE 4 PLANNING, DESIGNING, IMPROVING, OR ADVOCATING FOR PERFORMANCE MANAGEMENT SYSTEMS AND THEIR USE

Wellness, Social Media, and the Law

Mid Term Review of Project Support for enhancing capacity in advising, examining and overseeing macroeconomic policies

December A Teaching Moment. from Kansas. A cultural landscape memo and video. by the Topos Partnership

Chapter 1: Role of Performance Measurement in HUD CPD Formula Grant Programs

Customer Service QUICK EXPLORATORY SELF-ASSESSMENT GUIDE PRACTICAL TOOLS FOR SELF-ASSESSMENT. The Art of Service

Finance Mathematics. Part 1: Terms and their meaning.

Budgeting in a Chaotic Economic Environment Factors Leading to Improvement

Overseeing taxes in a new era

Spotlight: Robotic Process Automation (RPA) What Tax needs to know now

Performance Metrics and Budgeting. Paul L. Posner George Mason University May 18, 2011

Brief Introduction. To Completing the Logframe Matrix

REPORT ON FINANCIAL ACTIVITY FACILITATOR MANUAL WITH SIMULATED ONLINE BUSINESS ASSESSMENT BSBFIA402A

Office of the Superintendent of Financial Institutions Internal Audit Report on Insurance Supervision Sector

Natural Gas Demand Side Management Evaluation, Measurement, and Verification (EM&V) Plan

MANAGED INVESTMENT SERVICE

Charles Schwab Hong Kong Rising Affluent Survey

Creating A Program Budget

Oral History Program Series: Civil Service Interview no.: S11

Deferred Maintenance. Data Gathering and Presentation. By Tim McDonald 36 MAY/JUNE 2016 FACILITIES MANAGER

SAIIA 9 June Paul Baloyi. Development Bank of Southern Africa June 2011

Effective monitoring of outsourced plan recordkeeping and reporting functions

Congregational Socially Responsible Investing - Spectrum of Involvement

IFRS 4 Phase II Operational impacts

John F. Wegman Fund. Eligibility Questions. Rochester Area Community Foundation. Eligibility Conditions:*

How to Set Up Financial Ratios

The Multiplier Model

Focus on Energy Economic Impacts

Essential reading. If maximum possible points are scored at Disclosure level, 1 Leadership point is awarded (please see % Weightings tab for details).

Volunteer Instructor Notes

SMALL MARKET EMPLOYER SUMMARY

$$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$ $$$

Steps to Successful Backwards Planning: Make a list of everything that needs to be completed. The list should include:

Public Hearing on Solvency II Brussels Michael Koller. How does Solvency II compare with the Swiss Solvency Test

FOOD SAFETY RISK ANALYSIS

OPERATIONAL DUE DILIGENCE 3.0 RESPONDING TO A REGULATED AND INSTITUTIONAL ALTERNATIVE ASSET INDUSTRY

College Finance Department Program Review and Strategic Plan

Am unfunded mandate is a requirement that is passed

SIGMA Public Procurement Training Manual. Update 2015

Accounting. Overview 1/8/2019. Intro to Business and Technology

PERFORMANCE A NEW DIMENSION IN THE PURSUIT OF

Introduction to Financial Accounting

CALIFORNIA STATE UNIVERSITY Civic Engagement Questionnaire

Broad and Deep: The Extensive Learning Agenda in YouthSave

Emerson Electric: High-yield, Sound Valuation and 59 Consecutive Years of Dividend Increases, Part 2

WHEDA-Connect Administrators and Users Manual

Session Overview. Budgeting Skills Training - Instructor Notes. Thank you for teaching the Budgeting Skills Training Class :D

Development Cooperation Development Education Partnership Fair. Writing a successful co financing application

Delivery Execution. Case Studies in Operational Planning & Budgeting, Costing, and Technology Enablement

LUXEMBOURG ECONOMIC PROMOTION STRATEGY. Trade and Investment Board

Basic Introduction to Project Cycle. Management Using the. Logical Framework Approach

Modernization of WBC universities through strengthening of structures and services for knowledge transfer, research and innovation

Public Health Investment Fund

Analyzing the General Fund Reserve Risk Factors

Public Trust in Insurance

DC INVESTOR SURVEY. Biannual Report. Financial stress impedes employees ability to take action and hurts the corporate bottom line.

Allocation / Assessment

SAGE 100 ERP TIPS AND TRICKS BANK RECONCILIATION Presented by Target System Technology, Inc.

Leveraging Real-World Data and Analytics in the Device Industry. Tom Abbott Head, Healthcare Informatics Medical Device & Diagnostics

Professional Standards and Recognition Committee BUDGET REVIEWER S GUIDE Fiscal Year 2012/13

Fidelity National Title Insurance Company. Title Insurance Rates and Charges for the Commonwealth of Virginia

The Emergence of Economic Regionalism in Asia

NYISO Capital Budgeting Process. Draft 01/13/03

Blockchain: An introduction and use-cases June 12 th, 2018

From Cost to Value: Reframe How You Measure Travel. The Link Between Business Strategy and Travel Cost- Savings. How to Manage Hidden Travel Costs

Risk Management Plan PURPOSE: SCOPE:

REPORT 2015/178 INTERNAL AUDIT DIVISION. Audit of the United Nations Human Settlements Programme Regional Office for Arab States

SMALL RURAL HOSPITAL IMPROVEMENT GRANT PROGRAM (SHIP)

Resource Allocation and Decision Analysis (ECON 8010) Spring 2014 Foundations of Decision Analysis

JCT 2016 Edition of Contracts

New strategies to ensure sufficient food provisions in the case of crisis (NeuENV)

Benefits and Costs of the Health Targets for the Post-2015 Development Agenda

Key Risk Indicators (KRI) Survey September 2011

The Only Withdrawal Plan You Will Ever Need

PROJECT DESIGN FOR PSD:

Spreadsheet Risk - A new direction for HMRC? Don Price HM Revenue & Customs Eden House, Chester, CH4 9QY

City of Port Moody Citizen Survey. Presented by: Catherine Knaus, Ipsos Reid

Audit Report Internal Financial Controls. GF-OIG March 2015 Geneva, Switzerland

Transcription:

Introduction to Evaluation A N O V E R V I E W O F E V A L U A T I O N & T H E L O G I C M O D E L R e b e c c a S e r o, P h. D. E v a l u a t i o n S p e c i a l i s t W e b i n a r p r o d u c e d f o r W a s h i n g t o n S t a t e U n i v e r s i t y E x t e n s i o n A u g u s t 2 6 th, 2 0 1 5

Agenda Part 1: Introduction to Evaluation Why Evaluate? When to Evaluate? How to Evaluate? Part 2: Using a Logic Model Framework Steps in an Evaluation Process Part 3: Evaluation Tips

Research seeks to prove, evaluation seeks to improve M.Q. Patton

If the Goal of Evaluation is to improve a program Then an evaluation holds little worth unless findings are used.

Why Evaluate? To gain insight about a program and its operations to see where we are going and where we are coming from, and to find out what works and what doesn t To improve practice to modify or adapt practice to enhance the success of activities To assess effects to see how well we are meeting objectives and goals, how the program benefits the community, and to provide evidence of effectiveness To build capacity - increase funding, enhance skills, strengthen accountability

Why Evaluate: Extension? One of the primary purposes of evaluation within Extension is to improve the quality of the programs offered It allows stakeholders to determine the programs assets and weaknesses Make appropriate changes Extension programs, no matter how large or small, benefit from a review or assessment to see if they accomplished the stated objectives.

When to Evaluate? Planning a NEW program Assessing a DEVELOPING program Assessing a STABLE, MATURE program Assessing a program after it has ENDED Conception Completion The stage of program development and desired information influences the reason and type of program evaluation.

How to Evaluate: Types of Evaluation Needs Assessment Process Evaluation Outcome Evaluation Impact Evaluation

How: Needs Assessment Approaches to Needs Assessment: Determine what need is Allow stakeholders or others to help define the relevant factors Search for evidence of the problem or lack thereof Questions: What are the characteristics, needs, priorities of target population? What are potential barriers/facilitators? What is most appropriate to do

How: Process Evaluation Approaches to Process Evaluation: What an organization is supposed to do and what it actually does are two different things. Reveals how a plan is actually implemented. Keeping detailed data on the process allows you to say for whom the program is effective, for whom it is not, and sometimes why. Questions How is program implemented? Is delivery of service adequate, uniform? Are activities delivered as intended? Fidelity of implementation? Are participants being reached as intended? What are participant reactions?

How: Outcome Evaluation Approaches to Outcome Evaluation: Is used to determine whether change resulted because of participation in the program. Overall, it attempts to link the change to a specific part of the program. Questions To what extent are desired changes occurring? Goals met? Who is benefiting/not benefiting? How? What seems to work? Not work? What are unintended outcomes?

How: Impact Evaluation Approaches to Impact Evaluation: Examines the longer-term, deeper changes that are potentially a result of the program Questions To what extent can changes be attributed to the program? What are the net effects? What are final consequences? Is program worth resources it costs?

Using a Logic Model Framework Logic Model Courtesy of University of Wisconsin Extension: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Logic Model in Evaluation Provides the program description that guides evaluation process Helps match evaluation to the program Helps know what and when to measure Are you interested in process and/or outcomes? Helps focus on key, important information Prioritize: where will limited evaluation resources be spent? What do we really need to know??

Steps in an Evaluation Process What are the desired outcomes of this program? What are the goals? What is trying to be accomplished within the next month/quarter/year(s)? What activities will enable the program to reach its outcomes? How will it get there? What resources are available to help the program achieve the desired outcomes? What will be used to implement the program?

Steps in Establishing an Evaluation Inputs: The resources, raw materials, clients, and staff that go into a program. Program Process: The service delivered by a program. Outputs: The services delivered or new products produced by the program process. Outcomes: The impact of the program process on the state of the target population or the social conditions that a program is expected to have changed.

Evaluation Example For example, examining a food nutrition education program: Inputs: Program Coordinator, assistant, nutrition education manuals, videos, schools provide teen participants Program Process: Program provides in-school activities detailing how to make better food choices Outputs: Some number of teens attend programs at some number of schools. Outcomes: Initial/Short: Teens know more about nutritional food choices. Intermediate/Mid-term: Teens influence others to make better food choices. Long-term: Healthier community and lower obesity related rates.

Logical chain of connections showing what the program is to accomplish INPUTS OUTPUTS OUTCOMES Program investments Activities Participation Short Medium Longterm What we invest What we do Who we reach What are the results

Fully detailed logic model Logic Model Courtesy of University of Wisconsin Extension: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

OUTPUTS What we do Who we reach ACTIVITIES PARTICIPATION Train, teach Deliver services Develop products and resources Network with others Build partnerships Assess Facilitate Work with the media Participants Clients Customers Agencies Decision makers Policy makers Satisfaction

Steps in an Evaluation Process The activities are the interventions that your program will provide in order to bring about the intended outcomes. Programs offer all sorts of different activities to address their desired outcomes For the most part, program activities can be classified as any type of direct service or information that is provided to participants The participants are those on the receiving end of program activities.

OUTCOMES What results for individuals, families, communities.. SHORT Learning MEDIUM Action LONG-TERM Conditions Changes in Changes in Changes in Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation Behavioral intent Behavior Decision-making Policies Social action Conditions Social (well-being) Health Economic Civic Environmental C H A I N OF O U T C O M E S

Steps in an Evaluation Process Outcomes should be consistent with what could reasonably be accomplished Provide a foundation for all subsequent program implementation and evaluation activities Each of the outcomes should be evaluated

Logic model in evaluation What do you want to know? How will you know it? EVALUATION: check and verify

Logic Model & Common Types Of Evaluation

Evaluation Tips Outputs and outcomes may already be expected of or available to the program Review relevant documents from the funding agency and the larger organization to determine what outcomes are of greatest priority Ascertain what measures of effectiveness are used by other agencies (governmental, non-profit, etc.) and determine if it makes sense for the program to use the same or similar

Evaluation Tips When creating data collection instruments, you should revisit program s outputs and outcomes to develop questions that address: What you want to know What information you need to capture

Evaluation Tips When gathering data: It s essential to utilize data collection tools from the outset of the project If a data collection system is not developed early, it will be difficult to gather accurate data Streamline your process as much as possible Gather similar outputs and outcomes where possible Use technology to the greatest extent possible Excel spreadsheets, online data collection

Evaluation Tips For funding purposes: Be sure that outputs and outcomes are representative of the funding / grant A mismatch between desired outcomes and the funder s intentions could prevent funding. Once funded, a mismatch could jeopardize continued and/or future funding.

Evaluation Tips On an annual basis: Revisit the stated inputs, outputs, including activities, and outcomes to ensure they are focused, pragmatic, and provide a current roadmap for the program Reviewing these items will ensure that the program and its evaluation are proceeding as expected

Questions? Please feel free to get in touch if you have questions: r.sero@wsu.edu 509-358-7879