American Friends Service Committee (AFSC) Middle Atlantic Region (MAR) Washington DC Peace and Economic Justice Program E V A L U A T I O N O R I E N T A T I O N & E V A L U A T I O N M A D E E A S Y ( E V A M A E ) T r a i n i n g S e s s i o n T h u r s d a y F e b r u a r y 1 7, 2 0 1 1 1 1 : 0 0 a. m. 3 : 0 0 p. m. D r. L a v e r n e M o r r o w C a r t e r, P h. D., M P H R e s e a r c h a n d E v a l u a t i o n S o l u t i o n s, I n c. ( R E E S S I )
Goals and Objectives Goals To provide the staff and evaluation committee members with an overview of the evaluation activities. To provide the staff and evaluation committee members with a basic understanding of the purpose and strategies for assessing the effectiveness and efficacy of programs designed to mitigate social problems and issues. Learning Objectives Each participant will be able to describe the purpose, key activities, and deliverables of the evaluation project. Each participant will be able to define at least three activities that form the basis of program evaluation. Each participant will be able to list at least three types of evaluation activities and how they link to the stage of development of a program. Each participant will be able to prepare a program logic model with evaluation links.
AGENDA Time Actions 11:00 a.m. Introductions and Overview (Brooke McDonald) 11:20 a.m. The Evaluation Plan (L. Carter) Comments and Questions 11:50 a.m. EVAMAE Training (L. Carter) Pre-Test 12:00 12:45 p.m. 12:50-1:20 p.m. 1:00 2:40 p.m. 2:40-2:50 p.m. 2:45-3:00 p.m. 1) Problems, Issues, and Solutions Small Group Exercise 2) Key Elements of Program Planning Small Group Exercise 3) Measuring a Program s Success Small Group Exercise Review and Summary Post-Test Adjourn
Introductions & Overview Brooke McDonald Interim Associate Regional Director MAR American Friends Service Committee (AFSC)
Washington DC Peace and Economic Justice Program & Evaluation Plan 8-Week Project that began February 14, 2011 and will conclude April 7, 2011 The project plan consists of six key areas Project Start-up and Orientation Environmental Scan and Needs Assessment Identifying, Cataloguing and Assessing Associated Interventions Developing and testing process measures for current and future intervention Implementation assessment of the two current programs Recommendations for future program implementation activities Handout Evaluation Plan & Timeline
PRE-TEST
Key Components Problems, Issues, and Solutions Key Elements of Program Planning Measuring a Program s Success Handout Key Terms and Definitions
Problems, Issues, and Solutions Key Social Issues Poverty Low-No education and functional skills Low-No Jobs/Careers Low-No Housing Low-No Health Services Access Disease, Injury, & Disability High Chronic Diseases High Infectious Diseases High Intended and unintended injuries High Disabilities
Problems, Issues, and Solutions SOLUTIONS/PROGAMS A set of strategies and activities designed to improve the wellbeing of people (Kellogg Foundation, 1998) Planned learning experiences designed to facilitate volunteer changes in knowledge, attitudes, and behaviors (Green & Kreuter, 2005) Focus is usually at one or more levels of influence Individual Family Organization Community Policy
SMALL GROUP EXERCISE 1 Problems, Issues, and Solutions E A C H P E R S O N W A S P R O V I D E D A C O L O R E D C A R D W I T H A N U M B E R ( 1-3 ). P L E A S E G A T H E R W I T H T H E M E M B E R S O F Y O U R G R O U P. T H R E E A C T I V I T I E S W I L L B E C O M P L E T E D I N T H E S E G R O U P S. Y o u r t e a m w i l l h a v e 2 0 m i n u t e s t o r e v i e w a n d r e s p o n d t o t h e q u e s t i o n s i n W o r k s h e e t I, e n t i t l e d P r o b l e m s, I s s u e s, a n d S o l u t i o n s Handout Worksheet 1
Key Elements of Program Planning Population of Focus Important to assure appropriateness of strategies Age, Gender, Geography, Culture & Language, Socioeconomic Levels Is the strategy what the population desires and needs? The Goal and Vision Simple and practical statements that are easily written and spoken Goal=What are your trying to do? Vision=What does it look like when it is done? Program Objectives Administrative=resources, staffing, partners and collaborations, participant recruitment and activities Participant=numbers and reach, program doses, changes in KAB, and satisfaction levels
PROGRAM LOGIC MODEL HOW WHY Resources Outputs Short-term Outcome Long-term Outcome (Problem Solved) Activities Consumer- Beneficiary Intermediate Outcome IMPLEMENTATION RESULTS Adapted from McLaughlin & Jordan, 2004
SMALL GROUP EXERCISE 2 Elements of Program Planning Your team will have 20 minutes to review and respond to the questions in Worksheet 2, entitled Program Elements. Handouts Worksheet 2 and Sample Logic Model After Group Exercise
Measuring Program Success Defining Success Feeling Side Bessie Stanley, 1905 To laugh often and much; To win the respect of intelligent people and the affection of children; To earn the appreciation of honest critics and endure the betrayal of false friends; To appreciate beauty, to find the best in others; To leave the world a bit better, whether by a healthy child, a garden patch or a redeemed social condition; To know even one life has breathed easier because you have lived. This is to have succeeded. Peter Drucker, American Educator, Leadership and Management Guru-born in 1909 Efficiency is doing things right; effectiveness is doing the right things.
Measuring Program Success Defining Success Thinking Side 4 W s & the H What did you do? For Whom did you do it? How did you do it? How much did you do? Where did you do it? When did you do it? What changed (participants, organization, community)? What stayed the same? Where is the hard-core evidence of all of the above? What did it cost? (Future focus on cost-benefit)
Measuring Program Success Evaluation Functions Evaluation is a process for determining the merit, worth and value of something (Michael Scriven, 1991) Evaluation usually assesses one or more of six important domains (adapted from Rossi et al, 2004): Program Need (What are the needs?) Program Design (What is the program logic and/or theory?) Program Implementation and Service Delivery (How should the program operate?) Process Evaluation or Program Monitoring (Is the program operating as planned?) Program Impact or Outcomes (Is the program having its desired effects?) Program Cost Effectiveness (Are the program effects attained at a reasonable cost?) The Evaluation effort seeks to answer key questions under each domain through collecting and analyzing information and data related to each area in a structured and deliberate manner.
Measuring Program Success Process Assessment or Program Monitoring What is the fidelity and effectiveness of the program? How effective are: The administration and organization of the program The actual program activities How do the program activities compare to what was designed and planned? Program monitoring should involve an on-going system of accountability, not a one-shot review. Guidelines and personnel performance measures Plans and reports of activities Staff meeting agendas and minutes or notes Formal sign-in sheets Participant feedback and assessment documents
Measuring Program Success Outcome and Impact Program Level of Influence Individual Community Organization Policy Focus of Change Knowledge Attitudes Behaviors Particular Measureable Status Policies
Measuring Program Success Outcomes and Impact Short-term Outcomes (Up to 2 years) Knowledge (Surveys, Interviews, Case Studies) Awareness (Surveys, Interviews, Case Studies) Beliefs (Surveys, Interviews, Case Studies) Attitudes (Surveys, Interviews, Case Studies) Intermediate Outcomes (3-5 years) Individual Behaviors (Quantitative and real measures such as self-report, surveys, physical measures, personal records) Long-term Outcomes (6-10 years) Community Level Indicators (National reports and studies) Population Level Indicators (National reports) Policies (Government reports and legislative outcomes)
SMALL GROUP EXERCISE 2 Elements of Program Planning Your team will have 20 minutes to review and respond to the questions in Worksheet 3, entitled, Measuring Program Success. Handout Worksheet 3
REVIEW & SUMMARY OUR DEEPEST FEAR IS NOT THAT WE ARE INADEQUATE. OUR DEEPEST FEAR IS THAT WE ARE POWERFUL BEYOND MEASURE. IT IS OUR LIGHT, NOT OUR DARKNESS THAT FRIGHTENS US. Marianne Williamson Strength to Love, 1992 Handout Carter s Healthy Program Model
Program Evaluation Resources Books Rossi, Lipsey, & Freeman (2004). Evaluation: The Systematic Approach O Sullivan (2004). Practicing Evaluation: A Collaborative Approach Kapp & Anderson (2010). Agency-Based Program Evaluation: Lessons from Practice. Independent Publications W.K. Kellogg Foundation Logic Model Development Guide http://www.wkkf.org/knowledge-center/publications-and-resources.aspx CDC Evaluation Steps. http://www.cdc.gov/eval/steps.htm#evidence W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org/knowledge-center/publications-and-resources.aspx
POST-TEST
Laverne Morrow Carter, Ph.D., MPH Research and Evaluation Solutions, Inc. 6188 B Old Franconia Rd. Alexandria, VA 22310-3411 Lmcarter@reessi.com
Laverne Morrow Carter Bio Laverne Morrow Carter was born and raised in Kankakee, Illinois and is the oldest of ten children, raised by a single mother. She currently resides in Northumberland County, Virginia in the historic Northern Neck with her husband, Lawrence. Dr. Carter is a community health researcher and social entrepreneur, who possesses more than 30 years of experience in designing and evaluating community based health and social programs. She is the Founder, President and Chief Project Director of Research and Evaluation Solutions, Inc. (REESSI), a premier private research firm that is based in Alexandria, VA. Her clients include government and non-government agencies in multiple states. The firm evolved from Carter s 25 years of small business ownership experience and commitment to improving social conditions in vulnerable populations. REESSI s mission is to create, implement, and rigorously assess strategies and interventions that increase wealth, health, and life expectancy in communities of color around the world. The company s areas of focus are program planning and evaluation; intervention research; and community capacity building and engagement. During her professional career, she has been recognized for her innovative methods and programs by local policymakers and federal officials. Dr. Carter is a Fellow of the Salzburg Seminar (Salzburg, Austria) Session 406 on The Role of Non-Government organizations in Community Health (2003). From that experience, she and several colleagues in the United States established LiveFree, Inc., a non-profit, 501(c) 3 community based organization dedicated to the empowerment and training of coalitions of community members and frontline advocates to create and implement strategies that solve health and social problems at the community and policy levels. REESSI is the official program and evaluation consultant for LiveFree, Inc. In 1976, Dr. Carter received a Bachelor of Science degree with dual a major in Psychology and Communications from Illinois State University. She graduated from Washington University in St. Louis with a Master of Arts degree in Education (Counseling) in 1977. She received a Master of Public Health degree with a comprehensive concentration in Behavioral Health and Health Education in 2001 from St. Louis University. Dr. Carter completed a Doctor of Philosophy degree in Policy Analysis and Administration with a concentration in Health Policy Evaluation and Research at St. Louis University in 2005. Her greatest passion for this season of her career is training, mentoring, and empowering youth and young adults of color to create, implement, and rigorously assess strategies and interventions that increase wealth, health, and life expectancy in communities of color around the world.