Mid-Term-Evaluation on behalf of WHH Country Office Afghanistan 1 Introduction 15.06.2017 Country: Afghanistan Project title: Strengthening Food Security and Natural Resource Management in Jawzjan province in Northern-Afghanistan" Project No.: AFG-1177 Project holder: Welthungerhilfe Approved budget: Deutsche Welthungerhilfe e.v. Committed funds: 2.5 Mio EUR Co-financer (line): German Federal Ministry for Economic Cooperation and Development (BMZ Private Agencies) Project period: 01.12.2015 31.10.2018 Welthungerhilfe (WHH), formerly known German Agro Action (GAA),Is a German based, non-profit making, non-aligned international NGO that aspires to see a world without hunger and poverty. Welthungerhilfe's activities in Afghanistan began back in 1980 - immediately after the former Soviet Union's occupation - to provide emergency aid for refugees. In 1992, Welthungerhilfe further increased their emergency aid activities and services, as well as rehabilitation and reconstruction assistance. Currently, WHH s draws most of its funding from the German government as well as from donations of the German public with the aim to reduce the level of poverty while improving the livelihoods for the most vulnerable population. Since Dec 2015, North Regional Office of Welthungerhilfe Afghanistan in Jawzjan - in partnership with one local NGO (PRB Partners Revitalization and Building) has been implementing a project for food insecure people in three districts (Sheberghan, Khanaqa and Aqcha) of Jawzjan province. The project main objective is to improve the food and nutrition security in the three districts of Jawzjan. In particular the population of the three districts of Jawzjan is to be supported to regain and improve their livelihoods through activities in the area of agriculture and livestock breeding, internal production of food as well as restoration of and better handling of natural resources. Project Overall Objective (Impact): The food and nutrition security in the three districts of Jawzjan has improved. Project objective (Outcome): The population of the three districts of Jawzjan was able to restore and improve its livelihood. Project Outputs / Expected Results: 1- Farmers can improve their productivity, their diversity and their land management. (Availability, access and stability).
Target Indicators: a) By the project end, 40% of the 800 target farmers increase their productivity by at least 20% and cultivate more different plants (exact value placement takes place after the baseline study). b) By the project end, at least 15% more land is irrigated for production purposes than the current land area recorded in the baseline study. c) In the year after the PTD (participatory technology development), 50% of the 800 target farmers use techniques for increased diversity and productivity. (Behavioural change /economic status can hinder). d) By the project end, at least 60% of the 800 beneficiary farmers have introduced measures to protect soil and water, such as drip irrigation, erosion protection measures, dry crops. e) By the project end, 18 farmers (of 24 beneficiaries from the 4 pilot units) achieve at least a 20% higher income from the processing and marketing of selected marketable products. 2- Livestock owners can improve the productivity of livestock (availability and access). Target Indicators: a) By the project end, at least 50% of the 1,200 beneficiaries adopt improved management practices and receive better advisory services. b) In the year after the implementation of the FFS, 50% of the 1200 FFS graduates apply improved (including environmentally friendly) practices in livestock farming. c) By the project end, the livestock owners have restored a total of two hectares of private pastureland. d) By the project end, local veterinary assistants deliver services to 600 or 50% of the trained livestock owners. e) By the project end, 1,200 (800 male and 400 female) livestock owners have access to medicine and vaccines. f) By the project end, at least 8 monitoring reports and at least 8 financial and narrative reports have been submitted by the implementation partner all of them in accordance with international standards, accurate and in time. 3- Women (of child-bearing age) can ensure better health and nutrition in the household (use and utilisation). Target Indicators: a) By the project end, at least 40% of the 432 beneficiary households (HH = 7 members, according to UN) show a generally better nutritional diversity (use of more than 4 food groups), "improved food consumption score" based on the results of the baseline study. b) In the year after the training, at least 60% of the 432 beneficiary households continue to care for their kitchen gardens. c) In the year after the training, at least 30% of the 432 beneficiary households use SLM methods (mulching, composting, collecting of rainwater). d) In the year after the training, 40% of the 432 target households continue to use at least one food preservation method. e) At the end of each training course cycle, at least 90% of the 432 women know about nutrition at infant age and in early childhood and 60% apply this knowledge. f) At the end of the project, the hygiene in at least 60% of the addressed 432 households has improved (handwashing with soap, water storage, use of toilets). 2
4- Communal facilities and local authorities can better manage and administer the natural resources together (stability). Target Indicators: a) At least 24 functioning NRMCs, at least 30 teachers and students (male and female) of the Institute for Agriculture and Livestock, as well as ten members of the DAIL and DRRD have improved capacities and can better manage and administer the natural resources. b) By the project end, 24 NRMCs have administrative plans and processes (including financial plans/processes), in order to improve the usage and ownership structures of restored natural resources. c) By the end of the first year of the implementation, 24 villages create an inventory of their natural resources and multiple-year natural resources plans. d) By the end of the first year of the implementation, 24 villages create an inventory of their natural resources and multiple-year natural resources plans. e) By the project end, 24 villages produce management plans (including financial plans, if necessary), in order to preserve the restored natural goods. f) By the project end, 24 villages, civil society organisations and local authorities understand the community-based NRM and their roles in regard to the NRM. g) 24 NRMCs and CDCs present their management plans to the responsible ministries, which link the NRM projects with management structures at a higher level. h) Ten members of the DAIL and DRRD as well as at least 30 teachers and students (male and female) of the Institute for Agriculture and Livestock Breeding have participated together in four (theoretical and practical) training course meetings about the handling of natural resources and the knowledge increase, such as the linking of science and management, leads to a better understanding and appropriate national planning. Project Partner: The activities under result2 (livestock management) are being implemented in partnership with Partners Revitalization and Building (PRB) in three districts of Jawzjan province, and Welthungerhilfe is responsible for implementation of three other results which are being implemented in the same mentioned area. Objective of the Mid-Term Evaluation: The project holder would like to conduct a mid-term evaluation of the full project. The results will be used by the project holder to improve the work during the remaining half of the project and, pending on observations and recommendations, possibly also guide on whether replanning or planning adjustments have to be made for the remaining time of the project. The evaluation results will be shared with the donor. 2 Task description The main tasks for the evaluation team are drafted below. The suggested structure of the report is attached as an annex. 2.1 Questions of specific interest for Welthungerhilfe and its partners The objective of this Mid-Term evaluation is to provide a comprehensive assessment of the project and an opportunity to critically assess administrative and technical strategies and issues. The evaluation gives recommendations to improve the potential of the Project to achieve expected outcomes and objectives within the project timeframe. 3
The evaluation aims to determine: 1. To analyse the design of the project; and evaluate the effectiveness and efficiency of the projects implementation by WHH and partner (PRB) within the agreed contractual conditions, time- frame, work-plan, log frame, and budget. 2. To analyse the level of coordination and cooperation between WHH, and its partner (PRB) and find out the gaps, shortcoming, weaknesses and strength points. 3. To analyze that how the intervention improved food security of target groups in the targeted area so far. 4. To identify and record lesson learn, best practices, strengths and provide recommendations for improvement. 2.2 Criteria for the assessment of the project/ Questions of special interest For assessment of the project, the evaluation team uses the OECD/DAC criteria of relevance, effectiveness, efficiency, impact and sustainability (see annex 4). In addition to the OECD/DAC criteria, the evaluator or team should analyse any other pertinent issues that need addressing or which may or should influence future project direction. Based on the observations and findings, the evaluator should come up with clear recommendations for the remaining time of the project. Relevance: To what extent is the project suited to: o Is the Project addressing the right target group in the targeted area and likewise, is the project meet the priorities and needs of target group in the intervention area? o Are the Welthungerhilfe relevant standards, procedures and donor priorities considered? o Are the activities and outputs of the project consistent with the project primary goal and the attainment of the objectives? o Are the activities and output of the project consistent with the intended impacts and effects? Effectiveness: To what extent has the project achieved it s below stated purpose? o Farmers have improved access to irrigation water and enhanced their skills in agricultural production techniques, to what extent the intervention increased/will be increased their production and productivity and improved their livelihood. o Livestock holders and women have improved their skills through livestock management and kitchen gardening, to what extent the skills applied/will be applied in their daily life/livelihood. o To what extent the project helped/will help the target groups food & nutrition security and their livelihood extension. o Is the project improved the natural resource management in the area of intervention, to what extent the community NRM plans coordinated with line departments, and to what extent the community NRM plans integrated into provincial annual plan. o Are the project results (outputs) achieved/will be achieved in term of quantity and quality, and to what extent the project outcome achieved/will be achieved? o To what extend the accountability and transparency considered in the project. o What were the major factors influencing the achievement or non-achievement of the objectives? Efficiency: o o o How cost efficient were the project activities? Were the project objectives detailed above achieved/will be achieved on time? Was the project implemented in the most efficient way? 4
Impact: What are the positive and negative changes produced by the project, directly or indirectly, intended or unintended? o o What outcome (positive, negative) can be observed and what impacts expected? What recommendations for the future programme can be made on the basis of the lessons learnt and good practices in all part of the project, particularly on strengthening partnership? Sustainability: Assessment on whether the project will be environmentally as well as financially sustainable? Whether the benefits of the project activities are likely to continue after donor funding has been withdrawn. o o To what extent are the benefits of the project expected to continue once the donor funding has ceased? What are the major factors, which have so far influenced the achievement or non-achievement of sustainability of the programme or project 2.3 Description of context and project With regard to questions and criteria the evaluation team describes the following: context, status quo of the relevant professional discourse, important actors, the project, experiences of similar projects on the country or international level. This chapter provides information for the analysis in the chapter relevance. 2.4 Evaluation design and methodology As part of the assignment, the successful evaluator will apply a mixture of different methodologies, including primary data collection and review of existing resources: a) A desk review of all project-related documents, reports (narrative, financial and monitoring reports), review of proposal and budget and contract with donor and between WHH and its two implementing partners, b) Field data collection on a sample of outcomes (selection to be discussed by the evaluator with the project team) to verify data collected to date, c) Structured interview with direct & indirect beneficiaries with consideration of sufficient sample size. (Quantitative approach), d) Focus group discussions with beneficiaries including girls, boys, men and women, parents, and teachers, e) Key informants interview with community representative, relevant government officials and key staff. (Quantitative & Qualitative approach). As part of the assignment, the evaluator will provide a detailed planning on proposed methodologies. WHH will review the planned methodologies proposed by the evaluator and provide feedback before the evaluation process begins. We would also like the evaluator to suggest any particularly innovative or interesting methodologies they may employ. A critical assessment and discussion of the project s strengths and challenges is welcome. Welthungerhilfe is ready to learn from mistakes. The study design should follow international standards concerning accuracy and reliability of data. 2.5 Reporting and documentation a) Minutes of the kick-off meeting (a lighter form of an inception report), documenting decisions, specifications, agreed dates for field visits, reporting etc. emerging from the meeting. b) Debriefing paper, at the end of the field mission that outlines the most important findings of the evaluation, 1-2 pages 5
c) Evaluation report, at the latest four weeks after the field mission, 30-45 pages maximum, plus annexes. d) Draft management response; integration of the recommendations into Welthungerhilfe s management response form, together with the final version of the evaluation report. Details of the reports and documentations are included as annex 1. 3 Timetable and phases of the evaluation Evaluation to begin in first half of August 2017 (time-frame to be discussed with evaluator), and work in the field to be undertaken around an approximately 28 days timeline with the final report to be submitted by September 15, 2017 at the latest. 4 Profile of the evaluation team The evaluator should meet the following criteria: a) An advanced degree or equivalent experience in Agriculture, Natural resource management and/or relevant academic field. b) At least 5 years of experience working on and evaluating livelihood, food & nutrition security, and natural resource management projects. c) Knowledge of livelihood, Food security/natural resource management sectors in Afghanistan. d) Knowledge of partnership models and assessing their effectiveness, especially in a North-South relationship e) Clear, effective writer in English f) Experience of working with participatory methodologies 5 Quality Check of the evaluation report Welthungerhilfe will check the quality of the evaluation report. The applied grid is documented as an annex. 6 Proposal content, Deadline and Place of Submission: Consulting firms or individual freelancing evaluator that fulfill the requirement shall submit as PDF: a) Application letter, signed by lead applicant; b) Financial and technical proposal with schedule; c) Detailed CV of the consulting team; d) References Please note that applications in other format than PDF and applications with an application letter lacking a signature will not be considered. These should be submitted to afg.kab.hr@welthungerhilfe.de by 5pm July 5, 2017. The technical proposal for the content of the evaluation must at least contain: a) Methodology: detail of the following should be presented - Type of study design - Sampling technique - Data collection technique - Methods of data compilation, analyzing and interpreting b) Understanding of the TOR c) Organizational experience and capacity to undertake the assignment d) The physical input (e.g. computer, printer, fax, telephone and internet access) which would be used by the team of consultant 6
e) Work schedule f) Composition of the whole evaluation team including data collectors and supervisors Financial part The offer should contain the costs of personnel (lead and assistant consultant, enumerators, supervisors, etc) and inputs (stationary, photocopying, etc). Costs for unforeseen expenses should not be included in the calculation. When preparing budget, the following to be considered; Soft and hard copies of relevant documents will be provided by WHH Transportation will be provided by WHH WHH staff will facilitate community entry Laptop to be provided by the evaluator Preferred budget structure: Item Total Cost (AFN / USD) Evaluator s Fee: XX days @ XX AFN / USD Total = XXXX USD/AFN Other costs (please specify) Total 7 Proposal evaluation The evaluation will be done in two stages. During the first stage, the technical part of the proposal will be evaluated and rated. The technical part and the financial part will get 60% and 40% of the evaluation. 8 Termination If the Client finds that the CONSULTANT is not discharging its duties according to this terms of reference; it may at any time unilaterally terminate the contract and holds the consultants liable for all damages, financial and otherwise including advance payments. 9 Reimbursement and support for proposal preparation: The cost of preparation of the financial and technical proposals is neither reimbursable nor can be considered as a direct cost of the project. Annexes Annex 1: Structure of reports and documents Annex 2: Management Response to Project Evaluation Annex 3: Quality check of the evaluation report (Welthungerhilfe) Annex 4: Information about DAC Criteria 7
Annex 1: Final Report Outline Table of contents List of abbreviations and acronyms Lists of tables, illustrations, boxes, etc. Acknowledgement and disclaimer I. Summary 1 Brief description of the project and framework conditions 2 Relevance 3 Effectiveness 4 Efficiency 5 Outcomes and impacts 6 Sustainability 7 Most important recommendations 8 General conclusions and lessons learnt II. Main text 0 Prior remarks 0.1 Reason and aim of the evaluation 0.2 Evaluation process 0.3 Evaluation methods 1 Description of the project 1.1 Brief project description 1.2 Framework conditions 1.3 Brief description of the target group 2 Relevance 2.1 Relevance to core problems of the target group 2.2 Relevance to the objectives of Welthungerhilfe 2.3 Relevance to the objectives of the partner organisation and the partner country 3 Effectiveness 3.1 Achievement of project purpose 3.2. Project preparation 3.2.1 Quality of target group, actor and situational analyses 3.2.2 Extent self-help approaches and structures are taken into account 3.3 Project planning 3.3.1 Quality of the project planning matrix 3.3.2 Analysis of the results chain 3.3.3 Realism of project purpose 3.3.4 Appropriateness of staff, material and financial planning 3.4 Quality of the analysis of project holder (partner(s) or external structure of the Welthungerhilfe) 3.4.1 Organisational structure and focal points of work 3.4.2 Staff, equipment and logistics 3.4.3 Project steering and financial administration 3.5 Steering by the head office, regional office and cooperation with the project holder 3.6 Quality of the project execution 3.6.1 Presentation and evaluation of the project outputs and activities 3.6.2 Involvement of the target group and project cooperations 8
3.6.3 Overall evaluation of the execution 3.6.4 Occurrence of assumptions and risks 4 Efficiency 4.1 Observation on cost/benefit ratio for the overall project 4.2 Observation on cost/performance ratio for the individual measures (if possible) 4.3 Quality of the internal project M&E system 4.4 Other aspects 5. Direct and indirect, short, medium and long-term outcomes and impacts 5.1 Use of Outputs / Outcomes 5.1.1 Economic outcomes 5.1.2 Socio-cultural outcomes 5.1.3 Organisational and institutional/political outcomes 5.1.4 Environmental outcomes 5.2. Impacts 5.2.1 Economic impacts (if possible) 5.2.2 Socio-cultural impacts 5.2.3 Organisational and institutional/political impacts 5.2.4 Environmental impacts (if possible) 6 Sustainability 6.1 Economic-financial (if possible) 6.2 Socio-cultural 6.3 Organisational and institutional/political 6.4 Environmental (if possible) 6.5 Other aspects 7 Conclusions and recommendations 7.1 Project specific conclusions 7.2 Recommendations 8 General conclusions 8.1 Important lessons learnt 8.2 Positive examples from the project (good practice) Annexes Terms of reference Project planning matrix Travel and work schedule References,sources (resource persons, documents, surveys, etc.) Case Studies, special narrative, Minutes of the final on-site meeting, de-briefing, de-briefing workshop Maps Pictures, photographs and comments, explanations Others. 9
Annex 2 Management Response to Project Evaluation Project Name: Time evaluation was conducted: Response form filled in by: Project Number: Evaluator: Date of response: Evaluator: Please insert recommendations mentioned in the report into below section with green headings and if appropriate, suggest a timeframe. Country/ Project Office: This document serves as a working document for you to follow up on recommendations. Please fill in the fields with column headings in grey and send back a copy to MELA unit at headquarter 3-6 months after the evaluation. No Recommendation Directed To Partne r/ Organi sation WHH proje ct offic e WHH country office/ HQ Bonn Priorit y/ Timefr ame Recommend ation already implemente d? How? When was/ will the recommendation be implemented? Responsible person/ unit? The recommendatio n will not be implemented, because [Reference to chapter/ topic/ DAC criterion] [Reference to chapter/ topic/ DAC criterion] 10
Accuracy Utility Project information Annex 3: Quality Check of the evaluation report (Welthungerhilfe) Assessment by, date: No Criteria Categories Information / Assessment 1 Project title 2 Project No. 3 Co-financer 4 Project holder / partners 5 Local consultant 6 Cost (review and project) 7 ToR respected? a) fully b) mostly c) less than 50% 8 Quality of reporting (principal is clear; executive summary provides clear insides of the results, language is good etc.) 9 Methods of data collection / evaluation methods Quality of proceeding 10 (proceeding is transparent and comprehensible; facts, analysis, conclusions and recommendations are distinguished) 11 Technical terms are used correctly (DAC criteria, results chain, output, outcomes etc.) 12 Participation of target groups in the process of the review; 13 gender is considered in data collection and analysis deviations explained? a) very good/ good b) okay / acceptable c) rather poor a) review of documents b) interviews c) questionnaire d) group discussions e) other: Things that are countable are counted; things which are measurable are measured 1 : a) very good/ good b) okay / acceptable c) rather poor a) very good/ good b) okay / acceptable c) rather poor a) very good/ good b) okay / acceptable c) rather poor a) rather high b) okay / normal c) rather low a) rather high b) okay /normal c) rather low 1 For important aspects the consultant provides details in the sense of: 2 out of 10 partner organizations (interviewed) stressed that they want to focus on irrigation in 2015. Instead of: Some partner organizations stressed that 11
Fairness/ Utility 14 Important issues or/and strength & weaknesses are discussed. a) rather high b) okay / normal c) rather low 15 Conclusion: Quality of the report 16 Further remarks; interesting issues. a) very good/ good b) okay / acceptable c) rather poor 12
Annex 4: Information about DAC Criteria Source, 12 June 2015: http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm ; DAC Criteria for Evaluating Development Assistance When evaluating programmes and projects it is useful to consider the following criteria. Relevance The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a programme or a project, it is useful to consider the following questions: To what extent are the objectives of the programme still valid? Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives? Are the activities and outputs of the programme consistent with the intended impacts and effects? Effectiveness A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a programme or a project, it is useful to consider the following questions: To what extent were the objectives achieved / are likely to be achieved? What were the major factors influencing the achievement or non-achievement of the objectives? Efficiency Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a programme or a project, it is useful to consider the following questions: Were activities cost-efficient? Were objectives achieved on time? Was the programme or project implemented in the most efficient way compared to alternatives? 13
Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a programme or a project, it is useful to consider the following questions: What has happened as a result of the programme or project? What real difference has the activity made to the beneficiaries? How many people have been affected? Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions: To what extent did the benefits of a programme or project continue after donor funding ceased? What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? 14