EXTERNAL END OF PROJECT PERFORMANCE EVALUATION

Transcription

1 EXTERNAL END OF PROJECT PERFORMANCE EVALUATION 1. Background TERMS OF REFERENCE PT. Rainforest Alliance (RA) is leading a consortium group in implementing the Cocoa Revolution: High-Yielding Climate Smart Cocoa Farms project to contribute to promote sustainable cocoa production and improve smallholder livelihoods in Indonesia. More specifically, the project aims to contribute to poverty reduction and climate adaptation of 8,000 cocoa smallholders in North Kolaka and North Luwu districts in Sulawesi, by training and incentivizing them to apply on-farm techniques that will: 1) optimize sustainable yields; 2) improve quality; 3) provide access to national and global longterm marketing channels; and 4) introduce state-of-the-art climate-smart agriculture. In so doing, the project will also have contributed to mitigate climate change. The project started in July 2015, and will finalize field activities and deliverables on December The project is funded by matching funds between the Millennium Challenge Account Indonesia (MCA-Indonesia) and a consortium led by PT Rainforest Alliance in partnership with PT Olam Indonesia, GrowCocoa, the Indonesia Coffee and Cocoa Research Institute (ICCRI), local government, and cocoa smallholders, under the Millennium Challenge Account-Indonesia: Green Prosperity Project Partnership Agreement No. signed between MCA-Indonesia and PT. RA (Agreement). The Agreement establishes that an External End of Project Performance Evaluation (EEPPE) be carried out.based on MCA-I s post-agreement directives, all project activities are to end by December 31 st, meaning the EEPPE will be undertaken in quarter 10, starting October 2nd, 2017, and a final report delivered by December 21 st, The Agreement also establishes that carrying out the EEPPE is the responsibility of PT. RA, and is to be financed by the project s Monitoring and Evaluation (M&E) budget. EEPPE findings and recommendations will be reviewed and endorsed by the project s Steering Committee (SC). 2. Objectives The objectives of the EEPPE are to assess the project s progress and achievements against the Agreement and Implementation, identify any performance shortfalls, evaluate the project s efficiency, effectiveness, relevance and sustainability, and derive lessons learned as basic reference for future projects. The EEPPE is a participatory process through which the project beneficiaries, partners, stakeholders, MCA-Indonesia, the Project Manager and other PT. RA staff are consulted and also learn from experience to-date and develop consensus on how to improve design and management of projects in the future. The EEPPE will assess: 1. The project s relevance to the goals of MCA-Indonesia s Green Prosperity Program

2 2. The adequacy of project approaches towards stakeholder groups, project partners and beneficiaries to ensure good project performance and project success; 3. The project s effectiveness in terms of its capacity to complete (planned) activities and deliverables, in terms of quantity, quality, usefulness and timeliness; 4. The project s effectiveness in terms of its capacity to achieve promised objectives and outcomes; 5. The positive and negative (long term) changes produced by the project, directly or indirectly, intended or unintended, considering institutional, socio-economic and environmental dimensions. 6. The project s efficiency in terms of its capacity to cost-effectively use available human and financial resources to complete project activities and deliverables, and achieve outcomes; 7. The effectiveness of project technical and financial planning, management and oversight; 8. The effectiveness of the project s governance arrangements; 9. The effectiveness of project partners in implementing their commitments under the project; 10. The quality and timeliness of project PT. RA s technical and financial reporting. 11. The quality and timeliness of project partners technical and financial reporting. 12. The adequacy of project design, approach and management to ensure sustainability and replicability of project outputs and outcomes; 13. The capacity of the project to play a catalytic role, promote local ownership and own initiatives, and promote stakeholder involvement; 14. The appropriateness of the project s M&E system in facilitating quality and timely elements to support project management and reporting; 15. The quality of PT. RA s global staff backstopping of project implementation and reporting: 16. The quality of MCA-Indonesia s supervision and backstopping. Specific questions to be analyzed 1 and responded by the EEPPE regarding the project s key technical components will include, but will not be limited to, the following: 1. How effective were trainings provided by the project in terms of their contribution to practice adoption? 2. Did farmers consider trainings relevant to address the cocoa production, quality and marketing issues faced by them? 3. Did women farmers consider that they were given the opportunity to participate in trainings, and did they express that their skills and knowledge were improved? 4. What was the level of women s participation in trainings, and was this level representative of the female farmer population? 5. What was the level of poor farmers (those with farms < 0.5 hectares) participation in trainings, and was this level representative of poor farmer population? 6. How competent were trainers of lead farmers in delivering trainings to farmer trainers, including to lead farmers? 7. How competent were trainers of farmers, including lead farmers, in delivering trainings to farmers? 8. How effective were demoplots as teaching tools to promote farmers practice adoption 9. How effective was the project s mentoring program in promoting practice adoption? 10. How effective were solar dryers in promoting improvements in cocoa quality? 1 Analysis will include the why s and why nots, as well as a recommendation of what should have been done differently.

3 11. Were nursery arrangements appropriate as sources of quality shade-tree and cocoa seedlings to farmers, and how effective were they in promoting shade-tree planting and cocoa tree rejuvenation by farmers? 12. How effective was the project in supporting beneficiaries through the incentives program? 13. How effective were project incentives in promoting farmer s practice adoption, improved cocoa quality, and sales through GrowCocoa? 3. Person responsible for the review RA s Director of the M&E/Quality Assurance Unit will coordinate and oversee the EEPPE process, including selecting and the hiring of the Evaluator, and the review of the EEPPE deliverables. 4. Methods The project will be evaluated through: Desk review of project the project s Agreement, the Project Document, M&E tracking tools and supporting documentation (MoVs); Steering Committee meeting minutes, quarterly and annual progress reports, MCA-Indonesia documentation, PT. Olam and PT. GrowCocoa reports and deliverables, sub-grant agreements, and the Project Document; Expenditure reports against expected cash flow, including on co-finance delivery; Review of public communication activities, publications, media support, websites, etc.; Field visits to a representative sample of 6 of the project sites; Face-to-face and /phone interviews of selected project implementers, stakeholders, partners, and beneficiaries, including but not limited to: o PT. Olam and PT. GrowCocoa managers and field staff in both project sites 2 o PT. RA project manager and staff based in Makassar and in both project sites o PT. RA project financial manager o MCA-Indonesia managers supervising the project o Partners providing Trainers of Farmers o Individual Trainers of Farmers o Beneficiary farmers o Other stakeholders, including government institutions, community-based organizations and individuals involved in the project The EEPPE will be summarized in the Consolidated Report Template, provided in Annex I. 5. Field Visits The Evaluator will visit a minimum of 20 project sites; 10 in North Luwu, and 10 in North Kolaka. A PT. RA field staff will accompany the Evaluator to all visited project sites. It is recommended to schedule the visits in combination with stakeholder interviews, to the degree possible, in order to make the most efficient use of the Evaluator s and field staff s time. At each site there should be an opportunity to meet with the project beneficiaries responsible for implementing/managing the site, and to view ongoing activities. 2 North Luwu and North Kolaka

4 A detailed programme of field visits will be prepared by PT. RA, jointly with PT. Olam and the Evaluator. 6. Reporting The Evaluator will generate all reports in English, will be of no more than 50 pages (excluding annexes), will use numbered paragraphs, and will be clear and concise. They will present evidence-based findings, quantified ratings, conclusions, and lessons learned. All reports will include: i) Table of contents ii) List of acronyms iii) Executive summary iv) Introduction and background: A brief overview of the project, and scope and methodology applied for the EEPPE. v) Analysis: This section should include technical, financial and institutional aspects. vi) Preliminary conclusions (in the Progress Report) or final conclusions and ratings (in the Final Report) of project implementation success giving the evaluator s concluding assessments and ratings of the project against given evaluation criteria and standards of performance. The conclusions should provide answers to items covered in Annex I; vii) Sustainability and replicability (only in the Final Report) : A brief account on what has been found and what should be improved with regard to these aspects, as well as key requirements to sustain the project: What is required to sustain or enhance project approaches and methods, activities and workplans, as well as implementation arrangements? viii) Lessons learned and best practices ix) Annexes, including ToR, detailed methodology and survey/interview instruments, databases, list of interviewees, itinerary, etc. 7. Deliverables The Evaluator will produce and deliver the following products, according to the schedule presented in section 10, below: 1. A detailed program of field visits and stakeholder interviews 2. An EEPPE Progress Report 3. A 1st draft of EEPPE Final Report 4. A revised 1 st draft of EEPPE Final Report 5. A PowerPoint summarizing EEPPE results 6. Presentatin and discussion of EEPPE results at end-of-project workshop 7. 2 nd draft of EEPPE Final Report, including end-of-projectworkshop proceedings 8. Final (revised) EEPPE Final Report, including end-of-projectworkshop proceedings 8. Place, Level of Effort and Duration of Consultancy The Evaluator s level of effort; that is the effective time that he/she shal dedicate to this consultancy will be 63 work days, including the time that he/she will dedicate to

5 present the EEPPE results at the end-of-project workshop to take place the first week of December The Evaluator shall deliver the EEPE s Final Report, including end-of-project workshop proceeding by December 21st, Budget and Conditions for Payment (negotiable) The Evaluator will receive a lump sum remuneration for satisfactory delivery of the service, according to the schedule in section 10, below, amounting to no more than USD42,000. This remuneration covers the following concepts, and will be paid against approval by PT. RA of deliverables specified in these ToR, and signed receipts. a) An honorarium for at most USD37,500, corresponding to a daily rate of USD595 for 63 work days during the period spanning October 2 nd, 2017 to December 21 st, 2017; and b) Non-reportable expenses expected to be incurred by the Evaluator, estimated at US$4,500; 10. Summary Milestones and Payment Schedule for the EEPPE Milestones Who Finalized by: Contract signed and EEPPE initiated. Evaluator October 2nd, 2017 Detailed program of field visits and stakeholder PT. RA, jointly October 6 th, 2017 interviews 1 st payment (20% of total budget). with PT. Olam and Evaluator Field visits Evaluator October 9-20, 2017 Progress Report delivered for review 2 nd Evaluator October 23, 2017 payment (10% of total budget). Comments to Progress Report. PT. RA October 25 th, st draft of EEPPE Final Report delivered for Evaluator October 31 st, 2017 review Comments to 1 st draft of EEPPE Final Report Revised draft of EEPPE Final Report delivered. Approval of Revised 1 st draft of EEPPE Final Report. 3 rd payment (10% of total budget). PowerPoint summarizing EEPPE results presented, and discussed at end-of-project workshop EEPPE results presented, and discussed at end-of-project workshop 2 nd draft of EEPPE Final Report, including workshop proceedings, delivered Comments to 2 nd EEPPE Final Report, including workshop proceedings PT. RA and Steering Committee Evaluator PT. RA and Steering Committee November 6th, 2017 November 15th, 2017 November 30 th, 2017 Evaluator December 1st Evaluator Week of Dec. 4 th, 2017 Evaluator December 11th, 2017 PT. RA and December 13th, Steering 2017 Committee Final (revised) EEPPE Report, delivered Evaluator December 15th, 2017

6 Approval of Revised EEPPE Final Report, including workshop proceedings 4rth and last payment (60% of total budget). PT. RA and Steering Committee-approved EEPPE Final Report commented on by MCA-I EEPPE Final report + Addendum with MCA- Indonesia comments published PT. RA and Steering Committee MCA-Indonesia December 21st, 2017 December 31st PT. RA January 31 st Supervision, and Administrative and Logistical Support The Evaluator will be supervised by RA s Director of Monitoring, Evaluation and Quality Assurance. PT. RA and PT. Olam will provide the following administrative and logistical support to the Evaluator: Official letter of introduction vis-a-vis farmers and other stakeholders Full cooperation regarding staff availability for the discussion of project implementation and progress Unrestricted access to all documents, outputs and other project deliverables as required and as available Assistance in contacting farmers and stakeholders Simultaneous translation vis-a-vis farmers and stakeholders that do not speak Bahasa Arrangements for site visit(s), and associated accommodation and local transportation 12. Required Documentation, and Qualifications The aspirant shall send his/her curriculum vitae and a letter of interest in English, emphasizing his/her development project design and evaluation experience over the last 5 years, indicating countries where this work has been done. The Evaluator shall as well provide proof of his/her English and Bahasa literacy levels. Required and preferred qualification: Required, university degree in environment, natural resource management, agriculture, environmental/agricultural economics, or related area. A Master s degree is highly preferred. Required, at least 2 development project evaluations successfully completed in developing countries, preferably having played a leading role in the evaluation team. Preferred, development project design and evaluation experience in Asia. Preferred development project experience in Indonesia Preferred, experience in development project design and evaluation, involving international donors. Required, proven full command of Bahasa, and excellent writing and oral English skills.

7 Aspirant evaluation criteria: Criteria Ranges Points 1. Academic degrees in desirable knowledge areas Bachelors 10 Masters 15 2 projects Development project evaluations successfully 3-6 projects 20 completed > 6 projects Experience in project design and evaluation in Asia 4. Work experience in Indonesia 5. Experience in development project design and evaluation involving internatinal donors 1 project 15 2 projects 20 > 2 projects 25 1 year 10 2 years 15 > 2 years 20 No 0 Yes 10 Total 100

8 Annex I Review Criteria & Ratings (full details can be provided separately) Criterion A) Attainment of project objectives and results (overall rating) Sub criteria (below) Effectiveness Relevance Efficiency B) Sustainability of Project outcomes (overall rating) Sub criteria (below) Financial Socio Political Institutional framework and governance Ecological Achievement of outputs and activities C) Monitoring and Evaluation (overall rating) Sub criteria (below) M&E Design M&E Plan Implementation (use for adaptive management) Budgeting and Funding for M&E activities Catalytic Role Country ownership / driveness Stakeholders involvement Financial planning MCA-I s supervision and backstopping Overall Rating Reviewer s Summary Comments Rating A) RATING OF PROJECT OBJECTIVES AND RESULTS Highly Satisfactory (HS): The project had no shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency. Satisfactory (S): The project had minor shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency. Moderately Satisfactory (MS): The project had moderate shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

9 Moderately Unsatisfactory (MU): The project had significant shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency. Unsatisfactory (U) The project had major shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency. Highly Unsatisfactory (HU): The project had severe shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency. Please note: Relevance and effectiveness will be considered as critical criteria. The overall rating of the project for achievement of objectives and results may not be higher than the lowest rating on either of these two criteria. Thus, to have an overall satisfactory rating for outcomes a project must have at least satisfactory ratings on both relevance and effectiveness. B) RATINGS ON SUSTAINABILITY Sustainability will be understood as the probability of continued long-term outcomes and impacts after the project funding ends. The Terminal evaluation will identify and assess the key conditions or factors that are likely to contribute or undermine the persistence of benefits after the project ends. Some of these factors might be outcomes of the project, i.e. stronger institutional capacities, legal frameworks, socioeconomic incentives /or public awareness. Other factors will include contextual circumstances or developments that are not outcomes of the project but that are relevant to the sustainability of outcomes. Rating system for sustainability sub-criteria On each of the dimensions of sustainability of the project outcomes will be rated as follows. Likely (L): There are no risks affecting this dimension of sustainability. Moderately Likely (ML). There are moderate risks that affect this dimension of sustainability. Moderately Unlikely (MU): There are significant risks that affect this dimension of sustainability Unlikely (U): There are severe risks that affect this dimension of sustainability. All the risk dimensions of sustainability are critical. Therefore, overall rating for sustainability will not be higher than the rating of the dimension with lowest ratings. For example, if a project has an Unlikely rating in either of the dimensions then its overall rating cannot be higher than Unlikely, regardless of whether higher ratings in other dimensions of sustainability produce a higher average. C) RATINGS OF PROJECT M&E Monitoring is a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing project with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Evaluation is the systematic and objective assessment of an on-going or completed project, its design, implementation and results. Project evaluation may involve the definition of appropriate standards, the examination of performance against those standards, and an assessment of actual and expected results.

10 The Project monitoring and evaluation system will be rated on M&E Design, M&E Plan Implementation and Budgeting and Funding for M&E activities as follows: Highly Satisfactory (HS): There were no shortcomings in the project M&E system. Satisfactory(S): There were minor shortcomings in the project M&E system. Moderately Satisfactory (MS): There were moderate shortcomings in the project M&E system. Moderately Unsatisfactory (MU): There were significant shortcomings in the project M&E system. Unsatisfactory (U): There were major shortcomings in the project M&E system. Highly Unsatisfactory (HU): The Project had no M&E system. M&E plan implementation will be considered a critical parameter for the overall assessment of the M&E system. The overall rating for the M&E systems will not be higher than the rating on M&E plan implementation. ALL OTHER RATINGS, e.g. catalytic role etc., will be on the following six point scale. Performance Description HS = Highly Satisfactory Excellent Alternative description on the same scale S = Satisfactory Well above average MS = Moderately Satisfactory Average MU = Moderately Unsatisfactory Below Average U = Unsatisfactory Poor HU = Highly Unsatisfactory Very poor