Mid-Term Review of UNDP/GEF project: Strengthening the Operational and Financial Sustainability of the National Protected Area System

Transcription

1 Mid-Term Review of UNDP/GEF project: Strengthening the Operational and Financial Sustainability of the National Protected Area System Evaluation conducted by Alexandra Fischer GEFSEC Project ID: PIMS 3832 Agency s Project ID: Evaluation time frame- June July 2013 Date of Evaluation Report: July 12, 2013 (first draft); August 23, 2013 (final draft) Country: Jamaica GEF Operational Program: SP1 Strategic Priority: PA Executing Agency: National Environment and Planning Agency Acknowledgements: The evaluation consultant would like to thank the UNDP Jamaica Country Office and the Government of Jamaica for the support provided throughout the realization of this Mid-Term Review, as well as the many stakeholders who provided invaluable feedback on the project. i

2 Table of Contents Contents 1 Executive Summary... iv 2 List of Acronyms... x 3 Introduction Purpose of the Evaluation Key Issues Addressed Methodology of the Evaluation Structure of the Evaluation Project Description and Development Context Project Start and Duration Development Context and Problems that the Project Seeks to Address Immediate and Development Objectives of the Project Baseline Indicators established Expected Results Main Stakeholders Findings Project Design/ Formulation Project Implementation Project Results Conclusions Best practices LESSONS LEARNED MID-TERM REVIEW RECOMMENDATIONS ii

3 List of Tables Table 1: Project ratings xi Table 2: Summary of Expenditures by Outcome and Year 18 Table 3: Co-Financing Table 20 Table 4: Level of Progress on Project Indicators 24 iii

4 1 Executive Summary Project Summary Table: Project Title: Strengthening the Operational and Financial Sustainability of the National Protected Area System GEF Project ID: 3832 at endorsement (Million US$) UNDP Project ID: GEF financing: 2.77 Country: Jamaica IA/EA own: 0.20 Region: LAC Government: 0.50 Focal Area: PA Other: 4.35 FA Objectives, (OP/SP): Executing Agency: Other Partners involved: SP1 Total co-financing: 5.05 National Environment and Planning Agency Jamaica National Heritage Trust Forestry Department Fisheries Division Total Project Cost: 7.82 at completion (Million US$) ProDoc Signature (date project began): July 8, 2010 (Operational) Closing Date: Proposed: June 2016 Actual: June 2016 Overview of objective and methodology for Mid-Term Review This Mid-Term Review was undertaken between June and July 2013 and adhered fully to the UNDP/GEF guidelines and Terms of Reference for this consultancy. Key issues addressed were project relevance, effectiveness, efficiency, sustainability and impact. The methodology included a detailed review of all relevant project documentation; a 1.5 week mission involving interviews with a total of 37 stakeholders and a presentation of initial findings to the Project Steering Committee; follow-up phone interviews and correspondence; careful analysis of the findings; and preparation of the draft and final reports. Brief project description Jamaica's high levels of biodiversity are faced with a number of direct and indirect threats. The long-term solution is a consolidated National Protected Areas System (NPAS) supported by a strong institutional framework with the legislative mandate, management capacity and financial support needed for effective for protected area conservation. The project's objective of consolidating the operational and financial sustainability of Jamaica s national system of protected areas will be achieved through three components: (1) Strengthening of planning and revenue generation; (2) Rationalizing and integrating the national system of protected areas; and, (3) Increasing the effectiveness of protected area management The Project's Executing Agency is the National Environment and Planning Agency (NEPA). Other key stakeholders include the three other agencies charged with PA management, namely, Forestry Department, Fisheries Division and the Jamaica National Heritage Trust, TNC, the Protected Areas Committee, Ministry of Finance and Planning, Ministry of Water, Land, Environment and Climate Change, Planning Institute of Jamaica, and co-managing organizations, among others. iv

5 MAIN FINDINGS Project Execution The project execution rating is moderately unsatisfactory, owing to the slow project start-up, substantial delays in project execution, low delivery, and insufficient provision of managerial and technical support by NEPA to the project. Nevertheless, with renewed government commitment, coupled with the planned hiring of a Technical Adviser and other support to strengthen the Project Management Unit (PMU), execution can improve significantly. The recent steps taken by the EA to facilitate the provision of greater support from within NEPA and to plan for increased delivery are positive indications of change in the right direction. Procurement delays have been extensive, with the first consultants not coming on board until two years into the project in December 2012, and several key consultants still to be contracted. This is attributed to the slow procurement process in place, and limited national capacity leading to the frequent need to readvertise positions. Another factor that negatively affected procurement was the extended period of time it has taken for the TORs to be developed, reviewed by Project Steering Committee (PSC) members, and approved. Overlaps in the TORs of various consultancies have required additional discussions and have led to delays in approval of workplans. The procurement difficulties have also meant that consultants who need to work together have been hired at different periods of time, reducing the efficiency of their work. In general, the coordination of the consultants' work and facilitation of communication among consultants needs to be strengthened. The current PMU, composed of a project manager, an administrative assistant and supported in a parttime capacity by a Project Director and Branch Manager, has been clearly stretched to take on the multiple tasks of managing a project as large and complex as this one and adhering to the reporting and decision-making processes of both UNDP and NEPA. The PMU would benefit significantly from additional technical and administrative support. The small size of the PMU coupled with the slow decision making processes by the PSC and NEPA have contributed to delays in project progress and finalization of project deliverables. Adaptive management has been carried out on various occasions throughout the project, though it needs to be more timely. Examples of adaptive management include the revision of the SRF, the dedication of some project funds to a Small Grants Initiative to fund PA-site actions, and the modification of the TORs for some consultancies. In response to the negative ratings in the 2012 PIR, the PSC size was reduced and its mandate clarified to speed up progress. Project Implementation The UNDP has performed its functions as Implementing Agency satisfactorily, with a few issues that could be strengthened. It has maintained regular communication with the PMU. Monitoring and evaluation activities have been carried out, including through active participation on the PSC, contribution to annual PIRs, monitoring of project activities, and support for realization of this Mid-Term Review. UNDP has also provided technical support by reviewing project deliverables and providing feedback, though the provision of feedback could be more consistent and timely. UNDP has contributed to ensuring accountability and transparency by providing advice on UNDP and GEF procurement guidelines and participating in interview panels, among other tasks. The GEF funds have been managed adequately, with no issues of concern raised. Some interviewees commented on a lack of clarity on the decision-making responsibilities of the PSC versus the UNDP. In addition, UNDP's position on certain issues has not always been clear to PSC members, underlining the need for strong communication to avoid misunderstandings. Decision-making has also been slow at times, such as with regard to the hiring of a technical adviser and development of appropriate TORs. v

6 UNDP has elevated issues of concern to the UNDP CO Senior Management, the RTA and Headquarters and recently requested a meeting with the CEOs and Permanent Secretaries of the four agencies charged with PA management, which had positive impacts. It is advised that UNDP meet with high-level government officials more regularly and proactively to promote high-level engagement and commitment to the project. Some steps to build partnerships and to facilitate information exchange have been taken but additional actions could be taken to promote greater linkages among countries in the region. Project Results and Sustainability Progress toward the overall objective of strengthening the operational and financial sustainability of Jamaica's National Protected Areas System remains limited. One of the key elements in the project design to promote financial sustainability is the establishment of a national trust fund, which would receive interest from an endowment for Jamaica from TNC, KfW and GEF and from additional matching funds from new national financial inflows. At this point, decisions on where to house the National Protected Areas Trust Fund (NPATF) and the nature of its Board have been made, negotiations on the vertical agreement between the NPATF board the CBF are ongoing, and preliminary recommendations on feasible financial mechanisms that could be established at a national level have been made. In addition, a draft business planning framework and one draft business plan have been developed. Progress to rationalize and integrate the NPAS has been extremely limited, with regard to both the development of umbrella PA legislation and the gazetting of two new PAs. The international legal experts were only hired recently and have been reviewing co-management and conservation easement issues and the local expert has still not been hired. The PA management specialists have not been hired yet either, with the result that no PA management plans have been developed with project funds at this point. A small amount of funds was invested in five PA-level projects with the goal of contributing to increased management effectiveness. To support this outcome, a public awareness campaign is underway. Progress toward many of the SRF targets is limited as detailed in the table provided in the report itself, although it should be noted that the targets related to annual government funding for PAs and the capital for the NPATF surpassed those established in the ProDoc. The following provides a summary of main lessons learned and recommended corrective actions. The reader is advised to refer to the final section for detailed information on each of the recommendations. Lessons learned- project design 1. Allow sufficient time for establishment of PMU and engagement of consultants 2. Ensure structure of PMU adequate for implementation 3. Assess national capacity for key consultancies during PPG phase and plan for international consultancies or government agencies to implement some aspects if necessary 4. Review PPG implementation to identify issues that could reoccur during project implementation that need to be addressed 5. Ensure that all project indicators and targets are realistic and appropriate to the national context 6. Define PAs for targeted activities during PPG phase 7. Check baseline calculations for accuracy (e.g., METT scores) 8. Review PPG implementation to identify issues that could reoccur during project implementation that need to be addressed Lessons learned- project implementation 9. If working groups are established to support project outcomes, clarify possible conflicts of interest from the outset vi

7 10. Allocate sufficient time to produce outputs requiring stakeholder consultations (e.g., business plans) 11. Inclusion of small amounts of funding for local activities in PAs can have positive impacts and build stakeholder support Lessons learned- monitoring and evaluation 12. Utilize the project inception phase more effectively 13. Ensure that advisory role of Project Steering Committee is clarified and that its size does not impede project progress 14. Provide and enforce time limit for provision of feedback by PSC 15. Succinct minutes of PSC meetings need to be distributed on a timely basis with clear actions items identified MID-TERM REVIEW RECOMMENDATIONS Corrective Actions: Project Design 1. Correct any erroneous baseline data Corrective Actions: Project Implementation 2. Strengthen PMU as soon as possible with hiring of Technical Adviser 3. PMU should not postpone decision making or delivery during the period of recruitment of a Technical Adviser 4. Procure remaining consultants as efficiently as possible, and if necessary provide training to NEPA Procurement Committee 5. Promote more timely review and decision-making processes and greater technical support from within the four government agencies responsible for PA management 6. Garner greater support from within NEPA at both the technical and higher levels 7. Clarify decision making responsibilities of PSC vs UNDP and increase efficiency of decisionmaking within PSC 8. It is recommended that UNDP state its positions clearly and carry out more rapid decisionmaking 9. NEPA should facilitate increased coordination among consultants and increase communication with consultants 10. Strengthen organization of missions of international consultants 11. Communicate with stakeholders about status of Protected Areas technical working group and follow up as planned 12. Facilitate greater regional communication and information exchange among UNDP COs 13. UNDP Regional Service Centre to consider strengthening its procurement assistance to projects in the region Corrective actions: Project results 14. Maintain focus on strategic deliverables 15. Garner high-level support and promote mainstreaming of PA issue on an ongoing basis to achieve strategic results vii

8 16. Strengthen communication between PSC, Financial Working Group and other relevant actors and clarify decision making regarding the trust fund and proposed financial mechanisms 17. Establish clear timelines for all required steps and carry out regular follow-up regarding NPATF and financial mechanisms 18. Move quickly to begin discussions and arrangements for financial inflows into trust fund 19. Identify preliminary criteria to decide on projects to fund through the National Protected Areas Trust Fund 20. Promote strengthening and institutionalization of PAC to act as a coordinating body for the NPAS 21. Designate additional project funds for first months of implementation of prioritized business plans 22. Ensure that national protected area system management planning framework and guidelines are established before the individual PA management plans are developed under this project 23. Development of management plans must be linked with the identification of sources of funding to implement them 24. Agencies will need to dedicate time and build capacity to increase METT scores 25. Public awareness strategy needs to be discussed among stakeholders and follow-up actions prioritized and agreed upon 26. Clarify means of dissemination of PA information through Clearinghouse Mechanism or NEPA Protected Areas database 27. Maintain targets for PAs included in original project design, rather than including new PAs that have since been incorporated into the system in the SRF viii

9 Table 1: Project Ratings Rating Project Performance Criteria Rating Monitoring and Evaluation (Scale: Highly Satisfactory (HS), Satisfactory (S), Moderately Satisfactory (MS), Moderately Unsatisfactory (MU), Unsatisfactory (U), Highly Unsatisfactory (HU) Overall quality of M&E Moderately Satisfactory M&E design at project start up Satisfactory M&E Plan Implementation Moderately Satisfactory IA and EA Execution (Scale from HS to HU); Overall quality of Project Implementation/Execution Moderately Satisfactory (MS) Implementing Agency Execution Satisfactory (S) Executing Agency Execution Moderately Unsatisfactory (MU) Outcomes (Scale from HS to HU); Overall Quality of Project Outcomes Moderately Unsatisfactory (MU) Relevance Relevant Effectiveness Moderately Unsatisfactory (MU) Efficiency Moderately Satisfactory (MS) Sustainability; Scale: Likely (L), Moderately Likely (ML), Moderately Unlikely (MU), Unlikely (U) Where Likely refers to negligible risks to sustainability and Unlikely refers to severe risks to sustainability. Overall likelihood of sustainability Moderately Unlikely Financial resources Moderately Likely Socio-economic Moderately Likely Institutional framework and governance Moderately Unlikely Environmental Likely Impact; Scale: Significant (S), Minimal (M), Negligible (N) Environmental Status Improvement Minimal (M) Environmental Stress Reduction Minimal (M) Progress toward stress/status change Minimal (M) Impact of Overall Project Results Minimal (M) ix

10 2 List of Acronyms APR/PIR BJCMNP CBD CBF CO EA EFJ FACE GEF IA JCDT KfW LAC M&E METT MTR NEPA NEX NGO NPAS NPATF NRCA PAC PIOJ PIR PMU ProDoc PSC RSC RTA SGP TNC TORs UNDAF UNDP UNEP USAID Annual Progress Report/ Project Implementation Report Blue and John Crow Mountains National Park Convention on Biological Diversity Caribbean Biodiversity Fund Country Office Executing Agency Environmental Foundation of Jamaica Fund Authorization Certificate of Expenditure Global Environment Facility Implementing Agency Jamaica Conservation and Development Trust German government-owned development bank Latin America and the Caribbean Monitoring and Evaluation Management Effectiveness Tracking Tool Mid-Term Review National Environment and Planning Agency National Execution Non-governmental organization National Protected Areas System National Protected Areas Trust Fund Natural Resources Conservation Authority Act Protected Areas Committee Planning Institute of Jamaica Project Implementation Review Project Management Unit Project Document Project Steering Committee Regional Service Centre Regional Technical Adviser UNDP Small Grants Program The Nature Conservancy Terms of Reference United Nations Development Assistance Framework United Nations Development Program United Nations Environment Program United States Agency for International Development x

11 3 Introduction 1. The report includes five main sections: the Introduction, which describes the purpose and methodology of the evaluation and key issues addressed; a summary of the project description; an analysis of the project design; the main findings related to project execution/ implementation; and finally, the conclusions with key lessons learned and recommendations. The recommendations are intended to serve as a practical tool to help improve project execution and increase the likelihood of significant project impact and sustainability. 3.1 Purpose of the Evaluation 1. This Mid-Term Review (MTR) is a requirement of the United Nations Development Program (UNDP) and Global Environment Facility (GEF) and was initiated by the UNDP Jamaica Country Office. It was conducted according to the guidance, rules and procedures for such evaluations established by UNDP and GEF. 2. The overall objective of the MTR is to analyze the implementation of the project and review the achievements of the project thus far to deliver the specified objective and outcomes. It establishes the relevance, performance and success of the project, and provides a preliminary indication of the presumed sustainability of results. The evaluation also brings together and analyzes best practices, specific lessons learned and recommendations pertaining to the strategies employed, implementation arrangements and project results, in order to help guide project implementation for the remainder of the project. This information may also be of relevance to other projects in the country and elsewhere in the world. The MTR provides a comprehensive and systematic account of the performance of the project by assessing its project design, process of implementation and results vis-à-vis the project objective and outcomes. The MTR has four main objectives: Identify potential problems in the project design; Evaluate the level of progress in the achievement of project objectives; Identify and document lessons learned; Provide recommendations on specific actions that should be carried out to improve the execution of the project. 3.2 Key Issues Addressed 3. This evaluation analyzed the following five main criteria: Relevance. The extent to which the activities are suited to local and national development priorities and organizational policies, taking into consideration changes over time. Effectiveness. The extent to which the results have been achieved or how likely they are to be achieved. Efficiency. The extent to which results have been delivered with the least costly resources possible; also called cost-effectiveness or efficacy. Sustainability. The likely ability of an intervention to continue to deliver benefits for an extended period of time after completion. Projects need to be financially, socially and environmentally sustainable. 1

12 Impact: Verifiable improvements in ecological status, verifiable reductions in stress on ecological systems, or indications that progress is being made towards achievement of stress reduction and/or ecological improvement (through process indicators). 3.3 Methodology of the Evaluation 4. The Mid-Term Review employed qualitative data collection and analysis methods and included the following components: A) Evaluation Preparation: The consultant carried out an extensive review of documentation, including the Project Document and all other relevant information. The list of documents studied is provided in Annex 6 of this report; An MTR Inception Teleconference was held with the UNDP CO to plan the mission, confirm the TORs and establish any specific requirements for the mission. An Inception Report was prepared with a detailed mission programme, including the evaluation methodology to be followed. B) Evaluation Mission: At the outset of the mission an inception meeting was held with the Environment and Energy Unit of the UNDP Jamaica Country Office to obtain contextual information about the project and UNDP's impressions of the main challenges being experienced. were held with 37 stakeholders involved in the project (see Annex 3 of this report). Field visits were carried out to Mason River; Black River; and Blue and John Crow Mountains National Park. Additional material received during the mission was reviewed paying particular attention to project outcomes and outputs. The evaluator provided a debriefing of the main findings and recommendations to the UNDP CO Resident Representative, Deputy Resident Representative and Energy and Environment Program Adviser. The findings were presented through a PowerPoint presentation by the consultant to the Project Steering Committee and other key stakeholders who participated in the MTR, allowing for information exchange and feedback. C) Report preparation: Additional interviews that could not be scheduled during the mission were carried out after the mission by phone or by Skype; A detailed analysis of the data was undertaken; Follow-up phone calls were made and s sent to address information gaps; The information was consolidated and a draft report prepared. The draft was prepared in accordance with the guidelines and Terms of Reference for this Mid-Term Review (see Annex 1 of this report). Upon receipt of the reviewers comments, a final evaluation report was prepared. 2

13 5. No major methodological limitations are identified for this evaluation as the evaluator was able to speak with all the main stakeholders and obtain detailed feedback on the project, as well as review extensive project documentation. 3.4 Structure of the Evaluation 6. The structure of this evaluation followed the Terms of Reference provided by UNDP Jamaica and approved by the UNDP-GEF Regional Service Centre (RSC) (see Annex 1 of this report). UNDP Guidelines for Evaluators as well as GEF evaluation policies were followed as well as the specific expectations of the Implementing Agency (IA) and Executing Agency (EA- also referred to as the Implementing Partner). 4 Project Description and Development Context 4.1 Project Start and Duration 7. The Project Document was signed on July with a planned closing date of June 2016 (6 year duration). The process of hiring a project manager was lengthy and the individual did not come on board until December The inception workshop was held on February 10, The total GEF grant is USD 2,770,000 with USD 5,050,000 in planned co-financing, both in cash and in-kind. 4.2 Development Context and Problems that the Project Seeks to Address 8. Jamaica is a middle income, small island developing state (SIDS) with a population of approximately 2.7 million and an annual population growth is 0.47%. The Human Development Index (HDI) in 2007 was Low economic growth, large fiscal deficits, high unemployment, and weak export performance are all issues Jamaica is struggling to address. The country's economy is dependent on a number of sectors, including remittances, tourism, the bauxite industry, agriculture, and fisheries, which, with the exception of remittances, depend on the country's natural resources. Jamaica's biodiversity is faced with a number of different threats, including overexploitation of biodiversity (for example, as a result of overfishing); conversion and/or destruction of habitat (from activities such as mining, farming, grazing, forestry, tourism, and illegal land appropriation); invasive species; and the overarching threat of climate change. Underlying these direct causes of biodiversity loss are macroeconomic and policy factors, with the country strongly reliant on the exploitation of natural resources. The long-term solution is a consolidated National Protected Areas System (NPAS) supported by a strong institutional framework with the legislative mandate, management capacity and financial support needed for effective protected area conservation. There are, however, a number of barriers impeding this solution, including: i. Policy instruments are inadequate to support efficient and effective financial management; ii. Limited institutional capacity for the management of protected areas; iii. Inadequate funding sources and corresponding financial management mechanisms. The annual financing gap for basic management scenarios was estimated at USD

14 million; iv. Limited public support for PAs and understanding of the benefits of PAs; and v. Inadequate protected area coverage. The project intervention directly addresses these identified barriers. 4.3 Immediate and Development Objectives of the Project The project's objective is to consolidate the operational and financial sustainability of Jamaica s national system of protected areas. The objective will be achieved through three components: (1) Strengthening of planning and revenue generation; (2) Rationalizing and integrating the national system of protected areas; and, (3) Increasing the effectiveness of protected area management. 4.4 Baseline Indicators established 9. The baseline values for this project are specified in the Strategic Results Framework (SRF). The Project Design- Analysis of Logframe section provides detail on the appropriateness and quality of the project design, including the baseline values and expected results. 4.5 Expected Results 10. The Strategic Results Framework identifies the Project Objective and three Project Outcomes, as well as associated indicators, baselines and targets. Jamaica is a global conservation priority with more than 1,400 known endemic species. This project's efforts are expected to deliver global benefits associated with a national protected area system better equipped to conserve globally significant, but currently vulnerable, ecosystems and allied species. 4.6 Main Stakeholders 11. The main stakeholders are described in detail in Table 6 of the Project Document (ProDoc), including their role in the protected areas system. They include the four agencies charged with PA management, namely, NEPA (also the Executing Agency for this project), the Forestry Department, Fisheries Division, and Jamaica National Heritage Trust; Planning Institute of Jamaica; Ministry of Finance and Planning; Ministry of Water, Land, Environment and Climate Change, Environmental Management Division; the Institute of Jamaica and the Urban Development Corporation. Other relevant non-governmental actors include The Nature Conservancy (TNC); University of the West Indies; Jamaica Conservation and Development Trust; Montego Bay Marine Park Trust; Caribbean Coastal Area Management Foundation; Windsor Research Centre; Portland Environmental Protection Association; Negril Coral Reef Preservation Society; local forest or watershed management committees; Negril Area Environmental Protection Trust; parties interested in PA co-management 4

15 and concessions; as well as landowners, resource users, recreationalists, business sector, and others with social and/or economic interests within or near protected areas boundaries. Additional stakeholders that were not mentioned in the ProDoc include the Environmental Foundation of Jamaica and the Forest Conservation Fund. 5 Findings 5.1 Project Design/ Formulation Analysis of Logframe/ Results Framework (Project logic/strategy; indicators) 12. The original Strategic Results Framework included an appropriate project objective and three linked project outcomes. The selected indicators were generally appropriate to measure achievement of this objective and outcomes and make use of internationally recognized tools to measure protected area (PA) management effectiveness and financial sustainability, including the METT and the UNDP financial sustainability scorecard. It could be argued that Outcome 2, rationalizing and integrating the NSPA, lacked an appropriate indicator that would measure the impact of achievement of the output to develop national protected areas legislation and a supporting legal framework (the only related indicator was the number of PAs with legal agreements designating PA management authority). The indicators were generally "SMART" (specific, measurable, achievable, relevant and time-bound), with only a few that were considered to be overambitious in hindsight (in particular, the target of all 32 PAs benefitting from a 25% increase in METT scores, and all 32 PAs having legal agreements for management authority). Baseline indicators were presented for all but one indicator where the figures were to be confirmed at project inception. In general, established links between the planned outputs, outcomes and overall project objective are logical, although the evaluator considers that to fully achieve Outcome 3 (increasing management effectiveness) might have required more project funding for training in PA management for administrators. 13. The protected areas that would receive management and business plans through the project were not identified during the PPG phase and specified in the SRF. This would have saved time during project implementation and facilitated earlier action on the related deliverables. 14. The Strategic Results Framework underwent a revision by the Project Steering Committee to make the indicators and targets more appropriate and realistic. The revised framework was approved by the PSC in late No changes were made to the project objective, three Outcomes or outputs, however, the wording of some indicators was revised and several targets were changed. At the Objective level, the target number of PAs with high, medium and low METT scores was adjusted, and the indicator species for the monitoring of populations were revised. For Outcome 1, the target annual disbursement to the NPAS was reduced, while the other financial targets remained essentially the same. For Outcome 2, the previous target relating to the number of PAs with legal agreements to designate PA management authority was substituted with a target to develop umbrella PA legislation and a supporting legal framework. Finally, for Outcome 3, the target to increase the METT scores for 32 PAs by 25% was changed to increasing the METT score by 25% in 50% of the 32 PAs listed (i.e., 5

16 in a total of 16 PAs). The evaluator considers that these changes do not fundamentally alter the project's goals and are more feasible, particularly in light of the time it can take to modify METT scores, and the delays the project has experienced thus far. Assumptions and Risks 15. The ProDoc identifies the main risks as being: changes in political circumstances and economic priorities affect Government or other stakeholders - including NGO PA managers - financial commitment to NPSA; weak management and technical capacity undermines project outcomes; climate change, natural disasters, and other environmental impacts beyond national borders exceed current expectations; and critical legal and institutional framework necessary to improve management efficiency - including adoption of protected areas law and consolidation of NPAS management regime - will be resisted and not changed. Several additional assumptions are included in the project's Strategic Results Framework. However, three risks that ended up materializing and have significantly affected the project were not sufficiently recognized, namely, the limited national capacity to carry out some of the key consultancies envisioned for the project, the difficulty identifying the required international expertise, as well as the impact of the protracted government procurement policies and processes on the speed of execution. These issues have negatively affected previous projects in Jamaica as well. It would have been useful to identify possible specific mitigation measures, in addition to the longer project implementation period (see next paragraph). Lessons from Other Relevant Projects Incorporated Into Project Design 16. UNDP's Mid-Term Outcome Evaluation of its Energy and Environment Programme (2010) included within one of its recommendations the need to budget in extra time to compensate for time lost as a result of protracted procurement and start-up processes. This project design took this on board to a certain extent by including a six-year implementation period. Given the ceiling on GEF funds that can be used for project management, UNDP provided $200,000 in co-financing for project management to facilitate this longer project duration. Planned Stakeholder Participation 17. The project design clearly outlines the role of the different stakeholders in the PA system. The ProDoc also provides a summary of the expected inputs of the main stakeholders, including NEPA, UNDP/GEF, the Protected Areas Committee, TNC, and the Planning Institute of Jamaica. It might have been useful to provide more detail in this table to cover the other project stakeholders. The proposed institutional arrangements described in the ProDoc include a Project Board as well as possible technical sub-committees which may be established, though more information is not provided in terms of what kind of technical committees would be warranted. 18. Stakeholders participated actively in project design, with a PPG committee set up and regular meetings and workshops held, including a Results Framework workshop and a METT workshop in 6

17 November Some PSC members commented that they would have liked additional time for revisions and that the PPG had to be carried out in less time than originally planned. The other comment made was that more effort could have been spent on consultations at the PA-site level. Replication Approach 19. The ProDoc indicates that the project will fund two national-level workshops to disseminate project findings and activities and thus promote replication. These would involve local and national project managers, community members, government representatives, and protected area staff. The results of the two workshops would be gathered in two documents, one at mid-term and one at project close, and could serve as teaching guides to support replication. The project design also envisions the compilation of lessons learned from the various types of co-management across the PA system. Finally, the ProDoc indicates that each of the project outputs was designed to "generate a critical mass of demonstration effect". 20. At this point, the mid-term national-level workshop has not yet been carried out nor a summary document prepared. The PMU plans to hold this workshop around March 2014 when there are more deliverables to share. However, it should be noted that the broad participation in the project's steering committee has enabled considerable information exchange among various institutions involved in PA management and can be seen as a way of promoting future replication of project activities. In addition, the international legal consultants are beginning to compile relevant information on existing co-management arrangements. UNDP Comparative Advantage 21. UNDP Jamaica has a substantial comparative advantage to serve as Implementing Agency for this project. It has a long-standing physical office in the country, which has enabled the development of valuable relationships with stakeholders. This also facilitates the provision of administrative support and technical backstopping to the Executing Agency. In addition, UNDP has acted as Implementing Agency for other natural resource and environmental projects, such as the demonstration project aspect of the Jamaican component of the IWCAM (Integrated Watersheds and Coastal Area Management) regional project, and the Capacity Building for Sustainable Land Management in Jamaica project. UNDP Jamaica also implements the GEF Small Grants Program (GEF SGP), which provides funding to projects to support community-based and collaborative management arrangements for protected areas, among other areas of focus. Finally, it should be noted that UNDP has assisted with national reporting to the Convention on Biological Diversity (CBD) and to the NCSA (National Capacity Self-Assessment). Linkages between Project and Other Interventions within the Sector 22. The ProDoc includes an appropriate description of how the project will establish linkages with related projects, ensure complementarity, and heighten impact. For example, the GEF Early Action Grant will support the valuation of the NPAS to the national economy, which will complement the project's 7

18 activities to raise awareness of the importance of the NPAS and increase political support. The project will help Jamaica to meet the objective of the Caribbean Challenge of ensuring that Caribbean countries meet the CBD's Programme of Work on Protected Areas in terms of achieving 10% protection of marine ecosystems by The ProDoc also describes linkages with other initiatives such as the GEF Caribbean Large Marine Ecosystem project (CLME), the GEF regional invasive species project, and the USAID Global Sustainable Tourism Alliance project. Management Arrangements 23. The ProDoc establishes clear and appropriate management arrangements. The project is to be executed under the NEX (National Execution) modality, with the National Environment and Planning Agency (NEPA) taking on the role of the project's Executing Agency (EA). NEPA is accountable to UNDP for the efficient and effective use of project resources and achievement of the project objectives. The Project Management Unit (PMU) is to be established within NEPA and is responsible for directing, supervising, and coordinating the project implementation. A National Project Director would be nominated from among senior NEPA staff and a National Project Manager hired by NEPA, to take on the responsibility for day-to-day project implementation, leading and managing the PMU. The PMU would also include an administrative assistant and a part-time accountant. 24. A Project Board is to be established with strategic decision-making, non-executive powers. The ProDoc suggests that this could be composed of NEPA, UNDP and TNC-Jamaica. NEPA is to serve as the Board Executive and Chair of the Board. UNDP is the GEF Implementing Agency for this project and as such is responsible for provision of technical support to the PMU, budget revisions, donor reporting, advance of funds to the EA, and project monitoring. 5.2 Project Implementation Monitoring and evaluation design at entry and implementation Monitoring and evaluation design at entry (Satisfactory) 25. Details on the Strategic Results Framework are provided in the Project Design section, Analysis of Logframe, which provides comments on the baseline and indicators established. As mentioned, the project design was generally strong, with a relevant premise and logical identification of the project's objective and outcomes. This MTR noted that some of the targets were overly ambitious and a few critical risks were not identified. The Monitoring and Evaluation (M&E) plan as designed at entry and described in the ProDoc is comprehensive and includes all the main necessary elements, such as: project inception workshop, quarterly progress reports, annual project reviews/project implementation reports (APRs/PIRs), periodic monitoring through site visits, mid-term review, final evaluation, annual audits, learning and knowledge sharing. Each aspect of M&E is adequately described and justified in the ProDoc. The M&E plan was sufficiently budgeted with a total of USD 140,000 set aside for this work. 8

19 Monitoring and evaluation implementation (Moderately Satisfactory) 26. Overall, the implementation of the M&E plan is considered moderately satisfactory, with adequate reporting (though sometimes submitted late), an active Project Steering Committee (PSC) to help monitor the project, the implementation of adaptive management measures, and a Mid-Term Review underway. The follow-up to M&E activities has generally been too slow, which is the main reason for the Moderately Satisfactory instead of Satisfactory rating. Several elements could be strengthened as will be further explained in the next paragraphs. 27. As the first major M&E activity during project implementation, the inception workshop had strong participation with 50 individuals present, and it adequately served to introduce the project. However, it would have been beneficial for the workshop duration to have been longer than a half day, at least for PSC members, to increase their familiarity with the details of the project and its Strategic Results Framework (SRF) The PMU has been submitting adequate and sufficiently detailed progress and financial reports to the UNDP. There have been some delays in submitting quarterly and other reports and in making the necessary revisions. Also, workplans have taken some time to develop, particularly the first one, and have had to be revised several times (primarily because of the impact of procurement delays). According to interviewees, the SRF is increasingly being used as a tool to guide project implementation, particularly since its revision in 2012 (though there is still little progress toward many of the key targets). As outlined in more detail in the section on "Feedback from M&E", M&E activities, particularly the development of the 2012 PIR, have led to adaptive management measures, though these could sometimes have been implemented more rapidly. 29. The Project Steering Committee has been meeting regularly, with quarterly meetings, in addition to the occasional special stakeholder meetings. The PSC has been active in providing input. However, its original composition consisted of a total of 36 members from 23 different organizations. This extremely large size contributed to slow-decision making. In addition, interviewees commented that there was insufficient understanding of the role of the PSC, with members becoming too involved in micro, technical issues rather than focusing on providing overarching guidance. The PSC composition and mandate was reviewed in late 2012 and its membership reduced. Interviewees commented that recent meetings have been more focused on the provision of guidance. The problem of slow provision of feedback by many of the PSC members to consultancy deliverables remains an issue, despite a more recent decision to set a two week deadline for receipt of comments. This is contributing to slow project progress. 30. The minutes of the PSC meetings have tended to be too detailed and long, and do not generally contain a concise summary of the decisions made. A complete list of all absent members is not always provided. Action tables are included, but deadlines for the achievement of the different actions are not consistently identified. Furthermore, the minutes have been sent out late in the past and even 11 A recommendation to hold this type of workshop was made by UNDP once the PM was hired but was not followed up on until

20 the relatively recent decision to ensure that the final minutes are sent out one week before the next meeting could be improved upon. 31. This Mid-Term Review is being undertaken ahead of schedule, which is a positive move to enable the EA and Implementing Agency (IA) to benefit from recommendations sooner rather than later to increase delivery and project impact. It is not possible to comment on the level of consistency between PIR self-evaluation ratings and the results of the MTR as the 2013 PIR was not yet available to the evaluator. A comparison with the 2012 PIR would not be appropriate given the period of time that has elapsed and the significant progress made since then. UNDP and Implementing Partner implementation / execution - coordination, and operational issues UNDP Execution (Satisfactory) 32. UNDP execution for this project is rated as Satisfactory. The main responsibilities of Implementing Agency have been assumed conscientiously, although there are some issues that could be strengthened as described further on in this section. UNDP has maintained regular communication with the PMU and has been accessible and active in overseeing the project (with one interviewee describing their role as "hands-on"). UNDP has also been an active member of the PSC, providing advice on agenda items for meetings, reviewing minutes, providing recommendations on the structure of the minutes, and participating in the actual meeting discussions. 33. Support for monitoring and evaluation has been provided from the project inception workshop stage. M&E activities have included participation on the PSC, contribution to annual APR/PIRs including by highlighting issues of concern, monitoring of project activities and of PMU files, site visits, and support for realization of the Mid-Term Review. Potential risk factors are being monitored, such as through attendance at meetings for the Trust Fund establishment and follow-up on the issue of the Protected Areas System Master Plan. UNDP has been providing technical support by reviewing project deliverables and providing feedback and by promoting the review of these deliverables by other key stakeholders. The provision of this feedback could at times be more consistent and more timely. UNDP has contributed to ensuring accountability and transparency by providing advice on UNDP and GEF procurement guidelines and participating in the development of TORs and interview panels, among other tasks. UNDP has also offered its assistance in terms of procurement issues. 34. UNDP has been managing the GEF funds adequately, reviewing budget revisions, quarterly reports and requests for NEX advances, issuing advances in a timely manner, and providing support to the PMU on the reporting templates. NEPA has raised some concerns about the formats in use, particularly the Fund Authorization Certificate of Expenditure (FACE) form template which could be improved to make it easier to determine carry forward figures, as well as the fact that it is timeconsuming to review the CDRs produced by UNDP due to currency and exchange rate issues. As these templates are standard to the organization, any modifications would need to be considered at that level. UNDP CO organized a capacity building workshop in May 2012 for national project coordinators to train them in UNDP administrative procedures and reporting formats. Given some of the difficulties and delays experienced by the PMU and others in completing these forms, the 10

21 provision of additional training would be useful. In this respect, UNDP CO plans to carry out training with individual projects and the evaluator recommends that this be done as soon as possible. 35. An important element of UNDP's role is to raise issues of concern to higher levels of management. In this respect, the UNDP CO has been in communication with UNDP CO Senior Management, the RTA in the Regional Service Centre and Headquarters. At the end of 2012, UNDP CO requested, and received, a meeting with the CEO of the Executing Agency, NEPA and the UNDP Resident Representative in order to discuss the findings and the implications of the 2011/2012 APR/PIR. Again and more recently, UNDP CO requested another meeting with the CEO of NEPA and the Resident Representative in order to highlight low delivery rate, the serious delays in project implementation, the need for greater national ownership on the part of the partner agencies involved, the need to escalate key issues to the Ministerial level, and the need for greater action and commitment to improve delivery and project impact. The NEPA CEO was immediately responsive and organized a meeting with and the Permanent Secretaries of the four agencies responsible for protected areas management to share the issues raised by UNDP. These types of meetings are critical to enhancing country ownership, increasing the contribution of different government agencies to the project, and dealing with the various hurdles the project is facing. It is recommended that UNDP CO take on this role more regularly and proactively to raise issues with high-level government officials. 36. The interviews revealed that there was a lack of clarity among some stakeholders on the decision making responsibilities of the PSC versus the UNDP, and of the decision-making remit of the government as Executing Agency versus the UNDP. This may have been in part due to the inconsistency between the ProDoc, which indicates that UNDP has the final call to make decisions, and the Terms of Reference of the PSC, which specifies that the PSC has the final decision-making power. In this respect, it might be useful for Project Documents to provide more detail on the role of PSCs and UNDP, rather than using primarily standardized text, and to ensure that the issue is discussed and agreed upon during the PPG phase. 37. Another point raised during the MTR was that UNDP needs to state its position on key issues clearly from the outset. Interviewees commented that with regard to a few issues, such as where the Protected Areas trust would be housed, UNDP's position was not clear to them during the initial PSC discussions. Strengthening communication in this respect could avoid confusion among stakeholders and also ensure that time is not lost unnecessarily. 38. More rapid responses are required by UNDP CO in order to reduce project delays. For example, in March 2012, the project team and various stakeholders expressed their desire to strengthen the PMU with additional capacity to support the project manager. The UNDP CO had some concerns about possible overlap between the originally prepared TORs for the new position and the responsibilities of the project manager. The taking of a decision on this matter and revision of the TORs to make them acceptable to all parties has taken over a year and has still not reached its conclusion. 2 There were also some delays in terms of UNDP CO's response to the issue of the location of the NPATF. In 2 It should be noted that in June 2013, NEPA's CEO reiterated the request for a high-level technical adviser and UNDP responded within two weeks that it would accept this position. NEPA has now expressed its interest in hiring a project coordinator as well to support the PMU. 11