US$M): Sector Board : Social Development Cofinancing (US$M (US$M US$M): US$M):

Transcription

1 Public Disclosure Authorized IEG ICR Review Independent Evaluation Group Report Number : ICRR Project Data: Date Posted : 09/22/2014 Public Disclosure Authorized Public Disclosure Authorized Country: Timor-Leste Project ID : P Project Name : Timor Leste - Youth Development Project Appraisal Actual Project Costs (US$M US$M): L/C Number: CH414 Loan/Credit (US$M US$M): Sector Board : Social Development Cofinancing (US$M US$M): Cofinanciers : None Board Approval Date : 07/01/2008 Closing Date : 12/31/ /30/2012 Sector(s): Other social services (62%); Public administration- Other social services (38%) Theme(s): Other social development (50% - S); Participation and civic engagement (20% - S); Other rural development (15% - S); Conflict prevention and post-conflict reconstruction (15% - P) Prepared by : Reviewed by : ICR Review Coordinator : Hjalte S. A. Sederlof Roy Gilbert Christopher David Nelson 2. Project Objectives and Components: Group: IEGPS1 a. Objectives: To promote youth empowerment and inclusion in development by expanding the capacities of and opportunities for youth groups to initiate and participate in community and local development initiatives." (Financing Agreement -FA - Schedule 1 p.4; and Emergency Project Paper - EPP - p.1) Youth are defined as individuals between the ages of 16 and 30, as defined in Timor Leste s National Youth Policy. Public Disclosure Authorized Initially, there were three key outcome indicators : percent change in female and male youth from targeted communities who feel that there are more opportunities for youth participation in community and local development; percent change in female and male youth from targeted communities who participate in planning and managing community development activities in urban and rural areas. 3. Improved State Secretariat for Youth and Sports performance ratings (to satisfactory) During mid-term review, indicators 1 and 2 were modified to read as follows: percent of targeted communities where youth feel that there are more opportunities for youth participation in community and local development; percent of targeted communities that report increased youth participation in planning and managing community development activities in urban and rural areas. The two indicators were modified in the absence of a baseline, and insufficient resources for accurately assessing gender differences.(see also Section 10 on M&E design). The third key outcome indicator remained unchanged, although its target achievement was adjusted from "satisfactory" to "moderately satisfactory" during the mid-term review. b.were the project objectives/key associated outcome targets revised during implementation?

2 Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components: Component 1: Operationalizing the National Youth Policy (Appraisal estimate US$ 520,000; actual US$ 290,000). The component supported the Secretariat of State for Youth and Sports (SSYS) to operationalize the National Youth Policy by: (i) strengthening its institutional capacity to develop, implement and coordinate policy and programs relating to its mandate, including the establishment of a unit to administer a National Youth Fund within SSYS; and, (ii) enhancing its ability to work with youth groups, organizations and youth centers, and support youth initiatives. The component consisted of the following two sub -components: Sub- Component A : Institutional Strengthening. Drawing on the findings of a Bank supported capacity assessment of the SSYS, the sub-component would assist the SSYS to implement a series of capacity building workshops and conduct an annual progress report of its capacity. In addition, it would help SSYS to undertake an annual expenditure review of the government s budget allocation for youth programming; and it would help establish a unit in SSYS to manage and administer the National Youth Fund, a key element of the Government s National Youth Policy (NYP) and for which the SSYS has a significant budget allocation (about US$ 2.5 million was earmarked to be spent by the end of December 2008). Specifically, the sub-component would finance the design and implementation of guidelines and mechanisms to administer and monitor the National Youth Fund. Sub-Component B : Strengthening Youth Centers. This sub-component would help strengthen the capacity of the SSYS to work with and coordinate the activities of youth centers. This would include capacity building for the staff and management of select youth centers and youth associations to improve their management and programming. The SSYS would provide funding grants, or administrative grants, for select youth centers and youth associations from the National Youth Fund. Component 2: Youth for Local Development (Appraisal estimate US$ 1.6 million; actual US$ 1.8 million). The component would support the Ministry of State Administration and Territorial Management (MSATM) to establish a mechanism to distribute small grants to youth in rural villages (suco) and hamlets (aldeia) via the MSATM s existing local Local Development Program (LDP). The Project would provide support to pilot the component in five districts over the three year project period. Each district would have a minimum of two cycles of funding. The Government would scale up the Program to all 13 districts in Timor-Leste if the pilot was successful. The component had three sub-components: Sub-Component A : Youth Grants. The sub-component would provide earmarked funds for rural youth. These funds would be made available to youth groups at the village level to identify, prioritize, and implement sub-projects-such as livelihood activities, rehabilitating youth centers, social events, construction of small infrastructure, and life skills training-within their suco. The planning, implementation, and monitoring process would be driven by youth. The grants would be allocated based on total population so as to ensure fairness. The amount allocated was to be approximately $ 1.80 per person, resulting in grants of approximately $ 2,200 to $5,000 per suco depending on the population. Sub-Component B : Youth Facilitators and Capacity Building. The sub-component would provide support to train and staff a corps of Youth Facilitators who would support implementation of the component s activities at the village level. Such activities would include socialization of the project to suco youth, youth consultation meetings, and capacity building support to a Youth Implementation Team for each suco. These workshops would assist the Youth Implementation Teams to refine their proposals, plan their implementation, learn how to mitigate social and environmental impacts in sub -projects, and ensure that at least one member of each Youth Implementation Team is trained in the project s basic financial reporting obligations. The capacity building workshops under this sub-component would link with those under the National Youth Fund. Sub-Component C: Program Implementation and Technical Support. The sub-component would augment the project implementation capacity of the existing Project Management Unit within MSATM that supports the Local Development Program. This would include outreach, training, administrative support staff, equipment as well as technical assistance with documentation, training development and M&E. This sub-component would also include socialization and communication activities associated with the program. In addition, technical assistance would be provided to ensure that the use of the youth grants are gender inclusive.

3 The scope of activities under the two components was revised at mid -term review in October 2010 to address delays in project implementation and technical and capacity challenges in the implementation of the first project component. Changes in senior staff in the SSYS affected implementation capacity that in particular affected the operationalization of the National Youth Fund, which initially suffered repeated delays and finally was dropped at mid-term, as was a planned annual expenditure review. This in turn influenced the extent to which the project would be able to reach out to youth in urban areas and, more generally, the sustainability of the Fund approach. In parallel, activities under Component 2 were scaled-up with Youth Grants provided to an additional district, and adjustments in its operations were introduced to make grants more responsive to the needs of youth groups. In line with the changes in the two components, funding allocations were modified (reducing the former and increasing the latter); key indicators were modified as noted in Section 2a above; and the Closing Date was extended from December 31, 2011 to September 30, d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost and financing. Project cost at appraisal was estimated at US$ 2.1 million. Actual project cost was also US$ 2.1 million. There were adjustments to component costs, reflecting changes in the scope of the components at mid-term review (Section 2c). The total amount was financed by IDA. Borrower contribution. There was no Borrower contribution. Dates. The project was prepared as an Emergency Project, reflecting a fragile security situation and the need for urgent support to the Government to help stabilize a volatile political situation. It was approved on July 1, 2008, after a three-month preparation period that would prove to be too short taking into account limited institutional capabilities in the country; and it is reflected in the project becoming effective only on March 12, Following an extension of the Closing Date at mid-term review, the project closed on September 30, 2012 (see Section 2c). 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of objective is rated high. The PDO is consistent with the Government's National Priority Program and its National Youth Policy; and with the Bank s Country Assistance Strategy (CAS). The National Youth Policy responds to extensive violence that had erupted in Timor Leste in 2006, where youth had played a prominent role in engendering instability and crisis. With a significant number of young people feeling increasingly excluded from economic development and civic life, the policy emphasized empowerment and inclusion into local level decision -making as means of reintegrating them into mainstream society. While this approach is maintained and continues to receive donor (albeit not Bank) support, the emphasis of youth policy has undergone a shift towards job creation. Likewise, the Bank s approach has evolved. The Country Assistance Strategy (the CAS) stressed the importance of addressing youth issues across a broad front, including delivering sustainable services; creating productive employment; and strengthening governance, the emphasis in the latest Bank CAS for the period is on creating employable youth.(cas p. 17). b. Relevance of Design: Relevance of design is rated modest. Project design was consistent with project objectives. The project was to build up central government capacity to implement its youth policy of empowerment and inclusion (Component 1); and provide field-level financial and technical assistance to obtain immediate gains in applying it (Component 2). Empowerment and inclusion were defined as means to expand the capacities of youth and their opportunities to initiate and participate in community and local development initiatives (EPP p. 1). The risk analysis and related mitigating measures attempted to address the still uncertain political and policy environment. The results framework was weak: while it listed the outcomes that were expected to meet the PDO, there is no information on the inputs and outputs that were to generate the necessary outcomes, for instance, how was the intermediate objective of targeted youth centers provide services that respond to demands of youth to be achieved? Similar observations can be made for the other intermediate objectives. 4. Achievement of Objectives (Efficacy): OBJECTIVE - TO PROMOTE YOUTH EMPOWERMENT AND INCLUSION IN DEVELOPMENT BY EXPANDING THE CAPACITIES OF AND OPPORTUNITIES FOR YOUTH GROUPS TO INITIATE AND PARTICIPATE IN COMMUNITY AND LOCAL DEVELOPMENT INITIATIVES.

4 Efficacy is rated modest. Outputs 1. The establishment of a more effective central institution (SSYS) and a network of functioning youth centers : A performance monitoring system for SSYS was put into place, but functions with shortcomings The operationalization of the National Youth Fund was not implemented, removing a key tool of government outreach to youth groups; and The Youth Public Expenditure Review was not implemented All members of targeted youth center management committees that received training reported an increase in leadership, management and planning skills 2. Empowerment of young people to participate in community level activities by providing grants to youth groups at the village level, which would allow them to discuss their own priorities with community leaders, and participate in the planning and implementation of projects relevant to them. Some 22 percent of youth participated in youth decision -making meetings in target communities, compared with a target of 30 percent 289 projects were implemented by youth in target communities compared with a target of 200 projects 17 percent of youth participated in the implementation of youth development projects in target communities, compared with a target of 20 percent 100 percent of youth in targeted communities were satisfied with the goods and services delivered through project youth grants, compared to a 70 percent target Outcomes In 100 percent of targeted communities, youth felt that there were more opportunities for them to participate in community and local development, compared to target of 60 percent 100 percent of targeted communities reported increased youth participation in local community development activities, compared to a target of 60 percent. Performance of the Secretariat of State for Youth and Sports was rated moderately unsatisfactory, compared to an initial target rating of satisfactory and a mid -term revision to moderately satisfactory; based on a capacity building progress review by consultants In assessing to what extent the project development objective was attained, the following should be considered : does the Government have a more effective institution SSYS to support a network of functioning youth centers; as a result of the project; and is there a working mechanism that generates increased opportunities for youth participation at community level. In the former case, institutional strengthening was partly achieved, including partial improvements in basic management systems and capacity building for youth center management. At the same time, however, two project activities were cancelled (operationalization of the National Youth Fund, the public expenditure review), and a third one (performance monitoring), essential to better monitoring of SSYS performance, was only partly achieved. In the latter case, the project reached its objective of creating opportunities for youth inclusion and empowerment but was this driven by the availability of youth grants, or has the project given rise to a new paradigm of accepted participation by youth in community deliberations? Moreover, while the key outcome indicators confirm that opportunities had been created, the absence of a baseline makes it difficult to determine by how much they represent an improvement, i.e. the actual impact of that part of the project. 5. Efficiency: Efficiency is rated modest. Neither the project nor the ICR included an economic or financial analysis resulting in an internal rate of return. The ICR notes that based on available information, it was not possible to assess the efficiency of the project (ICR, p. 28). That said, the following three factors should have influenced efficiency, although precise indicators are lacking : (i) on the positive side, the use of existing MSATM structures and the Government's Local Development Project as platforms for activities at the village and hamlet level was an efficient approach of adapting existing institutions to project ends; (ii) on the negative side, efficiency may have been reduced by the capacity constraints encountered in SSYS and resulting in some elements of the project being cancelled (Section 4.1); and (iii) the disbursement profile shows actual disbursements consistently lagging original estimates.

5 a. If available, enter the Economic Rate of Return (ERR ERR)/Financial Rate of Return (FRR FRR) at appraisal and the re-estimated estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal ICR estimate No No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the project objective was rated high, as it directly supported government priority efforts to integrate youth into the economy and civil society. Relevance of design was rated modest, reflecting a weak results framework. Efficacy was rated modest - important parts of the project were not completed and in some instances, targets were not met. Efficiency was also rated modest in the absence of any strong positive efficiency indicators. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Project outcomes face a number of risks related to sustainability. At the political level, government priorities have shifted, and while empowerment and integration remain important, the focus now appears to be more on youth employment. At the technical level, (i) the modest success with strengthening SSYS, which was to play a key role in operationalizing the National Youth Policy (develop policies and programs; and work with youth groups ) makes effective implementation of the policy uncertain in the future; and (ii) inadequate quality of the majority of youth-initiated sub-projects (an outside review determined that only 20 percent of sampled projects were of acceptable quality) may undermine gains in youth participation that were achieved under the project, Continued limited capacity for supervision of works is unlikely to mitigate this in the near future (ICR p. 17). On the positive side, support to a continued engagement in youth development through inclusion and empowerment is being maintained through interactions between the government and donors a new Youth Development Policy is being developed, and further capacity building is being discussed, with UNICEF and GtZ as driving forces. While youth development in the sense of the project may not be front and center on the country s agenda, these initiatives provide some expectations that the efforts under the project will not grind to a halt. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project was strategically relevant : it responded to the country s need for quick support in a still fragile security environment where a significant number of alienated young people added to political volatility. Design focused on building capacity for achieving immediate gains in applying the country s youth policy. The project was quickly prepared, over a period of four months, which may have been too short for the uncertain environment that prevailed at the time; while limited institutional capacity had been recognized in the risk analysis, and mitigating measures had been introduced, they would prove insufficient and adversely affect implementation : implementation would take four years, including eight months for effectiveness, and revisions to design along the way, and component 1 would be only party implemented (Sections 2c and 4), and insufficient quality control of project works would raise quality issues in a majority of completed sub -projects under component 2. Preparation included neither economic nor financial analysis (this would be addressed during implementation ), nor discussion of the project s potential fiscal implications. The necessary fiduciary and procurement analysis for the project had been done, risks identified and preventive measures introduced. However, staffing difficulties throughout implementation would generate delays in procurement and disbursements. A monitoring and evaluation plan was drawn up; it was ambitious but appropriate in view of the team s intentions to seek external financing for some of the M&E activities. In practice, such financing for M&E would not materialize, requiring adjustments in key indicators during implementation (Section 10b). Quality-at at-entry Rating : Moderately Unsatisfactory b. Quality of supervision: Supervision included regularly scheduled supervision missions and technical supervision of sub -project works

6 by district staff of MSATM. According to the ICR (p. 5), five supervision missions were undertaken, During mid-term review a number of adjustments were introduced to address issues that arose during implementation : expected trust fund resources that were to finance baseline data collection for key indicators did not materialize and required adjustments to the indicators; capacity constraints, including continued staffing difficulties, generated delays in implementation, while changes in senior staff shuffled priorities, especially in SSYS, would contribute to only partial implementation of component 1. Faced with implementation delays and changing government priorities, the Bank team focused project energies post mid -term on interventions that were most likely to have a direct impact on empowerment/inclusion, that is, local development through sub -projects. The team was not able to tackle the quality issues that arose in sub -project development, at least in part due to capacity constraints in technical supervision, as the agency in charge, MSATM, was given new responsibilities by the Government. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government remained committed to the project throughout implementation, and project objectives remain relevant, as indicated by continued focus on working with donors (UNICEF, GTZ) to develop the national youth strategy. Still, the immediate urgency of addressing the exclusion of young people through community development initiatives appeared to cede to a greater concern about creating job opportunities and building skills for youth, as the immediate urgency of security concerns that underlay the project strategy diminished. Thus, a dedicated youth intervention in rural areas is not likely to be the preferred approach of the Government going forward (ICR p. 26). The difficulties encountered in resolving capacity constraints in the implementing agencies SSYS and MTSAM, and which negatively affected project implementation, may in part have been due to this change in emphasis; as may have been the case with delays in coming to agreement on adjustments to the project at mid-term review (ICR, p.33) Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The implementing agencies SSYS and MSATM were able to implement the project despite weak institutions, and high staff turnover in the case of SSYS, especially at senior levels. This severely limited implementation capacity in SSYS, and support for project sub -components. Initially, these problems were reflected in slow implementation, and subsequently, during mid -term review to decisions to not operationalize the national youth policy and consequently eliminate the planned public expenditure reviews for youth programs. The MTSAM was assigned responsibility by the Government for a new development program, limiting its capacity to monitor quality of youth sub-projects. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: M&E design included separate MIS for the two implementing agencies for regular monitoring of project progress; a quantitative impact evaluation; and a series of project -related thematic studies (infrastructure review, cost/benefit analysis, socialization and gender reviews, facilitation ). Donor funding was to be sought for the impact evaluation and the thematic studies. b. M&E Implementation: During implementation, M&E design was adjusted. Failing to secure donor financing, the impact evaluation had to be limited to a beneficiary assessment, and the thematic studies dropped. Key outcome indicators had to be

7 reformulated in the absence of baseline data that the task team had expected to collect in connection with the impact evaluation. c. M&E Utilization: The beneficiary assessment provided information that made it possible to understand improvements in youth participation rates at community level, and generated lessons learned. The Bank team believed that the information would have provided valuable information for a subsequent Government initiated community development program. It is not clear whether that in fact was the case. More generally, database maintenance and the production of progress reports was regularly carried out, and feedback was provided to implementing agencies. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project triggered OP 4.01 and an Environmental and Social Management Framework was developed to guide project implementation. b. Fiduciary Compliance: Financial Management : The recipient complied with the financial management (FM) conditions outlined in all of the financing agreements and satisfied the Bank's requirements under OP /BP The FM risk was rated as Substantial even after mitigating strategies to address these risks. There were significant shortcomings with financial management observed during project implementation, and financial management performance was frequently rated as Unsatisfactory" throughout the life of the project (ICR, p. 21). In addition, disbursements under Component 2 were temporarily suspended in 2009 based on FM review findings. Implementation support provided follow-up on problem sub-projects. In the exceptional cases where a satisfactory resolution was not reached by MSATM, these were cancelled/ and expenditures considered ineligible. All audit opinions were unqualified and all audits were accepted by the Bank. With the exception of the first year audit, there was no implementing agency response to audit recommendations. This lack of action resulted in continued shortcomings with financial management and contributed to the Unsatisfactory ratings. Procurement : Delays with procurement were frequent during project implementation and had a negative impact on disbursement and the speed of implementation. To a large extent these difficulties can be attributed to the lack of dedicated procurement staff while staff turnover reduced the effectiveness training provided by Bank team. c. Unintended Impacts (positive or negative): n/a d. Other: n/a 12. Ratings: ICR IEG Review Outcome: Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Significant Significant Outcome: Reason for Disagreement /Comments Bank Performance : Moderately Satisfactory Borrower Performance : Moderately Unsatisfactory Quality of ICR : Moderately Unsatisfactory Moderately Unsatisfactory Satisfactory A short preparation did not sufficiently take into account institutional constraints, extending effectiveness and implementation, and leading to only partly completed components

8 NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICRR gives rise to the following lesson: Emergencies and priorities. In line with Bank strategy at the time of classifying all Timor Leste projects as Emergency projects under OP/BP 8.00, this project was an Emergency Project. In emergency situations, priorities are bound to change over time, and country support for different project elements, or the project itself, may change. This underlines the importance of selecting interventions that have an immediate impact and can be implemented over relatively short periods of time. Grant size and supervision effort. The ICR indicates that resources for supervision were a function of grant size (ICR, p. 32), and turned out to be insufficient to cope with all the challenges of an emergency environment. Calibrating supervision resources to the challenge rather than the size of the Bank loan or grant may be a more rational approach. The following (edited) lessons are drawn from the ICR: Involving youth in the community. Offering youth opportunities to engage in community activities can reduce tensions and bring about a sense of belonging and shared responsibility for outcomes in their communities, Using technical assistance. Specialized technical assistance for livelihood activities is particularly relevant in contexts such as Timor-Leste, where sub-national level staff is likely to have limited technical expertise. Here, capacity constraints at central levels can serve as a good indicator of such bottlenecks that are particularly important to address when field-level interventions are key for success. Monitoring and evaluation. Adopting an M&E strategy using available data and resources should be considered in small-scale interventions; possibly combined with targeted studies and process evaluations that can generate results early on and inform timely corrective action. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: This is a thorough ICR that backs up its assessment with evidence and analysis. The logic of the discussion of the problems that the project faced during implementation is well set out and the arguments are internally consistent. On the results orientation, the ICR is clear in shaping the ex post framework and discussing outcomes based on it. The ICR is consistent with Bank guidelines. a.quality of ICR Rating : Satisfactory