US$M): Sector Board : ED Cofinancing (US$M US$M): Loan/Credit (US$M Sector(s): US$M):

Transcription

1 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 11/19/2007 Report Number : ICRR12797 PROJ ID : P Project Name : Bo- Education Quality Project Appraisal Actual Project Costs (US$M US$M): Country: Bolivia Loan/Credit (US$M US$M): Sector Board : ED Cofinancing (US$M US$M): Sector(s): Primary education (40%) Secondary education (39%) Sub-national government administration (20%) Central government administration (1%) Theme(s): Education for all (29% - P) Municipal governance and institution building (29% - P) Administrative and civil service reform (14% - S) Gender (14% - S) Participation and civic engagement (14% - S) L/C Number: C3096 Partners involved : Swedish International Development Agency (SIDA); Kredinstalt fur Wiederaufbau (KfW) Board Approval Date : 06/16/1998 Closing Date : 12/31/ /31/2006 Evaluator : Panel Reviewer : Group Manager : Group: Lisa N. Overbey Ridley Nelson Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The Project objectives as stated in the PAD were used for the purpose of evaluating the ICR as there are no significant differences between these objectives and those listed in the legal agreements. The PAD objectives were to increase the number of years of schooling completed (particularly primary level), to improve the quality of that schooling, and to develop approaches to reduce gender disparities in educational outcomes in selected municipalities where the differences are the greatest. The Development Credit Agreement stated that the Project objective was to assist in the implementation of the Borrower's educational reform program focusing on low -income rural and peri-urban areas, through targeted interventions aimed at improving the quality and equity of the Borrower's education system, facilitating the access of the target population to basic education, reducing desertion and repetition rates, and increasing the educational performance of boys and girls.

2 b.were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): A. Municipal Educational Development Program (Appraisal Estimate: US$ million, 67.7% of original project cost; Actual: US$ million). This component was planned to fund Nucleus Education Projects within the Curricular Transformation Program that would be selected through a municipal contest based on evaluations and reviews at the departmental and national levels subject to established co -funding ceilings. B. Institutional Strengthening (Planned: US$ million, 14.2% of original project cost; Actual: US$ million) Training, technical assistance and equipment for municipalities, the nucleus and equipment for the district head offices. Also included financing for an inventory of the condition of existing educational infrastructure an information subsystem, monitoring and evaluation and administrative costs. C. Social Investment Fund (FIS FIS) Investments (Planned: US$ million, 12.9% of original project cost; Actual: US$ million) To provide an immediate response to pressing needs in the short run, financing of some educational infrastructure projects that were identified and evaluated by the FIS but, because of a lack of financing, had to be suspended. This was financing for investments made before the first contest set forth by the project. D. Rural Girls' Schooling (Planned: US$ 1.00 million, 1.0% of original project cost; Actual: US$ 0.80 million) Research action component to establish a strategy that will allow for a sustained increase in female enrollment and retention within the education system in rural areas. E. Studies on Secondary and Tertiary Education (Planned: US$ 0.50 million, 0.4% of original project cost; Actual: US$ 4.60 million) Supported studies to evaluate and make recommendations on the expansion of the Education Reform Program to secondary and tertiary education. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The Rural Girls' Schooling component was revised in The expected outputs were adjusted due to challenges encountered in gathering information in targeted municipalities and analyzing it in a timely manner and the cost of implementing the strategies. Studies were carried out in 15 municipalities instead of 50 as planned in the PAD. The Credit financed these studies and a smaller pilot program to support rural girls' enrollment. The originally planned component activities were later implemented with alternative financing from UNICEF in partnership with the Government and IDA. The closing date was extended four times totalling 3 years. Four main reasons were given for the extensions to the closing date: (i) the choice to maintain involvement with the education sector during a period of changes in MOE staff;( ii) the implementation period for the rural girls' schooling component was longer than anticipated; (iii) the scope of tertiary education activities was expanded; and (iv) the significant amount of undisbursed funds that different government administrations required for completing programmed and committed activities. 3. Relevance of Objectives & Design: Relevance of Objectives : Substantial The project development objectives were and remain substantially relevant to Bank strategy and conditions within the country. They were consistent with the Government's development strategy of expanding access to basic education and improving the quality of education. Project objectives were also consistent with Bank strategy. The development objects were based upon the objectives of an ongoing Bank project, The Education Reform Project, and supported the 1998 Bolivia CAS objectives of improving the efficiency and quality of social services, improving access to social services and deepening decentralization. Project development objectives remain relevant in the context of the most recent CAS for Bolivia prepared in 2004, which plans to reorient expenditure towards programs that respond to underlying factors that led to social conflict, especially in sensitive geographical areas. Relevance of Design : Modest In some respects, project design was relevant to Bank strategy and country conditions but it did not allow for measurement of project progress and attribution which would have enabled better understanding of impact. 4. Achievement of Objectives (Efficacy): Achievement of Objectives : Modest Objective 1. Increase access to schooling and number of years completed : Modest

3 Inputs and outputs : Education improvement projects were implemented in all of the country's 314 municipalities. A total of 832 Nucleus Educational Projects (PENSs) developed in 6,151 rural schools (59% of target; ICR p. 15). 31 Indigenous Educational Projects implemented targeting indigenous vulnerable populations. No information was given on the number of beneficiaries of the 832 Nucleus Educational Projects that were implemented. Outcomes: An indicator was planned to monitor the average years of schooling for those leaving the school system in areas supported by the Project. Data related to the indicator was collected by the National Institute of Statistics (INE), reporting rates nationally. They reported an increase in the average years of schooling from 6.1 in 1992 to 7.4 in 2001 (no defined target). Gross primarily school enrollment increased from 99.8% in 1992 to 104.2% in Gross coverage for initial, primary and secondary levels rose from 75.5% to 88.4% over the same period. The primary completion rate increased from 61.8% in 2001 to 68.8% in The promotion rate increased 86.1% to 92.6% between 1998 and The drop out rate decreased from 7.9% to 5.2% and the repetition rate dropped from 6.1% to 2.1% over the same period. Impact: Reported indicators are for the entire population, not specifically for the project schools. In the absence of better M&E, it is therefore difficult to attribute improvements to the project or assess the impact of project activities on this project development objective. Objective 2. Improve the quality of schooling : Negligible Inputs and outputs : National Education Council and nine Departmental Education Councils were created. 274 district boards were created and 1,790 educational nucleus boards established. These boards elaborated and implemented integrated infrastructure and pedagogy projects. 313 Municipal Education Programs (PROMEs) and 1,500 Nucleus Education Projects (PENs) were elaborated by the 274 district boards. A total of 832 PENs were actually developed. There were 1,214,000 student beneficiaries of the Education Projects (PEs) and Municipal Education Programs (63% of total primary enrollment), 56,844 teacher beneficiaries (69% of total teachers) and 7,736 education units which benefited (65% percent of total). Outcomes: New education institutions were established and education programs financed in 100% of municipalities, intervening in at least 50% of the education nucleus in each municipality. It is unclear what specific changes or improvements occurred in the classroom and no data is provided on learning outcomes. Impact: The project planned to use the System to Measure the Quality of Education (SIMECAL) to assess improvements in quality of schooling. A planned indicator to asses changes in the quality of education was the change in the percentage of 3rd grade students with satisfactory academic achievement. Evaluations were carried out in 1997, 1999, and 2000 by SIMECAL but they were not done for the same grade in all three years. In 2001 the entire system was dismantled. It is difficult to assess the impact of the project on the quality of schooling or learning outcomes in project schools. No data exists on the academic achievements of 3rd graders in the project areas, a planned indicator for measuring improvements in the quality of schooling. The large number of teacher, student and education units exposed to project activities make it conceivable that the quality of schooling improved, however, there is no quantifiable data to support these claims. Objective 3. Reducing gender disparities : Negligible Inputs and outputs : In-depth studies were planned and carried out in 50 municipalities to diagnose the obstacles regarding girls' access to education, retention, attendance and drop -out rates. Interventions were carried out in 15 of the municipalities. The project originally planned to intervene in all 50 of the municipalities in which the studies were carried out but due to budget constraints in the MOE which delayed implementation, the component financing studies on girls' participation in education was revised. Interventions in the remaining municipalities were carried out by other international agencies. Outcomes: According to census data, average years of schooling for females nationwide increased from 5.2 in 1992 to 6.7 years in 2001 (as opposed to an increase from 7 to 8.2 years for males during the same time period). Impact: The data reported for the indicator measures gender differences in average years of schooling for those leaving the system as a whole and not specifically for the 15 municipalities. While the data shows that the difference in average years of schooling among male and female students decreased, there is no evidence that the gender gap closed in the 15 municipalities targeted by the project or that disparities changed faster for these 15 than among the country as a whole. Therefore, as noted in the ICR, it is difficult to attribute solely to the Project these improvements in outcomes and therefore the impact of the project on this development objective. 5. Efficiency (not applicable to DPLs): Efficiency is rated only modest mainly due to the attribution problem. A cost-benefit analysis was conducted during project appraisal. It was determined that direct benefits would accrue to the country through improvements in internal efficiency by reductions in repetition and drop out rates and by reduction of the numbers of years taken to complete primary education. While the Project was expected to have a positive impact on drop -out and repetition rates, no specific targets were set. The repetition rate dropped from a baseline of 9.8% in 1996 to 2.1% in The drop-out rate decreased from a baseline of 10% in 1996 to 5.2% with the project in These results, while better than those projected at appraisal, reflect changes nationally and it is therefore difficult to determine how much the improvements can be attributed to the project.

4 Additionally, there is a lack of data linking the inputs for each project development objective to the reported outcomes. All components were successfully implemented, however, there was a three year delay caused by social and political instability in the country which is likely to have impacted efficiency. a. If available, enter the Economic Rate of Return (ERR ERR)/Financial Rate of Return (FRR FRR) at appraisal and the re-estimated estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Substantial relevance but modest efficacy and efficiency suggest significant shortcomings. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The uncertain political and social context in the country pose the main risks to continuation and expansion of project outcomes. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Ensuring quality at entry: The project team responsible for preparation and implementation had substantial experience, having previously helped the Government with the preparation and implementation of the Education Reform Project. Design was based upon the lessons learned from the implementation of other projects in the sector within the country and the objectives relevant to Government and Bank country assistance strategies. The arrangements for M&E of project development objectives were weak. The IDA team chose not to include a specific component or subcomponent for M&E choosing instead to rely on national survey data provided by the country's Integrated System of Educational Information (SIMECAL) to monitor improvements in outcomes. No provisions were made, however, to make sure that the national data could be used to determine attribution of improvements to project activities. Ensuring quality of supervision: Coordination occurred between the IDA team and the Government's implementing agencies; the MOE and Social Investment Fund. Supervision in the context of instability in the MOE (seven ministers in three years) was especially challenging. More detail could have been provided in the ICR about the technical quantity and quality of supervision work (A similar comment was given in the Quality of Supervision Assessment). a. Ensuring Quality -at at-entry Entry:Moderately Unsatisfactory b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government Performance: The Government cooperated well with the IDA team and fulfilled its financial obligations. Performance weakened after 2004, suffering from lack of continuity and stability within the government team. The changes within the Government affected project implementation influencing the decision to extend the project closing date three times in three years. Implementing Agency Performance: The MOE and National Investment and Productive Social Fund were the two implementing agencies. The volatile social and political situation led to changes in MOE staff that affected the

5 agency's performance. The SIMECAL learning assessment system was dismantled after 2001 by leadership within the MOE. This negatively impacted the ability to measure Project achievements in terms of improvements in the quality of schooling. a. Government Performance :Satisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : Quality of M&E design was modest. The planned output indicators - average years of schooling for those leaving the system; percentage of students with satisfactory academic performance and gender differences in educational outcomes - were aimed at collecting the right data, given project objectives and data already available. Additional indicators measuring physical inputs and client satisfaction were also planned. M&E design, however, did not make provisions to ensure that data collected would reflect outcomes in the specific schools or groups targeted by the project. M&E Implementation : Quality of M&E implementation was negligible. The reported data on PDO indicators document improvements in outcomes which may or may not be attributable to the project. Reported data on intermediate outcome indicators was more comprehensive, documenting improvements between target and actual values measuring the quality of Education Projects, client satisfaction and implementation of physical inputs. M&E Utilization : As a result of the above, utilization cannot be assessed. a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Political and social instability in the country beginning in 2003 negatively impacted the project. It led to delays in the implementation of components and contributed to the weaknesses in M&E. 12. Ratings: Outcome: Satisfactory Risk to Development Moderate Outcome: ICR IEG Review Moderately Unsatisfactory Moderate Reason for Disagreement /Comments Weak Project M&E makes it difficult to assess the attributable achievement of objectives. Efficacy and Efficiency are both rated modest. Bank Performance : Satisfactory Borrower Performance : Satisfactory Quality of ICR : Moderately Unsatisfactory Moderately Unsatisfactory Satisfactory Quality at entry was compromised by the failure to include arrangements for adequate project M&E. The implementing agency's role in compromising the quality of Project M&E through its decision to dismantle the SIMECAL system after NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

6 13. Lessons: Nucleus Educational Projects (PENs) can be a useful tool to address diverse educational needs : In a situation of political instability, developing strong partnerships with local authorities at the sub -national level can be effective in maintaining stakeholder support. An adequate monitoring and evaluation system specific to the project area or interventions is important to assess a Project's progress and encourage informed decisions during implementation. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provided a clear description of the project development objectives and available outcomes. It also included a good analysis of the impact social and political instability within the country had on project implementation. However, it was very difficult based on the presentation of quantitative or qualitative information in the ICR, to differentiate between project achievements and what happened in the sector as a whole. The ICR lacked adequate discussion of other factors besides the Project, in particular, the ongoing Education Reform Project, which may have also influenced improvements in education outcomes. The ICR is rated satisfactory, but only marginally so. a.quality of ICR Rating : Satisfactory