ASIAN DEVELOPMENT BANK

Transcription

1 ASIAN DEVELOPMENT BANK PCR: PAK PROJECT COMPLETION REPORT ON THE SOCIAL ACTION PROGRAM (SECTOR) PROJECT II (Loan 1493-PAK[SF]) IN PAKISTAN December 2004

2 CURRENCY EQUIVALENTS Currency Unit Pakistan rupee/s (PRe/PRs) At Appraisal At Project Completion 31 August June 2002 PRe1.00 = $0.025 $0.017 $1.00 = PRs40.12 PRs58.00 ABBREVIATIONS ADB AOP EC FATA GDP IDA IMF MCH MDG MOE MOH MOPW MSU MTR NGO NSC NWFP PCC PCR PIHS PWP RRP RWSS SAP SAPPI SAPPII SMC SOE TMA TPV Asian Development Bank annual operational plan European Commission Federally-Administered Tribal Areas gross domestic product International Development Association International Monetary Fund maternal and child health millennium development goal Ministry of Education Ministry of Health Ministry of Population and Welfare Multi-donor Support Unit midterm review nongovernment organization national steering committee North-West Frontier Province project coordination committee project completion report Pakistan Integrated Household Survey Population Welfare Program report and recommendation of the President rural water supply and sanitation Social Action Program first Social Action Program (Sector) Project Second Social Action Program (Sector) Project school management committee statement of expenditure Tehsil municipal administration third-party validation

3 GLOSSARY Federally-Administered Tribal Areas Infant mortality rate Maternal mortality ratio Katchi class Tehsil Total fertility rate Union council A group of semi-autonomous tribal areas in the western part of the country, along the Afghanistan border, administered by the federal Government and internally governed by tribal laws. Number of deaths of infants under 1 year of age per 1,000 live births in a given year. Number of maternal deaths per 100,000 live births in a given year. A preprimary enrollment, similar to kindergarten. Subdistrict administrative unit. The average number of live children born per woman in her entire reproductive life, based on age-specific fertility rates. Subtehsil administrative unit, usually with a population of about 10,000 20,000 people. NOTES (i) (ii) The fiscal year (FY) of the Government and its agencies ends on 30 June. FY before a calendar year denotes the year in which the fiscal year ends, e.g., FY2005 ends on 30 June In this report, "$" refers to US dollars.

4 CONTENTS BASIC DATA MAP Page iv ix I. PROJECT DESCRIPTION 1 II. EVALUATION OF DESIGN AND IMPLEMENTATION 3 A. Relevance of Design and Formulation 3 B. Project Outputs 5 C. Project Costs 9 D. Disbursements 10 E. Project Schedule 10 F. Implementation Arrangements 10 G. Conditions and Covenants 11 H. Consultant Recruitment and Procurement 11 I. Performance of Consultants, Contractors, and Suppliers 11 J. Performance of the Borrower and the Executing Agency 12 K. Performance of the Asian Development Bank and Cofinanciers 12 III. EVALUATION OF PERFORMANCE 12 A. Relevance 12 B. Efficacy in Achievement of Purpose 12 C. Efficiency in Achievement of Outputs and Purpose 13 D. Preliminary Assessment of Sustainability 13 E. Environmental, Sociocultural, and Other Impacts 14 IV. OVERALL ASSESSMENT AND RECOMMENDATIONS 14 A. Overall Assessment 14 B. Lessons Learned 14 C. Recommendations 15 APPENDIXES 1. Project Framework Evaluation Methodology Targets and Outcome Socioeconomic Context Project Scope Project Outputs Costs and Financing Disbursement Implementation Schedule Implementation Issues Compliance with Major Loan Covenants Multi-Donor Support Unit Stakeholder Consultations Options for Social Sector Development 73

5 BASIC DATA A. Loan Identification 1. Country 2. Loan Number 3. Project Title 4. Borrower 5. Executing Agency 6. Amount of Loan 7. Project Completion Report Number B. Loan Data 1. Appraisal Date Started Date Completed 2. Loan Negotiations Date Started Date Completed 3. Date of Board Approval 4. Date of Loan Agreement 5. Date of Loan Effectiveness In Loan Agreement Actual 6. Closing Date In Loan Agreement Actual Number of Extensions 7. Terms of Loan Interest Rate (number of years) Maturity (number of years) Pakistan 1493-PAK(SF) Social Action Program (Sector) Project II Pakistan Planning and Development Division of the Ministry of Planning and Development SDR million PCR: PAK August September October October November January April April December June % per annum Disbursements a. Dates Initial Disbursement 25 August 1997 Effective Date 25 April 1997 Final Disbursement 12 December 2002 Original Closing Date 31 December 2000 Time Interval 64 months Time Interval 44 months

6 Category b. Amount ($ million) Original Allocation Last Revised Allocation FY1997 FY1998 FY1999 FY2000 FY2001 FY2002 Total a Federal Population Vertical Activities Foundations 0.70 Balochistan Education Health RWSS NWFP Education Health RWSS Punjab Education Health RWSS Sindh Education Health RWSS FATA Education Health RWSS SAP Support Activities Service Charge During Construction Total (in $) (in SDR) FATA = Federally Administered Tribal Areas, FY = Fiscal Year, NWFP = North-West Frontier Province, RWSS = Rural Water Supply & Sanitation, SAP = Social Action Program, SDR = special drawing rights. a SDR million ($ million) was canceled from the loan proceeds on 4 September v C. Project Data 1. Project Cost ($ million) Cost Appraisal Estimate Actual Foreign Exchange Cost 1, Local Currency Cost 5, ,319.7 Total 7, ,127.3

7 vi 2. Financing Plan ($ million) Appraisal Estimate Actualª Cost Foreign Local Total Foreign Local Total ADB-Financed IDA Netherlands DFID European Union Others Ongoing Projects Borrower-Financed 5, , , ,319.7 Total 1, , , , ,125.5 IDC Costs ADB-Financed Borrower-Financed Total 1, , , , ,127.3 = not available, ADB = Asian Development Bank, DFID = Department for International Development, IDA = International Development Assistance, IDC = interest during construction. ª For the period July 1998 to June Cost Breakdown by Project Component ($ million) Appraisal Estimate Actual Component Foreign Local Total Total Punjab , , ,953.9 Education , , ,532.0 Health RWSS Sindh , ,027.3 Education Health RWSS NWFP , Education Health RWSS Balochistan Education Health RWSS Federal Programs Federal Areas SAP Support Activities Service Charge During Project Life 1.8 Total Project Cost 1, , , ,127.3 NWFP = North-West Frontier Province, RWSS = Rural Water Supply and Sanitation, SAP = Social Action Program.

8 vii 4. Project Schedule Item Appraisal Estimate Actual Civil Works Date of First Award Sep 1996 Jan 1998 Completion of Work Jun 2000 Mar 2002 Policy and Institutional Reform Measures Start Jun 1996 Jan 1997 Completion Jun 2000 Jun 2002 Consulting Services Needs-Based Institutional Strengthening Start of Services Jan 1997 Mar 1998 Completion of Services Jun 2000 Jun 2002 Reporting, Monitoring, and Evaluating Start of Services Jun 1996 Mar 1998 Completion of Services Jun 2000 Jun 2001 Community Mobilization Start of Services Jun 1996 Jul 1999 Completion of Services Jun 2000 Jun 2000 Reorganization of SAP Sector Departments Start Jan 1997 Completion Jun 1998 Training Start Jan 1997 Mar 1998 Completion Sep 1999 Jun 2002 Procurement of Equipment and Materials Start Jan 1997 (World Bank) Completion Mar 2000 = not available, SAP = Social Action Program. 5. Project Performance Ratings Implementation Period Development Objectives Ratings Implementation Progress From 1 April to 31 December 1997 Satisfactory Satisfactory From 1 January to 31 December 1998 Satisfactory Satisfactory From 1 January to 31 December 1999 Satisfactory Satisfactory From 1 January to 31 December 2000 Satisfactory Satisfactory From 1 January to 31 December 2001 Satisfactory Satisfactory From 1 January to 31 December 2002 Satisfactory Satisfactory

9 viii D. Data on Asian Development Bank Missions Name of Mission Date No. of Persons No. of Person-Days Specialization of Members Fact-Finding Appraisal 13 Apr 7 May Aug 10 Sep a, b, c, d, e, f, g, h, i, a, c, d, f, g, h, i, j, k, l Inception May a, d, h, m Special Project Adm Aug n Review Sep d Special Project Adm Nov a Consultation Jan d Review May a Review 3 17 Jul 4 Aug d Review 4 27 Nov 10 Dec d Review 5 14 Apr 15 May c, d, h, m, o Special Project Adm Oct d Review 6 20 Jan 22 Feb d, h, i, m, o Consultation Apr b Consultation Jun d Midterm 18 Sep 17 Oct d, p Review 7 22 Jan 3 Feb d Review Jun d Review May d Project Completion Review Jan d, m Total = not available, Adm = administration. a a - senior project specialist/lead project administration specialist; b - manager; c - education specialist; d - health specialist/senior health specialist; e - project economist; f - senior programs officer; g - programs officer, Pakistan Resident Mission (PRM); h - project implementation officer, education (PRM); i - staff consultant (management and institutional development specialist, social development specialist, financing consultant); j - deputy director; k - counsel; l - young professional; m - control officer (PRM); n - project specialist; o - project implementation officer, rural water supply and sanitation (PRM); p - associate project analyst (headquarters). b The project completion review mission included V. de Wit, senior health specialist, T. Anwar, senior control officer, and A. Khan, disbursement analyst.

10

11 I. PROJECT DESCRIPTION 1. In 1992, the Government of Pakistan initiated the Social Action Program (SAP) to address the serious neglect of the social sector. The SAP aimed to accelerate the development of four basic social services: primary education, primary health care, population welfare, and rural water supply and sanitation (RWSS). The Asian Development Bank (ADB), as part of a development partner group led by the World Bank and including the Governments of the Netherlands and the United Kingdom, supported a 3-year time slice of the SAP (1993/ /96) through first Social Action Program (Sector) Project (SAPPI). 1 In view of the satisfactory performance of SAPPI and the lessons learned, the Government established a planning process headed by the chief minister s task force to delineate provincial and sector objectives and strategies, assess financial sustainability, and plan the second phase of the SAP. 2. Following a performance review of SAPPI and project preparatory technical assistance (TA), 2 ADB agreed to support Second Social Action Program (Sector) Project (SAPPII) 3 for a period of 4 years (1996/ /00). ADB s loan was approved on 28 November 1996 and became effective on 25 April The loan was closed on 30 June 2002 (1 year and 6 months later than scheduled). SAPPII loan of the World Bank, the European Commission, and the Netherlands Government was approved on 24 March 1998 and became effective and was closed on 28 May Within the overall 5-year plan and the SAP plan, SAPPII, like SAPPI, was driven by annual operational plans (AOPs) and reviews for each of the 27 subprograms covering Pakistan s basic social services, with external agencies financing a slice of expenditures for eligible programs. The project framework is in Appendix The Government s draft final report was received on 9 December The report is conceptually strong but has quantitative shortcomings. It rates SAPPII as having modest outcomes, poor policy reforms, likely sustainability, and modest institutional impact. ADB performance is rated as good, while the Borrower s and the World Bank s performances are rated as satisfactory, with qualifications. The Government s final report is still pending. 4. ADB fielded a joint project completion review mission 4 with the World Bank. The mission examined project performance in terms of inputs and outcomes compared to stated targets and how these may have been affected by project design, implementation, and environment. The mission collected input and output data with the help of a staff consultant and relied on the Pakistan Integrated Household Survey (PIHS) for coverage and impact assessment. Studies of the Multi-donor Support Unit (MSU) and other sources provided insights on many aspects. The mission applied the ADB grading system in terms of relevance, efficacy, efficiency, sustainability, and other impacts. The evaluation methodology, including a list of reference papers, is in Appendix At the time of appraisal, the rationale for ADB s support to Pakistan s SAP through SAPPII was threefold. First, Pakistan s human development status was very low, especially among the rural population, girls, and women, thereby constraining equitable and sustainable economic 1 ADB Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for Social Action Program (Sector) Project. Manila. 2 ADB Technical Assistance to Pakistan for Second Social Action Program Project. Manila. 3 ADB Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for Second Social Action Program (Sector) Project. Manila. 4 The Project Completion Review Mission comprised V. de Wit (sr. health specialist) and T. Anwar (sr. controller, Pakistan Resident Mission. The Mission was jointly carried out with the Department for International Development, European Commission, and the World Bank.

12 2 development. Second, the implementation performance and initial impact of SAPPI had been satisfactory. Third, the broad-based multisector program approach adopted jointly with the World Bank and other aid agencies was considered to be an effective way to continue policy dialogue, build institutional capacity, and increase the impact of other social sector projects. 6. The overall goal of the SAP was to raise Pakistan s human development status by improving basic social services education, health care, population welfare, and RWSS. SAPPII was to support the second phase of the SAP during the 1996/ /00 period. The targets and outcome are in Appendix SAPPII continued to support the multipronged approach of the first phase but reflected a shift in effort. Overall, SAPPII was to improve planning, implementation, and monitoring services, to increase project impact. The objectives were to (i) undertake activities to improve service delivery, from expanding access to improving quality of services; (ii) improve system design through policy reform, from macro-level reforms to improved district systems; (iii) increase fiscal effort and efficiency, from improving the overall fiscal effort to improving the financial flow and allocation efficiency; (iv) build planning and monitoring capacity; and (v) involve stakeholders, generating commitment from public representatives to local stakeholders and communities, as well as nongovernment organizations (NGOs) and the private sector. 8. The overall approach in each sector was to improve access, quality, management, and stakeholder participation in the sector. Compared to the SAP s first phase, the scope of the second phase was expanded. In education, the scope was expanded from primary education to elementary education including middle schools, and increased community ownership of schools. In health, it was expanded from primary care (services below hospital level) to primary health care (which includes first referral hospitals), while containing construction of new facilities. In population welfare, it included social marketing in the private sector, and decentralization of services. In RWSS, it remained focused on completing existing water schemes, with a moratorium on new schemes, and handing over schemes to communities. While the scope was expanded, the emphasis within this expanded scope was to be on consolidation, performance, decentralization, accountability, capacity building, and sustainability. 9. The first component of ADB s SAPPII supported the nationwide SAP covering the education and health and public health engineering sectors in the provinces of Balochistan, North-West Frontier Province (NWFP), Punjab, and Sindh and the Federally Administered Tribal Areas, as well as federal programs for education and health and population welfare. ADB supported 18 of 27 subprograms under the SAP. The World Bank and other aid agencies also supported other federal areas and the participatory development program. The second component of SAPPII financed SAP-supportive activities to strengthen planning and implementation capacities and reporting and monitoring functions. This component promoted specific studies, social mobilization, evaluation, and validation exercises. Separate grants and the MSU, respectively, financed and provided TA for the SAP At ADB appraisal, the total cost of the SAP for the period 1996/ /00 was estimated at $7.0 billion, of which the Government would finance $5.5 billion and aid agencies would contribute $785.0 million, while $815.0 million was anticipated from other aid-assisted projects under the SAP umbrella. Thus, $1.6 billion, or 23% of the total program cost (about half 5 ADB Technical Assistance to Pakistan for Multi-Donor Support Unit of the Social Action Program. Manila. ADB Technical Assistance Completion Report on Multi-donor Support Unit of the Social Action Program. Manila.

13 3 the incremental social sector expenditure), was to be financed by aid agencies, including 11% as direct program support. Of this amount, 63% was earmarked for education and 43% for Punjab. 11. The Planning and Development Division of the Ministry of Planning and Development was the Executing Agency, while federal line ministries, provincial departments, and federal area representatives were the implementing agencies. Project coordination was based in the Federal SAP Secretariat, and the SAP cells were established for provincial coordination in the Punjab Planning Board and the planning and development departments of the three other provinces. 12. The SAP process features were (i) 5-year plans providing the broad direction, targets, reforms, and investment plan, as proposed by the Government, in consultation with stakeholders, and approved by the National Economic Council; (ii) separate AOPs for each subprogram within the SAP framework, treated independently; (iii) multiple monitoring and evaluation instruments, including quarterly monitoring reports, third-party validation (TPV) surveys, PIHS, and special studies by the MSU or others; (iv) regular development partner reviews of plans and performance to determine the eligibility of subprograms and propose corrective actions, as needed; and (v) development partner financing of part of eligible expenditures, using high reimbursement rates for quality-enhancing expenditures. Reimbursement of sector expenditures was based on statements of expenditures. Imprest accounts were used for expenses of the Federal SAP Secretariat and provincial SAP cells, and for capacity building and monitoring and evaluation. 13. ADB's sector lending modality, adopted for SAPPI, was continued under SAPPII. The justifications for the sector lending approach were (i) a medium-term social sector development plan, including a financing plan; (ii) the program consisting of numerous subprojects that could not be planned in detail in advance; (iii) the Government and aid agencies preferring to pool resources, to avoid program fragmentation; (iv) ongoing policy dialogue and implementation; and (v) sector institutions capable of implementing the subprojects as agreed upon with the aid agencies. II. EVALUATION OF DESIGN AND IMPLEMENTAT ION A. Relevance of Design and Formulation 1. Design 14. SAPPII was highly relevant in the context of Pakistan s economic and social situation and to the declared priorities of the Government reflected in the 5-year plans. SAPPII was consistent with ADB s country assistance strategy, focusing on pro-poor social sector development and governance reforms, including potential benefits for women. It was also fully consistent with ADB s poverty reduction strategy, as stated in the medium-term strategic framework. At the Pakistan Development Forum 2003, the Government reconfirmed its commitment to improving the social sector, endorsed the millennium development goals (MDGs), and promised to substantiate these commitments with tangible results. 15. The development objectives were important and appropriate in addressing key constraints in the delivery of social services. They provided the right focus on basic social services, in particular for women and the poor, and a focus on improving efficiency and onservice delivery. The stated priorities in the education and health sectors were generally solid, but targets were optimistic, given trends. The crosscutting objectives were too general. In the water sector, coverage targets were unrealistic in view of the moratorium on new schemes.

14 4 16. The soundness of the design was examined in three ways: (i) approach taken; (ii) design of that approach; and (iii) mitigation of macrocondition risks. 17. In terms of overall approach, SAPPII followed the public sector model of SAPPI. The Government greatly expanded the network of public services to achieve access for all and wanted to continue operating public social services. Contracting NGOs and the private sector was not considered a viable option in view of limited government capacity with contracting, limited NGO capacity, and actual cost of services. At the same time, provincial line departments became comfortable with the SAP approach, identified clear priorities, and developed a routine of annual planning and budgeting. It was assumed that services and outcome would improve with better management and increased spending. However, planners failed to appreciate the serious governance problems. The increasing role of the private sector was also not incorporated. At project appraisal, ADB decided to opt out of funding NGOs, due to concerns regarding auditing and other issues. 18. Within the public sector model, three major design issues were identified during SAPPI evaluation: (i) complexity of design, (ii) delegation of authority, and (iii) lack of capacity building. First, the multisector and multiprovince approach was expected to have benefits of synergies, leverage, and mobilization of support. However, like SAPPI, SAPPII was complex in view of the large number of agencies, subprograms, and layers, as well as crosscutting reforms and sector objectives. Second, SAPPII had a stronger governance agenda than SAPPI but failed to properly analyze governance issues and formulate responses. SAPPII helped pave the way for devolution by raising public concern about social sector performance, making governance issues more transparent, improving accountability, and involving communities. 6 Third, as in SAPPI, capacity building was poorly planned. While the Government and external agencies made efforts in this field, capacity-building efforts were rather basic and poorly coordinated (Appendix 10). 19. For a project the size of SAPPII, risk mitigation should have been an important factor. Pakistan s public sector has a history of fragile political balances, weak institutions, feudal systems, and insufficient support for social services. During SAPPII, Pakistan experienced drought and political and economic crises, as well as accountability and austerity drives and fiscal restructuring, all of which resulted in a ban on staff member recruitment and in spending cuts on operations and development. Macroconditions eventually had a major impact on SAPPII performance. A more thorough risk analysis of political will, governance, and social dimensions could have helped make the design more robust. The socioeconomic context is in Appendix Formulation 20. Toward the end of SAPPI, officials were optimistic that things were moving in a positive direction, and the Government formed task forces and initiated workshops with a broad range of stakeholders to facilitate a participatory, bottom-up planning approach. The planning process correctly identified a range of issues to be addressed in the design of SAPPII but lacked depth and continuity and inadequately incorporated these recommendations in the project design. 6 Under the devolution in August 2001, the Governm ent devolved the primary responsibility for the education and health sectors to the district level, and the water and sanitation sector to the tehsil level.

15 5 21. ADB processed SAPPII 1 year ahead of the World Bank. 7 ADB staff and consultants prepared the project proposal in consultation with the Government and development partners. ADB s SAPPII objectives closely matched those of the development partners (Appendix 5). The differences did not create any problems during implementation but made project evaluation more difficult. Each year, the governments and aid agencies were to review SAPPII progress within the broad framework of SAPPII and make adjustments as necessary. 22. As a flagship undertaking of the Government and external agencies, SAPPII was continuously monitored and reformulated, with up to four review missions and 50 mission members covering all sectors and themes. The World Bank, as the leading agency, spent over $800,000 on SAP management in ADB s inputs were modest, with 1 2 staff participating in loan reviews. Review missions focused on making adjustments in plans and financing, as well as selected governance issues, but paid much less attention to capacity building, service delivery, and monitoring. ADB s midterm review (MTR) was delayed by 1 year, to synchronize with the World Bank s MTR. The joint MTR became an ambitious, drawn-out process covering 6 months. The Government prepared a plan for redirecting SAPPII. However, with the announcement of the devolution plans, the attention shifted to devolving social services. 23. In 2001, the World Bank reformulated SAPPII support. It had been compensating for the shortfall in funding from other agencies. The World Bank decided to continue supporting federal programs and coordinating SAPPII assistance, while ADB continued support for eligible subprograms. In a cabinet meeting on 11 May 2002, the Government decided to close the SAP on 30 June 2002, while committing to continue the SAP reform agenda. B. Project Outputs 1. Crosscutting a. Improve Implementation 24. Implementation was to be improved, with a focus on quality of services, by strengthening district management and enhancing staff performance through in-service training and supervision. District management capacity improved with the appointment of more qualified managers. SAPPII supported training of a large number of staff. Governance improvements reduced staff absenteeism. However, supplies only improved toward the end of SAPPII, and supervision improved during the accountability drive. Funds for supervision were lacking throughout most of SAPPII. The financial incentive of higher reimbursement levels for qualityrelated expenditures did not show much effect, perhaps because line departments did not receive reimbursements. District officers and field staff noticed only modest improvement of services. Evidently, the efforts at higher levels did not in a substantial way filter through, and districts and services were inadequately engaged. b. Improve System Design Through Policy Reform 25. The SAP promoted major policy reforms, such as forming school management committees, introducing contract appointments, and improving the pay and benefits packages of female staff, which were all implemented. The uniform water policy and moratorium on new schemes, initiated during SAPPI, were maintained. Major plans for integrating the ministries of 7 The World Bank s Second Social Action Program (Sector) Project loan was approved on 24 March 1998.

16 6 health and population welfare were changed to functional integration, except that village family planning workers were integrated with lady health workers. Progress was also made on meritbased recruitment and procurement. However, no major governance reforms accompanied the initiatives, which were therefore difficult to sustain. The SAP contributed to developing plans for devolving 12 sectors, including the social sectors. c. Increase Financing and Efficiency 26. SAPPII financial and efficiency performance was mixed. The SAP spending remained at 1.7% of gross domestic product (GDP), compared to a target of 2.5% (Table A7.1). In constant prices, spending on social services increased by 4.4% between 1996/97 and 2000/01. Education spending increased by more than 10.0%, health and population spending increased by 3.1%, and water and sanitation spending declined by 30.6%. The mix of spending deteriorated in the health sector, as spending on preventive health declined by 3.2%. The share of non-salary allocations increased but use reduced because of the accountability drive and late releases, subsequent cuts, and procedural uncertainties following devolution. Development spending, partly financed from salary savings, reduced fourfold during SAPPII. 27. During SAPPII, Pakistan experienced severe fiscal stress that required expenditure compression to stay within International Monetary Fund-guided limits on budget deficits. The SAP framework provided some protection (defense spending declined much more than education spending). However, the sector was already seriously underfunded by any standard, and the protection mainly applied to salaries. The International Monetary Fund provided an opening to increase social sector spending to 3% of GDP. However, the Government, despite its well-articulated priorities made at the Pakistan Development Forum, failed to support its stated commitment to human development in clear financial terms. The Government continued to give priority to economic self-sufficiency, including quick debt reduction within a very tight fiscal framework, leaving little space for pro-poor investments in the social sector. Perhaps this also reflected a lack of confidence in returns on social sector investments. As sector financing collapsed, competent managers could no longer be productive, and services deteriorated. d. Build Planning and Monitoring Capacity 28. Planning capacity generally improved and devolved during SAPPII. AOPs were regularly prepared and provided a clear set of objectives and strategic plan for the sector priorities. AOPs continued to be used after SAPPII. Considerable work was done to improve monitoring systems of the three sectors. A new education management information system was designed to monitor teacher and student attendance. The coverage of the health management information system was expanded. Monitoring in RWSS remained weak. However, management practices are not geared towards using data for making decisions. Regular supervision was only featured during the accountability period. Spot-checking of facilities proved useful in curbing staff absenteeism and identifying problems. The PIHS was carried out in , , and , providing a consistent data set to track impact indicators during SAPPII. Despite some shortcomings of the surveys, these were useful. TPV surveys were carried out four times from 1998 to 2001, to assess performance against selective governance indicators. e. Involve Stakeholders 29. Dissemination of the SAP objectives and strategies to a wider government and nongovernment audience was not a core program. The Canadian International Development Agency brought in a separate project to plan and implement a communication strategy.

17 7 However, this was a late initiative and didn t take off. It should be mainstreamed in future programs. SAPPII did make progress in the establishment of school management committees and village development associations (for water supply). MSU made some efforts to involve NGOs and the private sector. Fairly consistently throughout SAPPII, efforts were made to redress gender balance. Notable successes were the lifting of the ban on female staff toward the end of SAPPII, when the fiscal situation had improved; the increased number of rural women from Balochistan applying for training as paramedics; and an increased number of women in key management positions. While much more remains to be done in gender equity, these were important developments for the country. 2. Sector-specific a. Education 30. Pakistan s education sector needed to expand during SAPPII to cope with population growth and increased enrollment. The Government followed a modest expansion program. In 1997, the country had about 150,000 primary schools and 8,000 middle schools, which in 2002 increased to 165,000 primary schools and 11,500 middle schools. Urban schools and lower grades were often overcrowded, while rural village schools and higher grades remained underused, due to a shortage of teachers and high dropout rates. Hiring teachers was banned throughout most of SAPPII, and the ban was only lifted 2 months before its closing. This seriously affected school openings and instruction quality in rural areas. The provinces started to redistribute teachers. In Balochistan, for example, 150 out of 500 closed schools were made functional through the redistribution of teachers. Dropout rates remained high, mainly because of cost and children being unwilling to attend class. Parents were often not satisfied with the quality of instruction compared to costs. With the steady increase in private and NGO schools in rural areas, increasing numbers of parents opted for these schools. 31. Policy reforms emerged toward the end of SAPPII. The National Education Sector Reform Program provided a major boost to expanding the education sector. Several provinces have taken the initiative to develop innovative policies. For example, Sindh decided to hire only female teachers for primary schools, to make schools more accessible for girls. NWFP, after rationalization of posts, decided to hire 2,000 female teachers and retire surplus male teachers. After devolution, districts were made responsible for education planning. Non-salary allocations improved, but expenditures remained low, due to a ban on the procurement of durable goods. Financial constraints also affected the subsidy on textbooks for poor children and the annual national school census. Since SAPPII came to a close, subsidies have improved (e.g., textbooks for free; stipends for girls; and funds for the school management committee, for supplies and maintenance, based on the number of children in school). These are major improvements induced by SAPPII. A summary of education sector outputs is in Appendix 6. b. Health 32. The Government built a primary health care network in the 1980s that included basic health units, rural health centers, and tehsil and district hospitals. The SAP pursued a program of containing infrastructure, by improving quality of care through better staffing and supplies, and improving maternal and child health and family planning as well as communicable diseases control through national priority programs. About midway through SAPPII, the Government relaxed the ban on hiring staff and allowed the hiring of female paramedics based on sanctioned and vacant positions. The hiring of lady health workers after a development partner-imposed ban increased from about 45,000 to 70,000, thereby improving access to maternal and child

18 8 health and family planning in rural areas. Use of rural clinics increased, with the majority of those attending being female. However, this is still well below expected levels in view of population needs. Many prefer traveling to urban areas to attend private services believed to be of better quality. While exceptions exist, rural public health services on the whole have a poor profile due to the overall presentation and limited range of services and supplies. National programs showed some improvement in performance, notably for tuberculosis, but routine immunization services were constrained by frequent polio campaigns. 33. Governance problems continued to have a major impact on the performance of health services. One major problem is staff absenteeism in rural areas. Reasons for absenteeism are security problems, social isolation, poor housing, high transportation costs, lack of equipment and supplies, lack of income-earning potential and high opportunity costs, and supervisors rentseeking behavior. Staff member absenteeism was substantially reduced during the accountability drive, when the new Government came on board. However, such effort is not sustainable, unless incentives are changed. Devolution added transitional problems, including lack of clarity in the provincial-district relationship. District governments stopped reporting to provinces. A summary of health sector outputs is in Appendix 6. c. Population Welfare 34. The Population Welfare Program (PWP) operates its own network of family welfare centers and reproductive health services centers providing, respectively, primary and surgical family planning services with limited maternal and child care services. The PWP budget was kept at about 20% below projected levels. The PWP basically continued providing low-cost family planning services. Helped by high unmet demand and adequate supplies, the use of family planning services increased steadily. The contribution of the PWP to family planning coverage remained quite small, perhaps due to the limited number of outlets, which did not increase much during the SAPPII, as per government policy. In rural areas, about half of family planning users traveled for more than one hour to reach any family planning service, and only 11% of women were visited at home by a family planning worker. While private family planning services are expanding, the PWP remains the main source of family planning commodities in Pakistan. 35. Initially, the Government wanted to integrate the ministries of health and population welfare. This did not happen, except for community workers, and was replaced with functional integration, to provide reproductive health services in line with international commitments. However, cooperation between the two ministries remained poor during SAPPII. Several policy issues could not be resolved during SAPPII such as the pricing of family planning commodities (many of these are now being addressed). A monitoring system was put in place and appears to be operating well. A summary of population welfare sector outputs is in Appendix 6. d. Rural Water Supply and Sanitation 36. Based on the uniform policy, rural water supply schemes were to be handed over to communities. Providing safe drinking water in rural Pakistan is complex because of (i) scattered populations living in small hamlets; (ii) mostly saline ground water in Sindh, southern Punjab, and elsewhere; (iii) feudal system and related problems of rights and taxation; and (iv) limited social cohesion beyond the clan. Hence, the concept of a community-owned and -financed water supply is challenging, and adjustments need to be made for poor people living in inhospitable environments. According to the records, about half of schemes were handed over to communities. Local governments were provided training. However, work was handicapped due to inadequate funds for community mobilization and supervision, and a ban on the procurement

19 9 of pumps and other durable goods. About 30% of transferred schemes are reportedly functioning. In many schemes, people opted for going back to other (more polluted) sources of water. A major reason for this is high electricity charges. Some communities have switched to using diesel pumps, so that they can control pumping hours. The possibility of organizing water schemes through the local government and private sector should be further explored. Rural sanitation also improved during the SAPPII. While a part of the community mobilization program, rural sanitation probably mainly picked up because of private initiative. 37. On request of external agencies, the Government imposed a moratorium on new water schemes, to first finish incomplete schemes in the sector. Unfortunately, many old schemes were politically motivated, poorly designed, and mostly unwanted by communities. Several years into the SAP, many of the old schemes were stagnant. Following the drought, the shortage of water became acute. The Punjab government proposed to develop new low-cost schemes to improve access for the poor. However, SAPPII aid agencies did not agree. The Government bypassed the moratorium with several programs for rural infrastructure. Unfortunately, these programs, which were outside the SAP framework, did not follow the meritbased selection process adopted for SAPPII. A summary of RWSS outputs is in Appendix 6. C. Project Costs 38. The share of provincial governments in the divisible pool of federal taxes increased substantially in past years. Consequently, under revised federal-provincial financing arrangements, the provincial governments assumed greater responsibility from 1996/97, to allocate needed resources for the provincial SAP activities. Federal and provincial commitment was sought regarding the availability of adequate government funds (as the Government s share amounted to an increment of about $3 billion) and the timely release of budgeted funds. 39. The consortium of aid agencies provided direct SAPPII support. Of the original contributions, $26 million from ADB, $30 million from the World Bank, and $20 million equivalent from the European Commission was cancelled in mid-2001 and used for drought relief. The Department for International Development supported SAPPII with TA and participation in missions. The Canadian International Development Agency, the Norwegian Government, and the United Nations Children s Fund also provided significant TA. 40. SAPPII was to support a project period of 3 years and 8 months, from 1996/97 to 1999/2000, but was extended to 5 years and 4 months. The World Bank loan was approved on 24 March 1998, but retroactive financing was agreed from 1 January 1997, therefore covering 5 years and 6 months. The total expenditure incurred during SAPPII for the period 1997/ /02 based on Government data collected by ADB, was $5,126 million. Compared to the original plan of 21.4% development partner share to SAPPII, the actual development partner share was 15.7%, including 7.4% through other projects under the SAP umbrella and 8.3% from SAPPII aid agencies. The low development partner share was caused by economic sanctions, ineligible subprograms, and other problems. Given financial constraints, the Government made efforts to protect the SAP sector. However, such financial constraints often resulted in cuts and delayed releases that affected service efficiency. 41. In terms of ADB financing, the education and RWSS sectors used 10% and 6% less than their relative shares, respectively (Table A7.2). In terms of ADB disbursement across agencies, the relative shares were more or less as estimated. Balochistan, with a relative share of 6% against a target of 10%, was not eligible for some period. The Federally Administered Tribal Areas received more than their original share (Table A7.3).

20 10 D. Disbursement 42. Disbursement was slower than anticipated, due to ineligibility of subprograms, mainly resulting from inadequate allocations and underspending in relation to delayed budget releases and procurement problems. Procurement increased during the last year of SAPPII, after the fiscal situation improved, and newly created district governments managed reasonable use of funds in the first year of devolution. The World Bank loan was disbursed more quickly, as it compensated for a shortfall of bilateral funding and also financed major procurement for federal programs. For the last year of SAPPII, ADB was to finance a larger share of provincial activities but could even not fully contribute its own share, due to a paucity of funds. Imprest funds moved very slowly, except for expenses of the Federal SAP Secretariat and the provincial SAP cells. Imprest accounts were well maintained, useful, and closed in time (Appendix 8). E. Project Schedule 43. SAPPII became effective on 25 April 1997 and closed on 30 June 2002, compared with the original closing date of 31 December On request of the Government, SAPPII was extended three times, up to 31 January 2001, 30 June 2001, and 30 June No payment was retroactive. The effective project period was 5 years and 4 months (Appendix 9). F. Implementation Arrangements 44. Planning and Monitoring. The national and provincial committees met twice each year and provided general guidance. The national SAP committee functioned like a review committee. The provincial committees showed more active leadership and ownership. The Federal SAP Secretariat, provincial SAP cells, and the SAP executing agencies acted like coordinating units, as these were lacking the staff to perform data collection, aggregation, analysis, and reporting. Activities were driven by an overall policy framework and expenditure plan for each subprogram, which were translated into AOPs that were reviewed for consideration of development partner funding. Each AOP had a detailed implementation matrix and budget. AOPs provided the action plan for the reform program and were technically and financially balanced. Line departments regularly submitted quarterly monitoring reports. However, the high number of separate projects substantially complicated matters. The process of reimbursement through statement of expenditures was initially satisfactory but became more cumbersome with the introduction of a multitiered reconciliation procedure. A summary of implementation issues is in Appendix Audit. An unfortunate outcome of SAPP II was that it became a victim of the audit procedures built into the expenditure reimbursement mechanism. The hasty circulation of poorly drafted audit reports to aid agencies raised unwarranted alarms. These audit report issues made administrators extremely cautious in incurring expenditures for procurement and services. 46. TPV as a concept was appreciated, and the provinces particularly believed doing this had benefits. As shown in Appendix 3, Table A3.3, TPV shows no substantial improvement in terms of compliance with merit-based recruitment and procurement, staff member absenteeism, and site selection. Officials pointed out the weaknesses in terms of the methodology of TPV, including inconsistent sampling and poorly trained auditors. The Government also has doubts about the findings of TPV in later years, when procurement procedures and staff member presence was thought to have improved under the accountability drive. However, no alternative evidence of the performance in these areas is available.

21 11 G. Conditions and Covenants 47. Of a total of 55 loan covenants, 40 were complied with by the Government, 7 were partly complied with, and 7 were not complied with. One covenant was not applicable (Appendix 11). Covenants for the insurance of facilities and environmental standards were considered complied with based on government standards but could not be verified. Notably, the Government was regular in terms of audit reports. Partial compliance was generally the result of delays and underperformance. Noncompliance concerned unmet financial targets, low non-salary allocations, delayed releases of funds, banned recruitment of staff, delayed hiring of consultants, inadequate consultation of ADB in these matters, and late submission of the Government s project completion report. H. Consultant Recruitment and Procurement 48. ADB financed consulting services for the project preparatory TA (footnote 2), the Federal SAP Secretariat, and the MSU (Appendix 12, footnote 2). National consultants for the Federal SAP Secretariat were selected through Government procedures using advertisement in national newspapers. The selection procedure was found to be in order but slow. ADB supported consulting services for community participation, monitoring and evaluation, management information systems, and procurement. The consultants found working in the Government s system difficult (e.g., permission to proceed on field visits was difficult to obtain, and travel allowances were low). The contracts were terminated following the downsizing of the Federal SAP Secretariat. The consultants were technically competent but achieved little. ADB also hired a staff consultant to carry out a financial analysis of the SAP in preparation of the MTR. 49. Procurement in SAPPII included procurement of supplies for schools and health services. Construction was mostly limited to upgrading and expanding schools and facilities. Line departments used standard government procedures related to local competitive bidding. Procurement issues included (i) procedural rules that conflicted, (ii) red tape and cash flow problems, and (iii) improper procurement and misappropriation. World Bank procurement procedures were different from those of the Government, but also had to be followed in order to be eligible for aid. Under the project agreement with the World Bank, the Government hired a private firm to monitor compliance and process no-objection certificates. Procurement in the Government s system itself involved multiple steps, with little room for price adjustment. Combined with cash flow constraints, the result often was that procurement was rushed at the end of the fiscal year, causing slippage of procedures and audit queries. 50. Based on TPV, an average of 44% of procurement did not comply with agreed procedures (Appendix 3). This was usually due to slippage in procedures, not misappropriation. This observation reflects the prevailing audit culture in Pakistan, whereby auditing is done by poorly trained junior staff. As the SAP required full audit and reconciliation of all expenditures, the result was a large number of audit observations requiring a lot of time to sort out. ADB was refunded for all questionable expenditures. I. Performance of Consultants, Contractors, and Suppliers 51. ADB s support for consulting services was mainly through the related TA for MSU support of SAPPII. The Federal SAP Secretariat also contracted consultants, with ADB support, to help improve the capacity of the Federal SAP Secretariat. The Secretariat s performance was not satisfactory. It was technically competent but had limited capacity and was not able to deliver in the executing agency environment.

22 12 J. Performance of the Borrower and the Executing Agency 52. SAPPII received more attention than ordinary projects from the federal Government, but not the attention it needed for a program of this size and complexity. The provincial planning and development cells contributed little to the reforms initiated in SAPPII, which were mainly driven by line ministries and aid agencies. The cells also lacked analytical, coordination and project management capacity. Coordination focused on the SAP missions, while the SAP implementation was left to line departments. These line departments lacked consistent leadership, technical capacity, and sufficient funds to implement the ambitious agenda, all of which caused tensions. Staff capacity in line departments at the time of SAPPII appraisal declined during implementation and after devolution. Coordination among departments was also weak, which meant that subprograms operated independently and did not follow a comprehensive strategy under an integrated approach. K. Performance of the Asian Development Bank and Cofinanciers 53. To supervise SAPPII, the joint aid agencies would typically undertake four supervision missions per year. The World Bank coordinated these missions. Several aid agencies brought their own teams to cover all sectors, resulting in large missions of sometimes over 50 people. Despite these teams, consultation of district staff and field visits by missions were actually inadequate. ADB usually provided one or two people to these missions, trying to cover finance and governance, four sectors, and crosscutting issues. 8 Overall, ADB s contribution was consistent but limited in terms of range of expertise. Overall, the SAP missions lacked consistent inputs from governance and public-private partnership experts. The Government evaluated ADB s performance as good. SAPPII coordination by the World Bank office in Islamabad was generally good. World Bank leadership adequately consulted cofinanciers, but inadequately shared decision making, project work, and policy dialogue (e.g., queries regarding the appropriateness of the water sector policy). Overall, development partner coordination was better during the SAP than before and after the SAP. III. EVALUATION OF PERFORMANCE A. Relevance 54. SAPPII design was considered partly relevant. SAPPII was appropriate in terms of development objectives and policy environment. SAPPII, like SAPPI, followed the public sector delivery approach, reflecting the political realities at that time. However, it inadequately planned for governance and capacity-building challenges and risk mitigation. The targets were considered overambitious, especially in the water sector, where the adoption of the moratorium on new schemes constrained expansion of drinking water coverage. B. Efficacy in Achievement of Purpose 55. SAPPII is considered less efficacious. The public education sector did poorly. Improvement in literacy also benefited from private sector development. The health sector did well in terms of infant mortality but less well on immunization indicators (perhaps due to the polio campaign). The population welfare sector did well in fertility reduction and contraceptive prevalence. The water sector also showed some improvement, which, at least in part, should be 8 The health specialist was the SAP coordinator during most of SAPPII, working with the World Bank mission leader.

23 13 attributed to private sector initiatives. A discussion of efficacy and project targets and achievements is in Appendix 3. A discussion of data constraints is in Appendix Several factors contributed to the disappointing SAPPII performance. SAPPII coincided with a period of severe drought and political and economic crisis, resulting in serious fiscal constraints and austerity and accountability drives that affected the availability and quality of services. Increasing poverty further reduced demand for education, and many parents opted for low cost private schools. Lack of commitment undermined performance in the social sector. Although impact is disappointing, and may continue to be modest after devolution, SAPPII helped initiate a change expected to result in better impact in the medium-term. A summary of stakeholder consultations is in Appendix 13. C. Efficiency in Achievement of Outputs and Purpose 57. SAPPII is considered less efficient. SAPPII targeted basic services providing interventions of proven cost-effectiveness using standard strategies applied around the world. The expanded scope added higher level services that were more costly but these made the system more comprehensive and should have contributed to improving quality and demand. SAPPII financing was less than what was required to run the public system in an efficient manner and make proper use of available staff. Non-salary spending proportionally declined during SAPPII and was used in a less efficient manner, due to late releases and rushed procurement. The Government s austerity and accountability drive also curtailed procurement but helped reduce staff absenteeism and misuse of funds. Considerable effort was made to improve the quality of instruction and quality-related spending. Use of services and staff improved in the health and population welfare sectors. Cost recovery for rural water supply improved, with more households paying for water. Some policies, such as the containment of new health facilities and handing over schools to communities, probably helped improve efficiency. Others, such as the ban on new staff, may have increased inefficiencies. Many schools remained closed, and health facilities were without female staff until later in SAPPII, when the ban on female staff was partly lifted. While the handing over of water schemes to communities reduced maintenance costs for the Government, the subsequent closing of many schemes added to the economic costs. Unfortunately, more radical retrenchment of poorly performing staff was not done. 58. The Federal SAP Secretariat lacked resources to manage the SAP properly. The cost of the resulting micromanagement of external agencies was substantially higher. ADB probably underinvested in managing the SAP, even though ADB was not in a leading role. D. Preliminary Assessment of Sustainability 59. SAPPII results are likely to be sustainable. The pillars of the poverty reduction strategy, fiscal restructuring, devolution, and sector-specific reforms provide a strong basis for social sector development. The ban on hiring staff was relaxed for hiring female staff. The SAP sector, along with other sectors, was delegated to district governments for education, health and population welfare, and tehsil municipal administrations for RWSS, respectively. The Government supports the concept of a minimum level of social sector spending to operate the current infrastructure in an effective, equitable, and efficient manner. Much is now in the hands of local governments, which still need to go through the SAP learning process. They complained that funds are mostly tied up for salaries. Even then, some improvement in non-salary spending was noted. Development spending has also shown a major increase. This is more worrying, as existing infrastructure may still not be operational. Hence, in most districts spending needs to be

24 14 rationalized. An even more urgent need for strengthening the capacity of local governments exists now than did during SAPPII, in particular for tehsil municipal administrations. ADB is supporting this new development with a series of program loans and projects. E. Environmental, Sociocultural, and Other Impacts 60. SAPPII had substantial other impacts. First, SAPPII had a strong focus on gender equity. The Government has internalized this concern and is now actively promoting it. Second, SAPPII introduced support for school management committees and strongly supported the uniform policy for water schemes. As such, SAPPII brought about a positive change toward community ownership. Third, SAPPII provided some exposure for the Government in working with NGOs, in particular through the Participatory Development Program, a component of SAPPII not supported by ADB. While the overall attitude toward NGO participation changed little during SAPPII, the current trend is positive. Fourth, the Government procurement and expenditure system benefited. Financial management improved substantially during SAPPII. IV. OVERALL ASSESSMENT AND RECOMMENDATIONS A. Overall Assessment 61. SAPPII was rated partly successful. Its design was innovative, bringing together federal and provincial governments and aid agencies in a joint effort to uplift the social sector. Given the positive experiences in SAPPI, the overall design was considered relevant. The subsequent economic crisis and lack of funding would have been hard to predict. However, risk factors were inadequately analyzed and assessed, and the SAP targets were overambitious, given the political and financial setting. SAPPII was clearly constrained by the fiscal situation, resulting in limited non-salary allocations and funds for capacity building related to the quality of services. As was shown after SAPPII, improvement in social sector financing directly translated into better performance. The scale and procedures of SAPPII made it less efficient. SAPPII monitoring remained weak and driven by development partners, but periodic evaluation through the PIHS and other studies was successfully conducted. SAPPII did well in terms of sustainability and other impacts. SAPPII succeeded in uplifting the entire social sector, where smaller projects usually fall short. While core governance improvement may not be directly attributed to SAPPII, SAPPII helped pave the way for sustainable social sector development. The impact was modest and will probably continue to be affected by the process of devolution. Options for social sector development are indicated in Appendix 14. B. Lessons Learned 62. First, SAPPII showed that the performance of Pakistan s public social sector strongly depends on political will, governance, and funding, which are all nontechnical issues. Uplifting the sector therefore requires financial restructuring and profound institutional reforms in governance and civil service, which are partly beyond the scope of the social sector. 63. Second, SAPPII provides an example of the difficulties associated with implementing complex multisector and multiprovince projects. Keeping in view the limited implementation capacity of governments and services, future programs and projects should be limited in terms of scope and implementing agencies, supported by adequate leadership and capacity building, and allowed adequate time for implementation and consolidation.

25 Third, SAPPII demonstrated that these complex projects need careful design, risk analysis, and preparation and need to have strong capacity to adjust as necessary during implementation. The aid agencies, despite intensive coordination, failed to provide adequate technical inputs for institutional development and monitoring. ADB needs to carefully analyze management requirements that lead to good project performance. C. Recommendations 65. The first specific recommendation, given data constraints and ongoing reforms, is that a more general social sector analysis should be carried out, rather than a project performance audit report. Second, the Government s capacity, including sector monitoring, needs to be strengthened, as does ADB s local engagement in the sector (e.g., national sector specialists in the resident mission). Third, the Government should submit the final project completion report and settle any outstanding audit queries. No covenants need to be changed or put up for a waiver. ADB is already implementing the Decentralization Support Program and Sindh Devolved Social Services Program 9 and is preparing new loans for devolved social services that will provide further support for the initiatives under the SAP. Lessons learned have been incorporated in the new program design. 66. The first general recommendation is that social sector development, with widely scattered activities, should be handled at sub-national levels. Second, ADB should consider another modality of support to avoid micromanagement, keeping in mind that capacity for performance-based reimbursement is limited. Second, sound economic, political, and social analysis should be conducted to help contain risks in the project environment. Third, arrangements for organization, capacities of executing and implementing agencies, and arrangements for monitoring need upfront action. Fourth, plans, policies, and targets should be realistic, incorporating various risks and assumptions, and adjusted over time. 9 ADB Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for Decentralization Support Program. Manila; ADB Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for the Sindh Devolved Social Services Program. Manila.

26 16 Appendix 1 PROJECT FRAMEWORK (Summary of Original Project Frameworks for Four Sectors ) Design Summary Performance Indicators/Targets Monitoring Mechanisms Assumptions and Risk Purpose In accordance with the Eight 5-Year Plan, address the relative neglect of the social sector. - Increased political commitment to improving the quality of the public social sector - Increase the share of public social sector expenditure as percentage of gross domestic product - Increased public satisfaction with public services - Dialogue with concerned political representatives and ministries - Ninth 5-Year Plan - Annual household survey - Political and macroeconomic stability is maintained Goal Improve the social status of the population in general, and women and children in particular. - Increase literacy from 38% to 44% - Reduce population growth rate from 2.7% to 2.4% - Reduce total fertility rate from 5.7 to Reduce maternal mortality ratio from 300 to 200 per 100,000 - Reduce infant mortality rate from 95 to 80 deaths per 1,000 - Vital statistics - Annual household survey - No famine due to drought or floods or other major disasters Objectives Improve the coverage of basic social services, particularly those relating to girls education, maternal and child health care, and rural communicable diseases control. - Increase gross enrolment rate from 73% to 85% - Increase girls participation from 38% to 46% - Increase immunization from 55% to 85% - Increase contraceptive prevalence rate from 22% to 35% - Increase rural clean water supply from 73% to 93% - Increase sanitation coverage from 37% to 54% - Education and health management information systems - Annual household survey - Public leaders accept girls education and family planning - Adequate supplies can be made available - Staff absenteeism can be curbed - Female staff can be increased in rural areas Continued on next page

27 Appendix 1 17 Design Summary Performance Indicators/Targets Monitoring Mechanisms Assumptions and Risk Components 1. Strengthen the involvement of stakeholders, in particular at provincial, district, and community levels. 2. Increase social sector financing and improve allocative efficiency. 3. Strengthen the system design through policy reform. - Local leaders oriented in the SAP - Social action boards set up in 60% of districts, and 30% function adequately - 30% of districts have capable and active NGOs to assist in community organization - 5% of social sector services managed by the private sector and/or NGOs - Education and health foundations operational - Allocations increased as planned - Non-salary recurrent expenditures increased - Increased utilization - Allocative efficiency improved - Reorganization policy approved - Powers effectively decentralized - Guidelines for management available - Policy for staff incentives in place - Policy to reduce staff absenteeism in QMR QMR QMR and special reports Politicians and administrators will support social action boards, and policy for private management of public services approved Government can hold on to fiscal commitments, and policy for cost recovery and staff incentives approved Administrators accept decentralized powers and support junior staff in using these powers. 4. Improve implementation of services in terms of resources, management, access, and quality of care. 5. Build capacity for planning and monitoring the SAP. place - Policy for cost recovery in place - School infrastructure expanded based on mapping and/or merit - All schools have staff - Female staff in 10% of management positions - Staff absenteeism reduced by 20% - 60% basic health units provide maternal and child health/family planning services - 60% of villages have lady health workers - 20% of rural water schemes are community owned - Planning Cells are restructured in functional units and carry out most functions earlier done by MSU - Management information system operational in at least half of the districts - QMR submitted on time - Survey carried out on time - District management teams established in all districts and adequately functioning in 30% - Planning and monitoring activities satisfactory in 90% of districts QMR School census Health management information system Special report QMR Merit-based site selection Adequate female staff can be made available in rural areas. Planning cells are provided with adequate staff, accommodations, and communication facilities. Continued on next page

28 18 Appendix 1 Design Summary Performance Indicators/Targets Monitoring Mechanisms Assumptions and Risk Activities/Inputs 1. Involve stakeholders: - provide orientation - NGOs organize community interest groups - contract services to the private sector - Number of workshops for local leaders, managers, communities - Financial and technical support provided to NGOs - Technical support provided to health foundations QMR Politicians are willing to attend and get involved. 2. Increase health sector financing and allocative efficiency: - continue frequent dialogue between planning, finance, and line departments - carry out studies of allocation efficiency, cost recovery, and staff incentives - Budgets prepared and released on time - Studies initiated QMR Adequate NGOs of sufficient competency and stability can be identified for further training, and competent managers can be identified and selected. 3. Change the health system design through policy reform: - prepare guidelines - carry out training and follow-up to make devolved administrative and financial powers effective - Guidelines prepared QMR Fiscal situation 4. Improve implementation by updating HRD plan, training and recruiting staff, and improving support systems. - Plan updated - No ban on recruitment - Number of staff trained - Availability of supplies HRD plan and Study reports Proposed adjustments are acceptable to decision makers. 5. Build up planning and monitoring capacity: - restructure planning cells - increase staff, train staff, provide incentives, use fellowships - provide technical assistance through MSU or otherwise - Number of planning staff recruited - Number of training sessions conducted QMR Government is committed to capacity building and monitoring. HRD = human resources development, NGO = nongovernment organization, QMR = quarterly monitoring report, SAP = social action program.

29 Appendix 2 19 EVALUATION METHODOLOGY A. Method 1. A project completion report (PCR) is to provide a comprehensive evaluation of a project; examining inputs and outcome, including policy reforms and institutional strengthening, and interpreting these data in terms of internal and external factors. For a program the size and complexity of Second Social Action Program (Sector) Project (SAPPII), this is a challenging task. While the basic goals, objectives, and scope were quite clear, the Asian Development Bank (ADB) supported 17 of 27 subprograms, across four sectors and provinces and the Federally Administered Tribal Areas and federal programs. The Project had multiple crosscutting and sector-specific subobjectives. Each subprogram was a project of its own. Accordingly, within the data and resource constraints, the PCR can only provide a basic analysis of SAPPII performance. 2. ADB carried out a project completion review mission in January This was done jointly with the World Bank for the preparation of its Implementation Completion Report (ICR). The Project Completion Review/ICR Mission, a small team of four people, focused on getting qualitative feedback on the performance and problems of SAPPII and trying to identify successes and failures of SAPPII. The joint Mission visited the four provinces and held extensive consultations with about 200 district and field staff in three of four provinces (excluding Balochistan). The discussions were analyzed to identify common themes and views. Additional views of stakeholders, including external agencies and nongovernment agencies, were obtained during four workshops for devolving social services. 3. ADB s SAPPII preceded that of the World Bank by 1 year and had a somewhat different scope and targets (Appendix 5). This complicated the task of evaluation. In addition, the data requirements of the project implementation report are quite different. As reimbursement of government expenditures was shared between ADB and other agencies, and because other agencies supported the sector, ADB funds cannot be tied to specific impacts. 4. The Mission had to make special arrangements to collect input and output data, as the Social Action Program (SAP) cells were no longer existing and the Federal SAP Secretariat and the SAP cells lacked systematic SAP documentation. For the period after June 2002, information became even more difficult to obtain, as the information flow to the province was interrupted after devolution. The Mission hired a consultant, who visited the provinces three times but found retrieving a full set of required output data difficult. The collected data were aggregated and submitted to the Government for checking. The best possible data are reported in this PCR, but these do not fully match those of the Government s PCR, which was only received on 24 December The Mission also did an extensive literature review. The World Bank, as the lead agency, managed documentation and communication with the federal Government. ADB had limited capacity in the field to access and analyze reports, and the multi-donor support unit (MSU) was playing a key role for ADB by bridging this gap to the extent possible. The Project Completion Review Mission reviewed key documents, which are listed in this appendix. In addition, the Mission reviewed annual plans, quarterly monitoring reports, audit reports, provincial plans for capacity building, and numerous other reports, including MSU reports.

30 20 Appendix 2 6. In terms of coverage and impact data, considerable variation and different interpretations of key indicators were noted, such as for clean drinking water. One possibility is that the Pakistan Integrated Household Survey (PIHS) improved its methodology over the years. Another problem is that the baseline data in the report and recommendation of the President (RRP) do not fully match the available data set from the PIHS, and the RRP targets are for 1999/00, not SAPPII targets, as per RRP, also do not match the Government s targets. So, rather than using the RRP targets, the PCR focused on examining trends in indicators over the project period based on PIHS, which is a more or less consistent data set. The Government disagreed with some findings of the PIHS , in particular the finding that boys enrollment had not increased. However, experts generally agree that there was no substantial increase in boys enrollment. 7. The project performance audit report of the First Social Action Program (Sector) Project pointed out the importance of evaluating the counterfactual scenario, given the political and economic crisis during SAPPII (i.e., what would have happened with the social sector if SAPPII had not been there). This can be approached in several ways. One way is trend analysis of specific indicators (assuming that investments directly translate into measurable trends). Another way is to compare the performance of a basket of indicators that are highly dependent on government services (e.g., immunization and public school enrolment) with those more dependent on secular trends (e.g., malnutrition and literacy). While the SAP helped advance social sector reforms, data analysis suggests that the SAP impact was limited in the short term. This is no surprise, given the low levels of investment and implementation problems. However, more qualitative analysis suggests that where basic conditions were met (leadership, decision making authority, funds, and capacity), social sector transformation could occur, although often temporarily. The in-depth social sector analysis by the Operations Evaluation Department is expected to analyze this further. 8. It is important to consider the multiple impact of external events for the evaluation of SAPPII performance. For example, the economic crisis led to a fiscal crisis, which led to a ban on recruitment of new teachers, thereby keeping new schools closed. It also led to increased rural poverty, so that many parents could not afford to send their children to school. The population welfare program and vertical health programs received strong development partner and government support. Relative government and development partner priorities within the SAP probably led to some level of projection for these programs. 9. The Mission s project rating is shown in Table A2. Details of the rating are available on request. Table A2: Summary of Project Rating Criterion Assessment Rating (0 10) Weight (%) Weighted Rating Relevance Partly Relevant Efficacy Less Efficacious Efficiency Less Efficient Sustainability Likely Other Impacts Substantial Overall Rating Partly Successful Highly successful: Overall weighted average is 8.0 or more. Successful: Overall weighted average is in the range of 6.0 to 8.0. Partly successful: Overall weighted average is in the range of 4.0 to 6.0. Unsuccessful: Overall weighted average is less than 4.0. Source: ADB staff.

31 Appendix 2 21 B. List of Reference Papers for Second Social Action Program Evaluation 1. Project- Related Documents Asian Development Bank (ADB) Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for the Social Action Program (Sector) Project. Manila Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Pakistan for the Second Social Action Program (Sector) Project. Manila a. Project Completion Report on the Social Action Program (Sector) Project in Pakistan. Manila b. Draft Summary Project Implementation Plan for the Proposed Second Social Action Program (Sector) Project. Manila c. Back-to-Office Report of the Special Administration Mission, Second Social Action Program (Sector) Project. Manila a. Project Performance Audit Report on the Social Action Program (Sector) Project. Manila b. Aide Memoire of the Midterm Review Mission, Social Action Program (Sector) Project. Manila a. Pakistan Economic Update July 2002 June Manila b. Aide Memoire of the Project Completion Review Mission, Social Action Program (Sector) Project. Manila. Planning Commission, Government of Pakistan Planning Document for Social Action Program (Sector) Project, ). Islamabad. Planning and Development Division, Government of Pakistan Concept and Strategy Paper on Institutional Strengthening of Monitoring and Evaluation Capacity for Social Action Program (Sector) Project. Islamabad Project Completion Report on the Social Action Program (Sector) Project in Pakistan. Islamabad Restructuring of Second Social Action Program (Sector) Project. Islamabad Draft Implementation Completion Report on the Second Social Action Program (Sector) Project. Islamabad. World Bank. 1998a. Project Appraisal Report on the Second Social Action Program (Sector) Project. Islamabad..1998b. Implementation Completion Report on the Social Action Program (Sector) Project. Islamabad Implementation Completion Report on the Social Action Program (Sector) Project. Islamabad. Multi-Donor Support Unit Discussion Paper on the Role of Multi-Donor Support Unit. Islamabad Multi-Donor Support Unit Progress Report January June Islamabad Role of Multi-Donor Support Unit in Capacity Building of the Government for Improved Social Sector Services Under the Devolved System, a Proposal for Technical Assistance. Islamabad. 2. Social Sector Statistics Government of Pakistan, Federal Bureau of Statistics Round 2: Pakistan Integrated Household Survey. Islamabad Round 4: Pakistan Integrated Household Survey. Islamabad. Government of Balochistan Health Statistics of Balochistan1996. Quetta.

32 22 Appendix 2 Government of Sindh Sindh Education Profile 2001, Government Schools and Colleges. Sindh. Multi-Donor Support Unit Analysis of Provincial Social Action Program Expenditures, FY93 to FY99. Islamabad a. North-West Frontier Province Transfer of Rural Water Supply Schemes. North- West Frontier Province b. Public Expenditure on Social Sector in Pakistan. Islamabad Profiles of Education, Health and Water Sectors for All Provincial Districts. Islamabad. United Nations Educational, Scientific and Cultural Organization Education For All 2000 Assessment: Country Reports Pakistan. Islamabad.

33 Appendix 3 23 TARGETS AND OUTCOME Table A3.1: Second Social Action Program (Sector) Project Key Indicators, Targets, and Achievements Indicator A. Education 1. Primary School Enrollment Rate 2. Middle School Enrollment Rate 3. Girls Primary School Participation Rate 4. Female Literacy Rate (10 Years of Age and Over) B. Health 1. Infant Mortality Rate (per 1,000 Live Births) 2. Full Child Immunization Rate 3. Tetanus Toxoid Coverage Rate (Two Doses) for Women of Reproductive Age C. Population Welfare 1. Population Growth Rate 2. Total Fertility Rate 3. Contraceptive Prevalence Rate D. Rural Water Supply And Sanitation 1. Proportion of Households Having Access to Improved Water Supply a 2. Proportion of Households with Toilets 3. Proportion of Children Under 5 Years of Age Suffering from Diarrhea in Past 30 Days (%) Target Change (3) (1) (1) Change (ratio) a Including tapped and pumped water but not closed dug wells. The share of motor pumped water saw the largest increase, from 10% to 17% in 6 years, and is often drawn directly from a river or lake. Sources: Federal Bureau of Statistics Pakistan Integrated Household Survey Islamabad: Government of Pakistan. Federal Bureau of Statistics Pakistan Integrated Household Survey Islamabad: Government of Pakistan. National Institute of Population Studies Pakistan Fertility and Family Planning Survey 1996/97. Islamabad: Government of Pakistan. National Institute of Population Studies Pakistan Reproductive Health and Fertility Survey 2000/01. Islamabad: Government of Pakistan

34 24 Appendix 3 literacy rate gross enro girls partici infant mort immunizaticontracept baseline baseline Figure 38A3.1: Second 73 Social 38Action targets 22 targets Program 44 (Sector) 85 Project 46 Impact outcome 35 outcome literacy rate % literacy sanitation rate coveragegross enro girls partici gross infant enrollment mortrate immunizaticontracept class 1-5 baseline targets outcome rural safe drinking w ater coverage girls participation rate total fertility rate infant mortality rate contraceptive prevalence rate immunization rate Source: National Institute of Population Studies Pakistan Integrated Household Survey 2000/01. Islamabad: Government of Pakistan. 1. Primary Education. The education system consists of primary education (grades 1 5), middle school (grades 6 8, O level), secondary school (grades 9 10, A level), higher secondary education (grades 11 12, or college), and tertiary education (university). Most schools also have a katchi (kindergarten) class, for children below primary school age. The Government aimed to achieve 93% gross primary enrollment and 43% adult literacy by 1999, compared to Second Social Action Program (Sector) Project s (SAPPII s) 89% and 31%, respectively. By 2002, the Government s targets were universal primary enrollment and 70% adult literacy. In comparison, it achieved 71% gross primary enrollment and 45% adult literacy in and 72% primary enrollment and 45% literacy in At the most, SAPPII helped keep enrollment up with population growth, which is evident from the increased number of children attending school. Disaggregated by public and private sectors, public school enrollment stagnated, while private school enrollment increased. Anecdotal evidence suggests that more well-to-do children switched to private schools, while public schools absorbed more poor children. Girls participation did not improve either. Some of the reasons for these poor education trends are believed to be (i) increased rural poverty, with many poor children being taken out of school; (ii) continued ban on hiring teachers; (iii) high rates of teacher absenteeism, related to misbehavior and lack of payment of salaries, increases in transport costs, and travel problems for women; (iv) other governance problems; and (v) inadequate funds to properly supply, operate, and increase the number of schools. These areas will need to be addressed if public education is to be improved. With devolution, opportunities and risks in doing so have increased. 2. Health. Immunization services improved, but these services are separately funded, managed, and implemented and are therefore not good proxy indicators of general health services. Infant mortality declined substantially but is again dependent on many other interventions apart from health services. However, linking this with increased use of health services and quality proxy indicators, such as the supply of drugs, it is likely that public health services contributed to a reduction of infant mortality. Further impact of the sector on key indicators will depend on efforts to make female staff available and improve the quality of staff.

35 Appendix Population Welfare. Under the Eighth 5-Year Plan (1993/ /98), the Population Welfare Program aimed to reduce the population growth rate from 3.0% to 2.7% and the total fertility from 5.9 to 5.4 births per woman. The Population Welfare Program also aimed to increase the contraceptive prevalence rate from 14% to 24%. In comparison with these government targets, the sector has done well. 4. Rural Water Supply and Sanitation. Key indicators have definition problems. Based on the Pakistan Integrated Household Survey, access to clean water increased by 10% and sanitation increased by 11% during SAPPII. However, anecdotal evidence from departments and the field do not support this, and this possibly includes nonfunctioning water schemes and latrines. 5. After devolution, the Public Health Engineering Department and the Local Government and Rural Development Department merged with the Civil Works Department and became the Services and Works Department. In most provinces, old and foreign-funded schemes were retained at the provincial level. About 15% of the larger existing schemes with the Public Health Engineering Department are nonoperational. Major reasons are poor location; technical issues; low budget allocations for operation and maintenance, with emphasis on expansion instead of rehabilitation; inadequate community involvement; and poor cost recovery mechanisms. Sindh s policy of financing operation and maintenance costs for rural water supply by levying a cess on agricultural activities was difficult to implement. Balochistan has initiated the process of privatizing water tariff collection with some success. 6. Financing. Expenditure targets were not met. This has several dimensions. First, funds to operate services are inadequate for the large numbers of health facilities. Second, targets set at appraisal in terms of share of gross domestic product could not be maintained because of the fiscal crisis and were dropped during project implementation. Third, even minimal allocations agreed during review missions could often not be met. Fourth, whenever allocations were agreed, releases were often much less, seriously delayed, or both. Table A3.2: Trends in Social Action Project Expenditure Item Year 1992/ / / / / / / / / / / 03 SAP as Target Percentage Actual of GDP Government Target SAP Share Actual GDP = gross domestic product, SAP = Social Action Program. Source: World Bank appraisal reports for First Social Action Program Project and Second Social Action Program Project (target) and Federal SAP Secretariat (actual). 7. Participation. ADB deliberately stayed out of supporting the Participatory Development Program, due to concerns about government commitment and accounting and auditing procedures of grant-in aid. The program proved to be a nonviable mechanism to foster nongovernment organization (NGO) participation. Major issues were its centralized nature, lack of government ownership, mixed reputation of NGOs and associated government mistrust, and elaborate selection process. Complaints were raised about transparency from NGOs that were not selected, and the Auditor General raised audit issues. 8. Gender. As shown in Figure A3.3, Figure A3.4, and Figure A3.7, not much improvement occurred in terms of narrowing the gender gap. Some improvement was seen in terms of the girls enrollment rate and participation ratio in primary schools. In households that had boys in school, the tendency was to also put girls in school. In comparison, enrollment of boys also

36 26 Appendix 3 stagnated at 30% of the population living below the poverty line. However, security and social constraints remained major obstacles for girls enrollment in Pakistan, which could be addressed by making village schools operational. Health services statistics do not show a major discrepancy among boys and girls (e.g., for immunization). However, one could assume that major differences still exist in health outcomes, due to health and nutrition-related customs. 9. Institutional Capacity. Substantial capacity building was achieved in terms of overall enhanced capacity for policy dialogue, planning, management, financial management, and procurement. However, actual institutional capacity remained limited. One particular problem was the frequent transfer of staff. Another problem was the ban on hiring staff, including much needed female teachers and paramedics. 10. Governance. Overall district management did not show a dramatic improvement in governance during SAPPII, and any improvement may also be due to secular trends in the population. Selective SAPPII priorities did not show any improvement, perhaps because these were not based on major restructuring to change responsibilities, accountability, and incentives in the sector. Table A3.3: Compliance with Key Governance Indicators of Third-Party Validation Exercises, Rounds I IV (% of visited sites in compliance) Survey Round Recruitment (%) Procurement (%) Absenteeism (%) Site Selection (%) Round I, March Round II, March Round III, March Round IV, March Source: Government of Pakistan Third Party Validation Report, Round IV. Islamabad.

37 Appendix 3 27 Figure A3.2: Pakistan Urban Literacy Rate by Age Cohort, Sex, and Year Source: Pakistan Integrated Household Surveys Figure A3.3: Pakistan Rural Literacy Rate by Age Cohort, Sex, and Year Source: Pakistan Integrated Household Surveys

38 28 Appendix 3 Figure A3.4: Pakistan Net Primary Enrollment Rate by Sex and Year Source: Pakistan Integrated Household Surveys Figure A3.5: Pakistan Girls Net Primary Education Participation Rate by Place of Residence and Year Source: Pakistan Integrated Household Surveys

39 Appendix 3 29 Figure A3.6: Pakistan Government Schools Share in Primary Education by Place of Residence, Province, and Year Source: Pakistan Integrated Household Surveys Figure A3.7: Punjab Government Schools Share of Total Gross Primary Enrollment by Place of Residence, Sex, and Income Group Source: Pakistan Integrated Household Surveys