Public Disclosure Authorized. Public Disclosure Authorized. Public Disclosure Authorized. Public Disclosure Authorized

Transcription

1 38270 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized

2 THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT The Development of Monitoring and Evaluation Capacities to Improve Government Performance in Uganda ECD Working Paper Series 10 Arild O. Hauge Growing concerns about poor value for money from Uganda s public services are focusing attention on the need for better understanding of development effectiveness what works, what does not, in which contexts, and why. This issue is being addressed by efforts to strengthen Uganda s monitoring and evaluation (M&E) systems. October 2003 The World Bank

3 Copyright 2003 Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) Telephone: Facsimile: Evaluation Capacity Development (ECD) helps build sound governance in countries improving transparency and building a performance culture within governments to support better management and policymaking, including the budget process through support for the creation or strengthening of national/sectoral monitoring and evaluation systems. OED aims to identify and develop good-practice approaches in countries, and to share the growing body of experience with ECD efforts. The OED Working Paper series disseminates the findings of work in progress to encourage the exchange of ideas about enhancing development effectiveness through evaluation. An objective of the series is to get the findings out quickly, even if the presentations are somewhat informal. The findings, interpretations, opinions, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the Operations Evaluation Department or any other unit of the World Bank, its Executive Directors, or the countries they represent.

4 CONTENTS Foreword Abbreviations Executive Summary iv v vi 1. Introduction 1 2. Main Features of Uganda s M&E Systems 2 3. What Contribution Is M&E Making to National Development? 6 4. What Challenges Remain? 9 5. Lessons Learned from Uganda s M&E Experience Conclusions 16 Annex A: Poverty Monitoring and Evaluation Strategy: Priority Poverty and Sector Indicators 17 Annex B: Summary of OED M&E Diagnosis Findings, Annex C: Annex D: Annex E: OED s Contribution to M&E Capacity Development (ECD) in Uganda 19 M&E in Uganda PRSO3 Draft Policy Matrix (as of April 2003) 25 Select Findings from the Study of M&E in Uganda s Health, Education, and Water Sectors 28 References 31

5 FOREWORD As part of its mandate, the World Bank s Operations Evaluation Department (OED) has responsibility for encouraging and assisting developing member countries to build effective monitoring and evaluation (M&E) capacities and systems. OED plays a catalytic role, including support to a small number of countries with the intention that they will develop good-practice M&E approaches that can be adapted by other countries to suit their particular circumstances. In this context OED has provided active support for Uganda since The Ugandan government is committed to improving the development effectiveness of its activities by means such as the achievement of a stronger performance orientation in public management. Its efforts in this area, and those of many of its development partners, have been framed in the context of the Poverty Reduction Strategy. Uganda has pioneered access to the debt writeoff associated with the Highly Indebted Poor Country initiative, and it has also pioneered the Poverty Reduction Support Operation (formerly the Poverty Reduction Support Credit), a type of programmatic lending. This report provides a follow-up to a diagnosis of Uganda s M&E systems and functions, which was conducted in 2000 and published in January This latest report attempts to identify the contribution that M&E is making to national development and examine the remaining challenges to institutionalising M&E in a sustainable manner. A number of lessons from Uganda s efforts in this area are also discussed. A draft version of the paper was presented to OED and Bank staff in Washington, D.C., on June 1 2, 2003, and to members of the Uganda Evaluation Association in Kampala on June 19, We are grateful for comments received from a number of people, including Ismail Barugahara, Poul Engeborg-Pederson, Satu Kahkonen, Margaret Kakande, Sukai Prom- Jackson, and Peter Ssentongo. The strong support of Satu Kahkonen, of the Bank s Africa Region, has been most valuable. In addition, the generous financial support provided by the Ministry of Foreign Affairs of the Norwegian Government is gratefully acknowledged. The report was prepared by an OED consultant, Arild O. Hauge. The task manager was Keith Mackay (OED). Osvaldo Feinstein Manager Partnerships and Knowledge Programs, OEDPK iv

6 ABBREVIATIONS AfrEA African Evaluation Association AIDS Acquired Immunodeficiency syndrome BFP Budget framework paper CAPEP Capacity and Performance Enhancement Project CDF Comprehensive development framework CSO Civil society organization DEI Directorate of Ethics and Integrity DFID Department for International Development (United Kingdom) DWD Directorate of Water Development (of the Ministry of Water, Lands and Environment ECD (Monitoring and) evaluation capacity development GDP Gross domestic product GoU Government of Uganda HIPC Heavily indebted poor country HIV Human immunodeficiency virus HPPG Harmonized participatory planning guide IFMIS Integrated financial management system IPDET International Program for Development Evaluation Training JLO Justice, Law and Order LGDP Local Government Development Programme LGFC Local Government Finance Commission LGs Local governments M&E Monitoring and evaluation MDGs Millennium Development Goals MIS Management information system MOES Ministry of Education and Sports MOFPED Ministry of Finance, Planning and Economic Development MOH MOLG MOPS MOWLE MTEF NGO NSDS ODA OED OPM PAF PEAP PETS PMES PRSC PRSO PRSP ROM SWAP SWG TA UBOS UEA UPPAP USh Ministry of Health Ministry of Local Government Ministry of Public Service Ministry of Water, Lands and Environment Medium term expenditure framework Non-governmental organization National service delivery survey Official development assistance Operations Evaluation Department Office of the Prime Minister Poverty Action Fund Poverty Eradication Action Plan Public expenditure tracking survey Poverty Monitoring and Evaluation Strategy Poverty Reduction Support Credit Poverty Reduction Support Operation Poverty Reduction Strategy Paper Results oriented management Sector-wide approach Sector working group technical assistance Uganda Bureau of Statistics Uganda Evaluation Association Uganda Participatory Poverty Assessment Process Uganda shilling v

7 EXECUTIVE SUMMARY There is a growing awareness that Uganda s progress with poverty reduction does not match the rate of increase in budget resources for the social sectors; there are indications of poor effectiveness and value-for-money in public service delivery. These concerns are focusing attention on the priority for a better understanding of development effectiveness what works, what does not, in which contexts, and why. Uganda s monitoring and evaluation (M&E) systems have the potential to provide this much-needed understanding. A large volume of information is currently produced by different M&E systems and practices that have been established in Uganda emanating from domestic and external concerns with accountability, governance, public sector reform, and financial management. But the disparate information flows can create confusion about goals and complicate policy analysis. Parallel systems lead to duplication and waste in data collection, and the M&E workload diverts attention away from productive service delivery. The M&E systems also sometimes reward managers for good paperwork rather than contribution to poverty eradication. Although arguably documenting compliance with nominal accountability rules, Uganda s M&E systems need to increase the emphasis on the results that follow from public action. M&E data are often of poor quality, with missing, inaccurate, or outdated information. The distinction between observed reality and what is hoped-for is blurred. In this environment, donors when each brings a different set of rules and requirements have been part of the M&E problem rather than part of its solution. The Uganda experience helps to dispel the notion that increased M&E, in and of its own, will lead to improved results orientation. What Uganda needs is not more, but better, M&E. This priority is now being addressed by the Poverty Reduction Support Credit (PRSC) recently renamed the Poverty Reduction Support Operation (PRSO) process, which has become the vehicle for a systems approach to M&E development in Uganda. By marshalling donors toward budget support, it is inducing a greater reliance on the government s own M&E systems and practices. Equally important, the PRSCs/PRSOs have explicitly identified a series of actions to address four major M&E issues that remain. First, whereas M&E systems produce fairly accurate measures of expenditures on the one hand and poverty on the other, there are fewer systematic data on the intervening levels: that is, outputs, the targeting effectiveness of government service delivery, and outcomes. Second, M&E is not geared toward understanding causality and attribution between the stages of development change. Third, incentives are tied to compliance with reporting requirements, rather than the underlying performance revealed by M&E. Fourth, the requirements that accompany M&E arrangements for different national and donor funding schemes result in a complex and formidable burden of guideline familiarization, inspection activity, and data collection and reporting. It can be argued that the growing attention to M&E is already helping to bring greater rationality to public finances and development management by providing a more evidencebased foundation for policy, budgeting, and operational management. M&E-related initiatives are helping instill an orientation toward the government s poverty reduction goals (as vi

8 expressed in its Poverty Eradication Action Plan its poverty reduction strategy. They are also helping to reduce the government s transaction costs of donor coordination; and they are increasingly allowing civil society and Parliament to assume a more meaningful role in public affairs. A key lesson from Uganda is that technical constraints, such as skills, manuals, and logistical resources, are not the predominant bottleneck to capacity. M&E will flourish where there is strong demand from policymakers and advisers for M&E information, where managers seek an empirical foundation for decision-making, where the practice of M&E follows as a consequence of incentives embedded in public service systems, where rewards and sanctions are guided by achievement of results, and where managers collectively perceive a selfinterest in adopting tools of continuous assessment and learning. Making further headway with M&E will necessarily require coordination and operational action on several fronts. With an M&E working group established under the PRSC/PRSO, coordination is now being addressed, paving the way for the remaining M&E challenges to be addressed systematically. vii

9

10 1. INTRODUCTION Donor and developing country public management policymakers, analysts and practitioners alike, are closely watching Uganda. Over the last decade, the country has achieved a rate of growth unparalleled in the Sub-Saharan region. Uganda has pioneered access to the Heavily Indebted Poor Country (HIPC) initiative debt write-off and the Poverty Reduction Support Operation (PRSO) modality (PRSOs involve grants, and they recently replaced Poverty Reduction Support Credits (PRSCs), the earlier form of programmatic support for Uganda s poverty reduction strategy. Substantial and multifaceted reforms in public sector management have been undertaken. Parallel to the unfolding reform in Uganda, increased international attention is being given to the role that monitoring and evaluation (M&E) plays within public management results orientation in general and development effectiveness in particular. Moreover, in Uganda M&E has been identified as a priority area of cross-cutting public sector reform within the policy matrix of the PRSO series of operations that have been planned and implemented since early This paper reflects lessons learned from work undertaken as part of Operations and Evaluation Department (OED) support for national M&E capacity development within the PRSO context. The finalization of the paper has greatly benefited from discussions held with, and comments provided by, World Bank staff and Ugandan stakeholders. The objective of the paper is to take stock of the M&E initiatives and capacity development efforts that have taken place in Uganda, with a particular reference to those lessons that emerge as relevant to the wider development community. The analysis takes the functioning of M&E within the Uganda public sector, rather than the needs and requirements of the external donor community, as its main point of departure. The objective of M&E is seen as the improvement of the performance and effectiveness of government and its public service delivery system. M&E relates to more than the technical aspects of data collection: it ultimately derives its value not from the availability of accurate data per se, but from the way in which they are used to improve decision-making. M&E functions when it supplies managers with a flow of reliable information and analysis about what works and what doesn t as a basis for policy recommendations, optimising resource allocation, refining institutional strategies, changing the mix of public service products or suppliers, assessing performance of staff, or underpinning accountability requirements. Because M&E addresses the performance of governments in responding to their citizens needs and priorities, it is ultimately an issue of good public governance. 1

11 2. MAIN FEATURES OF UGANDA S M&E SYSTEMS If the output of M&E capacity development is more comprehensive production and availability of M&E information and findings, Uganda has made substantial progress. A number of new M&E systems and practices that enable production of data and analysis have been established within the last decade. These have often arisen from concerns associated with, for example, accountability (value for money, malpractice, and audit follow-up), governance (participation, transparency), public sector management (policy implementation, performance management), or financial management (budget execution, expenditure quality). Information is available, albeit to a variable degree of regularity and accuracy, that covers all the main bases of inputs, activities, outputs, outcomes, and poverty impact. The volume of data and analysis to support government operations has certainly improved over the period since Uganda initiated its Poverty Eradication Action Plan (PEAP), although there are still shortcomings in terms of scope, prioritisation, and data quality and use. PEAP/PRSP: The PEAP represents the point of departure for national M&E efforts. It is the national development planning framework that guides sector and district planning, as well as the budget process. It also serves as Uganda s Poverty Reduction Strategy Paper (PRSP). The PEAP goals and objectives 1 (annex A) have not been directly derived from, but are essentially congruent with, the Millennium Development Goals (MDGs). Its overarching goal is to reduce the headcount incidence of poverty in Uganda from 44 percent in 1997 to 10 percent by Every two years the government of Uganda (GoU) prepares the Poverty Status Report, outlining progress in reducing poverty and forming the basis for revision of the PEAP/PRSP. In addition, annual PRSP Progress Reports are produced, on the basis of the GoU s Background to the Budget document. Poverty surveys and data collection: Uganda has a good demographic and household poverty dataset, which allows monitoring of national progress against the MDGs and other development indicators (see table 2.1). The major source of data is the regular schedule of surveys undertaken by the Uganda Bureau of Statistics (UBOS). Household income and expenditure surveys are conducted every two years, demographic and health surveys every five years, and censuses every decade. Moreover, UBOS maintains a district resource endowment profiles database, comprising information about, for example, topographical characteristics, natural resources, and land use in the districts. The Uganda Participatory Poverty Assessment Process (UPPAP), implemented by Oxfam, provides a qualitative dimension to analysis of poverty through bringing the perspective of the poor into planning. 2 Client satisfaction with public services has been surveyed through a national service delivery survey, which was integrated within UBOS s responsibilities in However, although UBOS has received capacity development assistance from a half-dozen donors and is able to produce a relatively good series of statistics, indications are that much of its data are not effectively utilized. 1. As refined by the Ministry of Finance, Planning and Economic Development (MOFPED) in the Poverty Monitoring and Evaluation Strategy, Kampala, A first UPPAP was undertaken in and a second in

12 Table 2.1. Schedule of Poverty Data Collection Survey Scope/coverage Frequency National Household Survey Income and expenditures, all districts 2-yearly Participatory Poverty Assessment Qualitative aspects of well-being, focus groups 2-yearly National Service Delivery Survey Satisfaction with public service delivery 2-yearly Demographic and Health Survey Health and demographics, all districts 5-yearly Census Demographics, population survey 10-yearly Poverty Monitoring and Evaluation Strategy: Building on an informal Poverty Monitoring Network that has met since 2000 and comprises ministries, UBOS, Makerere University, nongovernmental organizations (NGOs), and donors, a Poverty Monitoring and Evaluation Strategy (PMES) has been formulated. The PMES is the product of a national effort, and it represents an overarching plan for M&E within the context of Uganda s PEAP/PRSP. The PMES identifies a set of 33 priority indicators for implementation of the PEAP, for which a systematic effort has been made to establish a baseline and target (annex A). The PMES also addresses institutional responsibilities for tracking and review of poverty. Moreover, it seeks to draw the linkage between poverty indicators on the one hand and the planning and operations of ministries and their service delivery chains on the other. Poverty Monitoring and Analysis Unit: This unit in the Ministry of Finance, Planning and Economic Development (MOFPED) has played a particularly important role in the development of Uganda s national M&E arrangements by virtue of its responsibility for the biennial Poverty Status Report, the establishment of the Poverty Monitoring Network, the formulation of the PMES, and its coordination of the UPPAP and other participatory poverty research in the country. The unit functions as an integral part of the ministry, although its staff is not part of the permanent government establishment. It has received support from various donors for specific tasks, and the United Kingdom s Department for International Development has provided core funding support. Budget planning and expenditure controls: Uganda s medium-term expenditure framework (MTEF) represents the interface between PEAP goals and operational planning and management. The budget is built through preparing sectoral and district level budget framework papers (BFPs) that spell out respective goals, achievements, and expenditure plans. A key element of reform has been the attempt to instill a greater degree of performance orientation by way of sectors being requested to relate their plans to the outputs and outcomes that follow from their spending. Training in results-oriented management has also pursued clarity of ministerial and departmental goals and expectations. 3 A further component of reform has been tighter control of spending. Unexpected revenue shortfalls are matched by expenditure reductions. The introduction of a commitment control system prevents buildup of arrears. Other measures to control spending include the 2002 Public Finance Accountability Act which provides that supplementary expenditure must be approved by Parliament before the release of funds. To reduce deviations between budgeted and actual annual expenditures, the GoU publishes a biannual Budget Performance Report. 3. See, for example, Ministry of Public Service (2002). 3

13 Poverty Action Fund: To fund poverty priorities within the PEAP, the Poverty Action Fund (PAF) was created in 1998, comprising a combination of savings under HIPC debt write-off, general and earmarked donor budget support, and the GoU s own resources. It provides a mechanism for ring-fencing expenditures that directly reduce poverty primary health, primary education, water and sanitation, agricultural extension, and rural roads. Within Uganda s system of decentralized management, local governments prepare activity-based annual and quarterly workplans and report separately for each of the 23 individual PAF and other conditional grant schemes. The PAF monitoring committee, which reviews policies and operations, meets quarterly and includes donors, line ministries, and NGOs. Coordinating role of sector working groups: Uganda s sector working groups bring together key central ministries, the respective technical agencies, NGOs, and the donor community, and they function as coordinating bodies for sector-wide strategy planning, budget preparation, stakeholder consultations, and liaison. Their biannual meetings include a comprehensive assessment of sector-wide progress. Most GoU ministries have research and documentation departments with skills and facilities for fairly advanced data collection. Several ministries have established their own management information systems (MISs). Following the decentralization of service delivery, their focus is on reporting and compiling data on sectoral PAF programs for which local authorities have responsibility. Civil society s monitoring: Civil society organizations (CSOs) are active participants in the national budget process, including the sector working groups. The Uganda Debt Network coordinates civil society monitoring of PAF activity in the districts. An independent NGO often critical of government and the donor community, it is also piloting a community-based M&E system, which mobilizes the public to verify expenditures and service delivery. A client scorecard system, to be piloted in the health sector, is currently under preparation. A particularly noteworthy public monitoring practice in Uganda is the posting of notices of funding allocations, which has been made mandatory for all PAF workplans and funds (MOFPED 2001), forcing transparency and accountability into administrators dialogue with clients or end users. This policy was introduced after the results of the first (1996) public expenditure tracking survey (PETS) showed a significant degree of leakage of government spending in the education sector. 4 Evaluations, studies and reviews: Uganda s government ministries, research institutions, and donor organizations are continuously undertaking a number of ad hoc poverty studies and reviews. At the sectoral level such studies and reviews are increasingly being undertaken under the auspices of the sector working groups. The Economic Policy and Research Centre and the Makerere Institute of Social Research, in particular, have strong capacities for research and evaluation. With support from the World Bank, the first PETS was conducted in the education sector in Since then expenditure tracking studies have been undertaken in the health, education, and water and sanitation sectors. However, the planning and conduct of such studies is not always well coordinated, and the PMES envisages a consolidated process of agenda setting for ad hoc poverty research and reviews. 4 See, for example, Reinikka and Svensson (2002). 4

14 Accountability and transparency: Uganda has a well-developed infrastructure of accountability institutions at the central, sector, and local government levels (Table 2.2). A national anticorruption strategy was announced in 2001 (Directorate of Ethics and Integrity 2001). The plan aims at building capacity for coordination, planning, and monitoring anticorruption efforts. In this connection GoU is reviewing all anticorruption laws to consolidate them into a single comprehensive legislation. The 2002 Leadership Code Act requires that senior government officials (ministers, presidential advisers, permanent secretaries, directors, and senior project managers and coordinators) declare their income, assets, and liabilities. Accountability institutions are partially funded by the PAF, under which 5 percent of resources have been allocated to monitoring and accountability. Nevertheless, and in spite of also being augmented by selective external assistance efforts, the accountability institutions suffer from continuing resource and staffing constraints. Table 2.2 Inspection and Audit Agencies Inspection Audit Central Office of the Prime Minister (OPM) Directorate of Ethics and Integrity Inspectorate General of Government MOFPED Treasury Inspectorate Public Accounts Committee of Parliament Office of the Auditor General MOFPED Office of Accounts and Internal Audit External donor audits Ministry of Local Government Inspectorate Sector Sector ministry inspectorates; Sector ministry internal audit District District inspectorates Local public accounts committees District internal audit External support for M&E skills and training: There have been many M&E capacity development efforts attached to individual donor-funded projects. More generic M&E capacity development support has also been provided to individual ministries and agencies. Examples include UBOS, the Auditor General, and various departments of Makerere University. These bodies have received external assistance from a range of donors, including Denmark, Norway, the United Kingdom, the United States, the World Bank, the UNDP, the UNPF, and the UNCF. With support from the World Bank s OED, a number of senior Ugandan civil servants have participated in international M&E seminars and training courses. With support from OED, about two dozen officials have taken part in OED conferences, the International Program for Development Evaluation Training (IPDET) at Carleton University, Canada, and the two conferences convened by the African Evaluation Association (AfrEA). Skills gained and networking through these events were instrumental in the establishment of the Uganda Evaluation Association (UEA) in Within the PRSO context, a series of M&E seminars, workshops, and consultations has been held with senior officials from Uganda s central and sector ministries, as well as CSOs and donors. Building on OED diagnostic guidance material, 5 donor analytical work and M&E advocacy under the PRSO have been captured by two separate studies: an initial M&E diagnosis in (Hauge 2001) 6 and a more detailed study of M&E in the health, education, and water sectors in 2002 (Hauge and others 2002). Although both these studies have presented a critique of Uganda s national M&E arrangements, their presentation was well received by government officials. An overview of OED s involvement in M&E capacity development in Uganda is presented in annex C. 5

15 3. WHAT CONTRIBUTION IS M&E MAKING TO NATIONAL DEVELOPMENT? In terms of the desired outcome of M&E capacity development 7 that M&E is effectively used to improve the quality of decision-making some progress is being made in Uganda, although it is inherently difficult to identify clear evidence of this. Attention to M&E, even if sometimes only arising as an implicit concern for example, in the context of broader public management reforms is one of the factors bringing greater rationality and discipline to Uganda s public finances and development management. M&E is providing the GoU with a more evidence-based foundation for policy formulation, budgeting, and operational management. The increased production and availability of information concerning national poverty eradication efforts is also allowing civil society and Parliament to assume a more meaningful role in public affairs. Improved economic management: The M&E that has accompanied Uganda s budget reforms has helped rationalization of public finances and achievement of sustained macroeconomic stability. First it has introduced a planning horizon that extends beyond individual projects and financial years that is, it is linked to the medium- and longer-term national development goals of the PEAP. Second, through building on a hard revenue constraint and expenditure monitoring, sector planning has been infused with realism and discipline, leading to relatively low levels of inflation and interest rates. Third, the establishment of the PAF has made it possible to increase the relative priority given to activities that have a direct bearing on poverty. The share of PAF expenditures within the national budget has risen sharply, from 18 percent to 37 percent between and , resulting in a major reorientation of national expenditure. Participation in public decision-making: The process of national decision-making in Uganda, in particular at the central government level, is generally characterized by relative transparency and openness. For instance, a number of measures to keep the public informed and involved throughout the budget cycle have been introduced (Gariyo 2000, MOFPED 2000). All key documents are publicly available, often at the draft stage. In addition, since 2001 a vigorous effort has been mounted to increase the effective participation of the legislature in the budget process. The 2001 Budget Act allows for the cabinet and the legislature to receive BFPs early enough to provide detailed comments before the budget is finalized. A result of opening the budget process and the greater availability of data on government operations has been that civil society increasingly plays a role in both the planning and review of national- and sector-level projects and programs. A wide cross-section of CSOs takes part in sector working group meetings that prepare BFPs. Decline in transactional costs of donor liaison: Both the value and the number of stand-alone, donor-supported technical assistance projects in Uganda have recently declined. Between 1998 and 2002 the number of health, education, and water projects has been reduced by more than half, from 111 to 54, while the value of annual disbursements has declined from nearly US$150 million to approximately US$25 million. The planning and management of projects that remain are increasingly being undertaken with reference to the PEAP framework and the respective sector-wide approaches. An increasing share of external assistance to Uganda, currently about 60 percent, is provided as budget support. A growing number of other donors, including Ireland, 6

16 Germany, the Netherlands, the United Kingdom, and the African Development Bank, provide parallel financing to the PRSO. With the shift toward programmatic assistance, alignment with national objectives is ensured and the transacational costs of liaison reduced and the GoU is mindful that donors would only substitute their own M&E systems for those of the government once they have faith that national arrangements maintain meaningful standards of accountability and fiduciary control. Despite the move to budget support, there are still 42 donors who in total have 524 active projects and who have entered into 825 separate agreements with the GoU. It is clear that even the remaining ad hoc, off-budget donor projects place a severe M&E burden on the GoU officials involved. Cost-effectiveness is still elusive: Although substantial gains in poverty reduction have been made for example, the proportion of people living below the national absolute poverty line decreased from 56 percent to 35 percent between and progress does not appear to match the rate of increase in budget resources for the social sectors. 8 It is not evident that the observed poverty reduction can be linked to any improvement in the quality of public sector management. There are indications of continuing poor effectiveness and value for money in public service delivery. For example, although the total flow of resources to the water sector tripled between and , the annual number of new water points dropped by 8 percent. Moreover, between 1995 and 2000 there was a reversal in infant and under-five mortality as well as immunization coverage. M&E as a possible constraint to effectiveness: A concern with M&E may translate into a call for expansion of M&E activity. However, the key challenge faced by Uganda is not more, but better, M&E. Although many useful M&E instruments and practices have been introduced, the M&E arrangements in place in Uganda have also in some ways constrained effectiveness in making progress with the PEAP: Disparate M&E information flows create confusion about goals and complicate policy assessment and budget analysis. Parallel sets of M&E arrangements lead to duplication and waste in data collection. Underused data discourage efforts to improve data quality. Excessive workload of data collection, inspection, and reporting can divert managerial attention from productive service delivery. When incentives are tied to conduct of M&E, as opposed to the underlying performance revealed by M&E, managers are rewarded for good paperwork rather than contribution to poverty eradication. In Uganda, there are examples of M&E harmonization initiatives that have added to, rather than simplified, M&E. New systems, even if logically sound, are unlikely to constitute any improvement unless they are also accompanied by simultaneous reductions in M&E elsewhere. In situations with several parallel information collection and reporting systems, data redundancy, duplication, and waste become more likely. Familiarity with the numerous guidelines and reporting formats becomes a major workload. Focus on downstream results is dissipated, and capacity development is undermined. Information that flows from different M&E streams cannot easily be aggregated or compared for purposes of broader, cross-sectoral policy and budget analysis, including for preparation of the PEAP/PRSP. 7

17 The objective of national M&E systematically yielding quality data to buttress a public management system oriented toward performance, cost-effectiveness, and results of service delivery remains an important challenge for Uganda. If synergy and simplification is brought to the set of M&E systems and practices that have been put in place in Uganda, it will constitute an important step in the right direction. Many senior government decision-makers and other key local stakeholders have understood that good M&E is important for the attainment of national development goals. In particular the PEAP and the PMES show evidence of a widening appreciation that sound M&E is not merely a data collection issue, but is also intrinsic to resultsoriented public management practice. The increased interest in and attention given to M&E in Uganda emanates from a combination of domestic and external sources of demand, founded on common concerns about performance and results orientation in public service delivery. If M&E has made a positive contribution to improving public management, it has often been more from its implicit importance as part of wider reform efforts than from technical M&E development efforts. An emerging understanding of M&E s role has helped raise the bar for assessment of GoU poverty eradication efforts. PEAP revision and PRSP progress reporting leave an impression of coherence and substance in analysis that is often lacking in other PRSP countries. At the sector level attention is shifting, albeit slowly, from the monitoring of ministry activities toward greater attention to service delivery targeting and outcomes. Cost-effectiveness questions are being raised and discussed, even if not yet resolved. Growing transparency and participation bring a degree of assurance that the GoU s actions are congruent with citizens own perceptions of need. 8

18 4. WHAT CHALLENGES REMAIN? Systemic M&E challenges: In terms of results-oriented M&E, four sets of problems remain: 1. The missing middle : Uganda s national M&E systems produce relatively accurate measures of expenditures and activities on the one hand and poverty or impact on the other, but there are relatively few systematic data available at the middle level of outputs, reach (the coverage of services and other outputs), and outcome. (This is a common problem in PRSP countries (Booth and Lucas 2001). Service providers, for instance, rarely collect client satisfaction data. 2. Link between stages of results chain: Within Uganda s public sector, existing M&E systems are generally not geared toward understanding causality and attribution between the stages of development change. The evaluation function is relatively undeveloped. Although tracing causality through the results chain always involves methodological challenges, an understanding of the processes of producing results from government and donor spending is, nevertheless, at the heart of the performance concept, whether in terms of contribution to outputs, outcomes, or impact. 3. Feedback and use of M&E data: In Uganda incentives for civil servants are often tied to nominal compliance with reporting requirements, rather than responding to the underlying performance revealed by M&E. More broadly the demand for evidencebased decision-making is not always present. Poor performance and misconduct, for instance, are rarely sanctioned. Also, little feedback is provided on data collected through ministerial inspection. What matters with M&E is not so much the data that is collected or the facts that are available, but how the data are used to inform choices in the different stages of planning and public service delivery. 4. Multiple, parallel data collection systems: In Uganda planners and managers at the sector, district, and facility levels currently have to relate separately to the numerous different M&E arrangements that have been established for the different conditional grants and national funding schemes, as well as the different donor requirements that remain. The result is a complex and formidable burden of inspection activity, indicator data collection, and reporting formats. These problems have, in turn, led to poor-quality M&E data in terms of missing, inaccurate, or outdated information. The distinction between observed reality and what is hoped for is often blurred. Although the M&E systems and practices that are in place arguably provide a reasonable accountability framework, their contribution to substantive learning is more limited. Uganda s M&E system produces a large volume of data on compliance with rules and regulation, but it is often of poor quality or not fully used and does not yield a sufficiently clear basis for assessment of value for money and cost-effectiveness in public service delivery. Poor data quality and lack of demand for performance information are mutually reinforcing in undermining efficacy of M&E systems. 9

19 Making further headway with M&E will necessarily be an ongoing and long-term process of awareness building, institutional liaison, systems adjustment, and skills formation, which will require coordination and operational action on several fronts. Further reforms will be necessary at the individual sector or ministry level, but cross-sectoral coordination will also be needed to bring coherence, focus, and synergy to Uganda s M&E systems. The PRSO as entry point for M&E reform: The PRSO supports national M&E capacity development in two main ways. First the PRSO emphasizes close liaison between donors and the GoU through budget support, thus inducing more reliance on the government s own M&E systems and practices and less emphasis on the multiple and disparate donor M&E rules and requirements. Second M&E is identified under the PRSO as an area of cross-cutting governance reform, together with changes in the related areas of pay reform, budget management, accountability, and anti-corruption, and engagement with civil society. The focus of M&E work is the PRSO policy matrix, which represents a rolling plan of issues to be addressed and specific actions needed for national M&E systems and practices to bring forth data that are of high quality, are fed into decision-making, and can bring a focus to cost-effectiveness and value for money issues. 9 In this respect the PRSO constitutes the national M&E reform agenda. To coordinate the review work and to follow up on implementation of the PRSO M&E benchmarks, an M&E working group has been formed, under the leadership of the Office of the Prime Minister (OPM) and comprising MOFPED, and the Ministries of Public Service and Local Government. M&E responsibilities and coordination arrangements: The Ugandan government has realized that stronger coordination is needed to ensure that M&E helps guide public actions toward greater cost-effectiveness in pursuit of poverty eradication. Coordination is needed to improve information standards, requirements, and systems that support different levels of national decision-making. Because assignment of responsibility for overall M&E coordination can ultimately only be resolved from the political level, the M&E working group is preparing a strategy for national M&E coordination, which has recently been submitted for review by the Cabinet. As a starting point for establishment of an integrated national M&E system, the GoU has reached agreement on a structure of responsibilities for implementation of cross-cutting public management reforms under the PRSO as a whole. At the same time the National Planning Authority, with a possible role in M&E coordination, has recently been established. Building on a clear and agreed set of M&E responsibilities, the development of systems and skills also needs to be subject to coordination. In this respect the World Bank s Uganda Capacity and Performance Enhancement Project (CAPEP), a US$100 million facility currently under preparation, may represent a useful entry point. Clarity of targets for outputs and outcomes: Over the last few years, significant improvements have been made to sector-level goals and targets, as reflected in budget requests, sector strategies, and policies. However, much still remains to be done in terms of the results orientation and consistency of sector planning and performance management. Specifically, several ministries have done their planning on the basis of measures that essentially relate to inputs, activities, and processes that is, with a focus on the spending side, as distinct from the outputs and outcomes that are meant to result. Nevertheless, improvements have been made in, for example, the health, education, and water sectors by identifying output and outcome targets in the budget cycle. The GoU has similarly committed to a special effort to identify output and outcome targets for the 10

20 agriculture and the justice, law, and order sectors. For the 2003 Public Expenditure Review, the GoU and its partners are evaluating the results orientation of the overall budget and, specifically, the progress made in five sectors: health; education; water and sanitation; agriculture; and justice, law, and order. The currently ongoing second revision of the PEAP is also expected to lead to an improvement in the clarity and results orientation of sector goals and targets. Common district-level data collection: Currently the various line ministries in Uganda have separate and parallel systems and routines for compiling and reporting data from districts and local governments. As reported in annex E, there is considerable overlap between these systems, and much of the data collected appear to be either redundant or of poor quality. The combined workload is in any case a substantial burden on district planners and sector program coordinators. There are also definitional differences in basic concepts and terminology of M&E under the various M&E systems. Until recently the data burden on districts and ministries had been expected to increase as a result of a new data system planned by the Ministry of Local Government under the Local Government Development Programme (LGDP). However, under the aegis of PRSO liaison, it has been agreed that the GoU will instead seek to build a common platform for local-level information collection and data entry, where a single system will serve the most essential information needs of both the central and the line ministries. Consolidation of inspection: Uganda s districts and frontline service delivery units are burdened by excessive visits by supervision, support, and inspection teams from various ministries and agencies these teams often have different objectives but frequently ask for similar information. Those visited perceive inspection as unnecessary policing, rather than constructive support or quality assurance, and also observe that it reduces the time available for actual service delivery. There is poor feedback from inspection visits, and little of the data collected are used for any decision-making at the local level. Inspectors are believed to be often motivated by per diem remuneration for travel and to give little attention to service facility client concerns, and they are rarely able to address value-for-money issues. Most of the inspection activity been conceived in terms of people from the centre making physical visits to the district or facility, with an objective of verifying the content of district reports. A very large workload is involved in catering to inspectors. In the health sector alone, this potentially involves as much as the equivalent of 1,400 staff-years annually. There are opportunities for reducing both the inspection activity that takes place within ministries and the inspection undertaken by the different national accountability institutions. One option would be to consolidate inspection activity into coordinated multidisciplinary teams of technical ministry personnel and central accountability representatives. There are also a number of inspection tasks that can be undertaken by CSOs or communities themselves, such as verification of classroom or health clinic construction. Agenda setting for conduct of evaluations, ad hoc studies, and reviews: Evaluation is not systematically embedded in the GoU s management practices. Most evaluations have been instigated by donors. The lack of coordination of studies and reviews has led to either overlap between different research exercises or reports of such work being poorly fed into decisionmaking. As part of the PRSO policy matrix, and as envisaged by the PMES, the GoU intends to establish a mechanism for agenda setting for evaluations, ad hoc studies, and reviews. Because evaluation addresses issues such as actual progress in attainment of program objectives, costeffectiveness and value for money, it responds to some of the aspects of Uganda s M&E system that are most critically lacking. Expenditure tracking studies in the Uganda health and education 11

21 sectors have brought about a better understanding of various budget execution problems by identifying substantial problems with leakage, delays in flow of funds, and capacity problems related to financial and operational management. To date, the expenditure tracking studies have largely been instigated and funded by donors, although their usefulness has since been attested to by the GoU itself. Other sectors also need to do tracking studies to improve flow of funds and enable efficient use of limited resources. Use of public service client satisfaction data: To rectify the lack of systematic information on client satisfaction with public services, a national service delivery survey (NSDS) was conducted in 2000 and will be repeated in Client satisfaction and public perceptions of service delivery quality are important issues with which civil service managers at both the facility and ministry levels should be concerned. The findings of the NSDS can be adopted as a barometer of government service reach and used for comparing performance at different locations and over time, for gauging the targeting effectiveness of government service delivery, and for performance target setting. More generally, changes in the level of user satisfaction can indicate to managers where they need to make changes, and levels of client satisfaction can provide the yardsticks of performance for which managers can be held accountable. A further, specific potential use of the NSDS would be the articulation of client service charters commitments for facility-level and ministry performance. However, the first NSDS has not been well used as a basis for performance improvement. One reason is that the surveyed coverage did not correspond well with functional areas of service delivery responsibility. Strengthening public accountability: Although popular dissatisfaction with Uganda s public service delivery remains widespread, many communities still feel that they are not empowered to question their leaders or service delivery managers. Bottom-up accountability is embedded in Uganda s Constitution and Local Government Act and is an aspiration in sector policies, but putting in place actual instruments of accountability has been weak. At the local level people still do not feel involved in decision-making. 5 The mandatory posting of notices on PAF allocations has been patchily practiced, presumably in part because there are no sanctions for civil servants who fail to comply with this requirement. The GoU intends to review how direct, bottom-up public accountability can effectively complement the established internal accountability mechanisms of government itself. An initiative is under way to establish a system of facility-level client scorecards to be managed by CSOs. Uganda s Parliament has recently assumed a strengthened role in the government budget process but is still considered relatively weak compared with the civil service. The UEA is emerging as a professional body with a capacity for contributing to the independent assessment of government actions and performance. The challenges that remain will need to be addressed at several levels. Individual institutions and ministries will have to take the lead in, for example, clarifying their goals for contributing to national poverty eradication and measuring client satisfaction with their service delivery. At the same time central coordination will be needed to ensure common approaches to such tasks and identify overlaps and potential synergies among different planning, data collection, and reporting systems. 5 See, for example MOFPED (2003) and Brock, McGee and Ssewakiryanga (2002). 12