Using Public Expenditure Tracking Surveys to Monitor Projects and Small-Scale Programs /A Guidebook

Transcription

1 Public Disclosure Authorized Using Public Expenditure Tracking Surveys to Monitor Projects and Small-Scale Programs /A Guidebook Margaret Koziol and Courtney Tolmie blic Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized

2

3 Using Public Expenditure Tracking Surveys to Monitor Projects and Small-Scale Programs A Guidebook Margaret Koziol and Courtney Tolmie Washington, D.C.

4 2010 The International Bank for Reconstruction and Development/The World Bank 1818 H Street NW Washington DC Telephone: Internet: All rights reserved This volume is a product of the staff of the International Bank for Reconstruction and Development/The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: ; fax: ; Internet: www. copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: ; pubrights@worldbank.org. ISBN: e-isbn: DOI: / Library of Congress Cataloging-in-Publication data has been requested.

5 iii Table of Contents Acknowledgments v Overview What is a Public Expenditure Tracking Survey? Step 0. Defining the Objective Step 1. Mapping Resource Flows Step 2. Collecting and Analyzing the Data Step 3. Identifying the Issues Step 4. Recommending Solutions Step 5. Dissemination and Advocacy Conclusions Annex A: World Bank PETS Annex B: CSO-Led PETS Annex C: Resources for Power Calculations References

6

7 v Acknowledgments This guidebook is the product of a collaborative effort between the World Bank Human Development Network s Office of the Chief Economist and the Transparency and Accountability Program (TAP) of the Results for Development Institute. The goal of this partnership is to support the capacity of civil society organizations (CSOs) to monitor and evaluate government programs in the human development sectors. The World Bank contributes to this partnership by offering support to civil society organizations participating in the program, including contributing technical expertise on Public Expenditure Tracking Surveys (PETS) and absenteeism methodologies, preparing training materials, reviewing research instruments and reports, and supporting workshops and courses for the researchers involved in the studies. The Bank also takes an active role in making connections between the CSOs supported by TAP and task teams to build links with operations. Established in 2006 with support from the William and Flora Hewlett Foundation, TAP focuses on strengthening the demand side of governance by providing technical and financial support to civil society organizations (e.g., non-governmental organizations [NGOs], think tanks, and universities) involved in promoting the accountability of public services to civil society and to citizens through analysis and advocacy. TAP s activities aim to address the major barriers that civil society organizations face in turning actions into results: weak institutional capacity and skills, non-existent or adversarial relationships with government and policymakers, and lack of access to information. TAP helps organizations overcome these barriers by supporting access to information, providing both the incentive and tools to develop analytic skills and capabilities, and providing technical support and peer learning opportunities to help connect

8 vi Using PETS to Monitor Projects and Small-Scale Programs CSOs with government and stakeholders so that their policy recommendations might be heard. This guidebook benefitted from comments by and conversations with many colleagues, including Zafar Ahmed, Deon Filmer, Ariel Fiszbein, Bernard Gauthier, Markus Goldstein, Charles Griffin, Kai Kaiser, Dena Ringold, and many current and former TAP program participants. Please send comments to and

9 1 Using Public Expenditure Tracking Surveys to Monitor Projects and Small-Scale Programs Overview Most often, Public Expenditure Tracking Surveys (PETS) are referred to in the context of a large, nationally sampled public expenditure review, or in the context of similar studies conducted by organizations such as the World Bank. A tried and tested methodology, PETS have been shown to be effective in identifying delays in financial and in-kind transfers, leakage rates, and general inefficiencies in public spending. Recently, civil society organizations have successfully taken up this methodology as part of a push for budget accountability from the ground up, an effort supported in part by the creation of this guidebook. Civil society organizations comparative advantage resides in their ability to take the temperature on the ground and to act on those issues that are most heated in the minds of the citizens whom they represent. In many cases, service delivery in the education and health sectors is a top priority. One way of improving service delivery, by keeping both governments and service providers accountable, is through the monitoring of budgets and efficiencies in public spending. PETS, when used by civil society organizations, offer an opportunity to carefully monitor specific programs or public spending in targeted districts and regions. Moreover, this instrument may be used to monitor World Bank and other projects. This brand of accountability strengthens the voice of citizens and focuses the discussion on problems that may be micro-level in nature, or specific to a particular region of a country.

10 2 Using PETS to Monitor Projects and Small-Scale Programs The goal of this guidebook is to serve as a starting point for civil society organizations, as well as Bank teams interested in conducting Public Expenditure Tracking Surveys, both on a small and larger scale. It is designed to lead a research team from idea inception to results dissemination, while emphasizing the importance of utilizing evidence to influence policy, regardless of whether it is on a macro or microlevel. Though the World Bank has been at the forefront of efforts to measure the effectiveness of service delivery, it is hoped that Bank teams and civil society alike will take the research one step further and empower citizen users to keep service providers accountable through information dissemination and citizen engagement efforts. The Power of PETS Experiences with PETS point to the strength of this particular methodology when used to monitor programs, projects, and public spending. Some illustrative examples of findings from these studies include the experiences of Ghana, where 50 percent of non-wage education expenditures and 80 percent of non-wage health expenditures did not reach intended beneficiaries; of Zambia, where leakage rates of discretionary non-wage expenditures reached 76 percent; and of Chad, where less than 1 percent of the non-wage budget officially allocated to regions actually reached health centers. 1 In Ghana, researchers found that incidences of leakage were more prevalent when the value of in-kind materials distributed to beneficiaries was unknown. Researchers in Zambia found evidence to suggest that a few select schools with greater bargaining power were able to capture the majority of the funds. And, in the case of Chad, it was estimated that, had all resources officially budgeted to regions actually reached service providers, the number of patients seeking primary health care would have more than doubled. These examples are only a small indication of what researchers can do by strategically utilizing the PETS methodology. 1 Ye and Sundharshan 2002, Gauthier and Wane 2006, and Das, et al 2004a.

11 A Guidebook 3 What is a Public Expenditure Tracking Survey? Public Expenditure Tracking Surveys are tools in a methodology used to track the flow of public resources (including human, financial, or inkind) from the highest levels of government to frontline service providers. PETS first identify a research objective, then employ an extensive mapping exercise in order to understand the flow of funds through the different levels of government. Once the resource flows are mapped, budget data are collected and analyzed, and often complemented with a facilities survey and qualitative research. PETS can help civil society and policy makers alike to understand funding flows, identify areas of leakage, and make informed policy decisions based on their findings. Goals and Uses PETS were developed to gather information beyond official data and administrative records to understand what actually happens to money that is appropriated for service delivery. We propose a two-tiered typology to describe how PETS can be applied in the human development sectors. Analytical PETS PETS with an Applied Research Focus The first PETS conducted by the World Bank was a diagnostic tool to support a Public Expenditure Review (PER) in Uganda. At the national level, funding had been provided for non-salary expenditures to each school in the form of a capitation grant. Normally a PER would take note of the amount attributed to each school and how it compared to what was needed for non-salary inputs. In this case, the PETS allowed analysis of what actually happened in the allocation; the result was startling. On average, 87 percent of the grant did not reach the school and was diverted along the way. The Ugandan government subsidized an information campaign alerting service users to the findings, whose results were later correlated to a decrease in the amount of the grant being diverted in subsequent years. Outside agencies have come to see PETS as a good instrument for

12 4 Using PETS to Monitor Projects and Small-Scale Programs ascertaining what actually happens to money or goods that are supposed to reach service delivery points. PETS, and other similar methodologies, 2 are important in showing systemic problems related to how a budget is executed, which with auditing and evaluation is typically the weakest element of government performance (as shown by PEFA surveys and the Open Budget Initiative). 3 In the case of Uganda, the discovery of leakages and the public campaign to alert parents seem to have adequately addressed the problem. 4 One way of using this approach to PETS, especially on a more micro level, might be to consider whether there are higher rates of leakage and capture in certain districts than others, or to focus the research by targeting a specific question, program, or geographical location. Yet another design might examine different types of education or health facilities and then compare outcomes. When using PETS as an analytical tool, researchers or civil society organizations can utilize the findings to identify causes, which they can then use as evidence to inform policy decisions; advocacy supported by robust results can be a powerful tool, regardless of whether a research team is working at the grassroots or national level. In many cases consulting firms or universities are contracted to conduct analytical PETS because the function of the research is to derive generalizations at a fairly high level. However, developing capacity among civil society organizations interested in scaling up research efforts should also be considered. Monitoring PETS PETS with a Governance Purpose There is another way to use PETS, which is the primary purpose of this guide. Monitoring PETS, which are also analytical in nature, ask whether 2 Quantitative Service Delivery Surveys, Public Expenditure Reviews, Public Budgeting Analysis, Benefit-Cost Analysis, etc. 3 and 4 Reinikka and Svensson 2001.

13 A Guidebook 5 an intervention is working as intended and other narrow questions about service delivery at the local level. Typically, they do not seek to solve national problems or devise new laws or regulations that could change incentives countrywide, though this sometimes ends up being the result. World Bank teams, for example, may have an interest in ensuring that the intended beneficiaries of a project are able to monitor whether, and what, the project is delivering. In that case, it would not make much sense to turn the job over to a consulting firm, a university, or a government agency, because part of the purpose is to generate knowledge among those who have an interest in demanding better results the users and to increase their ability to monitor and advocate for improvements. Monitoring PETS can also amplify the voice of civil society organizations and user-citizens by giving them an empirical tool to demand, on behalf of end-users, improvements in governance, management, and delivery of services, at least from the standpoint of resource provision and the reduction of financial waste. Because they are based on facts, PETS-derived lessons can help civil society organizations build credibility through knowledge-based constructive engagement with governments and seek feasible changes on the local level. CSOs are able to shift from pure advocacy for causes, for higher spending, or for rights to an analytically-based approach that demands accountability for results and for delivery on promises already made. An example of bridging the gap between analytical and monitoring PETS exists in Uganda where the first step was diagnosis through an analytical PETS, next was a public media campaign as a solution, and finally a follow-up survey to ascertain what occurred. In Uganda, using PETS identified a particular problem, action was taken to address the problem, and the result was monitored. The monitoring PETS combines diagnosis, action, and follow-up. If performed locally by a CSO that is interested in monitoring governmental performance, the PETS can also contribute to institutional development in this area. 5 5 Reinikka and Smith Gauthier 2006.

14 6 Using PETS to Monitor Projects and Small-Scale Programs One example of a CSO-led PETS incorporating diagnosis, action, and follow-up as well as discernable policy change comes from Centro de Investigaciones Económicas Nacionales (CIEN) in Guatemala. Responding to the call from a newly elected Minister of Education to improve education in her first 100 days in office, CIEN designed a PETS to identify existing problems in six primary school financing programs including support for textbooks and school meals. While CIEN found little evidence of leakage during its study, the organization identified significant delays in the distribution of resources to schools. Interviews and focus group discussions carried out in conjunction with the study allowed researchers to diagnose one possible cause for these delays: a school calendar that overlapped with the fiscal calendar and led to backlogs in approving resource allocation and delivery to schools. CIEN took action by presenting findings and recommendations to a willing Minister of Education. CIEN continually followed up with the Minister to encourage adoption of its recommendations, which contributed to the Ministry s decision to shift the school calendar in 2009 in order to address the delays. It is hoped that this manual will serve as a guide for organizations and research teams alike; the general procedure is the same regardless of the purpose. Scaling it up for an analytical PETS would be straightforward, but here there is an emphasis on choosing a narrow purpose, developing a small sampling framework, carrying out the work inexpensively, and taking the important final step of not shelving the study but of advocating for change, based on the findings. An important item to note is that, by focusing on a discrete and measurable objective, as we suggest in this guidebook, researchers are likely to be looking at a very small portion of the overall government expenditure. This was the case with the first PETS conducted in Uganda, where the capitation grant tracked represented approximately 3 percent of the recurrent expenditure on education. This does not in any way diminish or lessen the importance and impact that a small-scale exercise may potentially have. Nonetheless, understanding the limitations of these types of studies is essential.

15 A Guidebook 7 Step 0. Defining the Objective The first step in conducting a PETS defining its objective can be the most significant step in determining whether the implementing organization succeeds in conducting a valid and valuable study and whether the work has a short-term or long-term impact on public spending and service delivery. Defining a clear objective for a PETS will drive not only the scope of the study, but will also determine what an organization or research team can do with the study s results. That said, thinking about the way in which the research results will be used can also be a powerful tool for determining the objective and scope of the study. For either category of PETS analytical or monitoring defining the objective involves making critical decisions about the study, including selecting the topic and scope, developing research questions, and identifying target audiences. Selecting the Topic and Scope The success of any research team in conducting a high-quality and highimpact study depends first on how well the topic and scope of the research are defined. A list of PETS conducted by the World Bank alone (Annex A) illustrates the breadth of possible study topics. Comparing these studies with those conducted by CSOs (Annex B) further demonstrates the difference in scopes among successful PETS. It is critical that implementers choosing to conduct a PETS make decisions about the specific scope and focus of their study. Table 1 presents a non-exhaustive list of questions and examples of decisions made in response. When answering these questions, research teams should consider what they are trying to achieve with the PETS project (both the study itself and the advocacy and dissemination of results) and how to design the project to best achieve these goals. While the project design and choice of topic should take into consideration a number of different factors, there are three questions that we recommend all research teams ask during the project design process. We call this the FIR criteria:

16 8 Using PETS to Monitor Projects and Small-Scale Programs Is the proposed topic FEASIBLE for your organization? CSOs are wellplaced to implement PETS because of characteristics that they share namely, being based in-country and having an understanding of the local context. However, individual CSOs have differing strengths, and the most successful PETS are designed to complement an organization s strengths. For example, the Centre for Regional Information and Studies (PATTIRO), 6 a research advocacy organization in Indonesia, designed a tracking study of education program grants that supplemented basic quantitative findings with qualitative results from focus group discussions and interviews. On the other hand, the Romanian Academic Society (SAR), 7 a think tank, conducted a more rigorous quantitative analysis of school spending in Romania. Both of these CSOs built upon the existing strengths of researchers rather than attempting to conduct a study that fell outside their areas of expertise. Is the proposed topic of INTEREST to the target audiences? A common mistake made with PETS and other public expenditure studies is that they are seen by project implementers as purely research exercises. To maximize the likelihood of improving policies and the effectiveness of public spending, CSOs and project teams should choose a topic with an eye towards getting the attention of stakeholders and using their results to influence positive change. One way of approaching this is for implementers to choose a topic that is of interest to the people that they hope to influence. For example, focusing the research on an area that is being focused on by the national government. Furthermore, and perhaps even more essential, research teams can increase the likelihood of successfully impacting policy change by identifying a policy entry-point or a well-placed champion for the cause, or by packaging the results in such a way that the research team is targeting the right audience and being heard by the right people. Is the proposed topic RELEVANT to the organization s mission and long-term agenda? Target audiences such as policymakers, NGOs, and the public are not the only ones that should buy into

17 A Guidebook 9 the project design and topic; the implementing CSO or project team should choose a topic that fits into its mission and agenda for the future. While project funding can be attractive to CSOs, it can also have the unintended effect of taking an organization off its missiondesignated path. Project topics should not be donor-driven; the most successful PETS are designed to address the mission and goals of the lead organization, as well as critical needs identified on-the-ground. Table 1. Selecting a PETS Topic Question What facility level? What facility type(s)? What category of spending or resource flow? What program or policy? What is the geographic scope? Examples Primary schools versus secondary schools Public, Private, NGO-led Salaries, Capital spending Scholarship program, Textbook fund, Vaccine spending or in-kind transfers Focus on entire country, specific province(s), specific district(s) A Case for Targeted PETS While the World Bank and individual country governments have undertaken large-scale, sector-wide PETS in the education and health sectors, the most successful and most frequently cited studies have focused on specific programs or policies. The inceptive PETS conducted in Uganda in 1996 followed the allocation and disbursement of capitation grants for non-wage spending in Ugandan primary schools. While many factors contributed to the success of this PETS in lowering the level of leakage in the Ugandan program, the value of focusing on one program cannot be overstated. Armed with unambiguous evidence of leakage points in the capitation grants program, the public and media could easily transition to monitoring improvements in grant distribution and holding the correct officials accountable for any continuing problems.

18 10 Using PETS to Monitor Projects and Small-Scale Programs Recent PETS conducted by civil society organizations have taken similar approaches and shown early signs of success in developing a sustainable system of monitoring spending in specific programs. Focusing on a specific program or policy provides many advantages for smaller scale exercises: Implementation is less complicated. Rather than having to disentangle the large and often complicated web of sector-wide spending, a research team can focus on the small portion of the web that goes to its program of interest. Further, program-specific PETS generally require fewer data sources, increasing the likelihood that implementers will manage to gain access to all necessary data in a timely manner. Audiences are easier to target. A well-implemented PETS will generally identify areas for improvement (if not points of leakage) at each stage of the expenditure chain. Successful dissemination of results and recommendations includes bringing results back to government officials and service providers at each of those links in the chain. In a sector-wide study, the number of individuals or departments that would need to be targeted for dissemination can be daunting. In a PETS that focuses on a particular program or policy, the audiences for dissemination are smaller in number and generally easier for a team to target. Recommendations are often targeted, concrete, and more attractive to policymakers. Large-scale PETS take on the difficult task of evaluating the effectiveness of public spending for an entire sector, and the recommendations that come out of the studies often reflect this. For example, without focusing on specific components of education spending, project implementers generally identify system-wide problems and recommend solutions that reform the wider public expenditure management system. While these recommendations are often valuable, they are not always as attractive to policymakers (especially those facing short political terms or upcoming elections). On the other hand, a PETS that focuses on a textbook spending program can identify very specific problems and develop tangible and frequently inexpensive solutions. When presented with this type of recommendation by a credible CSO or team of researchers, policymakers are much more likely to be amenable to implementing the proposed solution.

19 A Guidebook 11 Box 1. World Bank Health PETS Nigeria 8 Motivation: To assess a decentralized delivery of health services under a federal system. Findings: Large-scale leakage of public resources in Kogi State. 42 percent of health staff had not received salaries in over six months, despite the fact that budget allocations were sufficient to cover costs. To supplement salaries, public health providers charged for services illegally, while expropriating drug supplies and selling them to patients. Follow Up: Though the study suggested that providing service users and citizens with more information about budget resources so they could hold providers accountable, no documented post-pets experiences currently exist. Developing Research Questions Like topic and scope, the list of possible research questions that a PETS can be used to answer is extensive. The most cited findings from expenditure tracking studies are leakages in spending. Results of leakage also tend to command the most attention in the media and the public because they can explicitly point out instances of corruption. Budgeting and expenditure analysis are often described as unglamorous topics; however, demonstrating corruption in the form of stealing public money is likely to grab the attention of those who are not normally interested in public spending. Implementers can also take the next step to explore patterns in leakages across geographic areas or time of year and determine potential causes of these problems. 8 Das Gupta, Gauri, Khemani, 2004.

20 12 Using PETS to Monitor Projects and Small-Scale Programs In addition to leakages, though, PETS studies can uncover many other problems with the functioning of the public expenditure management system and service delivery. PETS, along with review of financial records at facility and government levels, can also uncover delays and inefficiencies in resources reaching their intended beneficiaries. These issues can be due to poor planning, simple accounting errors, pharmaceutical expiration dates leading to medicines going unused, or inexplicable losses due to human error. In other words, leakage does not necessarily mean corruption. Past PETS have also frequently been expanded to include a collection of data on service provider absenteeism, mismatches between facility needs and allocations, and user satisfaction with public services. 9 Although these topics are not a part of traditional PETS, research teams and civil society organizations often bring one or more of these additional questions into their budget tracking studies. Before developing surveys and other methodological tools, it is valuable for the implementers to determine what questions they are hoping to answer with the study. Examples of questions (investigative and analytical) are listed in Table 2. We make the same case for research questions as we do for topic and scope; taking a targeted approach increases the feasibility of the study and the likelihood of successful dissemination and advocacy of the PETS results. While it can be tempting for implementers to attempt to answer all of these possible research questions with one study, narrowing the list of questions allows researchers to focus on the issues that are most important and investigate those few issues in a much more rigorous way. Limiting the list of research questions also allows implementers to design and administer shorter surveys that are less burdensome to respondents, increasing the likelihood of a high response rate. 9 Information on methodology like quantitative service delivery surveys is available online at

21 A Guidebook 13 Table 2. Examples of Diagnostic and Analytical Research Questions for PETS Studies Investigative Questions What is the level of leakage? At what stages of the expenditure chain do leakages occur? Analytical Questions How does the level of leakage vary across districts? Are schools with X characteristic less likely to experience leakages? Are there delays in resources reaching beneficiaries? What is the level of absenteeism? Does the funding requested by facilities match the funding allocated to them? Are there institutional factors or characteristics of the expenditure system that may be causing delays in resource disbursement? How does absenteeism vary between different facility types? Many PETS implementers find it helpful to utilize a research matrix to carefully define their topic and research questions and to ensure a targeted and narrow set of objectives. The sample matrix in Figure 2 comes from ABANTU for Development, 10 a Nigerian CSO currently implementing a PETS in the education sector, with the support of a grant from the Transparency and Accountability Program. A matrix such the one in Table 3 allows implementers to translate objectives into specific research questions, variables, and necessary sources of data. Identifying Target Audiences and Developing a Dissemination Plan The final critical component of defining project objectives is to identify the target audience for the study and develop a dissemination plan. A common criticism of PETS is that, while the study may identify critical 10

22 14 Using PETS to Monitor Projects and Small-Scale Programs Figure 1. Conceptualizing the PETS Determine Research Questions Define Objectives Identify Target Audience for Dissemination Conceptualizing the PETS issues, the results are not utilized to improve the efficiency of public spending or service delivery. One way in which implementers can ensure that that PETS results gain traction after the project is completed is to begin the PETS process with a plan of how results will be used to enact change. Research teams should begin by identifying and consulting with the project s target audiences before it has been implemented. Gaining buy-in from stakeholders at the project design phase will secure the interest of audiences in the topic and increase the probability that they will implement the recommended changes following the completion of the study. In addition to identifying target audiences at the beginning of the PETS, implementers should also develop a strategy to keep these stakeholders informed and involved throughout the project s duration. This is particularly

23 Table 3. Sample Research Matrix for PETS Objectives Questions Variable of Interest Data Source 1 To understand the mechanism of resource allocation in the education sector at the state and LGC levels and identify major issues in budgetary allocation in 3 LGAs in Kaduna State To identify problems with resource allocation and transfers What are the procedures with regard to resource allocation and transfers? What are the spending patterns at each level? What are the procedures with regard to resource transfers? What are the spending patterns at each level? What was budgeted? What was approved? What was released to the sector? What are the methods of disbursement of funds at the two levels? What is the funding gap to sector needs? Actual funds received at each level? Efficiency in disbursement? Budget discipline? Transparency and Accountability of funds at all levels? Challenges and constraints? State budget LGA budget SUBEB budget MOEP Financial records of the tier Finance and administration officials at the two levels 1 LGA local government area; SUBEB State Universal Education Board; MOEP Ministry of Economic Planning From Research Matrix developed by ABANTU for Development based on materials presented by Delius Aslimwe, July Method of Data Collection Desk review Extraction In-depth interview Data Collection Tool Extraction forms/checklist Extraction forms/checklist In-depth Interview schedule A Guidebook 15

24 16 Using PETS to Monitor Projects and Small-Scale Programs true for studies that are implemented over a long period of time. While stakeholders may show enthusiasm at the beginning of a project, this interest can wane if there are significant stretches of time during which the research team does not provide these individuals with updates. At the same time, researchers should be aware of and adapt to changes in their constituency and government counterparts, as contexts such as the political climate and government representation may change. While interacting with stakeholders, it is best for research teams to know what their dissemination and advocacy goals are. However, strategizing about what advocacy and communications activities will be undertaken, who will be targeted, and what will be achieved is a valuable exercise to undertake before the first pieces of data are collected. Choosing a Topic Major Take-Aways Consider whether your topic is feasible given the skills of your project team, of interest to policymakers and other key audiences, and relevant to the mission of your organization. Narrowing the focus of the PETS can make implementation and subsequent advocacy easier and more effective. In addition to selecting a topic, develop research questions that you hope to answer with your study and keep these questions in mind as you create your methodology and instruments. Identify key audiences before beginning the work plan to ensure buy-in and willingness to listen to results and recommendations. Step 1. Mapping Resource Flows A core idea behind Public Expenditure Tracking Surveys is that government officials and facilities have an incentive to misrepresent the amount of resources going to service users if some fraction of these resources is being used for purposes for which the funds are not intended (including padding the pockets of politicians and diverting to other public spending

25 A Guidebook 17 line items). To deal with this problem of incentives, PETS implementers triangulate budget data by looking at records from all of the places where money changes hands for particular resource flows. Viewing district level records of money allocated to a primary school by itself cannot identify potential corruption; comparing these records to ones kept by the primary school itself and seeing that the two records do not match, however, can point to a place where money may have leaked. Before triangulating budget data however, implementers must prepare to collect the necessary data. One critical step that research teams must take before actually gathering the data is determining what data are necessary. Most public social sector funds go through many hands before reaching the service user, and accurately determining both the level and location of leakages requires that project teams start with a complete and correct picture of what the expenditure chain looks like. This exercise in and of itself can be a major contribution of a PETS. Though various entities (government ministries and agencies) understand their own budgets and transfers systems, very few, if any, truly understand how frontline service providers are actually funded. A well-mapped PETS can serve as a vehicle for establishing this information. Mapping the flow of resources also provides a good list of sources from which to gather the data needed to conduct the PETS. Researchers should also note that different entities along the expenditure chain may call the same pot of money by different names, potentially making data collection and analysis more challenging if not noted and accounted for at early stages of research. The resource map may be a relatively simple flow or a complex web with multiple sources of funding and links in the chain. A 2008 tracking study by the Institute for Policy Analysis and Research (IPAR) in Kenya followed the flow of secondary education bursary funds from the central government to the secondary school level. 11 IPAR s background work 11 Oyugi, Riechi and Anupi 2008.

26 18 Figure 2. Flow of Funds for Secondary Education Bursary Scheme in Kenya The main source of bursary funds received in secondary schools Central Government Ministry of Education The custodian of the Secondary School Bursary Scheme Charged with the responsibility of evaluating and awarding bursaries to students at the constituency level in accordance with the guidelines issued by the Ministry of Education Constituency Bursary Committees Secondary Schools Recipient of the funds Ultimate beneficiary Students Using PETS to Monitor Projects and Small-Scale Programs

27 A Guidebook 19 mapping the flow of funds demonstrated a simple flow through four links in a chain: the central Government, Ministry of Education, Constituency Bursary Committees, and the secondary schools (Figure 3). All of the funding for this scheme goes through the same channels, providing the research team from IPAR a clear list of data that it would need to collect. Another example of a reasonably easy to follow resource flow is found in the 2002 Zambia PETS conducted by the World Bank, as seen in Figure 3. In this analysis, six main flows of funds were identified and categorized by their sources, types, and discretionary powers in fund allocation. This PETS tracked non-wage financial flows from the Ministry of Education and donors at the district, provincial, and facility levels. Unlike many other PETS, in-kind transfers and salaries were excluded from the analysis. The main objective of this study was to determine: Figure 3. Map of Resource Flows to Schools in Zambia 1 Province (Centralized) Ministry of Education Case IV Donors (primarily the Program for Advancement of Girls Education) Province (Centralized) District (Centralized) District (Decentralized) Households Schools Key to flows Discretionary Flows Rule-based Flows 1 Das et al. 2004a.

28 20 Using PETS to Monitor Projects and Small-Scale Programs 1) whether schools received their allocated lump-sum payments; 2) if and how provinces and districts supported schools through discretionary spending; and 3) how decentralization affected fund allocation. Not all programs or sectors follow a relatively simple linear pattern like that of the Kenyan secondary education bursary scheme, or in the case of Zambia. Budget flows can be complex and tangled, as illustrated in the resource flow mapping for Peru s health sector from the Research Center of the University of the Pacific (Figure 4). A mapping of resource flows to Peruvian health care facilities demonstrates the myriad ways in which resource flows can prove to be complicated. First, funds originate from many different sources, unlike the bursary funds in Kenya that all start at Figure 4. Map of Resource Flows to Health Care Facilities Donors PAAG Ministry of Finance (Treasury) Ministry of Health (BU) Health Insurance (SIS)(BU) Key to flows Monetary Salaries Supplies and pharmaceuticals Administration Unit Health Directorates (EU) SISMED Fund (DISA) Health Networks (EU) Facility 2 Facility 1 Out-of-pocket payments Note: EU: expenditure unit ; BU: budget unit. See the section on CIUP in Chapter 2. Source: Alvarado, Betty, and Eduardo Moron The Route of Expenditures and Decision Making in the Health Sector in Peru. Centro de Investigacion de la Universidad del Pacifico: Lima.

29 A Guidebook 21 the central Government level. Second, the direction of flows is not always one way. Third, the resources that flow through the health expenditure chain are not only monetary but also in-kind. Despite the complexity of the resource mapping, an expenditure tracking exercise requires collecting data from each of these links and knowing how each link in the chain is connected in order to accurately identify leakages and other problems in the system. One example of the problems that can arise from an incorrect resource flow mapping is provided in Box 2. There is no algorithm for uncovering how resources are supposed to flow through a system, often making the exercise of mapping funds a tedious one. Some resource flows may be common knowledge to those working on public expenditure analyses. In other cases, research teams may have to interview facility-level administrators to uncover the source of their funding and follow the chain up to the funding source. Regardless of the method used to map the flow of funds, the importance of starting with an accurate mapping cannot be overstated. Box 2. The Importance of Mapping: Tanzania Not mapping the resources correctly can have detrimental effects on the accuracy and impact of a PETS. In 2003, an education expenditure tracking exercise in Tanzania found minimal leakages of funds (5 percent) for the Primary Education Development Project. 12 However, further investigation uncovered that the actual level of leakage was much higher than that estimated in the 2003 PETS. The reason for this miscalculation was that the consultants carrying out the PETS had only looked at funds coming from the Ministry of Finance. By leaving out two additional main sources of funds for the program (Ministry of Education and Ministry of Local Government), the implementers had ignored a significant fraction of the expenditure chain, resulting in skewed findings. 12 Sundet 2007.

30 22 Using PETS to Monitor Projects and Small-Scale Programs Gathering the Budget Data After completing the map of resource flows, PETS implementers have a complete list of budget sources to check, compare, and triangulate in order to provide an accurate picture of inefficiencies in the expenditure chain. The research teams tracking the flow of funds described in Figures 3 and 4 had to compile financial data from each of the sources of recipients appearing in the resource maps. Data for all links in the expenditure chain above the facility level will be secondary data, often obtained directly from the respective government office. Facility level data are generally gathered using survey instruments that the implementing team develops. Gaining access to necessary data can require some amount of finesse. On the one hand, engaging with government officials from the beginning can help research teams overcome the problem of uncooperative government offices; on the other, it can serve as a signal to government officials that it is time to start keeping an extra set of budget books. In the experience of TAP-supported CSOs, however, when officials see that the study is being conducted out in the open and not under the radar of policymakers, they sometimes feel more comfortable providing researchers access to data (Box 3 has additional strategies from CSOs). However, in many cases, and despite all efforts, research teams are faced with the challenge of lacking access to data. Gathering budget data itself, however, can also reveal interesting trends in public expenditure management. First, the challenges that civil society and research teams more generally may face in gaining access to data from government officials can demonstrate the lack of transparency that exists in the system and the need for a remedy to opaque budgeting procedures (Box 3). Second, without looking at the numbers, many implementers find from the process of collecting budget records that there is significant miscommunication between different levels of policymakers about budgeting responsibilities and funding requests. Although uncovering these problems is not a goal of PETS, it is an

31 A Guidebook 23 Box 3. Lack of Access to Data One of the major challenges that research teams face in conducting PETS is access to reliable data. This problem can occur in a number of different forms, including: Data exists but the government does not want to provide it to researchers Data exists but is of very poor quality and is unreliable or in an unusable format Data just does not exist (a particularly common problem at the facility level) Centro de Análisis y Difusión de la Economia Paraguaya in Paraguay (CADEP) encountered all of these problems when it conducted a PETS of the primary education sector in CADEP found that policymakers were very hesitant to provide the researchers access to data. When CADEP was finally able to obtain data from the government and the schools in their sample, they found that the financial records (especially those kept by school administrators) did not provide adequate information on money received and money spent within schools in Paraguay. Although these data problems prevented CADEP from completing a traditional expenditure tracking exercise, they did provide the research team with evidence of the significant problems with transparency of budget information in the country. The discovery also led CA- DEP to conclude that better tools were needed if civil society was to monitor the efficiency of public spending in educational facilities. At the current time, CADEP is working with active parent associations to develop ways to monitor transparency of budgeting within the schools and training modules to improve the financial record-keeping skills of school administrators.

32 24 Using PETS to Monitor Projects and Small-Scale Programs important finding that research teams can use when developing policy recommendations and advocating for change after the completion of their studies. Moreover, sharing this type of information with governments can represent an easy-win for research teams; inefficiencies in the system may be easier to address than outright corruption. Additionally, an important consideration in implementing any PETS is timing. Ideally, data collection should include annual data and span a series of years for the sake of comparability and to maximize data quality. The key is to be certain to collect the necessary data before it is destroyed or archived in such a way that researchers will have a difficult time gaining access to it. Preparing to Follow the Money PETS is an excellent tool designed not only to examine budget allocations, but more importantly to trace the flow of financial, human, and in-kind resources all the way down to frontline service providers such as teachers and health care workers. Mapping the landscape of budget allocations, spending, and receipts alone is only the first step in determining the quality and quantity of service delivery on the ground. We have begun to explore how PETS implementers can determine what budget data they need to collect and begin compiling that data; following the money generally requires the collection of secondary and primary data, including: National level budget data Sub-national level budget data Facility budget data Data on spending and service delivery on the ground The latter two types of data, in particular, are generally collected through the use of facilities surveys, interviews, and direct observation. Once the research team has mapped resource flows and determined the necessary sources of data, it can prepare to follow the money by selecting a sample for the facility survey and designing questionnaires.

33 A Guidebook 25 Selecting a Sample While it would be valuable to identify all incidence of leakage in the selected program or system that the PETS is exploring, this is generally not feasible given the limited time and resources of the implementing team. Instead, research teams should select a sample of facilities that benefit from the program funds and could be suffering from inefficiencies and corruption in the program. Selecting a statistically valid sample is a critical part of any PETS, and this is especially true for micro-level PETS such as those often undertaken by civil society organizations. Some basic power calculations can help inform research teams about how statistically significant their results will be; more subjects in a sample yields higher power. For civil society organizations, we suggest utilizing online resources and open-source programs that can help with the technical aspects of power calculations. 13 PETS with small samples can still be informative, diagnose problems such as leakages, and point to possible ways to mitigate these problems. However, limiting the scope of the study decreases the likelihood that one or two non-typical facilities in a sample will drive the results, and allows the CSO to make more conclusive statements about problems with public spending. General rules for limiting the scope of a smaller study include: Select one type of facility for the sample. For small samples, the facilities included should be largely homogeneous but should exhibit one or two differing characteristics across which you are interested in comparing leakages. For example, if a research team wants to consider differences in leakages between urban and rural schools, then the team will need to select a sample containing both urban and rural schools. However, the sample schools should be otherwise homogeneous. For this reason, it is best to focus on one type of facility, for example primary schools, or primary health clinics. 13 See Annex C for Power Calculations Resources.