Evaluation Study of SPARC support and the resultant progress made in implementing the MDGs Conditional Grants Scheme

Transcription

1 Evaluation Study of SPARC support and the resultant progress made in implementing the MDGs Conditional Grants Scheme December 2010

2 The opinions expressed in this report are those of the authors and do not necessarily represent the views of the Department for International Development

3 Content List Abbreviations and Acronyms Executive Summary Section One: Introduction... 1 Background... 1 SPARC support to the CGS... 2 Purpose of the evaluation study... 2 Section Two: Approach & Methodology... 5 Evaluation questions and sub-questions... 5 Theory of Change... 6 Rationale for selection of state visits... 7 Limitations of the study... 8 Section Three: Evaluation Findings... 9 Alignment of SPARC support... 9 The strategy of the Conditional Grant Scheme... 9 Relevance to SPARC Changes in the working practices of CGS Overview of CGS working practices Key changes in CGS working practices Adoption of SPARC s technical support and impacts Efficiency of processes of award, distribution and management State Government commitment to the CGS The provision of counterpart funds Commitment to the MDGs The contribution of SPARC s State-level support CGS and meeting the MDG targets: beneficiaries, gaps, additionality CGS beneficiaries and average cost Estimated cost of meeting the MDGs targets CGS and the additionality question Addressing the MDGs funding gap Section Four: Conclusions and Recommendations Strategic Relevance

4 Working practices & efficiency Evidence of impact Sustainability and impact Appendix One: Terms of Reference SPARC TOR: PRG-M&E-3-C Appendix Two: List of Stakeholders Appendix Three: List of References Appendix Four: Methods of data collection and analysis Appendix Five: Comparison of CGS strategic objectives Appendix Six: Summary of SPARC support (outputs) and adoption (outcomes) Appendix Seven: Mapping of linkages between CGS and SPARC State-level support (Kaduna) Appendix Eight: Summary of Assessments of Functional CGS Projects (from Links Study) The 7 projects visited

5 Abbreviations and Acronyms CGS Conditional Grant Scheme DFID Department for International Development, UK DM Database Manager DRG Debt Relief Gain ESSPIN Education Sector Support Programme in Nigeria KSDP Kaduna State Development Plan, LEEDS Local Economic Empowerment and Development Strategy LG Local Government M&E Monitoring and Evaluation MDA Ministry, Department or Agency MDG Millennium Development Goal MIS Management Information System MoH Ministry of Health MoU Memorandum of Understanding MoWR Ministry of Water Resources MTSS Medium Term Sector Strategy NCCGS National Committee on the Conditional Grants Scheme OSSAP-MDG Office of the Senior Special Assistant to the President on MDGs P&S Policy and Strategy PATHS2 Partnership for Transforming Health Systems PC Programme Coordinator PDG Programme Development Group, SPARC PEFA Public Expenditure and Financial Accountability PFM Public Financial Management PSM Public Sector Management SAVI State Accountability and Voice Initiative SDP State Development Plan SEAT State Evaluation and Assessment Tool SEEDS State Economic Empowerment and Development Strategy SG State Government SHoA State House of Assemblies SLPs State Level Programmes SPARC State Partnership for Accountability Responsiveness Capability TOR Terms of Reference

6

7 Executive Summary In 2005, Nigeria received international debt relief from the creditors of the Paris Club, amounting to around USD 1 billion in savings from debt servicing and interest payments. Nigeria undertook to spend the savings on pro-poor projects that would help achieve its Millennium Development Goals (MDGs) and subsequently the Office of the Senior Special Assistant to the President on Millennium Development Goals (OSSAP-MDGs) was established to manage and report on the use of these funds. The OSSAP-MDGs managed Conditional Grant Scheme (CGS) was established in 2007 as a special-purpose vehicle to accelerate the progress towards the achievement of the MDGs. Through the establishment of the scheme, Nigeria s Federal Government seeks to support the other tiers of government with additional funding to scale-up activities aimed at achieving the MDGs by The CGS, which operates as a matching grant, has three main objectives, including: i) invest in the MDGs at the sub-national level and ensure local ownership and sustainability, ii) empower state and local governments to carry out their constitutional responsibilities and iii) leverage public sector reform, public expenditure reform, and national planning for service delivery. The State Partnership for Accountability, Responsiveness and Capability (SPARC) has supported the CGS scheme since late 2008 in a number of ways, including: i) support for the evaluation of applications from States (2008 CGS Round), ii) the sensitisation workshop (2009 CGS Round), iii) development of criteria and evaluation of applications from the States (2009 CGS Round), iv) analysis of MDG-Expenditure in Nigeria s Federal, State and Local Government Budgets, vi) a study on strengthening the links between MDG activities and State reform efforts and vii) the provision of a Database Manager for the CGS, and latterly the Programme Coordinator. The purpose of this evaluation study is to, conduct an assessment of SPARC support and the resultant progress made in implementing the MDG Conditional Grants Scheme (CGS). One of the key conclusions from the study is that, overall, SPARC s Federal-level objectives align very well with those of the OSSAP-MDG and the CGS in particular. However, it will be important that this external support (through SPARC and others) helps the CGS better anticipate and rise to the challenge of achieving an impact on the MDGs, and in a way that is sustainable and at scale. Another key conclusion is that the CGS has made significant strides in improving its working practices and SPARC has played an instrumental role in many of these changes. In particular, SPARC has played a key role in the development of the qualifying criteria and improving the rigour of the bid evaluation process, helping to improve the quality of applications and CGS transparency. In general the support of SPARC has been highly valued, although there is a risk

8 that the CGS becomes overly dependent on such support for its routine work (e.g. the funding of workshops, the provision of consultants and technical staff). At the State level there has been significant progress, with the States showing an increasing commitment to the CGS providing not only 50% counterpart funding, but also a rising number of applications and approvals (which far outstrips the available funds). CGS application scoring criteria are very project-specific and less about the broader governance reforms and there is also little robust evidence of a spill-over effect in terms of stimulating better project performance across the MDAs. This raises concerns about the extent to which the CGS can leverage reforms. The main recommendations related to the strategic relevance of SPARCs support to the CGS scheme are: i) the CGS should develop an internal/ operational CGS strategy that sets specific priorities, helps monitor performance and refines action plans, ii) the existing strong cooperative relationship between the CGS Office and SPARC should be sustained without compromising independence and objectivity, iii) as 2015 draws closer, it will be important that the support provided by SPARC and others should help the CGS to better anticipate and rise to the challenge of achieving an impact on the MDGs, and in a way that is sustainable and at scale, and iv) SPARC should consider providing support that helps Government address strategic issues related to the MDGs, including supporting the Government s response in a more institutionalised way, and increasingly, to the agenda that will emerge for the post-2015 period. The main recommendations related to CGS working practices and efficiency are: i) further work on the qualifying criteria should focus on refinement rather than adding additional requirements, ii) some level of automation of the process would be desirable for improved efficiency and to enhance the sustainability of the process. Efforts should also be made to further involve civil servants from the key MDAs, iii) efforts should also be made to define appropriate metrics to aid the analysis of process efficiency and assist management decisions and iv) continued attention needs to be paid to ensuring that there is sufficient skills transfer from the SPARC-supported positions of the Programme Coordinator and the Database Manager. The main evidence of impact of SPARC support to the CGS scheme include: i) the two studies funded by SPARC have been useful, but more evidence and analysis is needed over the coming years and ii) data availability on CGS operations should be significantly improved to facilitate a more robust evaluation of the scheme s progress and achievements in future as well as impact of external support. The main recommendations related to the sustainability and impact of SPARC support to the CGS scheme include: i) projects funded in the 2007 and 2008 rounds should be followed up in order to learn lessons and ensure that the gains being made against the MDGs are cumulative, ii) it is recommended that a capacity assessment is undertaken to determine the gaps to be bridged in reaching the MDGs through the CGS and iii) more work is needed to enhance the commitment and capacity of the States to address the MDGs and how the CGS can better leverage increases in MDG expenditure. 8

9 Section One: Introduction Background In 2005, Nigeria received international debt relief from the creditors of the Paris Club. This amounted to some USD 1 billion in savings from debt servicing and interest payments. 1 As part of the agreement, Nigeria undertook to spend the savings (known as Debt Relief Gain, or DRG) on pro-poor projects that would help achieve its Millennium Development Goals (MDGs). 2 In the past, studies have shown that progress towards the achievement of the MDGs has been generally slow in Nigeria, and that unless drastic action was taken, Nigeria was unlikely to achieve the MDG targets by Moreover, there were indications that collaboration between the different tiers of government (Federal, State and Local) was generally weak; a situation that would hinder the achievement of the MDGs. In 2006, the Federal Government established the Office of the Senior Special Assistant to the President on Millennium Development Goals (OSSAP-MDGs) to manage its expenditure and ensure that it can report progress to the group of creditors. The Conditional Grant Scheme (CGS) was established in 2007 as a special-purpose vehicle to accelerate the progress towards the achievement of the MDGs. Through the establishment of the scheme, Nigeria s Federal Government seeks to support the other tiers of government with additional funding to scaleup activities aimed at achieving the MDGs by The CGS, which operates as a matching 4 grant, has the following specific objectives: Invest in the MDGs at the sub-national level and ensure local ownership and sustainability Empower state and local governments to carry out their constitutional responsibilities Leverage public sector reform, public expenditure reform, and national planning for service delivery 5 1 Applications Evaluation Report, October 2009, page 3. 2 See SPARC-OSSAP-MDG, Links Study, May 2010, page 12; and SPARC Report on MDG Budget Analysis, June 2010 page Davidson Iriekpen, June OSSAP-MDG, Countdown Strategy, page As Searle (2008) notes, many countries have Conditional Grant programmes that have similarities to the Nigerian CGS, but few place as much emphasis on the last objective as the Nigerian one does. 1

10 SPARC support to the CGS The DFID-funded State Partnership for Accountability, Responsiveness and Capability (SPARC) started operations in Nigeria in August 2008 although through its predecessor has supported the CGS office since the scheme started in Primarily, SPARC support is directed at the State level to achieve governance reforms that will enhance the efficiency and effectiveness in the use of public resources. SPARC currently operates in the five States of Enugu, Jigawa, Kaduna, Kano and Lagos. The programme works through three main workstreams: (a) Public Financial Management (PFM); (b) Public Service Management (PSM); and, Policy and Strategy combined with Monitoring and Evaluation (P&S/ M&E). The Federal component of SPARC works selectively with key Ministries, Departments and Agencies (MDAs) that have the potential to influence and support improved performance of State Governments. 6 Importantly, SPARC is only mandated to work with Federal-level MDAs that are able to provide support and incentives for better performance and governance reforms at the State level. The aim is to provide additional incentives for change whether by facilitating policy and regulatory development, providing leadership and examples, and by creating an appropriate framework to stimulate reforms. This particular focus derives from the belief that governance reforms will lead to the better use of Nigeria s resources, which are critical for reducing poverty and accelerating the progress towards the attainment of the MDGs. 7 In terms of Federal level support to the CGS, SPARC has focused on a number of key outputs. The contribution and impact of these is explored later in this report. For now though, the summary of the key outputs is: Support for the evaluation of applications from States (2008 CGS Round) The sensitisation workshop (2009 CGS Round) Development of criteria and evaluation of applications from the States (2009 CGS Round) Analysis of MDG-Expenditure in Nigeria s Federal, State and Local Government Budgets A study on strengthening the links between MDG activities and State reform efforts The provision of a Database Manager for the CGS, and latterly the Programme Coordinator Purpose of the evaluation study As the implementation of SPARC moves beyond its first year, it has become increasingly important to assess the impact of SPARC-supported governance reforms and their contribution to achieving the programme s objectives. 8 With this in mind, this pilot evaluation study is the first of a series that are planned for the coming years. 9 Such evaluation studies are intended to provide a synthesis of the available evidence and inform decision-making within the programme. The aim of this evaluation study is to, conduct an assessment of SPARC support and the 10 resultant progress made in implementing the MDG Conditional Grants Scheme (CGS). The evaluation has two main features: Firstly, the study is not an evaluation of the Conditional Grant Scheme (CGS) per se, but rather it focuses on the technical outputs supported by SPARC, and the lessons that can be learned for future support though in practice, the separation is less distinct. And secondly, because the evaluation study is a pilot, it is important 6 SPARC Overarching Concept Guidance Note 2009, page 5. 7 DFID Nigeria, Concept Note for Contracts Cntr & The term impact is used here to refer to its more common usage to mean a transformative change, rather than the more technical distinction used in M&E circles, and which differentiates between outcomes (the update, adoption and use of the outputs by the target group; a change in behaviour, practice or attitude) and impacts (a longer-term, wider societal effect). 9 A Strategy for Evaluation Studies, SPARC internal document, June SPARC Terms of Reference (PRG-M&E-3-C). 2

11 to learn lessons about how to improve the process as well as the end product (within time and resource constraints). These lessons have been summarised in a separate Innovation Diary. 3

12

13 Section Two: Approach & Methodology This chapter provides an overview of the approach and methodology used to undertake this study. Firstly, there is a summary of the evaluation questions, then the theory of change, the methods and tools used, and the limitations of the approach. Further details of the methodology are provided in the Inception Report, and in Annex 4. Evaluation questions and sub-questions Body The evaluation study focuses on two of the five OECD/DAC evaluation criteria. These are concerned with relevance (was the strategy and interventions of SPARC well aligned to the priority needs of the recipients?) and effectiveness (were the objectives of SPARC s support satisfactorily achieved?). The latter criteria have been sub-divided into two main parts: The first is concerned with the extent of the achievement, while the second is focused on the contributory factors, including the importance of SPARC s support. Table 1 provides a summary, with further details provided in the Terms of Reference (Annex 1). Table 1: Summary of TOR questions Relevance: The extent to which the intervention is the right approach, given the target audience needs and the political, institutional and economic context. Was the approach appropriate to the governance context (problems) of the state? How well aligned are the strategies and plans of SPARC support with those of the Office of the Senior Special Assistant to the President on MDGs (OSSAP-MDGS) and the MDG Conditional Grants Scheme (CGS)? Effectiveness: The extent to which the objectives were achieved, or are expected to be achieved To what extent were the objectives achieved / are likely to be achieved? What major factors What changes have been made to MDG CGS working practices and processes since early 2009? What improvement has there been in: a) development and application of criteria for accessing MDG CGS funds and b) the efficiency of processes by which grants are awarded, distributed and managed? How many beneficiaries of MDG CGS interventions have there been and what has been the average unit costs (e.g. CGS capital costs, OSSAP recurrent costs and SPARC support); compared to estimated needs within relevant MDG goals and estimated funding of the MDG CGS? What level of commitment has there been by SG in allocating 5

14 influenced the achievement or nonachievement of the objectives? their own resources in support of CGS funded initiatives? To what extent have/will Change Programme reform initiatives enable SG in SPARC supported states to meet the requisite criteria, in various areas (e.g. PFM, M&E), for accessing MDG CGS funding? To what extent have technical outputs delivered with the support of SPARC been adopted and what contribution have they made to changes in working practices and processes identified above? Theory of Change In broad terms, an evaluation can be approached by assessing performance against the intended design of the intervention. Underpinning the design is a theory about the intervention, and the changes that will result (known as the Theory of Change). 11 For SPARC s Federal-level support, this theory of change is articulated in several places (including the National Programme Logical Framework, the Overarching Concept Guidance Note, and the Federal Support Guidance Note). While SPARC primarily works with State Governments, the rationale for working at the Federal level is that, in Nigeria, there is a disconnect between Federal wishes and States compliance, and the need for negotiation, persuasion and incentives, since sanctions is not an option. 12 Therefore alongside SPARC s support to the States the aim is to add further incentives for change through parts of the Federal Government. In this way, Federal influence can have a critical role in respect of facilitating policy and regulatory development, providing leadership and examples, and creating an appropriate framework of incentives. But, it is important to be clear that SPARC is mandated to work with Federal Government agencies only to, assist in providing support and incentives for better performance in State level governance. 13 Figure 1: Simplified Theory of Change for SPARC Support to the CGS MDG-CGS Conditional Grants (Naira) CGS application & other processes B State Govt MDG projects Improved governance Improving MDG indicators A OUTPUTS Federal-level support Focus for evaluation study SPARC C Technical work streams OUTPUTS 11 A Theory of change is a term closely related to implementation logic a term that is often used to describe the links between inputs, the implementation strategy and the intended outputs and outcomes. It describes the assumed or desired causal relationship between the activity or policy and its (intended) goals. Accurate and clearly stated theories of change are seen as necessary for effective programming, as well as providing a useful basis against which to evaluate performance. 12 SPARC Federal Support Guidance Note, May 2009, page SPARC Overarching Concept Guidance Note, 2009, page 5. 6

15 Our understanding of the theory of change for SPARC s support to OSSAP-MDG and the CGS is summarised in Figure 1. This is a reconstructed intervention logic based on a review of the key documents, the TOR and discussions with the M&E Technical Lead and the Federal-level work stream. Its purpose is to help clarify the focus of the evaluation study, and provide a framework for interpreting the TOR evaluation questions. In summary, it is SPARC s Federal-level support to the OSSAP-MDG that is the main focus for the study (rather than the CGS per se). The emphasis of this study is on the contribution this support has made to improve the working practices of the CGS [A]. The rationale for SPARC s support is based on the expectation that the CGS can be used to leverage governance reforms in the States [B], as well as contribute towards attaining the MDGs. SPARC also operates in five states (Lagos, Enugu, Kaduna, Kano and Jigawa) and through its technical work streams the programme aims to improve governance at the State-level [C]. This may indirectly lead to State Governments being better able to access CGS funds. 14 Based on this framework, we can group the TOR evaluation questions accordingly: Table 2: Linking TOR questions with the Theory of Change Summary of topic/ question Link to theory of change TOR Question Alignment of SPARC Federal-level support [A] Q1 Evolution of the CGS s working practices Q2, Q3 Efficiency of CGS processes Q3 Adoption of SPARC s technical support & impact [A] Q7 State-level commitment to CGS Q5 SPARC s support and State Government access to CGS [C] Q6 CGS and meeting the MDG targets (gaps, [B] Q4 additionality) In short, this study is expected to provide evidence-based findings on the relevance and effectiveness of SPARC supports to CGS and the associated lessons to decision makers for internal reprioritisation and adjustment of the implementation of the governance reforms plans. Rationale for selection of state visits Based on initial discussions, the evaluation team has developed methods to answer each of the TOR questions, including the identification of key sources (secondary data, documents and interviews), and the process for analysis and validation. See Annex 4 for details. As part of the methodology, visits were undertaken to two States. The states of Lagos and Kaduna were visited to follow up on issues to do with the CGS Project Support Unit and Focal Persons, as well as with the technical work streams of the SPARC state teams. The selection of Lagos and Kaduna States (among SPARC s five focal states) is based on the following criteria: States from the North (Kano, Kaduna, Jigawa) and South (Lagos, Enugu). States considered high (Lagos, Kano, Jigawa) and lower (Kaduna, Enugu) performing by the CGS evaluation bid criteria States covered (Lagos, Jigawa) and not covered (Kano, Kaduna, Enugu) by the CGS Links Study a study that has more in-depth material on the links between CGS and State reform Logistics and accessibility taking into consideration the time and resource constraints 14 It is important to note that SPARC does not provide support to enable States to access particular funds. 7

16 Limitations of the study As with any study, there are a number of limitations due to methodological and resource constraints. When reading the report, these should be taken into account. The most important of these was the lack of secondary data. Indeed, while there is useful data on the scoring of the evaluation of bids by the CGS office, other sets of data were unavailable to the team. The most notable of these were the limited availability of data on efficiency (such as time and costs at each stage of the CGS process) and a proxy assessment was made based on the number of CGS staff and consultants used. Also, data on the Project Support Units in the States could not be circulated to the team due to the sensitive nature of this information but such data would have enabled a more detailed assessment of differences in the capacity and efficiency at the State level. And lastly, much of the financial data is based on budget allocations rather than actual expenditure and therefore any conclusions drawn from such figures need to be considered with care. 8

17 Section Three: Evaluation Findings This chapter presents the key findings from the evaluation study. The chapter starts with an assessment of the alignment of SPARC s support to the strategic priorities of the OSSAP-MDG and the CGS. This section is concerned with whether SPARC is providing the right support to the CGS, and the extent to which this support aligns to its own objectives (section 3.1). Following on from this, the chapter considers the effectiveness of the CGS, focusing on SPARC s contribution to improvements at the Federal level. Specifically, the section explores the evolution of the working practices of the CGS (section 3.2), the contribution that SPARC has made to these changes (section 3.3), and issues around the efficiency and sustainability of these processes (section 3.4). Then, the chapter turns to the State level. From here onwards, the chapter considers the commitment of the States to the CGS (section 3.5), and the extent to which SPARC support at the State level helps the State Governments meet the requisite criteria (section 3.6). Finally the chapter ends with an assessment of the extent to which the CGS provides a means for attaining the MDG targets (section 3.7). This provides an assessment of the scale of the MDG challenge, and the relative contribution of the CGS. Alignment of SPARC support 15 This section considers the extent to which SPARC support is aligned to the strategies and plans of OSSAP-MDG, and specifically the CGS. The objectives of the CGS are set out in several sources, and there is no single CGS-specific strategy document for the period of the evaluation (2007 to 2010). It is therefore necessary to derive the main objectives and strategic priorities from several sources (see Annex 5). Of these, the Countdown Strategy is now regarded as the de facto strategy for CGS and, although the Strategy is forward-looking beyond 2010 and refers to the MDGs, it does however provide a reasonable summary of the priorities to date. The strategy of the Conditional Grant Scheme According to the Countdown Strategy, the objectives of the CGS are threefold: (i) Investing in the MDGs at the sub-national level and ensuring local ownership and sustainability; (ii) Empowering state and local governments to carry out their constitutional responsibilities; and, (iii) Leveraging public sector reform, public expenditure reform [and] national planning for service delivery. 16 Of these, the third objective encompasses several different elements that are stated in more detail in the qualifying criteria of 2007, and subsequent revisions in This responds to question 1 of the TOR: How well aligned are the strategies and plans of SPARC support with those of the Office of the Senior Special Assistant to the President on MDGs (OSSAP-MDGS) and the MDG Conditional Grants Scheme (CGS)? 16 Countdown Strategy 2010 to 2015: Achieving the MDGs, pages

18 and Although described in different ways, the most important aspects of leveraging improvements in State-level governance (objective 3) are: Human resources, capability and planning Public expenditure management, accountability and transparency Harmonisation with MTEFs and national and state -level plans Projects that relate to state policy priorities Strengthening partnerships between the three tiers of government Investing in projects that are in areas of greatest need, and in consultation with LGs Commitment to maintaining the projects While the objectives of the CGS are clearly stated in the Countdown Strategy, they are not as concisely worded in the 2009 Implementation Manual. 17 While the manual is an improvement on previous guidelines, this mirrors earlier concerns about the need to be totally transparent in the CGS guidelines. In 2007, the Searle report wrote: 18 There are several sources that provide statements of objective for the MDG-CG Program. The main one seems to be the 2007 Guidelines for the program and the Memorandum of Understanding the Federal and State Governments signed before any funds were transferred. While not explicitly stated as such.... The report recommends that, States and Local Governments are entitled to know exactly what the Federal Government is trying to achieve through this Program... A clear and concise statement of objectives should be included in the introduction to the 2008 Guidelines. More importantly though, there is a difference between the objectives of the grant scheme (as set out in the Qualifying Criteria and the Countdown Strategy) and the strategic priorities of the CGS office to deliver against these objectives. This can be viewed as the difference between what the CGS aims to achieve, and how the office seeks to achieve it. While it is argued that the strategies of the CGS office are synonymous with those of achieving MDGs in general - as set out in the Countdown Strategy the CGS represents only a relatively small proportion of Nigeria s efforts for achieving the MDGs. Stakeholders cite several reasons for not having a formal CGS strategy, including that: (a) the CGS does not have the status of an MDA, with no permanent staff, and with the possibility that the National Assembly could end the scheme in any one year; (b) a lack of a formal strategy provides the Senior Special Advisor (SSA) with a freer hand to be responsive to political pressures; and, (c) regular OSSAP-wide retreats are in any case used to expand and provide focus for the work of the CGS. 19 Nonetheless, there is still value in having an internal/ operational CGS strategy that guides the office and is available to development partners. This does not necessarily have to be a formal or published document, but rather it could be an internal paper to aid management and staff. The essence of such a strategy would be to maximise the possible contributions of CGS given the limited resources allocated to the scheme. Without such a strategy it is difficult to set specific priorities, monitor performance and refine action plans, in common with best management practice. This gap also makes it difficult for development partners, including SPARC, to know how best to support the CGS office with the risk that external support has a tendency to be driven by donor preferences rather than those of the CGS. Relevance to SPARC The lack of a specific CGS operational strategy also makes it problematic to undertake a systematic assessment of how well external support aligns with the CGS. Nonetheless at a broad level, SPARC s Federal objectives align very well with those of the OSSAP-MDG and the 17 Implementation Manual 2009, pages Searle Report, 2008, page Based on interviews with staff of the CGS office and the SPARC Federal workstream. 10

19 CGS. 20 This is particularly the case for CGS s third objective (leveraging governance reforms at State-level), which is a means to achieve CGS first objective (investing in the MDGs at the sub-national level). This mirrors closely SPARC s purpose-level objective (i.e. The efficiency and effectiveness of selected state level governments use of public resources is enhanced ) and its overall goal (i.e. Nigeria s own resources used efficiently and effectively to achieve MDGs ). This is corroborated by the mapping exercise undertaken by SPARC 21, which concluded amongst the different possibilities considered that the OSSAP-MDG and especially the CGS were highly recommended for support. It is also the manner of SPARC s support that has helped create close alignment with the priorities of the CGS office. SPARC s Federal support has been highly responsive to demand, addressing the perceived needs and requests from the CGS (e.g. sensitisation workshops, refining the qualifying conditions, improving the bid evaluation process, etc). It is a generally held view that the support has been provided in a collaborative manner. Indeed, this is so much so that some of SPARC s support is almost inseparable from CGS operations (e.g. provision of Database Manager; joint staff/ consultant teams for bid evaluations; provision of 22 staff member for the LG Track). In the next 3 to 5 years the CGS is likely to face increasing scrutiny, particularly around whether the CGS actually does make a difference to the MDG indicators and issues around impact, sustainability, attribution and scale will come to the fore. At the moment, there seems to be a mostly positive consensus that the CGS is making good use of the DRG, and that it is leveraging change and additional resources at the State level though actual evidence that it is making significant and sustainable in-roads towards the MDGs is less conclusive (see also Box 3). As 2015 draws closer, the extent to which Nigeria s MDG targets are met (or not met) will undoubtedly put the CGS under greater pressure to deliver more and to deliver it better. Harnessing external support that can help the CGS better anticipate and address these challenges will become increasingly important. Key findings: At a broad level, SPARC s Federal objectives align very well with those of the OSSAP-MDG and the CGS. The manner of SPARC s support and the cooperative relationship with the CGS office has helped foster this close alignment. While there are reasons given for not having a formal CGS strategy, there is value in having an internal/ operational CGS strategy that sets specific priorities, helps monitor performance and refines action plans. It may also help mitigate a tendency for external support to be driven more by donor preferences than those of the CGS. Over the next 3 to 5 years, as 2015 draws closer, it will be important that the external support (through SPARC and others) helps the CGS better anticipate and rise to the challenge of achieving an impact on the MDGs, and in a way that is sustainable and to scale. Changes in the working practices of CGS 23 This section examines the key changes that have been made to the CGS s working practices and processes since the inception of the scheme in In the first part, there is an overview of the working practices and processes, and this is then followed by an assessment of the key developments in past three years; particularly focusing on changes in the qualifying criteria and its application. 20 A view supported by the DFID Output-to-Purpose Review, May Mapping of Federal Support and Incentives to States, March 2009, page Based on several interviews with CGS staff (6/10/10 and 12/10/10) and the SPARC Federal workstream. 23 This section responds to question 2 and 3a of the TOR: Q2: What changes have been made to MDG CGS working practices and processes since early 2009?, and, Q3: What improvement has there been in the development and application of criteria for accessing MDG CGS funds? 11

20 Overview of CGS working practices The working practices and processes of the CGS comprise of the following key elements: Qualifying criteria: The criteria are in two parts, namely State qualifying indicators and project qualifying specific indicators. The former are the conditions that a State must meet in order to qualify for CGS funds while the latter are the conditions which a project must meet to be admitted for funding. The criteria have been refined over the years since the inception of CGS in 2007 to make them more rigorous and explicit. 24 The qualifying criteria were incorporated into CGS Guidelines during 2007 and 2008 and into the Implementation Manual in Sensitisation workshops: Following the development of the qualifying criteria and the guidelines/ implementation manual, a workshop is organised to assist State officials in understanding the qualifying criteria and the CGS procedures. The objective of the workshop is to enable States to develop and submit quality applications along with the requisite supporting evidence. Call circular: A circular calling on States to submit applications for Conditional Grants is issued by the Secretariat of National Committee on the Conditional Grants Scheme (NCCGS). The call circular will state the total amount approved for disbursement to all States for the year of the application; the maximum amount each State will be able to apply for; the eligible project areas for which applications are invited; the time-limit for the submission of applications; and any other information that will serve as requirements for States in preparing their applications for the year. 25 Evaluation of bids: The evaluation of applications is carried out by a technical team comprising OSSAP-MDGs/ CGS staff and SPARC consultants; and from 2009, some experts from the MDAs. Award of funds and funds disbursement: Subject to the approval of the Federal Executive Council (FEC), awards are made to States for projects that met the qualifying criteria. The disbursement of funds is conditional on a State fulfilling certain requirements, including: a signed Memorandum of Understanding (MoU); a Bank Guarantee Bond; submission of e-copies of Location Sheets and e-copies of work plan; evidence of the State Cash Counterpart Contribution; and a signed letter of authority to access directly the statement of Bank account of CGS Project Account from the Bank where the account is domiciled. 26 Oversight, monitoring and evaluation: The main platform for monitoring and evaluating CGS projects is the Overview of Public Expenditure on NEEDS (OPEN). The OPEN aims at ensuring effective spending of the Debt Relief funds, including monitoring and evaluation of the outputs and impact of the expenditures. M&E is yet to be institutionalised in many of the States despite it being a requirement of CGS. Monthly report requirement: The financial report and project report are required to be prepared by the Project Support Unit (PSU) and submitted to the Secretariat of the NCCGS on a monthly basis. The Implementation Manual specifies the minimum contents of the reports. 27 Monthly stakeholders meetings: A meeting of the CGS Focal Persons is held monthly to take stock of progress of projects implementation and to review challenges. The meetings also serve as capacity building/ training and knowledge sharing forums as presentations are made on pertinent issues. The meetings are rotational across different geopolitical zones of Nigeria. 24 For details of the qualifying criteria for : CGS Guidelines 2007 and 2008, plus the Implementation Manual CGS Implementation Manual, August 2009, page CGS Implementation Manual, August 2009, page CGS Implementation Manual, August 2009, pages

21 Key changes in CGS working practices There have been a number of considerable changes in the working practices of the CGS since its inception in One of the most significant has been the tightening of the qualifying criteria in 2008, with more conditions added in Over and above ensuring discipline and operational effectiveness of the CGS, these qualifying criteria are seen as a powerful tool for influencing governance changes at the State level. 28 The main changes in the qualifying criteria have included: For the 2008 Round: 50% cash counterpart funding requirement coupled with the provision for the 50% cash counterpart contribution to be passed by the State House of Assembly (SHoA) in the 2008 budget. Together these changes have had a significant effect in ensuring that States commit to providing counterpart funding. In 2007 the amount of counterpart funding was mostly in kind (e.g. office space, personnel) and was unrecorded at the Federal level. By 2008 and 2009 the cash contribution has risen to Naira 24.4 and 26.8 billion, respectively. Whether this has increased the total commitment of States to expenditure on the MDGs (as opposed to CGS projects specifically), remains elusive (see section 3.5). Maximum amount accessible reduced from Naira 3 billion to Naira 1 billion. This change in the ceiling amount accessible by each State appears to have had no perceptible effect on the number of States applying with applications actually increasing from 28 in 2007 to 35 in Content of application package enhanced with the requirements for evidence of: (a) An enabling institutional structure 29 and capacity to account for CGS funds in a transparent manner; (b) Human capacity development, facilities and equipment. This is said to have improved projects implementation. Projects justifications to include relevant sections of SEEDs, LEEDs and/ or equivalent policy documents as well as gap analysis. The aim is to compel the articulation and documentation of state strategies, policies and plans by States that did not have such documents. This is said to help projects align to policy priorities, though in practice. A promissory note duly signed by the governor outlining detailed plans for maintenance of the project. The aim is to promote project sustainability, although it is too early to tell whether this has been achieved. Evidence of consultation and collaboration between different tiers of government in project selection and location. The aim is to avoid duplication of effort, and ensure that project selection is optimised. Designation of CGS state focal person from Ministry of Economic Planning and Budget was compelled, with Project Support Units (PSU) established in most States. For the 2009 Round: Detailed project specification, designs and costing made compulsory for all projects. The aim is to ensure that cost information on project affordability is better assessed and taken into account when selecting bids. More stringent conditions specified for fund disbursement. This includes the introduction of the Bank Guarantee bond; evidence of the State s cash counterpart being lodged in the account; and, a letter authorising the NCCG to have direct access to the statement of the bank account of each CGS project. This was introduced to further ensure that the 50% counterpart funding is made available as part of the State s commitment to the implementation of the project. 28 A view supported particularly by the Links Study, but also from interviews with CGS staff, SPARC consultants, and staff of the SPARC Federal-level component. 29 As a result of this requirement, dedicated Project Support Units have been established in all States. 13

22 The Implementation Manual was developed in 2009, providing a more comprehensive and explicit set of information to the States than from the guidance given in previous years. In addition to the qualifying criteria, there have been significant changes in the process for evaluating the bids. SPARC has had a dominant influence in this process, particularly in terms of resource inputs. In 2009 evaluation for example, SPARC provided eleven consultants as against three OSSSAP-MDGs/ CGS staff. 30 More recently though, experts from other MDAs have been used to assess the applications. This is both less expensive than external consultants, less reliant on external donor funding, and therefore probably more sustainable. There have also been changes in the enabling institutional structures, such as the Project Support Unit at the State level, plus the stakeholder monthly meetings (which are rotational). These are said to have proved valuable in increasing capacity and share knowledge between 31 States and the CGS office, but actual evidence of this change is harder to verify. Key findings: There have been many changes in the CGS qualifying criteria over the past three rounds, particularly in 2008 and consolidated in the 2009 Implementation Manual. These changes have helped make the application process more robust and have seemingly improved the quality of the applications. One of the most significant of these changes has been the requirement for 50% counterpart funding along with the Bank Guarantee. This has led to a dramatic increase in counterpart funding from the States although whether this has also led to a corresponding commitment to increasing expenditure on the MDGs remains to be seen. Apart from the qualifying criteria itself, the process of evaluating the applications has also changed significantly. In particular, additional resources and expertise have been introduced through the use of joint CGS staff/ consultant teams. This has resulted in a more rigorous and transparent process, and one that has not been challenged by the applicants (to date). In the 2009 Round, the evaluation of bids has also included experts from MDAs a more cost-effective, and perhaps more sustainable, alternative to external consultants. Apart from the qualifying criteria and the improvements in the bid evaluation process, the other important improvements have included improvements in the State-level institutional structures (through the PSU), and the introduction of stakeholder monthly meetings which are said to have improved capacity and knowledge sharing. Adoption of SPARC s technical support and impacts 32 SPARC support for CGS effectively started in Since then, SPARC and the CGS office have worked closely together on a number of key processes. In particular these include the development of the qualifying criteria, the sensitisation/ training workshops and the evaluation of applications as well as two studies, and the provision of the Database Manager (DM) and Programme Coordinator (PC). The support is provided directly to CGS operations at the Federal level with the expectation that the effect of this support will cascade down to the States in the form of governance reforms. In the absence of any clearly defined and documented operational strategy for CGS, a case for support is usually made by the Scheme s management on an ad hoc basis which is then considered and approved or rejected by SPARC (see also section 3.1). 30 CGS Evaluation Methodology, page Based on discussions with the CGS office, PSUs and other stakeholders in Kaduna and Lagos States. The evaluation team were however unable to get access to the PSU assessments, which are said to show a more varied picture. Plus, there seems to be no systematic monitoring or feedback that could be drawn upon to assess the monthly meetings. 32 This section responds to question 7 of the TOR: To what extent have technical outputs delivered with the support of SPARC been adopted and what contribution have they made to changes in working practices and processes identified above? 14

23 Over the period under this evaluation, SPARC has produced a number of technical outputs in furtherance of its goal for CGS. This section examines the technical support with particular reference to the extent of their adoption and their impacts based on available evidence and interviews with stakeholders. The key technical areas of support provided are summarised as follows: Funding of some key positions in CGS. SPARC is currently funding two positions in CSG, namely Programme Coordinator (PC) and Database Manager (DM). The former provides technical expertise to ensure best practice operations of CGS. For example, the PC was a member of the Working Group for the evaluation of the 2009 round of CGS applications and he led the study on the analysis of MDGs expenditures in Nigeria s Federal, State and Local government budget (as described below). The DM is anchoring the management of the emerging massive data in CGS and has facilitated the design of uniform templates for reporting. CGS office has confirmed that the work of the DM is facilitating prompt responses to urgent data/information requests, especially from the SSAP-MDGs though observations of the database show that much more work is required. Both of these positions are now recognised as indispensable to CGS, with appropriate civil servants (apparently) acting as understudies (with a view to taking over upon the expiration of their contracts). This succession arrangement was confirmed as an integral part of the contracts for the respective SPARC consultants working with CGS. It is however too early to assess whether this will lead to the sustainable transfer of skills. Qualifying criteria development support: It is the development and refinement of the qualifying criteria that SPARC has had the greatest involvement, and perhaps impact. Over the years, SPARC has participated in the refinement of the qualifying criteria to make it progressively more effective. The qualifying criteria are being used as a tool to further governance reforms at the State level. Indeed, there is some evidence that this is occurring, particularly in terms of PFM reforms (see Box 1). But, there are other examples of change that have occurred and for which it is harder to establish the extent to which this is due to the CGS. For example, many states now have audited accounts as a way of meeting the CGS condition, and procurement legislation has either been enacted or in the process of being enacted. It is debateable whether these changes are really due to the CGS. Nonetheless, it is generally thought that the CGS process has become more transparent and objective while the evaluation results have become incontrovertible as a consequence of SPARC support. Sensitisation workshop: This is a training forum for State Government officials responsible for preparation of CGS applications. The workshop is meant to assist the officials in understanding the criteria specified in the Implementation Manual and to encourage them to develop well designed applications that clearly articulate MDGs investments and ensure that investments can be readily monitored and accounted for. 33 Stakeholders confirmed that the quality of applications from States has improved due to the sensitisation workshops, especially in the areas of costing and provision of supporting evidence. 34 The proportion of successful applications reinforces this overall notion that the quality of applications has improved. Only 19 (or 68%) of the 28 states that submitted applications in the 2007 round of CGS were successful; but the proportion rose to 97% and 94% in 2008 and 2009 respectively. 35 It is also believed that the knowledge gained by the civil servants that attended the workshop is being institutionalised in their respective offices through hands-on. Evaluation of applications: SPARC has supported the bid evaluation process by supplying consultants to provide technical expertise. SPARC provided 11 consultants for the 2009 CGS round; the use of consultants has been accepted as vital to give credibility to the process. It is worth noting that there has not been any petition from other quarters since the inception of the CGS; which is laudable especially given the grant sizes involved. The use of consultants may also have insulated the OSSAP-MDGs from external criticism. Plus, the transfer of skills from SPARC consultants is likely to have led to some capacity building gains by CGS staff. In 33 See: SPARC TOR: FED/MDG/1/A, Sensitisation Workshop for 2009 CGS Round, August Based on discussions with CGS staff, SPARC consultants, staff of the SPARC Federal-level component, and stakeholders met at the State level. 35 See: Links Study, pages 18-21; The Story of CGS, page

24 the 2009 CGS round, experienced civil servants from MDAs were incorporated into the process. In future, it is anticipated that this may reduce dependence on SPARC consultants and this may enhance the sustainability of the process. This is explored in more detail in section 3.4. Analysis of MDG expenditure: This study was supported by SPARC to ascertain how CGS is influencing government spending on MDGs. SPARC provided 5 consultants for the study. As a result of the findings, the CGS office has decided that future sensitisation training workshops will include M&E experts to underline its importance and to develop capacity. The study provides information on the funding gap for meeting MDGs targets; a vital piece of information in the development of the OSSAP-MDG Countdown Strategy. The CGS office plans to replicate the study to cover more States and Local Governments, because as it currently exists the study has a number of limitations (particularly the use of budget data rather than actual expenditure, the extrapolation of figures based on not all States and even fewer Local Governments, and the lack of trend data). The Links study: SPARC supported this study that was completed in May As a result of the study, CGS management is better informed about the impact CGS activities are having on the wider governance reforms at the State level and the areas where improvements are needed though some of the evidence of the resulting changes is more tenuous, and a few stakeholders felt that the study was more useful for donors than the CGS. Nevertheless, the study has called attention to the need to refine the qualifying criteria relating to PSM, P&S and M&E to make them more specific and direct (i.e. like those relating to PFM). In this regard, there is a plan to incorporate the study recommendations into the next round of the qualifying criteria refinement. Box 1. The links between the CGS and governance reforms The Link Study was conducted in 2009 to assess whether the CGS (and particularly the qualifying criteria) was having an effect in terms of leveraging governance reforms at the State level. The study claims that the CGS approach has impacted on wider governance and public sector reforms in significant ways, albeit with varying degrees amongst the States. A more careful reading of the case study examples in the report, plus our own visits to the States, suggest that the claim of impacting in significant ways may be overstated given that many of the links are somewhat tenuous or difficult to evidence in an objective manner (see Annex 8). Nonetheless, the key findings of the study were: Public Financial Management: The specificity of CGS criteria meant that the impact could be more readily assessed than for other areas of the criteria. The study claims that the criteria have resulted in the deepening PFM reform in the States. Policy and strategy: Under the criteria, CGS activities are required to be linked to the policy and strategy of each State, including linkages to the MTSSs under preparation (if applicable). But elsewhere in the report, the evidence is not so clear-cut. For example, in Jigawa State. Public Service Management: In contrast to the specificity of key PFM issues, CGS guidelines only require evidence of a plan for human capacity development in respect of public service management. The report recommends the need for greater specificity in this respect. Monitoring and Evaluation: In their CGS applications, the States are required to provide M&E arrangements and ensure a certain degree of commitment. Translating this into practical action has been a challenge and while there are efforts at the project level, progress is much slower in terms of institutionalising M&E reforms. In conclusion, the support of SPARC in providing consultants to better specify the qualifying criteria in 2008 and 2009 have played an important role in these changes, alongside the provision of consultants to the sensitisation workshops and the evaluation of applications. Together, this has helped to improve the rigour and utilization of the criteria to bids. Overall, most observers agree that the grant assessment process is now systematic and much more 16

25 rigorous improving both the quality and the transparency of the CGS. 36 As the CGS has expanded and become more prominent, this has been important to ensure fairness in the selection process and to avoid disputes about the allocation of funds. Key findings: SPARC and the CGS office have worked closely together on a number of key CGS working practices. In particular the support of SPARC has been instrumental in the development of the qualifying criteria and the rigour and utilization of the criteria during the bid process. Most observers agree that the grant assessment process is now much more systematic and robust, leading to improvements in the quality and the transparency of the CGS. SPARC s funding of key positions the Database Manager and the Programme Coordinator is also said to have been indispensible, although it is too early to assess the full impact in terms of skill transfer. The Programme Coordinator is a relatively new position and while the Database Manager has been in place for longer, observations of the database indicate that much more work is still required. The two studies funded by SPARC have been constructive attempts to get to grips with two central issues: Does the CGS working practices leverage governance reform at the State level, and, are the State increasing their commitment to the MDGs in terms of increased expenditure? The paucity of data however limits the utility of these studies and particularly the MDG study which is based on budget allocations (which are not credible in many States). The Links Study has usefully brought to attention the need to refine the criteria relating to PSM, M&E and P&S, although it too has limitations in terms of the sometimes heroic claims that the CGS has led to direct governance reforms. While the support of SPARC has been highly valued, there is a risk that the CGS becomes overly dependent on such support for its routine work (e.g. workshop funding, provision of consultants and technical staff) Efficiency of processes of award, distribution and management 37 This section examines the efficiency of processes by which grants are awarded, distributed and managed. While clearly there have been a number of improvements in the CGS process, as described in the preceding section, the additional rigour may also have had a trade-off in terms of efficiency. Furthermore, this section touches on some related issues, in terms of the reliance on external consultancy support and the likely sustainability of the processes in subsequent years. Firstly though, it is important to define how efficiency is assessed for the purposes of this study. Process efficiency can be assessed using different metrics depending upon the purpose of the study and data availability. The conventional metrics include: the time it takes to complete a process vis-à-vis best practice benchmark; the resource inputs (labour, facilities, time, etc) divided by the cost of completing a process, compared to a best practice benchmark; or cost of completing a process during different time intervals. The unavailability of data is a major challenge for assessing the efficiency of the Conditional Grant process (from applications and award, to disbursement management and oversight). In this case, only human resource input data was available to the team, 38 and even then, for limited activities or sub-processes of CGS working practices, namely: 36 As confirmed by several staff of the CGS office and SPARC-supported consultants. See also: Report on the evaluation of the bids 2008 (page 13) and 2009 (page 18). 37 This section responds to question 3b of the TOR: What improvement has there been in the efficiency of processes by which grants are awarded, distributed and managed? 38 The human resource data are in terms of number of CGS staff/civil servants, number of CGS staff/civil servants days, number of consultants, and number of consultants days per the respective activities/subprocesses. 17

26 Criteria and guidelines development Sensitisation workshop Applications evaluation Post-evaluation workshop and funds disbursement Furthermore, a caveat is necessary at this stage: Given the evolution of CGS working practices and processes discussed in the preceding section, the job contents of the activities/subprocesses listed above would have changed from year-to-year. Accordingly, changes in resource inputs for the respective activities/ sub-processes for each year may not be strictly due to changes in the efficiency (or inefficiency) of operations. The analysis shows that the activities are getting increasingly intensive in terms of the number of CGS staff (see Figure 2). The only exception is the post-evaluation workshop and fund disbursement, for which staff inputs remained constant during 2008 and The evaluation of bids has been the most staff intensive both in absolute and in incremental terms. This may be due to the larger number of applications and projects to evaluate. For instance: The number of States applying for CGS rose from 28 in 2007, to 35 and 36 in 2008 and 2009 respectively The number of projects increased from 226 in 2008 to 344 in 2009, or a 52.2% increase 39 The qualifying criteria have become increasingly stringent, necessitating equally rigorous and time consuming evaluations of the bids (e.g. the project cost information required in the 2009 round) In general, number of consultants has increased but average number of days per consultant has dropped, leading to overall drop in the aggregate consultants days except for bid evaluation. For the bid evaluation process, the consultants input increased throughout 2007 to 2009, both in terms of number of consultants and consultant days. This raises the possibility that the evaluation of bids process is becoming increasingly dependent on consultant s inputs. So while, the process itself has become more rigorous and has made better use of expertise, it is important that ways are sought to ensure that the process becomes mainstreamed within CGS and the civil service and likewise, does not become overly dependent on external support. There are early indications that this is beginning to happen, but further work will be required. 39 See: (a) The Linkage Study, pages 16-21; and (b) The Story of CGS, page

27 Figure 2: CGS Working Practices and Human Resource Inputs Table 3 presents the consultants to CGS staff ratios for two of the sub-processes for which all necessary data were available, namely sensitisation workshop and evaluation of applications. The ratio of consultants to CGS staff/ civil servants increased slightly in 2008 and dropped in 2009; while the ratio of consultants days to CGS staff days dropped throughout 2007 to 2009; suggesting there is some reduction in the dependence of CGS process on consultants/ SPARC. Table 3: Consultants to CGS staff ratio for sensitisation workshop & application evaluation Number of CGS Staff/Civil Servants Number of Consultants Number of CGS Staff/Civil Servants Days Number of Consultants Days Consultants to CGS staff/civil Servants (%) 120% 133% 83% Consultants Days to CGS staff/civil Servants Days (%) 138% 113% 75% Source: Conditional Grant Office, Abuja. Based on the prior analysis, it is reasonable to conclude: Firstly, that it is difficult to be categorical about the trend of process efficiency regarding CGS working practices because the job contents of the respective activities/ sub-processes have been changing over time. Secondly, of all the activities/ sub-processes examined, the evaluation of applications has become more intensive in terms of consultant inputs and CGS 19

28 staff/civil servants time. It appears that some level of automation of the process would be desirable for improved efficiency and to enhance the sustainability of the process. However, the ratios of consultants to CGS staff and consultants days to CGS staff days have reduced in the last round, suggesting a lessoning of the dependence of CGS process on consultants (SPARC). This apparent reduction in the dependence on consultants (SPARC) could be due to increasing use of MDAs experts. And lastly, data paucity has limited the robustness of the analysis of process efficiency in this study. In future, efforts should be made to define appropriate metrics for CGS process analysis and to systematically collect data on such metrics to aid the analysis of process efficiency. To further aid the analysis of CGS process efficiency for management decisions, a detailed process map of the working practices needs to be developed showing the main activities and tasks as well as their interrelationships and resource inputs requirements. Key findings: The analysis shows that the key processes of the CGS are getting increasingly intensive in terms of the number of CGS staff and the number of consultants. The ratio of consultants to CGS staff/ civil servants increased slightly in 2008 and dropped in 2009; while the ratio of consultants days to CGS staff days dropped throughout 2007 to 2009; suggesting that perhaps there is some reduction in the dependence of CGS processes on consultant inputs (and SPARC support). Over the years, the CGS has made better use of expertise to increase the rigour of its working practices, but it is important that ways are sought to ensure that these practices become mainstreamed within the CGS and the civil service more broadly (e.g. the use of MDA experts). To this end, it appears that some level of automation of the process would be desirable for improved efficiency and to enhance the sustainability of the process. Efforts should also be made to define appropriate metrics to aid the analysis of process efficiency and assist management decisions. For instance, a detailed process map of the working practices needs to be developed showing the main activities and tasks as well as their interrelationships and opportunities to resource inputs requirements. State Government commitment to the CGS 40 There are several ways to assess the level of commitment of State Governments in allocating their own resources in support of CGS-funded initiatives. Firstly, this section reviews the provision of counterpart funding, and the extent to which this represents a commitment. The section then goes on to consider the State s allocation of funding towards the MDGs more broadly and to assess whether the CGS is having a displacement effect or instead is leveraging greater allocations towards attaining the MDGs. Finally, it was hoped that an assessment could have made use of the State s commitment in terms of institutional support through the Project Support Unit (PSU), and through policy and strategic commitments to the MDGs. This has not been possible as the data on the PSU assessments could not be made available, and a thorough review of policy commitments (i.e. beyond a superficial review of the documents) was not possible within the constraints of this assignment. Instead, the visits to Kaduna and Lagos provide added insights into these aspects. 40 This section responds to question 5 of the TOR: What level of commitment has there been by SG in allocating their own resources in support of CGS funded initiatives? 20

29 The provision of counterpart funds In 2007 the criterion for counterpart funding stated that it should be, either in cash as a percentage of the project cost, or as a contribution in kind towards the execution of the project. This was judged to be most unclear, and as a consequence only a small number of States provided counterpart funds said to range from 10% to 20% (for example, Anambra, Bauchi and Ekiti States). 41 Following a Presidential Directive in 2008, the qualifying criterion was modified so that all States were required to provide a 50% cash counterpart contribution. This 50% requirement has been accompanied by other provisions to ensure that it takes place (including a Bank Guarantee as evidence that the cash counterpart contribution has been lodged into the account, plus a letter authorizing the NCCG to have direct access to the statement of bank account where it is located). It seems that the 2008 Presidential Directive has not affected the demand from the States. Indeed, quite the reverse. The number of applications and approvals has increased (see Figure 3). There has been an Figure 3: CGS Applications and Approvals Applications and Approvals under the CGS, (Naira, billions) Applications by Value (Naira, billion): Sources: Links Study (pages 18-21) and, The Story of CGS (page 18). Note the values include counterpart funding. increase in approved CGS funds over the years from 2007 to 2009; from Naira 18.4 billion to Naira 53.6 billion respectively (or 26.8 billion in 2009 with the 50% counterpart funding not included). Demand for the grants has substantially increased. The number and total value of the applications for the CGS has far outstripped the available funds, increasing from Naira 74.1 billion in 2007 to Naira billion in 2009 and from 226 project applications in 2008 to 344 applications in By 2009, every State had participated in at least one round of the CGS, with the number increasing to virtually the entire Federation and over 90% of State accessing the funds every year. 42 Commitment to the MDGs Although the counterpart funding and demand for the CGS has increased, this does not necessarily equate to an increased commitment to the MDGs at the State level with additional funds being allocated to key MDG sectors. It could for example be that funds are being reallocated. The Budget Allocation Study shows that in 2009, the total commitment of 23 States to the MDG sectors was Naira 494 billion, or 22% of the budget allocation. This is then calculated for all States as: Naira 848 billion. But study would need to be repeated to understand the trends and whether States are increasing their allocations to key MDG sectors. Furthermore, as the study is based on budget allocations and not expenditure figures, the discrepancies with actual budget execution can be significant and in some cases budget figures can be misleading (see Box 2). As the Links Study itself finds: 43 The inadequacy of data that is consistently reflected in assessments of progress towards achieving the MDGs since 2006, was confirmed in all the States - consensus on the imperative 41 Links Study 2009, pages The Story of CGS, page Links Study, page 9. 21

30 of quality data both for planning and for monitoring implementation progress in respect of MDG programmes and projects as well as governments development programmes and projects. Box 2. MDG allocations in Kaduna State 44 In Kaduna State, the recent annual budgets have proven financially unrealistic, and systematically over-estimate sector budgets and underestimate central overheads a real budget credibility problem. While they may reveal planning aspirations that might materialise over a lengthy time period, these annual budgets provide very little useful information on which to calculate expenditure against the key MDG sectors. This presents serious challenges to the MDG Budget Allocation Study and the figures used for projecting the MDG+10 estimations. Yet ironically, Kaduna State has a relatively positive story to tell about MDG expenditure, particularly in the health and education sectors. Figures for recurrent and capital expenditure show that the State Government has spend steadily increasing amounts on health (11% of total expenditure in 2009, compared to 6% in 2004). The expenditure on education is more variable, but has risen over the same period (to 19% of total expenditure in 2009, compared to 15% in 2004). In other sectors, such as water, agriculture and roads, the expenditure figures are more variable, and subject to political vagrancies. More recently Kaduna State is adding a MDG tracking code to its annual budgets, and although many items are difficult to allocate, 45 this should allow better tracking of budgets and expenditure against the MDGs. Key findings: Since the Presidential Directive in 2008 that required the States to provide 50% in counterpart funds, demand for the CGS has not been affected. Indeed, the numbers of applications and approvals have increased. The number and total value of the applications for the CGS have far outstripped the available funds. While counterpart funding and the demand for the CGS have increased, these do not necessarily equate to an increased commitment to the MDGs at the State level (i.e. with additional funds being allocated to key MDG sectors, and prioritisation given to the MDG targets). The evidence based on the MDG Expenditure Study is potentially misleading as it is based on budget allocations which can differ substantially to expenditure in some States. More work is needed to explore the commitment and capacity of the States to address the MDGs, and how the CGS can better leverage increases in MDG expenditure. The contribution of SPARC s State-level support 46 This section considers the extent to which SPARC s work at the State level supports the CGS in terms of leveraging governance reforms and helping the State Government to meet the requisite qualifying criteria. This is an important distinction, as it is not part of the mandate of SPARC to provide direct support to the CGS to directly increase the State Government s access to the fund (such as through the MDG Office or PSU located in the State). 44 Sources: Stakeholder interviews in Kaduna State, plus analysis draw from the SPARC PFM Factsheet 02: Kaduna State Government Payments Although some may argue that the overuse of the miscellaneous category may in the end render the analysis useless. 46 This section responds to question 6 of the TOR: To what extent have/will Change Programme reform initiatives enable SG in SPARC supported states to meet the requisite criteria, in various areas (e.g. PFM, M&E), for accessing MDG CGS funding? 22

31 At a broad level, SPARC s support to governance reforms closely fits the ideals of the CGS (see section 3.1 for details). But at the State level, the extent to which SPARC-supported reforms will enable States to meet the requisite criteria of the CGS is less pronounced. In both Lagos and Kaduna States, there was no evidence of any significant involvement of SPARC in CGS activities. SPARC works jointly with the State Governments through the Change Programme. As one stakeholder observed, If there is any compatibility between the Change Programme and the requirements of the CGS, then this is unintentional. 47 For Kaduna State, this evaluation undertook a detailed mapping assessment to identify the links between the CGS qualifying criteria and SPARC s State-level support (see Annex 7). The qualifying criteria are split into State Indicators (a possible 100 points) and Project Indicators (again, 100 points), and provide a list of items against which the CGS application is scored. The findings of this assessment show that only about 25% this requisite criteria could potentially be affected by SPARC support and perhaps less than 10% are likely to be strongly affected. This indicates that even if SPARC was performing well, then still its ability to enable the State Government meet the qualifying criteria of the CGS is relatively limited. This is because: Much of the scoring criteria is project-specific rather than about the broader governance reform of the State. Even within the State Indicator criteria (which is meant to be less project-specific), there are 10 points that are allocated based on the assessed capacity of the PSU, and 18 points for providing evidence for the 50% cash contribution. A further 12 points are given for evidence of a commitment to fund the operational costs, and so on. These few examples alone represent 40 out of a possible 100 points for the State Indicator criteria that would not score higher because the State Government is reforming. Where the criterion is about wider governance reforms, many of them are imprecise (especially for PSM, P&S and M&E). 48 For example one criteria focuses on, the existence of a State policy that prioritises the MDGs an existence that makes no reference to quality of the policy or the extent to which it drives prioritisation and resource allocation (both areas where SPARC might make a contribution). Similarly, other criteria are more focused on the CGS project per se. So for example, the M&E criteria is specifically for project rather than whether the M&E is improving and being better institutionalised within the MDA which again could be more directly facilitated by SPARC support. In contrast, it is only the PFM criteria that are precise, and as a consequence it is easier to match the criteria with specific areas of SPARC support at the State level. And lastly in some areas like auditing, SPARC is not working at the moment (according to the workplan, year 1) and so there might be a relevance over the life of the programme, this is not a priority in the first few years. In Kaduna State, the CGS is seen as a success due to high completion rates, short timeframes for delivery and the involvement of the grassroots. Compared to the bureaucracy of the State projects (delivered through the traditional civil service system), the CGS has a quick delivery time and high rates of completion indicating strong performance in terms of project implementation. But, the spill-over effect in terms of stimulating better project performance across the MDAs and governance reforms (in terms of better M&E, PFM and PSM practices) is less obvious. There are some claims, such as the Due Process Office drawing on some of the CGS processes, and some MDAs are attempting to use a CGS-type gap analysis for project selection. But in general, there are other initiatives that are addressing these issues in a more substantive way (e.g. the setting up of the PFM Office, corporate planning by some MDAs, etc), and the claims are somewhat tenuous in terms of substantiated evidence. Furthermore in terms of CGS improved practices being taken up by MDAs (e.g. reforming 47 The Change Programme is the main tool through which SPARC jointly plans with the State Government and provides its technical support. 48 A finding supported by the Links Study, pages

32 project implementation, monitoring, etc), then stakeholders said this was limited. The reasons given included: The faster procurement and contracting of the CGS was more due to the advance payment of resources than the procedures themselves, and this could not be easily replicated within the State Government s system (where financial releases are much slower against the budget). The short timeframe and rapid implementation rates are seen by some as being more to do with the pressures of different stakeholders being involved (PSU, MDA, National Committee, plus OPEN). Some elements could be replicated by the number of supervision visits and level of scrutiny would be difficult to replicate without additional resources. The State Government s MDA projects are often politicised due to the pressure to locate within particular LGs (where as CGS is spread more evenly across the LGs). Overall, while the improved practices of the CGS were laudable (like greater grassroots involvement, M&E by different stakeholders, quick budget releases, and a lack of political pressure in guiding selection), the view was that they would either be too costly or too difficult to replicate within the civil service. Key findings: At a broad level, SPARC s support to governance reforms closely fits the ideals of the CGS, but at the State level, the extent to which SPARC-supported reforms will enable States to meet the requisite criteria of the CGS is less pronounced and mostly tenuous. This is because many of the scoring criteria are project-specific rather than about the broader governance reforms of the State, and because many of the criterion about wider governance reforms are too imprecise (especially for PSM, P&S and M&E). The evidence of a spill-over effect in terms of stimulating better project performance across the MDAs and governance reforms (in terms of better M&E, PFM and PSM practices) is less clearly substantiated. In the example of Kaduna State, the CGS has a quick delivery time and high rates of completion compared to the bureaucracy of the MDA projects (delivered through the traditional civil service system). While this indicates strong performance in terms of project implementation, the adoption of the CGS improved practices by MDA projects is said to be limited and difficult to replicate within the constraints of the civil service. CGS and meeting the MDG targets: beneficiaries, gaps, additionality 49 This section is about looking at the bigger picture, and placing the CGS within the total MDG requirements. The section seeks to determine the number of beneficiaries of the CGS interventions and their average unit costs (e.g. CGS capital costs, OSSAP recurrent costs and SPARC support) compared to estimated needs within relevant MDGs goals and estimated funding of the CGS. In addressing this evaluation question, this section is about understanding the potential contribution that the CGS can make to the MDGs and conversely the likely shortfall. The section is structured as follows: Firstly an assessment of the number of CGS beneficiaries and the average cost. This is followed by an estimate of meeting the MDGs, and an assessment of the additionality that CGS provides. Then finally, the section looks at the gap between the CGS and the resources estimated to attain the MDGs. 49 This section responds to question 4 of the TOR: How many beneficiaries of MDG CGS interventions have there been and what has been the average unit costs (e.g. CGS capital costs, OSSAP recurrent costs and SPARC support); compared to estimated needs within relevant MDG goals and estimated funding of the MDG CGS? 24

33 CGS beneficiaries and average cost Reliable data was not available on number of beneficiaries by State, Local Government or by project. 50 The analysis here is based on the aggregate beneficiaries data presented in the Countdown Strategy. Moreover, there was no available data on OSSAP-CGS recurrent costs and SPARC support costs. The approved CGS funding was used to calculate the average unit cost per beneficiary in Table 4. Table 4: CGS beneficiaries and average cost Total Total Number of Beneficiaries (million) Conditional Grant Scheme Funding (N'billion) Unit cost per beneficiary (N) 3,488 2,176 3,644 2,848 Source: Average cost per beneficiary calculated from: Countdown Strategy 2010 to 2015, Annex 1, P. 64; Links Study, Pp ; The Story of CGS, Page 18. The total number of beneficiaries does show a consistent trend; rising dramatically in 2008 and dropping in 2009 (see Table 4). The drop in 2009 was in spite of the 10% rise in the level of funding. In the same vein, the average unit cost per beneficiary shows a fluctuating trend; dropping in 2008 (perhaps due to high number of beneficiaries), and rising in On the whole, the unit cost of servicing a CGS beneficiary during the period from 2007 to 2009 was less than Naira 3,000 or US $20. Estimated cost of meeting the MDGs targets The projected MDGs requirements, projected funding, and funding requirements are presented in Table 5. The projections are based on the following assumptions: Federal Government MDGs expenditure is assumed to grow at an average annual rate of 12.4% during ; that being the historical growth rate of Federal government expenditures during (See base data in SPARC Expenditure Analysis Report, June 2010, P. 28, Tables 3.2). The 36 States & FCT MDGs expenditure is assumed to grow at the same rate as that of the Federal Government during ; the base value being the estimated 2009 expenditure (See SPARC Expenditure Analysis Report, June 2010, P. 43, Tables 4.7). CGS funding is assumed to grow at 10%, being the increase in approved funding from 2008 to 2009 (See base data in OSSAP-MDGs, The Story of CGS, P. 18). Exchange rate of N150 to US$1 was used to convert all naira values into dollars. The Projected costs of achieving the MDGs targets were obtained from OSSAP-MDGs, Countdown Strategy, Table 7.1, P.47. There are three main findings from the analysis (See also Table 5). Firstly, the projected costs of meeting the MDGs targets are huge and rising - representing over 70% of the projected Federal Government budget in 2010, and 80% in Secondly, the total sum of the projected funding by the Federal, States and CGS still leaves a deficit of US $8 billion in 2010; plus, the projected deficit continues to widen reaching nearly US$18 billion by And lastly, the CGS represents an insignificant share of the projected costs of meeting the MDGs 50 There are figures for beneficiary numbers by project as estimates are provided as part of the State s CGS application. Some of this information is held on the CGS database. However in discussions with CGS staff, this was not considered reliable because: (i) it is provided ex ante, based on assumptions rather than the actual numbers reached by an intervention; and, (ii) the methodology for estimating the numbers was not consistent nor robust, and there was likely to be double-counting. 25

34 targets; being only 2% in 2010 and declining to 1.7% by CGS is currently the success story of MDGs intervention (see Box 3). But despite this, the impact of CGS in meeting the MDGs targets is likely to be minimal given the relatively small size of the Fund. It is therefore only if the CGS can leverage additional funds and governance reforms that it can provide a meaningful contribution to achieving the MDGs. And in the end, it may be that the CGS turns out to be a more significant vehicle for effecting governance change, than for making any major inroad into the achieving the MDG targets. Table 5: Projected MDGs funding requirements and funding gap Projected MDGs funding requirements ($'billion) Projected Federal Government MDGs expenditure ($'billion) Projected 36 States & FCT MDGs expenditure ($'billion) Projected CGS Funding ($'billion) Projected total funding (excluding LGAs) ($'billion) Funding requirement deficit ($'billion) Projected MDGs funding requirements ($'billion) as % of Projected Federal Govt Budget (8.0) (9.8) (11.9) (13.7) (16.0) (17.6) 71.3% 74.1% 76.6% 77.5% 78.7% 77.9% Projected CGS as % of projected MDGs funding requirements 2.0% 1.9% 1.8% 1.7% 1.7% 1.7% Sources: Calculated from data in: SPARC Expenditure Analysis Report, June 2010, Pp. 28, 30 and 43; Tables 3.2, 3.3 and 4.7; OSSAP-MDGs, Countdown Strategy, table 7.1, P.47; SPARC, Links Study, Pp ; OSSAP-MDGs, The Story of CGS, P. 18. Note: CGS funding for 2008 and 2009 included Counterpart funding. Box 3. The successes of the Conditional Grant Scheme In much of the documentation (the Countdown Strategy, the Links Study, etc), the CGS is presented as a remarkable success story. Most stakeholders seem to concur, citing figures of high completion rates and anecdotes from communities visited. The figures for the CGS show extremely high completion rates of 98% completion in 2007, and 88% in The CGS projects themselves are relatively straightforward outputs, such as the rehabilitation of boreholes, the construction of hand pumps, and the rehabilitation of a primary healthcare centre. While not downplaying the significance of these achievements, there are still important issues about the coverage, usage and access by the target groups, plus the ongoing management, maintenance and sustainability. While this evaluation was unable to make field visits to project sites, a review of the Links Study highlights many of the issues (see Annex 8). In the Links Study, seven projects were visited. It is likely that these were better examples, being either close to the main conurbation or recommended by PSU staff. As such, the following findings are likely to be more positively biased and not a true representation of the range of issues and their relative importance. Also, the visits were visual inspections, and did not attempt to assess the quality of the physical structures or gauge a representative view from the target group of beneficiaries. Despite these caveats, the findings are interesting: Firstly, there was a generally good level of completion of CGS projects (with functional boreholes, buildings 51 Countdown Strategy, page

35 having been constructed, etc). This is consistent with the completion figures. Secondly, the anecdotal evidence showed that users were mostly satisfied (though this was far from being a comprehensive study of access, use and user satisfaction). And lastly, there were significant issues highlighted about sustainability (reliance on motorised boreholes, irregular electricity, lack of trained PHC staff), and this included: the need to address recurrent costs (e.g. periodic maintenance of generators) and complementary inputs (e.g. LGA to provide trained staff) and misuse of buildings (e.g. the healthcare facility being used for other purposes the immigration office). CGS and the additionality question The CGS was conceived as an additional resource to scale-up the funding of MDG projects at the sub-national levels. Unless the funds are treated in this manner by the recipients, the intended impact will not be realised. In addition to the interviews with the stakeholders during our visits to the States, we examined the available budget/ expenditure data to gain insight into how States have been treating the CGS funds. We examined the actual expenditure data of the 36 states of the Federation plus the Federal Capital Territory (FCT) for the period as published by the Central Bank of Nigeria (CBN). We also examined the expenditure data of the two SPARC focal States (Kaduna and Lagos) visited during the study. 52 The analysis is summarised in Figure 4. There are three key findings to this analysis. Firstly, for the 36 States and the FCT as a whole, there is no evidence to suggest that the CGS was treated as additionality although it is difficult to confirm otherwise. MDGs expenditure increased in absolute terms throughout 2004 to 2008; but the expenditure as a proportion of total expenditure dropped from 40.6% in 2004 to 28.6% in Secondly, the story was similar in Kaduna although the data are budgets rather than actual expenditure. Budgeted expenditure on MDGs rose continuously during and almost tripled by 2009; but percentage wise, it declined slightly from 73% in 2007 to 69% by If the actual expenditure reflects this budget position, Kaduna State would be regarded as one of the most MDGs focused states in Nigeria given the very high percentage allocation to the MDGs (see also Box 2 for details on the limitations of the budget figures). And finally, for Lagos State, the actual expenditure on MDG rose in 2008 and declined in 2009 despite the fact that total spending actually rose by slightly (by 2%) in The relative performance of MDGs expenditure was even worse; the proportion dropped throughout the period from 52% in 2007 to 39% in This finding is contrary to the glowing tribute paid to Lagos State by stakeholders as MDGs focused State. 52 The selection of MDGs core sectors followed the approach adopted in the MDG Needs Assessment (See: OSSAP-MDGs & UNDP, MDGs Needs Assessment, Nov. 2008). The guidelines for determining MDGs expenditures were further exemplified in: SPARC, Analysis of MDGs Expenditure, June 2010 Pp ). See annexes 2 4 for the lists of sectors included in the respective analyses (36 states plus FCT, Kaduna and Lagos analyses). 27

36 Figure 4: States and FCT MDGs Expenditures Source: Kaduna State data extracted from the State's submissions to the CGS office; Lagos State data provided on CD by the Ministry of Economic Planning and Budget, Budget Department ; Central Bank of Nigeria, Annual Report & Financial Statements for the Year Ended 31st December 2008, P Notes: 2007 figures are the revised approved estimates ; Lagos State data are actual expenditures. Addressing the MDGs funding gap Given the enormous funding gap that is apparent from the previous analysis, the MDG Needs Assessment report concluded that, Nigeria is unlikely to achieve all the MDGs by Moreover, in view of the widening of the Federal Government s budget deficits over the years, 54 it will be unrealistic to expect any dramatic increase in funding from the Federal Government. Various options have been suggested for closing the funding gap. Increased taxation and internally generated revenue (the current tax revenue of 10% of the GDP is considered too low); but the prospect of increased tax revenue in a poverty ravaged society is not very bright. 55 The options include: Debt financing borrowing from banks and capital markets; the existing $28.2 billion (or N4.2 trillion) domestic debt is already posing a great threat56; except sufficient discipline can be mustered to borrow only for self-financing projects. Bond financing for long-term capital for infrastructural development. Use of the excess crude oil revenue and foreign exchange reserves. 53 OSAP-MDGs, Countdown strategy, page SPARC, MDG Budget Analysis, June 2010, Figure 3.1, page See for example the Countdown Strategy, page See FG Moves to Curb Foreign, Local Debts in The Punch, Thursday 28 October 2010, P.17; and Oluwaseyi Bangudu, Nigeria s rising domestic debt, threat to private sector 22 October 2010 (Internet). According to The Punch source, the Minister of Finance, Mr. Olusegun Aganga stated that: Nigeria s outstanding debt as at September 2010 stood at $32.8 billion; consisting of $4.5 billion (14 per cent) external and $28.2 billion (86 per cent) domestic debts (sic). 28

37 Private sector contribution, e.g. through increased corporate social responsibility (CSR) activities. Support from development partners (this may be in the area of capacity building/strengthening). These options focus mainly on the financial requirements for meeting the MDG targets. An equally important consideration is the issue of capacity in areas such as project design, execution, financial accountability and M&E. If the required funds are made available one way or the other, do the implementers have the capacity/capability to execute the relevant projects diligently as well as monitor and account for the funds? 57 This is an important question that must be answered as part of the medium term strategies for meeting the MDGs targets. Accordingly, it is imperative to carry out a comprehensive assessment of the capacity gap with a view to developing effective strategies for addressing any identified weakness. This could facilitate supports from development partners, the type currently being provided by SPARC. Key findings: The total number of beneficiaries does not show a consistent trend; rising dramatically in 2008 and dropping in 2009 with the drop in 2009 being in spite of the 10% rise in the level of funding. In the same vein, the average unit cost per beneficiary shows a fluctuating trend. In terms of the estimated costs of meeting the MDGs, the projected costs are huge (representing over 70% of the projected Federal Government budget in 2010, and 80% by 2015). The CGS represents an insignificant share of the projected costs of meeting the MDGs targets (being only 2% in 2010, and declining to 1.7% by 2015). While the CGS is currently the success story of MDG interventions in Nigeria, its impact is likely to be minimal given the relatively small size of the fund. It is therefore only if the CGS can leverage additional funds and governance reforms that it can provide a meaningful contribution to achieving the MDGs. In terms of additionality the CGS being treated as extra to existing State resources the available evidence neither confirms nor denies that CGS is being used as additional funding in support of the MDGs. More evidence and analysis of expenditure is needed to assess this issue. Given the enormous funding, it seems unlikely that through the current mechanisms that Nigeria will achieve all its MDG targets by 2015, and particularly so as the Federal Governments budget deficits increase over the years. Various options have been proposed to increase the resources available, but it is also important to consider how capacity can be better addressed in terms of project design, execution, financial accountability and monitoring and evaluation. The CGS has a potential role in addressing these issues. 57 The capacity issue was noted in the MDGs Needs Assessment report (See OSSAP-MDGs and UNDP, MDG Needs Assessment, Nov. 2008, P. 107). 29

38 Section Four: Conclusions and Recommendations Overall, SPARC s Federal-level objectives align very well with those of the OSSAP-MDG and the CGS in particular. And, the manner of SPARC s support and its cooperative relationship with the CGS office has helped foster this close alignment. As 2015 draws closer, it will be important that this external support (through SPARC and others) helps the CGS better anticipate and rise to the challenge of achieving an impact on the MDGs, and in a way that is sustainable and at scale. This will be a significant challenge. Over the past few years, the CGS has made significant strides in improving its working practices and SPARC has played an instrumental role in many of these changes. In particular, SPARC-supported consultants have played a key role in the development of the qualifying criteria and improving the rigour of the bid evaluation process. This has helped make the grant assessment process much more systematic and robust, leading to improvements in the quality of applications and the transparency of the CGS. Through funding key positions, such as the Database Manager and the Programme Coordinator, SPARC has also assisted the capacity of the CGS office - although it is too early to assess the full impact of this in terms of skills transfer. Plus, the studies funded by SPARC have been constructive attempts to grapple with key issues around the leverage of the CGS on State-level reforms (the Links Study) and the additionality in terms of MDG expenditure (MDG Study). These studies have usefully brought attention to areas that need refining in the qualifying criteria, although the paucity of data highlights that there is much more research required (e.g. on MDG expenditure trends). In general the support of SPARC has been highly valued, although there is a risk that the CGS becomes overly dependent on such support for its routine work (e.g. the funding of workshops, the provision of consultants and technical staff). This is particularly pertinent in terms of the efficiency of the CGS and its processes; which are getting increasingly intensive in terms of the number of CGS staff and the number of consultants. It is clear that the CGS has made good use of expertise to increase the rigour of its working practices, but it is also important that ways are sought to ensure that these practices become mainstreamed within the CGS and the civil service more broadly (e.g. the use of MDA experts) and some of the dependence is reduced. At the State level there has been significant progress, with the States showing an increasing commitment to the CGS providing not only 50% counterpart funding, but also a rising number of applications and approvals (which far outstrips the available funds). While the demand for the CGS has increased, the evidence is inconclusive as to whether this equate to an increased commitment to the MDGs by State Governments. The extent to which SPARC s State-level support enables State Governments to meet the requisite criteria of the CGS is less pronounced and indeed the State Teams do not deliberately target work towards the PSU (or MDG Office) that is responsible for the conditional grants. Furthermore, the CGS scoring 30

39 criteria is very project-specific and less about the broader governance reforms, with many of the criteria being too imprecise (especially for PSM, P&S and M&E). There is also little robust evidence of a spill-over effect in terms of stimulating better project performance across the MDAs. This raises concerns about the extent to which the CGS can leverage reforms. And lastly, the estimated costs of meeting the MDGs are huge, and the CGS represents an insignificant share of the projected costs. Therefore, while the CGS is currently the success story of MDG interventions in Nigeria, its impact is likely to be minimal given the relatively small size of the fund. It is only if the CGS can leverage additional funds and governance reforms that it can provide a meaningful contribution to achieving the MDGs. Here, the evidence of additionality is inconclusive but the CGS could have an important role in increasing capacity and improving project implementation. The main recommendations arising from this study are shown below. Strategic Relevance A. The CGS should develop an internal/ operational CGS strategy that sets specific priorities, helps monitor performance and refines action plans. It is likely that the CGS office will receive increasing attention and scrutiny over the coming years as 2015 nears, and having an operational strategy may also help ensure that CGS priorities are kept to the fore (rather than being swayed by the preferences of particular donors). B. The existing strong cooperative relationship between the CGS Office and SPARC should be sustained without compromising independence and objectivity. Both parties appear to have developed mutual confidence in each other which has been instrumental in CGS achievements to-date. C. As 2015 draws closer, it will be important that the support provided by SPARC and others should help the CGS to better anticipate and rise to the challenge of achieving an impact on the MDGs, and in a way that is sustainable and at scale. This will require a change in strategic objective from merely ensuring the effectiveness of CGS as it currently stands, to promoting the sustainability of a scaled-up CSG. D. SPARC should consider a gradual withdrawal from routine functions such as workshop funding and provision of consultants and technical staff and focus more on strategic support, such as strategy development, capacity building, scalingup options, etc. so as to ensure sustainability. Working practices & efficiency E. Further work on the qualifying criteria should focus on refinement rather than adding additional requirements (focusing particularly on P&S, M&E and PSM as appropriate). Undoubtedly the work on the qualifying criteria so far has improved it immeasurably, but there is a risk that it becomes too resource-intensive and burdensome. F. Some level of automation of the process would be desirable for improved efficiency and to enhance the sustainability of the process. Efforts should also be made to further involve civil servants from the key MDAs. This is because there is an increasing dependence on externally-funded consultants for routine CGS processes (and particularly the bid evaluation process) which raises important issues around dependency and potentially, inefficiency. G. Efforts should also be made to define appropriate metrics to aid the analysis of process efficiency and assist management decisions. For instance, a detailed process map of the working practices needs to be developed showing the main activities and tasks as well as their interrelationships and opportunities to resource inputs requirements. 31

40 H. Continued attention needs to be paid to ensuring that there is sufficient skills transfer from the SPARC-supported positions of the Programme Coordinator and the Database Manager. The database in particular requires much more work (and data validation) if it is to have wider utility for monitoring and analysis of the CGS and this requires CGS staff to better understand the usefulness of the data and its limitations. Evidence of impact I. The two studies funded by SPARC have been useful, but more evidence and analysis is needed over the coming years. In particular this includes a follow-up to the study on MDG expenditure which was limited because only budget allocation data was available at the time, and also because it was unable to assess trends. Learning from work such as the use of MDG codes in Kaduna State may also help improve usefulness of the available data. J. Data availability on CGS operations should be significantly improved to facilitate a more robust evaluation of the scheme s progress and achievements in future as well as impact of external support. Key areas to focus in this regards will include credible and consistent data on: CGS Office s recurrent expenditure by activity; cost of SPARC/ external support; number of beneficiaries; time and costs at each stage of the CGS process; and data on PSUs in the States. Sustainability and impact K. Projects funded in the 2007 and 2008 rounds should be followed up in order to learn lessons and ensure that the gains being made against the MDGs are cumulative. There is a seemingly shortage of robust information on the sustainability of projects; and M&E, supervision and site visits should emphasize recurrent and maintenance issues to be sure that projects are fully functional in years to come. L. It is recommended that a capacity assessment is undertaken to determine the gaps to be bridged in reaching the MDGs through the CGS. It is one thing increasing the funding available for the MDGs, it is another to make sure that there are the capacities to undertake sound project design, execution, financial accountability and M&E. M. More work is needed to enhance the commitment and capacity of the States to address the MDGs and how the CGS can better leverage increases in MDG expenditure. Ways should be found to encourage States to increase their MDGs expenditures in relative terms and to effectively monitor and evaluate progress towards the attainment of the MDGs. 32

41 Appendix One: Terms of Reference SPARC TOR: PRG-M&E-3-C 1. Name of Consultant: Chris Barnett - Team Leader 2. Overarching TOR Ref: PRG-M&E-3 (Attached) 3. Overarching TOR Version: 3 4. Contribution to specific Dimension outputs The specific Dimension outputs within the overarching TOR which this input will contribute towards: 2. A consolidated long-list of proposed 2011 evaluation studies ready for submission to the Evaluation Committee, alongside guidance on selection and screening criteria. 5. An approach and methodology for producing Time Series Studies approved by the Evaluation Team, implemented in all SPARC supported states and state level Time Series Studies prepared and updated on a regular basis. 6. For each evaluation study scheduled for completion during 2010/2011, a short (10 pages) Inception Report approved by the Evaluation Team which details the methodology to be used, consistent with the overarching SPARC approach to conducting evaluation studies (see section 14 of TOR), incorporating: a) understanding of the assignment, b) emerging issues and challenges, c) approach and methodology, d) response to the Six Guiding Principles, e) workplan (activities, time schedule) and f) annexes: terms of reference; initial list of stakeholders; list of secondary sources; evaluation matrix; table of contents for evaluation report; draft tools (checklists & questionnaires). 7. For each evaluation study scheduled for completion during 2010/2011, an Evaluation Report (30 pages) approved by the Evaluation Team, incorporating: a) introduction & purpose of the evaluation, b) approach & methodology (including limitations), c) evaluation findings (with supporting evidence), d) key lessons learned, e) recommendations and f) annexes. 5. Activities The consultant will work closely with the SPARC Evaluation Team to implement/deliver the following activities/outputs needed to deliver overarching TOR outputs 6 & 7: 1. Conduct an assessment of SPARC support and the resultant progress made in implementing the MDGs Conditional Grants Scheme (CGS). Key areas of assessment should include: Relevance: The extent to which the intervention is the right approach, given the target audience needs and the political, institutional and economic context. Was the approach appropriate to the governance context How well aligned are the strategies and plans of SPARC support with those of the Office of the Senior Special Assistant to the President on MDGs (OSSAP-MDGS) and the MDG Conditional Grants Scheme (CGS)? 33

42 (problems) of the state? Effectiveness: The extent to which the objectives were achieved, or are expected to be achieved To what extent were the objectives achieved / are likely to be achieved? What major factors influenced the achievement or nonachievement of the objectives? What changes have been made to MDG CGS working practices and processes since early 2009? What improvement has there been in: a) development and application of criteria for accessing MDG CGS funds and b) the efficiency of processes by which grants are awarded, distributed and managed? How many beneficiaries of MDG CGS interventions have there been and what has been the average unit costs (e.g. CGS capital costs, OSSAP recurrent costs and SPARC support); compared to estimated needs within relevant MDG goals and estimated funding of the MDG CGS? What level of commitment has there been by SG in allocating their own resources in support of CGS funded initiatives? To what extent have/will Change Programme reform initiatives enable SG in SPARC supported states to meet the requisite criteria, in various areas (e.g. PFM, M&E), for accessing MDG CGS funding? To what extent have technical outputs delivered with the support of SPARC been adopted and what contribution have they made to changes in working practices and processes identified above? The assessment should be conducted in line with the overall methodology for SPARC evaluation studies detailed in the main TORs for this assignment. 2. Prepare an Inception Report for the evaluation study. The Inception Report should be consistent with the overall methodology for SPARC evaluation studies given in the main TORs for this assignment and should respond to the key questions given in (1) above, alongside any additional questions identified during inception. The Inception Report should be a maximum of 10 pages, plus annexes, and include the following: Understanding of the assignment. Emerging issues and challenges. Approach and methodology. Response to the Six Guiding Principles. Workplan (activities, time schedule). Annexes: Terms of Reference; Initial list of stakeholders; List of secondary sources; Evaluation Matrix; Table of Contents for evaluation report; Draft tools (checklists & questionnaires). 3. Review and adjust the Inception Report in consultation with the Evaluation Team. 4. Develop the structure of the Evaluation Report for the study and provide guidance within this structure which study team members can use to draft the Evaluation Report sections. The Evaluation Report should be prepared as detailed in the overall methodology for SPARC evaluation studies given in the main TORs for this assignment. The Evaluation Report should be a maximum of 30 pages, plus annexes, and incorporate the following: Introduction & purpose of the evaluation Approach & methodology (including limitations) Evaluation Findings (with supporting evidence) Key Lessons Recommendations Annexes 5. Review and adjust the proposed Evaluation Report structure in consultation with the Evaluation Team. 6. Provide quality assurance of the Evaluation Repor through reviewing and editing the draft report prepared by study team members. 7. Review and adjust the Evaluation Report in consultation with the Evaluation Team. The consultant will work closely with the SPARC M&E Team Leader to implement/deliver the following 34

43 activities/outputs needed to deliver overarching TOR outputs 2 & 5: 8. Facilitate discussions with the SPARC Evaluation Team in the following areas: a) potential improvements to study planning and management, b) guidance on selection and screening criteria required for 2011 programme of evaluation studies, c) implications of revised evaluation study workplan and budget on overall approaches and methodology, d) potential approach and methodology for producing Time Series Studies and e) emerging approaches to measuring VFM in SPARC. 9. Prepare a summary report which: a) records and assesses lessons learned from implementing the two pilot evaluation studies, b) makes recommendations for improving the overall study methodology and study planning and management and c) makes recommendations on selection and screening criteria for the 2011 programme. 10. Prepare a concept paper which provides a workable approach and methodology for producing Time Series Studies, including: a) how to regularly synthesise various sources of information (e.g. MIS, innovation diaries, P&E research), b) proposed roles and responsibilities of relevant SPARC staff (e.g. STLs, M&E Team Leader, M&E/KM TOs) and c) anticipated outputs from the time series process and an indication of how they will be used during the main programme of evaluation studies. 6. Inputs and Deadlines The anticipated inputs/resources needed to implement the activities, together with deadlines for delivering the stated outputs are 22 Days in total, consisting of. 12 days Nigeria inputs in order to complete activities/outputs 1-5 and 8 and begin activity/output 6, over 4 th Oct 15 th Oct days home based inputs in order to complete activities/outputs 6, 7 and 9, by end of Oct days home based inputs in order to complete activity/output 10 by mid Dec

44 Appendix Two: List of Stakeholders 1 Dr Ishyaku Mohammed Head, Conditional Grant Scheme, OSSAP-MDG 2 Chinedu Eze Programme Coordinator CGS-LG Track, OSSAP-MDG 3 Mr Jude Nwankpa Assistant Director, Conditional Grant Scheme, OSSAP-MDG 4 Jonathan Edwards Economist, OSSAP-MDG 5 Muhammad Sani Economist, OSSAP-MDG 6 David Allu Project Manager, Conditional Grant Scheme, OSSAP-MDG 7 Maduka Okafor Database Manager, Conditional Grant Scheme, OSSAP-MDG 8 Adija Momoh Assistant, Conditional Grant Scheme, OSSAP-MDG 9 Chris Perry M&E Technical Lead, SPARC 10 Oluwatosin Akomolafe M&E Technical Officer, SPARC 11 Joe Abah Deputy Programme Manager, SPARC 12 Hadiza Elayo Federal component, SPARC 13 Nuhu Moses Head, CGS Project Services Unit, Kaduna 14 Paul Kagoro Mechanical Engineer, CGS Project Services Unit, Kaduna 15 Mohammed T. Soba Internal Auditor, CGS Project Services Unit, Kaduna 16 Joseph G. Graba Project Accountant, CGS Project Services Unit, Kaduna 17 Arc. Chori Head of M&E, CGS Project Services Unit, Kaduna 18 Mark Walker State Team Leader, SPARC, Kaduna 19 Chris Umebese Senior Technical Coordination Manager, SPARC, Kaduna 20 Umaru Adamu MDG Desk Officer, Project Implementation Unit, Ministry of Health, Kaduna 21 Engineer Markus MDG Desk Officer, Ministry of Water Resources 22 Mr. E. O. Oyegoke Director of Planning, Ministry of Economic Planning and Budget (MEPB) and Head of CGS, Lagos State 23 Mr. Abdul Munim Ladan CGS Desk Officer, MEPB, Lagos State 24 Mr. Ajibola Akeem Deputy Director, MEPB and Deputy Head of CGS, Lagos State Balogun 25 Mrs. Adenike Mustapha Principal Planning Officer and SPARC Desk Officer, Lagos State 26 Arc. Tunde Orelaja Architect on CGS Projects, Lagos State 27 Mr. E. A. Fatade Lagos State Tenders Board (LSTB) 28 Mr. Oladotun Basorun Lagos State Tenders Board (LSTB) 29 Mr. Ajibola Ariyibi Director, M&E Department, MEPB, Lagos State 36

45 30 Omotayo Takoluju Assistant Chief Administrative Officer, M&E Department, MEPB, Lagos State 31 Mr. Balogun Moses Abiodun Chief Statistical Officer, M&E Department, MEPB, lagos State 32 Fesoeme T. Husein Assistant Chief Administrative Officer, M&E Department, MEPB, Lagos State 33 Omorodion B. Titilayo Assistant Chief Administrative Officer, M&E Department, MEPB, Lagos State 34 Ifeanyi Peters Ugwuoke SPARC Technical Coordination Manager, Lagos State. 35 Dr. A. E. Erinosho Secretary, Primary Healthcare Board (PHB), Lagos State. 36 Mrs. Muinat Olabimpe Ojikutu-Balogun Principal Administrative Officer, Ministry of Local Government Affairs, Lagos State. 37

46 Appendix Three: List of References CDD (no date) Learning Lessons: Monitoring and Evaluating Nigeria's MDGs Expenditures CGS Office (2009) Implementation Manual for the MDGs 2009 CGS, OSSAP-MDG DFID (2010) SPARC: Output-to-Purpose Review, Main Report, May 2010 Govt of Nigeria (2010) Countdown Strategy-2010 to 2015: Achieving the MDGs Govt of Nigeria (2010) MDG Report 2010 Govt of Nigeria (2010) Nigeria MDGs +10: We're making progress and accelerating to meet the 2015 target date Kaduna SG (2008) Public Financial Management Performance Report and Performance Indicators, Final Report, March 2008, Kaduna PEFA Self-Assessment Core Team, Kaduna Kaduna SG (2009) Detail Report of the Accountant General: Kaduna State (for the year ended 31st December 2009), Kaduna State Government, Kaduna Kaduna SG (2009) Kaduna SG Change Programme 'Complete Version', Reforming our public service, our financial management, and our planning & monitoring functions to effectively deliver the Eleven Point Agenda (EPA), Office of the Head of Service, Kaduna Kaduna SG (2009) Policy and Strategy, Information Management/M&E Self-Assessment, Final Report, March 2009, Kaduna P&S/M&E Self Assessment Team, Kaduna Kaduna SG (2009) Public Service Management Self-Assessment, Final Report, March 2009, Kaduna PSM Self Assessment Team, Kaduna Kaduna SG (2009) Summary Report of the Accountant General: Kaduna State (for the year ended 31st December 2009), Kaduna State Government, Kaduna Kaduna SG (2010) Change Programme Progress Report: Policy and Strategy, Including M&E, June 2010, Ministry of Economic Planning, Kaduna Kaduna SG (2010) Change Programme Progress Report: Public Financial Management, Including M&E, June 2010, Ministry of Economic Planning, Kaduna Kaduna SG (2010) Change Programme Progress Report: Public Service Reform, Including M&E, June 2010, Ministry of Economic Planning, Kaduna Lagos SG (2008) 2008 Budget - The Great Leap Budget, Ministry of Economic Planning & Budget, Lagos Lagos SG (2009) 2009 Budget - The Budget of Accelerated Growth, Ministry of Economic Planning & 38

47 Budget, Lagos Lagos SG (2010) 2010 Budget - Budget of Consolidation, Ministry of Economic Planning & Budget, Lagos Lagos SG (2010) LASG Sector Budgets HOA Approved, Ministry of Economic Planning & Budget, Lagos OSSAP-MDGs (2007) Guidelines for conditional grant in budget 2007 OSSAP-MDGs (2008) 2008 Conditional Grant Scheme Guideline OSSAP-MDGs (2009) MDGs CGS Implementation manual OSSAP-MDGs (2009) Quarterly Brief on the MDGs office, October 2009 OSSAP-MDGs (2009) Report of the technical team Evaluation of States submissions for 2009 CGS OSSAP-MDGs (2010) Communicating the MDGs OSSAP-MDGs (2010) Making Debt relief work for the poor: Achieving the MDGs in Nigeria OSSAP-MDGs (2010) MDG Factsheets (various) OSSAP-MDGs (2010) Partnering to achieve the MDGs: The story of Nigeria's CGS PCM (2006) Presidential Committee PPT_June2006 PCM (2007) Presidential Committee presentation FINAL (2007) ex post corrected PCM (2008) Presidential Committee presentation 2008 v22. PCM (2009) Presidential Committee April 2009 Q1 Final without photos. PCM (2009) Presidential Committee on the MDGs-2nd/3rd Quarter Report 2009 PCM (2009) Presidential Committee presentation August 2009 FINAL PCM (2009) Quarterly Brief on the Activities of MDG Office, 04/10/09 PCM (2010) Presidential Committee on the Assessment and Monitoring of the MDGs-4th Quarter 2009/1st Quarter 2010 Report PCM (2010) Presidential Committee presentation April 2010 FINAL PCM (2010) Presidential Committee presentation June 2010 FINAL PCM (no date) Final PCM Committee on strategy and Prioritisation of the MDG Phillips, J. (2009) Restoring Confidence and Increasing Responsiveness in State and Local Government through Conditional Grants: The Experience of Nigeria, World Bank, Abuja PSU (2010) CGS in Kaduna State: A Presentation to Kaduna State LG Officers on the September 2010, Project Support Unit, CGS, Kaduna Searle, B. (2008) Achieving Nigerian Policy Objectives through Conditional Grants: Scope for Greater Use of Incentive-based Instruments, World Bank/ DFID, Abuja SPARC (2008) 2008 CGS implementation process, Consultants Reports SPARC (2009) 2009 Conditional Grants Scheme Sensitisation Workshop, Consultants reports SPARC (2009) Federal Support: Guidance Note, May 2009 SPARC (2009) Mapping of Federal Support and Incentives to the states, March 2009 SPARC (2009) Quarterly Progress Report - Work Plan Review and Adjustment Work Plans, No 3 October to December 2009, SPARC, Abuja 39

48 SPARC (2009) SPARC Overarching Concept: Guidance Note, May 2009 SPARC (2009) TOR FED-MDG-1A: Sensitisation Workshop for 2009 CGS Round, August 2009 Abuja SPARC (2009) TOR FED-MDG-1B: 2009 CGS Round: Evaluation of Applications from States, September 2009 Abuja SPARC (2010) Analysis of MDGs Expenditure in Nigeria's Federal, State and Local Governments Budgets SPARC (2010) Assessment of the links between Conditional Grant Scheme Activities and State Reform Efforts SPARC (2010) Final Report: Database Manager, 30th August 2010 SPARC (2010) Lagos Education Budget analysis , SPARC, Lagos SPARC (2010) Lagos Health Budget analysis , SPARC, Lagos SPARC (2010) Looking Closer, Digging Deeper: A Strategy for Evaluation Studies, June 2010 SPARC (2010) National Logframe, versions 05/02/10 and 27/09/10 Abuja SPARC (2010) Visit Report: Analysis of MDGs Expenditure in Nigeria s Federal, State and Local Government Budgets, FED-FED-1-E SPARC (2010), 2010 Budget Analysis and Budget Information, version 1.2, SPARC, Lagos 40

49 Appendix Four: Methods of data collection and analysis Sub-questions Data sources Methods Question 1: How well aligned are the strategies and plans of SPARC support with those of the Office of the Senior Special Assistant to the President on MDGs (OSSAP-MDGS) and the MDG Conditional Grants Scheme (CGS)? What are the strategies / plans of the OSSAP- Interviews with MDG Office staff: (a) What are the OSSAP-MDG s MDG? objectives? (b) What actions have been taken to achieve these What are the strategies / plans of the CGS? What are the strategies/ plans of SPARC (Federal-level)? objectives? [NB: Initial discussions suggest that there is no documented strategy for the OSSAP-MDG & CGS, and that a de facto strategy will need to be derived] MDG Office/ SPARC Federal-level workstream Overarching SPARC Approach Paper; Federal-level Approach Paper; First year Work plan; Second year Work plan; Quarterly progress report; SPARC Programme Logframe; Other documents setting out a vision or strategy for support to MDG office? Question 2: What changes have been made to MDG CGS working practices and processes since early 2009? What are the MDG working practices and processes in: 2007, 2008 and 2009? Define practices and processes : Evaluation criteria; Call for bids; Sensitization workshops; Evaluation of bids; Award and disbursement; Oversight (M&E) Implementation Manual and Guidelines for CGS for 2007, 2008 and 2009 Report on the linkages between CGS and State Reform (Section 3) Evaluation of bids 2008 and 2009 Interviews with MDG office staff on working practices at Federal 41 Review and summarise the documents Interview the OSSAP-MDG office to derive a de facto strategy (of objectives/ priorities) including background on the rationale for not having such a strategy. Assess the alignment of the strategies/ plans Crosscheck initial findings with the MDG office staff to identify gaps in support (if any) Validate with Chris P, Joe A and Hadiza Check our understanding of the practices and processes (Hadiza, documents) Review the practices and processes (including mechanisms for review & adjustment to working practices). Summarise the changes during Evaluation the adoption and impact of these changes and State-levels (changes and resultant effects) Question 3: What improvement has there been in: a) development and application of criteria for accessing MDG CGS funds and b) the efficiency of processes by which grants are awarded, distributed and managed? What is the efficiency of the grant award process (time/ HR per application 2007 to 2009)? What is the efficiency of the disbursement of CGS funds (time/ HR 2007 to 2009)? MDG Office Hadiza/ Chris P to facilitate data collection (letter to OSSAP- MDG, Etc) Are there Annual Reports for the MDG Office? Validate the proposed approach with the MDG office (does the approach make sense, given the available data and the way the CGS works?) Collect data on time/ human resource inputs at each stage of

50 Sub-questions Data sources Methods What is the efficiency of the grant management process by SGs (time/ HR 2007 to 2009)? 42 the CGS process (through meetings with the MDG Office) Review of annual reports/ reviews (if available) Analysis of data on time/ human resources (staff/consultant inputs), and complexity of process Draw conclusions from the analysis Question 4: How many beneficiaries of MDG CGS interventions have there been and what has been the average unit costs (e.g. CGS capital costs, OSSAP recurrent costs and SPARC support); compared to estimated needs within relevant MDG goals and estimated funding of the MDG CGS? What is the number of beneficiaries: - By state/ LGA? - By project? - By MDG/ sector? What are the total costs (CGS capital costs; OSSAP recurrent costs; SPARC support)? What is the estimated need per MDG goal in terms of the number of beneficiaries? What is the estimated cost of meeting the MDG needs? How important is CGS relative to the needs (i.e. the gap analysis of requirements to meet the MDGs by 2015)? How to address the gap in CGS scaling-up (+ leveraging State Government increases in spending to reach the MDGs by 2015)? MDG Office/ Hadiza MDG database? MDG CGS office/ focal person at the state-level MDG+10 and Countdown Strategy MDG+10 and Countdown Strategy (key resources for this analysis) Quarterly Progress Reports from OSSAP-MDG Find out from meetings with MDG Office & Consultants: How can the CGS be used to incentivise the States to scale-up spending on the MDGs? Question 5: What level of commitment has there been by SG in allocating their own resources in support of CGS funded initiatives? How much counterpart funding is provided by Review and summarise MDG+10 and Countdown Strategy Explore alternative data sources (if necessary) Analyse and summarise key findings Meetings with MDG Office and consultants on scaling-up Summarize scaling-up options and conclusions (such as opportunities for increasing leverage at the state-level financing strategies, policies, etc). SGs (by project)? MDG Office/ Hadiza Collect and analyse data on counterpart funding Are there dedicated and effective units for CGS in the states, including M&E activities? [Proxy for commitment] What are the budget allocations to core MDG sectors (trend analysis)? State/ MDG Office Report on MDG expenditure in Nigeria Collect and review high-level policy documents of 2 states Interview with MDG/CGS office in the States Collect and analyse budget data 2007 to 2009 (by sectoral allocation) Collect and analyse budget implementation status for 2 states

51 Sub-questions Data sources Methods Question 6: To what extent have/will Change Programme reform initiatives enable SG in SPARC supported states to meet the requisite criteria, in various areas (e.g. PFM, M&E), for accessing MDG CGS funding? Identify SPARC reform initiatives by workstream: PFM, PSM, P&S, M&E Map SPARC reform initiatives to CGS criteria SPARC Change Programme documents (dimensions) Change Programme Joint Review documents Talk to SPARC staff in the States Report/ document on the Linkages between CGS and State Reforms (Jigawa and Lagos states) Review the Change Programme document and joint review document Summarise SPARC reform initiatives by work stream (use a matrix approach as much as possible: rows = CGS criteria; columns = SPARC reform initiatives by workstream) Follow-up (interviews) on findings from 2 SPARC states to identify enabling and constraining factors Draw conclusions and recommendations Question 7: To what extent have technical outputs delivered with the support of SPARC been adopted and what contribution have they made to changes in working practices and processes identified above? What technical outputs have been delivered (to the OSSAP-MDG office for the CGS)? What has been adopted by OSSAP-MDG office? What contributions have the adopted outputs made to working practices and processes? Any gaps and lessons for the future of the CGS? SPARC Federal-level TOR Visit (progress) Reports Consultant outputs/ reports Interview with MDG office and consultants Speak with Tim/ Hadiza/ Joe on feedback Cover in the interview with MDG office and consultants As above Check with Hadiza if all the technical outputs are available (SPARC Federal-level TOR, Visit/progress reports) Summarise outputs in a matrix format (columns outputs by title; rows objectives, consultants, duration, inputs, results, etc) Hold interviews with MDG office and consultants on the adoption, effects, lessons, gaps, etc Crosscheck initial findings with Tim/ Hadiza/ Joe Summarise findings and conclusions 43

52 Appendix Five: Comparison of CGS strategic objectives 2010: Countdown Strategy, p : Story of the CGS, p ) Investing in the MDGs at the sub-national level and ensuring local ownership and sustainability 2) Empowering state and local governments to carry out their constitutional responsibilities 3) Leveraging public sector reform, public expenditure reform [and] national planning for service delivery Through the qualifying criteria, established 2007 and revised in 2008 and 2009, the aims are to ensure: 3(a) Human Resources, capability and planning Investing in the MDGs at State & LG levels Empowering State & LGs to carry out constitutional responsibilities 2009: Implementation Manual, p : Searle Report, p13 Ultimately, it [the CGS] is an opportunity to leverage spending towards supporting State and Local Government programmes that are fully aligned with national policy objectives and the MDGs. Through the Conditional Grants Scheme, the Federal Government seeks to build the capacities of States and Local Governments in order to improve governance, service delivery, financial management, transparency, and accountability as well as the attainment of the MDGs. The CGS is therefore one of the Federal Governments way of intensifying its efforts at achieving the MDGs at all levels of government in the country. 2007/2008: The primary objective of the Program was, of course, to increase spending in pro-poor areas of sub-national governments budgets, and use a counterpart funding condition to get a contribution of extra State expenditure equal to the Federal allocation. The States understood this and accepted it as a mechanism to achieve the MDG targets /2008: The secondary objectives, outlined by the SSAP-MDGs in a number of public statements and in her Office s Guidelines for the Implementation of Conditional Grants in Budget 2007 however, were probably just as important. Through them, the Federal Government hoped to get more resources for and achieve much more efficiency in the use of public sector inputs. 58 Many countries have Conditional Grant programs that have similarities to the Nigerian MDG-CG Program, either with or without the matching requirement. Few, however, place as much emphasis on the secondary objectives as the Nigerian one does. The referenced paper by Steffensen and Larsen hold details for Bangladesh, Kenya, Nepal, Tanzania and Uganda. The OSSAP has been provided with complete documentation for the Local Government Development Grants Program in Sierra Leone. 44

53 2010: Countdown Strategy, p : Story of the CGS, p10-11 Adequate institutional structures to support the implementation of projects and programmes Development of planning for human resources, facilitates, equipment and supplies Promoting improvements in public service delivery 3(b) Public expenditure management, accountability and transparency Adequate institutional structures to account for expenditure (public expenditure reforms, modernisation of state budget processes) Encouraging improvements in public expenditure management 3(c) Harmonisation with MTEFs and national and state -level plans 3(d) Projects that relate to state policy priorities Projects relate to state priorities as declared in state development policy strategy documents 3(e) Strengthening partnerships between the 3 tiers of government Consultation with federal MDAs on project rationale, specifications, cost and due process mechanisms Strengthening partnership between the 3 tiers of govt, and involving local communities 2009: Implementation Manual, p : Searle Report, p13 In addition, the scheme is a unique opportunity to build a platform on which States and Local Governments harmonize their budgets fully with the nation s Medium Term Fiscal Framework. The CGS is an opportunity to maximize the use of information and expertise at every level of government and share the burden of expenditure responsibility among different levels of government. The scheme also seeks to foster genuine consultations and commitment among the Federal, State, Local Governments, communities and civil society. This will engender sustainability. Improve human resource capacity and management Improve financial management and government accounting Improve transparency at all levels of government Develop a philosophy of results-oriented budgeting Harmonise the budgets of all governments towards the MTFFs, and National and State Development Plans Achieve better collaboration between governments in policy development and planning Increase internally generated revenue at the sub-national level 45

54 2010: Countdown Strategy, p : Story of the CGS, p (f) Investing in projects that are in areas of greatest need, and in consultation with LGs Projects or programmes are in a key area for interventions (determined annually and communicated to the states) Projects relate to and/or demonstrate areas of greatest needs Consultation with local governments to identify projects and select locations 3(g) Commitment to maintaining the projects States commit to maintain the projects 2009: Implementation Manual, p : Searle Report, p13 46

55 Appendix Six: Summary of SPARC support (outputs) and adoption (outcomes) Objective SPARC support Extent of adoption Outcome Output 1: 2008 CGS Round: Evaluation of Applications from States To evaluate applications submitted by states for the 2008 CGS. Output 2: Sensitisation Workshop for 2009 CGS Round To assist States in understanding the criteria specified in the Implementation Manual, and to encourage them to develop well designed applications that clearly articulate MDGs investments and ensure that investments can be readily monitored and accounted. Deliverables: Workshop, presentation materials, report on training SPARC support: Four consultants funded by SPARC, with others funded by the MDG office Timescale: Aug 2009 Output 3: 2009 CGS Round: Development of Criteria and Evaluation of Applications from States To develop the qualifying criteria, plus to evaluate applications submitted by states for the 2009 CGS Deliverables: Scoring scheme; Evaluation report; Reports on each state that does not qualify; a database of projects, locations, approved project costs, deliverables and work plan; Any other key outputs/analysis completed during the course of the evaluation process e.g. costing benchmarks and variations, sustainability analysis. SPARC support: Lead Consultant, plus other specialists (11 consultants in total). 47 Use of consultants accepted as vital to provide technical support to the process. Seen as an indispensable means of enabling state officials to understand the qualifying criteria and to apply them correctly SPARC has helped in making the qualifying criteria tougher and more rigorous to meet The criteria have been accepted as a tool to further governance reforms at state level. Capacity building for civil servants that participated in the evaluation due to skills transfer from SPARC consultants. Experienced civil servant being incorporated for sustainability of the process. Improved quality of applications from states, especially in the areas of costing and supporting evidences. Institutionalisation of the knowledge gained by the civil servants that attended the workshop. Governance changes are taking place at states level due to the application of the qualifying criteria since states must meet the criteria in order to obtain the fund; e.g.: Availability of audited accounts. Due process in procurement. Evidence of budgets M&E. CGS process has become more

56 Timescale: Sept 2009 Output 4: Analysis MDG-Expenditure in Nigeria s Federal, State and Local Government Budget To estimate the proportion of public funds/scale of resources being invested in MDG projects and programmes at the three tiers of government toward poverty reduction, among other MDGs, especially the Debt Relief Component. Deliverables: A report detailing the nature of the MDG-related expenditures in each budget; A discussion of the methodological assumptions and concerns that arose during the assessment; update baseline guidelines of what constitute MDG related projects. SPARC support: Five consultants, including 2 PFM specialists, budget analysis specialist, SG budget specialist and a LG budget specialist. Timescale: 20 May to 18 June 2010 Output 5: Strengthening the links between MDG Activities and State Reform Efforts To find ways of drawing closer link between MDG activities and wider reform efforts in states. Deliverables: A full report, with a chapter on each state. The report, any cross-cutting or common issues and clear recommendations SPARC support: Four consultants Timescale: February to mid-march 2010 Output 6: MDG - Conditional Grant Scheme Database Manager To deliver the CGS database to full functionality. Deliverables: Verification and reconciliation of database totals; Collaboration on design of webaccessible database; Populating the database with 2008 location details; Populating database with 2007 CGS information; Ensuring readiness of database for uploads for future CGS data; Training of CGS team on use of database; etc. SPARC support: 1 National Database IT Expert Timescale: April 2010-April 2011 Source: Review of project documents and interview with stakeholders/fact-holders. Due to study findings on governance, future sensitisation training workshops will include M&E experts. Study findings also being used to improve the contents of the implementation manual. CGS managers are better informed about the impact CGS activities are having on the wider governance reforms at the states level and the areas where improvements are needed. Dm now recognised as indispensable to CGS; a civil servant is understudying the dm with a view to taking over from him upon the expiration of the dm s contract with SPARC and to ensure sustainability. Has promoted the design of uniform templates for reporting. transparent and objective; evaluation results have become incontrovertible. There is a plan to replicate the study to cover more states and local governments. The report is being used as an advocacy manual (how???) There is a plan to revise the qualifying criteria for greater impact on other areas of governance reforms, particularly PSM and M&E. Facilitates prompt responses to urgent data/information requests, especially from the OSSAP-MDGs. Potential to utilise the database for research by CGS staff and members of the public. 48

57 Appendix Seven: Mapping of linkages between CGS and SPARC State-level support (Kaduna) CGS Qualifying Criteria SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 STATE INDICATORS SCORING: A Formal Application Letter 5 Score SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) Linkage B Commitment to fund operational costs (procurement, contract management, financial management and monitoring processes) Promissory Note signed by Governor undertaking to fund operational costs 4 points Detailed operational plans provided 4 points Operational plans Budget provided 4 points 12 None None None C Evidence of institutional capacity to implement CGS projects 20 In deciding the allocation of funds within the CGS, the NCCGS will take into account whether the State has adequate Relevant MDAs 59 to implement CGS projects are in place in the State 3points None 59 Relevant MDAs were interpreted as follows: A. Primary Health Care Ministry of Health, PHC Departments, HMB, etc; B. Water Ministry of Water Resources, Rural Water and Sanitation Board, etc; C. Skills and Economic Emp. Ministry of Youths, Ministry of Commerce and Industry and Coordinating Unit for Skills Acquisition Centres, etc.d. CCT Relevant State MDA that will be responsible for CCT Initiative E. Agric Ministry of Agric, ADP, etc. 49

58 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 institutional structures to support implementation, i.e. relevant MDAs, State and LG Joint Planning Board, SEEDS Implementation Committee, etc. with appropriate staffing, operating environment and systems for the purpose of implementing CGS projects. States are not expected to create new institutional structures for implementing CGS, rather they should utilize existing institutional framework. They must also show that there are existing functional institutional structures that will be utilized for implementation of CGS projects.(paragraph 3.4.1) There are budgetary provisions for operational costs of the relevant MDAs in the 2009 State budget 3 points The State has a duly constituted Committee or Board that will oversee the States CGS project implementation 4 points PSU Assessment 10 points PI-7 Extent of unreported government operations SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) PFM: Output 2, Dimension 1:Budget Preparation [Note: It is assumed that SPARC support on improving the budget accuracy, should reduce off-budget provision of operational costs] None None Limited Linkage D Evidence of institutional capacity to account for CGS funds 35 The National Committee will also take into account whether there are adequate institutional structures in the State to account for grant expenditures, for example, public expenditure management 50% of Actual Score of the State in Measures 2B, 2F, 2G and 4A of 2006 Benchmarking Exercise 11 points 50

59 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 reforms and changes in state budget processes. The basis of this decision will be the SEEDS bench-marking exercise (particularly measures 2B, 2F, 2G and 4A) performed in 2006 by the National Planning Commission. Where States have evidence of their own improved systems since the 2006 SEEDS Benchmarking, they are welcome to send documents supporting their case with their application (paragraph 3.4.2) Measure 2B The budget demonstrates a clear link between resource allocations and the poverty reduction strategies defined in SEEDS; Measure 2F Budget allocations are released on schedule and there are timely and accurate reporting and dissemination of in-year budget execution reports; P&S (1): Provision by Politicians of Policy and General Budget Envelope Guidelines P&S (8): Sector Policies are Financially Feasible PI-16 Predictability in the availability of funds for commitment of expenditures PI-24 Quality and timeliness of in-year budget reports PI-25 Quality and timeliness of annual financial statements P&S (B1). Provision by politicians of policy guidelines and realistic budget envelopes P&S (A1): Medium Term Plans are Revised Annually, and support the policy agenda PFM (A1): Medium term sector strategies and three year rolling forecasts PFM (D3): Expenditure commitment controls PFM (F4): Internal monitoring reports for budget execution SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) P&S: Output 1, Dimension 1: Develop Medium-Term Renewable Plans [Notes: The MTSS approach is the main approach through which SG/ SPARC aims to strengthen the link between resource allocations & policies] PFM: Output 2, Dimension 3: : Expenditure Control & Cash Management [Notes: The CGS criteria focuses on the release of budget allocations, but SG/SPARC is also working on the accuracy of the budgets in the first place] Linkage Strong Strong 51

60 CGS Qualifying Criteria Score 52 SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Measure 2G Accounts are prepared on time, audited according to statutory requirements and audit recommendations are acted upon; and Measure 4A Plans to ensure due process, particularly in procurement procedures should be developed and positively assessed against Federal Government due process standards as well as punish corrupt acts. Evidence of Budgetary classification system capable of tracking poverty reducing expenditures separately 3 points Evidence of 2008 and first quarter and second quarter 2009 budget monitoring reports 3 points PI-22 Timeliness and regularity of accounts reconciliation PI-21 Effectiveness of internal audit PI-19 Competition, value for money and controls in procurement PI-5 Classification of the budget PI-6 Comprehensiveness of information included in budget documentation PI-24 Quality and timeliness of in-year budget reports PFM (C1): Coverage and quality of internal audit PFM (C2): Understanding of and compliance with regulations and procedures by officers in MDAs PFM (L2): use of open competition PFM (A4): budget classification PFM (A5): budget documentation PFM (F4): internal monitoring reports for budget execution SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) PFM: Output 2, Dimension 5: Accounting & Reporting [Notes: The SPARC workplan is not working on auditing, but instead the focus is more on the reconciliation of the below the line accounts ] [Notes: The SPARC workplan is not working on procurement issues in Yr 1] PFM: Output 2, Dimension 1:Budget Preparation [Note: SPARC is supporting work on budget classifications] PFM: Output 2, Dimension 5: Accounting & Reporting [Notes: From the workplan SPARC does not appear to be working on budget monitoring reports] Limited? Strong None Limited Linkage

61 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Evidence of 2007 and 2008 State Audited Financial Statement 3 points Evidence of Submission of State Fiscal Responsibility Bill to the State House of Assembly 2 points Evidence of Published Due Process or Public Procurement Manual 3 points Regularity of Reporting and Auditing of CGS 2007 and 2008 projects 10 points E Evidence of human capacity development, facilities and equipment 10 States are required to submit evidence of a plan in the areas of human capacity development, provision of facilities, equipment and supplies. For example, a proposal to enlarge health-care facility would require the hiring and training of more health attendants and professionals (doctors, nurses, etc) and technical assistance requirements for knowledge Evidence of Training Programme in areas relevant to the CGS Project Areas 3 points PI-26 Scope, nature and follow-up of external audit PFM (F6): submission of financial statements to auditor general PSM (Dimension 8): Manpower Development Policy and Management System Training and Capacity Building SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) PFM: Output 2, Dimension 5: Accounting & Reporting [Notes: According to the SPARC workplan, it is not supporting external audit in Yr 1] PSM: Output 3, Dimension 8 Manpower Development Policy & Management System [Note: the CGS criteria relates to training in support of the project, while SPARC working on a more systematic approach to human capacity across the civil service] Linkage Limited None? None None Limited more project specific 53

62 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 transfer etc. The proposal should include sources of funds required to actualize the plans. (paragraph 3.4.4) Evidence of Budgetary provision for the training programme 3 points Evidence of plan to recruit additional staff for running of the proposed project(s) at completion 2 points Evidence of provision of additional facilities and equipment for the proposed project(s) 2 points. F Evidence of 50% cash counterpart contribution provided in Budget 18 P&S (9) Sector Strategies are consistent with the human and organisational capacity to implement them P&S (D5): Sector Strategies are consistent with human and organisational capacity to implement them SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) PSM Output 3, Dimension 1: Mandates, Statutory Functions & Structure of KDS Public Service [Note: the CGS criteria relates to the project, rather than SPARC s support which is across the civil service] Linkage None Limited more project specific None To reinforce State commitment to ownership and project implementation, States will be required to show evidence of budgetary provision for the 50% cash counterpart contribution of the project costs under the First Track of the CGS (paragraph 3.4.3) Evidence of 50 % of total proposed Projects provided for in States 2009 Budget 18 points or Supplementary Appropriation Bill presented to the HA 15 points or Letter of undertaking from governor to submit Supplementary Appropriation Bill 12 points or None None None 54

63 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Evidence of process at an advanced stage 10 points SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) Linkage None PROJECT INDICATORS SCORING: 1. Issues of Project Submission Form 10 For a project to be eligible for funding under the conditional grants scheme, a comprehensive work plan and cash flow for each project must be submitted. The application template (i.e. annex 2) must be completed in full for each project... Detailed specification, designs and costing/bill of quantities should be prepared for all projects submitted for funding. Costing should be comprehensive, disaggregated against each activity and items. Costs should be realistic and reasonable as well as taking into consideration all statutory payments expected to be incurred... (paragraphs ) Code provided 1 point Proper Identification of project 1 point Proper description of project 1 point Location of project stated in recommended form 1 point Work plan of project 2 points Cash Flow Statement 1 points Deliverables 1 point Indicators of Progress 2 points 2. Justification and Policy context of Project 35 None project specific 55

64 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Projects sponsored under the CGS must be related to priorities stated in policy documents at the Federal, State and LG levels. Specifically, applications for the funds must exhibit evident expenditures that relate to highly-ranked State priorities in SEEDS, LEEDS or equivalent policy documents which also are consistent with the 7-Point Agenda, Federal Government Development Plan, Vision 20:2020, and the MDGs...The relevant sections of a State policy documents should be forwarded as part of the State s application... (paragraphs ) Priority of the State (e.g. high MMR, under five mortality, above prevalence rate of HIV/AIDS, low access to water, high level of marginalized and vulnerable groups, etc) as indicated in SEEDS or LEEDS or similar policy strategy document 15 points Proposed project is the best approach to address the identified problem (e.g. renovation of Health Facilities as a means of addressing high MMR and under five mortality, provision of equipment for VCT centres as means of addressing above prevalence rate of HIV/AIDS, etc) P&S (3): Research and Baseline Information used in Policy Definition (p17) P&S (D2): Research and baseline information is used in Policy and Strategy Definition (p69) P&S (A1): Medium Term Plans are Revised Annually, and support the policy agenda (p66) SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) P&S/M&E: Output 1, Dimension 5: Increased availability of data analysis and information P&S/M&E: Output 1, Dimension 1: Develop Medium-Term Renewable Plans (p21) [Note: the CGS criteria is more about the MDGorientation of the policies, rather than their quality, such as being based on a solid evidence-base and realistic resource envelopes] Linkage Limited more about existence of policy documents None project specific 56

65 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Projects funded under the CGS should be physically located in areas of greatest need to be informed by baseline data on State service delivery infrastructure and poverty maps. The State should show evidence of infrastructure needs by including in its application a State gap mapping of projects related to those for which it is applying... (paragraph 3.3.6) The project is located in areas of greatest need. Evidence that proper GAP analysis has been conducted 15 points or Spread of project locations across the LGAs only (12 points for Health Supplies; 10 points for CCT; 10 points for Health Facilities Rehabilitation & Construction; 7.5 points for Water, Agric and Skill & Economic Empowerment) 3. Proper management, maintenance/ sustainability of project at completion To show the ability to operate, monitor, evaluate, maintain and sustain CGS projects, States must forward along with their application a standalone binding promissory note, signed by the Governor which outlines in detail their plans for the operation, monitoring, evaluation, maintenance and sustainability of the projects proposed for funding (paragraph 3.39) Promissory Note on proper management and maintenance/sustainability of project signed by Governor 8 points Detailed management and maintenance/sustainability plan provided 10 points The plan is costed 7 points Funding source of the plan is shown 5 points 4. Consultation with Federal MDAs for project rationale, location, specification, cost, due process mechanism, etc M&E 5: Performance management processes in support of policy, strategy and program adjustment. Dimension F. Increased availability of data analysis and information for M&E, Policy Formulation and Strategic Planning (p76) SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) M&E (Dimension 5): Increased Availability of Data Analysis and Information (p22) Linkage Limited Some link if MDAs are making better use of data None project specific 57

66 CGS Qualifying Criteria Score SEAT Dimensions Change Matrix Implementation Manual 2009 Evaluation of Bids 2009 Kaduna State Change Programme, May 2009 Formulation of the project proposals should also be a product of close collaboration between State Government and Federal MDAs; this is to ensure that there is coherence in policies, planning and implementation of projects at the Federal, State and Local Government levels. This will also harmonize development efforts as well as enhance consistency in project location, technical specifications, and costs, thus avoiding duplication of investments (paragraph 3.4.5) Evidence of ongoing communication with relevant Federal MDA on the projects 10 points 5. Consultation with Local Government in project identification, location, costing, etc It is mandatory for States to work in close collaboration with their Local Governments to determine the priority projects for which conditional grants will be made. All LGAs applying for funds to undertake projects under the First Track of the CGS should therefore forward their proposals to State Government, who shall after proper collaboration with LGs apply for funds under CGS. (paragraph 3.4.5) LG applies through State Government 15 points or Evidence of consultative meeting between relevant State and Local officials 15 points or Evidence of correspondence between State and LG 15 points 15 SPARC Workplan Dimensions SPARC Annual Work Plan (Year 1, revised, December 2009) Linkage None project specific None project specific 58

67 Appendix Eight: Summary of Assessments of Functional CGS Projects (from Links Study) The 7 projects visited Green highlight Achievements (outputs such as buildings or functional boreholes) Yellow highlight - Satisfaction levels from users (patients, staff, etc) Blue highlight Key challenges, such as to do with sustainability, recurrent costs and need for other agencies to provide complementary inputs (e.g. staff). Adamawa Edo Imo Jigawa Kwara Lagos Motorised Borehole Project, Dakre: The team visited a 2008 CGS project, a borehole attached to a PHC in Dakre community in Girei Local Government area. This borehole was meant to serve the PHC. Though the water project was approved for funding, the PHC was not approved. The same PHC has procured EOC materials from CGS funds but the maternity ward is dilapidated. The Oregbeni PHC: Located in Ikpoba- Okha LGA, Oregbeni PHC was one of the healthcare facilities that benefitted from the 2007 CGS. Included in the proposal are residences for staff and borehole for the facility. The facility is quite spacious with enough rooms for treatment and observation of patients. The borehole is still functional. Unfortunately, the team were unable Amaruku Motorised Water Scheme: This project is located in Amaruku town in Isiala-Mbano LGA. It is reticulated to serve about 15 villages within a radius of six kilometres. The grant process and implementation timelines are summarized below [Not included here]. The water tank has a capacity for 312,000 litres. It is powered by a Perkins 60KVA. Because of the poor supply of electricity in the area, the operation of the water scheme is virtually totally dependent on the generator. In the circumstance, water supply is provided at intervals of 2-4 days to reduce the burden on the generator. However, the beneficiaries expressed satisfaction with the project that saves them from expensive water vendors and/or sourcing water from unwholesome ponds and brooks. PHC Model Health Centre: In 2007, the World Bank and Africa Development Bank (ADB) jointly financed the construction of an additional building for an existing state-owned Health Clinic, operated by the Isiala-Mbano LGA that is responsible for the staff. The two donors also built quarters for one doctor (a three-bedroom 59 Basic Health Clinic, Sakwaya: The MDG-CGS financed the construction and equipment of the Clinic as an upgrade of an existing Health Post. The facility was one of the 2007 CGS projects in the state. It was completed during the third quarter of 2008 and commissioned in November The building is solid and the design provides adequate space for the needs of male and female patients, including space for delivery, ante-natal care, and immunization. The BHC has a laboratory and a drug revolving fund store. Significantly, in late 2008, the State Government directed LGAs to use the CGS BHC design for their own Comprehensive Health Centre: The facility is located in Ijagbo in Oyun LGA. It was built as a Basic Health Care Centre by the state government and commissioned in February It was converted into a Comprehensive Health Care Centre in the late 1990s. The renovation/upgrade of the Centre was undertaken with 2008 CGS funds. The collapsing parts of the roofing were fixed, the floor tiles in the entire building were replaced and the building was repainted. CGS also Primary Healthcare Centre, Igando, and Alimosho LGA: Of the three PHCs visited, this was the only site where the team was able to find users of the facility. The facility was renovated with support from the 2007 CGS awards. The centre is spacious (this was common with all the project sites visited). Discussions with a user who was familiar with the centre before renovation indicated that the improved centre is a welcome development as community members were hitherto forced to utilise expensive private services. The user, a

68 borehole is functional but yet to be branded. Though the PHC is yet to be fully connected to the borehole, communities around the facility are benefiting from the water project. to meet with any of the users. The team noted that two rooms within the healthcare facility have been allocated for use on a temporary basis by the State Immigration Office without the knowledge of the Focal Person. bungalow) and another bungalow with two apartments for nurses/midwives. In 2009, CGS fund (under the 2008 round) was used to finance the costs of equipment to assure quality Emergency Obstetrics Care (EOC), the purchase of an emergency ambulance and a 10KVA generator as well as the construction of VIP latrines. This is an example of CGS complementing interventions by state and local governments and two development partners to enhance the PHC s capacity to deliver better services to the public. The PHC staff and beneficiaries interviewed expressed satisfaction with the value added to the PHC through CGS s interventions. However, the inability of the LGA to provide trained staff able to use some of the equipment provided by CGS means that they have remained unused and it is uncertain when the needed staff would be recruited. The PHC staff met were of the view that the upgrading of the PHC s structures together with the substantial quality equipment provided by CGS qualify it for a secondary health facility status. We advised the State MDG-CGS Focal Person to communicate the request to the government for consideration. Two key messages emerged from the discussions held with the different stakeholders at the two projects visited. First, quasi-total reliance on generators for a motorized water scheme and a public health clinic is not sustainable. The cost of purchasing diesel and the inevitable periodic maintenance of the generators result in interrupted services of varying duration. Without reliable electricity supply from PHCN, neither of the two projects is able to provide continuous satisfactory service for the respective beneficiaries. The point about how poor electricity supply undermines the quality and sustainability of these two projects would certainly apply to similar projects in Imo state and in other states of the federation. constructions and MDG-CGS made it available to the state s Ministry of Local Government There is arrangement for a visiting doctor (twice a week) and emergencies that require a doctor s intervention are transferred to the nearest secondary health hospital. There were patients on admission. A few out-patients who were interviewed in local language (Hausa) expressed satisfaction with the quality of services provided but would like the doctor s visit to be more frequent. A Hand Pump Borehole provided by CGS in the vicinity of the BHC serves both the Clinic and the inhabitants of the surrounding settlements. Both the Community Health Extension Worker who is the permanent staff at the facility and a couple of patients who were interviewed expressed satisfaction with the functionality of the Clinic. The arrangement for meeting recurrent costs that GUNDUMA management had explained to the visiting team was confirmed. However, an examination of the Clinic s financed VIP Latrines and a motorized borehole to complement an existing borehole that relies exclusively on electricity from PHCN that is very irregular. The CGS interventions were jointly supervised by the MDG-CGS Office and the state s Ministry of Health. The interventions were completed in November According to the Centre s staff, attendance in the Centre has increased significantly since the renovation was completed. Patients come from about six surrounding towns and villages for immunization, anti-natal care, deliveries and outpatient care. The Centre has adequate staff (13 in all, including a community health officer, a nurse/midwife, a pharmacy technician and six community health extension workers), all paid by the state government, with the exception of the security officer who is an food seller who runs her business within the premises, confirmed increased patronage of the facility. 60

69 The second message is the critical importance of communication between the beneficiaries of CGS projects on the one hand and the staff of the projects and MDG-CGS office on the other. There is need for a well-publicized channel of communication that would include the use of the local language. It is through such a mechanism that complaints about interruptions in service delivery or generally poor services can be made to the appropriate quarters for attention. record on deliveries revealed underutilization for which the Community Health Extension Worker did not have a satisfactory explanation. employee of the LGA. However, there is need for a visiting doctor and the perimeter of the Centre needs to be fenced to assure the security of both the staff and the patients, especially as the Centre is open 24/7. Summary of analysis: Generally good level of completion of CGS projects (functional boreholes, buildings, etc) Anecdotal evidence of users being satisfied (though this was far from being a representative study of access, use/ attendance, and user satisfaction) But, significant issues with sustainability (reliance on motorised boreholes, irregular electricity, lack of trained PHC staff), including: the need to address recurrent costs (e.g. periodic maintenance of generators) and complementary inputs (e.g. LGA to provide trained staff) and misuse of buildings (e.g. the healthcare facility being used for other purposes the immigration office) 61

70

71

72