Actual Project Name : Rural Poverty Reduction Project - Rio Grande Do Norte (US$M. Project Costs (US$M US$M):

Transcription

1 Public Disclosure Authorized IEG ICR Review Independent Evaluation Group Report Number : ICRR Project Data: Date Posted : 04/29/2013 Public Disclosure Authorized Public Disclosure Authorized Country: Brazil Project ID : P Appraisal Actual Project Name : Rural Poverty Project Costs (US$M US$M): Reduction Project - Rio Grande Do Norte L/C Number: L4667; L7489 Loan/Credit (US$M US$M): Sector Board : Agriculture and Rural Development Cofinancing (US$M US$M): Cofinanciers : Board Approval Date : 06/27/2002 Closing Date : 12/31/ /31/2010 Sector(s): Micro- and SME finance (60%); Sub-national government administration (30%); Central government administration (10%) Theme(s): Small and medium enterprise support (34% - P); Rural non-farm income generation (33% - P); Poverty strategy; analysis and monitoring (33% - P) Prepared by : Reviewed by : ICR Review Group: Coordinator : John R. Heath Ridley Nelson Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: (I) ORIGINAL PROJECT The statement of project development objectives in the Project Appraisal Document (PAD) is similar but not identical to that in Loan Agreement (LA). Public Disclosure Authorized According to the PAD, "The project aims to assist the State of Rio Grande do Norte to reduce currently high levels of rural poverty by: (a) improving well-being and incomes of the rural poor through better access to basic social and economic infrastructure and services and support for productive activities, using proven community-driven development (CDD) techniques; (b) increasing the social capital of rural communities to organize collectively to meet own needs; (c) enhancing local governance by greater citizen participation and transparency in decision-making, through creation and strengthening of community associations and Municipal Councils; and (d) fostering closer integration of development policies, programs and projects at the local level, by assisting Municipal Councils to extend their role in seeking funding, priority-setting and decision-making over resource allocation" (p. 3). According to the LA, "The objectives of the Project are: (a) to increase social and economic opportunities for the Municipalities' rural poor by improving access to basic, social and economic infrastructure through Community Subprojects; (b) to increase the social capital of rural communities to organize collectively and meet their own needs ; and (c) to foster local governance and citizenship through creation and strengthening of Municipal Councils, at the same time forging links with governmental and nongovernmental agencies, civil society, financial institutions and the private sector" (Schedule 2, page unnumbered). (II) ADDITIONAL FINANCING The additional financing that was approved in 2007 had the same objectives as the original project. The statements

2 of project development objectives in the Project Paper and the Loan Agreement were not identical. The Project Paper that was prepared in support of additional financing modified sub-objective (a) of the PAD by adding one word ("employment"): "Improve wellbeing, employment and incomes of the rural poor through better access to basic social services and economic infrastructure and services, and support for productive activities" (p. iv). The LA for additional financing contained the following statement of objectives (which, by adding objective (d), came closer to the PAD than the LA for the original project: "The objectives of the Project are: (a) to promote wellbeing and income-generation for the rural poor by improving access to basic, social and economic infrastructure as well as support for productive activities through Community Subprojects; (b) to increase the social capital of rural communities to organize collectively and meet their own needs; (c) to enhance local governance and citizenship through creation and strengthening of Municipal Councils; and (d) to foster closer integration of development policies and programs in rural areas at the local level by strengthening Municipal Councils in their fund-raising, prioritization and decision-making capacity, at the same time forging links with governmental and non-governmental agencies, civil society, financial institutions and the private sector" (Schedule 1, p. 6). IEG evaluates the project against the statement of objectives contained in the PAD, because this makes clear that the project's overarching objective is to reduce rural poverty: sub-objectives (a)-(d) in the PAD are treated as intermediate outcomes. All five outcomes will be assessed. b.were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1. Community Subprojects (Expected cost at appraisal, US$26.0 million; Additional financing, US$25.5 million; Actual cost at closing, US$50.4 million). This component provided matching grants to rural community associations to finance approximately 1,500 infrastructure, productive and social subprojects (up to US$50,000 each) identified by these groups as priority investments that would improve community well-being. Subprojects were intended to benefit 83,000 families, or about 370,000 people. This component included three subprograms: State Community Schemes (PAC). Under the PAC subprogram, rural communities submitted their investment proposals directly to the State Technical Unit (the project implementing agency), which screened and approved them and released funds to the beneficiary associations. Municipal Community Schemes (FUMAC). Under this subprogram, decision-making on investment proposals was delegated by the State Technical Unit to project Municipal Councils, composed of community members and representatives of civil society and municipal authorities. At least 80 percent of Council voting members were potential project beneficiaries and civil society representatives. The Municipal Councils discussed, and sought to build consensus on priorities and approve community proposals, in the context of an indicative annual budget amount determined by the state. Pilot Municipal Community Funds (FUMAC-P). The State Technical Unit established an annual budget envelope, according to a distribution formula based on clear and measurable criteria (rural population, poverty levels and previous year's performance). Based on this budget, Municipal Councils submitted an Annual Operating Plan (Plano Operativo Anual) for review by the State Technical Unit. Upon approval, funds were transferred to the Municipal Council, which was then responsible for managing their distribution to community associations and assisting them with implementation of subprojects. 2. Institutional Development (Expected cost at appraisal, US$1.5 million; Additional financing, US$1.7 million; Actual cost at closing, US$5.0 million). This component financed technical assistance and training to build the capacity of the Community Associations, the Municipal Councils and the implementing agency. It also included funding for technical assistance to support state modernization and reform issues relevant to poverty reduction.

3 3. Administration, Supervision, Monitoring and Evaluation (Expected cost at appraisal, US$1.5 million; Additional financing, US$1.5 million; Actual, US$3.3 million). This component financed the incremental costs (excluding salaries) of project administration and coordination including supervision, monitoring and impact evaluation. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs Under the original project, the total cost was estimated as US$30.0 million at appraisal, falling to US$28.8 million at closing. Under the additional financing, the total cost was estimated as US$30.0 at appraisal and was US$30.1 million at closing. The average cost of individual subprojects rose during project implementation owing to a higher than expected proportion of productive projects (which tended to cost more than infrastructure projects), depreciation of the US dollar, and increases in the cost of goods and materials. Disbursements totaled 99 percent of the combined commitment under the original project and additional financing, the shortfall resulting from delays in subproject approval that reflected the capacity limitations of the State Technical Unit, the implementing agency for the project. Financing The loan agreement was amended three times to accommodate reallocation of funds between components and extensions of the closing date; none of these amendments involved Board approval. Owing to rising average subproject costs and strong demand for subprojects, US$3.3 million in 2006, and a further US$0.9 million in 2007, was shifted to the Municipal Community Schemes Grants (FUMAC) window. In 2010, under additional financing, US$1.1 million was reallocated to Consultant Services and Training, and Project Supervision and Monitoring. Borrower Contribution Adding funding from the original project to the additional financing, the expected Borrower contribution was US$14.6 million, and the actual contribution was US$14.2 million. The Borrower contribution included both the counterpart from the state government and the funds contributed by Community Associations (who were responsible for paying 10 percent of the investment cost). Dates For the original project, on May 26, 2006, the expected loan closing date (December 31, 2006) was extended by one year to permit the commitment and disbursement of the remaining loan balance. It was further extended to December 31, 2010 when the additional financing was approved in Relevance of Objectives & Design: a. Relevance of Objectives: (Rating: Substantial) Throughout the implementation period the relevance of this project s objectives was consistently endorsed in the Borrower and Bank's strategy. The overarching objective of rural poverty reduction was relevant because, when the project was appraised in 2001, 43 percent of rural families in the state of Rio Grande do Norte lived in poverty. Government data showed that 56 percent of rural households in Rio Grande do Norte lacked piped water supply, 20 percent were without sanitation, and 14 percent had no electricity. The Borrower attached particular importance to the aim of using the subproject process to leverage financing from sources outside the project (sub-objective (d) in the Project Appraisal Document) a new departure for this series of community-driven development projects, one that was relevant because it increased prospects for sustaining the drive to reduce poverty. The Bank s FY Country Assistance Strategy (dated May 24, 2001) identified poverty and inequality reduction as the core of Bank assistance efforts and stressed well-targeted, decentralized programs, social capital formation and local integration of programs. The Additional Financing accorded with the FY CAS (dated November 10, 2003), which called for successive projects under the Northeast community-driven development program to finance basic infrastructure for the rural poor, support income generation, and promote closer integration of state and federal rural initiatives in participating municipalities. The Country Assistance Strategy that was current when the loan closed (covering the period 2008 to 2011)

4 acknowledges the validity and the success of the series of Northeast community development operations of which this project is a part, "both in building social capital and in enabling poor communities to get access to water and electricity" (p. 10). However, the FY08-11 CAS did signal a change of emphasis, indicating that in response to demands from state governments in the Northeast, the Bank would henceforth address the development needs of rural municipalities by prioritizing the promotion of productive subprojects connected to high-value supply chains (e.g. contracts with supermarkets). This signaled a shift of emphasis away from the rural community infrastructure investments, promotion of which was central to the project. b. Relevance of Design: (Rating: Substantial) Design relevance was enhanced by the incorporation of lessons learned from the two earlier community-driven development projects in this series of operations for Rio Grande do Norte. The project components were sufficiently few in number, clear in conception, and flexible in implementation to make it more likely that the objective and intermediate outcomes would be achieved. Attainment of project objectives was facilitated by a process that promoted community commitment to the investments by involving would-be beneficiaries in identifying subprojects, choosing between investment alternatives, and cofinancing of the subprojects that were selected. The components included the training and technical assistance to Community Associations and Municipal Councils that was needed to make them more effective and more sustainable agencies for local planning. Equally important was the leaning by doing that implementation entailed. The availability of a pre-existing menu of tried and tested subproject options helped to make these small-scale investments more feasible. The existence of three distinct windows for sub-project financing made it possible to tailor the project process to the institutional capacity of the various communities, with greater delegation of decision-making to communities with greater capacity (using the FUMAC and FUMAC-P windows, explained in Section 2c above). The transparent and participatory process of subproject selection helped to ensure that decisions reflected the will of the majority of would-be beneficiaries, reducing the scope for elite capture. Project design ensured that around 90 percent of project funds flowed directly to beneficiaries, increasing the scope for attaining the overall objective of poverty reduction. The approach taken to targeting the poor by area was simple and relatively sound: of the 167 municipalities in the state, 12 with higher income levels were excluded from the Original Project; the eligible municipalities were then divided into three groups according to their scores on the Human Development Index (HDI), with more funds per municipality being reserved for the poorest group (PAD, p. 8). However, under additional financing the number of eligible municipalities was increased to 165 (although the weighting of resources by HDI grouping was retained). 4. Achievement of Objectives (Efficacy): (a): Improve well-being and incomes of the rural poor through better access to basic social and economic infrastructure and services and support for productive activities, using proven community-driven development techniques. (Rating: Substantial) Outputs Adding the results under the original project to those achieved under additional financing, subproject investments benefited 106,857 families (80 percent of the combined target) and 2,610 subprojects were delivered (83 percent of the combined target). (Under the additional financing, the subproject target was cut from 1653 to 810 in November 2010, because the average cost per subproject exceeded the initial estimate and the larger than expected number of families per subproject; 801 subprojects were delivered.) Subprojects were distributed among the following categories: infrastructure (48 percent), productive (44 percent), and social (8 percent). Communal water supply accounted for the largest number of subprojects (36 percent), followed by raising of goats or cattle (21 percent) and dams (9 percent). Subprojects were selected from a menu of tried and tested options, helping to ensure they were technically sound. In the course of subproject appraisal, 26 percent of the proposals made were turned down for financing, for diverse reasons (ICR, p. 27). Physical performance studies in 2005 and 2010 confirmed that, in most cases, subprojects were being soundly operated although beneficiaries expressed the need for better technical assistance 75 percent reported that they had not received any assistance (ICR, pp ). A 2010 study found that 83 percent of the subprojects funded under the original project were functioning regularly. Outcomes Targeting to the poorest areas increased over time: under the Original Project, 28 percent of project resources flowed to the poorest 24 percent of project municipalities; under the Additional Financing, 42 percent of project

5 resources went to the poorest 36 percent of project municipalities. (The project covered poor rural municipalities generally, and was not targeted to the poorest of the poor). The data that the ICR presents does not show significant income and employment gains. Economic returns on productive activity were very low, as were incomes. About 36 percent [of beneficiaries] said their experience with the project had resulted in income gains (ICR, p. 48). Annex 3 of the ICR reports on income gains attributable to the project for certain subprojects, noting however that the data derive from only 12 subprojects implemented in Rio Grande do Norte, whose representativeness cannot be vouched for (ICR, p. 38). However, based on conservative estimates, the ICR reports that 9,581 jobs were created, mainly from productive subprojects (milk production, irrigated agriculture, fishing and crafts) (ICR, p. iv): taking into account total beneficiaries from the three categories of subproject, this translates into 0.1 jobs per beneficiary family. A study by Binswanger and others compared beneficiary and control households for three states including Rio Grande do Norte, comparing circumstances in 2005 (when the survey was conducted) with interviewees recall of circumstances in (The loan for the Rio Grande do Norte project was made effective in September 2002). This study found that, with respect to income and household assets, there was no statistically significant difference between the treatment and the control group. The project team provided additional data showing that real rural household per capita declined from then subsequently rose from R$147 in 2003 to R$249 in These figures would include project areas and other rural areas in Rio Grande do Norte. It is unclear how much of this improvement would be due to the project as opposed to general improvements in the economy or expanding social protection programs. The Binswanger study found that access to clean water increased by 12 percent between 2002 and 2005, citing this as a key reason why, for the treatment group, there was a 38 percent fall in infant mortality and a 70 percent drop in the incidence of diarrhea relative to the control group (ICR, p. 58); attribution of the health benefits to the project was speculative and did not take into account other determinants of diarrhea and infant mortality. Further, these results are aggregated across three states in the survey, In Rio Grande do Norte, a 2005 Results Evaluation (SETHAS/IICA 2005) of randomly-selected subprojects, conducted in December 2004, found that 45 percent of respondents reported diverse health benefits of water supply investments, and 35 percent stating that the incidence of diarrhea had decreased as a result of clean, piped water. There was no study to confirm the health benefits, their magnitude, or attribution to the project, however. The project team provided the additional information that between 2002 and 2009, access to potable water in rural areas of Rio Grande do Norte increased from about 44% to nearly 70%. These results include but are not limited to the project areas and would have been affected by other contemporaneous government water programs, but the project s outputs likely contributed to the trends. (b): Increase the social capital of rural communities to organize collectively to meet own needs. (Rating: Substantial) Outputs The number of Community Associations and Municipal Councils formed under the project, and the training they received, may be construed as evidence of increased collective organization. 2,057 Community Associations were created (there was no target). 70 Municipal Councils were created; the target was 76. The number of training courses delivered to Community Associations and Municipal Councils was respectively 21 percent and 18 percent above the target. Outcomes The study by Binswanger and others referred to above found that, relative to the control group, the treatment group experienced, between 2002 and 2005, a bigger increase in community participation in associations, collective decision-making and collective activities, and that beneficiaries viewed Community Associations as inclusive, participatory and democratic institutions where decisions were made by the majority (ICR, p. 50). Other studies showed more mixed results (ICR, p. iv). A Social Capital Index was part of the project design but the necessary data were not collected to assess change between the start and finish of project implementation. (c): Enhance local governance by greater citizen participation and transparency in decision-making, through creation and strengthening of community associations and Municipal Councils. (Rating: Modest) Outputs For the Community Associations, 21 percent of the members were women, and for the Municipal Councils the proportion was 27 percent. Women led 19 percent of the Community Associations and 18 percent of the Municipal Councils. However, none of the Municipal Councils created were of the FUMAC-P type (see Section 2c above for explanation), which was, according to the project design, the most advanced model of decentralized decision-making and administration; the target was to create 4 of these councils. Outcomes A 2010 survey found that 85 of the Community Associations created under the original project still continued to

6 function formally three years after funding of their respective subprojects had been completed. However, Municipal Councils "remain dependent on the mayors/unions for logistical support (premises, transport and other). This combined with modest 2 percent "fee" available from each subproject, limited their autonomy" (ICR, paragraph 3.2.4). The study by Binswanger and others found that, relative to the control group, the treatment group reported, between 2002 and 2005, only a modest increase in the extent that Municipal Councils provided information on the actions of the municipal administration, and in under 30 percent of cases the Municipal Councils did not deliberate on federal, state and municipal programs (that is, initiatives outside the project). A later report (COPES 2011) found that 40 percent of all Councils demonstrated the capacity to understand and debate other state and federal programs. The most decentralized model of Municipal Councils (FUMAC-P) was not implemented. (d): Foster closer integration of development policies, programs and projects at the local level, by assisting Municipal Councils to extend their role in seeking funding, priority-setting and decision-making over resource allocation. (Rating: Modest) Outputs There is no evidence that Municipal Councils facilitated integration by serving as a focal point for discussions on different policies and programs, by helping to set priorities or by leveraging funds. "Most Councils still had limited capacity to respond to the project's integration vision or to formally collect relevant data at the local level" (ICR, paragraph 3.2.5). The project implementing agency leveraged complementary financing of R$96.6 million [about US$58 million] from many federal, state and local programs for the project's targeted poor. This was less than envisaged at appraisal but still 129 percent of the loan amount (combining Original Project and Additional Financing). The contribution of Councils to leveraging funds from outside sources is not clear. A 2010 survey found that 15 percent of Community Associations operating under the original project and its forerunner had prepared new subprojects, with one-third of these new subprojects funded from other programs; but the role of the Municipal Councils in leveraging these outside funds was not studied (ICR, p. vi). Outcomes Overall, the Community Associations and Municipal Councils were not able to plan and execute investments with funds from outside the project, suggesting that the "integration" that was sought was not achieving, making it possible that planning capacity nurtured by the project would not be sustained once the project ended. (e) Assist the State of Rio Grande do Norte to reduce currently high levels of rural poverty. (Rating: Substantial) The project team provided additional evidence that the incidence of extreme rural poverty in Rio Grande do Norte rose slightly from , after which it declined from 30% in 2003 to 13% in There are no doubt many factors affecting this reduction including the expansion of social safety net and rural water supply programs and a general economic improvement in Brazil. However, the extent of targeting and the evidence of improved access and incomes suggest that the project activities likely contributed to poverty reduction in the project area. 5. Efficiency: Benefit-cost analysis. The ICR gives an economic rate of return of 27 percent compared to the appraisal estimate of 28.5 percent; but this is not credible, given the small number of case studies (only 13 cases, 1 percent of the total), the non-random nature of their selection, and the incompleteness of the economic analysis for those subprojects included as case studies. The 13 subprojects chosen spanned the range of subproject types but only 6 were subjected to full economic and financial analysis. The project team provided the additional information that the subprojects were selected to be typical, but it is not clear whether this comprised only successful projects. The share of unsuccessful projects is not reported. The ICR notes that "Statistical extrapolations for the entire universe of subprojects were neither intended nor possible" (p. 39). Cost-effectiveness. Subsequent to the ICR, some comparative cost figures were provided by the project team suggesting that public investments of a similar type were more costly. Evidence obtained from the Borrower after the ICR was written showed that for family water cisterns average cost under the project was R$1,600, compared to R$3,500-$4,000 under state and federal programs (SEMARH/FUNASA); for simplified water supply systems, costs per household were respectively R$1,171 and R$5,000 for comparable systems; for small-scale irrigation systems, the project cost was R$42,000 compared to R$123,000 for a comparable system (SEMARH); for rural electrification, average costs per household were R$1,377 under the project, compared to R$2,620 under the state program (COSERN). The actual cost of project management (Component 3) was 6 percent of total project cost, consistent with claims in

7 the PAD about the limited overheads involved in this approach to community-driven development. Combining original and additional financing, actual spending on subprojects (US$51.5 million) was 98 percent of the expected amount, while 80 percent of the expected number of families benefited from the project. The evidence on the precision of targeting is limited, raising the possibility that project funds leaked to the non-poor. Fiscal savings. One significant area of cost savings is foregone expenditures for public water distribution via tanker trucks. In Rio Grande do Norte, 53,000 families benefited from 949 water subprojects at a combined cost of US$26.6 million. The monthly cost of water trucks for those families the only viable alternative without the project would have been about US$2.7 million. Efficiency is rated modest. a. If available, enter the Economic Rate of Return (ERR ERR)/Financial Rate of Return (FRR FRR) at appraisal and the re-estimated estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 28.5% 100% ICR estimate Yes 27% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The objective of reducing rural poverty was substantially relevant in terms of conditions in Rio Grande do Norte, and in relation to Bank and Borrower strategy. The relevance of design was also substantial, because the project components were tried and tested, and they were appropriate for improving access and incomes, increasing social capital and local governance, and reducing poverty. Three of the objectives were substantially achieved and two were modestly achieved. The analysis of efficiency is incomplete and the estimates of economic rate of return are based on a small, non-random sample. a. Outcome Rating : Moderately 7. Rationale for Risk to Development Outcome Rating: Subprojects were selected from a tried and tested menu of options, increasing the prospect that they would be sustainable. Environmental screening processes were apparently adequate. Provisions for operation and maintenance were included in the contract between the Community Association and the State Technical Unit. A 2010 survey conducted by the implementing agency found that 85 percent of the subprojects funded under the original project were still operational three years later. However, the project area is subject to drought, which may undermine the sustainability of the productive projects (ICR, p. 42), these accounting for 44 percent of all subprojects financed. The project made special provision for agricultural extension to productive subprojects (ICR, p. 19), which may have enhanced their viability. On the other hand, the ICR cites a 2005 survey (p. 18) which found that 75 percent of interviewees claimed not to have received technical assistance. Also, the ICR presents no evidence that the beneficiaries of productive subprojects were able to obtain loans from credit institutions, creating further doubts about sustainability. The evidence that Municipal Councils played an active role in leveraging funds outside the project is mixed, possibly reducing the scope for sustaining project benefits. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project's objectives were relevant to Borrower and Bank strategy at the time of appraisal and afterwards; and

8 its design was relevant to the attainment of those objectives. However, there were five respects in which implementation arrangements could have been stronger. First, the arrangements did not, at first, allow for local project offices (in addition to the State Technical Unit, located in the state capital, which had overall implementation responsibility). This omission reduced the scope for close and timely subproject oversight. Following the mid-term review, regional implementing offices were introduced. Second,, in Rio Grande do Norte, there was less scope than in other states for the Municipal Councils supported by the project to serve as a central clearing house and a single conduit for funds flowing from different sources, thus facilitating local-level integration of programs and policies. "The PAD over-stated the Councils' and communities' track record on and propensity/capacity for integration" (ICR, paragraph 2.1.6). Third, risk assessment was incomplete. The ICR notes that, "Risks associated with project targeting might have mentioned the importance of intensive consultation, mobilization and organization of vulnerable populations from the preparation phase forward to ensure their capacity to demand access on an equal footing with better-organized communities. Also, the pervasive lack of technical assistance in rural areas should have been acknowledged and a mitigation framework designed and implemented early on, especially in light of the project's productive goals and consequent need for sustained, specialized support" (ICR, paragraph ). Fourth, subproject targets were not met because the demand for infrastructure projects was overestimated at appraisal (see Section 3 above), owing to the displacement effect created by other poverty reduction initiatives; this suggests that appraisal didn't sufficiently consider developments outside the project. Fifth, the design of monitoring and evaluation was weak. The shortage of baselines and targets in the project's results framework compromised preparation of any final reckoning of project impact. This was particularly noteworthy given that this project was the latest in a long series that had not previously been subject to formal impact evaluation. The PAD overestimated the capacity for the Municipal Councils to develop into significant arbiters of the local development process (ICR, paragraph 2.1.6). Quality-at at-entry Rating : Unsatisfactory b. Quality of supervision: The ICR reports that field visits were based on a random selection of communities, Municipal Counties and subprojects. There was a substantial mid-term review, leading to preparation of a comprehensive action plan and appropriate follow-up. Careful field inspections helped to ensure that safeguards were well implemented. However, there were several unexplained supervision gaps of 9-12 months each under the original project. This was a significant oversight given the insufficient field-level presence of the implementing agency, a weakness that was picked up at mid-term review (ICR, footnote 9). Missions often consisted of just one person. Supervision did not sufficiently tackle the shortcomings of the Municipal Councils. Also, the implementing agency was not sufficiently encouraged to explore ways of redressing the lack of technical assistance in rural areas. There was a delay in restructuring the additional financing to reduce subproject targets to more realistic levels. The additional financing was declared at risk in 2010 because of slow implementation, but this was not reflected in the Implementation Progress rating (ICR, paragraph 2.2.8). The monitoring and evaluation "baseline" study was conducted seven years into project implementation. Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government of Rio Grande do Norte provided steady support. Counterpart funding was reliably provided. The government granted considerable autonomy to the State Technical Unit which, according to the ICR, helped to smooth project execution. However, the inadequate detail on targeting and the shortfalls in transparency (both

9 reported in the next section) suggest that closer oversight by state authorities was called for. Government Performance Rating Moderately b. Implementing Agency Performance: The ICR attributes several failings to the State Technical Unit, which had overall responsibility for implementation: " excessive bureaucracy; uneven communications with and operational support for [Municipal] Councils and associations; weaknesses in technical assistance; and, lack of adequate, specialized manpower and at times the correct structure and internal arrangements for its role. Its regional field presence was not established until quite late" (paragraph 5.2.2). The implementation of targeting was mixed. Under the original project there are no data to indicate what proportion of subprojects went to the poorest group of municipalities (ICR, p. 31). Under additional financing, 39 percent of the 801 subprojects went to Area A (poorest municipalities), 40 percent went to Area B, and 21 percent went to Area C (least poor municipalities). The completion report suggests that there was some lack of transparency in the operation of the State Technical Unit: "It is not known whether any of the R$96.6 million in complementary resources leveraged was channeled through the [Municipal] Councils, and the additional amounts and beneficiaries are not counted in project data" (ICR, paragraph 2.1.6). Implementing Agency Performance Rating : Unsatisfactory Overall Borrower Performance Rating : Moderately 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project results framework could have been stronger: some indicators (such as, social capital) were hard to measure; and appropriate baselines or targets were not set. There was no provision at the design phase for measuring attainment of the ultimate objective of reducing rural poverty: While an adequate Management Information System was inherited from the previous project, there does not appear to have been much effort to develop commitment to detailed evaluation by the implementing agency or state government. b. M&E Implementation: The Management Information System performed well, generating detailed information about subproject inputs and outputs. There was a web-based platform that permitted real-time data entry direct from the field. Evaluation was less satisfactory. The "baseline" study was not conducted until 2010: "seven years late" (ICR, footnote 14). Other studies comprised: reports on the "physical performance" of subprojects in 2005 and 2010; a 2005 "Evaluation of Results"; and a 2009 quasi-experimental study of projects of the same CDD series in three Northeast states, including Rio Grande do Norte (ICR, p. 46). The ICR notes that "a final impact evaluation is under completion" (paragraph 2.3.2). c. M&E Utilization: There is no detail in the ICR to suggest that M&E results facilitated improvements to project design or implementation; or that there was any attempt to compare the overall effectiveness of this project, relative to other poverty reduction initiatives carried out in the state. M&E Quality Rating : Modest 11. Other Issues a. Safeguards:

10 The project was Category B, with an Environmental Management Plan, which remained the same under additional financing. All subprojects were systematically screened for consistency with the environmental provisions set out in the Operational Manual. Under the Additional Financing a system of environmental licenses was introduced for all projects with potential negative impacts. The only safeguard triggered was Environmental Assessment (OP 4.04, BP 4.01, and GP 4.01). During implementation, the project had no adverse environmental impacts although, in the longer term, the cumulative impact of the water resource subprojects calls for study (ICR, p. 12). b. Fiduciary Compliance: Although audits were delivered on time and were unqualified, the final rating of the project's financial management was moderately satisfactory, mainly owing to complications caused by the implementing agency's use of a parallel information technology system (ICR, paragraph 2.4.4). c. Unintended Impacts (positive or negative): Unintended positive impacts included: enrollment of productive subprojects (although it is not clear what proportion) in Fair Trade schemes that promoted sale of small-farm products in national and international markets; use of subproject funding to contribute to pioneering anti-desertification initiatives; and jobs created for rural youth through establishment of local orchestras (ICR, paragraph 3.5.8). d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement /Comments Outcome: Moderately Moderately Risk to Development Moderate Significant The sustainability of productive Outcome: subprojects may be compromised by the prevalence of drought; and there is no convincing evidence that Community Associations and Municipal Councils were sufficiently strengthened to ensure enduring poverty reduction. Bank Performance : Moderately Unsatisfactory Quality at entry was compromised by the significant weaknesses in project design and supervision appears not to have been sufficiently intensive or rigorous. Borrower Performance : Moderately Quality of ICR : Moderately NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR observes that: Allocating a portion of subproject cost for technical assistance will not, by itself, resolve the problem that in poor rural areas in general (and in Northeast Brazil in particular) these services are often scarce and of

11 mediocre quality; an intense effort needs to be made to build up technical assistance over time. "Studies consistently show that technical assistance quality and quantity are among the top priorities for rural development stakeholders" (ICR, p. 22). IEG adds: The appraisal of CDD projects needs to weigh the costs and benefits relative to other programs with similar objectives that are already underway. Intermediate outcomes need to be properly quantified with well-designed indicators, baselines, and targets. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The report is well written and comprehensive, distinguishing clearly between what was achieved under the original project and under additional financing. There is a thorough assessment of project design. A number of specific and informative lessons are proposed. However, given the project objective of reducing high levels of rural poverty in the state, there is limited evidence presented on poverty outcome (perhaps reflecting the weak M&E). The economic analysis should have included data on subproject unit costs relative to other programs and the efficiency analysis should have been based on a larger number of subprojects, randomly selected so as not to exclude unsuccessful subprojects. a.quality of ICR Rating :