Millenium Villages Project

Transcription

1 Millenium Villages Project 1

2 Few Evalua8ons available When Does Rigorous Impact Evalua8on Make a Difference: The Case of Millenium Villages By Michael Clemens and Gabriel Demombynes 2

3 MVP Project Started with Millenium Summit (2000, NY) Largest gathering of heads of state in modern history 147 Presidents, prime ministers, monarchs Pledged to meet 8 Millenium Development Goals 3

4 To eradicate extreme poverty and hunger To achieve universal primary educa8on To promote gender equality and empower women To reduce child mortality To improve maternal health To combat HIV/AIDS, malaria, and other diseases To ensure environmental sustainability To develop a global partnership for development 4

5 Villages Villages project with Jeff Sachs in charge spun- off in Large interven8ons in clusters of poor villages in rural Africa Nearly 400,000 people in 79 villages. 12 groups across 10 African countries (Ethiopia,Ghana, Kenya, Malawi, Mali, Nigeria, Rwanda, Senegal, Tanzania, and Uganda), Located in different agro- ecological zones that reflect the range of farming, water, and disease challenges facing the con8nent. 5

6 Process Started in Sauri Kenya (2004) Broad package of interven8ons in each village, including distribu8on of fer8lizer insec8cide- treated bednets, school construc8on, HIV tes8ng, microfinance, electric lines, road construc8on, piped water and irriga8on lines, and others. The precise mix differs in each village cluster. Villages were selected based on community buy- in. 6

7 Goal break the villages out of poverty traps, and make the investments in human capital and infrastructure required to achieve self sustaining economic growth. Over a 5- year period, community commilees and local governments build capacity to con8nue these ini8a8ves and develop a solid founda8on for sustainable growth MVP

8 Cost $150 per villager per year over 5 years i.e. 400,000*150*5 = 300 Million US dollars Big i.e. roughly the size of per capita income in each village 8

9 Assessments MVP called itself a solu8on to extreme poverty (MVP 2007a) Recommended that its interven8on be massively scaled up across Africa. 9

10 Jeffrey Sachs (2005: 236) called for foreign aid to Kenya to increase fiqeen- fold in order to provide $1.5 billion per year for Millennium Village interven8ons in that country alone. MVP(2008b) applauded plans to expand the MVP in several countries. Concluded, The MVP has therefore created a powerful pressure to expand as a result of its notable successes. John McArthur CEO 10

11 Policy Should these be scaled up? Should they be conducted elsewhere? Is this a long- term solu8on to poverty? Is this a development strategy? 11

12 Mid- term MVP evalua8on Report June 2010 first public results released for 5 sites gained from surveys of these Sauri, Kenya; Ruhiira, Uganda; Pampaida, Nigeria; Bonsaaso, Ghana; and Mwandama, Malawi Project impact assessed by before- and- aqer comparisons (Sanchez et al. 2007). I.e. MVP evalua8on report compares the treated villages before the interven8on with how they are doing aqer it. Tyler Cowen 12

13 Problems Some, all, or more benefits might have occurred in the absence of the MVP How would we design a study to tell us whether the MVP works? I.e. Answer the following ques8ons: 13

14 Op8mal Project Design 1. What should the project villages be compared with to determine the impact of the project? E.g., Country average, typical village? 2. How to choose which villages will receive the projects? 3. Where, when and what outcomes should be measured? 14

15 Actual Project Design 1. Villages selected to be part of the project are compared before and aqer the project was undertaken. Half way through the project, addi8onal comparator villages were inves8gated 2. Par8cipatory villages were chosen based on desire to sample wide range of villages, and on buy- in. 3. Outcomes for all MD Goals measured at 8me 0, Mid way and end of project in all project villages. 15

16 Alterna8ve (not perfect) comparison Compare change in villages with changes in same country overall, rural regions of same country overall, same region of the country overall Able to conduct this analysis for three of the ini8al 14 village clusters. Two criteria: 1) covered in the MVP (2010c) mid- term evalua8on of June ) located in a country for which publicly- available Demographic and Health Survey (DHS) data can be used to establish broad trends between two recent 8me points in most of the development indicators reported in the This yields three village clusters: Sauri in Kenya, Bonsaaso in Ghana, and Pampaida in Nigeria. 16

17 Results from alterna8ve Figures and Tables Final report will compare MV with selected comparison villages 17

18 18

19 19

20 20

21 21

22 22

23 23

24 24

25 25

26 26

27 Comparison protocols Five years aqer first Millennium Village released details of plans for evalua8on of the project s effects. The study protocol (MVP 2009) calls for assessing child mortality as primary indicator for program and also details a number of secondary indicators. Core of the evalua8on is data collec8on in one comparison village cluster for each of 10 MV sites (the protocol excludes the Sauri, Kenya and Koraro, Ethiopia sites from the evalua8on plan) 27

28 Comparison Villages Choice of candidate villages for comparison was at random, and informed by both scien8fic and opera8onal issues A detailed matching process is undertaken to adequately characterize village alributes Matching process involves collec8ng data on village- This inventory is collected both for the interven8on and comparison villages and updated in parallel with each round of surveys. Comparison villages had to be logis8cally accessible yet far enough away from the interven8on site that spillover from the MVP would be unlikely. 28

29 Why are these s8ll not perfect Problem 1. Selec8on: evalua8ons? MV sites not selected at random. Issues of feasibility, poli8cal buy- in, community ownership and ethics also featured prominently in village selec8on for par8cipa8on in large scale development programs such as MVP. 29

30 Problem 2: Subjec8vity Matching criteria comprise a handful of village traits such as size, distance to a paved road, and agro- ecological zone. Protocol does not indicate how closely villages will need to match, what weights the different traits will have, or how it can be known that comparison villages were not chosen using addi8onal criteria not that do not appear on the list. Other factors such as the level of community organiza8on and willingness and ability of community leadership to work well with outside agencies are omiled. Even if the comparison villages were perfectly matched on the listed criteria, their usefulness as a rigorous comparison group would remain unclear 30

31 Randomiza8on up to three comparison village candidates are assessed for poten8al inclusion The one is randomly chosen Does this help in overcoming subjec8vity or omiled category biases? No 31

32 Problem 3 No baseline data in comparison villages Didn t assess the indicators (the before ) in the comparator villages Protocol acknowledges this but states [f]or many indicators, in the absence of new interven8ons it would be unlikely that substan8ve year on year changes would be witnessed, such as malaria prevalence, ins8tu8onal delivery rates or child nutri8on. WHAT!!!! 32

33 Problem 4. Small sample size. One comparison village for each of the 10 village clusters being evaluated 300 households in each survey clustered but each subject to the same common treatment component Cannot thus compare efficacy at any one site. At best finds average effects across sites. 33

34 Would have to be big to find effects: Power calcula8ons based on detec8ng changes in child mortality, note that the planned sample size will only be able to reliably detect a minimum drop of 40 percent at interven8on sites. In five years requires the child mortality rate to fall at more than double the rate called for in the Millennium Development Goals. Smaller changes could not reliably be dis8nguished from zero. Similarly large drops would be needed in other indicators measured as propor8on (such as school alendance and access to sanita8on 34

35 How to do it right 1. MATCHING Choose pairs of village clusters using a systema8c matching procedure to resemble each other on a number of observable traits popula8on, average educa8on, and a wealth index. Want to be sure that we are comparing like with like Perhaps also want to select on par8cular characteris8cs (e.g. most poor, or most needed etc.) 35

36 2. RANDOMIZATION Select only one cluster per pair of clusters at random to receive the package interven8on. This eliminates the possibility of subjec8vity in the choice of untreated comparison villages. Eliminates selec8on based on percep8on of where it will work, or on contacts Eliminates biases on the part of researchers in finding what they want 36

37 3. MEASURING Survey all the treated and untreated clusters, at mul8ple 8me points such as 0 (baseline), 5, 10, and 15 years aqer the interven8on began Survey on all of the outcomes in the MDG Survey on income, assets, employment as well Pre- state and publically post dimensions which will be surveyed 37

38 This design would allow measurement of short- and medium- run indicators, make the choice of treated villages clear and objec8ve, make the choice of comparison villages clear and objec8ve, allow for complete baseline data on both groups. allow rigorous measurement of the overall impact of the en8re package, across all treatment sites, rela8ve to no interven8on. Stop data- mining 38

39 Costs of Evalua8on Iden8fying clusters (close to zero) Randomizing within (close to zero) Baseline survey in comparison villages (since these are already being compared) small Pre- sta8ng measures (close to zero) 39

40 Why not do this? Ethics? Economics? Poli8cs? Ego? 40