Randomized Evaluation Start to finish

Transcription

1 TRANSLATING RESEARCH INTO ACTION Randomized Evaluation Start to finish Bruno Crépon Abdul Latif Jameel Poverty Action Lab povertyactionlab.org

2 Course Overview 1. Why evaluate? What is evaluation? 2. Outcomes, indicators and measuring impact 3. Impact evaluation why randomize 4. How to randomize 5. Sampling and sample size 6. Implementing an evaluation 7. Analysis and inference 8. Randomized Evaluation: Start to finish

3 TRANSLATING RESEARCH INTO ACTION Micro credit in rural Morocco Bruno Crépon Florencia Devoto Esther Duflo William Parienté Abdul Latif Jameel Poverty Action Lab povertyactionlab.org

4 The setting: Al Amana Al Amana is one of the largest Microfinance institution in Morocco Active loans Cumulated served loans Loans $ 232,440,000 Large number of branches 464

5 The setting: Al Amana s expansion to rural Morocco Mostly operated in urban areas up to 2006 New policy started : expansion in rural Morocco An area where almost no financial services existed 10% have access to credit 6% through informal loans

6 The need Many reasons for which people would like to borrow Start / expand new business Absorb schocks Consumption durable/non durable Reduced borrowing possibilities People rely on informal loans or do not borrow

7 Intervention Al Amana opens a new branch in remote rural areas Usually in a small town Well identified nearby villages Offer Al Amana microcredit products in the town and villages Loan officers visit villages, organize focus groups Al Amana microcredit product Need an investment project Not consumption loans Need to have two activities Switch from group lending to individual lending during the experiment

8 Theory of change No access to financial services Households decisions about their activity are made in a constrained environment Supply of microcredit changes this environment by relaxing the constraint Many potential effects

9 Theory of change Existing investment project not realized because of financial constraints Take the microcredit Do the investment Reorganize household s work effort Change in production, resources Repay the loan Change in savings and consumption Can be different in the short run and the long run

10 Theory of change : side effects What about the quality of the initial project Problems in loan repayments Negative effect on consumption or savings What about education decision Potential long term negative effect if reduced school attendance Woman empowerment Business started by women who get therefore their own money and autonomy

11 Theory of change : What type of investment project Investment project can be an entrepreneurship project Common view is that the poors are all potential talented entrepreneurs They have the desire and the skills to run their entrepreneurship projects It can also be an investment to reorganize the work effort Painful work effort : mainly daily laborers There are ways to reorganize the work effort but they are costly and require investments Purpose is not to make business it is to have a better life now

12 Theory of change A substitute to insurance : no insurance products available Shocks : economic lives in rural villages subject to large shocks Absorption of these shocks frequently implies to take on household s assets Either monetary or physical assets Microcredit is a way to accommodate these shocks Taking a microcredit in case of a shock allows to keep household s asset

13 Theory of change : intertemporal constraints removed Current decisions can be taken with in mind the knowledge that financial constraints may occur in the future Even if people do not take a credit now the environment in which they take their decision has changed Potential effect also on non takers

14 Why Evaluate? Strong debates surrounding the impact of microfinance For some the silver bullet to fight poverty For other a path to overindebtedness Need evidence based study

15 Why Evaluate? Almost no knowledge about microcredit effect Strong selection effect Individuals self select into microfinance programs Microfinance institutions select alsoindividuals Difficult to find suitable empirical strategies to deal with selection biases Some attempts using non RCT methods but not convincing Large value added by RCTs

16 Why Evaluate? Already existing RCT: India (Banerjee & al, 2010), Philippines (Karlan and Zinman,2011) In urban areas Both studies take place in areas where there exists several alternative borrowing possibilities Cheaper credit made available

17 Why Evaluate? No knowledge about how people adapt their decisions and working life when the financial constraint is relaxed The setting here is unique Compare A world without financial services With a word in which these services are made available

18 Design In 2006 Al Aamana decided to expand progressively in remote rural areas Progressive move Process is to have a new branch located in a small town Serving the town and well identified nearby villages

19 Design Al Amana Progression in waves Schedule was to have a first wave in march 2006 with 10 new branches One additional wave in July 2006 with 30 branches One last wave in October 2006 with 40 branches

20 Design Design is the following For each new branch select a paire of villages within the set of villages served by the new branch Randomly assign one village of the paire to be a treatment village : microcredit is offered The second village of the paire is the control village The offer of microcredit services is postponed for two years

21 Design How to select the villages They have to be close to the border of the area served by the new branch Get a map of the area with roads and villages and identify potential villages They have to be quite similar Do a survey to collect all suitable information: size, activity, # farmers, wealth, Match the potential villages

22 Selection of villages

23 Encouragement design All the households will not become micro credit clients Some will but some other not We followed randomly selected people in treatment and control villages

24 Encouragement design T NT Treatment Village NT Control Village This is for one paire We have many paires Clustered experiment: we need lots of clusters Follow everybody randomly selected in T and C villages

25 Design Power calculation Two questions: How many people do we need to follow in each village How many pairs of villages Two important unknown parameters Correlation intra village: villages from a same pair share a lot in common Micro credit take up: real unknown parameter Use a guess value based on what the microfinance institution was expecting : 70%

26 Design Power calculation We are doing a test with apha=0.05 We want to detect a standardized effect of 20% We want a power of 80% Rho was chosen low 0.05 Take up assumed to be 70% Choose to survey 25 households in average at the village level

27 Design Power calculation Run optimal design Get the number of paires of villages 80 paires 162 villages An order of magnitude to keep in mind Risk : No real knowledge about the take up Power strongly sensitive to take up

28

29 entify key palyers op management at Al Amana Fouad Abdelmouni head of Al Amana strongly supportive Other people working in Rabat. We mainly had to work with them Al Amana a large institution with already bureaucratic procedures Not a 100% responsive environment but however things went well

30 entify key players : field staff ey they understand the experiment Need to go very often in the field to monitor, listen, understand Check they understand what they have to do Getting the maps was not easy: they didn t know the area They just started a new activity with 100 s of things to do Experiment was just an additional thing, a bit weird o strong incentives to go to the treatment villages Remote villages Take sometime one day to go Obtain from Al Amana they get reimbursed for travel expenses and

31 entify field staff : donor he study was financed by AFD he agency in charge of development rograms in France hey have a large field experience mportant to have them involved all along the rocess

32 easurement wo large surveys conducted baseline survey and an endline survey 2 ears after he survey lasted more almost 2 hours ased on existing household questionnaire sed by many institutions arge number of information

33 easurement: Intermediate outcomes ending : we want to know whether offering icrocredit indeed made a change in the amount ousehold borrowed e want to know if the amount borrowed from arious sources Informal, formal, formal IMF e want to know also the repayment burden

34 easurement: Final outcomes ctivity : very detailed information, know the roduction of cherries, figues, olives, carrots same or livestock, same for business now if activities are run at a very detailed level now also the amount self consumed, the amount old, the amount stored now by activity all expenses at a very detailed level Wages, input now also the investment now productive assets owned by the household

35 easurement: Final outcomes now each household member labor effort inside nd outside the household now if young people attend school now the consumption of very detailed consumption tems et information about women autonomy

36 easurement: Implementation irst RCT we did in Morocco ifficult to implement surveys dministrative procedure to access to villages eed to get the authorization from local authorities sk a survey firm to do the job or the RCT we are now conducting in Morocco we refer to organize our own enumerator teams

37 anning l Amana progress in rural areas chedule and reality nitially planned to have a three waves in arch, July, October 2006 n the end four waves in March, October 2006, ebruary and July months delay: not bad in fact for such an rganization!

38 ming l Amana send us the list of new branches ew branches are created and a loan officer come here raw a rough map of the area, with villages and oads. Town is served but no villages dentify a list of potential villages end Team Maroc to survey the villages hoose the paire ell Al Amana to serve all villages but the paire

39 ming end Team Maroc to do the surveys in the paires of illages raw the treatment and control within paire ell Al Amana which is the control l Amana goes intensively in the treatment village to erve microcredit products

40 sult : loans lmost no credit available in control group ffering microcredit lead to a substantial increase in oans l Amana clients : +16% oans (from the survey): +13% ood but far from what was expected : 70% ower at risk

41 sults loans arge increase in borrowed amounts ompute the difference between treatment and ontrol villages nown as ITT estimate : 777 Dhs ean that the additional amount for clients is 777/0.16=4662 Dhs nly look at ITT difference between mean outcome in treatment and control villages eep in mind that only a small fraction get additional

42 sult : new activities icro finance marginally helps the start of ew businesses, very disaggregated level umber of activity in agricultural sector 2.3 control: +0,06 ns in treatment umber of different animal raised 2.6 control : +0.11** in treatment

43 bstantial increase in acticvity ll self employment activity Production control : treatment +3574** Expenditures control : treatment +2402** Profit 4930 control : treatment Self consumption 5626 control : treatment 565* re these numbers large? ecall this is ITT. Effect is obtained by dividing y take up. Here for production : 3574/0.16=22337(=90%of control mean)

44 bstantial increase in acticvity griculture Production 8120 control : treatment ** Expenditures 5296 control : treatment +460 ns Profit 1160 control : treatment +930 ** Self consumption 380 control : treatment 280 ns

45 bstantial increase in acticvity ivestock Production 6120 control : treatment + 650* Expenditures 6230 control : treatment +378 ns Profit 120 control : treatment +337 ns Self cons 1966 control : treatment +181 ns ncrease in the value of the livestock After 1 year : all animals cattle ** After 2 years : all animals * cattle + 920*

46 bstitution among income urces tal income control : treatment 200ns riculture 1163 control : treatment 931** estock 120 control : treatment 337ns siness 1917 control : treatment 322ns ages control : treatment 1071 ** mittances 820 control : treatment 122 ns hers 6746 control : treatment 900 ns

47 bstitution among income urces riculture 24% control : treatment +2%** vestock 11% control : treatment 0%ns siness 8% control : treatment 0%ns ages 47% control : treatment 2%** ain effect is to do a substitution between main income urces ouseholds are mainly daily laborers ey shift their activity from daily labor to self ployment

48 rtial substitution in hours of ork hours of work per member per day tal 4.63 control : treatment 0.05 ns ousehold activity 1.46 control : treatment utside 1.13 control : treatment 0.09** ores 2.05 control : treatment 0.00 rpose is to reallocate working hours partially to self ployment activity so a reduction ns in total hours

49 nsumption Do not see large effect on consumption A small ns reduction of total consumption Located in some specific items (social events)

50 nclusion Not a huge effect of microcredit Far from ideas that take up will be very high and households will all become entrepreneur Making business is not the story Household use microcredit to do small improvements Just marginal reallocation of work from probably painful daily labor to household activity However no improvement in income. Why? Bad expectation? Painful daily labor?