5IE475 Program Evaluation and Cost-Benefit Analysis

Transcription

1 5IE475 Program Evaluation and Cost-Benefit Analysis LECTURE 12 Instrumental Variable Approach (contd) Qualitative program evaluation Klára Kalíšková

2 EXAMPLES OF INSTRUMENTAL VARIABLES STUDIES (CONTD) 2

3 Example 3: Public health insurance Motivation Currie, Gruber (1996) Health Insurance Eligibility, Utilization of Medical Care, and Child Health. The Quarterly Journal of Economics, 111(2), Effect of public health insurance eligibility (X) on utilization of health care by children (Y) Taking advantage of policy reforms Fraction of children eligible for public insurance (Medicaid) doubled Different extent of reforms in different states Natural experiment approach with IV 3

4 Example 3: Public health insurance Health system in the U.S. Eligibility for public health insurance for children: Medicaid (X) Only low-income families are eligible Coverage of health insurance Some people are covered by public insurance (Medicaid, Medicare), some by private, some by none Increase in eligibility for Medicaid might not increase coverage for everyone (people have private insurance and only switch from private to public) Utilization of health care (Y) How often children visit doctors, hospitals, If they are covered by health insurance, visits are mostly for free, and thus likely to be more frequent 4

5 Example 3: Public health insurance Do we need an IV? Can we estimate the impact by a simple comparison of health care usage of those who are Medicaid eligible/ineligible? Probably not, Medicaid eligible are different in many ways, likely also in their health care usage There are unobserved factors (W) that predict both the eligibility for Medicaid (X) and the health care usage (Y): E.g. socioeconomic background -> it affects the family income (and thus the child s Medicaid eligibility) and also the health care usage (children with a better parental background have different approach to health care) We need an IV! 5

6 Example 3: Public health insurance Instrumental variable approach They use IV that focuses on exogenous variation in Medicaid eligibility laws They create a simulated instrument variable Select a national random sample of 300 children of a given age and calculate the fraction of children in this sample who would be eligible for Medicaid in a given state and year This simulated IV (fraction of children eligible) captures the generosity of Medicaid policy in a given state and year 6

7 Example 3: Public health insurance Is this a good IV? Is it relevant? = Correlated with X (eligibility) Yes, it should explain well individual eligibility, as individual eligibility is given by state laws, and the IV capture differences in state laws Is it exogenous? = Not correlated with Y (health care usage) Yes, share of eligible children (generosity of the Medicaid) has no direct effect on individual health care utilization (the only effect is through individual Medicaid eligibility, and that is OK) 7

8 Example 3: Public health insurance Estimation: impact on utilization of care Outcome: utilization of health care (Utilization) Treatment: eligibility for public health insurance (Eligible) Instrument: share of children eligible -> indicator of generosity of the Medicaid laws in a given state (Fraction_eligible) Estimation (2SLS): Eligible = α 0 + α 1 Fraction_eligible + X β + u Utilization = γ 0 + γ 1 Eligible + X δ + e 8

9 Example 3: Public health insurance Results: impact on utilization of care 9

10 Examples of other IV studies Y (outcome) X (determinant) Z (instrument) paper Earnings, happiness College education Proximity to college Card (1993) Earnings Military service Vietnam era draft lottery Angrist (1990) Crime rate Police force Timing of elections Levitt (1997) Economic performance (GDP) Institutions Type of colonization policy Acemoglu, Johnson & Robinson (2001) 10

11 QUALITATIVE PROGRAM METHODS 11

12 Quantitative vs qualitative methods Quantitative: Numbers Breadth Generalizations Controlled Qualitative: Words Depth Specific Flexible It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted. William Bruce Cameron (1963) Informal Sociology: A Casual Introduction to Sociological Thinking 12

13 When should we employ qualitative/quantitative methods? Quantitative methods: When we are interested in the impact of the policy. When we want to know what the outcomes are. When we are interested in generalizability of the results. How did the policy affected fertility rate? Qualitative methods: When we want to know why the program is/is not working. When we want to know how it is successful what the most important components are. Why did people choose to have more children? How did the program influence family planning? 13

14 Data collection Quantitative methods Surveys, questionnaires Existing databases Qualitative methods Interviews Focus groups Direct observation Document review Participatory assessments 14

15 Interviews Asking program participants about the program and recording their point of view on its benefits, shortcomings, One-to-one interviews are suitable when: The subject is sensitive. People are not willing to discuss the topic in front of others. Illiteracy of subjects 15

16 Interviews: types (1) Structured interview Interview schedule, set of written questions The same exact questions in the same exact order No flexibility in wording Semi-structured interview Using interview guide series of questions or topics Not necessary all questions asked to all participants and not necessarily in the same order Unstructured interview the interview proceeds at the respondent s pace the subject can vary and can be chosen by the interviewee 16

17 Interviews: types (2) Face-to-face interview: Researcher can elicit more in depth responses, fill in information if a participant does not understand a question, Can use visual aids, photographs, Telephone interview: Impersonal Needs short introduction, well-prepared questions Response rate may be lower Advantage: lower costs 17

18 Interviews: implementation Avoid long questions Avoid leading questions Try not to influence the response in a certain way Do not use strongly positive or negative language Do you think this policy has increased your chances to find a job? How do you think this policy has influences your chances of finding a job? Establish time frame for the interview Sequence topics Keep neutral language 18

19 Focus groups Structured small group interviews Participants are usually similar in some way (certain characteristic, experience) Participants are required to discuss particular topic, discussion is quided by a moderator Ideally: 6-12 people and a moderator (note taker) Purpose: To solicit perceptions, views and opinions In-depth analysis of certain topic 19

20 Focus groups: considerations One focus group is one observation (even if it includes 12 people) Need a series of focus groups Size of the group: Smaller groups if the topic is difficult or sensitive Homogeneity and anonymity of the group People are more likely to talk openly in front of people who are similar (like-minded), but who they do not know Leading the discussion too much vs too little Stay on topic, but provide enough room for participants to share their views 20

21 Observations Watching people, programs, events, Involves all 5 senses (not only sight) Providing information about real-life situations Helps to understand behavior, process or situations Disadvantages: Observer s presence can influence situation Potential for bias, misinterpretations, and overlook of important aspects Difficult to analyze 21

22 Qualitative data analysis Search for emerging themes How many times has a certain issue/topic/word been mentioned? How many people mentioned it? Emotional response to certain questions, topics. What details has been most commonly mentioned? 22

23 Summary of qualitative methods They can help us answer the why and how questions They provide details, big picture, and context. Can help us replicate successful programs. Illustrates which aspects most likely contributed to the success of the program. 23

24 SHORT REVIEW OF IMPACT EVALUATION 24

25 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 25

26 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 26

27 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 27

28 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 28

29 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 29

30 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and a before/after data? Difference-in-differences 30

31 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and before/after data? Difference-in-differences 31

32 How to chose method for impact evaluation? Can you influence the design of the policy? If yes, is (any) randomisation feasible? Randomised control trial Is there a strict treshold for eligibility? Regression discontinuity Is there something that predicts participation, but not outcome? Instrumental variable Is there a reasonably similar control group and before/after data? Difference-in-differences 32

33 EXAMPLE: EXECUTIONS AND PERSONAL BANKUPTCY 33

34 Motivation: Share of people in execution in the CZ regions 34

35 Motivation: Share of people in execution in the CZ regions 35

36 Personal bankruptcy Evaluate impacts of a law that introduces personal bankruptcy Personal bankruptcy: If you get into troubles paying off your debts, and it is unlikely you will ever be able to pay them off, you can call personal bankruptcy > if you do your best to pay off as much as you can from you debt and live with minimum living standard in the following five years, the rest of the debt will be written off Eligible only if you can pay off at least 30% of your debt within the five years 36

37 Impact on unemployment rate Evaluate the impact on unemployment rate Idea: people who have too many debts are usually in execution and thus their income from work is taken by the executor to pay off the debt, and they are left with very little -> have little incentives to work If you are in personal bankruptcy, you have more incentives to work and pay off part of your debt, because after the five years, you can have a fresh start Personal bankruptcy should motivate indebted people to work and should thus decrease official unemployment rate 37

38 How to evaluate this policy? (1) Personal bankruptcy was introduced in 2008 in the Czech Republic. Can we compare unemployment rate in the Czech Republic before and after 2008 and use this to evaluate the impact of this new policy? 38

39 How to evaluate this policy? (2) Your team is asked to conduct the evaluation and your colleague suggests to look at unemployment rates of those debtors who declared personal bankruptcy and those who did not. What do you think of this approach? Is this going to tell you what is the impact of personal bankruptcy on unemployment? 39

40 How to evaluate this policy? (3) Imagine you have detailed regional data on the share of debtors eligible for personal bankruptcy and unemployment rates. What method of impact evaluation can you use here? How would you proceed? 40