Regression Discontinuity Design

Transcription

1 Regression Discontinuity Design Aniceto Orbeta, Jr. Philippine Institute for Development Studies Stream 2 Impact Evaluation Methods (Intermediate) Making Impact Evaluation Matter Better Evidence for Effective Policies and Program September , ADB Headquarters

2 Outline Basic Characteristics Validity Test Specification Decisions Estimation methods Classic examples

3 Basic Characteristics There is an continuous indicator / forcing variable that can order observation units in some manner Poverty Index (e.g. PMT) Impact of development projects to communities/households above a poverty incidence threshold (e.g. Pantawid Pamilya (Philippine CCT)) Age Impact on access to public goods of discounts to senior citizens (above 60 years old) Exam scores Impact of remedial school program mandatory for children whose score is less than some cut off level on a test Impact of migration when eligibility is based on qualifying exam scores

4 Basic Characteristics: RDD before and after intervention Before Intervention After Intervention Impact Gertler et al. (2011) Impact Evaluation in Practice. World Bank

5 Validity Tests Shown mostly by scatter plots No jump in outcome before treatment. Without the intervention, there exist a smooth relationship between the outcome and the predictor of treatment assignment (plot of outcomes against running variable) No jumps in relevant covariates at cut off. Only the treatment variable should cause a jump in the outcome variable; there should be no other covariate that exhibits similar jump at the threshold or it will invalidate the design (plot of covariates vs running variable) Eligibility cannot be manipulated McCrary (2008) density test (density plot of running variable around the cutoff)

6 Decision to Make: Choice of functional form RDD requires a break in outcome at the threshold point Panel A Linear - Will require linear model Panel B Non-linear - Will require a non-linear model Panel C Not a RD - There is no break only a non-linear relationship

7 Decision to make: Choice of How wide from the threshold should be included? trade-off between bias and variance The wider BW: more bias but smaller the variance and viceversa Optimum BW i.e. optimality based on MSE Common practice: Provide several estimates at different BW to reveal bias Bandwidth (BW)

8 Data-Determined Optimum Bandwidth Trade-off between bias and variance Imbens and Kalyanaraman (2012) Calonico, Cattaneo, Titiunik (2014) Cross Validation (Ludwig & Miller, 2007)

9 Two Types of RD When treatment is determined completely by predictor / forcing variable (called sharp); if partly (called fuzzy) Outcome Y i Yi (0) if Wi 0, Yi (1) if Wi 1 Where W i is the treatment assignment Let X be the predictor and c threshold value Sharp: W i =1{X i c} ( deterministic assignment ) Fuzzy: P(W i =1 X i =x) from below c P(W i =1 X i =x) from above c rather than having treatment jumping, it s the probability of getting treatment that jumps ( probabilistic assignment )

10 Impact Estimates Sharp: Sharp E[ Y (1) Y (0) X c] Fuzzy: Fuzzy E[ Y (1) X c] E[ Y (0) X c] E[ W (1) X c] E[ W (0) X c] measured from above (1) and from below (0) of threshold c

11 Importance of Graphical Analysis Validity test are shown through graphs Specification are often revealed by a graph If there is no visible jump in the graph, chances are there is significant impact Of course, a graph cannot give us a precise numerical estimate of the impact

12 Estimation Global Polynomial Local Linear Local Randomization

13 Global Polynomial Naïve (assumes constant treatment effect with equal slope ) Y * W ( X c)*... ( X c) * p i sharp i 1 i p i Flexible p Y * W ( X c)*... ( X c) * i sharp i i i p W *( X c)*... W *( X c) * p i 1 i p i

14 Local-Polynomial Estimate separately a weighted regression (using kernel weights) locally (within a bandwidth, h) Left of cutoff [-h,cutoff] Right of cut-off [cutoff,h] Yi ( X i c)* _ _, i Y ( X c)*, Or combined i i i ˆ ˆ_ ˆ ( h ) sharp Y * W ( X c)* W *( X c)* i sharp i i 1 i i 1 i

15 Local Randomization Idea: Near the cutoff treat as if randomly assigned Find a window [-h<x<h] such that for all Xi, Wi is independent of outcome - Y(0), Y(1) Employ RCT methods in the window, i.e. test of difference in means below and above the cutoff

16 Classic Examples Impact of social assistance on labor market outcomes (Lemeiux and Milligan, 2008) (Sharp RD) Impact of Class Size on Achievement (Angrist and Lavy, 1999) (Sharp and Fuzzy RD)

17 Example 1 Impact of social assistance on labor market outcomes Lemeiux, T. and K. Milligan (2008) Incentive effects of social assistance: A regression discontinuity approach Journal of Econometrics (also NBER Working Paper 10541)

18 Evaluation Issue: What is the incentive effect of social assistance? What impact of social assistance on labor market behavior? Evaluation Model: Output Social assistance rule Intermediate outcomes Amount of social assistance Final outcomes Labor market outcome

19 Background - Before 1989, childless social assistance recipients in Quebec under age 30 received much lower benefits than recipient over age 30 - Used this policy rule to estimate the effects of social assistance on labor market outcomes

20 Social Assistance benefits for under 30 and over 30, Benefits are indeed very different before 1989 between those under 30 and 30 and above

21 Data - Canadian census a detailed questionnaire (long form) is assigned to approximately 20% of households, consisting of questions on labor market characteristics and participation, education, income, and the demographics of respondents - From the 20% sample, obtained samples of men around 3,000 high school dropouts for each age group, keep only men without children for each age group around the discontinuity at age 30 - Employment data employment status during the reference period, hours worked - Labor force survey provide too small number of sample per age group; used to provide data on the labor market context

22 3-year moving average of employment rate by agegroup (25-29; 30-34), Observations -Cyclicality of employment rates => need for a control group to separate business cycle effects from policy effects of interest -The pattern of employment for those aged and aged are similar => labor market conditions are similar to the two groups -Quebec has lower employment rate compared to rest of Canada

23 Empirical Model Y 1 TREAT ( a) ia o ia ia Where Y ia =outcome for individual i of age a δ(a)=effect of age on outcome variable TREAT=treatment dummy 0 if a<30 and 1 if age>=30 β 1 is the parameter of interest

24 Key identifying assumption δ(.) is a smooth (continuous) function We have a sharp RD design since the treatment variable is a deterministic function of the regression variable (age) The assumption that δ(.) is a continuous function means that differential benefits are the only discontinuity in outcomes around age 30

25 Threats to the validity of the assumption Employment rate by age is a well-known profile Violations can happen when Some people can find ways to cheat on their age by, for example, falsifying their birth certificates difficult to do because can be easily verified Differential benefits are only for individuals without dependent children to the extent that fertility and living arrangements decisions (live with your children or not) are endogenous, this generates a problem of non-random selection

26 Outcomes Employment rate during reference week Employment rate based on the fraction of weeks worked in the previous year [did not define how many weeks working constitutes employment]

27 Estimation Estimated a variety of the specification for the regression function δ(a) Linear Quadratic Cubic Linear spline Quadratic spline Used age-specific cell averages in the estimation Use weighted OLS using the inverse of the sampling variance as weights which is similar to having the number of observations per cell as weights

28 Graphical Evidence: Employment at census week, Quebec, 1986 A very strong evidence that employment drops abruptly once the individual is eligible for higher social assistance Employment also tends to trend down faster (steeper) as a function of age especially after age 30

29 Graphical Evidence Employment rate in previous year, Quebec, 1986 Similar patterns observable using employment rate based on weeks worked in previous year

30 Estimation Results RD estimate of impact of social assistance on labor market outcome, Quebec, 1986 Employment impacts are more precisely estimated (lower standard errors) by the first four models compared to quadratic spline The impact is even stronger using employment rate based on status during reference week compared to that based on weeks worked during the past year; more precisely estimated Similar estimates except for quadratic spline the estimate is from to Goodness of fit test shows that even simpler models (linear and linear spline) fit the data very well fitted not significantly different from actual

31 Example 2 Impact of Class Size on Achievement Angrist and Lavy (1999) Using Maimonides Rule to Estimate the Effect of Class Size on Scholastic Achievement, Quarterly Journal of Economics, 144(2),

32 Evaluation Issue: Impact of school output on school outcomes Impact of class size on test scores Logic Model: Output Intermediate outcome Maimonides rule Class size Test score Final outcome

33 Background Class size in Israeli schools is capped at 40. Student in a grade with up to 40 students can expect to be in classes as large as 40, but grade with 41 students are split into two classes, grade with 81 students are split into three classes, and so on Called Maimonides Rule the rule was proposed by the medieval Talmudic scholar Maimonides

34 Implications of Maimonides Rule on class size Maimonides Rule implies that the predicted class size (in a given grade) assigned to a class c in schools s (m sc ) is m sc es ( es 1) int 1 40 e s =enrollment in grade Int(a)=interger part of (a) Enrollment 1-40 in a single class; enrollment split into 2 classes; enrollment into three classes, and so on m sc is an increasing function of e s, making it an important control

35 Maimonides Rule and actual class size, 5 th Grade Actual data on class size reveal, Maimonides rule was not strictly followed (or there are other factors determining class size) but it clearly is largely determined by the rule While class sizes is not in multiples of 40, class size increases with enrollment size; and drops sharply at integer multiples of 40

36 Class size and (reading) test scores Test scores are generally higher in school with larger enrollments (positive relationship) Showing, in part, a mirror image upand-down pattern Apparent positive correlation is partly attributable to the fact that larger schools are more likely to be located in relatively prosperous cities, while poorer schools are more likely to be located in relatively poor town outside of major urban centers

37 Model Y X n isc o s sc c s isc Y isc =student i test score in school s and class c, X=vector of school characteristics n sc =the size of this class c in school s Use class-level estimating equations

38 Fuzzy RD Implementation Fuzzy version: m sc is an instrument to n sc n X m sc s 0 sc 1 sc

39 OLS Estimates, 1991 With no controls show strong positive correlation between class size and achievement test scores (0.221 for reading; for math) When the percentage disadvantage children was added as control, the estimate falls to but insignificant; coefficient of math score remains to be positive Neither does the addition of enrollment significantly affect estimates

40 IV Estimate, 5 th grade With IV estimates, large class size are associated with lower test scores Impact of class size on reading without controls is (0.04); with linear and quadratic controls for enrollment size ranges from (0.08) and (0.07) Impact on math scores is virtually zero without enrollment control; with linear and quadratic controls, the impact on math scores is (0.09) to (0.11) Even bigger impact with RD sample Note: Full sample includes everyone; discontinuity sample include only those in the vicinity of the thresholds class size +/- 5.

41 Strength of RDD Strong internal validity as we get near the threshold, the treatment and comparison groups are as if chosen by randomized assignment to treatment; strongest among the quasi-experimental methods Less ethical issues - no need to exclude eligible units from receiving treatment

42 Issues with RDD External validity impact is valid only around the threshold and not for the whole population; should not be used if policy issue is the impact on the whole population Statistical power - Requires larger sample size than the others (e.g. approx. 3 times than RCT*); there may not be enough sample around the threshold and expanding the band around the threshold weakens internal validity Functional form dependence may require additional functional form assumption to obtain credible impact as one gets farther from the threshold *Lee, H. & Munk, T. (2008), 'Using Regression Discontinuity Design in Program Evaluation', Survey Research Methods.; Schochet, P. Z. (2009), 'Statistical Power for Regression Discontinuity Designs in Education Evaluations', Journal of Educational and Behavioral Statistics 34(2),

43 Basic References Lee, D. S. & Lemieux, T. (2010), 'Regression Discontinuity Designs in Economics', Journal of Economic Literature 48(2), Imbens, G. W. & Lemieux, T. (2008), 'Regression discontinuity designs: A guide to practice', Journal of Econometrics 142(2),

44 Thank You

45 Reproducing Angrist and Lavy (1999)

46 Estimate to reproduce

47 Data Available at /data/anglavy99

48 Preliminaries use AL1999_final5, clear lab var c_size "Enrollment lab var tipu "Percent disadvantage lab var classize "Class size lab var avgverb "Score, Reading lab var avgmath "Score, Math ** Variables generation replace avgverb= avgverb-100 if avgverb>100 replace avgmath= avgmath-100 if avgmath>100 g func1= c_size/(int((c_size-1)/40)+1) g func2= cohsize/(int(cohsize/40)+1) replace avgverb=. if verbsize==0 replace passverb=. if verbsize==0 replace avgmath=. if mathsize==0 replace passmath=. if mathsize==0 keep if 1<classize & classize<45 & c_size>5 keep if avgverb~=. * RD sample indicator g byte disc= (c_size>=36 & c_size<=45) (c_size>=76 & c_size<=85) (c_size>=116 & c_size<=125) g c_size2= (c_size^2)/100 * GENERATE TREND g trend= c_size if c_size>=0 & c_size<=40 replace trend= 20+(c_size/2) if c_size>=41 & c_size<=80 replace trend= (100/3)+(c_size/3) if c_size>=81 & c_size<=120 replace trend= (130/3)+(c_size/4) if c_size>=121 & c_size<=160

49 Descriptive stats. summ avgverb avgmath classize func1 tipu c_size schlcode, sep(0) Variable Obs Mean Std. Dev. Min Max avgverb avgmath classize func tipuach c_size schlcode

50 Full sample (col 1). ivregress 2sls avgverb (classize=func1) tipu, vce(cl schlcode) // col 1 Instrumental variables (2SLS) regression Number of obs = 2019 Wald chi2(2) = Prob > chi2 = R-squared = Root MSE = (Std. Err. adjusted for 1002 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize tipuach _cons Instrumented: classize Instruments: tipuach func1 Note: Angrist and Lavy (1999) used Moulton (1986) correction, here we used cluster option in ivregress Moulton (1986) Random group effects and the precision of regression estimates, J. of Econometrics, 32,

51 Full sample (col 2). ivregress 2sls avgverb (classize=func1) tipu c_size, vce(cl schlcode) // col 2 Instrumental variables (2SLS) regression Number of obs = 2019 Wald chi2(3) = Prob > chi2 = R-squared = Root MSE = (Std. Err. adjusted for 1002 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize tipuach c_size _cons Instrumented: classize Instruments: tipuach c_size func1

52 Full sample (col 3). ivregress 2sls avgverb (classize=func1) tipu c_size c_size2, vce(cl schlcode) // col 3 Instrumental variables (2SLS) regression Number of obs = 2019 Wald chi2(4) = Prob > chi2 = R-squared = Root MSE = (Std. Err. adjusted for 1002 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize tipuach c_size c_size _cons Instrumented: classize Instruments: tipuach c_size c_size2 func1

53 Full sample (col 4). ivregress 2sls avgverb (classize=func1) trend, vce(cl schlcode) // col 4 Instrumental variables (2SLS) regression Number of obs = 1961 Wald chi2(2) = Prob > chi2 = R-squared =. Root MSE = (Std. Err. adjusted for 990 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize trend _cons Instrumented: classize Instruments: trend func1

54 Discontinuity Sample (+/- 5) (col 5). ivregress 2sls avgverb (classize=func1) tipu if disc==1, vce(cl schlcode) // col 5 Instrumental variables (2SLS) regression Number of obs = 471 Wald chi2(2) = Prob > chi2 = R-squared = Root MSE = (Std. Err. adjusted for 224 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize tipuach _cons Instrumented: classize Instruments: tipuach func1

55 Discontinuity Sample (+/- 5) (col 6). ivregress 2sls avgverb (classize=func1) tipu c_size if disc==1, vce(cl schlcode) // col 6 Instrumental variables (2SLS) regression Number of obs = 471 Wald chi2(3) = Prob > chi2 = R-squared = Root MSE = (Std. Err. adjusted for 224 clusters in schlcode) Robust avgverb Coef. Std. Err. z P> z [95% Conf. Interval] classize tipuach c_size _cons Instrumented: classize Instruments: tipuach c_size func1