Wilbert van der Klaauw, Federal Reserve Bank of New York Interactions Conference, September 26, 2015

Transcription

1 Discussion of Partial Identification in Regression Discontinuity Designs with Manipulated Running Variables by Francois Gerard, Miikka Rokkanen, and Christoph Rothe Wilbert van der Klaauw, Federal Reserve Bank of New York Interactions Conference, September 26, 2015 The views presented here are those of the author and do not necessarily reflect those of the Federal Reserve Bank of New York, or the Federal Reserve System

2 On the applicability of RD approach Many applications of RD design over past 20 years not infrequently in cases where RD assumptions appear violated Much work on non-parametric estimation and testing in RD designs Optimal bandwidth choice Quantile treatment effects Additional tests for internal and external validity Test based on pseudo outcomes or test for jumps at nondiscontinuity points Test for discontinuity in density = test of non-manipulation assumption

3 On the applicability of RD approach (cont) Application of approach has been expanded Allowing for some imperfectly controlled manipulation Regression Kink Design discontinuity in slope of running variable treatment rate relationship Identification of treatment effects away from the discontinuity point eg. by exploiting noise in measurement of running variable

4 On the applicability of RD approach (cont) In general RD only identifies a local average treatment effect: LATE effect on compliers at the cutoff, representing those in a neighborhood of the cutoff whose treatment status switches if the eligibility cutoff score (or threshold for participation) switched from just above their score to just below their score Limited external validity away from cutoff Effect generally policy relevant for cases where policy question involves changing the location of the threshold

5 This paper Challenges conventional wisdom that RD approach is not applicable when agents can manipulate running variable Generally leads to non-comparability of those left and right of cutoff Eg., in case of a known test score threshold with option to retake the test, individuals retaking the test may be a selected sample Proposes a partial identification approach to identify treatment effects in RD designs with potentially manipulated running variables Derives sharp bounds on treatment effects Proposes methods to construct confidence intervals for treatment effects that have good coverage properties

6 Main ideas Classify agents by unknown type: manipulators and nonmanipulators Manipulators have preference for being on right of cutoff Manipulators have perfect control of running variable Non-manipulators do not behavior in line with standard RD setting Implies that Left of cutoff only have non-manipulators Jump in density identifies share of manipulators near cutoff Can identify distribution of pre-treatment characteristics of manipulators and non-manipulators by comparing distributions left and right of threshold Bound treatment effects by considering scenarios in which manipulators have highest or lowest values of outcome variable and trim respective observations as in Lee (2008).

7 Main ideas (cont) Can sharpen bounds if willing to assume that manipulators include greater fraction of compliers & always takers Estimation involves standard nonparametric estimators of density, conditional quantile, and (truncated) conditional expectation functions using local polynomial regression In contrast to standard RD design need to estimate functions separately for treated and non-treated left and right of cutoff Sample truncation implemented using polynomial truncation approach Apply bootstrap techniques to compute confidence intervals [Imbens and Manski 2004, Chernozhukov, Lee and Rosen 2013] Illustrate approach with an application evaluating the impact of UI eligibility on the duration to next formal job

8 General comments Approach is intuitive; identification and estimation methods fairly standard Would be useful to conduct Monte Carlo simulation exercise to analyze performance of estimator and confidence bounds Looks like you could relax identification conditions and get tighter bounds in case of one-sided sharp RD where cutoff represents eligibility condition (no treated cases on left) My main comments focus on applicability of approach which types of manipulation allowed and who are non-manipulators? policy relevance and interpretation of partially identified treatment effect

9 Applicability of approach Applicability limited to cases where only have one-sided manipulation Problematic if expect heterogeneity of treatment effect including people with negative perceived net benefits (large stigma) may lead to two-sided manipulation. Not concern for paper s application Case of perfectly controlled or deterministic manipulation appears very restrictive and unrealistic, but your approach appears applicable under broader definition of manipulation Would be good to consider range of manipulation scenarios (and data availability requirements) and discuss under which the approach is applicable or not

10 Perfectly controlled manipulation Before manipulation ƒ(x) After manipulation ƒ(x) M L = 1 M L = 1 M R = 1 M R = 1 M L = 0 M R = 0 M L = 0 M R = 0 C X C X Discontinuity in density measures share of manipulators Lose comparability between those left and right of cutoff Partially identified: LATE on non-manipulators with score close to c

11 Imperfectly controlled manipulation eg. test re-taking Before manipulation After manipulation ƒ(x) ƒ(x) M L = 1 M L = 1 M R = 1 M L = 1 M R = 1 M L = 0 M R = 0 M L = 0 M R = 0 C X C X Discontinuity in density measures share of manipulators (test re-takers) Manipulation again causes non-comparability of those on left and right Partially identified: LATE for mix of non-manipulators and manipulators with (final) score close to c Extends to case where can retake test multiple times

12 LATE - policy relevance Approach identifies LATE for non-manipulators (or mix of nonmanipulators and manipulators with imperfect control) at threshold Increased concerns about external validity (relative to RD without manipulation) Who belongs to subgroup is likely endogenous: nonmanipulators may become manipulators and manipulators with imperfect control may try harder when program benefits increase

13 LATE - policy relevance (cont) In presence of manipulation does estimated LATE still approximate the total local impact of a small change in the cutoff score? No, unless policy change had zero impact on outcomes of manipulators Unlikely to hold: decision to manipulate (e.g. retake test) may lead to substantial change in value of running variable and in outcomes Impact on manipulators likely to depend on whether policy change was pre-announced and known to manipulators, and on scope for immediate and longer-term manipulation In general, to assess total impact on those near the cutoff (not just non-manipulators) would require estimates of policy s impact on behavior and outcomes of manipulators near cutoff

14 Final comment Interesting and thought-provoking paper!

15 Other comments I believe you need left continuity of E[g(W) X=x, M=1] as well in Corollary 4 (as in assumption 2) Recent literature points to gains from simultaneous bandwidth selection (eg. Arai and Ichimura 2013,2014) for estimating difference between two functions at a point. Can similar gains be achieved here? Under what conditions is presence of discontinuity in density itself evidence of non-zero treatment effect?