7th European Commission Evaluation Conference The Result Orientation: Cohesion Policy at Work Methodologies to assess the overall effectiveness of EU cohesion policy: a critical appraisal and (Sapienza, Univ. of Rome) (Univ. of Westminster) Sofia, Bulgaria 16 & 17 June 2016
Structural and Cohesion Funds (SF) is the EU s flagship development programme to narrow the gap between the rich and poor regions of Europe and to contribute to European economic growth. Given the increasing share of the EU budget devoted to Regional Policy, a large and growing body of literature has investigated the policy s contribution to economic growth and convergence. However, after more than 30 years of policy intervention, so far no consensus has been reached: In a paper reviewing the existing econometric evaluation analyses, Mohl and Hagen (2010, p. 1) conclude: the empirical evidence has provided mixed, if not contradictory, results. Page 2
One major drawback of this literature is that the results strongly depend on model specification, econometric techniques adopted and the dataset used (period under study, country considered, level of payments etc.). (Dall Erba and Fang, 2015) In this presentation we briefly discuss the problems related to the use of different models and methodologies for the evaluation of the impact of European Structural Funds on regional economic growth We also present some new results based on the work done for the European Commission, Directorate General for Regional and Urban Policy, Work Package 14 on Ex post evaluation of the ERDF and CF programmes. Obviously, the opinions expressed herein are only ours. Page 3
A first issue in the evaluation of SF is the identification of the impact of Regional Policy from the confounding effects induced by other factors. In the econometric studies regarding SF, growth is usually modelled according to the logic of the neoclassical model. The dependent variable is usually the growth rate of per capita GDP. This is a function of a number of factors including initial GDP level, a variable representing Cohesion Policy (the actual level of transfers or a dummy variable) and a limited number of other factors. The choice and number of explanatory variables for regression differs widely between studies Page 4
However, the number of potential explanatory variable in the equation could be infinite. The problem is not only statistical: would the omitted variables, not included in the equations, actually better explain growth than the variables included in the equations? (Pien kowski and Berkowitz, 2015) There is a trade-off between an arbitrary selection of a small number of variables which may give rise to omitted variable bias, and introduction of a large set of variables which may make it difficult to identify important variables: However, the results are often not robust to change in the model specification. A way to deal with the problem of estimating the causal effect of a policy is the use of modern evaluation techniques for causal inference, scarcely used in the field of regional policies. Page 5
The new econometric techniques to assess the impact of Cohesion Policy on regional growth are based on the counterfactual methodological approach The proposed methodologies are mainly based on the the regression discontinuity design (RDD), and the generalised propensity score (GPS) The RDD is used in the papers of Becker et al. (2010, 2013, 2016) and Pellegrini et al. (2013, 2015). The authors build on the allocation rule of Objective 1 funds to compare the effect on the regions with a per capita GDP level just below the eligibility threshold (75% of the European Union average) with the per capita GDP of the regions just above since they did not get this type of funding. Page 6
An example of RDD from Pellegrini et al (2013): non-parametric approach (logarithmic scale), 1995-2006
Advantages: The method requires limited information (regional per capita GDP level and per capita GDP average annual growth rate); No model is required. Therefore it bypasses many of the concerns related to model specification; Results are robust and comparable to that of a randomised experiment (high internal validity). Limitations: Results cannot be extended to regions far from the threshold (low external validity of the design); We estimate only an average policy impact, without explaining the link between policy and the rise in regional growth. Page 8
The GPS is a non-parametric method to estimate treatment effects conditional on observable determinants of treatment intensity. The GPS addresses the problem of a continuous treatment, as it should be able to correct for selection bias into different levels of treatment intensity by comparing units that are similar in terms of their observable characteristics. The GPS methodology allows the estimation of the doseresponse function, that shows the relationship between the amount of transfers and regional growth However, all the estimators based on the matching approach suffer the strong heterogeneity of regions, which is hardly captured by the observed covariates. Page 9
Since RDD mimics a block randomization within the neighbourhoods of the cut-off point, in the presence of small sample the researcher has to check whether or not treated and non-treated units in the neighbourhood of the cut-off point are indeed balanced with regards to the other relevant confounding factors (different from the forcing variable). In these cases, GPS is a viable alternative to RDD, because it can exploit all the regions included in the common support. Two recent papers used the GPS in this context: Mohl and Hagen (2010) shows that SF payments have a positive, but not statistically significant, impact on EU regions growth rates ; Becker et al. (2012), applying GPS to NUTS3 regions, find that, overall, EU transfers enable faster growth in the recipient regions as intended. Page 10
Page 11
Another factor that affects the evaluation of the impact of Cohesion policy is the high heterogeneity in transfer intensity. Actually, although SF payments should be the main variable of interest, many studies in the literature use only a dummy (binary) variable indicating whether a given region is eligible for SF transfers or not. Using dummy variables for SF payments neglects substantial differences in aid intensities between regions. Transfers intensity varied from below 1 % of GDP in some Objective 1 regions to above 10 % in the others (Pieńkowski and Berkowitz,2015). High heterogeneity of transfer intensity across regions suggests that the intensity of allocated funds between regions is a potentially important source of variability of the SF impact. Page 12
For instance, in the period 1994-06 the region of North-Holland received an annual average per capita transfer close to 9, whereas the Regia o Auto noma dos Ac ores (PT) almost 85 time more ( 773). Limiting the analysis to the regions with Objective 1 (Ob. 1) status during the entire period 1994-2006, and excluding those of Sweden and Finland, the lowest amounts of the capita transfers (in the regions of Burgenland AT - and Merseyside UK) is more than eight and half times lower than the maximum. Pellegrini - Cerqua UE Commission, Sixth report on economic, social and territorial cohesion, 2014 Page 13
The differences in the intensity of SF reflect the choice to allocate more resources to the regions that are particularly in need of assistance. However, the relationship between the aid-intensity and the impact of the SF is not known. Economists and policy makers ignore whether the marginal efficiency of transfers is constant or increasing or decreasing in some parts of the relationship. There are 2 main reasons to suggest that the dose-response function of the SF transfers may not be linear: diminishing returns to investment and limited absorptive capacity of SF transfers. Page 14
An important question is the normalization of the EU regional expenditure. The method used by the Commission in the allocation of resources for each Member State is based on a financial allocation per inhabitant per year, to be applied to the population living in the eligible regions The average population by region seems the natural normalization variable. However, in the literature, the beginning-of period GDP (1994 here) has been used (Mohl and Hagen, 2010; Becker et al., 2012). The reason is that this ratio could be interpreted as a minimum target of the impact of SF on the economy. METHODOLOGIES TO ASSESS THE OVERALL EFFECTIVENESS OF EU Page 15
SF Intensity => (SF/Population) SF Intensity (SF/GDP in 1994) => Page 16
Regional distribution of SF (Intensity = SF / Population)
Data reliability is a central issue in the analysis of the effects of EU regional policy. On the one hand, the measurement of the treatment (the amount of financial resources by region) has to consider actual expenditure (in the EU jargon, certified expenditure ). On the other hand, economic growth measures must be comparable across time. However, the recent literature does not fully take into account these data issues. For instance, several studies use as main outcome variable the average annual growth of per capita GDP in purchasing power parity (PPP). Unfortunately, PPPs are spatial indices, used to make spatial volume comparisons of GDP or GDP per head. This point is raised in the Eurostat-OECD (2006, pp. 32, 33), that signals: the rates of relative growth derived from the (PPP) indices are not consistent with those obtained from the constant price estimates of GDP of countries and concludes that the use of PPPs as a means of constructing national growth rates is not recommended. Page 18
The analysis that we present is based on a new, reliable and comparable dataset, stemming from several sources. The construction of the final dataset has been the result of a joint work with Daniele Bondonio, Flavia Terribile, Daniele Vidoni and other DG Regio staff members. The spatial grid used in our work is defined by EU-27 regions at level 2 of the NUTS classification. We use the NUTS 2006 classification with adjustments to include data from 1994-1999 programming period: Data on EU Structural and Cohesion Funds payments to Member States, broken down by programming period (1994-1999, 2000-2006, 2007-2013) and region per year, has been provided by the European Commission-DG REGIO. The originality and relevance of this dataset arises from its internal coherence (EU payments by operational programme per year) and extensiveness (it covers all the main funds, including the Cohesion Fund, the European Regional Development Fund (ERDF), the European Social Fund (ESF), EAGGF and FIFG EU payments to operational programmes are a proxy of payments to beneficiaries which, in turn, are a proxy to effective project implementation. Page 19
We present a study based on RDD and on the regional expenditure referring to the two programming cycles from 1994 to 2006 Our dataset excludes from the analysis 4 regions whose level of per capita GDP in the period 1988 1990 was above 75% of EU average, but were included in Ob. 1 for political reasons : Prov. Hainaut (BE), Corse (FR), Molise (IT), Lisboa (PT). Moreover, two regions (Aragón in Spain and Dytiki Makedonia in Greece) were clear outliers and were dropped. In line with the RDD approach, we also selected a restricted sample, which includes the regions closest to the discontinuity. In order to still maintain a sufficient number of degrees of freedom, we have eliminated the lowest quarter for treated regions (in terms of initial level of per capita GDP) and the upper quarter for the non-treated regions. the restricted sample is then equal to 152 regions (40 treated and 112 non-treated regions). This smaller sample will be used for the main part of the analysis. Page 20
Eligible areas and treated and non-treated regions (Task 1)
The idea of the use of the RDD in the evaluation of the Cohesion policy exploits the source of local randomness due to the sharp discontinuity in the assignment of different transfer intensity (75% of average GDP criterion). We also propose in our paper the use of the continuous RDD, which for the first time allows a compelling evaluation strategy in the presence of a continuous treatment. A nice aspect of RDD is that we can plot the data and see if the data support our hypothesis. In our case we can show the relationship between 3 variables (per capita GDP 1988-90, GDP growth rate 1994-2006, intensity of transfers) using a 3D graph. Page 22
Page 23
The graphs show the typical shape of the RDD. On average, Objective 1 regions show higher growth rates than other EU-15 regions, and in the graph this is represented by a clear discontinuity. However, we are interested in the relationship between intensity and growth. The most interesting aspect of the figure is the concavity that is created in the surface along the intensity axis: the relation between intensity and growth is not linear: first it is steady and then growing among the non-treated regions; while, it increases and then decreases for treated regions. Thus, there is evidence that NUTS 2 regions receiving lower SF intensity are much more sensitive to SF intensity changes than NUTS 2 regions receiving higher SF intensity levels Page 24
Our idea is to extend the RDD to the case of continuous treatment, considering the intensity as a cause of the impact heterogeneity The idea is to compare differences from the mean treatment level for treated and not treated regions around the cut-off. However, when the number of observations is finite and limited, the heterogeneity in covariates can dominate the heterogeneity in the treatment. One alternative is to combine designs and to assume that, after conditioning on covariates, treatment assignment (in differences from the mean) is as-if randomized for those regions near the discontinuity. Our approach is a combined design, considering heterogeneity in RDD after balancing in pre-treatment covariates. A similar approach, although adopted in a different framework, is presented in Keele et al. (2015). Here I do not go through the model, and I am going to describe only the results in brief by some graphs Page 25
RESULTS: SF Intensity (SF/population) Page 26
The main result of this study is that the positive and statistically significant impact of SF on regional growth is confirmed. In the case of the fully specified RDD model, over the period 1994-2006, in the EU 15 regions, the impact of receiving the higher average intensity of the Obj1/Convergence regions is equal to +0.7 percentage point (p.p) in the annual growth of per-capita GDP. This is compared to a counterfactual status of receiving the lower average intensity of the non-obj1/non-convergence regions. In Pellegrini et al. (2013), with a different model specification, such average impact is equal to +0.9 (p.p.), close to our preferred RDD estimate. When we normalize the EUF intensity in terms of share of the initial (1994) GDP, our impact estimates are in the order of 1.1% of 1994 GDP. The results from the PSM models are similar, with impact estimates ranging from +0.3 to +1.0 percentage points in the annual growth of per-capita GDP caused by the higher average SF intensity of the Ob.1/Convergence regions. Page 27
However, the RDD results show that the positive impact of the SF intensity on the growth of the Ob. 1 regions is decreasing the higher are the regional transfers Thus, the data suggest that the NUTS-2 regions with lower levels of SF show a bigger impact on GDP per head of increases in SF intensity than the NUTS-2 regions with higher levels of SF. After a certain intensity threshold, additional SF transfers are not, on average, associated with significantly higher regional GDP growth. Similar results are estimated by the RDD model applied to the same EU-27 extension of the analysis. The impact of the higher EUF intensity on the Objective 1/convergence regions is positive but not statistically significant. However, adding into the analysis the last programming period, where the economic crisis is evident, has a substantial effect on the EUF impact, which is almost halved to +0.4 percentage points per year. Page 28
A richer dataset is needed if the counterfactual evaluation wants to tackle the issue of the multifaceted outcome of the EU regional policy. Currently, GDP growth is just one of the many dimensions of EU regional policy, that is oriented to reduce economic and social disparities across European regions. Therefore effects on GDP are important but cannot exhaust the purpose of the SF intervention. The recent papers of Becker et al. (2016) and Cerqua and Pellegrini (2015) go in this direction. Finally, we need additional information (beyond the year 2011) for a more robust empirical analysis of the last programming period (2007-2013), where the heterogeneity across regions is higher, due to the presence of new Member States and the largest economic crisis in Europe since WWII was in action. The empirical findings for this programming period will have to be confirmed when the complete data become available. Page 29