ADMINISTRATIVE DATA FOR RESEARCH AND EVALUATION

Transcription

1 MUTUAL LEARNING PROGRAMME: PEER COUNTRY COMMENTS PAPER - NORWAY ADMINISTRATIVE DATA FOR RESEARCH AND EVALUATION Peer Review on Evaluation of Labour Market Policies and Programmes: the use of data-driven analyses Belgium, November 2012 A paper submitted by Erling Barth in consortium with ICF GHK Consulting and CERGE-EI Date: 25/10/2012

2 This publication is supported for under the European Community Programme for Employment and Social Solidarity ( ). This programme is managed by the Directorate-General for Employment, Social Affairs and Inclusion of the European Commission. It was established to financially support the implementation of the objectives of the European Union in the employment and social affairs area, as set out in the Social Agenda, and thereby contribute to the achievement of the Lisbon Strategy goals in these fields. The seven-year Programme targets all stakeholders who can help shape the development of appropriate and effective employment and social legislation and policies, across the EU-27, EFTA- EEA and EU candidate and pre-candidate countries. PROGRESS mission is to strengthen the EU contribution in support of Member States' commitments and efforts to create more and better jobs and to build a more cohesive society. To that effect, PROGRESS will be instrumental in: providing analysis and policy advice on PROGRESS policy areas; monitoring and reporting on the implementation of EU legislation and policies in PROGRESS policy areas; promoting policy transfer, learning and support among Member States on EU objectives and priorities; and relaying the views of the stakeholders and society at large For more information see: The information contained in this publication does not necessarily reflect the position or opinion of the European Commission.

3 CONTENTS 1 APPROACH TO EVALUATION OF LABOUR MARKET POLICIES AND PROGRAMMES EVALUATION USING ADMINISTRATIVE DATASETS ASSESSMENT OF THE SUCCESS FACTORS AND TRANSFERABILITY QUESTIONS... 9 ANNEX 1: SUMMARY TABLE ANNEX 2: REFERENCES... 11

4 1 APPROACH TO EVALUATION OF LABOUR MARKET POLICIES AND PROGRAMMES This paper has been prepared for a Peer Review within the framework of the Mutual Learning Programme. It provides information on Norway s comments on the policy example of the Host Country for the Peer Review. For information on the policy example, please refer to the Host Country Discussion Paper. Norwegian researchers and evaluators have been using administrative register data for evaluation purposes for quite some time. Evaluation of labour market policies and programmes is an integral part of labour market policy making and debate. Evaluation studies have both been financed through research programmes, directly commissioned regarding specific policy tools or reforms, and been undertaken internally by the agency responsible for policy implementation. Responsibility for labour market policy lies with the Ministry of Labour. The main policy areas of the Ministry of Labour are divided into a Working Environment and Safety Department, a Department of Welfare Policy, a Pensions Department and a Department of Labour Market Affairs. The Department of Labour Market Affairs is responsible for the promotion of a well-functioning labour market, for policies and measures taken to help the unemployed, and for some of the measures aimed at disabled individuals. Within the Department of Labour Market Affairs, the Analysis and Income Policies Unit is responsible for employment policies, labour market measures and income policies. It also has responsibility for the coordination of research related to the labour market and income policies. Responsibility for implementing a large part of labour market policy is allocated to the Norwegian Labour and Welfare Service (NAV). Their objective is to facilitate matching in the labour market between job seekers and vacant positions, and to ensure comprehensive follow-up of people who need work-oriented assistance to find employment. They organize labour market measures aimed at increased participation in employment, reduced unemployment and through combating exclusion by helping people with problems on the labour market to find work and become active. They also administer the Norwegian rehabilitation benefit system. Resources for research and evaluation The Ministry allocates yearly in total about NOK 350 million to knowledge development. This includes about NOK 120 million allocated through the Norwegian Research Council for research programs that cover the area of its policy areas, including labour market policies. One large research program who gets about 60 million per year is the Welfare, Working life and Migration,(VAM) programme, including as diverse topics as ageing, consequences of increased wealth, international migration, family and society, economic growth and organisation of working life, and the popular support for, organising and governing of the welfare society. They also finance evaluations of large public sector reforms, such as the NAV reform and the pension reform through the Norwegian Research Council. The Ministry also finances the National Institute of Occupational Health (STAMI) whose main tasks are to create knowledge and inform about the connection between work, illness and health, to monitor occupational health factors, evaluate risk factors, and suggest preventive measures. The Ministry also commissions evaluation studies, most of which uses register data. The evaluation efforts commissioned directly by the Analysis Unit of the Ministry of Labour are directed towards effect evaluations of particular programs or labour market policy measures. 4

5 The Ministry also funds research and development mainly organized and commissioned by the policy implementing agency, the Labour and Welfare Service (NAV). The FARVE program, administered by NAV is funded with approximately NOK 30 million per year The evaluation efforts commissioned by the Labour and Welfare Service are directed towards the efficient implementation of policy goals, regarding both organization and implementation of particular policies and their effects 1. This is done through both experiments, research and development efforts. Ongoing projects are found within the areas of the labour market, working environment, ethnical minorities, disability studies, senior workers, sickness absence and youth, and include projects on various programmes and for different groups. NAV commissions projects to external research groups, but it also has its own analysis unit of approximately 10 analysts who work particularly with NAV s data bases. The Minstry is also involved in both the development and commissioning of analyses using randomized trials. 2 EVALUATION USING ADMINISTRATIVE DATASETS The Ministry funds research and evaluation using administrative datasets both by the Analysis Unit commissioning external evaluation projects themselves, in particular regarding national policy measures. Examples of such projects include assessment of the potential extent of social dumping following increased immigration, using wage statistics and evaluations of the Including Work Life Agreement which is a tri-partite agreement implemented to reduce the extent of sickness absence, through research funds allocated through the Norwegian Research Council, and through NAV. In addition, the Labour and Welfare service (NAV) may undertake their own analysis, using their own data. Most data including benefit, services and benefit status originate and is owned by NAV (see below). Using their own data, they may also choose to use the detailed information on which their data are based, for instance underlying data from applications or detailed accounting data. External researchers and evaluators who perform the external analyses generally obtain the data through Statistics Norway, after having obtained permission from various data owners (such as NAV) and the Data Directorate. Matching of various data sources is done by the Statistics Norway, and external institutions and researchers that are approved by the Norwegian Data Protection Authority may obtain matched data set with internally consistent, but encrypted identifiers, to use in their analysis. Data An important feature, and potential problem, with administrative data is that they require collaboration between several agencies because the data is collected for various purposes across a host of different bodies and agencies. All these data need to be matched in sensible ways for research and evaluation purposes. The data available through Statistics Norway consists of several data registers. A description of which is available at from which the below information is taken: It comprises Register of companies The Central Register of Establishments and Enterprises Income register data The Norwegian National Education Database (NUDB), dynamic data 1 5

6 Register-based employment file Social security and national insurance FD-Trygd, dynamic data Population Family and generation data Population and Population and Housing Censuses of which the register based employment data and Social security and national insurance data are the most important ones for labour market policy research and evaluations. The register-based employment data files are produced by combining data from the following sources (Norwegian names of registers in parentheses): The Central Register on Employers and Employees (Arbeidstaker-Arbeidsgiverregisteret) End of the Year Certificate Register (Lønns- og trekkoppgaveregisteret) The Registers of Conscripts and of Conscientious Objectors (Verneplikts- og Siviltjenesteregisteret) The Central Coordinating Register for Legal Entities (Enhetsregisteret) The Central Register of Establishments and Enterprises (Bedrifts- og foretaksregisteret) The Register of Job Seekers (Arbeidssøkerregisteret ARENA) Several registers of government employees and employees of local and regional authorities Wage statistics for employees in the private sector The Sick Leave Register (Sykefraværsregisteret) The Register for Personal Tax Payers (Selvangivelsesregisteret) The registers used for classification of salaried employees are the Central Register on Employers and Employees (the EE-register), the End of the Year Certificate Register, and the Registers of Conscripts and of Conscientious Objectors. Identification of self-employed individuals is done on the basis of the Register for Personal Tax Payers and the Central Coordinating Register for Legal Entities. The EE-register is linked to the Norwegian public health insurance system. All employers are required to register with the employers' part of the EE-register. Furthermore, employers are required to register all employees with the EEregister. FD-Trygd is a historical event database where the main topics included are demography, social conditions, social security, employment, search for work, state employees, income and wealth. The statistical unit is the person, and information in the database consists of registrations of events in each personal life span. These registrations can be put together to individual event histories, and for a group of persons or a whole population, event data will provide a relevant overview of movements between continuous events/ changes in circumstances. FD-Trygd contains information for the whole population from 1992 and onwards. For each person in the database it is possible to connect all people belonging to the same family. The family definition is coherent with the system for personal registration. The information in FD-Trygd is organized according to topics in the database (Oracledatabase). It contains one data file for each topic or each scheme. Each set of data provides information on when a person enters a scheme (entry), and when a person leaves a scheme or parts of a scheme (exit)... The database contains information from following sources: Registers in Statistics Norway, NAV (the Norwegian Labour and Welfare Organisation, earlier called Rikstrygdeverket (National Insurance administration) and Aetat 6

7 (Employment directorate) and Skattedirektoratet (Taxation). It contains event data on different schemes such as: Demography (settlement, immigration, families, marriage and movement) Pension and benefits (disability, old age and contractual pension, rehabilitation allowance, maternity- and sickness benefit, child benefit, single parent benefit), Employment and job seekers, and Social assistance. In addition FD-Trygd contains information on income, which is are not on an event basis (transfers, income and tax amounts). Event history within education is also constructed upon the same model as FD- Trygd, and it's possible to match data from FD-Trygd and NUDB (National education data base). See for further details. It is important to note that administrative register data also allows for the use of smaller samples. The registers may be very large and difficult to manage. A smaller sample may allow for smoother work, and following standard sampling theory, there is no need to work with the full universe of data. The Norwegian use of administrative register data actually started off with a sample drawn from several administrative data; the KIRUT data base which contains a 10 percent sample of the Norwegian population. Advantages of using administrative register data In the article Administrative Registers: Unexplored Reservoirs of Scientific Knowledge? Røed and Raaum (2003) discuss the merits of using administrative register data. They mention five areas where administrative register data may be particularly useful for labour market research and evaluation. The first is matching and the effect of social programs where they find administrative data to be particularly promising for non-parametric analyses using matching techniques for at least three reasons. First, an observationally identical person that can be used to predict the counterfactual outcome is 'always' available. Second, register data ensure that participants and non-participants belong to the same local labour market at the same time, with similar pre-programme labour market history and that the personal characteristics as well as performance criteria are measured in exactly the same way. When different surveys are used to collect information on the two groups, treatment effects are likely to be biased. Third, register data allow a maximum degree of flexibility regarding heterogeneity in individual effects, i.e. the extent to which the causal effects interact with explanatory variables, such as the state of the business cycle. the second is structural modeling and the identification of complex budget sets, where the full coverage of the population allows for very flexible modeling strategies. The also stress the possibility of disentangling micro and macro phenomena: Since the micro units covered by (complete) administrative data typically add up to the macro numbers, register data often provide the 'missing link' between the micro and the macro level of the economy. This makes it possible to decompose macroeconomic or aggregate patterns into their appropriate micro phenomena. The size of the data also allows for careful studies of small groups and rare events. Finally, administrative register data may allow for analyses of intergenerational issues based on matches between parents and children and siblings that are rare and typically small in surveys. The Host country paper discusses several advantages of using administrative register data. I would like to add emphasis to two points. The first is accuracy; an important reason why administrative data are accurate, is that they typically are entered for a purpose, for instance related to accounting purposes, tax reporting etc. that are under strict rules of control and auditing. This ensures accuracy compared to survey data where the entries typically do not matter for the person giving information. (of course, a caveat is in order: In some instance the fact that the entries actually matter may induce purposely wrong figures as well, as in the case of tax evasion, also mentioned by the host paper). Another point is related to non-response as mentioned by the host country paper; this issue is particularly relevant when we are evaluating effects of social programs, because non response is likely to be much bigger of a problem for individuals who are poor, do not have a home, have addiction problems etc. inducing serious biases in survey based studies in this area. On the other hand, as discussed by the host country paper, surveys may be necessary to obtain 7

8 information on attitudes, emotions, motivation and outcomes other than those recorded in administrative registers, such as happiness or self-reported well-being. Another advantage of administrative register data is that the researchers are in control of the sampling procedure as well as the population. This makes it possible to use administrative data to design various forms of samples and experiments, and have full control of issues such as attrition, non-response etc. Renewed emphasis has been made on exactly the topic of research design (see eg. Angrist and Pischke, 2010). This is an important advantage of the register data. Examples of recent evaluations using administrative register data. One example of evaluation is given by the evaluation of the Norwegian youth guarantee effort from 1995, basically involving increased effort in active labour market programs for youth (see Hardoy et al 2006). They use administrative records on unemployment spells, including information on demography, labour market status, education, benefits and wages for the period Using two different methods, one based on a difference in difference approach, and the other on duration analysis, they conclude that the youth guarantee led to higher program participation and a more rapid transition into employment for unemployed youth. Another example is given by Markussen et al (2012) who study the effect of graded absence certificates on sickness absence, using data from administrative registers including encrypted information about all citizens and their primary care physicians from 2001 and onwards. The data include detailed longitudinal information on employment and social security spells, and annual information on earnings, as well as longitudinal information on all certified absence spells from 2001 through 2005, including diagnosis and grade (graded or not graded). These data are merged with different administrative registers (employer-employee, education, demography, social security, income/taxes) Their analysis shows that activation requirements not only bring down benefit claims, they also reduce the likelihood that long-term sickness absence leads to inactivity. Absentees who are assigned graded (partial) absence certificates by their physician have shorter absences and higher subsequent employment rates than they would have had on regular sick leave. 3 ASSESSMENT OF THE SUCCESS FACTORS AND TRANSFERABILITY The extensive use of administrative data in Norway has been enabled by a collaboration between researchers within and outside of Statistics Norway, supported by research funding agencies, such as the Norwegian Research Council and the Ministry of Labour, as well as by collaboration with, and a research oriented attitude among several key administrative data owners. A key success factor in the Belgian case was the Crossroads Bank project. In Norway, Statistics Norway is the agency that collects, to a large extent maintains, and matches large data bases, either from own collections, or from the various data owners. The fact that Statistics Norway has a strong research orientation, with its own research department and a commitment for maintaining and supplying research data, has been an important factor behind the extensive development and use of administrative register data. The role of the Norwegian Data Protection Authority has also been positive, in that they have allowed for extensive use of such data and at the same time provided a guarantee for the protection of privacy. 8

9 4 QUESTIONS Do you have any suggestions as to how to improve on the timing of data delivery to researchers and analysts in order to improve the quality and relevance of evaluations? Have you considered the combination of randomized experiments and the use of administrative register data? What sources would be best for international comparisons, surveys or registers? The success factors seemed to rely partly on good will on part of some individuals. Could you think of institutional factors that would promote the ability to use such data for these purposes in countries where administrative data are currently less available for research and evaluation purposes? 9

10 ANNEX 1: SUMMARY TABLE Approach to evaluation of labour market policies and programmes Extensive use of administrative register data has a relatively long history in Norway. Research and evaluation is conducted both by external research institutes, evaluators, and by the policy implementing unit NAV Evaluation using administrative datasets Use of administrative register data is a cost effective, high quality source of data for research Administrative data allows researchers to have better control of research design The use of administrative register data requires matching routines and data sharing Assessment of success factors Statistics Norway collects and matches data registers from various sources and makes them ready for research purposes Use of administrative register data may alleviate several methodological challenges associated with evaluation studies and the analysis of economic behaviour in general Questions Do you have any suggestions as to how to improve on the timing of data delivery to researchers and analysts in order to improve the quality and relevance of evaluations? Have you considered the combination of randomized experiments and the use of administrative register data? What sources would be best for international comparisons, surveys or registers? The success factors seemed to rely partly on good will on part of some individuals. Could you think of institutional factors that would promote the ability to use such data for these purposes in countries where administrative data are currently less available for research and evaluation purposes? 10

11 ANNEX 2: REFERENCES Angrist, J. and Pischke, J - S. The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics, Journal of Economic Perspectives, 24(2):3-30. Hardoy, Ines, Knut Røed, Hege Torp, and Tao Zhang (2006) Virker Ungdomsgarantien (Is the youth guarantee working?) Søkelys på arbeidsmarkedet 1/2006. Vol 23: Markussen, Simen, Arnstein Mykletun, Knut Røed, 2012: The Case for Presenteeism - Evidence from Norway s Sickness Insurance Progra, Journal of Public Economics, Forthcoming Røed, Knut and Raaum, Oddbjørn, 2003: Administrative Registers - Unexplored Reservoirs of Scientific Knowledge?, Economic Journal, vol 113, f258-f281 11