Effectiveness of government interventions at labour markets: the case of women and youth in Serbia

Transcription

1 MPRA Munich Personal RePEc Archive Effectiveness of government interventions at labour markets: the case of women and youth in Serbia Jovan Zubović and Ivana Domazet Economics Institute, Belgrade, Serbia 15. May 2012 Online at MPRA Paper No , posted 13. February :00 UTC

2 Effectiveness of government interventions in the labour markets the case of women and youth in Serbia 1 Dr Jovan Zubović, Economics Institute, Belgrade Dr Ivana Domazet, Institute of Economic Sciences, Belgrade Abstract: In this paper there has been evaluated the effectiveness of the active labour market policies on two largest vulnerable groups in Serbian labour market - women and youth population. By the means of using adapted methodology of other authors we concentrate on in-depth empirical research within the above named target groups with a goal of determining what policies bring most gains. Moreover by using econometric matched pair design methodology we have undertaken a microevaluation of several different ALMP used in Serbia with a goal of obtaining precise information on the difference in effects among measures. Results that we have achieved are to a certain extent surprising, showing that widely utilised matching methodology can be altered and improved. On the other hand we found that women and youth perform better than average in effectiveness of active labour market policies. Key words: Women and Youth in Labour Markets, ALMP, Evaluation, Matching Model JEL Classification: J08, J21, J18, J68 Introduction Government interventions against falling employment and rising unemployment include Active Labour Market Policies (ALMP). The evaluation of their economic impact is widely discussed topic by academics worldwide for several decades. However, improved databases and modern statistical software facilitate more precise analysis and evaluation of the economic impact which they create. Several factors including recession have led the employment in Serbia to fall to a historical minimum in Expenditure on ALMP in Serbia equalled to only 0.1% of GDP in 2009 and 2010, substantially lower than 0.76% which is an average in EU27. Increase of the expenditure to 0.14% in 2011 generated positive results measured by econometric tools (Zubović, Subić 2011; Eunes 2011), but it has not been accompanied by comprehensive research on the vulnerable groups. In this paper we aim to implement an in depth research on the effectiveness of ALMP in Serbia on two largest vulnerable groups - female and youth. Research is based on the methodology introduced in the project supported by RRPP (2012), adjusted to targeted population groups. Moreover by the means of econometric tools we evaluate the difference of the effects of targeted policies on women and youth as compared to general population in the Serbian labour market. The paper consists of five parts: The first part reviews literature on evaluation of active policies. In the second part we present the research methodology used in the paper, while in third we deliver the results collected form the Serbian labour market. In the fourth part we analyse and discuss our findings. Finally we give conclusions and recommendations for further research. 1 This paper is prepared in partial fulfilment of the project "Macroeconomic analysis and empirical evaluation of active labour market Policies in Serbia" supported by Regional Research Promotion Programme in the Western Balkans (RRPP) of the University of Fribourg

3 Literature review on active labour market policies and their evaluations Labour market policies were initially introduced through Public Works at the beginning of the twentieth century as an answer to growing unemployment. Economists of that era, most of all Keynes, had worked on development of the (un)employment theory and models for managing labour market trends. The theory of multipliers introduced by Kahn (1931) was used by Keynes (1936) to prove that public works and government interventions can help fight unemployment. First ALMPs were introduced after the WWII and have significantly changed since. Firstly introduced in the Scandinavian countries they served as an integral part of the model of economic and social change. They were used to reduce short-term inflationary impact of high employment along with solving the problem of fast-growing demand for labour (OECD, 1964, Barkin, 1967). Mostly positive impact of initial measures was presented in several papers (OECD 1993, Katz 1994 etc.). Estevao (2003) and Betcherman et al. (2004) pointed out that the constant increase in the unemployment rate in the 1970s and 1980s, which came after the oil shock crisis, was a consequence of a mismatch in labour supply and demand. Unemployment in OECD countries increased by 3% in 1988 (Martin, 2000). Emancipation of women and young people has led to significant growth on the supply-side in labour market. Therefore it was necessary, at first in France, Germany and the United States, to introduce new programs targeting labour supply, specifically vulnerable groups. Interventions were extensively used to facilitate adjustment of labour to market needs. In that stage the ALPMs became a part of the employment strategies in transition countries in the form of public works or training programs (OECD, 1990). Growth of the expenditure on ALMP made evaluation of their effectiveness a necessity. According to Harrell et al (1996), there are four basic types of evaluations: performance monitoring, impact evaluation, cost-benefit analysis and the process evaluation. Zubović and Subić (2011) presented results of research based on cost-benefit model conducted in Serbia for the period Several other papers define methodological framework for the evaluation of the impact of ALMP (Fay 1996; Dar and Tzannatos 1999; Daguerre and Etherington 2009; OECD 1993). The first scientific papers on the evaluations, like Calmfors (1994) brought very puzzling results. However, advanced information systems eased analysis of data, therefore Lehman and Klueve (2010) claim that using improved research methodology it is possible to prove that ALMPs have positive effect both on individual likelihood of exiting unemployment and on aggregate employment growth. In this paper we will use the principle presented by de Koning and Peers (2007). Our focus will be on use of matching model comparing participants results with the ones of the control group. Over the past fifteen years there was a significant increase among researchers in countries of Central and Eastern Europe. These studies helped us to better understand how labour markets act in the new economic environment introduced by transition (Lehmann, Klueve 2010). In those countries budgets available for ALMP are very limited, and for that reason it is important that the effects are properly assessed in order to make the right distribution among different types of measures. Evaluations in transition countries include several papers (Zubović, Simeunović (2012); Zubović, Subić (2011); Lehman, Klueve (2010); Ognjenovic (2007); Bonin, Rinne (2006); Betcherman, Olivas, Dar (2004); Spevacek (2009) and many others). Methodology Zubović and Simeunović (2012) analysed the effects of ALMP on the whole population of registered persons at NES at the beginning of 2008 and 2009 without using econometric models to prove causality of the effects. In order to make results more robust it is necessary to enrich the

4 methodology by creation of a valid control group and performing matching test in order to determine what exact effects the investments in ALMP in Serbia result with. Zubović and Subić (2011) note that classically designed experimental evaluations start with creation of a randomly selected sample (or use a complete population) of unemployed persons before they were exposed to any active policy. If the sample is large enough and if there is a proper control group, by changing the independent variable (in this case participation in any type of ALMP), we may measure the change in the achieved results. Such changes can be attributed to participation in ALMP. The matching methods create a subset of the control group whose members are paired with participants in the factors measured, and thus get precise and robust results. According to Garson (2010) a matched pairs design selects different but somewhat similar participants according to any important characteristics that might affect performance. In order to avoid such problems there have been developed new quasi-experimental models. They differ from experimental models in a method of selection of experimental and control group. Instead of a random sample they are selecting participants after the measures have been implemented. By using econometric techniques of matched pairs it is possible to correct the disparities between the two groups, and with the low cost to evaluate the effects independently of the implementation of the policies. The sample in our research consist of the individuals who have participated in ALMP (experimental group), and a control group of individuals with whom we have compared five observable variables 2 (characteristics) prior to exposure to the treatment (active measures) in the period prior to In order to facilitate selection of control group, we have shortened our experimental group to 17,943 persons who participated in ALMP (and exit from them) in the period 01 Jan June In evaluation we compared their results with the results achieved by the control group of unemployed persons of the same size, who had equal chance of being selected, but did not participate in the implementation of active measures. Thus, the average effect of ALMP was defined as the difference in employment rates achieved by two groups of persons, after we ensure consistency across the observed variables. Effectiveness of the measure was made based on the results of comparison of achieved results measured by two different outcomes. The first is employment status 3 months after the exit from the measure (Yes/No). The second is the number of days person was employed in the period of 6 months after the exit from the measure. That was made by using the access to interlinked databases of NES and the Institute of Social Insurance, by which we were able to trace and distribute all persons from our experimental and control group who have found jobs into 21 sub-groups with total days of employment according to business sectors in NACE rev.2 classification. Using the data sorted according to NACE classification helped us to perform a cost-benefit analysis published in Zubović and Simeunović (2012). Combination of the results from this paper and the one named above will be the basis for the further research necessary to be conducted in order to complete robust cost-benefit analysis with precise information on the netfinancial benefits of the investments in ALMP. Moreover, another novelty of this paper is an attempt to cope with a problem of data on ALMP in Serbia which is recorded and sorted according to the national classification that significantly differs from the EC methodology (EC 2006) which divides them on 6 categories (Training, Job rotation and job sharing, Employment incentives, Supported employment, Direct job creation and Start-up incentives). Since data sets available from the NES are not comparable to the EC Methodology we 2 The first criterion is Gender which has 2 categories: Men, Female. Second criterion is the "Region", which has 30 categories. The third criterion is "The level of education" which has 10 categories. The fourth criterion is "Age" which has 10 categories: 15-19, 20-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, Final fifth criterion is "Occupation", which has 19 categories.

5 have in cooperation with IT centre in NES developed a software module which gives possibility to rearrange data to so as comply with EC methodology, therefore making our results easily comparable with other research in Europe. Similar methodology was applied in other research conducted in Serbia, like Eunes (2011) and Ognjenović (2007). In both of those there has been evaluated the impact of certain group of measures using matching methodology. However there has not been analysed the effectiveness of the whole set of measures implemented by NES, nor have been the results presented in such a way so as to be comparable with the studies in other countries. Research results Aggregate data on ALMP in EU and Serbia Economic reforms in countries with a centrally planned economy (transition economies) since the beginning of the nineties had significantly increased the level of open unemployment, and raised aggregate unemployment to above the EU-15 average. For that reason, budgets for ALMP increased until In the period they have been slowly diminishing in the periods of fast average GDP growth, followed by a rapid increase in 2009 and 2010 when most countries faced growth in unemployment as a result of recession. Table 1 - Expenditure on ALMP (2-7) in EU transition countries (% of GDP) GEO/TIME EU n/a EU n/a EU 10 (transition countries)* Bulgaria n/a Czech Republic Estonia Latvia Lithuania n/a Hungary n/a Poland n/a Romania Slovenia n/a Slovakia n/a SERBIA * unweighted average 1 incomplete data Source: Eurostat (2012) and Own calculations based on MERR (2011) Consolidated data on expenditures on ALMP in Serbia go back to 2008, which coincides with the end of the development of new Information system in NES. In order to make a comparison of the expenditure on ALMPs, we have used data from Eurostat database (Table 1). Data for Serbia is available for 2011 as well and it amounts to 0.17% of GDP. Unlike in other transition countries, ALMP budgets have been steadily increasing for the whole observed period. However, despite such growth they are still substantially lower than average of 0.23% in other transition countries of EU.

6 Macro data from the empirical research As explained in the methodological section, we have rearranged data from NES classification to EC methodology. The data on the number of persons included in different types of LM measures according for the period are listed in table 2. Table 2 - Persons included in ALMP measures ALMP Code ALMP Measures** No ALMP 825, , , , With ALMP 24,438 27,241 23,262 29, Training 1,851 2,699 4,312 3, Training 2,963 7,773 5,706 9,870 3 Job rotation and job sharing Employment incentives 12,482 7,309 6,486 7,138 5 Supported employment ,585 6 Direct job creation 3,854 6,150 3,471 4,034 7 Start-up incentives 2,701 2,967 2,236 3,275 Combined* Total*** 850, , , ,109 * Persons participating in over 1 measure ** Further on we will not list the names of the measures but only their codes *** Includes all persons listed at NES as of Jan 1 st of the current year Further on we have summed all expenditures for the groups of persons listed in table 2, by types of ALMP (table 3). Table 3 -A LMP expenditures (in ) ALMP Measures No ALMP With ALMP 15,555,102 25,685,579 23,958,642 44,156, , ,211 2,459,720 2,621, ,615 9,944,044 7,663,284 14,461, ,224,734 4,859,086 4,900,848 11,749, , ,827 2,957, ,525,069 7,940,810 4,922,759 6,852, ,286,840 2,008,454 2,960,905 4,888,984 Combined 565, , , ,504 Total 15,555,102 25,685,579 23,958,642 44,156,260 Using the methodology presented above, we have analyze the effectiveness of the funds spent for the ALMP. For that reason we have collected information on employment of people from our sample in two fold manner. For those who have not used any ALMP we gathered information on number of persons being employed during respective year (for at least one day) and the total number of days of employment. For persons who have participated in any ALMP, we gathered the same information but for the period of 12 months after entering the measure. Results are presented in table 4.

7 Table 4 - Employment by number of persons and working days Persons employed Days working Persons employed Days working Persons employed Days working Persons employed Days working No ALMP 110,063 19,694,841 72,591 12,460,764 82,253 14,327, ,717 16,881,759 With ALMP 18,064 5,162,352 22,005 6,123,127 14,846 4,186,389 20,600 5,640, , , ,784 1, , , ,328 7,592 2,363,666 4,280 1,408,452 6,048 1,904, ,841 3,561,978 6,895 2,287,067 5,915 2,004,103 7,083 2,338, , ,078 1, , , ,593 6,051 1,087,727 3, ,640 4, , , , , ,291 Combined , , , ,842 Total 128,127 24,857,193 94,596 18,583,891 97,099 18,513, ,317 22,522,578 After completing collection of aggregated data, we could summarize it in three sentences. Over the four year period there has been an increase of 20% in number of persons who were included in some (or several) programs of active labour market policies. In the same period total expenditure on financing those policies has increased by nearly 200%. However the effectiveness of that increase in expenditure was accompanied by a modest increase of around 10% measured in number of persons who were employed and total number of days in a respective year they have been employed. Discussion on micro data and econometric testing In order to give robustness to our research we have conducted a test using a matched pair design explained in the methodological section. From the total population, we have used a sample of 17,943 persons who have exit from ALMP they participated in the period 01 Jan June In table 5 there is information on the matching process and similarity of experimental and control groups measured by 5 variables. Table 5: Similarity test of experimental and control groups by 5 variables Rank Variable Number of Matching Mismatching categories share (in %) (in units) 1 Gender % 0 2 Region % 0 3 Education % 24 4 Age % 0 5 Occupation % 522 All five variables 96.99% 540 As seen in table 5, for only 3% of persons from the sample of experimental group were not possible to find the absolute match by all five variables. In that case as suggested by Ognjenović (2007, p. 30) it has been used a method of nearest neighbor. We can conclude that the matching process has been successfully completed. In this way we have created 17,943 pairs for whom we gathered information on two different outcomes. First one, as noted before, is the employment status 3 months after the date of exit from the program (table 6). The second is the number of days (and business sector by NACE 2.rev classification) person was employed in the period of 6 months after exiting the program (table 7).

8 Table 6: Frequency and structure of the first outcome (employment 3 months after exit) of matched pairs in Serbian NES, 2011 Number of persons ALMP code Group 3 months after exit Total Employed Unemployed Frequency Share (%) 67.7% 32.3% 100.0% Frequency Share (%) 15.4% 84.6% 100.0% 2.4 Frequency Share (%) 77.3% 22.7% 100.0% Frequency Share (%) 16.9% 83.1% 100.0% 3 Frequency Share (%) Frequency Share (%) Frequency Share (%) 76.7% 23.3% 100.0% Frequency Share (%) 35.2% 64.8% 100.0% 5 Frequency Share (%) 88.5% 11.5% 100.0% Frequency Share (%) 17.3% 82.7% 100.0% 6 Frequency Share (%) 50.0% 50.0% 100.0% Frequency Share (%) 0.0% 100.0% 100.0% 7 Frequency Share (%) 41.7% 58.3% 100.0% Frequency Share (%) 29.7% 70.3% 100.0% Total Frequency Share (%) 67.9% 32.1% 100.0% Frequency Share (%) 26.2% 73.8% 100.0% We must draw attention that in this analysis we have not determined the level of deadweight, substitution effects and displacement effects. That is something which remains to be performed in the continued research in order to make results the most robust as possible. Despite those drawbacks results shown in table 6 prove very high effectiveness of the measures financed on Serbian NES. The weakest effects are observed in the category of Start-up incentives (7) where the members of experimental group had employment on the day 3 months after exiting from the treatment by 40% higher than the control group. Best results are present in the group Training Special support for apprenticeship where it is registered over 400% higher employability 3 months after exit from the program.

9 Table 7: Frequency and structure of the second outcome (days employed in the period of 6 months after exit) of matched pairs in Serbian NES, 2011 ALMP code Total Group Number of days period of 6 months after exit Total Employed Unemployed Frequency , Share (%) 55,2% 44,8% 100,0% Frequency , Share (%) 14,2% 85,8% 100,0% Frequency Share (%) 70,7% 29,3% 100,0% Frequency Share (%) 16,2% 83,8% 100,0% Frequency Share (%) Frequency Share (%) Frequency Share (%) 21,1% 78,9% 100,0% Frequency Share (%) 31,4% 68,6% 100,0% Frequency Share (%) 78,9% 21,1% 100,0% Frequency Share (%) 15,6% 84,4% 100,0% Frequency Share (%) 29,6% 70,4% 100,0% Frequency Share (%) 0,0% 100,0% 100,0% Frequency Share (%) 21,4% 78,6% 100,0% Frequency Share (%) 26,2% 73,8% 100,0% Frequency Share (%) 40,2% 59,8% 100,0% Frequency Share (%) 23,7% 76,3% 100,0% Significantly different results are generated when calculating the second outcome (see table 7). If instead of looking at the employment status on the day of 3 months after exiting the treatment we observe the number of days the same person has been employed in the period of 6 months (182,5 days) after the exits we can see that effects of the ALMP are reduced. As long as 67.9% of persons treated were employed 3 months after exit, they have been employed for only 40% of days in the period of 6 months after the exit. If we look at the control group, the difference is significantly smaller. There is a drop from 26% of people employed on the exact day 3 months after exit, as long as they have been working for around 24% of days in the period of 6 months. Hence, we can conclude that the effectiveness of ALMP measured by the first outcome which showed gains of nearly 160% is significantly lower if measured by the second outcome and it equals 69%. That is certainly not unimportant effect, but it does not cover above named effects of deadweight,

10 substitution and displacement. At this stage we were not able to estimate the impact of those three effects, so we will continue with our analysis as it is. Setting the second outcome as more reliable, we will continue with our analysis only by using that data. Further on we will show the effectiveness of ALMP on women and youth population (15-24). We will also exclude data on frequency and continue with presenting only the share in total. First of all let us see the distribution of women and youth among participants in ALMP (table 8). Table 8: Distribution of women and youth population among participants in ALMP ALMP code Total Women Youth Frequency 1,633 1, Share (%) 100.0% 64.5% 22.7% 2.4 Frequency 5,694 3,017 3,121 Share (%) 100.0% 53.0% 54.8% 3 Frequency Share (%) Frequency 6,340 3,197 2,146 Share (%) 100.0% 50.4% 33.8% 5 Frequency Share (%) 100.0% 42.3% 48.1% 6 Frequency Share (%) 100.0% 33.3% 27.8% 7 Frequency 4,206 1, Share (%) 100.0% 39.5% 10.8% Total Frequency 17,943 8,960 6,126 Share (%) 100.0% 49.9% 34.1% As one can see from table 8 there are significant differences in distribution in different types of active measures. That is not surprising since the design of some treatments is strictly made for women or youth population. However it is interesting to note that women comprise large majority of participants in Training, whereas in other types are mostly below 50% in share. In average women are evenly included in measures as men. Regarding youth population, they comprise only 1/3 of total participants in ALMP. Their share is high only in Special support for apprenticeship and Supported employment. Let us now review the effectiveness of women and youth participants compared to total (table 9). Table 9: Comparison of the effectiveness of ALMP on women and youth against total ALMP Total Women Youth Group code Employed Unemployed Employed Unemployed Employed Unemployed Exp. 55,2% 44,8% 56,0% 44,0% 55,0% 45,0% Con. 14,2% 85,8% 13,2% 86,8% 17,1% 82,9% 2,4 Exp. 70,7% 29,3% 70,4% 29,6% 68,4% 31,6%

11 Total Con. 16,2% 83,8% 16,6% 83,4% 13,0% 87,0% Exp Con Exp. 21,1% 78,9% 21,5% 78,5% 22,0% 78,0% Con. 31,4% 68,6% 31,9% 68,1% 38,2% 61,8% Exp. 78,9% 21,1% 63,9% 36,1% 70,8% 29,2% Con. 15,6% 84,4% 18,1% 81,9% 13,9% 86,1% Exp. 29,6% 70,4% 16,8% 83,2% 10,2% 89,8% Con. 0,0% 100,0% 0,0% 100,0% 0,0% 100,0% Exp. 21,4% 78,6% 21,6% 78,4% 18,2% 81,8% Con. 26,2% 73,8% 26,5% 73,5% 26,8% 73,2% Exp. 40,2% 59,8% 42,1% 57,9% 47,6% 52,4% Con. 23,7% 76,3% 23,5% 76,5% 23,1% 76,9% When looking on the last two rows of the table it is easy to see that both women and especially youth show better results as compared to total sample of 17,934 persons who participated in ALMP in the first half of This is not a result of achieving better results than average in any specific group of measures, but it was rather a result of greater participation in measures which in general showed better results, like Special support for apprenticeship and Supported employment. Conclusions This comprehensive and extensive research has shown some very important results which might significantly alter the decision process on selection of persons to participate in ALMP. Firstly we need to draw attention to the fact that there is significant difference in results of two different outcomes, where looking at the number of days person taking part in ALMP has been employed in the period of 6 months after exit from treatment gives much more precise results. At second we have shown that matching process results in very high level of similarity of the experimental and control group members with nearly 97% marching of 5 observed variables. Further on we have seen that there are substantial differences in achieved effectiveness among persons taking part in different groups of measures. Especially poor results, with control group performing much better are shown for ALMP groups 4: Employment incentives and 7: Startup incentives, as long as all types of Training show extremely positive results. Finally we have shown that women and youth are playing important role in the active policies, and that the results they achieve are better that average, mostly due to a fact that they are not massively taking part in treatments that in general show weak results. Finally, it is important to note that finding from this research in some extent coincide with the findings of the researches conducted in a different environment, i.e. in more developed countries and by using different methodologies. For example Kuttim et al (2011) note that the best way of developing human capital is... through combination of education and experience. This opens a completely new field to perform comparative study in order to verify our findings. It is also important to note that results from this research need to be combined with the results presented in paper written by Zubović and Simeunović (2012) in order to gain precise information on the cost-benefit effectiveness of the ALMP performed in Serbia in the period of deep recession.

12 References 1. Barkin S (1967) Meeting the demands of an active manpower policy with the assistance of the academic disciplines, De Economist, Vol. 115, Issue 6, Springer Netherlands 2. Betcherman, G., Olivas, K., Dar, A. (2004), Impact of Active Labour Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries, Social Protection Discussion Paper Series 0402, the World Bank, January Bonin, H., Rinne, U. (2006), Evaluation of the Active Labour Market Program Beautiful Serbia, IZA Discussion Paper Calmfors, L. (1994), Active Labour Market Policy and Unemployment: a Framework for the Analysis of Crucial Design Features, OECD Economic Studies 22, Summer Dar A, Tzannatos Z (1999) Active labour market programs: A review of the evidence from Evaluations, Working paper, World Bank, (accessed on ) _854C0.pdf 6. Daguerre, A., Etherington, D. (2009), Active Labour Market Policies in International Context: What Works Best? Lessons for the UK, Department for Work and Pensions, Working Paper EUNES (2011) Evaluacija neto efekata aktivnih mera tržišta rada, EUNES IPA Project 2008, Belgrade, available on (assessed on ) 8. Eurostat (2012) Eurostat on-line database: (accessed ) 9. Fay, R. (1996): Enhancing the Effectiveness of Active Labour Market Policies: Evidence from Programme Evaluations in OECD Countries, Labour Market and Social Policy Occasional Papers, OECD. 10. Garson D (2010) Research Designs, available on assessed on Harrell, A., Burt, M., Hatry, H., Rossman, S., Roth, J., Sabol, W. (1996). Evaluation strategies for human services programs: A guide for policymakers and providers. Washington, DC: The Urban Institute. 12. Estevão M (2003) Do Active Labour Market Policies Increase Employment?, International Monetary Fund (IMF) Working Paper 03/234. Washington, DC: IMF. Available at: (accessed ) 13. European Commission (2006), Labour Market Policy Database: Methodology, Revision of June 2006, Office for Official Publications of the European Communities, Luxembourg. (accessed ) 14. Lehmann, H., Kluve, J. (2010), Assessing Active Labour Market Policies in Transitional Economies in The Labour Market Impact of the EU Enlargement, Caroleo, F.E., Pastore, F. (eds.), Springer- Verlag Berlin Heidelberg, pp Kahn R.F. (1931) The Relation of Home Investment to Unemployment, Economic Journal, June Keynes J.M. (1936) The General Theory of Employment, Interest and Money Macmillan Cambridge University Press, for Royal Economic Society in 1936; available at: (accessed ) 17. de Koning J, Peers Y (2007) Evaluating ALMP evaluations, SEOR Working Paper No. 2007/2, Roterdam 18. Kuttim M, Venesaar U, Kolbre E (2011) Enterpreneurs' Human Capital in Creative Industries: A Case of Baltic Sea Region Countries, Actual Problems of Economics No 12(126), 2011, pg

13 19. OECD (1964), Recommendation of the Council on Manpower Policy as a Means for the Promotion of Economic Growth, OECD, Paris. 20. OECD (1990) Labour Market Policies for the 1990s, OECD, Paris. 21. OECD (1993) Active Labour Market Policies: Assessing Macroeconomic and Microeconomic Effects, chapter 2, pp , in Employment Outlook, OECD: Paris. 22. Ognjenović, K. (2007), Using Propensity Score-Matching Methods in Evaluation of Active Labour Market Programs in Serbia, Economic Annals 52 (172), pp , Faculty of Economics, Belgrade. 23. RRPP (2012) Macroeconomic Analysis and Empirical Evaluation of Active Labour Market Policies in Serbia, project supported by RRPP, available on westernbalkans.net/en/research/current- Projects/Serbia/mainColumnParagraphs/05/text_files/file/RRPP_12_Macroeconomic_Analysis.p df 24. Spevacek AM (2009) Effectiveness of Active Labor Market Programs: A review of Programs in CEE and CIS, USAID konowledge service center 25. World Bank (2006), Republic of Serbia: Assessment of Labour Market, Report No YU, Washington D.C. 26. Zubović J, Subić J (2011) Reviewing Development of ALMP and the Evaluation Techniques in ed. Andrei et al. The Role of LAbour Markets and Human Capital in the Unstable Environment, Karta Graphic, Ploiesti, Romania 27. Zubović J, Simeunović I (2012) On the New Methodology in a Cost-Benefit Analysis of ALMP The Case of Serbia in ed. Zubović and Domazet New Challenges in Changing Labour Markets, Institute of Economic Sciences, Belgrade