OECD-JRC Second workshop on Composite Indicators of Country Performance. Paris, February 26-27, Stefano Tarantola (JRC)

Transcription

1 OECD-JRC Second workshop on Composite Indicators of Country Performance Paris, February 26-27, 2003 Using composites for policy-making Three ongoing works in the Commission Stefano Tarantola (JRC) Applied Statistics Group 1

2 Summary Ongoing works of the Commission on composite indicators: I. The CI of the knowledge-based economy developed by DG RTD II. the Internal Market Index developed by DG MARKT III. the e-business Readiness CI developed by DG ENTR 2

3 Phases to build composite indicators stakeholder involvement 1a. decide on the phenomenon being measured 1b. draw an analytical conceptual model 2. select sub-indicators 3. investigate statistical properties of sub-indicators 4. check data availability (devise imputation strategies) 5. set-up normalization methods 6. set-up weighting methods 7. assess robustness of the composite indicator 8. investigate presentational issues (related uncertainties) 3

4 Robustness assessment Space of alternatives scenarios conceptual models EU US JP visualisation imputation normalisation weighting EU US JP 10 0 EU US JP 4

5 CI of the knowledge-based economy KEY FIGURES By DG RTD Key Figures provides a set of indicators to take stock of Europe s position in Science, Technology and knowledge economy. Two composite indicators of the knowledge-based economy (investment and performance) are included. 5

6 CI of the knowledge-based economy 1a. decide on the phenomenon being measured The composites aim to capture country progress in the transition towards the knowledge-based economy, by focusing on the knowledge dimension of the Lisbon agenda. 1b. draw an analytical conceptual model Dimensions of investment: creation and diffusion of new knowledge Elements of performance: productivity, scientific and technological performance, use of the information infrastructure, effectiveness of the education system 6

7 CI of the knowledge-based economy 2. select sub-indicators Investment: R & D efforts (GERD), investment in highly skilled human capital (S&T PhD, RSE), capacity and quality of education system (education spending, LLL), purchase of new capital equipment that can contain high tech (GFCF), modernisation of public services (e-government) Performance: labour productivity (GDP/cap), EU and US patents, scientific publications, e-commerce, schooling success rate. key indicators reviewed and assessed by experts Member States + Adhesion countries + US + JP coverage

8 CI of the knowledge-based economy 4. check data availability (devise imputation strategies) Total new Science and Technology PhDs (RTD benchmarking exercise) pays v i 1 v i2 v i3 v i4 v i5 v i6 v i7 B DK D EL E F IRL I L NL A P FIN S UK EU ~ 5% data missing Linear regression where applicable When it is not possible imputing 8

9 CI of the knowledge-based economy 4. check data availability (cont.) Construct provisional composite indicators countries indicators Remove the country from list yet all sub - indicators Remove the indicator from list yet all countries Comparability across countries is guaranteed Missing value Three versions of composite 9

10 CI of the knowledge-based economy 8. investigate presentational issues 1. Full composite for EU 14 (EU15 Lux) B % % DK % % D % % EL % % E % % F % % IRL % % I % % NL % % A % % P % % FIN % % S % % UK % % EU % % GERD/cap 2/24 PhD/cap 4/24 RSE/cap 2/24 GFCF/cap 3/24 E-gov 3/24 Educat 7/24 LLL 3/

11 CI of the knowledge-based economy 8. investigate presentational issues (cont.) B % DK % D % EL % E % F % IRL % I % NL % A % P % FIN % S % UK % EU % CY % CZ % EE % HU % LT % LV % 2. Partial composite to compare EU 14 (EU15 Lux) and some Accession Countries E-government and LLL excluded GERD/cap 2/24 PhD/cap 4/24 RSE/cap 2/24 GFCF/cap 7/24 E-gov 0 Educat 9/24 LLL 0 PL % SK % 11 EU %

12 CI of the knowledge-based economy 8. investigate presentational issues (cont.) 3. Reduced composite to compare EU with US, JP D % % E % % F % % I % % UK % % EU % % US % % JP % % e-government, Education expenditure, and LLL excluded (no data for US and JP) GERD/cap 2/12 PhD/cap 4/12 RSE/cap 2/12 GFCF/cap 4/12 E-gov 0 Educat 0 LLL

13 investigate presentational issues (cont.) 1. Full composite for EU 14 (EU15 Lux) Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 2000 and annual growth rate (empty symbol). Relative country positions in 2001 and annual growth rate (full symbol). Annual growth rate ( , empty) Annual growth rate ( , full) 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% 2% 1% 0% -1% -2% -3% EL P IRL E FIN DK S EU-14 I NL A S B D B EL F UK IRL NL FIN P A DK E EU-14 UK I F D Investment level in 2000 (empty); Investment level in 2001(full) 13

14 investigate presentational issues (cont.) 2. Partial composite to compare EU 14 (EU15 Lux) and some Accession Countries Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 2001 and annual growth rate % Annual growth rate ( % 8% 6% 4% 2% 0% -2% LV LT PL HU EL SK EE CZ CY E P EU-22 I EU IRL D UK NL F B A FIN S DK -4% Investment level in

15 investigate presentational issues (cont.) 3. Reduced composite to compare EU with US, JP Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 1999 and annual growth rate (empty symbol). Relative country positions in 2001 and annual growth rate (full symbol). 8.0% Annual growth rate ( , empty) Annual growth rate ( , full) 7.0% 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% E I E I EU-14 EU-14 UK UK F D D F JP JP US US 0.0% Investment level in 1999 (empty); Investment level in 2001(full) 15

16 CI of the knowledge-based economy 7. assess robustness of the composite indicator Composite indicator on investment in a knowledge-based economy Relative country positions in 2001 and growth rate % 8% Uncertainties coming from imputation (linear regression) Annual growth rate % 4% 2% 0% EL P E I UK EU-14 IRL NL F B A FIN DK S -2% -4% D Relative country position in 2001 Ranking not clear 16

17 The Internal Market Index 1a. decide on the phenomenon being measured Objective: provide some measure of the effects of the Internal Marke policies in terms of MS progress in the functioning of the Internal Marke 1b. draw an analytical conceptual model Consider Internal Market policies in terms of free circulation of goods, services, capital and workers within the EU. Model will be revisited (workshop in April) to align the Index more closely with the realities of the IM. 2. select sub-indicators TWELVE sub-indicators. SIX of them are Commission s Structural Indicators in the domain Market Reform. 17

18 The Internal Market Index 2. select sub-indicators Intra EU Foreign Direct Investment /GDP Decreased greatly after technology bubble burst. No relation with internal market policy. Need to revise Intra-EU FDI sub-indicator FDI stocks (rather than FDI flows) seem to improve the significance of the sub-indicator to measure progress towards Internal Market (consultation with experts). 18

19 The Internal Market Index Macro di Roman 19

20 The Internal Market Index 5. set-up normalization methods Thanks Ralf. 20

21 The Internal Market Index 6. set-up weighting methods The relative importance of the 12 sub-indicators was decided by canvassing the members of the Internal Market Advisory Committee (IMAC), the group of Member State officials who advise the Commission on Internal Market matters. used) (budget allocation was 21

22 Budget Allocation Expert... allocate 100 points An example A1: % firms using Internet A2: % firms having www A3: % of firms with 2 security facilities A4: % employees using PC A5: % firms with broadband conne. Expert 1 Expert 2 Expert

23 The Internal Market Index 6. set-up weighting methods 23

24 The Internal Market Index 8. investigate presentational issues 1992 = 100: to measure progress since 1992 (ten years market without frontiers) 24

25 The e-business readiness composite 1a. decide on the phenomenon being measured The composite indicator aims to monitor country progress in the implementation of the e-europe 2005 Action Plan... Which calls for a benchmarking of the target that by 2005, Europe should have ( ) a dynamic e-business environment, and specifying that e-business comprises both selling and buying on-line and restructuring of the business processes to make best use of digital technologies. 25

26 List of indicators approved by the Council Adoption of ICT a1 - % of firms that use Internet a2 - % of firms that have web/home page a3 - % of firms using 2 security facilities a4 - % of employees using computer a5 - % of firms with broadband connection a6 - % of firms with LAN Use of ICT b1 - % of firms purchasing products/serv. via Internet b2 - % of firms receiving orders via Internet b3 - % of firms with IT linked with other internal IT b4 - % of firms with IT linked with external IT b5 - % of firms using Internet for banking b6 - % of firms selling products via Internet 26

27 2. select sub-indicators 4. imputation strategies Enterprise surveys in 2001, 2002, 2003 (4 missing values) Adoption a1 a2 a3 a4 a5 a6 BE DK DE ES IE IT LU NL AT FI SE IS NO PT EU

28 2. select sub-indicators 4. imputation strategies Imputation via regression analysis Use b1 b2 b3 b4 b5 b6 BE DK DE ES IE IT LU NL AT FI SE IS NO PT EU

29 The e-business readiness composite Adoption of ICT EU BE DK 0.80 PT DE 0.50 NO ES a1 a IS IE a3 a5 a4 a6 SE IT FI LU AT NL 29

30 The e-business readiness composite Use of ICT EU BE DK 0.80 PT DE 0.50 NO ES b1 b2 IS IE b3 b4 b5 b6 SE IT FI LU AT NL 30

31 3. investigate statistical properties of sub-indicators No changes in correlation before and after imputation a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 Adoption of ICT a a a a a Use of ICT b b b b b b

32 6,7. set-up normalization and weighting methods Data are in percentages: No scaling required Equal weights (simplest approach) Based on statistical analysis (e.g., reduce weights to highly correlated indicators) Policy relevance (elicitation of experts via budget allocation) 32

33 The e-business readiness composite Equal weights 0.40 Use of ICT 0.35 NL FI 0.30 BE DK IS 0.25 AT IE EU NO SE 0.20 ES LU DE 0.15 PT IT Adoption of ICT 33

34 The e-business readiness composite The panel of experts on: e-business Support (e-bsn) has participated to the budget allocation exercise. 34

35 The e-business readiness composite Weights obtained from Member States representatives of the e-bsn. A D DK FIN F IRL a a a a a a b b b b b b

36 The e-business readiness composite Weights obtained from Member States representatives of the e-bsn. I L NL P S E a a a a a a b b b b b b

37 The e-business readiness composite Results from budget allocation SCORE EU BE DK DE ES IE IT NL AT PT FI SE IS NO CI (panel) CI (equal) RANKING EU BE DK DE ES IE IT NL AT PT FI SE IS NO panel equal

38 The e-business readiness composite Uncertainty analysis on budget allocation 38

39 The e-business readiness composite Sensitivity of country rankings to the omission of one sub - indicator at a time Sensitivity analysis shows whether the composite indicator is well balanced in its sub - indicators 39

40 The e-business readiness composite Results of sensitivity analysis baseline a1=0 a2=0 a3=0 a4=0 a5=0 a6=0 b1=0 b2=0 b3=0 b4=0 b5=0 b6=0 FI FI FI FI FI FI FI FI FI DK FI FI FI DK DK DK DK DK NL DK DK DK FI DK DK DK SE SE NL SE SE DK SE SE SE SE SE NL NL NL NL SE NL NL SE NL NL NL NL NL SE SE BE BE BE BE BE IS BE BE BE BE BE BE BE IS IS IS IS IS BE IS IS IS IS IS AT IS AT AT AT AT AT AT AT AT AT AT AT NO AT NO NO NO NO DE NO NO NO NO NO NO LU NO DE LU LU DE NO DE DE DE DE DE DE IS DE LU DE DE LU LU LU LU LU LU LU LU DE LU EU EU EU IE EU IE EU EU EU EU EU IE IE IE IE IE EU IE EU IE IE IE IE IE EU EU ES ES ES ES ES IT IT ES ES IT ES IT ES IT IT IT IT IT ES ES IT IT ES IT ES IT PT PT PT PT PT PT PT PT PT PT PT PT PT 40

41 Conclusions Considerable experience is being gained with these ongoing works on composite indicators These tests will make a useful contribution to the OECD/JRC handbook on useful practices for constructing composite indicators 41