OECD-JRC Second workshop on Composite Indicators of Country Performance Paris, February 26-27, 2003 Using composites for policy-making Three ongoing works in the Commission Stefano Tarantola (JRC) Applied Statistics Group http://www.jrc.cec.eu.int/uasa 1
Summary Ongoing works of the Commission on composite indicators: I. The CI of the knowledge-based economy developed by DG RTD II. the Internal Market Index developed by DG MARKT III. the e-business Readiness CI developed by DG ENTR http://www.jrc.cec.eu.int/uasa 2
Phases to build composite indicators stakeholder involvement 1a. decide on the phenomenon being measured 1b. draw an analytical conceptual model 2. select sub-indicators 3. investigate statistical properties of sub-indicators 4. check data availability (devise imputation strategies) 5. set-up normalization methods 6. set-up weighting methods 7. assess robustness of the composite indicator 8. investigate presentational issues (related uncertainties) http://www.jrc.cec.eu.int/uasa 3
Robustness assessment Space of alternatives scenarios conceptual models 20 10 0 EU US JP visualisation imputation normalisation weighting 20 10 0 20 EU US JP 10 0 EU US JP http://www.jrc.cec.eu.int/uasa 4
CI of the knowledge-based economy KEY FIGURES 2003-2004 By DG RTD Key Figures provides a set of indicators to take stock of Europe s position in Science, Technology and knowledge economy. Two composite indicators of the knowledge-based economy (investment and performance) are included. http://www.jrc.cec.eu.int/uasa 5
CI of the knowledge-based economy 1a. decide on the phenomenon being measured The composites aim to capture country progress in the transition towards the knowledge-based economy, by focusing on the knowledge dimension of the Lisbon agenda. 1b. draw an analytical conceptual model Dimensions of investment: creation and diffusion of new knowledge Elements of performance: productivity, scientific and technological performance, use of the information infrastructure, effectiveness of the education system http://www.jrc.cec.eu.int/uasa 6
CI of the knowledge-based economy 2. select sub-indicators Investment: R & D efforts (GERD), investment in highly skilled human capital (S&T PhD, RSE), capacity and quality of education system (education spending, LLL), purchase of new capital equipment that can contain high tech (GFCF), modernisation of public services (e-government) Performance: labour productivity (GDP/cap), EU and US patents, scientific publications, e-commerce, schooling success rate. key indicators reviewed and assessed by experts Member States + Adhesion countries + US + JP coverage 1995 2001. http://www.jrc.cec.eu.int/uasa 7
CI of the knowledge-based economy 4. check data availability (devise imputation strategies) Total new Science and Technology PhDs (RTD benchmarking exercise) 1995 1996 1997 1998 1999 2000 2001 pays v i 1 v i2 v i3 v i4 v i5 v i6 v i7 B 666 686 679 667 650 681 752 DK 391 337 428 378 392 393 373 D 8607 8868 9339 9571 9557 9820 9164 EL 198 225 217 275 300 319 345 E 1819 2071 2205 2241 2162 2361 2350 F 6601 7413 7333 6981 6477 6545 6047 IRL 207 283 290 275 290 282 309 I 1605 1561 1561 1438 1476 1629 1555 L NL 942 995 931 933 884 842 920 A 630 701 805 746 734 752 805 P 291 341 325 360 379 424 463 FIN 442 398 432 687 656 720 666 S 1175 1319 1398 1445 1537 1565 1693 UK 5409 5495 5580 5845 5807 6111 6179 EU-14 28984 30693 31523 31842 31301 32444 31621 ~ 5% data missing Linear regression where applicable When it is not possible imputing http://www.jrc.cec.eu.int/uasa 8
CI of the knowledge-based economy 4. check data availability (cont.) Construct provisional composite indicators countries indicators Remove the country from list yet all sub - indicators Remove the indicator from list yet all countries Comparability across countries is guaranteed Missing value Three versions of composite http://www.jrc.cec.eu.int/uasa 9
CI of the knowledge-based economy 8. investigate presentational issues 1. Full composite for EU 14 (EU15 Lux) 2000 1995-2000 2001 2000-2001 B 0.87 3.5% 1.24 3.5% DK 3.08 5.5% 3.37 1.3% D 0.76 3.3% 0.75-2.1% EL -1.31 11.4% -1.15 2.6% E -0.21 6.4% -0.06 0.7% F 1.03 2.8% 1.17 0.4% IRL 0.67 8.5% 0.93 3.1% I 0.07 4.0% 0.24 0.2% NL 0.95 4.3% 1.22 2.6% A 1.27 3.8% 1.50 1.7% P -0.24 8.4% -0.12 1.4% FIN 2.23 6.4% 2.42 2.3% S 3.06 4.4% 3.67 5.9% UK 0.78 3.0% 0.94 1.3% EU-14 0.64 4.1% 0.78 0.6% GERD/cap 2/24 PhD/cap 4/24 RSE/cap 2/24 GFCF/cap 3/24 E-gov 3/24 Educat 7/24 LLL 3/24 http://www.jrc.cec.eu.int/uasa 10
CI of the knowledge-based economy 8. investigate presentational issues (cont.) 2001 2000-2001 B 1.47 3.5% DK 3.62 1.3% D 0.80-2.1% EL -1.37 2.6% E -0.21 0.7% F 1.17 0.4% IRL 0.70 3.1% I 0.28 0.2% NL 1.40 2.6% A 1.65 1.7% P -0.22 1.4% FIN 2.39 2.3% S 3.72 5.9% UK 0.95 1.3% EU-14 0.80 0.6% CY -0.91 3.6% CZ -0.91 4.6% EE -1.21 6.9% HU -1.36 4.0% LT -1.91 2.3% LV -1.94 8.5% 2. Partial composite to compare EU 14 (EU15 Lux) and some Accession Countries E-government and LLL excluded GERD/cap 2/24 PhD/cap 4/24 RSE/cap 2/24 GFCF/cap 7/24 E-gov 0 Educat 9/24 LLL 0 PL -1.50 9.8% SK -1.15 9.3% http://www.jrc.cec.eu.int/uasa 11 EU-22 0.22 0.9%
CI of the knowledge-based economy 8. investigate presentational issues (cont.) 3. Reduced composite to compare EU with US, JP 1999 1995-1999 2001 1995-2001 D 1.18 4% 1.26 3% E -0.37 7% -0.19 6% F 1.20 3% 1.40 2% I -0.10 3% 0.14 3% UK 0.68 3% 0.72 2% EU-14 0.66 4% 0.83 3% US 2.70 5% 2.59 3% JP 1.81 1% 1.80 1% e-government, Education expenditure, and LLL excluded (no data for US and JP) GERD/cap 2/12 PhD/cap 4/12 RSE/cap 2/12 GFCF/cap 4/12 E-gov 0 Educat 0 LLL 0 http://www.jrc.cec.eu.int/uasa 12
investigate presentational issues (cont.) 1. Full composite for EU 14 (EU15 Lux) Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 2000 and annual growth rate 1995-2000 (empty symbol). Relative country positions in 2001 and annual growth rate 2000-2001 (full symbol). Annual growth rate (1995-2000, empty) Annual growth rate (2000-2001, full) 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% 2% 1% 0% -1% -2% -3% EL P IRL E FIN DK S EU-14 I NL A S B D B EL F UK IRL NL FIN P A DK E EU-14 UK I F D -2-1 0 1 2 3 4 Investment level in 2000 (empty); Investment level in 2001(full) http://www.jrc.cec.eu.int/uasa 13
investigate presentational issues (cont.) 2. Partial composite to compare EU 14 (EU15 Lux) and some Accession Countries Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 2001 and annual growth rate 2000-2001. 12% Annual growth rate (2000-2001 10% 8% 6% 4% 2% 0% -2% LV LT PL HU EL SK EE CZ CY E P EU-22 I EU-14-3.0-2.0-1.0 0.0 1.0 2.0 3.0 4.0 IRL D UK NL F B A FIN S DK -4% Investment level in 2001 http://www.jrc.cec.eu.int/uasa 14
investigate presentational issues (cont.) 3. Reduced composite to compare EU with US, JP Composite indicator of investment in the transition to a knowledge-based economy. Relative country positions in 1999 and annual growth rate 1995-1999 (empty symbol). Relative country positions in 2001 and annual growth rate 1995-2001 (full symbol). 8.0% Annual growth rate (1995-1999, empty) Annual growth rate (1995-2001, full) 7.0% 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% E I E I EU-14 EU-14 UK UK F D D F JP JP US US 0.0% -1.0-0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Investment level in 1999 (empty); Investment level in 2001(full) http://www.jrc.cec.eu.int/uasa 15
CI of the knowledge-based economy 7. assess robustness of the composite indicator Composite indicator on investment in a knowledge-based economy Relative country positions in 2001 and growth rate 2000-2001 10% 8% Uncertainties coming from imputation (linear regression) Annual growth rate 2000-2001 6% 4% 2% 0% EL P E I UK EU-14 IRL NL F B A FIN DK S -2% -4% D -2-1 0 1 2 3 4 Relative country position in 2001 Ranking not clear http://www.jrc.cec.eu.int/uasa 16
The Internal Market Index 1a. decide on the phenomenon being measured Objective: provide some measure of the effects of the Internal Marke policies in terms of MS progress in the functioning of the Internal Marke 1b. draw an analytical conceptual model Consider Internal Market policies in terms of free circulation of goods, services, capital and workers within the EU. Model will be revisited (workshop in April) to align the Index more closely with the realities of the IM. 2. select sub-indicators TWELVE sub-indicators. SIX of them are Commission s Structural Indicators in the domain Market Reform. http://www.jrc.cec.eu.int/uasa 17
The Internal Market Index 2. select sub-indicators Intra EU Foreign Direct Investment /GDP Decreased greatly after technology bubble burst. No relation with internal market policy. Need to revise Intra-EU FDI sub-indicator FDI stocks (rather than FDI flows) seem to improve the significance of the sub-indicator to measure progress towards Internal Market (consultation with experts). http://www.jrc.cec.eu.int/uasa 18
The Internal Market Index Macro di Roman http://www.jrc.cec.eu.int/uasa 19
The Internal Market Index 5. set-up normalization methods Thanks Ralf. http://www.jrc.cec.eu.int/uasa 20
The Internal Market Index 6. set-up weighting methods The relative importance of the 12 sub-indicators was decided by canvassing the members of the Internal Market Advisory Committee (IMAC), the group of Member State officials who advise the Commission on Internal Market matters. used) (budget allocation was http://www.jrc.cec.eu.int/uasa 21
Budget Allocation Expert... allocate 100 points An example A1: % firms using Internet A2: % firms having www A3: % of firms with 2 security facilities A4: % employees using PC A5: % firms with broadband conne. Expert 1 Expert 2 Expert 3 15 15 10 30 30 25 25 20 15 15............... http://www.jrc.cec.eu.int/uasa 22
The Internal Market Index 6. set-up weighting methods http://www.jrc.cec.eu.int/uasa 23
The Internal Market Index 8. investigate presentational issues 1992 = 100: to measure progress since 1992 (ten years market without frontiers) http://www.jrc.cec.eu.int/uasa 24
The e-business readiness composite 1a. decide on the phenomenon being measured The composite indicator aims to monitor country progress in the implementation of the e-europe 2005 Action Plan... Which calls for a benchmarking of the target that by 2005, Europe should have ( ) a dynamic e-business environment, and specifying that e-business comprises both selling and buying on-line and restructuring of the business processes to make best use of digital technologies. http://www.jrc.cec.eu.int/uasa 25
List of indicators approved by the Council Adoption of ICT a1 - % of firms that use Internet a2 - % of firms that have web/home page a3 - % of firms using 2 security facilities a4 - % of employees using computer a5 - % of firms with broadband connection a6 - % of firms with LAN Use of ICT b1 - % of firms purchasing products/serv. via Internet b2 - % of firms receiving orders via Internet b3 - % of firms with IT linked with other internal IT b4 - % of firms with IT linked with external IT b5 - % of firms using Internet for banking b6 - % of firms selling products via Internet http://www.jrc.cec.eu.int/uasa 26
2. select sub-indicators 4. imputation strategies Enterprise surveys in 2001, 2002, 2003 (4 missing values) Adoption a1 a2 a3 a4 a5 a6 BE 0.91 0.62 0.89 0.59 0.49 0.38 DK 0.97 0.75 0.92 0.64 0.69 0.31 DE 0.95 0.71 0.89 0.37 0.42 0.22 ES 0.82 0.33 0.35 0.44 0.51 0.26 IE 0.86 0.59 0.45 0.46 0.19 0.26 IT 0.83 0.47 0.83 0.42 0.31 0.17 LU 0.85 0.58 0.81 0.57 0.39 0.30 NL 0.86 0.61 0.83 0.65 0.37 0.38 AT 0.89 0.66 0.86 0.50 0.48 0.27 FI 0.97 0.70 0.94 0.66 0.65 0.33 SE 0.95 0.80 0.92 0.64 0.62 0.38 IS 0.97 0.68 0.92 0.54 0.20 0.30 NO 0.88 0.63 0.85 0.57 0.47 0.24 PT 0.70 0.25 0.64 0.32 0.27 0.14 EU 0.86 0.57 0.69 0.42 0.39 0.24 http://www.jrc.cec.eu.int/uasa 27
2. select sub-indicators 4. imputation strategies Imputation via regression analysis Use b1 b2 b3 b4 b5 b6 BE 0.22 0.20 0.46 0.12 0.69 0.02 DK 0.23 0.18 0.35 0.12 0.79 0.02 DE 0.11 0.09 0.30 0.17 0.66 0.01 ES 0.03 0.02 0.36 0.08 0.67 0.00 IE 0.24 0.14 0.32 0.17 0.60 0.01 IT 0.04 0.03 0.12 0.07 0.48 0.00 LU 0.17 0.13 0.40 0.15 0.49 0.01 NL 0.31 0.27 0.60 0.17 0.70 0.04 AT 0.21 0.12 0.34 0.13 0.70 0.01 FI 0.16 0.18 0.63 0.17 0.82 0.03 SE 0.23 0.13 0.23 0.06 0.77 0.03 IS 0.15 0.12 0.31 0.09 0.92 0.01 NO 0.21 0.13 0.31 0.10 0.67 0.02 PT 0.09 0.03 0.21 0.14 0.53 0.01 EU 0.11 0.08 0.29 0.10 0.62 0.01 http://www.jrc.cec.eu.int/uasa 28
The e-business readiness composite Adoption of ICT - 2003 EU BE 1.00 0.90 DK 0.80 PT 0.70 0.60 DE 0.50 NO 0.40 0.30 ES a1 a2 0.20 IS 0.10 0.00 IE a3 a5 a4 a6 SE IT FI LU AT NL http://www.jrc.cec.eu.int/uasa 29
The e-business readiness composite Use of ICT - 2003 EU BE 1.00 0.90 DK 0.80 PT 0.70 0.60 DE 0.50 NO 0.40 0.30 0.20 ES b1 b2 IS 0.10 0.00 IE b3 b4 b5 b6 SE IT FI LU AT NL http://www.jrc.cec.eu.int/uasa 30
3. investigate statistical properties of sub-indicators No changes in correlation before and after imputation a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 Adoption of ICT a2 0.92 a3 0.63 0.71 a4 0.61 0.66 0.51 a5 0.47 0.44 0.36 0.57 a6 0.60 0.63 0.36 0.87 0.44 Use of ICT b1 0.38 0.62 0.35 0.70 0.17 0.71 b2 0.48 0.61 0.44 0.79 0.26 0.80 0.89 b3 0.30 0.26 0.15 0.63 0.33 0.67 0.48 0.76 b4-0.04 0.07-0.02-0.01-0.11 0.07 0.33 0.46 0.60 b5 0.76 0.62 0.41 0.58 0.37 0.59 0.36 0.44 0.39-0.12 b6 0.31 0.47 0.42 0.72 0.43 0.63 0.78 0.82 0.63 0.34 0.43 http://www.jrc.cec.eu.int/uasa 31
6,7. set-up normalization and weighting methods Data are in percentages: No scaling required Equal weights (simplest approach) Based on statistical analysis (e.g., reduce weights to highly correlated indicators) Policy relevance (elicitation of experts via budget allocation) http://www.jrc.cec.eu.int/uasa 32
The e-business readiness composite Equal weights 0.40 Use of ICT 0.35 NL FI 0.30 BE DK IS 0.25 AT IE EU NO SE 0.20 ES LU DE 0.15 PT IT 0.10 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 Adoption of ICT http://www.jrc.cec.eu.int/uasa 33
The e-business readiness composite The panel of experts on: e-business Support (e-bsn) has participated to the budget allocation exercise. http://www.jrc.cec.eu.int/uasa 34
The e-business readiness composite Weights obtained from Member States representatives of the e-bsn. A D DK FIN F IRL a1 25 10 16.66 10 15 15 a2 15 10 16.66 10 15 10 a3 10 20 16.66 0 15 15 a4 15 15 16.66 0 10 20 a5 25 20 16.66 40 25 30 a6 10 25 16.66 40 20 10 b1 10 10 16.66 10 15 15 b2 15 15 16.66 10 15 15 b3 20 20 16.66 40 25 25 b4 15 25 16.66 40 25 30 b5 25 20 16.66 0 10 10 b6 15 10 16.66 0 10 5 http://www.jrc.cec.eu.int/uasa 35
The e-business readiness composite Weights obtained from Member States representatives of the e-bsn. I L NL P S E a1 20 30 10 20 5 35 a2 25 20 10 10 5 38 a3 10 5 5 10 15 1 a4 10 15 20 30 20 15 a5 15 10 25 20 25 5 a6 20 20 30 10 30 3 b1 20 25 25 20 15 16 b2 20 25 25 20 15 16 b3 15 15 25 20 10 16 b4 15 5 10 20 30 16 b5 10 15 5 10 5 20 b6 20 15 10 10 25 16 http://www.jrc.cec.eu.int/uasa 36
The e-business readiness composite Results from budget allocation SCORE EU BE DK DE ES IE IT NL AT PT FI SE IS NO CI (panel) 0.35 0.45 0.48 0.39 0.32 0.35 0.29 0.45 0.41 0.26 0.50 0.46 0.31 0.40 CI (equal) 0.37 0.47 0.50 0.39 0.32 0.36 0.31 0.47 0.43 0.27 0.52 0.48 0.34 0.42 RANKING EU BE DK DE ES IE IT NL AT PT FI SE IS NO panel 9 5 2 8 11 10 13 4 6 14 1 3 12 7 equal 9 5 2 8 12 10 13 4 6 14 1 3 11 7 http://www.jrc.cec.eu.int/uasa 37
The e-business readiness composite Uncertainty analysis on budget allocation http://www.jrc.cec.eu.int/uasa 38
The e-business readiness composite Sensitivity of country rankings to the omission of one sub - indicator at a time Sensitivity analysis shows whether the composite indicator is well balanced in its sub - indicators http://www.jrc.cec.eu.int/uasa 39
The e-business readiness composite Results of sensitivity analysis baseline a1=0 a2=0 a3=0 a4=0 a5=0 a6=0 b1=0 b2=0 b3=0 b4=0 b5=0 b6=0 FI FI FI FI FI FI FI FI FI DK FI FI FI DK DK DK DK DK NL DK DK DK FI DK DK DK SE SE NL SE SE DK SE SE SE SE SE NL NL NL NL SE NL NL SE NL NL NL NL NL SE SE BE BE BE BE BE IS BE BE BE BE BE BE BE IS IS IS IS IS BE IS IS IS IS IS AT IS AT AT AT AT AT AT AT AT AT AT AT NO AT NO NO NO NO DE NO NO NO NO NO NO LU NO DE LU LU DE NO DE DE DE DE DE DE IS DE LU DE DE LU LU LU LU LU LU LU LU DE LU EU EU EU IE EU IE EU EU EU EU EU IE IE IE IE IE EU IE EU IE IE IE IE IE EU EU ES ES ES ES ES IT IT ES ES IT ES IT ES IT IT IT IT IT ES ES IT IT ES IT ES IT PT PT PT PT PT PT PT PT PT PT PT PT PT http://www.jrc.cec.eu.int/uasa 40
Conclusions Considerable experience is being gained with these ongoing works on composite indicators These tests will make a useful contribution to the OECD/JRC handbook on useful practices for constructing composite indicators http://www.jrc.cec.eu.int/uasa 41