Task Force on quality of business survey data

Transcription

1 Fnal Report V Task Force on qualty of busness survey data Task force 5: wegtng approaces

2 . Introducton Busness tendency surveys (BS) dffer from most busness surveys by ter early release, te qualtatve nature of ter questons and te orgnalty of te derved ndcators (balances of opnons, ndcators of confdence ). Ter specfcty s also attaced to ter non-ntrnsc utlty; te nterest of busness tendency surveys s based on ter ablty to forecast macroeconomc evolutons ence te queston of ow responses of te ndvdual frms sould be quantfed and combned. A debate on te best way to aggregate ndvdual answers as occurred from te begnnng of 5 s. A consensus seemed to be found n te 7 s by economc forecasters; balances of opnons were ten consdered as te most effcent way to quantfy and combne responses of ndvdual frms. However, te ssue ow to wegt te results remans an open queston. Te most common way to wegt ndvdual BS answers s to take nto account bot te relatve sze of te frm (wegtng at frm level) and te economc mportance of ts sector (wegtng at stratum level). Te queston s ow to coose te most effcent varables at frm and stratum levels. Te queston s also wat effcent means. Te queston of te real mpact of wegtng scemes may also be asked. Te followng statement can be found n te OECD andbook on busness tendency surveys: Furtermore, practcal experence as sown tat te balances are not very senstve to te coce of wegtng varables. In practce t s suffcent to use a sngle varable reflectng te general economc mportance of te enterprse n wegtng all te survey answers. However, te small sze of samples and potentally evolvng eterogenety among frms suggest tat ts statement needs to be revsted.. Wy and ow to wegt ndvdual answers? Two knds of wegtng scemes can be used n processng te answers of busness tendency surveys: sample wegts and sze wegts. In some cases, results are processed wtout wegts... Sample wegts Most BS samples are stratfed samples. Processng results by usng sample wegts means tat balances of opnons (B), calculated as te dfference between percentages of respondents gvng favourable (P + ) and unfavourable answers (P - ) are estmated by te classc estmator of stratfed samplng. Let s consder tat te BS answers are assocated wt an auxlary varable : = f te answer s + ncreased denoted + = f te answer s = remaned uncanged denoted = = - f te answer s - decreased denoted - B P P ( ) B s te average of te varable. By usng te classc estmator of an average for a stratfed sample, te estmator of B (denoted Bˆ ) s as follows. Bˆ H y H ( p p ) were s te reference ndex of stratum, te number of entreprses n stratum of te sample frame and n te number of reportng unts n stratum of te sample. All enterprses surveyed n stratum ave terefore te same wegt w. n s te sample wegt of stratum. n Busness Tendency Surveys: A Handbook (3), page 37, 5

3 However, ts way of wegtng ndvdual answers s generally speakng not used for busness tendency surveys. Indeed, t s dffcult to tnk tat attrbutng te same wegt to eac frm watever ts sze s te best coce for lnkng BS to macroeconomc data. Sze wegts are terefore generally preferred to sample wegts n processng BS answers because te mportance of te answers s expected to depend on te sze of te reportng unts... Sze wegts Te dea of usng sze wegts at frm level s tat answers from a large frm carry more nformaton on economc actvty tan answers from a small one. If S refers to a frm sze crteron (turnover, employment, ), te estmator of balances of opnons obtaned n a stratum s ten defned as follows: Bˆ n s y s y s s s s were n s s s + (resp. s - ) s te total sze of enterprses tat declare + ncreased n stratum (resp. - decreased ) In a second tme, results at stratum level need to be wegted n order to correct te nequalty of sample rates n eac stratum and ten to reflect te structure of te economy. Te so-called Balance of opnons s ten obtaned as a wegted average of te above estmator by stratum. were V and V H V Bˆ employment of te consdered economc sectors. H V Bˆ V are non-random values. In general, V refers to value added, turnover or Ts estmator s qute dfferent from te classc estmator used for stratfed samples. It s not based on sample wegts but on wegts calculated from a varable lnked to frms actvty (turnover, employment ). Enterprses surveyed n stratum ave terefore an ndvdual wegt w dependng on ter sze: 3. Metadata and qualty ndcators V s w V. s 3.. Wegtng procedures used n te Member States Te answers of BS partners to te metadata questonnare sent by DG-ECFI llustrate te dversty of metods mplemented for wegts at frm level as well as for wegts at stratum level (see tables below). Tey confrm te fact tat sample wegts are generally not used: few BS partners don t wegt te answers. Turnover or employment data are te man varables used to aggregate ndvdual answers.. At stratum level, value added s preferred to turnover. Te majorty of BS partners use a sngle varable to wegt all te questons.

4 Industry: On frm level On stratum level Wegt umber of umber of Wegt reples reples Turnover (only) 4 Turnover (only) 3 Employment (only) 8 Employment (only) 5 Value added (only) 3 Value added (only) 8 Turnover or employment 4 Turnover or employment o wegt (due to samplng metod Sold producton Servces: o wegt Output, employment or exports o answer or no detal 4 Employment or exports o wegt o answer 5 On frm level On stratum level Wegt umber of umber of Wegt reples reples Turnover (only) 6 Turnover (only) 3 Employment (only) 9 Employment (only) 4 Value added (only) Value added (only) 8 Turnover or employment 5 Turnover or employment o wegt (due to samplng metod Sold producton Constructon: o wegt o wegt o answer or no detal o answer or no detal 7 On frm level On stratum level Wegt umber of umber of Wegt reples reples Turnover (only) 3 Turnover (only) 4 Employment (only) 3 Employment (only) 6 Value added (only) Value added (only) 5 Turnover or employment 3 Turnover or employment o wegt (due to samplng metod Sold producton Retal trade: o wegt o wegt o answer or no detal o answer or no detal 6 On frm level On stratum level Wegt umber of umber of Wegt reples reples Turnover (only) 8 Turnover (only) 5 Employment (only) 6 Employment (only) 4 Value added (only) Value added (only) 4 Turnover or employment 5 Turnover or employment Gross proft margn Sold producton o wegt (due to samplng metod o wegt o wegt o answer or no detal 8 o answer or no detal 3

5 Te coce of varables used to wegt ndvdual answers s lmted to te nformaton avalable for eac reportng unt. In contrast, a wder range of nformaton s avalable at stratum level, comng from eter te samplng frame or from an external source suc as natonal accounts for example. Watever te nformaton source, te coce of te varable may ave an mpact on te results; te wegt of economc sectors usually dffers n terms of producton, value added or employment. Te lower te level of dssemnaton wll be, te more mportant te mpact of te wegtng varable. To llustrate ts pont, let s take an example comng from Frenc ndustry survey. In Frenc ndustry survey, stratum level corresponds to te tree-dgt level of ACE. Wen calculatng seres at manufacturng ndustry level, balances calculated at stratum level are wegted by te wegt of eac sub-sectors n manufacturng ndustry (see table ). Te wegt of sub-sector 3.3 wll vary to 3 to 7, dependng on te wegt used: value added or turnover. Table : Sare of value added and producton of Manufacture of oter transport equpment sub-sectors n manufacturng ndustry, n France ACE Value added Producton 3. Buldng of sps and boats 3. Manufacture of ralway locomotves and rollng stock 3.3 Manufacture of ar and spacecraft and related macnery Manufacture of transport equpment n.e.c. Manufacturng ndustry Source: Insee, atonal Accounts Wen calculatng seres at Manufacture of oter transport equpment level, balances calculated at stratum level are wegted by te wegt of eac sub-sectors n manufacture of oter transport equpment (see table ). Te wegt of sub-sector 3.3 wll ten vary to 74 to 86, dependng on te wegt used: value added or turnover. Table : Sare of value added and producton of Manufacture of oter transport equpment sub-sectors n manufacture of oter transport equpment, n France ACE Value added Producton 3. Buldng of sps and boats 6 3. Manufacture of ralway locomotves and rollng stock Manufacture of ar and spacecraft and related macnery Manufacture of transport equpment n.e.c. 3 Manufacture of oter transport equpment Source: Insee, atonal Accounts For te Frenc ndustry survey, te coce as been made to wegt at stratum level balances related to actvty by producton. Balances related to employment are wegted by employment. 3.. Qualty ndcators DG-ECFI as calculated two qualty ndcators for eac confdence ndcator (COF): correlatons between COF and reference seres and monts for cyclcal domnance (MCD). Remnder: calculaton of COF ndcators IDU SERV BUIL RETA Balances taken nto account n te COF Q - Assessment of order-book levels Q4 -Assessment of stocks of fnsed products Q5 - Producton expectatons for te monts aead Q - Busness stuaton development over te past 3 monts Q - Evoluton of te demand over te past 3 monts Q3 - Expectaton of te demand over te next 3 monts Q3 - Evoluton of your current overall order books Q4 - Employment expectatons over te next 3 monts Q - Busness actvty (sales) development over te past 3 monts Q - Volume of stock currently old Q4 - Busness actvty expectatons over te next 3 monts COF formula (Q - Q4 + Q5) / 3 (Q + Q + Q3) / 3 (Q3 + Q4) / (Q - Q + Q4) / 3 4

6 Correlatons between COF and reference seres Because of ter early release, BS results are commonly used n economc forecasts. However, forecasters are requred to brdge te gap between te qualtatve nature of BS ndcators and te quanttatve nature of ard economc ndcators. Te econometrc tecnques used, called brdgemodels, are usually lnear, ence te mportance to fnd balances or composte ndcators wc are te most correlated to economc varatons. To track performances of BS ndcators, correlatons between COFs and references seres ave been calculated by DG-ECFI. Remnder: lst of references seres COF IDU SERV BUIL RETA Reference seres Producton n ndustry - Percentage cange compared to correspondng perod of te prevous year Gross Value added n wolesale and retal trade, transport, accommodaton and food servce actvtes, nformaton and communcaton, fnancal and nsurance actvtes Constructon producton ndex - Trend cycle Houseold and PISH fnal consumpton expendture Te table below sows tat correlaton coeffcents are larger tan.75 for a trd of countres. On te oter and, correlatons are smaller tan.5 for a trd of countres n retal trade survey. Tey are even negatve for one country n buldng survey. Correlaton coeffcent IDU SERV BUIL RETA [- ; [ [ ;.5[ 5 8 [.5 ;.75[ [.75; ] ote: - correlaton coeffcents of COF ndustry wt reference seres are between.5 and.75 - data are not avalable for all BS partners Tese results ave neverteless to be qualfed because of bot te specfc way to calculate COF ndcators and te coce of reference seres. For nstance, t s not obvous tat te ndustral producton sould be calculated on a year on year bass snce frms are supposed to assess te evoluton of ter producton over te last tree monts compared to te tree prevous ones and not compared to te same monts of last year. Besdes, te weaker correlaton observed for retal trade COF may not be a surprse for forecasters wo commonly notce tat brdge models are better suted to forecast some supply varables suc as ndustral output tan to forecast demand components suc as ouseolds fnal consumpton. To forecast fnal consumpton, beavoural equatons are usually preferred to brdge models. Monts for cyclcal domnance (MCD) Te MCD s a measure of sort-term volatlty n tme seres. It s defned as te sortest span of monts for wc te I/C rato s less tan unty. I s te average mont-on-mont cange (wtout regard to ts sgn) of te rregular component of te seres and C s te trend-cycle component of te seres. Te Monts for Cyclcal Domnance (MCD) s used to determne te mnmum number of monts before an mprovement/deteroraton n te tme seres can be nterpreted wt reasonable confdence as an mprovement/deteroraton n economc sentment. Te ger te MCD, te ger s te volatlty of tme seres. Te table below sows tat on a mont-to-mont bass, te average cange n te rregular factor s larger tan tat n te cyclcal factor n te majorty of countres. Over four-mont ntervals, t s larger n a mnorty of countres except for te retal trade survey, and for te buldng survey to a certan extent. ote: MCD IDU SERV BUIL RETA MCD= n COF ndustry seres 5

7 Once agan tese results ave to be qualfed because of bot te specfc way to calculate COF ndcators and te real am of reducng MCD. Indeed, f for some seres te measurement errors are already relatvely small t may not be possble to reduce MCD. MCD ndcates te number of monts at wc te average ampltude of te cycle-trend component wll overtake te rregular one. Bend rregular component, two knd of resdual erratc fluctuatons can be dstngused: te part of te rregular fluctuatons tat s due to measurement errors (response errors, samplng errors, processng errors, suc as non effcent wegt of responses, ) and te part tat s due to real fluctuatons (unusual weater, strkes, ). Te frst part of te rregular sould be consdered as nose by forecasters wereas te second part sould be consdered as real nformaton. Balancng te costs and gans of reducng MCD wll be possble only f te measurement errors are known. 4. Furter emprcal results Even f sze wegts are commonly used, tere s no teoretcal evdence on te most effcent varable to use at frm and stratum level. Ts paper does not ntend to gve te defnte soluton to te queston of wegtng BS results. More modestly we provde some experments on te mpact of a cange n wegtng scemes on tme seres n terms of correlatons wt reference seres and MCD. However, a smple vsual nspecton of tme seres may already lead to some mportant conclusons. More precsely, te rest of ts paper presents te results of some experments made on Frenc ndustry survey data. Frst, seres ave been recalculated wt no wegt at frm level (denoted _). Secondly, seres ave been recalculated wt no wegt neter at frm level nor stratum level (denoted ). As recalculated seres may not ave te same long term average, graps presentng tese tree seres are based on standardzed data (.e. [-average]/standard error). 4.. Impact of te wegtng sceme on qualty ndcators Te grap below sows te tree seres of COF ndustry for France lmted to te perod 5-. A quck vsual nspecton sows tat n average all tese curves vary n te same way. Ts s confrmed by correlaton coeffcents of eac COF wt te reference seres: tey are almost uncanged around.7. COF ndcator : Frenc Industry survey Standardzed data IDU IDU_ IDU Seres IDU_ and IDU seem to smoot te orgnal seres IDU. Ts s confrmed by te vsual nspecton of rregular components I (see graps below). If eac curve glgts te end of 6

8 8 perod as an rregular one, I seems to ave softened all te rregular fluctuatons even f tey are due to real fluctuatons. Irregular components of te tree COF ndustry seres I I_ I Te results of calculaton of MCD for eac seres confrm tat MCD of IDU s lower tan MCD of IDU (see table 3). MCD of IDU and IDU_ s equal to. However, te rato of te average ampltude of te rregular component to te cycle-trend one s lower for IDU_. Table 3: Average ampltudes of rregular and cyclcal components for to monts span of monts span Wegt at frm and stratum levels (IDU) MCD= o wegt at frm level (IDU_) MCD= o wegt neter at frm level nor stratum level (IDU ) MCD= C I I C C I I C C I I C I : average percentage cange (wtout regard to sgn) n rregular component C : average percentage cange (wtout regard to sgn) n cycle-trend component 4.. Vsual nspecton of tme seres Te consequences of a cange n wegtng sceme on qualty ndcators needs a qualfcaton. A vsual nspecton of tme seres s an oter way to assess te relatve performance of wegted or unwegted seres. Let s take te example of ndustry survey queston Q5: Producton expectatons for te monts aead and let s pont out two perods: end 8-begnnng 9 and end 9-begnnng (see grap below). Perod : end 8-begnnng 9 One can observe tat te unwegted curves (Q5_ and Q5 ) were muc below ter long term average (correspondng to zero on te grap) tan te wegted one Q5. Ts could gve an advantage to unwegted seres, as t s well known tat busness tendency surveys dd not fully measure te ntensty of 8 crss. However, we can also notce tat te wegted seres restarted to ncrease before te unwegted ones, wc s more n lne wt te evoluton of te ndustral producton. Tat tends to sow tat bggest enterprses may ave antcpated a recovery before te smallest ones. Ts last result gves a notable advantage to wegted seres. 7

9 Perod : end 9-begnnng Te wegted seres reaced ts long-term average n Q4 9 before stablsng. At te same tme, one could observe te same penomena n te ndustral producton seres. Once agan, te wegted seres seem to be more sutable to detect te evoluton of producton n ndustry. Q5 : Producton expectatons for te monts aead (advanced for 3 monts) janv-5 janv-6 janv-7 janv-8 janv-9 janv- janv- janv- Q5 Q5_ Q5 Ref_sere standardzed data Ts paper never ntended to gve te defnte soluton to te queston of wegtng BS results. But t elped to sow tat te mpact of dfferent wegtng scemes on tme seres s not neglgble. From a teoretcal pont of vew, t would ave been nterestng to develop metods tat am at determnng te most effcent wegts n order to optmze te forecast of te macroeconomc evolutons. However, we ave to keep n mnd tat we need to fnd wegtng scemes easy to mplement and above all, tat we ave to guarantee te avalablty of long tme seres, wc could not be te case f new wegtng scemes ad to be mplemented. 5. References DG ECFI manual (7), te jont armonsed EU Programme of busness and consumer surveys OECD manual (3), Busness Tendency Surveys: A Handbook Bau O., Ferrar. (6), Téore de l opnon, faut-l pondérer les réponses ndvduelles? Fansten M. (976), Introducton à une téore matématque de l opnon. Mtcell J., Smt RJ., Weale MR. (), Effcent aggregaton of panel qualtatve survey data. Sskn J. (96), How accurate? 8