Saisics Denmark, Prices and Consumpion Jonas Mikkelsen JOM@DST.dk Exending he Danish CPI wih scanner daa - A sepwise analysis Inroducion In 2011 Saisics Denmark (DST) go access o scanner daa from he larges Danish supermarke chains. The main focus has been on undersanding how scanner daa can be incorporaed ino he curren CPI producion sysems and processes. Scanner daa is differen from he informaion receive in he regular price collecion. This paper focuses on hese differences presening a sepwise analyical approach moving from radiional price collecion o a scanner daa based CPI. The firs par of his paper focuses on he change of price concep ha is ineviable when using scanner daa. In order o analyse he isolaed effec from his change he curren produc baske is idenified in he scanner daa. In his way he colleced shelf prices have been exchanged wih prices from scanner daa defined by weekly uni prices. The second par of his paper is concerning he definiion of he iem baske. The goal is o generae a represenaive and manageable baske ha can be used in he curren CPI producion sysem. Using he urnover daa makes i possible o obain a se of represenaive producs for each COICOP group. Bu high ariion raes of iems require effors when dealing wih missing prices when iems leave he scanner daa on discoun. Creaing a manageable baske is a challenge boh regarding he large amoun of prices and regarding he ariion of iems. In his paper i is analysed wheher a longer daa collecion period can limi he problems wih missing values. Secondly, limiing he amoun of prices in he baske is aemped by aggregaing sore specific prices o chain specific prices. Thirdly, he effec of reducing he number of producs in he baske is invesigaed. This is carried ou by limiing he baske only o include he op 3 besselling produc for each COICOP 8-digi caegory. The daa Since January 2011 Saisics Denmark (DST) has received scanner daa from he larges supermarke chains on a weekly basis accouning for approximaely 60% of he Danish sales of food and beverages. The produc ID In scanner daa each produc is idenified by a produc code eiher called European Aricle Numbering (EAN) or Produc Lookup Code (PLU). The EAN number is defined by he producer and normally has 13 digis while he PLU number is ofen defined by he supermarke chain and ofen has less han 13 digis. When working wih scanner daa he EAN/PLU 1 number is used as he produc idenificaion which is used o secure mached prices. Example of scanner daa from he supermarkes Dae Sore EAN number Turnover Volume Uni Quaniy per uni Produc number Produc descripion 1 7894 2920080800007 3402,70 211 Gram 300 976003 Sliced bacon 2x150 G. 1 7895 2920080800007 2119,65 163 Gram 300 976003 Sliced bacon 2x150 G. 1 78 2920080800007 1516,05 108 Gram 300 976003 Sliced bacon 2x150 G. 1 7897 2920080800007 1478,13 105 Gram 300 976003 Sliced bacon 2x150 G. 1 7214 2921056000005 302,50 14 Gram 200 956001 Chicken Fille 1 7215 2921056000005,50 5 Gram 200 956001 Chicken Fille 1 Hereafer EAN/ PLUs are referred o as EAN.
The dae field consiss of a 2 digi year number and a 2 digi week number. The sore number is unique for he specific supermarke sore. From his number i is also possible o idenify he supermarke chain i belongs o. The prices are derived from dividing he weekly urnover wih he weekly volume for each EAN number for each sore. The quaniy per uni is ofen defining he size of he produc in grams or kilograms. Unforunaely, i someimes holds he informaion e.g. 10 apples which is no a precise measure of he acual size of he produc. The produc number is very imporan as i links o he produc hierarchy of he supermarke chain. This produc hierarchy is indispensable when linking he EAN number o he COICOP. For each EAN here is a produc descripion creaed by he supermarke chain. The price concep Scanner daa volailiy Possible bias deriving from he scanner daa volailiy One imporan difference beween radiional price collecion and scanner daa is he price concep. In radiional price collecion you are dealing wih shelf prices. In scanner daa he price is a uni price calculaed from urnover and sales volumes. Because of he huge amoun of daa ha is generaed in a supermarke each day we receive scanner daa aggregaed on a weekly basis. Scanner daa consiss of weekly aggregaed urnover and volume for each specific EAN number for each sore in he chain. One of he shorcomings wih he weekly aggregaion is ha i is no possible o idenify volume discouns e.g. 3 packages for 2 s price in he daa. The price will be a mix of boh single packages and volume discouns. The derived challenge wih volume discoun prices is ha he mached iem crieria may no be enirely me. A single package is no sricly he same produc as a volume discoun iem of 3 packages. Anoher challenge of using scanner daa for CPI purposes is he abiliy o coninuously monioring producs over ime. Every week here is a number of producs enering or leaving he supermarke sore. Many producs have a shor life cycle bu also changes in he produc packaging resul in new EAN numbers. Consequenly, here are many missing prices hrough he year. These missing prices can creae a bias in he indices if hey are no deal wih properly. I is possible ha he scanner daa based CPI has a downwards or an upwards bias when eiher new producs ener he iem baske or when producs are leaving he iem baske. We have observed biases in wo ways: A produc eners he baske on discoun he firs monh and is hen se o he normal price in he monhs afer. This leads o an arificial increase in he index ha will no ge levelled ou. A produc leaves he baske on discoun. This leads o a persisen decrease in he index. When a large proporion of he above incidens happen he bias becomes problemaic in he indices over ime. The upwards or downwards biases ofen happen when he producs are emporarily ou of he scanner daa. The many missing prices in daa series are impossible o foresee and herefore mus be deal wih ex pos. If a produc is emporarily ou of he scanner daa or persisenly ou of he scanner daa he decisions on impuaion hen becomes imporan. The problem wih impuing is ha i can lead o a bias owards index. Hisorically, Saisics Denmark has reaed missing prices manually securing he impuaion bias is minimized. If a price is missing in monh 0 (p(0)) he previous monh s p(-1) is evaluaed wih he p(-2). If p(-1) is differen from he p(-2) he missing price p(0) is se o p(-2). If he produc price is also missing in he fuure monhs i is replaced by a new represenaive iem. In he implemenaion of scanner daa in he CPI i is aemped o coninue processing he missing prices as described above. Page 2 of 29
The analysis In he sepwise analysis i is emphasised o change as lile as possible in each sep in order o isolae he effec of each change. This paper does no deal wih analysing effecs from impuaions or effecs from filers. The curren producion flow The on-going produc updae The incremenal seps of he analysis In appendix A he curren monhly producion flow is shown ogeher wih he new scanner daa processes. The curren producion process allows use of 2 weeks scanner daa per monh. Scanner daa inroduces new processes such as mainenance of he key beween he COICOP and he EANs and mainenance of he iem baske securing a suiable coverage of he oal urnover. When inroducing he scanner daa i is imporan ha hese new processes can fi ino he ime frame of he producion window of less han 2 weeks. The curren producion includes an on-going produc updae procedure. When a produc is no sold by he supermarke chain i is changed o a new similar produc eiher by DST or he supermarke. Tradiionally his has been emphasised in order o allow for an iem baske ha is aligned wih he acual sales in supermarkes and o minimize he use of impuaions. The basis for all analyses in his paper is he currenly used index formula on elemenary aggregae level a weighed Jevons index. We have separaed he analysis in wo main seps going from he curren daa collecion based CPI o he scanner daa based CPI. Addiional hree seps are carried ou aemping o limi issues wih missing prices and o opimise he manageabiliy of he scanner daa in he producion processes. The analyical seps are incremenal which means ha he parameers ha seems o work are carried over o he nex analyical sep: 1. The change of price concep CPI where producs from he curren publish CPI index are found in he scanner daa and replaced in our curren CPI producion sysem. 2. The change of produc selecion. 3. The change of daa collecion period. 4. Chain aggregaion. 5. Limiing he size of he iem baske. The producs analysed in he paper Five differen COICOP groups each wih differen characerisics have been chosen for he analysis. In he curren CPI producion he chosen produc prices are colleced in differen ways. These differences are imporan when comparing he curren CPI wih he scanner based indices. The curren collecion mehods are described below: 1. Rice presumably a generic produc and price sable caegory. A shelf price lis is provided by supermarkes. DST chooses he rice producs from he lis. The chosen rice ypes are based on he Household Budge Survey (HBS). 2. Red wine is ypically bough on discoun. A shelf price lis is provided by supermarkes. DST chooses he red wine producs from he lis and also collecs some produc prices from large wine dealers on he inerne (all non-bag-in-box). The red wines chosen from he supermarke are mainly from France, Ialy and Spain whereas he producs from he inerne are colleced wihou counry specificaions. 3. Coffee is based on 1 kg produc. Producs are chosen by price collecors The coffee produc prices are based on 1 kg. The coffee producs are colleced by price collecors in he sore. The price collecors choose producs having he mos shelf space. 4. Minced beef a produc ofen bough on volume discoun. The minced beef produc prices are based on 1 kg. The minced beef producs are colleced by price collecors in he sore. The price collecors choose he producs based on he conen of fa. Producs wih less han 15% fa are included. Page 3 of 29
5. Apples a seasonal good. The apple produc prices are based on 1 kg. There is a mix of prices from one 1 kg bags and single apples ha are weighed and recalculaed o represen 1 kg prices. The apple producs are colleced by price collecors in he sore. In he curren price collecion new similar producs are included in he baske when earlier producs disappear. In all cases excep coffee he price collecion includes nonvolume discouns. The differences in he daa collecion mehods are essenial when comparing he curren published indices o he scanner daa based indices. In order o limi he size of his paper only he main rends from he analyses will be discussed in he chapers. The res of he indices can be found in appendix C. The curren published CPI All analyses are based on our curren CPI index formula on elemenary aggregae level. This is a he COICOP 8-digi level (hereafer referred o as c8 level). The index is calculaed as described below: (1) w i i ( p ) i ( p0 ) i Jv p i I0 : = =, = 1 i i w w p0 w i The index formula is a weighed Jevons. The weighing is made on wo levels; on he COICOP 8-digi level and a he sore level. The c8 level weighs are based on he household budge survey and he COICOP specific sore weighs are based on he supermarkes urnover. I has been chosen o use he same sore weighs on c8 level as in he curren published CPI in order o make he sepwise analysis as clear as possible. The specific calculaions for he sore prices, produc prices, and basis prices can be found in appendix B. 1. The change of price concep In his analysis he quesion o be answered is: Are he curren CPIs on elemenary level idenical wih indices based on scanner daa prices covering he exac same iems from he exac same sores? Mached iem crieria Idenifying he curren iem baske in he scanner daa One obvious commen regarding he change in price concep is ha a produc from he scanner daa will have a weekly average price which could level ou he weekly price changes. This could be a problem since volume discouns would be mixed wih non-discoun wihin he same EAN code which implicae ha he mached iem crieria may no be enirely me. I was expeced ha rice producs would be easy o idenify when picking he curren iem baskes rice producs in he scanner daa. However, i was challenging o idenify he exac same rice producs as here is a huge variey of producs. The acual rice marke is differeniaed wih a very wide price span. Since he scanner daa iem descripions are limied he mehod of jus picking he curren baske in he scanner daa is ime consuming. In general he iem from he curren baske has been picked in he exac same sore in he exac same weeks from he scanner daa. When here were many possible maching producs we chose he ones ha had he bes price mach. In his sep of he analysis only rice, red wine and coffee has been analysed due o he ime-consuming ask of manually idenifying he exac curren baske iems. Page 4 of 29
The Consumer Price on Rice from January o December 2011 82 SD curren baske 81 80 79 78 77 76 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The published CPI on rice is relaively sable whereas he curren baske picked in scanner daa (SD curren baske) on rice is more volaile. This is plausible since he curren colleced prices are shelf prices and he scanner daa consiss of acual sales prices ha are per definiion more volaile. The SD curren baske seems o have a slighly negaive rend whereas he published CPI is slighly increasing hrough 2011. We expec ha he rends will show a paern close o each oher on a longer ime frame. The Consumer Price on Red Wine from January o December 2011 SD curren baske 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The published CPI on red wine has a sable slighly increasing rend hrough 2011 whereas he SD curren baske on red wine is again more volaile. The SD curren baske seems o be volaile around he and herefore seems o have similar overall rends as expeced. The main difference beween he wo ses of prices is ha he scanner daa include volume discouns whereas he only includes prices for 1 bole. I is currenly no possible o separae non-volume discouns in he received scanner daa. Page 5 of 29
The Consumer Price on Coffee from January o December 2011 150 145 140 135 SD curren baske 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Besides he increased volailiy in he scanner daa based coffee index he overall rend is diverging more exensively han he indices on rice and red wine. This divergence may be based on a change in consumer preferences when he general coffee prices wen up in he firs half of 2011. As he published CPI is based on prices on he shelf he price index has a clear increase whereas he weekly urnover based price index from he scanner daa reflecs a change on consumer preference owards buying a larger amoun of coffee on discoun. Conclusion In general he differen price concep linked o scanner daa inroduces more volailiy o he indices mainly due o discouns. In he shor erm he flucuaions can be large whereas he rend is expeced o be similar on he longer erm. Wih produc groups of rice or red wine he rend may be close o he published CPI by picking he same producs in scanner daa. However, large differences seem o appear when consumer produc preferences change because of large price changes. Hence, a change of price concep inroduces new index behaviours which are in realiy closer o he acual marke siuaion experienced by he consumer. 2. The change of produc selecion Inroducing scanner daa in he CPI producion eliminaes he curren feedback from he supermarke chains regarding he produc selecion. Scanner daa presumably offers a superior way of selecing a proper iem baske. The baske will reflec wha is acually bough by he consumers when using he urnover from he scanner daa. A consequence, however, is ha he amoun of selecable producs is huge and he ways of defining he baske are many. The quesion o be answered in his sep of he analysis is: Wha is he effec on he indices when defining a represenaive iem baske by he scanner daa urnover? All he presened analyses in his chaper will have a flexible iem baske. I is aemped o benchmark how differen selecion parameers impac on he indices. The goal is o define a durable iem baske ha can be handled in he curren producion sysem. In he firs par of he analysis only 1 week of daa per monh (1w/m) is used. The baske is defined from monh o monh wih he following crieria: The iems mus have been sold in he 4 previous monhs. The urnover regarded for each specific iem in each sore consiss of 4 monhs aggregaed urnover. Page 6 of 29
The iems seleced for he iem baske add up o 50% of he COICOP 8-digi urnover for each supermarke chain 2. In he index calculaion each supermarke chain on he COICOP 8-digi level is weighed securing ha a supermarke chains wih lower urnover have less impac on he index han sores wih higher urnover 3. The Consumer Price on Rice from January o December 2011 84 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa 82 80 78 76 74 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Wih he urnover defined iem baske of rice (SD urnover def baske) he volailiy is large especially in February and Augus. These exreme decreases are due o discouns. The longer erm rend seems o go back around he published CPI whereas he curren baske from scanner daa (SD curren baske) shows a downwards rend. The index wih scanner daa defined baske is in general more volaile because of discouns ha are no capured in he shelf prices. The Consumer Price on Red Wine from January o December 2011 108 94 92 90 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The urnover based baske on red wine is volaile as he curren baske in scanner daa. However, he las 4 monh of 2011 i has a larger increase. This increase sems 2 When defining he iem baske he firs goal was o be able o selec he mos imporan produc wihin he COICOP group. This has been done wih a selecion idenifying he produc wih he highes urnover of all he supermarke chains wihin he COICOP group on he 6-digi level. A derived challenge from his mehod is ha some supermarke chains are excluded from he daa series. In some cases a supermarke chain had discoun on a produc one monh resuling in very large sales excluding he iems in oher supermarke chains. The nex monh anoher supermarke chain would have a discoun resuling in an iem baske wih a very volaile selecion of supermarke chains. The soluion was o selec he iems wih op 50% urnover wihin he COICOP 6-digi level wihin each supermarke chain. 3 See appendix B for furher informaion Page 7 of 29
from wines ha are sold on a lower inroducory price ha in a laer period is sold on a normal price before i again leaves he daa (is ou of sock). Looking ino he micro daa here are a lo of daa series only having 3-5 monhs of sales in each sore. The iem exis combined wih price increases in he monh before resuls in an upwards bias. The Consumer Price on Coffee from January o December 2011 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa 150 145 140 135 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The urnover based coffee index is closer o he published CPI han he SD curren baske. The difference beween he urnover based index and he SD curren baske is he produc selecion. The coffee producs wih high sales urnover are differen from he seleced coffee producs in he curren baske from he published index. One of he main reasons is ha he curren collecion of coffee is only based on single packages and no volume or bundled coffee packages. The published coffee index is based on 1815 prices in 2011 whereas he urnover based iem baske is based on,8 prices from all he differen sores in he supermarke chains. The number of price observaion for each analysis can be found in appendix D. Scanner daa volailiy and he iem baske In order o secure ha he scanner daa iem baske is more robus o he producs enering and leaving he scanner daa 4 we have added he crieria ha an iem in he baske mus have been in he scanner daa for a leas 4 monhs. In his way producs ha are sold for a very shor ime are excluded limiing he problems wih bias from ingoing and ougoing producs on discoun. Even hough he crierion of 4 monhs in daa seems o be adequae i is analysed below wha difference a 3 monhs in daa crieria makes. This shows how sensiive he indices are o his ype of crieria. 4 Only 74.3% of he EAN numbers represened in January 2011 are sill represened in he scanner daa of December 2011. Page 8 of 29
The Consumer Price on Red Wine from January o December 2011 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa 108 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The red wine indices are quie similar. From January o Sepember he index wih he 4 monhs crieria is slighly higher han he index wih 3 monhs crieria. The reason could be ha he 4 monh crieria lead o fewer producs leaving he daa on discouns. The Consumer Price on Apples from January o December 2011 108 94 92 90 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec In he analysis of apples he 4 monh crieria seems o resul in less volailiy in he index which is mainly because of he apples seasonaliy. The mos seasonal apples are excluded by he 4 monhs crierion which is no desirable. When dealing wih seasonal iems i seems o be bes o have a less resricive crierion han 3 or 4 monhs in he baske. In mos of he analysed produc groups here is lile impac on he index by going from he 3 o he 4 monhs crieria. The indices are slighly less volaile when requiring ha he iem baske candidaes mus have been raded a leas four monhs insead of hree monhs. The remaining produc groups analysed regarding he 3 or 4 monhs crieria can be found in appendix C, page 21. Flexible versus fixed basked iem basked There are advanages and disadvanages wih a flexible baske. An advanage is ha he monhly baske sample will include many new producs enering scanner daa every week. The supermarkes inroduced 8,783 new EAN numbers during 2011 which is an increase of 32.9%. A disadvanage wih he flexible baske is ha many new producs have a shor lifeime which could lead o daa series se wih many missing prices. Page 9 of 29
In fuure works we will aemp o creae a sysem ha can suppor a semi-fixed baske wih he opporuniy o manually monior and include new imporan upcoming producs in he iem baske. Conclusion I is challenging o make an iem baske based on scanner daa even hough he urnover informaion is available. To answer he quesion regarding he impac of using urnover informaion as a selecion parameer i generally leads o a differen baske of iems han he iem baskes of he currenly published indices. In order o limi he number of missing prices in he daa series i is imporan o pick producs ha are sold in a longer ime period. The analyses show ha he 4 monhs crieria is beer han 3 monhs regarding missing prices. Wih non-seasonal goods he index differences are quie small. Seasonal goods, however, should be reaed wih less resricive crieria. The goal is o use a semi-fixed baske so ha new imporan producs are included in he CPI sample. Therefore i becomes essenial o build a sysem ha keeps rack of he curren baske and add new imporan producs o he baske. 3. The change of daa collecion period Using scanner daa in he curren CPI producion flow has resricions. Firs of all here is limied ime for processing he daa. Secondly, he weekly aggregaed daa are ofen spli beween monhs in he firs and he las week of a monh ofen have daa ha belong o boh he previous and he following monh. These resricions allow using 2 weeks of daa per monh. In his sep of he analysis he quesion is: How will change from 1 week of daa per monh o 2 weeks of daa per monh impac he indices? An obvious commen is ha 3 weeks per monh should be esed as well. Because he curren producion window is limied i is a his momen no realisic o incorporae 3 weeks. To limi his paper he 3 weeks analysis has been lef ou. The Consumer Price on Red Wine from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 108 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Regarding red wine here is a limied impac on he index by changing from 1 week o 2 weeks of daa per monh. The reason is ha he wines seleced for he iem baske in he paricular monh are ofen sold in he 2 weeks a similar prices. Using 2 weeks of daa per monh leads o fewer missing prices. One week of daa per monh includes 353,927 prices of red wine whereas wo weeks of daa per monh include 410,795 prices. Page 10 of 29
The Consumer Price on Minced Beef from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 99 97 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec There is a downwards rend in he index of minced beef when only including 1 week of daa per monh. The main explanaion is ha here are a lo of missing prices in he daa series. When a produc emporarily leaves he daa on a discoun in one week he index will no have he corresponding upwards impac when he produc reurns o he normal price. Ofen he iem will be sold again in laer monhs a he normal price and hen again exi a volume discoun. Changing from one week o wo weeks of daa per monh has a large effec on he minced beef index. Especially in he 4 h quarer of 2011 he bias seems o be removed when using 2 weeks of daa per monh. The reason for his is ha wo weeks of scanner daa per monh leads o less missing prices. If he produc is ou of sock in one of he weeks and i is sold in he oher week he produc will be represened in he daa series. The same rends are seen in he rice and apple indices 5. Conclusion In general wo weeks of daa limi he level of bias semming from producs ha leaves he iem baske on discoun. The effec of increasing he daa period is larger wih rice, minced beef han wih red wine and coffee. To limi he emporarily ou of sock problem furher and limi he missing prices i would be desirable o use 3 weeks per monh which will hopefully be possible in he fuure. 4. Supermarke chain aggregaion In his sep of he analysis he focus is o deal wih he missing prices from e.g. emporarily sock ous in he specific sore bu also o aemp o limi he amoun of prices o process in he producion flow. Aggregaing he supermarke chains for each EAN number could be a soluion. The quesion is herefore: Wha happens o he indices when aggregaing produc prices on he supermarke chain level? Can his aggregaion limi he amoun of missing prices? When aggregaing he specific producs prices per supermarke chain he number of prices in 2011 is reduced from 83,477 o 536 in he analysis wih rice. The aggregaion is carried ou by simply summing he urnovers and volumes of all he chains sores for each EAN. 5 The oher indices can be found in appendix C. Page 11 of 29
The Consumer Price on Rice from January o December 2011 84 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 83 82 81 80 79 78 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The wo indices of rice have similar rends, however, he index levels are generally higher when comparing he sore aggregaed CPI s wih he non-sore aggregaed. This is mainly because he supermarke chain aggregaion eliminaes many of he missing prices. The Consumer Price on Minced Beef from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 94 92 90 88 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The supermarke chain aggregaed minced beef index have similar rend as he nonaggregaed index. However, he level of he chain aggregaed index is higher han he non-sore aggregaed index. The difference beween he levels saring in June is mainly due o difference in missing prices. When he June prices on discoun are missing in July a bias is inroduced. By aggregaing prices on supermarke chain level he bias is limied. The number of missing prices is especially decreasing in he analyses of minced beef and o a lesser exen wih rice, red wine, and coffee. Conclusion In general i seems o be a good idea o aggregae each EAN on chain level as i limis he level of missing prices he poenial derived bias. Moreover he amoun of daa o handle is limied which speeds up he performance of he IT sysems and he handling of missing prices. 5. Limiing he size of iem baske Using 2 weeks per monh of daa insead of 1 week mos likely leads o beer indices and sore aggregaion secures ha missing prices are more seldom. Wih hese Page 12 of 29
parameers i is ineresing o observe if a smaller number of iems in he baske would resul in he same overall index rend. An incenive o limi he size of he iem baske even furher is he increase efficiency in he producion processes. Wha is he impac on he indices when limiing he number of iems per COICOP 8-digi group o hree? In his analysis he op 3 iem indices have he following crieria: The daa consis of he 2 middle weeks of scanner daa per monh. The iems have o be presen in he daa in previous 4 monhs o candidae for he iem baske. The 3 bes-selling producs in each supermarke chain per COICOP 8-digi level are included in he baske. The indices named SD Turnover def baske from he earlier analyses have he following crieria: The daa consis of he 2 middle weeks of scanner daa per monh. The iems mus have been sold in he 4 previous monhs. The iems seleced for he iem baske add up o 50% of he COICOP 8-digi urnover for each supermarke chain. The Consumer Price on Rice from January o December 2011 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr 86 SD Turnover def baske, 2w/m, Supermarke chain Aggr 85 84 83 82 81 80 79 78 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Insead of using 536 prices in 2011 wih he 50% urnover crieria he use of op 3 crieria includes 324 prices in 2011. The reducion in he number of observaions does no impac he index significanly before Augus. The differences in Augus, November and December mainly sems from iems on discoun. As he published CPI rice index consiss of only single packages he price volailiy is more limied. The scanner daa prices include volume discouns which are he main explanaion for he volailiy in he indices. Page 13 of 29
The Consumer Price on Red Wine from January o December 2011 114 112 108 94 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr SD Turnover def baske, 2w/m, Supermarke chain Aggr Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Using he op 50% crieria leads o 2,758 prices in 2011 whereas he op 3 crieria include 326 prices in 2011. The scanner daa based indices on red wine have similar rends, bu a level difference of 1-2 pc. The difference is caused by changes in he produc selecion and he embedded differences in price developmens. The difference beween he published CPI on red wine and he scanner daa indices is ha scanner daa conains prices on bole in bags as hey have a large sales urnover. The published CPI only conains prices of single boles of red wine. Hence, he main explanaion for he difference is he produc selecion. The Consumer Price on Minced Beef from January o December 2011 105 103 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr SD Turnover def baske, 2w/m, Supermarke chain Aggr 101 99 97 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Using he op 3 crieria leads o a lower level minced beef index han he 50% urnover and 4 monhs in baske crieria. In general he minced beef and apple indices have significanly fewer prices when using he 50% crieria han when using he op 3 selling producs. Typically here are 2 producs per supermarke chain dominaing he sales 6. Therefore he analyses on hese produc groups do no fi he originally purpose of limiing he iems in he baske. This opens an imporan discussion wheher o se a fixed arge on he number of producs included in he baske for all COICOP groups. The differen COICOP groups have differen marke dynamics. The amoun of producs available and heir volailiy are differen from 6 The minced beef in scanner daa consiss of 150 producs wih a oal urnover of 388,770,820 DKK in December 2011. By using he 4 monh in baske crierion only 74 iems are included add up o 192,043,344 DKK of he urnover. When applying he op 3 besselling producs per supermarke chain only 24 iems are included adding up o a urnover of 116,447,638 DKK. When applying he 50% urnover per supermarke chain only 13 iems are included which accoun for 86,256,8 DKK. Page 14 of 29
produc group o produc group. Ideally he number of producs seleced for each COICOP group should be decided in relaion o he COICOP characerisics. The furher work will coninue in he direcion of defining appropriae selecion crieria for he COICOP group. Conclusion Rice, Coffee and Red wine have similar rends when eiher using he op 3 selecion or he op 50% urnover. An ineresing issue appeared when analysing he minced beef and apple indices because of large changes EANs included in he baske over ime. These very volaile daa series requires specific inclusion crieria. In general i depends on he COICOP group wheher he op 3 producs by urnover or 50% urnover crierion are suiable o describe he COICOP group. Conclusion Scanner daa is indeed differen from he informaion receive in he regular price collecion. This paper shows ha large differences appear when only changing he price concep. This is parly due o volume discouns ineviably included in he scanner daa whereas he curren daa collecion excludes he volume discouns. Anoher change inroduced wih scanner daa is he index volailiy which is a naural elemen when going from he more sable shelf prices o he acual sales prices. The volailiy of scanner daa leading o somewha divergen rends is plausible on he shor and medium erm whereas he long-erm effec of changing price concep should be small. I will be ineresing o analyse he long-erm rends when wo years of scanner daa becomes available. Defining a represenaive iem baske based on urnover informaion from scanner daa is no rivial. I became clear ha a fixed baske accompanied wih a flexible aribues during he year is desirable. In his paper only he flexible baske mehodology is used in order o cope wih he high ariion raes. In general his delivers a quie volaile bu represenaive iem baske. When inroducing scanner daa in he Danish CPI producion especially wo issues calls for soluions which are he missing prices and he number of iems o process in he producion sysems. I became eviden ha bias from missing prices could parly be deal wih by using a 2 weeks of daa per monh insead of 1 week per monhs. Also he supermarke chain aggregaion removed some of he bias. Regarding he specific parameers defining he iem baske i became clear ha he differen COICOP groups have diverse marke dynamics. Especially seasonal goods should ideally no have a resricion requiring he iem o exis in scanner daa he previously 3 or 4 monh. Also producs wih a lo of discouns wih emporarily sockous should be handled wihou he 3 or 4 monh crierion. As his may no be sufficien o preven hin series he urnover selecion should include more iems han a 50% urnover rule or he op 3 bes-selling iems would include. Furher analyses of he COICOP group characerisics will be carried ou in fuure sudies which are expeced o resul in represenaive iem baske suied for he curren CPI producion. Page 15 of 29
Appendices Appendix A - The Danish monhly CPI Workflow
Appendix B formulas Sore prices (1) f r i 1/ r 1 1/ r 2 1/ r r 1/ r = ( ρ ) = ( ρ ) ( ρ )... ( ρ ) i= 1 f ρ i : sore price in period : iem prices in period from sore i=1,,r Produc prices (2) e m g 1 2 m g s 1 s 2 s m s g = (f ) = (f ) (f )... (f ), s = 1 g= 1 e g f s g : produc price in period : sore prices in period : weigh for sore group g = 1,,m Basis prices (3) p n k 1 2 n k v 1 v 2 v n v k = (e ) = (e ) (e )... (e ), v = 1 k= 1 p j e v k : price on elemenary level in period : produc prices in period : weigh for produc group k = 1,,n Monhly index on elemenary level (4) I 1: = p p 1 I -1: : on elemenary level from -1 o p : on elemenary level in period p -1 : on elemenary level in period -1
Appendix C Resuls of he analyses 1. Change of Price Concep The Consumer Price on Rice from January o December 2011 82 SD curren baske 81 80 79 78 77 76 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 SD curren baske 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 150 145 140 135 SD curren baske 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 18 of 29
2a. Iem baske definiion, 1 week daa per monh The Consumer Price on Rice from January o December 2011 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa 84 82 80 78 76 74 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa 108 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 SD curren baske SD Turnover def baske, 1w/m - iem 4 monhs in daa 150 145 140 135 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 19 of 29
The Consumer Price on Minced Beef from January o December 2011 SD Turnover def baske, 1w/m - iem 4 monhs in daa 105 103 101 99 97 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Apples from January o December 2011 SD Turnover def baske, 1w/m - iem 4 monhs in daa 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 20 of 29
2b. Iem in scanner daa crieria; 3 monhs vs. 4 monhs The Consumer Price on Rice from January o December 2011 84 82 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa 80 78 76 74 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa 108 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 SD Turnover def baske, 1w/m - iem 3 monhs in daa 150 145 SD Turnover def baske, 1w/m - iem 4 monhs in daa 140 135 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 21 of 29
The Consumer Price on Minced Beef from January o December 2011 94 92 90 88 86 84 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Apples from January o December 2011 108 94 92 90 SD Turnover def baske, 1w/m - iem 3 monhs in daa SD Turnover def baske, 1w/m - iem 4 monhs in daa Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 22 of 29
3. Daa collecion period; 1 week per monh vs. 2 weeks per monh The Consumer Price on Rice from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 84 83 82 81 80 79 78 77 76 75 74 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 108 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 150 145 140 135 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 23 of 29
The Consumer Price on Minced Beef from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 105 103 101 99 97 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Apples from January o December 2011 SD Turnover def baske, 2w/m SD Turnover def baske, 1w/m 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 24 of 29
4. Supermarke chain aggregaion The Consumer Price on Rice from January o December 2011 84 83 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 82 81 80 79 78 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 114 112 108 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 150 145 140 135 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 25 of 29
The Consumer Price on Minced Beef from January o December 2011 105 103 101 99 97 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Apples from January o December 2011 108 SD Turnover def baske, 2w/m SD Turnover def baske, 2w/m, Supermarke chain Aggr 94 92 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 26 of 29
5. Top 3 producs per supermarke chain in he COICOP group The Consumer Price on Rice from January o December 2011 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr 86 SD Turnover def baske, 2w/m, Supermarke chain Aggr 85 84 83 82 81 80 79 78 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Red Wine from January o December 2011 114 112 108 94 92 90 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr SD Turnover def baske, 2w/m, Supermarke chain Aggr Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Coffee from January o December 2011 150 145 140 135 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr SD Turnover def baske, 2w/m, Supermarke chain Aggr 130 125 120 115 105 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 27 of 29
The Consumer Price on Minced Beef from January o December 2011 105 103 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr SD Turnover def baske, 2w/m, Supermarke chain Aggr 101 99 97 95 93 91 89 87 85 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec The Consumer Price on Apples from January o December 2011 SD Baske Top 3 per Supermarke chain & c8 Baske, 2w/m, Supermarke chain Aggr 108 SD Turnover def baske, 2w/m, Supermarke chain Aggr 94 92 90 Jan Feb Mar Apr May Jun Jul Aug Sep Oc Nov Dec Page 28 of 29
Appendix D Number of prices in he analyses Rice Publ CPI SD curren baske SD Turnover def baske, 1w /m - iem 4 monhs in daa SD Turnover def baske, 1w /m - iem 3 monhs in daa SD Turnover def baske, 2w /m - iem 4 monhs in daa SD Turnover def baske, 2w /m, Supermarke chain Aggr - iem 4 monhs in daa SD Baske Top 3 per supermarke chain & c8 Baske, 2w /m, Supermarke chain Aggr Number of price observaions Min' - 'Max' obs per monh Toal in 2011 12 144 1978-2726 26687 5884-7487 78846 5535-7810 82218 6359-7791 83477 40-50 536 27 324 Red wine Publ CPI SD curren baske SD Turnover def baske, 1w /m - iem 4 monhs in daa SD Turnover def baske, 1w /m - iem 3 monhs in daa SD Turnover def baske, 2w /m - iem 4 monhs in daa SD Turnover def baske, 2w /m, Supermarke chain Aggr - iem 4 monhs in daa SD Baske Top 3 per supermarke chain & c8 Baske, 2w /m, Supermarke chain Aggr Number of price observaions Min' - 'Max' obs per monh Toal in 2011 147-154 1815 14615-17760 1949 27755-30568 353927 28370-32084 361240 32666-35205 410795 220-239 2758 27-28 326 Coffee Publ CPI SD curren baske SD Turnover def baske, 1w /m - iem 4 monhs in daa SD Turnover def baske, 1w /m - iem 3 monhs in daa SD Turnover def baske, 2w /m - iem 4 monhs in daa SD Turnover def baske, 2w /m, Supermarke chain Aggr - iem 4 monhs in daa SD Baske Top 3 per supermarke chain & c8 Baske, 2w /m, Supermarke chain Aggr* *3 normal coffee producs and 3 insan coffee producs Number of price observaions Min' - 'Max' obs per monh Toal in 2011-115 1351 2823-3016 35412 7717-8433 8 7732-9232 092 8308-9103 105678 51-53 621 53-54 645 Minced beef Publ CPI SD curren baske SD Turnover def baske, 1w /m - iem 4 monhs in daa SD Turnover def baske, 1w /m - iem 3 monhs in daa SD Turnover def baske, 2w /m - iem 4 monhs in daa SD Turnover def baske, 2w /m, Supermarke chain Aggr - iem 4 monhs in daa SD Baske Top 3 per supermarke chain & c8 Baske, 2w /m, Supermarke chain Aggr Number of price observaions Min' - 'Max' obs per monh Toal in 2011 115-121 1407 832-2423 21676 828-2580 22121 1013-2561 230 10-15 155 23-26 291 Apples Publ CPI SD curren baske SD Turnover def baske, 1w /m - iem 4 monhs in daa SD Turnover def baske, 1w /m - iem 3 monhs in daa SD Turnover def baske, 2w /m - iem 4 monhs in daa SD Turnover def baske, 2w /m, Supermarke chain Aggr - iem 4 monhs in daa SD Baske Top 3 per supermarke chain & c8 Baske, 2w /m, Supermarke chain Aggr Number of price observaions Min' - 'Max' obs per monh Toal in 2011-117 1358 1827-2656 27066 1477-2710 29175 1857-2704 27806 10-13 145 26-28 325 Page 29 of 29