A Framework for Large Scale Use of Scanner Data in the Dutch CPI

Transcription

1 A Framework for Large Scale Use of Scanner Daa n he Duch CPI Jan de Haan Sascs Neherlands and Delf Unversy of Technology Oawa Group, 2-22 May 215

2 The basc dea Ideally, o make he producon process as effcen as possble, a lmed number of fully or sem-auomaed mehods would be used. The purpose of hs paper s o propose a framework supporng hese plans. Our basc aggregaon formula s wha we refer o as a qualyadjused un value ndex, whch s equal o he value ndex dvded by a quany ndex ha s defned as he rao of qualy-adjused or sandardzed quanes. Tme dummy regresson models play an mporan role n he esmaon of he qualy-adjusmen (sandardzaon) facors. There are wo exreme cases. If nformaon on all relevan em characerscs s avalable, hen he use of me dummy hedonc models s preferred. When characerscs nformaon s lackng, he use of me-produc dummy (fxed-effecs) models s proposed. (De Haan, 215) 2

3 Oulne Expandng he use of ransacons (and onlne) daa The framework Sarng pon: real chans Un values, un value ndexes and qualy-adjused un value ndexes Esmang he qualy-adjusmen facors weghed me dummy hedonc regressons weghed me-produc dummy (fxed effec) regressons Accounng for revsons: wo splcng mehods An example usng NZ consumer elecroncs scanner daa Fxed effecs and lack of machng Onlne daa Conclusons 3

4 Expandng he use of ransacons (and onlne) daa Budge cus - reducon of feld prce collecon Expandng he use of secondary daa - new projec jus sared scanner daa provded by real chans (supermarkes, drugsores, DIY sores, deparmen sores, ec.) onlne daa obaned hrough web-scrapng New mehods requred Lmed number of dfferen mehods Lower levels: rends more mporan han shor-erm changes Weghed (superlave-ype) ndexes Hedonc qualy adjusmen Focus on mullaeral approaches; maxmum use of maches n he daa whle beng (approxmaely) free of chan drf 4

5 Real chans and elemenary aggregaes Elemenary aggregaes: cross-classfcaon of sore ypes and produc caegores Each real chan (and feld collecon) reaed as a separae sore ype Produc caegores: COICOP classfcaon a publcaon level Below lowes publcaon level: chan-specfc elemenary aggregaes Typcally a sngle prce ndex mehod used for a parcular chan (no requred ype of produc more mporan han ype of chan) Mos elemenary prce ndexes currenly unweghed Aggregaon whn and across chans: scanner daa where possble 5

6 Un value ndex Tme perods,...,t, where s he base perod or sarng perod of he me seres o be consruced p... q... T p T q T s Prces, quanes, expendure shares (Dynamc) sample of ems purchased/sold T S...S s... For a homogeneous em, he average ransacon prce or un value s he approprae measure of prce and he un value ndex P UV S S p p q q S 1 S q q S S s s ( p ( p ) ) s he approprae measure of prce change beween perods and ( 1,..., T ) 6

7 7 Qualy-adjused un value ndex Un value ndex s no approprae for heerogeneous producs Sandardzaon: quany of each em expressed n uns of an arbrary base em b usng sandardzaon facors or qualy-adjusmen facors Equvalen o adjusng he prce of each em for dfference n qualy wh he base em: Qualy-adjusmen facors assumed consan across me Qualy-adjused un value ndex / b ) ( ~ ) ( ~ S S QAUV p s p s P b p p / / ~

8 8 Qualy-adjused un value ndex and quany ndex Implc quany ndex Smple rao of quanes expressed n consan-qualy uns; easy o nerpre change n number of qualy-adjused sales Quany ndex smplfes o he rao of number of (unadjused) sales when all ems or of he same qualy Quany ndex and qualy-adjused un value ndex are boh ransve Quany ndex sasfes deny es n mached-em conex bu qualy-adjused un value ndex doesn / / 1 S b S b QAUV S S q q P q p q p Q

9 Esmang he qualy-adjusmen facors Mullaeral regresson-based approach Tme dummy regressons based on pooled daa for perods Weghed leas squares: expendure shares serve as weghs o reflec ems economc mporance,...,t Model 1) Tme dummy hedonc model ln p T K D 1 k 1 z k z wh em characerscs and me dummy varables k k D Qualy-adjusmen facors esmaed by ˆ exp K b pˆ / / pˆ p pˆ / pˆ ˆ b k ( zk zbk ) k 1 9

10 Tme dummy hedonc ndex ˆ~ p p / ˆ / b Usng and, he me dummy ndex can be wren as ˆ~ ˆ p p / / b Pˆ TD S S ( pˆ~ ( pˆ~ ) ) s s Tme dummy ndex: rao of expendure-share weghed geomerc means of qualy-adjused prces Qualy-adjused un value ndex: rao of expendures-share weghed harmonc means of (he same) qualy-adjused prces The wo ndexes are ransve, hence free of chan drf, and ndependen of choce of base em 1

11 Tme dummy ndex and qualy-adjused un value ndex Relaon beween he wo ndexes ˆ S QAUV Pˆ TD P S s s exp( u exp( u ) ) u ln( pˆ / p ) are regresson resduals Brackeed facor - changes he geomerc qualy-adjused un value ndex (.e., he me dummy ndex) no he desred arhmec qualy-adjused un value ndex - depends on varance of resduals and s expeced o be very small 11

12 Tme-produc dummy (fxed effecs) ndex Many scanner daa ses: no enough characerscs nformaon avalable for hedonc regressons Model 2) Tme-produc dummy or fxed effecs model ln p T 1 D N 1 1 D wh dummy varables (ndcaors) for he varous ems D The em-specfc fxed effecs of he unknown hedonc prce effecs can be vewed as approxmaons K k 1 k z k 12

13 Tme-produc dummy (fxed effecs) ndex The me-produc dummy or fxed effecs (FE) model s a specal case of he me dummy hedonc model, so - same choce of regresson weghs (expendure shares) - smlar relaon wh qualy-adjused un value ndex - TPD/FE ndex s ransve/free from chan drf TPD/FE ndex - mgh be called a qualy-adjused prce ndex because uses a form of overlap prcng for new and dsappearng ems - needs a leas wo observaons for an em o be non-rvally ncluded; for example, new ems n he las perod (T) are no ncluded! 13

14 Decomposon of me dummy hedonc and FE ndexes Because he weghed me dummy ndexes are ransve, hey can be wren as chaned perod-on-perod ndexes A sngle ndex movemen can be decomposed as follows: adjacen-perod Tornqvs ndex X effec of dsappearng ems X effec of new ems X resdual facor (o make he chaned ndex ransve) 14

15 Accounng for revsons All mullaeral ndexes, ncludng me hedonc, FE, and qualyadjused un value ndexes, suffer from revsons - when he sample perod s exended and new daa s added, he resuls for all perods wll change Rollng wndow approach esmaon wndow (fxed lengh) s moved forward and ndexes (re-) esmaed Two ssues: - how should he esmaes from he mos recen wndow be lnked o he exsng me seres,.e. wha s he preferred splcng mehod? - wha s he opmal wndow lengh? 15

16 Two splcng mehods Every splcng mehod mpars he ransvy propery of mullaeral prce ndexes, so chan drf n he lnked me seres canno be compleely ruled ou 1) movemen splce afer movng forward he wndow one monh and re-esmang he model, he mos recenly esmaed monh-on-monh movemen of he ndex s splced on o he exsng me seres 2) (Frances Krsnch s) wndow splce splces he enre newly esmaed 13-monh seres on o he ndex level peranng o 12 monhs ago (by consrucon no chan drf n annual changes) 16

17 Two splcng mehods Movemen splce s ypcally used for me dummy hedonc ndexes (and GEKS ndexes, an alernave mullaeral approach wh no characerscs nformaon) Wndow splce s probably more useful for FE ndexes Choce of esmaon wndow lengh - a leas 5 quarers (or 13 monhs) o nclude srongly seasonal ems - no oo long; assumpon of fxed (underlyng) characerscs parameers Wh wndow splcng, he esmaon wndow may dffer from he splcng wndow 17

18 Qualy-adjused un value ndexes: an example New Zealand scanner daa from marke research company GfK Seven consumer elecroncs producs Monhly daa from md-28 o md-211 Close o full coverage of New Zealand consumer marke Iems defned as unque combnaon of brand, model and avalable se of physcal characerscs Daa aggregaed across oule ypes Hgh degree of churn (new and dsappearng ems) 18

19 Qualy-adjused un value ndexes: an example (Adjused) R square values for WLS me dummy hedonc regressons range from.964 (DVD players) o.989 (porable meda players) Hgh R squares are parly due o aggregaon over goods (barcodes) wh dencal characerscs We also esmaed me dummy models usng OLS, he resulng me dummy ndexes and correspondng qualy-adjused un value ndexes (Adjused) R squares for OLS regressons range from.859 o.913 (dgal cameras) 19

20 Tme dummy hedonc and qualy-adjused un value ndexes Porable meda players TD (WLS) QAUV (WLS) TD (OLS) QAUV (OLS) RYQAUV Dgal cameras TD (WLS) QUAV (WLS) TD (OLS) QUAV (OLS) RYQAUV 2

21 Indexes of number of sales and qualy-adjused sales 3.5 Porable meda players Unadjused Adjused Dgal cameras Unadjused Adjused 21

22 Fxed effecs and lack of machng EAN (GTIN) can be oo dealed o be useful as em denfer - dfferen EANs may relae o he same em - lack of machng over me - rae of em churn overesmaed - hdden prce changes wll be mssed n mached-model ndexes, ncludng FE ndexes Aggregang across EANs peranng o he same em requred n Ausrala: SKU No a bg problem for me dummy hedonc ndexes (alhough resuls wll be ncreasngly model-based) 22

23 Onlne daa Prces colleced from realers webses va web scrapng Many ssues nvolved - coverage of ems sold (webses versus physcal sores) - prce dfferences beween webses and sores? - onlne versus offlne purchases - characerscs nformaon? - changes n webses Quanes/expendures are unobservable - no un values across he monh or quarer, jus average prces - qualy-adjused un values and weghed ndexes no possble 23

24 Conclusons Sascs Neherlands: ncreasng use of scanner daa and onlne daa For effcency reasons: lmed number of fully or sem-auomaed mehods Regresson-based mehods mos promsng - me dummy hedonc models (scanner daa) - me-produc dummy / FE effecs models (scanner daa, onlne daa) Proposal: qualy-adjused un value ndexes raher han nal ndexes (scanner daa) New projec sared research and mplemenaon; nernaonal collaboraon? 24