A Test for Sparsity. Junnan He Department of Economics Washington University in St. Louis. October 15, Abstract

Transcription

1 A Test for Sarsity Junnan He Deartment of Economics Washington University in St. Louis October 5, 08 Abstract Many roerties of sarse estimators rely on the assumtion that the underlying data generating rocess DGP is sarse. When this assumtion does not hold, a sarse estimator can erform worse than non-sarse estimators such as the ridge estimator. We roose a test of sarsity for linear regression models. Our null hyothesis is that the number of non-zero arameters does not exceed a small reset fraction of the total number of arameters. It can be interreted as a family of Bayesian rior distributions where each arameter equals zero with large robability. As the alternative, we consider the case where all arameters are nonzero and of order / for all number of arameters. Formally the alternative is a normal rior distribution, the maximum entroy rior given zero mean and the variance determined by the ANOVA identity. We derive a test statistic using the theory of robust statistics. This statistic is minmax-otimal when the design matrix is orthogonal, and can be used for general design matrices as a conservative test. Keywords: Sarsity; Variable Selection; Robust statistic. JEL: C For the newest version lease go to htts://junnanhe.weebly.com. The author is grateful to Werner Ploberger for the insiring discussions and guidance. The author thanks Yuing Chen, Siddhartha Chib, George-Levi Gayle, Nan Lin for their helful feedback.

2 It can scarcely be denied that the sureme goal of all theory is to make the irreducible basic elements as simle and as few as ossible without having to surrender the adequate reresentation of a single datum of exerience. Albert Einstein, Oxford 933, The Herbert Sencer Lecture. Introduction The increase in availability of data has boosted a fast growing literature on variable selection. The goal of which is to search for a small set of variables from a vast number of different combinations that can sufficiently exlain the resonse variable. An estimator that contains many zeros in the estimated coefficients is called a sarse estimator. Such estimators include but not limited to AIC Akaike, 974, BIC Schwarz, 978, LASSO Tibshirani, 996, SCAD Fan and Li, 00, Elastic Net Zou and Hastie, 005 etc. When the data generating rocess DGP is sarse, i.e. when the resonse is only significantly affected by a diminishing fraction of the variables, sarse estimators can consistently find the imortant variables and estimate them efficiently Meinshausen and Buhlmann, 006; Zhao and Yu, 006; Zhang and Huang, 008. However, it is well known that a sarse estimator does not always dominate a non-sarse one. If the underlying DGP is not sarse, using a sarse method may result in inefficient estimates. Tibshirani 996 observed in simulations that, when the DGP is dense, i.e. a large number of small effects, the sarse estimator LASSO is significantly less efficient than the ridge regression, a dense estimator. Aart from the loss in efficiency, consistency can also be comromised. When alied to a dense DGP, sarse estimators can be selection inconsistent by selecting significantly too few variables. Figure.

3 Figure.: In each grah, we simulate 000 times the regression model Y = Xβ+u and estimate the LASSO with the tuning arameter determined by otimizing the BIC Zou, Hastie and Tibshirani, 007. Then we lot the histogram of the number of non-zero coefficients estimated. The design matrix is always simulated from a multivariate standard normal and u N 0,. On the left enal, β =,,,,,...,,,,, / 50 is a vector of length 50 and the number of observation is n = 00. On the right enal, β =,,,,,...,,,,, / 00 is a vector of length 00 and the number of observation is n = 500. shows that LASSO selects a very low dimensional model when the true DGP is dense. While all variables have non-zero coefficients in both simulations, 40% of the estimated models are of dimension less than four in the first anel, and 5% in the second anel. Tyically, when the number of regressors diverges and the DGP remains sarse, many sarse estimators have the roblem of being too liberal that they select too many irrelevant variables Chen and Chen, 008. On the other hand, the above simulation shows when the DGP is dense, sarse methods are likely too stringent. Moreover, there is evidence that many economic variables may not have a sarse DGP. Giannone, Lenza and Primiceri 07 used a Bayesian aroach to estimate the model dimensions for a number of regressions with economic variables. Their osterior distributions were found to concentrate in high dimensional models for all the macroeconomic and financial examles in their aer. Such evidence suggest caution in alying sarse estimator to economic variables. When the underlying DGP is dense, alying a dense estimator such as the ridge estimation is more efficient see e.g., Hsu, Kakade and Zhang, 04. Hence one should determine the data sarsity when choosing between 3

4 a dense estimator and a sarse estimator. In this aer, we rovide a test to distinguish whether the DGP is sarse or dense for linear regression models. The test can be used as a validation or diagnostics before or after alying sarse estimators. We focus the analysis on the linear regression framework because it is the most oular statistical tool, and many sarse estimation techniques were first roosed for the regression context. In comarison to ure Bayesian techniques such as Giannone et al. 07, our null hyothesis consists of a large family of sarse data generating rocesses. Hence when the null is rejected, it is not due to the secification of the rior distribution. One can interret our test as a test between two families of Bayesian riors. The null hyothesis is a set of rior distributions that each coefficient of interest is zero with high robability, and the alternative hyothesis is the rior that each coefficient is of the same magnitude. Informally, our rocedure works in the following way. Let n and resectively be the number of observations and the number of arameters to be estimated. Suose < n while both n, are allowed to diverge to infinity. The test statistic summarizes the number of coefficients estimated to be significantly far away from zero. Since the OLS is n consistent, all but a few estimated coefficients are close to zero of magnitude / n under the null. When the alternative is true, the estimated coefficients are the sum of their values lus the estimator noise, hence they are of magnitude / + /n. We borrow techniques from robust statistics to distinguish this difference, and reject the null when the estimated values are overall far away from zero. The organization of the aer is as follow. We formally introduce the hyotheses in the next section. The test statistic is derived in Section 3. The rejection region of the test is simulated. The simulation method and a sufficiency result on asymtotic consistency is given in Section 4. We discuss issues related to imlementation of the test in Section 5. Section 6 describes some simulation results and Section 7 rovides two emirical alications of the test. Lengthy roofs are ostoned to the aendix. For nonlinear roblems, when the log-likelihood is sufficiently smooth, many estimators e.g. maximum likelihood can be locally aroximated by a linear estimator. Generalizations to these roblems are ossible but outside the scoe of this aer. 4

5 Figure.: The histogram of the standardized estimated values from a single regression. We standardize the OLS estimates of the model Y = Xβ+ u where u N 0, 6 and X is simulated from multivariate normal with correlation ρx i, x j = 0.3 i j. The true arameter β is of dimension 00 and all but the first 0 entries are 0. The first 0 entries of β are s. The curve is a standard normal density suer-imosed to the histogram. The Hyotheses Consider the classical linear regression model Y = Xβ + u where X is indeendent from u i where u i iid N 0, σ for i =,..., n. The dimensions of Y and β are resectively n and, both diverging to infinity. Without loss of generality we assume that Y and X are standardized to have mean zero and variance. Our null hyothesis can be thought of a large family of rior distributions each describes β as a sarse vector. Let F be the set of all dimensional distributions over the reals in R. Formally, the null hyothesis H 0 ɛ : i, β i = z i γ i γ,..., γ F for some distribution F F; z i is indeendent Bernoulli with success robability ɛ. In other words, each β i = 0 whenever z i = 0, which has robability ɛ. When z i 0, β i = γ i which can be drawn from any distribution over the reals. When we have a fixed disersed distributions F, the distribution for β is similar to a so-called sike-and-slab rior in Mitchell and Beaucham 5

6 988. Nonetheless, F can be an arbitrary distribution including a Dirac-delta measure, in which case each β i is either 0 or a fixed constant, which may be better described as a sike-and-sike rior. The above null hyothesis can be interreted as a family of Bayesian riors for the vector β given the knowledge that about at least ɛ fraction of the entries in β are zero. Since β i = 0 with robability ɛ, each such rior is rather informative about the location of each β i. Naturally the alternative hyothesis should describe the contrary, a lack of information about the recise location. To this end, we take the alternative to be a maximum entroy distribution see e.g. Jaynes, 968. Imosing homogeneity and symmetry, each β i is indeendent and identically distributed around zero. We take the second moment of the β i s by the ANOVA identity E[Y Y ] = E[β X Xβ] + E[u u], or equivalently Var[Y ] = n E[β X Xβ] + σ. These conditions in down the alternative rior distribution for β. Formally, H a : i, β i iid N 0, σ. Under this alternative, not only all β i s are non-zero, but also nearly all arameters are of order. Hence this rior can naturally be interreted as a hyothesis that there is a large number of small effects. Moreover, under the alternative hyothesis, the otimal estimator for square-loss is exactly a ridge regression estimator. This is in accordance with the observation that that the ridge regression is more efficient for a dense DGP. 3 The Test Statistic Assuming the matrix X X is invertible, we derive a test statistic that bases on the OLS estimates. The OLS estimator has variance σ X X. Let the ith diagonal element of the matrix X X/n be s i, and the resective For discussions about the case > n, refer to Section 5. 6

7 ositive square-root be s i. Our test statistics come from the following intuitive observation. We standardize the estimated values to n σs ˆβi i, so that the sequence of normalized estimated values have the same marginal variance conditional on β. When β is sarse, all but a few of the entries are zero. If we remove the indices i s where β i 0, the remaining estimated values all centers around 0 with variance. As shown in Figure, other than a few n σs ˆβi i s for which β i 0, most other entries of normalized ˆβ lies under the standard normal density. Under some regularity conditions, Azriel and Schwartzman 05 Theorem shows that the emirical distribution of the estimated values, i.e. n σs ˆβi i for which β i = 0, converges to the standard normal distribution. In our context, when the null is true, the standardized estimated vector can be thought of a normal vector with a few outliers. This allows us to interret our null to be the following. Each n σs ˆβi i is drawn from a standard normal with ɛ robability, and with ɛ robability, it is drawn from an arbitrary unknown distribution. To ut differently, the estimated values as random draws from a distribution in the esilon-contamination neighborhood of standard normal distribution, as defined in Huber 004. Hence a natural test statistic for the above hyotheses is the robust test for esilon-contaminated neighborhood. Before deriving the test statistics, we first examine the marginal distribution for ˆβ i under both the null and the alternative. Under the H 0 ɛ, it is easy to derive that ˆβ i equals in distribution to the following distribution ˆβ i = d z i u i s i n + z i γ i where u i N 0, σ. Therefore, ˆβi follows N 0, σ s i /n with ɛ robability, and with ɛ robability following some arbitrary distributions. For this reason, we say that under H 0 ɛ, ˆβ i is a random variable in the ɛ-contaminated neighborhood of N 0, σ s i /n. On the other hand, under the H a we have i, ˆβi = d N 0, σ + σ s i /n. To derive a likelihood-ratio tye statistic for ˆβ i under the null and the 7

8 alternative, we start with the likelihood ratio without ɛ-contamination. This likelihood ratio between N 0, σ to +σ s i /n and N 0, σ s i /n is roortional x x ex +. σ + σ s i n σ s i n Since the ratio is monotonically increasing in x, the normal variable squared, it is without loss of generality that we analyze only the squared variables according to Huber 004. Denote by P 0 the cumulative distribution function CDF of the square of the N 0, σ s i /n variable, and by P a the square of the N 0, σ +σ s i /n variable.3 Observe that both P 0 and P a are CDFs of some variables with different scaling factors. Write their resective densities as χ 0 xdx = e x σ s i /n dx and a xdx = e dx πσ s i /nx πvx x v where v = σ + σ s i /n. Every element in the ɛ-contamination neighborhood of P 0 can be written as ɛp 0 + F where F is a distribution over [0,. Since under the null, ˆβ i follows an ɛ-contaminated N 0, σ, s i /n, we have ˆβ i Q where Q is a CDF on [0, such that Qx ɛp 0 x x 0. For convenience, in the following of this section, we denote by H 0 ɛ the set of distributions {Q is a CDF on [0, Qx ɛp 0 x}. As in Huber 004, within H 0, we choose the following distribution rere- 3 We suress the index i here for simlicity. 8

9 sented by density q 0 ɛ 0 x for x x q 0 x = c a x for x > x for some constants x and c such that q 0 = and ax q 0 x = c. The next lemma shows that for each s i, there is a unique air of and c i that satisfies these restrictions. The would serve as a cut-off value to determine if ˆβ i is too large. For each ˆβ i, the log-likelihood ratio statistic between a and q 0 is σ s i /n v min{ ˆβ i, x i } u to a deterministic constant. We take the following normalization of the average statistic over i {,..., } as our test statistic. T := i= σ n σ n + σ s i min{ ˆβ i σ s i /n, σ s i /n}, where solves erf x σ s i /n + vi σ s ex i /n v i σ s i /n x x erfc = v i ɛ. The following Lemma describes the asymtotics of these cut-off values. Lemma Let v := σ + σ s /n. When v > σ s /n ɛ, there is a unique air of x and c that simultaneously solves the equations q 0 = and ax q 0 x = c, and x satisfies x σ s /n σ s + σ n vn 4 ɛ σ ln n s σ. π ɛ Let, s, σ and ɛ be functions in n. Suose ɛ 0, and for some constants κ, κ 0,, κ < σ < κ, and for some constants κ 3 > 0, s n ln ɛ < κ 3. Then the solution s bounded below by σ s + σ n σ n vn ɛ ln s σ ɛ C x σ s /n, 9

10 whenever C is some constants indeendent of, s, σ, ɛ and n. From now on, we define to be the solution of the equation ɛ erf x vi σ s i /n + ɛ σ s i /ne σ s x i /n v i x erfc =, v i where v i := σ + σ s i /n. When the solution does not exists, we set x i = 0. 4 Rejection Regions and Asymtotic Consistency Recall that in the test statistic T := i= σ n σ n + σ s i min{ ˆβ i σ s i /n, σ s i /n}, each term in the sum is derived from the likelihood ratio between the densities a and q 0 for ˆβ i under the alternative and the null resectively. Since the null contains a family of distributions each ˆβ i, the choice of the density q 0 is not arbitrary. Huber 004 showed that this articular choice ensures that the likelihood ratio between a and q 0 is a max-min statistic for each ˆβ i. In articular, when the design matrix is orthogonal, ˆβ i are indeendent conditional on β. In this case the test statistic T is max-min otimal. The exact distribution of T under the null is difficult to exress, however the following result allows us to simulate the rejection region. Theorem Under H 0 ɛ, T is first order stochastically dominated by S := i= σ n σ n + σ s i z i min{e i, σ s i /n} + z i σ s i /n, where z i iid Bernoulliɛ and n n n n e N 0, diag s,..., s X X diag s,..., s. 0

11 In articular, this first order stochastic uer bound is tight. Therefore, a roer alha level of the test can be defined as the region T t α, where PrS t α α. This region can be simulated. The order of the rejection region can be easily bounded using the Markov s inequality. Proosition 3 The random variable S is of order Proof. Observe that S = i= i= O E[S] O + ɛ i= σ s i /n σ n σ n + σ s i z i min{e i, e i + z i σ s i /n. Let Σ be the covariance matrix for e, we have E e i =E[e e] = E[e Σ / ΣΣ / e] =. i= for the trace of Σ is. Markov s inequality.. σ s i /n} + z i σ s i /n The rest of the roosition follows directly from Usually asymtotic consistency means a test rejects the null with robability aroaching as n diverges. However, since we allow both and the design matrix hence s i s to vary with n, the asymtotic consistency of this test means the rejection robability aroaches for a sequence of null and alternatives. In articular, the sequence of null is a sequence of models that are asymtotically sarse. Since can increase as n increases, the number of non-zero coefficients in the sequence of models can otentially increase as a result. However we need to avoid the athological case where the number of non-zero coefficients increases faster than n. Following Meinshausen and Buhlmann 006, Zhao and Yu 006 and Huang et al. 008, we assume the fraction of non-zero coefficients goes to zero, and the number of non-zero

12 coefficients grows at a rate less than one. Mathematically, we define asymtotic sarsity as follow. Condition 4 As n increases, there exist constants α > 0, α [0, such that ɛ = α n α. This condition has imlication for the test statistic under the null. The cut-off values are imlicitly affected by the above assumtion. Proosition 5 Let n, if there exists a ositive constant κ such that n s i κ for all i, then Condition 4 imlies that there exists c and c such that 0 < c < c and for all i, c ln n x i σ s i /n c ln n. And hence S = O + ɛc ln n = O asymtotically. Since under the null, S dominates the test statistic, therefore Condition 4 imlies the test statistic under the null is of finite order. To obtain a consistency, we can show that the test statistic diverges to infinity under the alternative when some sufficiency condition holds. One sufficient condition is that nλ κ ln n where λ is the minimal eigenvalues of X X/n. Since X X/n is a normalized, its minimal eigenvalue can be thought of as a measure of multile-colinearty of the design matrix. The above condition requires the effective number of observations er coefficient diverges slowly. Theorem 6 Let the minimal eigenvalues of X X/n be λ. Suose there exists some constant κ > 0 such that nλ κ ln n always holds. Suose Condition 4 holds. Then under H a, T diverges to in robability as n,. Hence the test is consistent.

13 5 Further Discussions In this section, we discuss three questions related to the alication of the test. They include the cases when σ is unknown, when > n and the choice of ɛ.. Unknown σ. When σ is unknown, we can lug in the residual mean-squared error ˆσ from OLS estimates. Denote the diagonal matrix by := diag n,..., n. s s The lug-in test statistics is then ˆT := i= ˆσ n ˆσ n + ˆσ s i min{ ˆβ i ˆ ˆσ s i /n, ˆσ s i /n}, where ˆ solves ɛ erf x ˆσ s i /n + ɛ ˆvi ˆσ s /ne v i ˆσ s x i /n i erfc x ˆv i =, and ˆv i = ˆσ + ˆσ s i /n. An alication of Theorem shows that under the null, the above test statistic is st order dominated by the following random variable. S := i= ˆσ n ˆσ n + ˆσ s i z i min{e σ i ˆσ, ˆ ˆ ˆσ s i /n} + z i ˆσ s i /n where z i iid Bernoulliɛ and e N 0, X X. Since σ is unknown, the rejection region is simulated from the random variable Ŝ := i= ˆσ n ˆσ n + ˆσ s i z i min{e i, ˆ ˆ ˆσ s i /n} + z i ˆσ s i /n The following sufficiency result shows the difference between S and Ŝ can be asymtotically negligible. Proosition 7 Suose ˆσ is a function in n that satisfies ˆσ σ = O/ n. Let be a diagonal matrix, := diag n,..., n and let z s s i iid Bernoulliɛ and e N 0, X X. Let the random variables Ŝ and S be defined as above. We have Ŝ S 0 as max i{s i } V arŝ n 0.. > n. 3.

14 For roblems involving a data set where > n, our test can be used as a ost-selection test. Under the null hyothesis of a sarse DGP, one can slit the data into two disjoint subsets erform any desired screening rocedures to the first subset. For examle, the Dantzig selector Candes and Tao 007 and Sure Indeendence Screening Fan and Lv 008 can be used to screen all the imortant variables while reduces the number of arameters to less than the number of observations. Methods to obtain a n-consistent estimate for σ is available in the literature as well. 4 Our test can be subsequently alied to the second art of the data. 3. Choice of ɛ. If one has some reconcetion about which sarsity level to test for, one can fix such ɛ level and erform the test. When there is little reconcetion about the sarsity level, our test can be turned into a confidence set about the sarsity of the underlying model. See emirical alication section for more detail. 6 Simulations In this section we rovide three simulation exeriments for our test. In each case we simulate datasets from the following model Y = Xβ + N 0, σ for various sizes and dimensions. In all of them, the covariates x i i =,..., n is simulated indeendently from a dimensional multivariate normal where is the dimension of β. The airwise correlation between x ij and x ik is 0.5 j k for all j, k =, Simulation Under Alternative In this simulation, we let β =,,,,,,,. In erforming the 5% level tests, we use ɛ = /3 for there are three larger variables. Other 4 See Reid et al. 06 for a survey and comarison of these estimators. 4

15 settings are the same as the revious simulation. The noise standard deviation σ =,, 3, 4, 5, 6, corresonding to SNRs 9.45, 4.86,.6,., 0.78 and For each n = 0, 50, 90, we conduct a 5% level test on 500 datasets and reort the rejection frequency. The horizon axis is the base- log of SNR, whereas the vertical axis is the test owers estimated from simulation. The color green, blue and urle corresonds to the samle size n = 0, 50, 90. From the figure we can see that the the ower generally increase as data size increases, and as SNR increases. 6. Simulation Under Alternative In this simulation, we let = 40 and the vector β consists of a five-time reetition of the sequence,,,,.5,.5,,. The correlation structure of the redictors is as before excet now each x i is of dimension 40. For each simulation, we first aly the LASSO in conjunction with BIC to the data and record how many arameters are selected. In order to better illustrate the ower of our test, each time we aly the test by setting ɛ equal to the ratio of number of selected to the total number of arameters 40. We reort both the mean number of selected rounded to nearest integer and the rejection robability rounded to nearest ercentage of the BIC minimizing sarsity level. Different combinations of σ and n are used with σ = 4, 6, 8, 0 corresonding to aroximate SNRs,, 0.5, 0.3. The number of observations ranges from n = 80, 0, 00. Under each arameter setting, 500 simulations are made and the results are resented in the following table. n=80 n=0 n=00 Noise Select Reject% Select Reject% Select Reject% σ = σ = σ = σ = From the above table, one can see that the BIC selection is close to true model only when the noise level is low and observation is large. Our test do not reject the selection in this case with % rejection since the selection is 5

16 close to the true model. When the samle size is moderate, the BIC misses more than half of the variables and our test rejects the selected sarsity levels with large robability 50% or more. When σ 6, the signal to noise level is less than or equal to. Under this arameter setting, the selection misses most of the variables. Our test reject the selection with close to robability for moderate samle sizes. For small samle sizes, our test still rejects the selected sarsity level with fair robability. 7 Emirical Illustrations 7. Illustration I We aly our test to the a cross country growth data set. In the data subset, there are observation on 35 countries each with observation of 67 characteristics lus the resonse variable, GDP growth rate from 60 to 96. Since there are many missing observations in the data, we aly our test to only a subset of the samle. We use the subset of the samle where there is no missing observations on the following 8 variables East Asian dummy EAST Primary schooling 960 P60 Investment rice IPRICE GDP 960 log GDPCH60L Fraction of troical area TROPICAR Poulation density coastal 960 s DENS65C Malaria revalence in 960 s MALFAL66 Life exectancy in 960 LIFE060 Fraction Confucian CONFUC African dummy SAFRICA Latin American dummy LAAM Fraction GDP in mining MINING Sanish colony SPAIN Years oen YRSOPEN Fraction Muslim MUSLIM00 Fraction Buddhist BUDDHA Ethnolinguistic fractionalization AVELF Government consumtion share 960 s GVR6 which are a number of economic and olitical factors, geograhical and historical dummies, and several demograhic characteristics that were described 6

17 as otential imortant factors in exlaining long-run GDP growth in Sala-I- Martin et al There are 94 observations that have no missing observations in the above listed variables. These countries or regions are Algeria Benin Botswana Burkina Faso Burundi Cameroon Cent l Afr. Re. Congo Egyt Ethioia Gabon Gambia Ghana Kenya Lesotho Liberia Madagascar Malawi Mali Mauritania Morocco Niger Nigeria Rwanda Senegal Somalia South Africa Tanzania Togo Tunisia Uganda Zaire Zambia Zimbabwe Canada Costa Rica Dominican Re. El Salvador Guatemala Haiti Honduras Jamaica Mexico Nicaragua Panama Trinidad & Tobago United States Argentina Bolivia Brazil Chile Colombia Ecuador Paraguay Peru Uruguay Venezuela Bangladesh Hong Kong India Indonesia Israel Jaan Jordan Korea Malaysia Neal Pakistan Philiines Singaore Sri Lanka Syria Taiwan Thailand Austria Belgium Denmark Finland France Germany, West Greece Ireland Italy Netherlands Norway Portugal Sain Sweden Switzerland Turkey United Kingdom Australia New Zealand Paua New Guinea 7

18 Many economic models focus analyses on a coule of factors and their relation with long-run growth. For examle Sala-I-Martin et al. 004 focuses their arguments on rimary schooling enrollment, investment rice and initial GDP levels. Therefore, in this numerical exercise we will set the sarsity arameter to be ɛ = 3/8, interreted as whether the variation in longrun growh can be sufficiently exlained by 3-variable linear regression model. We simulate 0k random draws from the uerbound distribution of the null see Thm 7. The 5% rejection is defined as the uer 5% quantile of the simulated samle. The test statistic calculated from the data is above the 5% quantile and has a -value of less than.7%. Hence we reject the null that the cross country long run GDP growth can be exlained by a three factors or fewer sarse linear model, and accet the alternative that a non-sarse model of multile 8 small effects is better suorted by the data. It might be interesting to know which variables ass our robust thressholds. They are Primary schooling enrollment in 960, initial GDP level 960, investment rice, life exectancy in 960 and fraction of GDP in mining. One can interret it as an indication that these variables may be more imortant than others in determining long run GDP growth. Although we set the benchmark case to be ɛ = 3/ from a modelling ersective, it would be interesting to see how the test would conclude if we aly a less strict sarsity arameter. To this end we reort the -values for several different sarsity below. ɛ value % The above -values shows strong evidence for at least 9 non-zero variables, indicating that the underlying DGP is not sarse. Nonetheless, it does not contradicts the roosal of Sala-I-Martin et al. 004 that rimary schooling, initial GDP level and investment rices are very imortant factors. It suggest that long-run GDP growth is a comlex high dimensional object that is affected by many different country-level characteristics. 8

19 7. Illustration II Ludvigson and Ng 009, 00 found that the excess return of U.S. government bonds is redictable using macroeconomic fluctuations. They found that macroeconomic fundamentals contain information about risk remia beyond those embedded in bond market data. In this section, we aly our test to their rediction roblem. The macro-factor data is taken from the udated data file is rovided on Ludvigson s website. The eight factors, f,..., f 8 each have different interretations based on their loading of the samle series. According to Ludvigson 009, f is the factor of economy activity in real-terms; f loads on interest rate sreads; f 3 and f 4 are rice factors; f 5 is mainly a combination of interest rates but not so much of interest rate sreads; f 6 loads on housing; f 7 on money suly; and f 8 loads mainly on stock-related series. The resonse variable r n t+, the continuously comounded log excess return on an n-year discount bond in year t+, is also taken from Ludvigson s website. The resonse data san from -year excess return to 5-year excess return. Due to the availability of the resonse series and the lag- month regression, we have in total 468 observations. Following their aers, we regress r n t+ on CP t, the forward rate factor used in Cochrane and Piazzesi 005, and the eight macro factors lus their interaction terms u to the third order. In other words, as redictors we have CP, f,... f 8 and all of the f i f j, and f i f j f k where i j k 8, totaling to 09 redictors. Ludvigson and Ng 009 uses BIC and searched through low dimensional models, and conclude that the best model consists of CP, f, f 3, f 3, f 4 and f 5. Our test results for excess return for different eriods are reorted below. ɛ resonse: r t resonse: r 3 t resonse: r 4 t resonse: r 5 t Table : -values % for various sarsity levels 9

20 Overall, the 5% level test rejects extremely sarse models. The 95% confidence interval varies, covering from as less as models with dimensions 7, to as much as models of dimensions 5. The size of the model found by Ludvigson and Ng 009 barely lies in these confidence intervals. A closer examination of the confidence interval also indicate that the longer the maturity of the Treasury bonds, the sarser the regression model becomes. Potentially short term returns can be affected by more factors whereas in the longer terms, only the most imortant facts has lasting effects. Nonetheless, our test suorts the use of a sarse model to redict such excess returns. 8 Aendix: Proofs 8. Proof of Lemma Proof. We devide the roof into three stes. The first ste is to show there is a unique solution. The second ste is to construct an uerbound and the third ste is to construct a lower bound. Ste. We show the set of equations have a unique solution. Observe that ax q 0 x is now increasing in x and reaches maximum /c on [x,. Let v := σ + σ s /n. The relation between x and c can be solved from ax q 0 x = c to be x ex σ s /n x v = ɛ c v σ s /n. So there is an inverse relationshi between x and c. q0 = we get Substituting into x 0 ɛ 0 tdt + ɛ v σ s /n e v σ s x /n x a tdt =. By differentiating the LHS with resect to x, we see that the LHS becomes 0

21 e x σ s /n v ɛ πσ s /nx ɛ σ s /n e v σ s x e x v /n πvx v + ɛ σ s /n e v σ s x /n v σ s a sds /n x v = ɛ σ s /n e v σ s x /n v σ s a sds < 0, /n because v > σ s /n. Since the LHS of the equation is decreasing in x, in order for a solution to exist, it is necessary that when x = 0 we have LHS >. In otherwords, x ɛ > σ s /n v v > σ s /n ɛ. This is guarrenteed when ɛ 0. Before ste, we observe from the x equation that x ɛ 0 ɛ e t/σ s /n πσ s /nt dt + ɛ x σ s /n 0 x ɛ erf σ s /n e t dt + ɛ π + ɛ v σ s /n e v σ s /n e v σ s /n e v x σ s /n x /v v x σ s /n v x σ s /n x v e t/ πt dt = π e t dt = x erfc =. v where erfcx := π x e t dt and erfx := erfcx. Ste. We now construct an uer bound. To construct the uerbound for x, we first substitute the following into the LHS of the above equation x = vσ s /n vn σ / ln a s σ = σ s n σ s + σ n σ n vn a ln s σ,

22 where a deends on ɛ and is to be determined later. We have LHS = ɛ erf where ξ = σ s σ n + ξ ln a + /ξ + ɛa / erfc ξ ln a + /ξ. Since LHS is monotonically decreasing in x, it suffices to show that for certain a, LHS <. Then we conclude that the solution for x would be less than this value. In the following, we use the following bounds for the function erfc derived from Formula 7..3 of Abramowitz and Stegun 964. When x 0, π e x x + π e x x + x + < erfcx π e x x + x + 4/π < e x π x, and in articular, it is well-known that erfcx π e x for x 0. LHS ɛ = erf + ξ ln a + /ξ + a / erfc ξ ln a + /ξ ex + ξ ln a + /ξ π + a / π ex ξ ln a + /ξ + + ξ ln a + /ξ = a / π Now let a satisfy a / π = LHS ɛ = ɛ ex + /ξ / ex ξ ln a + /ξ + + ξ ln a + /ξ + ξ ln a + /ξ ɛ + ɛ ex ξ ln a + /ξ ɛ = + ɛ ex ξ ln a + /ξ ɛ ɛ ɛ, i.e. a = 4 π ɛ, ɛ and get a / π + /ξ + + ξ ln a + /ξ + /ξ + + ξ ln a + /ξ ex ξ ln a + /ξ

23 Since a > for all ɛ small enough and ξ > 0, we have ex ξ ln a + /ξ, and 0 < and therefore Hence we conclude that LHS < ɛ + x σ s /n σ s + σ n σ ln n + /ξ <, + + ξ ln a + /ξ ɛ ɛ =. vn 4 s σ π ɛ. ɛ Ste 3. Now we establish an lowerbound for x. To do this, we show that by substituting in x = vσ s /n vn σ / ln a s σ = σ s n σ s + σ n σ n vn a ln s σ, for some other values of a deending on ɛ than before, we have LHS > 3

24 asymtotically. As before, we start with LHS ɛ = erf + ξ ln a + /ξ + a / erfc ξ ln a + /ξ ex + ξ ln a + /ξ π = a π + /ξ a π + /ξ = + a e ξ lna+/ξ π = + a +ξ e ξ ln+/ξ π + a / erfc + ξ ln a + /ξ ξ ln a + /ξ ex ξ ln a + /ξ + a erfc + ξ ln a + /ξ ξ ln a + /ξ ex ξ ln a + /ξ a + + ξ ln a + /ξ π / + ξ ln a + /ξ ex ξ ln a + /ξ / + / + ξ ln a + /ξ By assumtion, ξ 0. We have e ξ ln+/ξ π / π from below. In order to give a lower bound for the LHS, we also need to bound the following term from below. = = / + Let a satifies ξ ln a + /ξ + ξ ξ ξ ln a + /ξ ξ ln a + /ξ ξ / ξ ln a + /ξ. ξ ln a + /ξ / + ξ ln a + /ξ + ξ / + ξ ln a + /ξ + ξ a / e κ κ κ 3 / + κ κ κ 3 + κ 3 + κ 3 = ɛ ɛ. ξ ln a + /ξ + /ξ + ξ ln a + /ξ ξ. + ξ ξ ln a + /ξ 4

25 It is clear that a ɛ. ɛ Additionally, since ξ = σ s σ n and κ < σ < κ ln ɛ < κ 3, we have and s n / + ξ ln a + /ξ + ξ ξ ln a + /ξ / + κ κ κ 3 + κ 3 + κ 3 where the last inequality holds asymtotically. Again since κ < σ < κ and s n ln ɛ < κ 3, it follows that LHS > ɛ + a +ξ e ξ ln+/ξ π / + κ κ κ 3 + κ 3 + κ 3 > ɛ + a / e κ κ κ 3 / =. + κ κ κ 3 + κ 3 + κ 3 This shows that x σ s /n σ s + σ n vn σ ln n s σ a, where a = ɛ ɛ C for some constant C. 8. Proof of Theorem Proof. By definition, n n n n diag σ s,..., ˆβ σ s = d e + diag σ s,..., σ s β 5

26 where e is as defined above and β i = z i γ i where γ i F for some F F. Consider the i-th term in the test statistic ˆβ i min{ σ s i /n, σ s i /n}. Under the null hyothesis, conditional on z i = 0, β i = 0 for β i = z i γ i. And ˆβ i x i n min{ σ s i /n, σ s i /n} = min{e i + e i σ s i β i + β i σ s i /n, σ s i /n} = z i min{e i, σ s i /n} + z i σ s i /n. On the other hand, conditional on z i =, we have ˆβ i min{ σ s i /n, σ s i /n} x i σ s i /n = z i min{e i, σ s i /n} + z i σ s i /n. Since this holds for each i =,...,, we conclude that under H 0 ɛ, T st S. To see the bound is tight, consider in H 0 ɛ a sequence of {F k } k N F that diverges to infinity: for all k N, F k k F k k = 0. For each i =,...,, conditional onz i = 0, ˆβ i min{ σ s i /n, σ s i /n} = z i min{e i, σ s i /n} + z i σ s i /n as before. Conditional on z i =, min{ ˆβ i σ s /n, i σ s i /n} converges in robability to σ s i /n = z i min{e i, σ s /n} + z i i σ s i /n for β i = γ i converges in robability to infinity along the sequence F k. Since such a convergence holds for all i =,...,, the statistics T converges in distribution to S along F k. 8.3 Proof of Proosition 5 We first introduce a lemma. Lemma 8 Let the minimal eigenvalues of X X/n be λ and let the ith diagonal entry of X X/n be s i. Then s i /λ for all i =,...,. 6

27 Proof. Let λ i for i =,..., be the eigenvalues of X X/n. Write the eigenvalue decomosition as X X/n = QΛQ where Λ = diagλ,..., λ. The ii-th entry of X X/n is = j= Q ij λ j. Similarly for X X/n = QΛ Q, we have s i = j= Q ij /λ j. By the inequality of weighted harmonic mean and arithmetic mean, we have s i = j= Q ij /λ j Q ijλ j =. j= The other inequality follows from Schur-Horn Theorem see Schur 93 and Horn 957 that /λ max i {s i }. Now we roceed to the roof of Proosition 5. Proof. On one hand,by Lemma, for some c > 0. x i v i n ɛ +b σ s ln i /n s i σ ɛ + b ln ɛ c ln n On the other hand, by Lemma, we have x i σ s i /n = O vi n ɛ σ n ln s i σ ɛ O ln σ + ɛ = Oln n, where in the inequality we used s i from the Lemma 8, and in the last equality we used the assumtion of asymtotic sarsity. 8.4 Proof of Theorem 6 We need to first reare a lemma. This lemma may be of interest in its own. It states that the emirical distribution of ˆβ i /v i converges to the cdf of the χ distribution. Lemma 9 Let the minimal eigenvalues of X X/n be λ and let v i = σ + σ s i n nλ. Suose there exists some constant κ > 0 such that κ. Then under 7

28 the alternative, the emirical distribution of ˆβ i /v i converges to the cdf of χ as n. Proof. Since under the alternative, the asymtotic distribution for ˆβ is ˆβ N 0, σ I + σ X X. Under scaling by D := diag v /,..., v /, the distribution becomes D ˆβ N σ 0, D I + σ X X D. It is clear that in the above variance matrix, all entries on the main diagonal are. Let the minimal eigenvalues of X X/n be λ, it is clear that for any δ > 0, σ σ D I + σ X X D D + σ I D nλ σ + σ /κ σ I. Let the eigenvalues of the variance matrix of D ˆβ be r,... r. The normalized Frobenius norm of the above variance matrix is given by σ / D I + σ X X D := ri i= σ + σ / /κ σ 0 as. Now we aly Theorem of Azriel and Schwartzman 05, and conclude that the emirical distribution of the entries of D ˆβ converges to the standard normal, and hence the emirical distribution of ˆβ i /v i converges to χ as n,. Now we roceed to the roof of Theorem 6. Proof. We have seen that Lemma 8 imlies s i /n /κ. Let v i := σ + i= 8

29 σ s i n. There exists a constant K 0 > 0, such that T = σ n σ n + σ s i min{ ˆβ i σ s i /n, σ s i /n} ˆβ i K 0 min{ σ s i /n, σ s i /n} K 0 σ s i /n K 0 c ln n. i= i= ˆβ i v i x i v i ˆβ i v i x i v i Since Lemma and Lemma 8 imlies there exists constant K that for all i, σ v i σ nλ ln + σ nλ ɛ σ ɛ K nλ ln + σ nλ σ + K nλ ln n K /κ as nλ ln by the Condition 4 and the assumtion that nλ κ ln n. Therefore + σ nλ σ 0 T K 0 c ln n Prχ K /κk 0 c ln n, ˆβ i v i K /κ where ˆβ i K v /κ Prχ K /κ by Lemma 9. Since we have shown i reviously that the test statistics under the null is of order O, the roof is comlete. 8.5 Proof of Proosition 7 Before roving the roosition, we need to reare the following lemma. Lemma 0 Let c, c be two ositive constants, and the random vector x, y t 9

30 [ ] ρ N 0, M where M :=. For any c, c 0, we have Cov min{x, c }, min{y, c } ρ 0. Proof. The density of x, y can be written as fxfy x = ex x π ex y ρx ρ π ρ By definition, Cov min{x, c }, min{y, c } =E [ min{x, c } min{y, c } ] E[min{x, c }]E[min{y, c }] ex = x ex y ρx min{x, c } min{y ρ, c } dydx E[min{x, c }]E[min{y, c }] R π R π ρ = min{x, c } min{ ρ s + ρx, c }dφs dφx E[min{x, c }]E[min{y, c }] R R = min{x, c }hxdφx min{x, c }dφx min{y, c }dφy R R where Φ is the standard normal c.d.f., and hx := R min{ ρ s + ρx, c }dφs. It is clear that hx = h x, we can write hx = h x. So with a change of variable x = t, we have R Cov min{x, c }, min{y, c } = min{t, c }h tdχ t R R min{t, c }dχ t min{t, c }dχ t R where χ is the Chi-square c.d.f of one degree freedom. Since we have R h tdχ t = R min{y, c }fxfy xdydx = R min{y, c }dφy, Chebyshev s sum inequality imlies Cov min{x, c }, min{y, c } > 0 as long as if h t is an increasing function in t for t > 0. To see h t is indeed increasing, observe that if s is a standard normal random variable, then s + ρ ρ t follows a noncentral Chi-squared dis- 30

31 ρ t ρ tribution χ with noncentrality arameter ρ t. Since the noncentrality arameter has the monotone likelihood ratio roerty, it follows that ρ for any 0 t < t, the distribution χ ρ t ρ first order stochastically dominates the distribution χ ρ t ρ. Since h t := where s := R h t is increasing in t. min{ ρ s + ρ t, c }dφs = E[min{ ρs, c }] s + ρ ρ t is some χ ρ t ρ Now we can roceed to the roof. ˆσ n ˆσ n+ˆσ s i random variable. Hence Proof. Since for each i, is bounded above by, Ŝ S i= e i σ. Since ˆσ is n-consistent, Ŝ ˆσ S is of order / n using Cauchy Schwarz. For the rest of the roof, it suffices that to show that V arŝ is of order at ˆσ least /. For i =,...,, denote S i := n ˆσ n+ˆσ s z i min{e i, ˆ i ˆσ s /n} + z i i By definition V arŝ = i,j= CovS i, S j. When i j, we have ˆ ˆσ s i /n. ˆσ n ˆσ n + ˆσ s i ˆσ n ˆσ n + ˆσ s j CovS i, S j ˆ =Cov z i min{e i, ˆσ s i /n} + z i ˆσ s i /n, z j min{e j, =Cov z i min{e ˆ i, ˆσ s i /n}, z j min{e ˆx j j, ˆσ s j /n} =Cov min{e i, ˆ ˆσ s i /n}, min{e j, ˆ ˆx j ˆσ s j /n} 0 where the last inequality follows from Lemma 0. Therefore ˆx j ˆσ s j /n} + z j ˆx j ˆσ s j /n V arŝ = CovS i, S j i,j= V ars i. i 3

32 ˆx Since i ˆσ s i /n is uniformly bounded away from 0 by Lemma. And as max i{s i } n ˆσ 0, we have n ˆσ n+ˆσ s i uniformly over i. Therefore V ars i C for some constant C > 0. Hence V arŝ i= C = C. This comletes the roof. References [] Abramowitz, M. and Stegun, I.A. 97. Handbook of Mathematical Functions,Dover Publications, 98. [] Akaike, H A new look at the statistical identification model. IEEE. Trans. Auto. Control [3] Azriel, D., and A. Schwartzman 05 The Emirical Distribution of a Large Number of Correlated Normal Variables Journal of the American Statistical Association 0:5, 7-8 [4] Candes, E., and T. Tao 007 The Dantzig selector: Statistical estimation when is much larger than n Ann. Statist. Volume 35, Number 6, [5] Chen, Jiahua, and Zehua Chen 008 Extended Bayesian Information Criteria for Model Selection with Large Model Saces. Biometrika, Volume 95, Issue 3. Se [6] Cochrane, J. H., and M. Piazzesi Bond Risk Premia. American Economic Review 95:38 60 [7] Fan, J., and Li, R. 00, Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Proerties, Journal of the American Statistical Association, 96, [8] Fan, J. and Lv, J. 008 Sure indeendence screening for ultrahigh dimensional feature sace J. R. Statist. Soc. B 70, [9] Giannone, D., M. Lenza and G.E. Primiceri 07 Economic Predictions with Big Data: the Illusion of Sarsity working aer [0] Horn, A. 954, Doubly stochastic matrices and the diagonal of a rotation matrix, American Journal of Mathematics 76,

33 [] Huang, J., Ma, S. and Zhang, C., 008 Adative LASSO for Sarse High-dimensional Regression Models, Statistica Sinca 8, [] Huber, P. 004 Robust Statistics. New Jersy. John Wiley & Sons, Inc. [3] Hsu, D., Kakade, S. M. and Zhang, T 04. Random Design Analysis of Ridge Regression Found. Comut. Math [4] Jaynes, E.T Prior Probabilities. IEEE Trans. on Systems Science and Cybernetics. 4 3: 7. [5] Ludvigson, S. and S. Ng, 009 Macro Factors in Bond Risk Premia. The Review of Financial Studies, 009, : [6] Ludvigson, S. and S. Ng, 00 A Factor Analysis of Bond Risk Premia. Handbook of Emirical Economics and Finance, 00, e.d. by Aman Uhla and David E. A. Giles, [7] Meinshausen, N. and Buhlmann, P High dimensional grahs and variable selection with the Lasso. Ann. Statist [8] Mitchell, T. J., and J. J. Beaucham 988 Bayesian Variable Selection in Linear Regression, Journal of the American Statistical Association, 83, [9] Reid, S., R. Tibshirani and J. Friedman 06. A Study of Error Variance Estimation in LASSO Regression. Statistica Sinica Vol. 6, No [0] Schur, I. 93 Uber eine Klasse von Mittelbildungen mit Anwendungen auf die Determinantentheorie, Sitzungsber. Berl. Math. Ges., 9 0. [] Schwarz, G. E. 978, Estimating the dimension of a model, Annals of Statistics, 6 : [] Tibshirani, R Regression Shrinkage and Selection via the lasso. Journal of the Royal Statistical Society. Series B methodological. Wiley. 58 : [3] Zhang, C. H. and Huang, J The sarsity and bias of the Lasso selection in high-dimensional linear regression. Annals of Statistics, 36,

34 [4] Zhao, P. and Yu, B On model selection consistency of LASSO. J. Machine Learning Research [5] Zou, H The adative lasso and its oracle roerties Journal of the American statistical association 0 476, [6] Zou, Hui and Hastie, Trevor 005. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society, Series B: [7] Zou, H., Hastie, T. and Tibshirani R. 007 On the degrees of freedom of the lasso The Annals of Statistics Volume 35, Number 5,