Statistics and Probability Letters. Variance stabilizing transformations of Poisson, binomial and negative binomial distributions

Transcription

1 Statistics and Probability Letters 79 (9) 6 69 Contents lists available at ScienceDirect Statistics and Probability Letters journal homeage: Variance stabilizing transformations of Poisson, binomial and negative binomial distributions u Guan Deartment of Statistics, Zhejiang Forestry University, Lin an Zhejiang, 33, China a r t i c l e i n f o a b s t r a c t Article history: Received 4 February 8 Received in revised form 5 Aril 9 Acceted 7 Aril 9 Available online 3 May 9 Consider variance stabilizing transformations of Poisson distribution π(λ), binomial distribution B(n, ) and negative binomial distribution NB(r, ), with square root transformations for π(λ), arcsin transformations for B(n, ) and inverse hyerbolic sine transformations for NB(r, ). We will introduce three terms: critical oint, domain of deendence and relative error. By comaring the relative errors of the transformed variances of π(λ), B(n, ) and NB(r, ), and comaring the skewness and kurtosis of π(λ), B(n, ) and NB(r, ) and their transformed variables, we obtain some better transformations with domains of deendence of the arameters. A new kind of transformation (n + )/ sin ( n ) for B(n, ) is suggested. 9 Elsevier B.V. All rights reserved.. Introduction Let be a random variable with mean E() = µ and variance Var (). If the relationshi between µ and Var () was known, we could use this information to find a variance stabilizing transformation Z = T() such that Var (Z) C (constant). T() is exanded at the oint = µ into a Taylor series: T() = T(µ) + T (µ)( µ) + o( µ). By solving the differential equation T (µ) CVar () /, we obtain T(). In Proosition, we list some well-known transformations (see Chatterjee et al. () and Montgomery (5)). Proosition. () If π(λ) (Poisson distribution), T() = with Var ( ) 4. () If B(n, ) (binomial distribution), T() = n sin /n with Var ( n sin ( /n)) 4. (3) If NB(r, ) (negative binomial distribution), its frequency function is P( = k) = ( k+r r ) r ( ) k (k =,,,..., r > ), and then T() = r sinh /r with Var ( r sinh /r) 4. Bartlett (936, 947) first introduced variance stabilizing transformations, and roosed the transformation +.5 for π(λ). Anscombe (948) showed that + 3/8 is the most nearly constant variance transformation for π(λ) when has a larger mean λ, and sin +3/8 for B(n, ) similarly. Freeman and Tukey (95) suggested combined n+3/4 transformations ( + + ) for π(λ) and (n + )/ [sin + n+ sin + ] for B(n, ). The method n+ of combined transformation was generalized to NB(r, ) by Laubscher (96). Thacker and Bromiley () and Bromiley and Thacker () investigated the effects of stabilizing transformations on a Poisson distributed quantity and a binomial distributed quantity. Uddin et al. (6) resented a necessary condition for a variance stabilizing transformation to be an Tel.: address: guanyu@zjfc.edu.cn /$ see front matter 9 Elsevier B.V. All rights reserved. doi:.6/j.sl.9.4.

2 6 G. u / Statistics and Probability Letters 79 (9) / (+.) / [ / +(+) / ]/ (+.385) /.6.55 / (+.) / (+.385) / variance (+.385) / (+) / [ / +(+) / ]/ variance (+) / [ / +(+) / ]/ λ λ Fig.. Five transformed variance curves on π(λ) for λ [.5, ] (left anel) and λ [, ] (right anel). aroximate symmetrizing transformation. For more general data, the Box Cox transformation is often used (see Box and Cox (964), Box and Cox (98) and amamura (999)). For π(λ), Anscombe (948) showed that Var ( + a) = 3 8a { + + O(λ )} for large mean λ and 4 8λ 4 any constant a. On the other hand, it is clear that lim λ Var ( + a) = for any constant a. This imlies that + a is only a local variance stabilizing transformation. There is the same case for binomial variables and negative binomial variables. Our task is to find those transformations such that variances of transformed variables change less in larger domains of deendence, i.e. domains of arameters. Since exact formulas for transformed variances could not be obtained, numerical methods are alied to calculate these variances of various transformed random variables. By virtue of mass numerical comutation, we comare fluctuations of these variances and obtain some better transformations with their domains of deendence. The selection criterion is less fluctuation of variances with resect to a larger domain of deendence of arameters. In this aer, we will comute arameters λ, and a to the third lace of decimals. In Section, we study transformation + a and its combined transformations for π(λ), and introduce three concets: left critical oint, domain of deendence and relative error, to describe the stabilization of a transformation. In Section 3, we show that a new kind of transformation (n + )/ sin ( n ) is equivalent to (n + )/ sin +a for B(n, ). We also give the limit relation between NB(r, ) and π(λ), and investigate three transformations for NB(r, ) in Section 4. In the last section, the skewness and kurtosis of these three kinds of variables and variables transformed by various transformations are comuted and comared. Some better variance stabilizing transformations with domains of deendence of arameters on π(λ), B(n, ), NB(r, ) are suggested.. Poisson distribution Proosition. Let π(λ) and a > a ; then Var ( + a ) < Var ( + a ). Proof. Let δ = a a > and k = P( = k) = λk k! e λ (k =,,,...). We have E[( + a i ) ] = λ + a i (i =, ), and Var ( + a ) Var ( [ ] + [ ] + + a ) = δ k + a k k + a k k= k= < δ + δ k k= m>n δ m n = δ δ ( + k= k ) =. () Here, (m + a )(n + a ) (m + a )(n + a ) > δ, since [ (m + a )(n + a )] [ (m + a )(n + a ) + δ] = δ (m + a + n + a ) δ (m + a )(n + a ) >. Remark. Proosition tells us that Var( + a) is a monotone decreasing function of a. Note that there is only one constant a at most such that Var ( + a ) =.5 for each λ. By numerical comutation, we obtain the following (see Fig. and Table ).

3 G. u / Statistics and Probability Letters 79 (9) Table Variances under transformation by + a. a λ M V M.5 λ m V m.5 λ L V Table Some better variance stabilizing transformations on π(λ). a λ L (, ) (,.5) (,.3) (,.75) (., ) (., ) (,,.8) () When a [,.375], there exists a secial oint λ (a) such that Var ( (λ) + a) is a monotone increasing function of λ (, λ (a)] and a monotone decreasing function of λ (λ (a), + ). () When a [.376,.39] and the ositive arameter λ increases, Var ( (λ) + a) at first shows monotone increase, then monotone decrease, then monotone increase, and so on. (3) When a.393, Var ( (λ) + a) is a monotone decreasing function of λ (, + ). According to the characteristic of Var ( (λ) + a) curves, for a.39 we introduce a concet left critical oint λ L which satisfies the following conditions: () There exist both a maximum and a minimum of Var ( (λ) + a) for λ (λ L, ], but no maximum and no minimum of Var ( (λ) + a) for λ (, λ L ). () Var ( (λ L ) + a) Var ( (λ m ) + a) > Var ( (λ L.) + a). (3) Var ( (λ m ) + a) = min{var ( (λ) + a) λ (λ L, ]}, Var ( (λ M ) + a) = max{var ( (λ) + a) λ (λ L, ]}. When λ [λ L, ], the fluctuation of Var ( (λ) + a) is less. When λ (, λ L ) decreases, Var ( (λ) + a) decreases violently. Interval [λ L, ] or [λ L, + ) is called the domain of deendence of arameter λ. In order to describe the fluctuation of variances recisely, we define the relative error of the variance for arameter θ Θ as follows: = max { Var (θ ) Var (θ ) } θ,θ Θ. Var Here Var is the limit value of Var (θ) (θ Θ). For π(λ), B(n, ) and NB(r, ), by Proosition their Var are all /4. In Table, V M = Var (λ M ) + a, V m = Var (λ m ) + a, V λ = Var (λ) + a. The last column denotes the relative error for λ [λ L, ]. The last row denotes the transformation +.5; its variance is a monotone increasing function of λ >, and this means that +.5 is not a better transformation. When λ 5, the left critical oint of is the most roximal to 5, and thus is almost the best variance stabilizing transformation of π(λ) for λ 5. And so is + 3/8 for λ 4. Now we discuss the combined transformation b +a +b +a +a +b +a b +b () (b + b ). After much calculation, we are convinced that the combined transformation b might be seen as a comromise between b +b + a and + a. For examle, + + imroves the variance stabilization of and + extremely.

4 64 G. u / Statistics and Probability Letters 79 (9) n / sin [( n)/n]+(n+) / sin [( n)/(n+.5)]] / (n+.5) / sin [( n)/(n+.4)] variance (n+.5) / sin [( n)/(n+.77)] (n+.5) / [sin [( n )/(n+)]+sin [( n+)/(n+)]] /.75.7 (n+.5) / sin [( n)/(n+)] Fig.. Five transformed variance curves on B(n, ) for n = 5 and [.,.5]. Table lists some better transformations and combined transformations for a Poisson variable, where a, (a, a ) and (a, a, b) resectively denote the transformations + +a + +a a, error of the transformed variances λ [k, ] (k = 5, 4, 3,,,.5). +a +b +a and, and +b k reresents the relative 3. Binomial distribution Proosition 3. Let B(n, ); then T() = sin ( n ) is a variance stabilizing transformation and Var (sin ( n )) n n /n. Proof. We have dt d = n n { ( ) } / = n n { ( ) n n n } /. Therefore Var (sin ( n )) /n. n Proosition 4. Let B(n, ), T (, a) = sin +a, T (, a) = sin ( n ) and a [, ]. Then: () T (n, a) = π/ T (, a), T (n, a) = T (, a); () T (, a) = T (, a) π/; (3) Var (T i (n, a)) = Var (T i (, a)) (i =, ), Var (T (, a)) = 4Var (T (, a)). Proof. () Considering sin {sin ( +a )/ } + sin {sin ( n +a )/ } =, we have T (n, a) = π T (, a). Similarly, T (n, a) = T (, a). () Considering cos(t (, a)) = sin (T (, a)) = sin(t (, a)), we have T (, a) = T (, a) π/. (3) By items () and (), the equalities of item (3) are easily obtained. Remark. By Proosition 4, sin ( n ) is a negative symmetrical transformation with axis of symmetry = n/ and is equivalent to sin +a for the variance transformation for B(n, ). But the former formula is simler than the latter, so in this aer we suggest (n + )/ sin ( n ) with aroximate variance instead of (n + )/ sin n. It is well-known that π(λ) can be derived as the limit of B(n, ) as n aroaches infinity and aroaches zero in such a way that n = λ. Therefore, we study B(n, ) by numerical comutation like π(λ). While n and are fixed, the variance transformed by (n + )/ sin ( n ) is a monotone decreasing function of addend a also. See Fig. and Table 3. Freeman and Tukey (95) suggested the combined transformation (n+ )/ [sin n+ +sin + for B(n, ). n+ Laubscher (96) roosed the transformation n / sin n +(n+)/ sin +3/4 ]. Analogously, we can convert these n+3/ two transformations into (n + )/ [sin ( n ) + sin ( n+ )] and [n / sin ( n ) + (n + ) / sin ( n )] with n+ n+ n n+3/ aroximate variances 4. Table 3 lists four better transformations (n + )/ sin ( n ), (n + n+.77 )/ sin ( n ), (n + n+.75 )/ [sin ( n ) + n+ sin ( n+ )] and [n / sin ( n )+(n+) / sin ( n )] with their numerical results, where arameter is comuted n+ n n+3/ to the fourth lace of decimals when n =.

5 G. u / Statistics and Probability Letters 79 (9) (r.5) / sinh [(+.375)/(r.75)] / variance (r.5) / sinh [(+.385)/(r.77)] / (r.5) / sinh [(+.39)/(r.78)] / Fig. 3. Three transformed variance curves on NB(r, ) for r = and [.,.77]. Table 3 Some better variance stabilizing transformations on B(n, ). a n n L / / / / (, ) (, ) (, ) (, ) (, 3/4) (, 3/4) (, 3/4) (, 3/4) Negative binomial distribution Proosition 5. Let NB(r, ) and lim r + r( ) = λ (ositive constant); then lim r + (k+r r ) r ( ) k = λk k! e λ for all k. (3) Proof. When n +, n! π n n+/ e n. Then ( k+r r ) r ( ) k (k+r )k+r / e k (r ) r / r k k r )r / e k ( + k ) k (r( ))k ( r( ) ) r (r( ))k e r( ) for large r. r k! r k! r (r( )) k k! = ( + Remark. According to Proosition 5, NB(r, ) π(r( )) for large r. So there are transformations and geometrical characteristics similar to those for transformed variance curves on NB(r, ), like π(λ) and B(n, ), if we just regard q = ( ) in NB(r, ) as in B(n, ). Fig. 3 shows that.385 is the best numerical value of a for transformation (r )/ sinh +a on NB(r, ) for rq 5, r a aroximately. There are some cases for π(λ) and B(n, ) also. Anscombe (948) showed that on (r )/ sinh +a the otimum value of a is 3/8 when rq is larger and its variance r a is equal to + 4 O((rq) ). Laubscher (96) roosed the transformation r / sinh r + (r )/ sinh +3/4 n 3/.

6 66 G. u / Statistics and Probability Letters 79 (9) 6 69 Table 4 Some better variance stabilizing transformations on NB(r, ). a r rq L / / / / (, 3/4) (, 3/4) (, 3/4) (, 3/4) skewness.5 (+) / (+.385) /.5 / +(+) / / λ Fig. 4. Five curves showing the skewness of π(λ) and the skewness transformed by four transformations for λ [.5, ]. Table 4 shows three better transformations: (r )/ sinh +.385, (r r.77 )/ sinh +3/8 and r / sinh + r 3/4 r (r ) / sinh +3/4. The third column exresses the left critical oint of rq = r( ), with corresonding relative r 3/ errors in the fourth column. The last six columns denote relative errors of transformed variances for [., k/r] (k = 5, 4, 3,,,.5) resectively, excet for the case r = (marked by ) with [., k/r] (k = 5, 4, 3,,,.5) because of a calculated error roblem. 5. Skewness, kurtosis and conclusions Skewness is used as a measure of asymmetry of a random variable about its mean. Kurtosis can be used to detect that a symmetric distribution dearts from normality by being heavy-tailed or light-tailed or too eaked or too flat at the center. Utilizing skewness and kurtosis, we study the normality of these transformations. Let denote the random variable π(λ), or B(n, ), or NB(r, ). Let T((a)) (a [, ]) denote the variance stabilizing transformation + a for π(λ), or (n+ )/ sin ( n ) for B(n, ), or (r )/ sinh +a for NB(r, ). Let T((, )) r a denote a combined transformation such as + + for π(λ), or (n+ )/ [sin ( n )+sin ( n+ )] for B(n, ), n+ n+ or r / sinh + (r r )/ sinh +3/4 for NB(r, ). r 3/ By comaring the skewness and kurtosis for transformed and not transformed cases, we obtain some conclusions as follows (see Figs. 4 9). () T((a)) obviously imroves the skewness of rimary data. Aroximately, when λ 3 for π(λ), or n [3, n 3] for B(n, ), or rq 3 for NB(r, ), T((a)) exchanges the skew direction and diminishes its size. () T((a)) imroves the kurtosis of NB(r, ), esecially while a.3. But it has no effect on π(λ) and B(n, ).

7 G. u / Statistics and Probability Letters 79 (9) / kurtosis /+(+)/.5 (+)/.5 (+.385)/ /+(+)/ / λ Fig. 5. Five curves showing the kurtosis of π(λ) and the kurtosis transformed by four transformations for λ [.5, ]..8.6 skewness.4.. sin [( n)/(n+)].4 sin [( n)/(n+.77)].6 sin [( n )/(n+)]+sin [( n+)/(n+)].8 sin [( n)/n] Fig. 6. Five curves showing the skewness of B(n, ) and the skewness transformed by four transformations for n = 5 and [.5,.975]..5 kurtosis Fig. 7. Five curves showing the kurtosis of B(n, ) and the kurtosis transformed by four transformations for n = 5 and [.5,.975].

8 68 G. u / Statistics and Probability Letters 79 (9) skewness.4. sinh [(+)/(r )] / r / sinh (/r) / +(r ) / sinh [(+3/4)/(r 3/)] / sinh (/r) / Fig. 8. Five curves showing the skewness of NB(r, ) and the skewness transformed by four transformations for r = and [.,.95]..5 sinh (/r) / sinh [(+.385)/(r.77)] /.5 kurtosis sinh [(+)/(r )] /.5 r / sinh (/r) / +(r ) / sinh [(+3/4)/(r 3/)] / sinh (/r) / Fig. 9. Five curves showing the kurtosis of NB(n, ) and the kurtosis transformed by four transformations for r = and [.,.95]. (3) Combined transformation T((, )) behaves near T((.5)) and is not better than T((.385)) or T((3/8)) for normalizing random variables. (4) When λ or n [, n ] or rq, the skewness and kurtosis transformed by T((a)) are almost indeendent of a. When λ < or n (, ) (n, n] or rq <, the larger a behaves better than the smaller. In general, T((.385)) and T((3/8)) are referred variance stabilizing transformations for π(λ), B(n, ) and NB(r, ) when their means are not less than 3, namely and for π(λ) and λ 3, (n + )/ sin ( n ) n+.77 and (n + )/ sin ( n ) for B(n, ) and n [3, n 3], and (r n+.75 )/ sinh and (r r.77 )/ for r.75 NB(r, ) and nq 3. Here the corresonding relative errors of transformed variances are less than %. When their means are not less than 5, then {Var (T((.385)))} is less than.%. If all the means of the above three distributions are small enough (e.g. ) but larger than.5, combined transformations are favorable. They are and + + for π(λ), (n + )/ [sin ( n ) + sin ( n+ )] for B(n, ), n+ n+ and r / sinh + (r r )/ sinh +3/4 for NB(r, ). When the means are not less than (or.5), the relative errors r 3/ of the transformed variances are less than % (or 4%). Acknowledgement This research was artially suorted by the Oen Fund for a Key Key Silvicultural Disciline of Zhejiang Province Grant 64.

9 G. u / Statistics and Probability Letters 79 (9) References Anscombe, F.J., 948. The transformation of Poisson, binomial, negative binomial data. Biometrika 35, Bartlett, M.S., 936. The square root transformation in the analysis of variance. Sulement to the Journal of the Royal Statistical Society 3, Bartlett, M.S., 947. The Use of Transformations. Biometrics 3, Box, G.E.P., Cox, D.R, 964. An analysis of transformations. Journal of Royal Statistical Society, B 6, 43. Box, G.E.P., Cox, D.R, 98. An analysis of transformation Revisited. Journal of the American Statistical Association 77, Bromiley, P.A., Thacker, N.A.,. The effects of an arcsin square root transform on a binomial distributed quantity. Tina memo, -7. Chatterjee, S., Hadi, A.S., Price, B.,. Regression Analysis by Examle, 3rd ed. Wiley and Son, Inc. Freeman, M.F., Tukey, J.W., 95. Transformations related to the angular and the square root. The Annals of Mathematical Statistics, Laubscher, N.F., 96. On stabilizing the binomial and negative binomial variances. Journal of the American Statistical Association 56, Montgomery, D.C., 5. Design and Analysis of Exeriment, 6th ed. Wiley and Son, Inc. Thacker, N.A., Bromiley, P.A.,. The effects of a square root transform on a Poisson distributed quantity. Tina memo, -. Uddin, M.T., Noor, M.S, Kabir, A., Ali, R., Islam, M.N., 6. The transformations of Random variables under symmetry. Journal of Alied Sciences 6, amamura, K., 999. Transformation using (x +.5) to stabilize the variance of oulations. Researches on Poulation Ecology 4, 9 34.