Lecture Notes 6. Assume F belongs to a family of distributions, (e.g. F is Normal), indexed by some parameter θ.

Transcription

1 Sufficient Statistics Lecture Notes 6 Sufficiency Data reduction in terms of a particular statistic can be thought of as a partition of the sample space X. Definition T is sufficient for θ if the conditional distribution of the sample X n T does not depend on θ, where we use X n (X,..., X n ). Thus, f(x n t; θ) f(x n t). Sample X X n X,, X n F. Assume F belongs to a family of distributions, (e.g. F is Normal), indexed by some parameter θ. We want to learn about θ and try to summarize the data without throwing any information about θ away. Calculate some statistic T (X,, X n ) that contains all available information about θ in the sample. If so, we say T is sufficient. Theorem 2 (Theorem 6.2.2, CB) If f(x n ; θ) is the joint pdf or pmf of X n and q(t; θ) is the pdf or pmf of T (X n ), then T (X n ) is a sufficient statistic for θ if, for every x n in the sample space, the ratio is constant as a function of θ. f(x n ; θ) q(t (x n ); θ) Exercise 2: Binomial sufficient statistic Let X,, X n be iid Bernoulli random variables with parameter θ, 0 < θ <. We show that T (X n ) n i X i is a sufficient statistic for θ. Note that T (X n ) has Binomial(n, θ) distribution. Thus for t n i x i, we have f(x n ; θ) q(t (x n ); θ) n i θx i ( θ) x i ( n ) θt ( θ) t n t θ n i x i ( θ) n ) θt ( θ) n t ( n t i ( x i) ( n t θ t ( θ) ) n t θt ( θ) n t ( n t) ( ) n n i x i

2 Interpretation: The total number of s in this Bernoulli sample contains all the information about θ that is in the data. Sum of Poisson Random variables Theorem 3 (Theorem 4.3.2, CB) if X Poisson(λ) and if Y Poisson(µ) and X and Y are independent, then X + Y Poisson(λ + µ) Example and exercise! X,, X n Poisson(θ). Denote by X n (X,..., X n ) and x n (x,..., x n ). Let T n i X i. Clearly T Poisson (nθ) by Theorem 3. Hence for t 0, P (T t) e nθ (nθ) t. t! We have where p X n T (x n t) P(X n x n T (X n ) t) P (Xn x n and T t), P (T t) { 0 if n P (X n x n and T t) i x n t P (X n x n ) if n i x n t Thus we focus on the case t n i x i, where x i 0, i,..., n: P (X n x n ) by independence of X,..., X n. Now n e θ θ x i i x i! e nθ θ x i (xi!) e nθ θ t (xi!) p X n T (x n t) P (Xn x n ) P (T t) t! xi! n i ( ) xi t! n xi! n t when n x i t i and p X n T (x n t) 0 otherwise; Clearly this is the Multinomial Distribution with parameter t and p i /n for all i,..., n. Note that this distribution does not depend on θ. So T i X i is a sufficient statistic for θ following the definition. 2 Factorization Theorem Theorem 4 (Theorem 6.2.6, CB) Let f(x n θ) denote the joint pdf or pmf of a sample X n. A statistic T (X n ) is a sufficient statistic for θ if and only if there exists functions g(t θ) and h(x n ) such that, for all sample points x n and all parameters θ, f(x n θ) g(t (x n ) θ)h(x n ). 2

3 Homework: read proof of Theorem 4 on CB P.276. exponential family of distributions is. Try also to understand what an Example 5 Let X,, X n Poisson. Then f(x n θ) e nθ θ X i (xi!) e nθ θ i Xi (xi!) Example 6 X,, X n iid N(µ, σ 2 ) f(x n µ, σ 2 ) ( ) n 2 2πσ 2 ( (a) If σ known f(x n µ) 2πσ 2 X n is sufficient for µ. ) n 2 { } (xi x) 2 + n(x µ) 2 exp 2σ 2 { } (xi x n ) 2 exp 2σ }{{ 2 } h(x n ) { } n(xn µ) 2 exp 2σ }{{ 2 } g(t (x n ) µ) () (b) If (µ, σ 2 ) unknown, by staring at (2), we have T (X n, S 2 n) is sufficient. So is T ( X i, X 2 i ). Note:. T is jointly sufficient for (µ, σ 2 ); 2. and the sample is sufficient also. 3 Sufficient Partitions Data reduction in terms of a particular statistic can be thought of as a partition of the sample space X. Let T {t : t T (x n ) for some x n X } be the image of X under T (x n ). Then T (x n ) partitions the sample space into sets B t, t T, defined by B t {x n : T (x n ) t} The statistic summarizes the data in that, rather than reporting the entire sample x n, it reports only that T (x n ) t or x n B t. Example 7 Let X, X 2, X 3 Bernoulli(θ). Let T X i. 3

4 B t x n t p(x n t) B 0 (0, 0, 0) t 0 B (0, 0, ) t /3 (0,, 0) t /3 (, 0, 0) t /3 B 2 (0,, ) t 2 /3 (, 0, ) t 2 /3 (,, 0) t 2 /3 B 3 (,, ) t 3 8 elements 4 elements. A partition B,..., B k is sufficient if f(x n X n B j ) does not depend on θ for all j. 2. A statistic T induces a partition. For each t, B t {x n : T (x n ) t} is one element of the partition. T is sufficient if and only if the partition is sufficient. 3. Two statistics can generate the same partition: example: i X i and 3 i X i. We wish to consider the conditional probability P (X n x n T (X n ) t). If x n is a sample point such that T (x n ) t, then clearly P θ (X n x n T (X n ) t) 0. So we are interested in P θ (X n x n T (X n ) T (x n )). We must verify that for any fixed values of x n and t, the conditional probability P θ (X n x n T (X n ) T (x n )) does not depend on θ. Since {X n x n } is a subset of {T (X n ) T (x n ), P θ (X n x n T (X n ) T (x n )) P θ(x n x n and T (X n ) T (x n )) P θ (T (X n ) T (x n )) P θ (X n x n ) P θ (T (X n ) T (x n )) f(x n ; θ) q(t (x n ); θ). This is exactly the content of Theorem 2. Example 8 For the discrete version of the example above, we showed that for all sample points x n, and Y n (X, X 2,..., X n) such that the mechanism of producing Y n is based on the following conditional probabilities given T (X n ), P (X n x n T (X n ) T (x n )) P (Y n x n T (X n ) T (x n )) Exercise: Show that P θ (X n x n ) P θ (Y n x n ). 4

5 Example 9 X,, X n iid N(µ, σ 2 ) ( ) n { } f(x n µ, σ 2 2 (xi x) 2 + n(x µ) 2 ) exp 2πσ 2 2σ 2 ( ) n { } 2 (xi ) 2 exp 2πσ 2 2σ }{{ 2 } g(t (x n ) µ0,σ 2 ) i X2 i is sufficient for σ 2 when µ is known to be zero. Note:. T is jointly sufficient for (µ, σ 2 ); 2. and the sample is sufficient also. Example 0 (Example.9.2 from Testing Statistical Hypotheses by E.L.Lehmann and J.P.Romano) If X,, X n iid N(0, σ 2 ), the conditional distribution of the sample point over each of the spheres, defined as {x n : n i x2 i constant} is uniform irrespective of σ 2. Hint: to understand this, you need to use the density function defined in the previous example to obtain the conditional distribution defined immediately above. One can therefore construct an equivalent sample X,..., X n from the knowledge of T (X n ) n i X2 i and a mechanism that can produce a point randomly distributed over a sphere. Equivalence here is in the following sense, For Y n (X, X 2,..., X n), f X n(x n ; θ) f Y n(x n ; θ). 4 Minimal Sufficient Statistics (MSS) Want the greatest reduction in dimension. Example X,, X n N(0, σ 2 ). Some sufficient statistics: T (X,, X n ) (X,, X n ) R n (X, 2, Xn) 2 R n+ m n ( Xi 2, Xi 2 ) R 2+ i X 2 i im+ minimal sufficient 5

6 Definition T is a Minimal Sufficient Statistic if (i) T is sufficient and (ii) T is a function of all other sufficient statistics. (It yields coarsest partition.) Suppose U is sufficient. Suppose T H(U) is also sufficient. T provides greater reduction than U unless H is a transformation, in which case T and U are equivalent. Example 2 X N(0, σ 2 ). X is sufficient. X is sufficient. X is MSS. So are X 2, X 4, e X2. 4. How to find a MSS Theorem 3 Suppose that T has the following property: for each pair x n, y n, R(x n, y n ; θ) f(yn ; θ) f(x n ; θ) does not depend on θ if and only if T (y n ) T (x n ). Then T is a MSS. Example 4 Y,, Y n iid Poisson (θ). f(y n θ) e nθ θ y i yi, f(y n ; θ) f(x n ; θ) θ yi x i yi!/ x i! which is independent of θ iff y i x i. This implies that T (Y n ) Y i is a minimal sufficient statistic for θ. The minimal sufficient statistic is not unique. But, the minimal sufficient partition is unique. Exercise 3: Understand the following example. Example 5 Let X, X 2, X 3 Bernoulli(θ). Then T X is not sufficient. Look at its partition: x n t p(x n t) (0, 0, 0) t 0 ( θ) 2 (0, 0, ) t 0 θ( θ) (0,, 0) t 0 θ( θ) (0,, ) t 0 θ 2 (, 0, 0) t ( θ) 2 (, 0, ) t θ( θ) (,, 0) t θ( θ) (,, ) t θ 2 8 elements 2 elements 6

7 5 Digression: Order Statistics Assume that the sample X, X 2,..., X n has been ordered from smallest to largest: X () X (2)... X (n). Theorem 6 Let X (), X (2),..., X (n) denote the order statistics of a random sample X,..., X n from a continuous population with cdf F X (x) and pdf f X (x). Then the CDF and pdf of X ( k) is: Proof. F X(k) (x) f X(k) (x) n jk ( ) n [F X (x)] j [ F X (x)] n j j n! (k )!(n k)! f X(x) [F X (x)] k [ F X (x)] n k. Let Y be the random variable counting the number of X,..., X n less or equal to x. Then, by defining a success as the event {X j x}, we see that Y binomial(n, F X (x)) as F X (x) P (X x). F X(k) (x) P (X (k) x) P (Y k) P (at least k of the X,..., X n x) n P (exactly j of the X,..., X n x) Differentiate to find the pdf. jk n jk ( ) n [F X (x)] j [ F X (x)] n j j Corollary 7 Let random variables X and X 2 be independent and both from a continuous distribution with cdf F X and pdf f X (x). Let Z : max{x, X 2 } X (2). The pdf of Z is: and the density is F Z (z) P (X z, X 2 z) F 2 X(z) f Z (z) df 2 X(z)/dz 2F X (z)f X (z). 7

8 6 Best Unbiased Estimators What is the smallest variance of an unbiased estimator? This was once considered an important question. Today we consider it not so important. There is no reason to require an estimator to be unbiased. Having small MSE is more important. Example 8 We have learned two unbiased estimators: Let X,..., X n be a random sample from a population with mean µ and variance σ 2 <. That is. µ E(X i ) and σ 2 Var(X i ). Then E(X n ) µ, Var(X n ) σ2 n, E(S2 n) σ 2. In your homework, you are asked to compute Var(S 2 n) in case X,..., X n N(µ, σ 2 ). Example 9 Let X,..., X n N(θ, σ 2 ). Then The MSE s are E(X) µ, E(S 2 ) σ 2. E(X µ) 2 σ2 n, E(S2 σ 2 ) 2 2σ4 n. We now calculate the MSE for the MLE estimators σ 2, namely, σ 2 n n (X i X) 2. i using MSE B 2 + V, for which we have for n, Exercise 2: E( σ 2 σ 2 ) 2 2n n 2 σ 4 < 2σ4 n E(S2 σ 2 ) 2. Verify everything involved in the last equation. Hint, you need to compute B and V of σ 2 based on those of S 2. However, for completeness, we will briefly consider the question. An estimator W is UMVUE (Uniform Minimum Variance Unbiased Estimator) for τ(θ) if (i) E θ (W ) τ(θ) for all θ and (ii) if E θ (W ) τ(θ) for all θ then Var θ (W ) Var θ (W ). The Cramer-Rao inequality gives a lower bound on the variance of any unbiased estimator, which we will study later. 8

9 Theorem 20 The Rao-Blackwell Theorem. Let W be an unbiased estimator of τ(θ) and let T be a sufficient statistic. Define W φ(t ) E(W T ). Then W is unbiased and Var θ (W ) Var θ (W ) for all θ. Proof. We have E θ (W ) E θ (E(W T )) E θ (W ) τ(θ) so W is unbiased. Also, Var θ (W ) Var θ (E(W T )) + E θ (Var(W T )) Var θ (W ) + E θ (Var(W T )) Var θ (W ). Implications: Conditioning any unbiased estimator on a sufficient statistic will result in a uniform improvement, so we need consider only statistics that are functions of a sufficient statistic in our serach for best unbiased estimators. Exercises: Convince yourself that W is an estimator that you can construct from the sample, that is, its expression is independent of θ by sufficiency. When does the eqaulity hold in the above formula? 7 Exponential Family We cover material from the text, Section 3.4. Example 2 Here are some examples of the Exponential Family: Binomial distribution when n is fixed and p (0, ) Double exponential distribution when µ is fixed and σ > 0 Log-normal distribution where < µ < and 0 < σ < Normal distribution N(µ, σ 2 ) where < µ < and 0 < σ < Gamma distribution where α, β > 0, More examples: Poisson, Multinomial distribution when n is fixed... 9

10 We will prove Theorem next week. We proved Theorem regarding the sufficient statistics for the Exponential Family. We can rewrite an exponential family in terms of a natural parameterization. For k we have f(x η) h(x) exp{ηt(x) A(η)} define A(η) log h(x) exp{ηt(x)}dx For example a Poisson can be written as f(x η) exp{ηx e η }/x! where the natural parameter is η log λ. Theorem. Let X have density in an exponential family. Then E (t(x)) A (η), Var (t(x)) A (η). This convenient representation and link to moments holds also for k >. 0