STAT/MATH 395 PROBABILITY II Distribution of Random Samples & Limit Theorems Néhémy Lim University of Washington Winter 2017
Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem
Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem
Internet usage time in the United States 2010-2015 Time spent on selected platforms by digital population in the United States as of December 2015 (in billion minutes)
Definition (i.i.d. Sample) Let X 1,..., X n be collection of n N random variables on probability space (Ω, A, P). (X 1,..., X n ) is a random sample of size n if and only if: (1) X 1,..., X n are mutually independent (2) X 1,..., X n are identically distributed We say that X 1,..., X n are independent and identically distributed (abbreviated i.i.d.). Definition (Sample mean) Let (X 1,..., X n ) be a random sample.then the random variable is called the sample mean. X n = 1 n X i (1) n i=1
Definition (Independence) Let X 1,..., X n be n N rrvs on probability space (Ω, A, P). Then X 1,..., X n are said to be independent if and only if : (Discrete case) For all (x 1,..., x n ) X 1 (Ω)... X n (Ω) n p X1...X n (x 1,..., x n ) = p Xi (x i ), (2) i=1 (Continuous case) For all (x 1,..., x n ) R n n f X1...X n (x 1,..., x n ) = f Xi (x i ), (3) i=1
Property (Distribution of iid sample) Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P). Then (Discrete case) For all (x 1,..., x n ) X 1 (Ω)... X n (Ω) n p X1...X n (x 1,..., x n ) = p X1 (x i ), (4) i=1 (Continuous case) For all (x 1,..., x n ) R n n f X1...X n (x 1,..., x n ) = f X1 (x i ), (5) i=1 Example. Let (X 1,..., X n ) be a random sample of size n N from an exponential distribution with parameter λ. Give the joint pdf of the sample.
Definition (Expected Value) Let X 1,..., X n be n N rrvs on probability space (Ω, A, P) and let g : R n R. Then, the mathematical expectation of g(x 1... X n ), if it exists, is : (Discrete case) E[g(X 1... X n )] = (Continuous case) E[g(X 1... X n )] = x 1 X 1 (Ω)... x n X n(ω) g(x 1,..., x n )p X1...X n (x 1,..., x n ) g(x 1,..., x n )f X1...X n (x 1,..., x n ) R n (6) (7)
Theorem Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P) and let g 1,..., g n be n real-valued functions on R. Then, E[g 1 (X 1 )... g n (X n )] = E[g 1 (X 1 )]... E[g n (X n )] (8) provided that the expectations exist. Theorem (Variance of independent rrvs) Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P). Then, provided that the variances exist. ( n ) n Var X i = Var(X i ) (9) i=1 i=1
Property Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P) with mean µ = E[X 1 ] and variance σ 2 = Var(X 1 ) <. Then E[X n ] = µ (10) and Var(X n ) = σ2 n (11)
Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem
Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) and X be a rrv on the same probability space. Sequence (X n ) is said to converge in probability towards X if, for all ɛ > 0 : lim P ( X n X > ɛ) = 0 (12) n Convergence in probability is denoted as follows : X n P X Example Let X be a discrete rrv with pmf p X defined by : 1/3 if x = 1 p X (x) = 2/3 if x = 0 0 otherwise and let X n = (1 + 1 n )X.
Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) and X be a rrv on the same probability space. Sequence (X n ) is said to converge almost surely or almost everywhere or with probability 1 or strongly towards X if: ( ) P lim X n = X = 1 (13) n Almost sure convergence is denoted as follows : X n a.s. X
Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P). For any n, the distribution function of X n is denoted by F n. Let X be a rrv with distribution function F X. Sequence (X n ) is said to converge in distribution or converge weakly towards X if: lim F n (x) = F X (x) (14) n for all x R at which F X is continuous. Convergence in distribution is denoted as follows : X n D X Example. Let (X n ) n N be a sequence of rrvs with cdf F n ( ( F n (x) = 1 1 1 ) nx ) 1 n (0, ) (x)
Theorem Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) with respective mgfs M n. Let X be a rrv with mgf M X. If the following holds : lim M n (x) = M X (x) (15) n for all x R where M n (x) and M X (x) exist, then sequence (X n ) converges in distribution to X.
Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem
Property (Markov s Inequality) Let X be a rrv that takes only on nonnegative values. Then, for any a > 0, we have : P(X a) E[X] a (16) Property (Bienaymé-Chebyshev s Inequality) Let X be a rrv that has expectation and variance. Then, for any α > 0, we have : P ( X E[X] α) Var(X) α 2 (17)
Theorem (Weak law of large numbers) Let (X n ) n N be a sequence of i.i.d. rrvs, each having finite expectation. The weak law of large numbers (also called Khintchine s law) states that the sample mean X n converges in probability towards E[X 1 ], that is, for all ɛ > 0 : ( ) lim P X n E[X 1 ] > ɛ = 0 (18) n
Theorem (Strong law of large numbers) Let (X n ) n N be a sequence of i.i.d. rrvs, each having finite expectation. The strong law of large numbers (also called Kolmogorov s strong law) states that the sample mean X n converges almost surely towards E[X 1 ], that is: ( ) P lim X n = E[X 1 ] = 1 (19) n
Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem
Property Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P) with respective moment generating functions M 1,..., M n. Then the moment generating function of is : for all x R where M n (x) exist. n S n = X i i=1 n M Sn (x) = M i (x) (20) i=1
Corollary (Mgf of iid sample) Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P) with moment generating function M = M X1. Then the moment generating function of n S n = X i i=1 is : M Sn (x) = (M(x)) n (21) for all x R where M(x) exists.
Theorem (Central Limit Theorem) Let (X n ) n N be a sequence of i.i.d. rrvs, each having expectation E[X 1 ] = µ and finite variance Var(X 1 ) = σ 2 <. The Central Limit Theorem states that the sequence of variables (Z n ) n N defined by: Z n = X n µ σ 2 n converges in distribution towards Z following a standard normal distribution N (0, 1), that is: lim F Zn (x) = Φ(x), for all x R (22) n
Example Let (X 1,..., X 15 ) be a random sample with probability density function : f (x) = 3 2 x 2 1 ( 1,1) (x) What is the approximate probability that the sample mean X 15 falls between -2/5 and 1/5?
Example Let (X n ) be a sequence of i.i.d. Poisson random variables with mean 3. Estimate approximately how large n must be such that: ( P 3 + 1 ) n X i > 0.1 = 0.1. n i=1