STAT/MATH 395 PROBABILITY II

Size: px

Start display at page:

Download "STAT/MATH 395 PROBABILITY II"

Aldous Bishop
5 years ago
Views:

1 STAT/MATH 395 PROBABILITY II Distribution of Random Samples & Limit Theorems Néhémy Lim University of Washington Winter 2017

2 Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem

3 Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem

Internet usage time in the United States 2010-2015 Time spent on selected platforms

4 Internet usage time in the United States Time spent on selected platforms by digital population in the United States as of December 2015 (in billion minutes)

5 Definition (i.i.d. Sample) Let X 1,..., X n be collection of n N random variables on probability space (Ω, A, P). (X 1,..., X n ) is a random sample of size n if and only if: (1) X 1,..., X n are mutually independent (2) X 1,..., X n are identically distributed We say that X 1,..., X n are independent and identically distributed (abbreviated i.i.d.). Definition (Sample mean) Let (X 1,..., X n ) be a random sample.then the random variable is called the sample mean. X n = 1 n X i (1) n i=1

6 Definition (Independence) Let X 1,..., X n be n N rrvs on probability space (Ω, A, P). Then X 1,..., X n are said to be independent if and only if : (Discrete case) For all (x 1,..., x n ) X 1 (Ω)... X n (Ω) n p X1...X n (x 1,..., x n ) = p Xi (x i ), (2) i=1 (Continuous case) For all (x 1,..., x n ) R n n f X1...X n (x 1,..., x n ) = f Xi (x i ), (3) i=1

7 Property (Distribution of iid sample) Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P). Then (Discrete case) For all (x 1,..., x n ) X 1 (Ω)... X n (Ω) n p X1...X n (x 1,..., x n ) = p X1 (x i ), (4) i=1 (Continuous case) For all (x 1,..., x n ) R n n f X1...X n (x 1,..., x n ) = f X1 (x i ), (5) i=1 Example. Let (X 1,..., X n ) be a random sample of size n N from an exponential distribution with parameter λ. Give the joint pdf of the sample.

8 Definition (Expected Value) Let X 1,..., X n be n N rrvs on probability space (Ω, A, P) and let g : R n R. Then, the mathematical expectation of g(x 1... X n ), if it exists, is : (Discrete case) E[g(X 1... X n )] = (Continuous case) E[g(X 1... X n )] = x 1 X 1 (Ω)... x n X n(ω) g(x 1,..., x n )p X1...X n (x 1,..., x n ) g(x 1,..., x n )f X1...X n (x 1,..., x n ) R n (6) (7)

9 Theorem Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P) and let g 1,..., g n be n real-valued functions on R. Then, E[g 1 (X 1 )... g n (X n )] = E[g 1 (X 1 )]... E[g n (X n )] (8) provided that the expectations exist. Theorem (Variance of independent rrvs) Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P). Then, provided that the variances exist. ( n ) n Var X i = Var(X i ) (9) i=1 i=1

10 Property Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P) with mean µ = E[X 1 ] and variance σ 2 = Var(X 1 ) <. Then E[X n ] = µ (10) and Var(X n ) = σ2 n (11)

11 Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem

12 Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) and X be a rrv on the same probability space. Sequence (X n ) is said to converge in probability towards X if, for all ɛ > 0 : lim P ( X n X > ɛ) = 0 (12) n Convergence in probability is denoted as follows : X n P X Example Let X be a discrete rrv with pmf p X defined by : 1/3 if x = 1 p X (x) = 2/3 if x = 0 0 otherwise and let X n = (1 + 1 n )X.

13 Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) and X be a rrv on the same probability space. Sequence (X n ) is said to converge almost surely or almost everywhere or with probability 1 or strongly towards X if: ( ) P lim X n = X = 1 (13) n Almost sure convergence is denoted as follows : X n a.s. X

14 Definition Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P). For any n, the distribution function of X n is denoted by F n. Let X be a rrv with distribution function F X. Sequence (X n ) is said to converge in distribution or converge weakly towards X if: lim F n (x) = F X (x) (14) n for all x R at which F X is continuous. Convergence in distribution is denoted as follows : X n D X Example. Let (X n ) n N be a sequence of rrvs with cdf F n ( ( F n (x) = ) nx ) 1 n (0, ) (x)

15 Theorem Let (X n ) n N be a sequence of rrvs on probability space (Ω, A, P) with respective mgfs M n. Let X be a rrv with mgf M X. If the following holds : lim M n (x) = M X (x) (15) n for all x R where M n (x) and M X (x) exist, then sequence (X n ) converges in distribution to X.

16 Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem

17 Property (Markov s Inequality) Let X be a rrv that takes only on nonnegative values. Then, for any a > 0, we have : P(X a) E[X] a (16) Property (Bienaymé-Chebyshev s Inequality) Let X be a rrv that has expectation and variance. Then, for any α > 0, we have : P ( X E[X] α) Var(X) α 2 (17)

18 Theorem (Weak law of large numbers) Let (X n ) n N be a sequence of i.i.d. rrvs, each having finite expectation. The weak law of large numbers (also called Khintchine s law) states that the sample mean X n converges in probability towards E[X 1 ], that is, for all ɛ > 0 : ( ) lim P X n E[X 1 ] > ɛ = 0 (18) n

19 Theorem (Strong law of large numbers) Let (X n ) n N be a sequence of i.i.d. rrvs, each having finite expectation. The strong law of large numbers (also called Kolmogorov s strong law) states that the sample mean X n converges almost surely towards E[X 1 ], that is: ( ) P lim X n = E[X 1 ] = 1 (19) n

20 Outline Distribution of i.i.d. Samples Convergence of random variables The Laws of Large Numbers The Central Limit Theorem

21 Property Let X 1,..., X n be n N independent rrvs on probability space (Ω, A, P) with respective moment generating functions M 1,..., M n. Then the moment generating function of is : for all x R where M n (x) exist. n S n = X i i=1 n M Sn (x) = M i (x) (20) i=1

22 Corollary (Mgf of iid sample) Let (X 1,..., X n ) be a random sample of size n N on probability space (Ω, A, P) with moment generating function M = M X1. Then the moment generating function of n S n = X i i=1 is : M Sn (x) = (M(x)) n (21) for all x R where M(x) exists.

23 Theorem (Central Limit Theorem) Let (X n ) n N be a sequence of i.i.d. rrvs, each having expectation E[X 1 ] = µ and finite variance Var(X 1 ) = σ 2 <. The Central Limit Theorem states that the sequence of variables (Z n ) n N defined by: Z n = X n µ σ 2 n converges in distribution towards Z following a standard normal distribution N (0, 1), that is: lim F Zn (x) = Φ(x), for all x R (22) n

24 Example Let (X 1,..., X 15 ) be a random sample with probability density function : f (x) = 3 2 x 2 1 ( 1,1) (x) What is the approximate probability that the sample mean X 15 falls between -2/5 and 1/5?

25 Example Let (X n ) be a sequence of i.i.d. Poisson random variables with mean 3. Estimate approximately how large n must be such that: ( P ) n X i > 0.1 = 0.1. n i=1

Chapter 2. Random variables. 2.3 Expectation

Chapter 2. Random variables. 2.3 Expectation Random processes - Chapter 2. Random variables 1 Random processes Chapter 2. Random variables 2.3 Expectation 2.3 Expectation Random processes - Chapter 2. Random variables 2 Among the parameters representing