CS145: Probability & Computing Lecture 8: Variance of Sums, Cumulative Distribution, Continuous Variables Instructor: Eli Upfal Brown University Computer Science Figure credits: Bertsekas & Tsitsiklis, Introduction to Probability, 2008 Pitman, Probability, 1999
CS145: Lecture 8 Outline Ø Variance of Sums of Random Variables Ø Cumulative Distribution Functions Ø Limits and Continuous Variables
Variance Ø The variance is the expected squared deviation of a random variable from its mean (the following definitions are equivalent): Var[X] =E[(X E[X]) 2 ]= X x2x(x E[X]) 2 p X (x) " X # " X # 2 Var[X] =E[X 2 ] E[X] 2 = Ø The variance is always non-negative: x2x x 2 p X (x) x2x xp X (x) Var[X] 0 because (x E[X]) 2 0 for all x. Ø By definition, the standard deviation is the square root of the variance: X =Std[X] = p Var[X]
Geometric Variance via Total Expectation Ø A geometric random variable X has parameter p, countably infinite range: p X (k) =(1 p) k 1 p X = {1, 2, 3,...} Ø Memoryless: For any integer c > 0, if I observe that X > c, then Y=X-c has same geometric PMF: p X c (k X>c)=(1 p) k 1 p, k =1, 2,... Ø Compute second moment via two cases: E[X 2 ]=pe[x 2 X = 1] + (1 p)e[x 2 X>1] E[X 2 ]=p +(1 p)e[(x + 1) 2 ] E[X 2 ]= 2 p p 2 p p X (k) p X X>2(k) p(1-p) 2 p...... 1 k 3 k p X- 2 X>2(k) p... 1 k
Geometric Probability Distribution Ø A geometric random variable X has parameter p, countably infinite range: p X (k) =(1 p) k 1 p X = {1, 2, 3,...} Ø The mean and variance of the geometric distribution then equal: E[X 2 ]= 2 p 2 p 20 400 18 16 14 12 10 E[X] = 1 p 350 300 250 200 Var[X] = 1 p 2 p 8 150 6 100 4 2 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 50 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Wikipedia
Sums of Independent Variables Ø If Z=X+Y and random variables X and Y are independent, we have E[Z] =E[X]+E[Y ] Var[Z] = Var[X] + Var[Y ] For any variables X, Y. Only for independent X, Y. Ø Interpretation: Adding independent variables increases variance Var[Z] Var[X] and Var[Z] Var[Y ] Ø The standard q deviation of a sum of independent variables is then Z = 2 X + Y 2 X = p Var[X], Y = p Var[Y ], Z = p Var[Z] Ø Identity used in proof: If X and Y are independent random variables, E[XY ]=E[X]E[Y ] if p XY (x, y) =p X (x)p Y (y) This equality does not hold for general, dependent random variables.
Reminder: Bernoulli Distribution Ø A Bernoulli or indicator random variable X has one parameter p: p X (1) = p, p X (0) = 1 p, X = {0, 1} Ø For an indicator variable, expected values are probabilities: E[X] =p Ø Variance of Bernoulli distribution: Var[X] =E E[X 2 ]=p h (X p) 2i = p(1 p) Ø Fair coin (p=0.5) has largest variance Ø Coins that always come up heads (p=1.0), or always come up tails (p=0.0), have variance 0
Binomial Probability Distribution Ø Suppose you flip n coins with bias p, count number of heads Ø A binomial random variable X has parameters n, p: n p X (k) = k Ø If X i is a Bernoulli variable indicating whether toss i comes up heads, then X = P n i=1 X i Ø Then because tosses are independent: E[X] =np Var[X] =np(1 p) p k (1 p) n k X = {0, 1, 2,...,n}
CS145: Lecture 8 Outline Ø Variance of Sums of Random Variables Ø Cumulative Distribution Functions Ø Limits and Continuous Variables
Cumulative Distribution Function Ø Recall probability mass function (PMF): p X (x) =P (X = x) Ø The cumulative distribution function (CDF) is the cumulative sum of the PMF: F X (x) =P (X apple x) = X kapplex p X (k) PMF p X (x) p x (2) p x (2) 0 1 2 3 4 x 0 1 2 3 4 PMF p X (x) 0 x 0 1 1 CDF F X (x). CDF F X (x)...... x x Ø The CDF equals 0 below the range of X, 1 above the range of X, and is monotonically increasing: F X (x 2 ) F X (x 1 )ifx 2 >x 1. Ø The CDF allows quick computation of the probability of intervals: P (x 1 <Xapple x 2 )=F X (x 2 ) F X (x 1 )
Binomial Probability Distribution Ø You flip n coins with bias p, count number heads Ø A binomial probability mass function (PMF) has parameters n, p: p X (k) = n k p k (1 p) n k X = {0, 1, 2,...,n} Ø By definition, the binomial cumulative distribution function (CDF) equals F X (x) = bxc X k=0 n k p k (1 p) n k No simple closed form, evaluate numerically. Wikipedia
Geometric Probability Distribution Ø A geometric probability mass function (PMF) has parameter p, countably infinite range: p X (k) =(1 p) k 1 p X = {1, 2, 3,...} Ø The geometric cumulative distribution function (CDF) then equals: F X (x) =1 P (X >x)=1 (1 p) x Ø Note that the CDF is strictly less than 1.0 for all finite x, because the range is unbounded above Wikipedia
Quantiles of Distributions Ø For 0 < p < 1, the p-quantile of distribution of random variable X is the smallest x for which F X (x) p Ø The median is the 0.5-quantile. This is the center of the distribution, which sometimes (but not always) equals the mean Ø The 0.25-quantile and 0.75-quantile are sometimes called quartiles. Ø Often we are interested in extreme quantiles, which give the probabilities of rare events: p = 0.9, 0.99, 0.999, p = 0.1, 0.01, 0.001,
CS145: Lecture 8 Outline Ø Variance of Sums of Random Variables Ø Cumulative Distribution Functions Ø Limits and Continuous Variables
Discrete Uniform Distribution p X (x) = 1 n if a apple x apple b n = b a +1 bxc a +1 F X (x) = n if a apple x apple b Wikipedia
1 Scaled Uniform Distributions 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.2 n=10 0.1 n=50 0.1 n=100 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Ø Take a discrete random variable uniformly distributed between 0 and n-1, and multiply by 1/n to get a variable taking values between 0 and 1 Ø What does this random variable approach as n becomes large?