9 Expectation and Variance

Transcription

1 9 Expectation and Variance Two numbers are often used to summarize a probability distribution for a random variable X. The mean is a measure of the center or middle of the probability distribution, and the variance is a measure of the dispersion, or variability in the distribution. These two measures do not uniquely identify a probability distribution. That is, two different distributions can have the same mean and variance. Still, these measures are simple, useful summaries of the probability distribution of X. 9.1 Expectation of Discrete Random Variable The most important characteristic of a random variable is its expectation. Synonyms for expectation are expected value, mean, and first moment. The definition of expectation is motivated by the conventional idea of numerical average. Recall that the numerical average of n numbers, say a 1, a 2,..., a n is 1 n n a k. We use the average to summarize or characterize the entire collection of numbers a 1,..., a n with a single value. Example 9.1. Consider 10 numbers: 5, 2, 3, 2, 5, -2, 3, 2, 5, 2. The average is ( 2) We can rewrite the above calculation as = =

2 Definition 9.2. Suppose X is a discrete random variable, we define the expectation (or mean or expected value) of X by EX = x x P [X = x] = x x p X (x). (15) In other words, The expected value of a discrete random variable is a weighted mean of the values the random variable can take on where the weights come from the pmf of the random variable. Some references use m X or µ X to represent EX. For conciseness, we simply write x under the summation symbol in (15); this means that the sum runs over all x values in the support of X. (Of course, for x outside of the support, p X (x) is 0 anyway.) 9.3. Analogy: In mechanics, think of point masses on a line with a mass of p X (x) kg. at a distance x meters from the origin. In this model, EX is the center of mass (the balance point). This is why p X (x) is called probability mass function. Example 9.4. When X Bernoulli(p) with p (0, 1), Note that, since X takes only the values 0 and 1, its expected value p is never seen Interpretation: The expected value is in general not a typical value that the random variable can take on. It is often helpful to interpret the expected value of a random variable as the long-run average value of the variable over many independent repetitions of an experiment Example 9.6. p X (x) = 1/4, x = 0 3/4, x = 2 0, otherwise 109

3 Example 9.7. For X P(α), EX = i=0 = e α α α (α)i ie i! k=0 = α (α)i e i + 0 = e α (α) i 1 (α) i! (i 1)! α k k! = e α αe α = α. Example 9.8. For X B(n, p), n ( n EX = i i i=0 n = n Let k = i 1. Then, ) p i (1 p) n i = n n! i i! (n i)! pi (1 p) n i (n 1)! (i 1)! (n i)! pi (1 p) n i = n n ( ) n 1 p i (1 p) n i i 1 n 1 ( ) n 1 n 1 ( ) EX = n p k+1 (1 p) n (k+1) n 1 = np p k (1 p) n 1 k k k k=0 We now have the expression in the form that we can apply the binomial theorem which finally gives EX = np(p + (1 p)) n 1 = np. We shall revisit this example again using another approach in Example Example 9.9. Expected Profit Game #1: Flip a fair coin. H: You get $200 T: You lose $100 Game #2: Flip a biased coin with P ({H}) = H: You get $2,000,000 T: You lose $0 Game #3: Pay $50 to play the game. Flip a biased coin with P ({H}) = H: You get $2,000,000 T: You lose $0 k=0 110

4 Example Pascal s wager: Suppose you concede that you don t know whether or not God exists and therefore assign a 50 percent chance to either proposition. How should you weigh these odds when deciding whether to lead a pious life? If you act piously and God exists, Pascal argued, your gain eternal happiness is infinite. If, on the other hand, God does not exist, your loss, or negative return, is small the sacrifices of piety. To weigh these possible gains and losses, Pascal proposed, you multiply the probability of each possible outcome by its payoff and add them all up, forming a kind of average or expected payoff. In other words, the mathematical expectation of your return on piety is one-half infinity (your gain if God exists) minus one-half a small number (your loss if he does not exist). Pascal knew enough about infinity to know that the answer to this calculation is infinite, and thus the expected return on piety is infinitely positive. Every reasonable person, Pascal concluded, should therefore follow the laws of God. [13, p 76] Pascals wager is often considered the founding of the mathematical discipline of game theory, the quantitative study of optimal decision strategies in games. Example A sweepstakes sent through the mail offered a grand prize of $5 million. All you had to do to win was mail in your entry. There was no limit on how many times you could enter, but each entry had to be mailed in separately. The sponsors were apparently expecting about 200 million entries, because the fine print said that the chances of winning were 1 in 200 million. Does it pay to enter this kind of free sweepstakes offer? Multiplying the probability of winning times the payoff, we find that each entry was worth 1/40 of $1, or $0.025 far less than the cost of mailing it in. In fact, the big winner in this contest was the post office, which, if the projections were correct, made nearly $80 million in postage revenue on all the submissions. [13, p 77] 111

5 9.12. Technical issue: Definition (15) is only meaningful if the sum is well defined. The sum of infinitely many nonnegative terms is always welldefined, with + as a possible value for the sum. Infinite Expectation: Consider a random variable X whose pmf is defined by { 1 p X (x) = cx, x = 1, 2, 3, , otherwise Then, c = n=1 1 n 2 which is a finite positive number (π 2 /6). However, EX = kp X (k) = k 1 1 c k = 1 2 c 1 k = +. Some care is necessary when computing expectations of signed random variables that take infinitely many values. The sum over countably infinite many terms is not always well defined when both positive and negative terms are involved. For example, the infinite series has the sum 0 when you sum the terms according to (1 1)+(1 1)+, whereas you get the sum 1 when you sum the terms according to 1 + ( 1 + 1) + ( 1 + 1) + ( 1 + 1) +. Such abnormalities cannot happen when all terms in the infinite summation are nonnegative. It is the convention in probability theory that EX should be evaluated as EX = xp X (x) X (x), x 0 x<0( x)p If at least one of these sums is finite, then it is clear what value should be assigned as EX. If both sums are +, then no value is assigned to EX, and we say that EX is undefined. 112

6 Example Undefined Expectation: Let { 1 p X (x) = 2cx, x = ±1, ±2, ±3, , otherwise Then, EX = The first sum gives kp X (k) kp X (k) = The second sum gives 1 k= ( k) p X (k) = 1 k= k 1 2ck 2 = 1 2c kp X ( k) = ( k) p X (k). 1 k = 2c. k 1 2ck 2 = 1 2c 1 k = 2c. Because both sums are infinite, we conclude that EX is undefined More rigorously, to define EX, we let X + = max {X, 0} and X = min {X, 0}. Then observe that X = X + X and that both X + and X are nonnegative r.v. s. We say that a random variable X admits an expectation if EX + and EX are not both equal to +. In which case, EX = EX + EX. 113

7 9.2 Function of a Discrete Random Variable Given a random variable X, we will often have occasion to define a new random variable by Y g(x), where g(x) is a real-valued function of the real-valued variable x. More precisely, recall that a random variable X is actually a function taking points of the sample space, ω Ω, into real numbers X(ω). Hence, we have the following definition: Definition The notation Y = g(x) is actually shorthand for Y (ω) := g(x(ω)). The random variable Y = g(x) is sometimes called derived random variable. Example Let and p X (x) = { 1 c x2, x = ±1, ±2 0, otherwise Y = X 4. Find p Y (y) and then calculate EY For discrete random variable X, the pmf of a derived random variable Y = g(x) is given by p Y (y) = p X (x). x:g(x)=y 114

8 Note that the sum is over all x in the support of X which satisfy g(x) = y. Example A binary random variable X takes only two values a and b with P [X = b] = 1 P [X = a] = p. X can be expressed as X = (b a)i + a, where I is a Bernoulli random variable with parameter p. 9.3 Expectation of a Function of a Discrete Random Variable Recall that for discrete random variable X, the pmf of a derived random variable Y = g(x) is given by p Y (y) = p X (x). x:g(x)=y If we want to compute EY, it might seem that we first have to find the pmf of Y. Typically, this requires a detailed analysis of g which can be complicated, and it is avoided by the following result Suppose X is a discrete random variable. E [g(x)] = x g(x)p X (x). This is referred to as the law/rule of the lazy/unconcious statistician (LOTUS) [?, Thm 3.6 p 48],[8, p. 149],[?, p. 50] because it is so much easier to use the above formula than to first find the pmf of Y. It is also called substitution rule [17, p 271]. Example Back to Example Recall that { 1 p X (x) = c x2, x = ±1, ±2 0, otherwise (a) When Y = X 4, EY = 115

9 (b) E [2X 1] Caution: A frequently made mistake of beginning students is to set E [g(x)] equal to g (EX). In general, E [g(x)] g (EX). (a) In particular, E [ ] 1 X is not the same as 1 EX. (b) An exception is the case of an affine function g(x) = ax + b. See also (9.26). Example Continue from Example 9.4. For X Bernoulli(p), (a) EX = p (b) E [ X 2] = 0 2 (1 p) p = p (EX) 2. Example Continue from Example 9.7. Suppose X P(α). E [ X 2] = i=0 i 2 e ααi i! = e α α α i 1 i (i 1)! We can evaluate the infinite sum in (16) by rewriting i as i 1+1: α i 1 i (i 1)! = α i 1 (i 1 + 1) (i 1)! = (i 1) = α i=2 α i 1 (i 1)! + α i 2 (i 2)! + α i 1 (i 1)! = αeα + e α = e α (α + 1). Plugging this back into (16), we get E [ X 2] = α (α + 1) = α 2 + α. α i 1 (i 1)! (16) Continue from Example 9.8. For X B(n, p), one can find E [ X 2] = np(1 p) + (np)

10 1/3, x { 1, 1}, Example Let p X (x) = 1/6, x { 2, 2}, 0, otherwise. E [ X 2]. Find EX and Some Basic Properties of Expectations (a) For c R, E [c] = c (b) For c R, E [X + c] = EX + c and E [cx] = cex (c) For constants a, b, we have E [ax + b] = aex + b. (d) For constants c 1 and c 2, E [c 1 g 1 (X) + c 2 g 2 (X)] = c 1 E [g 1 (X)] + c 2 E [g 2 (X)]. (e) For constants c 1, c 2,..., c n, [ n ] E c k g k (X) = n c k E [g k (X)]. Definition Some definitions involving expectation of a function of a random variable: [ (a) Absolute moment: E X k] [, where we define E X 0] = 1 (b) Moment: m k = E [ X k] = the k th moment of X, k N. The first moment of X is its expectation EX. The second moment of X is E [ X 2]. 117

11 9.4 Variance and Standard Deviation An average (expectation) can be regarded as one number that summarizes an entire probability model. After finding an average, someone who wants to look further into the probability model might ask, How typical is the average? or, What are the chances of observing an event far from the average? A measure of dispersion/deviation/spread is an answer to these questions wrapped up in a single number. (The opposite of this measure is the peakedness.) If this measure is small, observations are likely to be near the average. A high measure of dispersion suggests that it is not unusual to observe events that are far from the average. Example Consider your score on the midterm exam. After you find out your score is 7 points above average, you are likely to ask, How good is that? Is it near the top of the class or somewhere near the middle?. Example In the case that the random variable X is the random payoff in a game that can be repeated many times under identical conditions, the expected value of X is an informative measure on the grounds of the law of large numbers. However, the information provided by EX is usually not sufficient when X is the random payoff in a nonrepeatable game. Suppose your investment has yielded a profit of $3,000 and you must choose between the following two options: the first option is to take the sure profit of $3,000 and the second option is to reinvest the profit of $3,000 under the scenario that this profit increases to $4,000 with probability 0.8 and is lost with probability 0.2. The expected profit of the second option is 0.8 $4, $0 = $3, 200 and is larger than the $3,000 from the first option. Nevertheless, most people would prefer the first option. The downside risk is too big for them. A measure that takes into account the aspect of risk is the variance of a random variable. [17, p 35] 118

12 9.30. The most important measures of dispersion are the standard deviation and its close relative, the variance. Definition Variance: Var X = E Read the variance of X [ (X EX) 2]. (17) Notation: D X, or σ 2 (X), or σx 2, or VX [?, p. 51] In some references, to avoid confusion from the two expectation symbols, they first define m = EX and then define the variance of X by Var X = E [ (X m) 2]. We can also calculate the variance via another identity: Var X = E [ X 2] (EX) 2 The units of the variance are squares of the units of the random variable Basic properties of variance: Var X 0. Var X E [ X 2]. Var[cX] = c 2 Var X. Var[X + c] = Var X. Var[aX + b] = a 2 Var X. 119

13 Definition Standard Deviation: σ X = Var[X]. It is useful to work with the standard deviation since it has the same units as EX. Informally we think of outcomes within ±σ X of EX as being in the center of the distribution. Some references would informally interpret sample values within ±σ X of the expected value, x [EX σ X, EX + σ X ], as typical values of X and other values as unusual. σ ax+b = a σ X σ X and Var X: Note that the function is a strictly increasing function. Because σ X = Var X, if one of them is large, another one is also large. Therefore, both values quantify the amount of spread/dispersion in RV X (which can be observed from the spread or dispersion of the pmf or the histogram or the relative frequency graph). However, Var X does not have the same unit as the RV X In finance, standard deviation is a key concept and is used to measure the volatility (risk) of investment returns and stock returns. It is common wisdom in finance that diversification of a portfolio of stocks generally reduces the total risk exposure of the investment. We shall return to this point in Example Example Continue from Example If the standard deviation of exam scores is 12 points, the student with a score of +7 with respect to the mean can think of herself in the middle of the class. If the standard deviation is 3 points, she is likely to be near the top. Example Suppose X Bernoulli(p). (a) E [ X 2] = 0 2 (1 p) p = p. 120

14 (b) Var X = EX 2 (EX) 2 = p p 2 = p(1 p). Alternatively, if we directly use (17), we have Var X = E [ (X EX) 2] = (0 p) 2 (1 p) + (1 p) 2 p = p(1 p)(p + (1 p)) = p(1 p). Example Continue from Example 9.7 and Example Suppose X P(α). We have Var X = E [ X 2] (EX) 2 = α 2 + α α 2 = α. Therefore, for Poisson random variable, the expected value is the same as the variance. Example Consider 2.4the Expectation two pmfs shown in Figure 15. The 85 random variable X with pmf at the left has a smaller variance The than variance the random is the average variable squared deviation Y withofpmf X about atits the mean. right The variance because characterizes more probability how likely itmass is to observe is concentrated values of the random near variable zero far (their from its mean. mean) For example, in the consider the two pmfs shown in Figure 2.9. More probability mass is concentrated near zero graph at the left than in the graph at the right. [8, p. 85] in the graph at the left than in the graph at the right. p ( i ) X p Y ( i ) 2 1 1/3 1/ i 2 1 1/3 1/ i Figure 15: Example 9.39 shows that a random variable whose probability mass is concentrated near the mean has smaller variance. [8, Fig. 2.9] Figure 2.9. Example 2.27 shows that the random variable X with pmf at the left has a smaller variance than the random variable Y with pmf at the right. Example Let X and Y be the random variables with respective pmfs shown in Figure We Compute havevar(x) already and var(y talked ). about variance and standard deviation Solution. as By a number symmetry, that both Xindicates and Y havespread/dispersion zero mean, and so var(x) of=e[x the 2 pmf. ] and var(y )=E[Y 2 ]. Write More specifically, let s imagine a pmf that shapes like a bell curve. As the value of E[Xσ 2 ]=( 2) ( 1) (1) (2)2 6 1 X gets smaller, the spread of = the 2, pmf will be smaller and hence the pmf would look sharper. Therefore, the and probability that E[Ythe 2 ]=( 2) random ( 1)2 variable 6 1 +(1)2 X 6 1 would +(2)2 3 1 take = 3. a value that is Thus, far from X andthe Y aremean both zero-mean would be random smaller. variables taking the values ±1 and ±2. But Y is more likely to take values far from its mean. This is reflected by the fact that var(y ) > var(x). When a random variable does not have121 zero mean, it is often convenient to use the variance formula, var(x) =E[X 2 ] (E[X]) 2, (2.17)

15 The next property involves the use of σ X to bound the tail probability of a random variable Chebyshev s Inequality: or equivalently P [ X EX α] σ2 X α 2 Useful only when α > σ X P [ X EX nσ X ] 1 n 2 Example If X has mean m and variance σ 2, it is sometimes convenient to introduce the normalized random variable Y = X m. σ Definition Central Moments: A generalization of the variance is the nth central moment which is defined to be (a) µ 1 = E [X EX] = 0. µ n = E [(X EX) n ]. (b) µ 2 = σ 2 X = Var X: the second central moment is the variance. 122