Modes of Convergence

Transcription

1 Moes of Convergence Electrical Engineering 126 (UC Berkeley Spring 2018 There is only one sense in which a sequence of real numbers (a n n N is sai to converge to a limit. Namely, a n a if for every ε > 0 there exists a positive integer N such that the sequence after N is always within ε of the suppose limit a. In contrast, the notion of convergence becomes somewhat more subtle when iscussing convergence of functions. In this note we briefly escribe a few moes of convergence an explain their relationship. Since the subject quickly becomes very technical, we will state many of the funamental results without proof. Throughout this iscussion, fix a probability space Ω an a sequence of ranom variables (X n n N. Also, let X be another ranom variable. 1 Almost Sure Convergence The sequence (X n n N is sai to converge almost surely or converge with probability one to the limit X, if the set of outcomes ω Ω for which X n (ω X(ω forms an event of probability one. In other wors, all observe realizations of the sequence (X n n N converge to the limit. We abbreviate almost surely by an we enote this moe of convergence by X n X. Of course, one coul efine an even stronger notion of convergence in which we require X n (ω X(ω for every outcome (rather than for a set of outcomes with probability one, but the philosophy of probabilists is to isregar events of probability zero, as they are never observe. Thus, we regar convergence as the strongest form of convergence. One of the most celebrate results in probability theory is the statement that the sample average of ientically istribute ranom variables, uner very weak assumptions, converges to the expecte value of their common istribution. This is known as the Strong Law of Large Numbers (SLLN. Theorem 1 (Strong Law of Large Numbers. If (X n n N are pairwise inepenent an ientically istribute with E[ X 1 ] <, then n 1 n i=1 X i E[X 1]. An example we will see later in the course is in the context of iscrete-time Markov chains: oes the fraction of time spent in a state converge to a value, an if so, to what value? Another example of convergence that we will stuy is the asymptotic equipartition property from information theory, an its relevance to coing. Finally, another question of interest comes from machine learning: if we use the stochastic graient escent algorithm to minimize a function, o the iterates converge to the true minimizer of the function? 1

2 2 Convergence in robability X, if for every 0. In other wors, for any fixe ε > 0, the probability that the Next, (X n n N is sai to converge in probability to X, enote X n ε > 0, ( X n X > ε sequence eviates from the suppose limit X by more than ε becomes vanishingly small. We now seek to prove that convergence implies convergence in probability. Theorem 2. If X n X, then X n X. roof. Fix ε > 0. Define A n := m=n { X m X > ε} to be the event that at least one of X n, X n+1,... eviates from X by more than ε. Observe that A 1 A 2 ecreases to an event A which has probability zero, since the convergence of the sequence (X n n N implies that for all outcomes ω (outsie of an event of probability zero, the sequence of real numbers ( X n (ω X(ω n N is eventually boune by ε. Thus, However, the converse is not true. ( X n X > ε (A n (A = 0. Example 1. The stanar example of a sequence of ranom variables which converges in probability but not is the following. First, set X n = 0 for all n N. Then, for each j N, pick an inex N j uniformly at ranom from {2 j,..., 2 j+1 1} an set X Nj = 1. Observe that (X n 0 = (N log2 n = n = 2 log 2 n 0, so X n 0. However, for any ω Ω, the sequence of real numbers (X n (ω n N takes on both values 0 an 1 infinitely often, so (X n (ω n N oes not converge an hence (X n n N oes not converge to X. As a consequence of the SLLN (Theorem 1 an Theorem 2, then if (X n n N are pairwise inepenent an ientically istribute with E[ X 1 ] <, then n 1 n i=1 X i E[X 1]. This is known as the Weak Law of Large Numbers (WLLN. The istinction between convergence an convergence in probability manifests itself in applications in the following way. If you have convergence in probability, then you know that the probability of a eviation of any particular size goes to zero, but you may inee observe such eviations forever; if you ha to pay a ollar for each ε-evation, you might en up paying up infinite ollars. In contrast, with convergence you are assure that for any observe realization, there will come a time in the sequence after which there will never be any such eviations, an thus you will only lose a finite amount of money. 3 Convergence in Distribution Finally, the last moe of convergence that we will iscuss is convergence in istribution or convergence in law. Here, X n X if for each x R such that (X = x = 0, we (X x. Notice that unlike the previous two forms of convergence, have (X n x convergence in istribution oes not require all of the ranom variables to be efine on the same probability space. 2

3 First we explain why we require (X n x (X x only at points x for which (X = x = 0. Example 2. Consier the sequence of constant ranom variables (X n n N, where we efine X n := 2 n. We woul like to assert that X n X, where X := 0. However, (X n 0 = 0 for all n N, whereas (X 0 = 1, so in particular (X n 0 oes not converge to (X 0. Notice that in this example, (X = 0 = 1, so we can fix this issue by only looking at points x for which (X = x = 0, i.e., points at which the CDF of X is continuous. In the following important special cases, convergence in istribution is easier to escribe: Theorem If (X n n N an X take values in Z, an if (X n = x (X = x for all x Z, then X n X. 2. If (X n n N an X are continuous ranom variables, an if f Xn (x f X(x for all x R, then X n X. Next we show that convergence in probability implies convergence in istribution. Theorem 4. If X n X, then X n X. roof. Let x be a point such that (X = x = 0. Fix ε > 0. We can write Similarly, we write (X n x = (X n x, X n X < ε + (X n x, X n X ε (X x + ε + ( X n X ε. (X x ε = (X x ε, X n X < ε + (X x ε, X n X ε Combining the two bouns, we have (X n x + ( X n X ε. (X x ε ( X n X ε (X n x (X x + ε + ( X n X ε. Since X n X, then ( X n X ε 0, so the boun above tells us that eventually the sequence ((X n x n N is trappe between (X x ε an (X x + ε. This is true for all ε > 0 an the CDF of X is continuous at x by assumption, so by taking ε 0, we conclue that (X n x (X x. The converse is not true: convergence in istribution oes not imply convergence in probability. In fact, a sequence of ranom variables (X n n N can converge in istribution even if they are not jointly efine on the same sample space! (This is because convergence in istribution is a property only of their marginal istributions. In contrast, convergence in probability requires the ranom variables (X n n N to be jointly efine on the same sample space, an etermining whether or not convergence in probability hols requires some knowlege about the joint istribution of (X n n N. Even when the ranom variables (X n n N are jointly efine, it is possible to construct counterexamples: 3

4 Example 3. Let X 0 Uniform[ 1, 1], an for each positive integer n, let X n := ( 1 n X 0. Then, X n = Uniform[ 1, 1] for all n N because the Uniform[ 1, 1] istribution is symmetric aroun the origin, so convergence in istribution hols (for a silly reason: all of the marginal istributions are the same. However, (X n n N oes not converge in probability (think about why this is true. Thus we have built a small hierarchy ( X n X T heorem 2 ====== ( X n ( X T heorem 4 ====== X n X. We now precisely state the Central Limit Theorem (CLT, which is an assertion about convergence in istribution. Theorem 5 (Central Limit Theorem. If (X n n N is a sequence of i.i.. ranom variables with common mean µ an finite variance σ 2, then n i=1 X i nµ σ n Z, where Z is a stanar Gaussian ranom variable. Explicitly, for all x R, ( n i=1 X i nµ σ n x x 1 2π exp ( z2 2 z. The CLT plays a huge role in statistics, where it is use to provie asymptotic confience intervals. Similarly, statisticians work towars proving convergence in istribution to other common istributions in statistics, such as the chi-square istribution or the t istribution. Another example of convergence in istribution is the oisson Law of Rare Events, which is use as a justification for the use of the oisson istribution in moels of rare events. Theorem 6 (oisson Law of Rare Events. If X n Binomial(n, p n where p n 0 such that np n λ > 0, then X n X, where X oisson(λ. In fact, many other situations (especially concerning balls an bins have oisson limits, an oisson limits are use in popular ranom graph moels. 4 Miscellaneous Results 4.1 Continuous Mapping One of the most useful results is presente below: Theorem 7 (Continuous Mapping. Let f be a continuous function. 1. If X n X, then f(x n f(x. 2. If X n X, then f(x n f(x. 4

5 3. If X n X, then f(x n f(x. A typical application is to analyze a sequence of ranom variables (X n n N by applying a log or exp transformation, which is useful when showing the convergence of (log X n n N or (exp X n n N is easier than showing that the original sequence (X n n N converges. 4.2 Convergence of Expectation In general, none of the above moes of convergence imply that E[X n ] an example, let U Uniform[0, 1] an let X n := n1{u n 1 }. Then, X n E[X]. As 0, but E[X n ] = 1 for all n. In more avance treatments of probability theory, convergence of expecte values is quite important, an there are a number of technical tools calle convergence theorems use to justify convergence of expectations. Although we will not nee them, we will state them here. Theorem 8 (Convergence Theorems. Suppose X n X. 1. (Monotone Convergence If 0 X 1 X 2 X 3, then E[X n ] E[X]. 2. (Dominate Convergence If there exists a ranom variable Y 0 with E[Y ] < an X n, X Y for all n, then E[X n ] E[X]. 5