Theoretical Statistics. Lecture 4. Peter Bartlett

Transcription

1 1. Concentration inequalities. Theoretical Statistics. Lecture 4. Peter Bartlett 1

2 Outline of today s lecture We have been looking at deviation inequalities, i.e., bounds on tail probabilities likep(x n t) for some statistic X n. 1. Using moment generating function bounds, for sums of independent r.v.s: Chernoff; Hoeffding; sub-gaussian, sub-exponential random variables; Bernstein. Today: Johnson-Lindenstrauss. 2. Martingale methods: Hoeffding-Azuma, bounded differences. 2

3 Review. Chernoff technique Theorem: Fort > 0: P(X EX t) inf λ>0 e λt M X µ (λ). Theorem: [Hoeffding s Inequality] For a random variable X [a, b] with EX = µ and λ R, lnm X µ (λ) λ2 (b a)

4 Review. Sub-Gaussian, Sub-Exponential Random Variables Definition: X is sub-gaussian with parameter σ 2 if, for all λ R, lnm X µ (λ) λ2 σ 2 2. Definition: X is sub-exponential with parameters (σ 2,b) if, for all λ < 1/b, lnm X µ (λ) λ2 σ

5 Review. Sub-Exponential Random Variables Theorem: ForX sub-exponential with parameters (σ 2,b), ( ) exp t2 2σ if0 t σ 2 /b, P (X µ+t) 2 exp ( ) t ift > σ 2 /b. 2b For independent X i, sub-exponential with parameters (σ 2 i,b i), the sum X = X 1 + +X n is sub-exponential with parameters ( i σ2 i,max ib i ). Example: X χ 2 1 is sub-exponential with parameters (4,4). 5

6 Sub-Exponential Random Variables: Example Theorem: [Johnson-Lindenstrauss] For m points x 1,...,x m from R d, there is a projection F : R d R n that preserves distances in the sense that, for all x i,x j, (1 δ) x i x j 2 2 F(x i ) F(x j ) 2 2 (1+δ) x i x j 2 2, provided that n > (16/δ 2 )logm. That is, we can embed these points inr n and approximately maintain their distance relationships, provided that n is not too small. Notice that n is independent of the ambient dimension d, and depends only logarithmically on the number of pointsm. 6

7 Johnson-Lindenstrauss Applications: dimension reduction to simplify computation (nearest neighbor, clustering, image processing, text processing). Analysis of machine learning methods: separable by a large margin in high dimensions implies it s really a low-dimensional problem after all. 7

8 Johnson-Lindenstrauss Embedding: Proof We use a random projection: F(x) = 1 n Yx, where Y R n d has independent N(0,1) entries. Let Y i denote theith row, for 1 i n. It has an(0,i) distribution, so Y T i x/ x 2 N(0,1). Thus, Z = Yx 2 2 x 2 2 = n i=1 ( Y T i x/ x ) 2 χ 2 n. 8

9 Johnson-Lindenstrauss Embedding: Proof SinceZ χ 2 n is the sum ofnindependent sub-exponential (4,4) random variables, it is sub-exponential (4n,4). And we have that for 0 < t < n, P ( Z 1 t) 2exp( t 2 /(8n)). Hence, for0 < δ < 1, ( Yx 2 ) 2 P n x 2 1 δ 2exp( nδ 2 /8) 2 ( ) F(x) 2 P 2 x 2 [1 δ,1+δ] 2exp( nδ 2 /8). 2 9

10 Johnson-Lindenstrauss Embedding: Proof Applying this to the ( m 2) distinct pairs x = xi x j, and using the union bound gives P ( i j s.t. F(x i x j ) 2 2 x i x j 2 2 ) [1 δ,1+δ] ( ) m 2 exp( nδ 2 /8). 2 Thus, for n > 16/δ 2 log(m), this probability is strictly less than 1, so there exists a suitable mapping. In fact, we can choose a random projection in this way and ensure that the probability that it does not satisfy the approximate isometry property is no more thanǫfor n > 16/δ 2 log(m/ǫ). 10

11 Concentration Bounds for Martingale Difference Sequences Next, we re going to consider concentration of martingale difference sequences. The application is to understand how tails of f(x 1,...,X n ) Ef(X 1,...,X n ) behave, for some function f. [e.g., in the homework, we have that f is some measure of the performance of a kernel density estimator.] If we write f(x 1,...,X n ) Ef(X 1,...,X n ) n = E[f(X 1,...,X n ) X 1,...,X i ] E[f(X 1,...,X n ) X 1,...,X i 1 ], i=1 then we have represented this deviation as a martingale difference sequence. 11

12 Martingales Definition: A sequencey n of random variables adapted to a filtrationf n is a martingale if, for all n, E Y n < E[Y n+1 F n ] = Y n. F n is a filtration means these σ-fields are nested: F n F n+1. Y n is adapted tof n means that each Y n is measurable with respect tof n. e.g. F n = σ(y 1,...,Y n ), theσ-field generated by the first n variables. Then we sayy n is a martingale sequence. e.g. F n = σ(x 1,...,X n ). Then Y n is a martingale sequence wrt X n. 12

13 Martingale Difference Sequences Definition: A sequence D n of random variables adapted to a filtration F n is a martingale difference sequence if, for all n, E D n < E[D n+1 F n ] = 0. e.g., D n = Y n Y n 1. E[D n+1 F n ] = E[Y n+1 F n ] E[Y n F n ] = E[Y n+1 F n ] Y n = 0 (because Y n is measurable wrt F n, and because of the martingale property). Hence, Y n Y 0 = n i=1 D i. 13

14 Martingale Difference Sequences: the Doob construction Define X = (X 1,...,X n ), X i 1 = (X 1,...,X i ), Y 0 = Ef(X), Y i = E[f(X) X1]. i n Then f(x) Ef(X) = Y n Y 0 = D i, where D i = Y i Y i 1. Also,Y i is a martingale w.r.t.x i, and hence D i is a martingale difference sequence. Indeed (because EX = EE[X Y]), i=1 E[Y i+1 X i 1] = E [ E[f(X) X i+1 1 ] X i 1] = E[f(X) X i 1 ] = Y i. 14

15 Martingale Difference Sequences: another example [An aside:] Consider two densities f and g, withg absolutely continuous w.r.t.f. Suppose X n are drawn i.i.d. from f, and Y n is the likelihood ratio, i=1 Y n = n i=1 g(x i ) f(x i ). Then Y n is a martingale w.r.t. X n. Indeed, [ n+1 ] E[Y n+1 X1] n g(x i ) = E f(x i ) Xn 1 = E = n i=1 g(x i ) f(x i ) = Y n, because E[g(X n+1 )/f(x n+1 )] = 1. [ ] g(xn+1 ) n f(x n+1 ) i=1 g(x i ) f(x i ) 15

16 Concentration Bounds for Martingale Difference Sequences Theorem: Consider a martingale difference sequence D n (adapted to a filtration F n ) that satisfies for λ 1/b n a.s.,e[exp(λd n ) F n 1 ] exp(λ 2 σ 2 n/2). Then n i=1 D i is sub-exponential, with(σ 2,b) = ( n i=1 σ2 i,max ib i ). ( ) 2exp( t 2 /(2σ 2 )) if0 t σ 2 /b P D i t 2exp( t/(2b)) ift > σ 2 /b. i 16

17 Concentration Bounds for Martingale Difference Sequences Proof: ( Eexp λ i D i ) = E E [ [ exp exp ( ( λ λ n 1 i=1 n 1 i=1 D i )E[exp(λD n ) F n 1 ] D i )]exp(λ 2 σ 2 n/2), ] provided λ < b. Iterating shows that i D i is sub-exponential. 17

18 Concentration Bounds for Martingale Difference Sequences Theorem: Consider a martingale difference sequence D i with D i B i a.s. Then ( ) ) P D i t 2exp ( 2t2. i B2 i Proof: It suffices to show that i E[exp(λD i ) F i 1 ] exp(λ 2 B 2 i/2) But D i B i a.s., so the conditioned variable (D i F i 1 ) B i a.s., so it is sub-gaussian with parameter σ 2 i = B2 i. 18

19 Bounded Differences Inequality Theorem: Suppose f : X n R satisfies the following bounded differences inequality: for all x 1,...,x n,x i X, f(x 1,...,x n ) f(x 1,...,x i 1,x i,x i+1,...,x n ) B i. Then P ( f(x) Ef(X) t) 2exp ( 2t2 i B2 i ). 19

20 Bounded Differences Inequality Proof: Use the Doob construction. Then Y i = E[f(X) X i 1], D i = Y i Y i 1, n f(x) Ef(X) = D i. i=1 D i = Y i Y i 1 = E[f(X) X1] E[f(X) X i 1 i 1 ] = E [ E[f(X) X1] f(x) ] i X1 i 1 B i. 20

21 For a set A R n, consider Examples: Rademacher Averages Z = sup ǫ,a, a A where ǫ = (ǫ 1,...ǫ n ) is a sequence of i.i.d. uniform {±1} random variables. Define the Rademacher complexity ofaas R(A) = EZ. [This is a measure of the size ofa.] The bounded differences approach implies that Z is concentrated around R(A): Theorem: Z is sub-gaussian with parameter 4 i sup a Aa 2 i. Proof: WriteZ = f(ǫ 1,...,ǫ n ), and notice that a change ofǫ i can lead to a change in Z of no more than B n = sup a A 2 a i. The result follows. 21

22 Examples: Empirical Processes For a class F of functions f : X [0,1], suppose that X 1,...,X n,x are i.i.d. onx, and consider Z = sup f F Ef(X) 1 n f(x i ) n = Pf } P {{ n f. } i=1 F emp proc If Z converges to0, this is called a uniform law of large numbers. Here, we show that Z is concentrated about EZ: Theorem: Z is sub-gaussian with parameter 1/n. Proof: WriteZ = g(x 1,...,X n ), and notice that a change ofx i can lead to a change in Z of no more than B n = 1/n. The result follows. 22