The Autocorrelation Function and AR(1), AR(2) Models

Transcription

1 The Autocorrelation Function and AR(1), AR(2) Models Al Nosedal University of Toronto January 21, 2016 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

2 Definition The sample autocovariance function is defined as ˆγ(h) = 1 n h (x t+h x)(x t x), n t=1 with ˆγ( h) = ˆγ(h) for h = 0, 1,..., n 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

3 Definition The sample autocorrelation function is defined as ˆρ(h) = ˆγ(h) ˆγ(0). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

4 Toy Example To understand autocorrelation it is first necessary to understand what it means to lag a time series. Y (t) = Y t =(3,4,5,6,7,8,9,10,11,12) Y lagged 1 = Y (t 1) = Y t 1 =(*,3,4,5,6,7,8,9,10,11) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

5 Toy Example (cont.) Find the sample autocorrelation at lag 1 for the following time series: Y (t) = (3, 4, 5, 6, 7, 8, 9, 10, 11, 12). Answer. ˆρ(1) = 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

6 R Code y<-3:12 ### function to find autocorrelation when lag=1; my.auto<-function(x){ n<-length(x) denom<-(x-mean(x))%*%(x-mean(x))/n num<-(x[-n]-mean(x))%*%(x[-1]-mean(x))/n result<-num/denom return(result) } my.auto(y) ## [,1] ## [1,] 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

7 An easier way of finding autocorrelations Using R, we can easily find the autocorrelation at any lag k. y<-3:12 auto.lag1<-acf(y,lag=1)$acf[2] auto.lag1 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

8 Series y ACF Lag ## [1] 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

9 Background It is natural to plot the autocovariances and/or the autocorrelations versus lag. Further, it will be important to develop some notion of what would be expected from such a plot if the errors are white noise (meaning no special time series techniques are required) in contrast to the situation strong serial correlation is present. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

10 Graph of autocorrelations A graph of the lags against the corresponding autocorrelations is called a correlogram. The following lines of code can be used to make a correlogram in R. ### autocorrelation function for ###a random series set.seed(2016); y<-rnorm(25); acf(y,lag=8,main="random Series N(0,1)"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

11 Graph Random Series N(0,1) ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

12 The AR(1) Model Autocorrelation and Autocovariance For the AR(1) model, x j = a 1 x j 1 + w j, x j = a1 2x j 2 + a 1 w j 1 + w j, and in general x j = a1 kx j k + k t=1 at 1 1 w j t+1. Furthermore γ(1) = E(x j x j 1 ) = E([a 1 x j 1 + w k ]x j 1 ) = a 1 σar 2 = a 1γ(0). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

13 The AR(1) Model Autocorrelation and Autocovariance More generally, γ(k) = E(x j x j k ) = E([a k 1 x j k + k t=1 at 1 1 w j t+1 ]x j k ) = a k 1 γ(0). So, in general ρ(k) = a k 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

14 The AR(1) Model Autocorrelation and Autocovariance For AR(1), the relationship of variance in the series to variance in white noise is σ 2 AR = E(x jx j ) = E([a 1 x j 1 + w j ][a 1 x j 1 + w j ]) = a 2 1 σ2 AR + σ2 w, so σ 2 AR = σ2 w 1 a1 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

15 The Stationary AR(1) is an MA model of infinite order Here we introduce the fundamental duality between AR and MA models. We can keep going backward in time using the first-order autoregressive model: x t = a 1 x t 1 + w t x t = a 1 (a 1 x t 2 + w t 1 ) + w t or x t = a 2 1 x t 2 + (a 1 w t 1 + w t ) = a 2 1 (a 1x t 3 + w t 2 ) + (a 1 w t 1 + w t ) = a 3 1 x t 3 + a 2 1 w t 2 + a 1 w t 1 + w t Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

16 The Stationary AR(1) is an MA model of infinite order Continuing back to minus infinity we would get x t = w t + a 1 w t 1 + a1w 2 t 2 + a1x 3 t which makes sense if it is the case that a1 k 0 as k rapidly enough for the series to converge to a finite limit. This is our stationary condition. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

17 The Stationary AR(1) is an MA model of infinite order The expression for x t is then x t = a1w i t i The AR(1) model can thus be written as an MA( ) model. i=0 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

18 The AR(1) process is not always stationary The variance of the AR(1) process is given by ( ) σar 2 = Var(x t) = Var a1w i t i σar 2 = Var(x t) = σw 2 (1 + a1 2 + (a1) (a1) (a1) ) If a 1 = 1 or if a 1 is larger, this variance will increase without bound. i=0 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

19 The Yule-Walker equations The development of these useful equations is not difficult. Write the general AR(p) model as x t = a 1 x t 1 + a 2 x t a p x t p + w t where we assume that x t is a zero-mean process (or that the mean has been subtracted) and that w t is a white-noise process and that E(w t x t k ) = 0 for k > 0. Once again, compute γ(k): γ(k) = E(x t x t k ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

20 The Yule-Walker equations γ(k) = E(x t x t k ) = E[(a 1 x t 1 + a 2 x t a p x t p + w t )x t k ] γ(k) = E(x t x t k ) = a 1 E[x t 1 x t k ] + a 2 E[x t 2 x t k ] a p E[x t p x t k ] + E[w t x t k ] Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

21 The Yule-Walker equations From the definition of the autocovariance γ(k) of a stationary process, it is a function only of the lag between observations. Thus, γ(k) = E(x t x t k ) = E(x t+s x t+s k ). We can use this fact to simplify γ(k) = E(x t x t k ) = a 1 E[x t 1 x t k ] + a 2 E[x t 2 x t k ] a p E[x t p x t k ] + E[w t x t k ] to obtain γ(k) = a 1 γ(k 1) + a 2 γ(k 2) a p γ(k p) + E(w t x t k ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

22 The Yule-Walker equations For k > 0 we know that the last term is zero. γ(k) = a 1 γ(k 1) + a 2 γ(k 2) a p γ(k p). If we divide by the variance of the series γ(0) = σar 2 and recall the definition of the autocorrelation ρ(k) = γ(k) γ(0), we obtain the Yule-Walker equations ρ(k) = a 1 ρ(k 1) + a 2 ρ(k 2) a p ρ(k p), k > 0. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

23 The Yule-Walker equations These are extremely important equations. For k = 0, we obtain γ(0) = σ 2 AR = a 1γ( 1) + a 2 γ( 2) a p γ( p) + E(w t x t ). We can find E(w t x t ) as follows: E(x t w t ) = E[a 1 x t 1 w t + a 2 x t 2 w t a p x t p w t + w 2 t ] = E(w 2 t ) = σ 2 w. (because E(x t k w t ) = 0 for k > 0.) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

24 The Yule-Walker equations Recalling that γ( s) = γ(s), we have that γ(0) = σ 2 AR = a 1γ(1) + a 2 γ(2) a p γ(p) + σ 2 w, and dividing by γ(0) on both sides, we obtain or 1 = a 1 ρ(1) + a 2 ρ(2) a p ρ(p) + σ2 w σ 2 AR σ 2 w = σ 2 AR (1 a 1ρ(1) a 2 ρ(2)... a p ρ(p). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

25 Why are Yule-Walker equations so important? We can use equations and ρ(k) = a 1 ρ(k 1) + a 2 ρ(k 2) a p ρ(k p), k > 0. σ 2 w = σ 2 AR (1 a 1ρ(1) a 2 ρ(2)... a p ρ(p). to estimate, say, a 1, a 2, σ 2 w from the autocorrelations if we knew that the order of the model is p = 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

26 AR(2) Process This process is defined by It can be shown that x t = a 1 x t 1 + a 2 x t 2 + w t. a 1 = a 2 = ρ(1)[1 ρ(2)] 1 ρ(1) 2 ρ(2) ρ(1)2 1 ρ(1) 2 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

27 Stationarity conditions for an AR(2) process We recently discovered that the equation for the variance of an AR(p) process was σ 2 AR = For an AR(2) process this reduces to σ 2 w 1 ρ(1)a 1 ρ(2)a 2... ρ(p)a p. σ 2 AR = σ 2 w 1 ρ(1)a 1 ρ(2)a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

28 Stationarity conditions for an AR(2) process Substituting our expressions for a 1 and a 2 can be shown to yield (after a little bit of algebra) σ 2 AR = (1 a 2 )σ 2 w (1 + a 2 )(1 a 1 a 2 )(1 + a 1 a 2 ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

29 Stationarity conditions for an AR(2) process Using the facts that autocorrelations must be less than 1, and each factor in the denominator and numerator must be positive, we can derive the conditions 1 < a 2 < 1 a 2 + a 1 < 1 a 2 a 1 < 1 The inequalities define the stationarity region for AR(2). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

30 Scatterplot to Illustrate the AR(1) Model The model AR(1) is about the correlation between an error and the previous error. First consider white noise. In this case there should be no correlation between the errors and the shifted errors. On the other hand, consider AR(1) errors with a 1 = 0.7. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

31 R Code n<-1000; error<-rep(0,n); a1<-0.7; # simulating white noise; set.seed(9999); noise<-rnorm(n,0,2); # simulating AR(1); error = filter(noise,filter=(0.7),method="recursive",init=0); plot(error[-n],error[-1],xlab="errors", ylab="shifted errors"); title("ar(1) errors, a=0.7" ); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

32 Scatterplot AR(1) errors, a=0.7 shifted errors errors Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

33 Autocorrelation Function acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

34 Autocorrelation Function Series error ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

35 Based on the formulas already derived and the choice used in the code (a 1 = 0.7): ρ(0) = 1 by definition ρ(1) = a 1 1 = 0.7 ρ(2) = a 2 1 = 0.49 ρ(3) = a 3 1 = ρ(4) = a 4 1 = Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

36 R Code acf(error,plot=false)[0:4]; ## ## Autocorrelations of series 'error', by lag ## ## ## Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

37 Examples of Stable and Unstable AR(1) Models Consider the following three cases of AR(1) data: 1. a 1 = a 1 = a 1 = 1.01 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

38 Case 1 n<-100; error<-rep(0,n); a1<- -0.9; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = -0.9, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

39 Case 1. Time Series a = 0.9, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

40 Case 1. Correlogram Series error ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

41 Case 2 n<-100; error<-rep(0,n); a1<- 0.5; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = 0.5, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

42 Case 2. Time Series a = 0.5, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

43 Case 2. Correlogram Series error ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

44 Case 3 n<-100; error<-rep(0,n); a1<- 1.1; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = 1.1, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

45 Case 3. Time Series a = 1.1, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

46 Case 3. Correlogram Series error ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

47 The AR(2) Model Autocorrelation and Autocovariance The model is: x t = a 1 x t 1 + a 2 x t 2 + w t, and the autocovariances can be characterized with recursive relationships. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

48 γ(1) and ρ(1) γ(1) = E(x j x j 1 ) = E([a 1 x j 1 + a 2 x j 2 + w j ]x j 1 ) = a 1 γ(0) + a 2 γ(1), so it is easy to see that ρ(1) = a 1 1 a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

49 ρ(2) and ρ(3) Doing something similar, and ρ(2) = a 2 1 (1 a 2 ) + a 2 ρ(3) = a3 1 + a 1a 2 (1 a 2 ) + a 1a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

50 σ 2 AR = γ(0) It can be shown that, for the AR(2) model, σ 2 AR = σ 2 W 1 a 1 ρ(1) a 2 ρ(2). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

51 Simulating Data for AR(2) Models n<-100; error<-rep(0,n); a1<- 0.8; a2<- -0.7; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

52 Correlogram Series error ACF Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

53 Autocorrelations Based on the formulas already derived and the choices used in the code (a 1 = 0.8, a 2 = 0.7): ρ(0) = 1, ρ(1) = a 1 1 a 2 = , ρ(2) = a 1 ρ(1) + a 2 ρ(0) 0.8(0.4706) 0.7(1) , ρ(3) = a 1 ρ(2) + a 2 ρ(1) 0.8( ) 0.7(0.4706) , etc. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

54 Stable and Unstable AR(2) Models Recall the AR(2) process is stationary when: i) a 1 + a 2 < 1, ii) a 2 a 1 < 1, and iii) a 2 < 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

55 Examples of Stable and Unstable AR(2) Models Stable: a 1 = 1.60 and a 2 = Stable: a 1 = 0.30 and a 2 = Stable: a 1 = 0.30 and a 2 = Unstable violates (i): a 1 = and a 2 = Unstable violates (ii): a 1 = and a 2 = Unstable violates (iii): a 1 = 0 and a 2 = Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

56 Unstable case a 1 = and a 2 = n<-100; error<-rep(0,n); a1< ; a2< ; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = 0.5 and a2=-0.505, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

57 Unstable case a 1 = and a 2 = a1 = 0.5 and a2=0.505, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

58 Unstable case a 1 = and a 2 = n<-100; error<-rep(0,n); a1< ; a2< ; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = and a2=0.500, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

59 Unstable case a 1 = and a 2 = a1 = and a2=0.500, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

60 Unstable case a 1 = 0 and a 2 = n<-100; error<-rep(0,n); a1<- 0; a2< ; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = 0 and a2= -1.05, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61

61 Unstable case a 1 = 0 and a 2 = a1 = 0 and a2= 1.05, n =100 error Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, / 61