The Autocorrelation Function and AR(1), AR(2) Models

The Autocorrelation Function and AR(1), AR(2) Models Al Nosedal University of Toronto January 21, 2016 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 1 / 61

Definition The sample autocovariance function is defined as ˆγ(h) = 1 n h (x t+h x)(x t x), n t=1 with ˆγ( h) = ˆγ(h) for h = 0, 1,..., n 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 2 / 61

Definition The sample autocorrelation function is defined as ˆρ(h) = ˆγ(h) ˆγ(0). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 3 / 61

Toy Example To understand autocorrelation it is first necessary to understand what it means to lag a time series. Y (t) = Y t =(3,4,5,6,7,8,9,10,11,12) Y lagged 1 = Y (t 1) = Y t 1 =(*,3,4,5,6,7,8,9,10,11) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 4 / 61

Toy Example (cont.) Find the sample autocorrelation at lag 1 for the following time series: Y (t) = (3, 4, 5, 6, 7, 8, 9, 10, 11, 12). Answer. ˆρ(1) = 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 5 / 61

R Code y<-3:12 ### function to find autocorrelation when lag=1; my.auto<-function(x){ n<-length(x) denom<-(x-mean(x))%*%(x-mean(x))/n num<-(x[-n]-mean(x))%*%(x[-1]-mean(x))/n result<-num/denom return(result) } my.auto(y) ## [,1] ## [1,] 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 6 / 61

An easier way of finding autocorrelations Using R, we can easily find the autocorrelation at any lag k. y<-3:12 auto.lag1<-acf(y,lag=1)$acf[2] auto.lag1 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 7 / 61

Series y ACF 0.5 0.5 0.0 0.2 0.4 0.6 0.8 1.0 Lag ## [1] 0.7 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 8 / 61

Background It is natural to plot the autocovariances and/or the autocorrelations versus lag. Further, it will be important to develop some notion of what would be expected from such a plot if the errors are white noise (meaning no special time series techniques are required) in contrast to the situation strong serial correlation is present. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 9 / 61

Graph of autocorrelations A graph of the lags against the corresponding autocorrelations is called a correlogram. The following lines of code can be used to make a correlogram in R. ### autocorrelation function for ###a random series set.seed(2016); y<-rnorm(25); acf(y,lag=8,main="random Series N(0,1)"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 10 / 61

Graph Random Series N(0,1) ACF 0.4 0.2 0.8 0 2 4 6 8 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 11 / 61

The AR(1) Model Autocorrelation and Autocovariance For the AR(1) model, x j = a 1 x j 1 + w j, x j = a1 2x j 2 + a 1 w j 1 + w j, and in general x j = a1 kx j k + k t=1 at 1 1 w j t+1. Furthermore γ(1) = E(x j x j 1 ) = E([a 1 x j 1 + w k ]x j 1 ) = a 1 σar 2 = a 1γ(0). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 12 / 61

The AR(1) Model Autocorrelation and Autocovariance More generally, γ(k) = E(x j x j k ) = E([a k 1 x j k + k t=1 at 1 1 w j t+1 ]x j k ) = a k 1 γ(0). So, in general ρ(k) = a k 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 13 / 61

The AR(1) Model Autocorrelation and Autocovariance For AR(1), the relationship of variance in the series to variance in white noise is σ 2 AR = E(x jx j ) = E([a 1 x j 1 + w j ][a 1 x j 1 + w j ]) = a 2 1 σ2 AR + σ2 w, so σ 2 AR = σ2 w 1 a1 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 14 / 61

The Stationary AR(1) is an MA model of infinite order Here we introduce the fundamental duality between AR and MA models. We can keep going backward in time using the first-order autoregressive model: x t = a 1 x t 1 + w t x t = a 1 (a 1 x t 2 + w t 1 ) + w t or x t = a 2 1 x t 2 + (a 1 w t 1 + w t ) = a 2 1 (a 1x t 3 + w t 2 ) + (a 1 w t 1 + w t ) = a 3 1 x t 3 + a 2 1 w t 2 + a 1 w t 1 + w t Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 15 / 61

The Stationary AR(1) is an MA model of infinite order Continuing back to minus infinity we would get x t = w t + a 1 w t 1 + a1w 2 t 2 + a1x 3 t 3 +... which makes sense if it is the case that a1 k 0 as k rapidly enough for the series to converge to a finite limit. This is our stationary condition. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 16 / 61

The Stationary AR(1) is an MA model of infinite order The expression for x t is then x t = a1w i t i The AR(1) model can thus be written as an MA( ) model. i=0 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 17 / 61

The AR(1) process is not always stationary The variance of the AR(1) process is given by ( ) σar 2 = Var(x t) = Var a1w i t i σar 2 = Var(x t) = σw 2 (1 + a1 2 + (a1) 2 2 + (a1) 2 3 + (a1) 2 4 +...) If a 1 = 1 or if a 1 is larger, this variance will increase without bound. i=0 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 18 / 61

The Yule-Walker equations The development of these useful equations is not difficult. Write the general AR(p) model as x t = a 1 x t 1 + a 2 x t 2 +... + a p x t p + w t where we assume that x t is a zero-mean process (or that the mean has been subtracted) and that w t is a white-noise process and that E(w t x t k ) = 0 for k > 0. Once again, compute γ(k): γ(k) = E(x t x t k ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 19 / 61

The Yule-Walker equations γ(k) = E(x t x t k ) = E[(a 1 x t 1 + a 2 x t 2 +... + a p x t p + w t )x t k ] γ(k) = E(x t x t k ) = a 1 E[x t 1 x t k ] + a 2 E[x t 2 x t k ] +... + a p E[x t p x t k ] + E[w t x t k ] Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 20 / 61

The Yule-Walker equations From the definition of the autocovariance γ(k) of a stationary process, it is a function only of the lag between observations. Thus, γ(k) = E(x t x t k ) = E(x t+s x t+s k ). We can use this fact to simplify γ(k) = E(x t x t k ) = a 1 E[x t 1 x t k ] + a 2 E[x t 2 x t k ] +... + a p E[x t p x t k ] + E[w t x t k ] to obtain γ(k) = a 1 γ(k 1) + a 2 γ(k 2) +... + a p γ(k p) + E(w t x t k ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 21 / 61

The Yule-Walker equations For k > 0 we know that the last term is zero. γ(k) = a 1 γ(k 1) + a 2 γ(k 2) +... + a p γ(k p). If we divide by the variance of the series γ(0) = σar 2 and recall the definition of the autocorrelation ρ(k) = γ(k) γ(0), we obtain the Yule-Walker equations ρ(k) = a 1 ρ(k 1) + a 2 ρ(k 2) +... + a p ρ(k p), k > 0. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 22 / 61

The Yule-Walker equations These are extremely important equations. For k = 0, we obtain γ(0) = σ 2 AR = a 1γ( 1) + a 2 γ( 2) +... + a p γ( p) + E(w t x t ). We can find E(w t x t ) as follows: E(x t w t ) = E[a 1 x t 1 w t + a 2 x t 2 w t +... + a p x t p w t + w 2 t ] = E(w 2 t ) = σ 2 w. (because E(x t k w t ) = 0 for k > 0.) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 23 / 61

The Yule-Walker equations Recalling that γ( s) = γ(s), we have that γ(0) = σ 2 AR = a 1γ(1) + a 2 γ(2) +... + a p γ(p) + σ 2 w, and dividing by γ(0) on both sides, we obtain or 1 = a 1 ρ(1) + a 2 ρ(2) +... + a p ρ(p) + σ2 w σ 2 AR σ 2 w = σ 2 AR (1 a 1ρ(1) a 2 ρ(2)... a p ρ(p). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 24 / 61

Why are Yule-Walker equations so important? We can use equations and ρ(k) = a 1 ρ(k 1) + a 2 ρ(k 2) +... + a p ρ(k p), k > 0. σ 2 w = σ 2 AR (1 a 1ρ(1) a 2 ρ(2)... a p ρ(p). to estimate, say, a 1, a 2, σ 2 w from the autocorrelations if we knew that the order of the model is p = 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 25 / 61

AR(2) Process This process is defined by It can be shown that x t = a 1 x t 1 + a 2 x t 2 + w t. a 1 = a 2 = ρ(1)[1 ρ(2)] 1 ρ(1) 2 ρ(2) ρ(1)2 1 ρ(1) 2 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 26 / 61

Stationarity conditions for an AR(2) process We recently discovered that the equation for the variance of an AR(p) process was σ 2 AR = For an AR(2) process this reduces to σ 2 w 1 ρ(1)a 1 ρ(2)a 2... ρ(p)a p. σ 2 AR = σ 2 w 1 ρ(1)a 1 ρ(2)a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 27 / 61

Stationarity conditions for an AR(2) process Substituting our expressions for a 1 and a 2 can be shown to yield (after a little bit of algebra) σ 2 AR = (1 a 2 )σ 2 w (1 + a 2 )(1 a 1 a 2 )(1 + a 1 a 2 ) Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 28 / 61

Stationarity conditions for an AR(2) process Using the facts that autocorrelations must be less than 1, and each factor in the denominator and numerator must be positive, we can derive the conditions 1 < a 2 < 1 a 2 + a 1 < 1 a 2 a 1 < 1 The inequalities define the stationarity region for AR(2). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 29 / 61

Scatterplot to Illustrate the AR(1) Model The model AR(1) is about the correlation between an error and the previous error. First consider white noise. In this case there should be no correlation between the errors and the shifted errors. On the other hand, consider AR(1) errors with a 1 = 0.7. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 30 / 61

R Code n<-1000; error<-rep(0,n); a1<-0.7; # simulating white noise; set.seed(9999); noise<-rnorm(n,0,2); # simulating AR(1); error = filter(noise,filter=(0.7),method="recursive",init=0); plot(error[-n],error[-1],xlab="errors", ylab="shifted errors"); title("ar(1) errors, a=0.7" ); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 31 / 61

Scatterplot AR(1) errors, a=0.7 shifted errors 5 0 5 10 5 0 5 10 errors Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 32 / 61

Autocorrelation Function acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 33 / 61

Autocorrelation Function Series error ACF 0.0 0.4 0.8 0 5 10 15 20 25 30 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 34 / 61

Based on the formulas already derived and the choice used in the code (a 1 = 0.7): ρ(0) = 1 by definition ρ(1) = a 1 1 = 0.7 ρ(2) = a 2 1 = 0.49 ρ(3) = a 3 1 = 0.343 ρ(4) = a 4 1 = 0.2401 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 35 / 61

R Code acf(error,plot=false)[0:4]; ## ## Autocorrelations of series 'error', by lag ## ## 0 1 2 3 4 ## 1.000 0.690 0.461 0.319 0.216 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 36 / 61

Examples of Stable and Unstable AR(1) Models Consider the following three cases of AR(1) data: 1. a 1 = 0.9 2. a 1 = 0.5 3. a 1 = 1.01 Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 37 / 61

Case 1 n<-100; error<-rep(0,n); a1<- -0.9; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = -0.9, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 38 / 61

Case 1. Time Series a = 0.9, n =100 error 10 0 5 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 39 / 61

Case 1. Correlogram Series error ACF 0.5 0.5 0 5 10 15 20 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 40 / 61

Case 2 n<-100; error<-rep(0,n); a1<- 0.5; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = 0.5, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 41 / 61

Case 2. Time Series a = 0.5, n =100 error 4 0 4 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 42 / 61

Case 2. Correlogram Series error ACF 0.2 0.4 1.0 0 5 10 15 20 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 43 / 61

Case 3 n<-100; error<-rep(0,n); a1<- 1.1; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=(a1),method="recursive", init=0); plot.ts(error,main="a = 1.1, n =100"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 44 / 61

Case 3. Time Series a = 1.1, n =100 error 0 60000 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 45 / 61

Case 3. Correlogram Series error ACF 0.2 0.4 1.0 0 5 10 15 20 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 46 / 61

The AR(2) Model Autocorrelation and Autocovariance The model is: x t = a 1 x t 1 + a 2 x t 2 + w t, and the autocovariances can be characterized with recursive relationships. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 47 / 61

γ(1) and ρ(1) γ(1) = E(x j x j 1 ) = E([a 1 x j 1 + a 2 x j 2 + w j ]x j 1 ) = a 1 γ(0) + a 2 γ(1), so it is easy to see that ρ(1) = a 1 1 a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 48 / 61

ρ(2) and ρ(3) Doing something similar, and ρ(2) = a 2 1 (1 a 2 ) + a 2 ρ(3) = a3 1 + a 1a 2 (1 a 2 ) + a 1a 2. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 49 / 61

σ 2 AR = γ(0) It can be shown that, for the AR(2) model, σ 2 AR = σ 2 W 1 a 1 ρ(1) a 2 ρ(2). Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 50 / 61

Simulating Data for AR(2) Models n<-100; error<-rep(0,n); a1<- 0.8; a2<- -0.7; set.seed(9999); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); acf(error); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 51 / 61

Correlogram Series error ACF 0.5 0.5 0 5 10 15 20 Lag Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 52 / 61

Autocorrelations Based on the formulas already derived and the choices used in the code (a 1 = 0.8, a 2 = 0.7): ρ(0) = 1, ρ(1) = a 1 1 a 2 = 0.8 1.7 0.4706, ρ(2) = a 1 ρ(1) + a 2 ρ(0) 0.8(0.4706) 0.7(1) 0.3235, ρ(3) = a 1 ρ(2) + a 2 ρ(1) 0.8( 0.3235) 0.7(0.4706) 0.5882, etc. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 53 / 61

Stable and Unstable AR(2) Models Recall the AR(2) process is stationary when: i) a 1 + a 2 < 1, ii) a 2 a 1 < 1, and iii) a 2 < 1. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 54 / 61

Examples of Stable and Unstable AR(2) Models Stable: a 1 = 1.60 and a 2 = 0.63. Stable: a 1 = 0.30 and a 2 = 0.40. Stable: a 1 = 0.30 and a 2 = 0.40. Unstable violates (i): a 1 = 0.500 and a 2 = 0.505. Unstable violates (ii): a 1 = 0.505 and a 2 = 0.500. Unstable violates (iii): a 1 = 0 and a 2 = 1.05. Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 55 / 61

Unstable case a 1 = 0.500 and a 2 = 0.505. n<-100; error<-rep(0,n); a1<- 0.500; a2<- 0.505; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = 0.5 and a2=-0.505, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 56 / 61

Unstable case a 1 = 0.500 and a 2 = 0.505. a1 = 0.5 and a2=0.505, n =100 error 10 4 0 4 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 57 / 61

Unstable case a 1 = 0.505 and a 2 = 0.500. n<-100; error<-rep(0,n); a1<- - 0.505; a2<- 0.500; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = - 0.505 and a2=0.500, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 58 / 61

Unstable case a 1 = 0.505 and a 2 = 0.500. a1 = 0.505 and a2=0.500, n =100 error 30 0 20 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 59 / 61

Unstable case a 1 = 0 and a 2 = 1.05. n<-100; error<-rep(0,n); a1<- 0; a2<- -1.05; set.seed(9); noise<-rnorm(n,0,2); error <- filter(noise,filter=c(a1,a2),method="recursive"); plot.ts(error,main="a1 = 0 and a2= -1.05, n =100"); Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 60 / 61

Unstable case a 1 = 0 and a 2 = 1.05. a1 = 0 and a2= 1.05, n =100 error 50 0 50 0 20 40 60 80 100 Time Al Nosedal University of Toronto The Autocorrelation Function and AR(1), AR(2) Models January 21, 2016 61 / 61