Markov Chains (Part 2)

Transcription

1 Markov Chains (Part 2) More Examples and Chapman-Kolmogorov Equations Markov Chains - 1

2 A Stock Price Stochastic Process Consider a stock whose price either goes up or down every day. Let X t be a random variable that is: 0 if the stock price goes up on day t, and 1 if the stock price goes down on day t. The probability that the stock price goes up tomorrow, given it goes up today, is 0.7. If the stock goes down today, the probability that it goes up tomorrow is 0.5. Does the stochastic process X t possess the Markovian property? What is the one-step transition probability matrix? Markov Chains - 2

3 A Stock Price Stochastic Process Consider a stock whose price either goes up or down every day. Let X t be a random variable that is: 0 if the stock price goes up on day t, and 1 if the stock price goes down on day t. The probability that the stock price goes up tomorrow, given it goes up today, is 0.7. If the stock goes down today, the probability that it goes up tomorrow is 0.5. Does the stochastic process X t possess the Markovian property? What is the one-step transition probability matrix? Stock behavior today Probability of stock going up tomorrow Probability of stock going down tomorrow Up Down Markov Chains - 3

4 A Stock Price Stochastic Process Now, suppose the probability of whether the stock goes up or down tomorrow depends on the stock s behavior today and yesterday Intuitively, can we define a stochastic process X t that possesses the Markovian property? Stock behavior yesterday Stock behavior today Probability of stock going up tomorrow Up Up 0.9 Down Up 0.6 Up Down 0.5 Down Down 0.3 Markov Chains - 4

5 A Stock Price Stochastic Process We can expand the state space to include a little bit of history, and create a Markov chain. Let X t be a random variable that has four states: 0, 0 if the stock price went up yesterday on day t-1 and up today on day t, 1, 0 if the stock price went down yesterday on day t-1 and up today on day t, 0, 1 if the stock price went up yesterday on day t-1 and down today on day t, 1, 1 if the stock price went down yesterday on day t-1 and down today on day t Intuitively, now the stochastic process X t satisfies the Markovian property Markov Chains - 5

6 A Stock Price Stochastic Process Now, the one-step transition probability matrix is 4x4 State Transition (t-1,t) to (t,t+1) 0,0 (up, up) 1,0 (down, up) 0,1 (up, down) 1,1 (down, down) 0,0 (up, up) ,0 (down, up) ,1 (up, down) ,1 (down, down) Markov Chains - 6

7 Multi-step Transition Probabilities So far, we have only focused on one-step transition probabilities p ij But these don t directly provide answers to some interesting questions For example, if it is sunny today, what is the probability that it will be sunny day after tomorrow? If the stock went down today, what is the probability that it will go down three days later? These are called multi-step (or n-step) transition probabilities. In particular, we want to find P(X t+n =j X t = i), which is denoted by p ij (n) The Chapman-Kolmogorov (C-K) equation is a formula to calculate n-step transition probabilities.

8 n-step Transition Probabilities If the one-step transition probabilities are stationary, then the n-step transition probabilities are written: P(X t+n =j X t =i) = P(X n =j X 0 =i) for all t = p ij (n) Interpretation: X t j j i i n t t+1 t+2 t+n Markov Chains - 8

9 Inventory Example n-step Transition Probabilities p 12 (3) = conditional probability that starting with one camera, there will be two cameras after three weeks Four ways that could happen: X t t Markov Chains - 9

10 Two-step Transition Probabilities for the Weather Example Intuition: to go from state 0 to 0 in two steps we can either Go from 0 to 0 in one step and then go from 0 to 0 in one step OR Go from 0 to 1 in one step and then go from 1 to 0 in one step Therefore, p 00 = P(X 2 = 0 X 0 =0) = p 00 p 00 +p 01 p 10 In short, p 1 = p0k p 00 k0 k = 0 You just wrote down your first Chapman-Kolmogorov equation using intuition Now use the above intuition to write down the other 2-step transition probabilities p 01, p 10, p 11 These four two-step transition probabilities can be arranged in a matrix P called the two-step transition matrix Markov Chains - 10

11 Two-step Transition Probabilities for the Weather Example P ( " 2 ) = p 00 # ( 2 ) ( 2 ) p 10 ( 2 ) p 01 ( 2 ) p 11 % " ' = p p + p p p p + p p % ' & # p 10 p 00 + p 11 p 10 p 10 p 01 + p 11 p 11 & Interpretation : p 01 is the probability that the weather day after tomorrow will be rainy if the weather today is sunny. An interesting observation: the two-step transition matrix is the square of the one-step transition matrix That is, P =P 2 Why? Recall matrix product to write down P 2 and confirm that it equals P above. Markov Chains - 11

12 Two-step Transition Probabilities for General Markov Chains For a general Markov chain with states 0,1,,M, to make a two-step transition from i to j, we go to some state k in one step from i and then go from k to j in one step. Therefore, the two-step transition probability matrix is, P =P 2 # # P = # # # " p 00 p 10 p 01 p 11 p M 0 p M1... p 0M... p 1M... p MM & & & & & % M k=0 with p = p p ij ik kj Markov Chains - 12

13 n-step Transition Probabilities for General Markov Chains For a general Markov chain with states 0,1,,M, the n-step transition from i to j means the process goes from i to j in n time steps Let m be a non-negative integer not bigger than n. The Chapman- Kolmogorov equation is: M p (n) = " p (m) p (nm) ij ik kj k=0 Interpretation: if the process goes from state i to state j in n steps then it must go from state i to some state k in m (less than n) steps and then go from k to j in the remaining n-m steps. In matrix notation, P (n) =P (m) P (n-m). This implies that the n-step transition matrix is the n th power of the one-step transition matrix (Why? - substitute m=1 and see what happens) Markov Chains - 13

14 Chapman-Kolmogorov Equations p (n) ij = k=0 p (m) (nm) ik p kj Consider the case when m = 1: X t M (n) p = ij M " for all i, j, n and 0 m n M k = 0 (n "1) (n "1) pik p = P # P kj k j i p ik (n "1) p kj 0 1 n t Markov Chains - 14

15 Chapman-Kolmogorov Equations The p ij (n) are the elements of the n-step transition matrix, P (n) Note, though, that P (n) = P P (n "1) = P P P(n " 2) = P P P"P = P n Markov Chains - 15

16 How to use C-K Equations To answer the following question: what is the probability that starting in state i the Markov chain will be in state j after n steps? First write down the one-step transition probability matrix. Then use your calculator to calculate the n th power of this onestep transition probability matrix Write down the ij th entry of this n th power matrix. Markov Chains - 16

17 Weather Example n-step Transitions Two-step transition probability matrix: P = = Sunny Sunny & 0.5 Rainy % 0.2 Sunny Sunny & 0.35 Rainy % 0.26 Rainy # 0.8 " Rainy 0.65# 0.74 " Markov Chains - 17

18 Inventory Example n-step Transitions Two-step transition probability matrix: P = = " % 2 ' ' ' # ' & & % # " Note: even though p 12 =0, p 12 >0 Markov Chains - 18

19 Inventory Example n-step Transitions p 13 = probability that the inventory goes from 1 camera to 3 cameras in two weeks = (note: even though p 13 = 0) Question: Assuming the store starts with 3 cameras, find the probability there will be 0 cameras in 2 weeks p 30 = Markov Chains - 19

20 (Unconditional) Probability in state j at time n The transition probability p ij (n) is a conditional probability, P(X n =j X 0 =i) How do we un-condition the probabilities? That is, how do we find the (unconditional) probability of being in state j at time n, P(X n =j)? The probabilities P(X 0 =i) define the initial state distribution M i=0 M i=0 P(X n = j) = P(X n = j X 0 = i)p(x 0 = i) = p (n) P(X = i) ij 0 Markov Chains - 20

21 Inventory Example Unconditional Probabilities If initial conditions were unknown, we might assume it s equally likely to be in any initial state: P(X 0 =0) = ¼ = P(X 0 =1) = P(X 0 =2) = P(X 0 =3) Then, what is the probability that we order (any) camera in two weeks? P(order in 2 weeks) = P(in state 0 at time 2) = P(X 0 =0)p 00 +P(X 0 =1)p 10 +P(X 0 =2)p 20 +P(X 0 =3)p 30 = ¼(0.249) + ¼(0.283) + ¼(0.351) + ¼(0.249) = Markov Chains - 21

22 Steady-State Probabilities As n gets large, what happens? What is the probability of being in any state? (e.g., in the inventory example, what happens as more and more weeks go by?) Consider the 8-step transition probability for the inventory example. P (8) = P 8 = & % # " Markov Chains - 22

23 Steady-State Probabilities In the long-run (e.g., after 8 or more weeks), the probability of being in state j is These probabilities are called the steady state probabilities lim p = n#" Another interpretation is that π j is the fraction of time the process is in state j (in the long-run) This limit exists for any irreducible ergodic Markov chain (Next, we will define these terms, then return to steady-state probabilities) ( n) ij j Markov Chains - 23