Lecture 2 Dynamic Equilibrium Models: Three and More (Finite) Periods. Introduction In ECON 50, we discussed the structure of two-period dynamic general equilibrium models, some solution methods, and their application to issues such as optimal consumption and savings and asset pricing. In this lecture, we increase the time horizon to three and more (but finite) periods. We will see that while increase in the time horizon does not change the structure of dynamic general equilibrium models, it increases the number of choice variables, which makes the Langarangian method of the previous lecture quite cumbersome (in many cases almost impossible). In this lecture, we will discuss an alternative method the method of dynamic programming to solve DGE models. After learning this method, we will see some of its application. Let us begin with an example. We will solve this example using both Langrangian and dynamic programming methods and then compare the structure of these two solution methods. Example Suppose that the economy lasts for three periods and the representative consumer wants to choose an optimal consumption plan subject to his budget constraint. The optimization problem is as follows: subject to max ln c + β ln c 2 + β 2 ln c 3 (.) c, c 2, c 3, a, a 2 c + a = a 0 ( + r) (.2) c 2 + a 2 = a ( + r) (.3) c 3 = a 2 ( + r) (.4) given a 0 and a 3 = 0. (.2)-(.4) are period budget constraints. a i is the asset level in period i and r is the real rate of interest. Let λ, λ 2, and λ 3 be the Langrangian multipliers associated with (.2), (.3), and (.4) respectively. The first order conditions are c : c 2 : c = λ (.5) β c 2 = λ 2 (.6)
c 3 : β 2 c 3 = λ 3 (.7) a : λ = ( + r)λ 2 (.8) Combining (.5), (.6), and (.8), we have Similarly, from (.6), (.7), and (.9), we have a 2 : λ 2 = ( + r)λ 3. (.9) c = β( + r) c 2. (.0) c 2 = β( + r) c 3 (.) (.0) and (.) are Euler equations linking consumptions in adjacent periods. Using (.0) and (.) together with three budget constraints, we can solve for five unknowns c, c 2, c 3, a, & a 2. From (.2), (.3), and (.0), we get After some manipulation, we have ( + r)a 0 a = β( + r) ( + r)a a 2. (.3) a 2 = ( + r)( + β)a β( + r) 2 a 0. (.4) From (.3), (.4), and (.), we have another expression for a 2 : a 2 = From (.4) and (.5) we get solution for a as a = β( + r) + β a. (.5) β( + β)( + r) ( + β( + β)) a 0. (.6) Once we have solved for a, we can solve for other choice variables using (.4) and three budget constraints. Let us look at the structure of the solution method. In Langrangian method, we choose all the choice variables c, c 2, c 3, a & a 2 simultaneously. This leads to five first order conditions (.5 to.9). In addition, we have three budget constraints. Using these eight equations we solve for eight unknowns c, c 2, c 3, a, a 2, λ, λ 2 & λ 3. In general, when we use Langrangian method, we get a system of simultaneous equations and the number of equations in the system is equal to the number of choice variables plus the number of constraints. From the structure of solution method, it is immediately 2
clear that as the number of choice variables increases (e.g. more time periods), the system of simultaneous equations becomes bigger and more complex making it tougher (if not impossible) to solve it. For instance, if we add one more time period in the previous example, we will need to solve system of simultaneous equations consisting of equations. Obviously then in order to solve a multiple period DGE model, we need another solution method which will reduce the dimension of system of simultaneous equation we need to solve. This leads to the method of dynamic programming. 2. Dynamic Programming: Finite Periods The method of dynamic programming provides a way of breaking down the big multiperiod optimization problem into a series of smaller optimization problems which are solved one at a time. Essentially, the optimization problem is broken down in a sequence of twoperiods optimization problems. The choice variables are chosen sequentially (recursively) rather than simultaneously. Under certain conditions (discussed below), the solutions derived are identical to Langrangian method. Let us solve the previous optimization problem using this new method. Example 2 We begin with the last period. We choose c 3 in order to maximize last period utility subject to the last period budget constraint. Let W 3 (a 2 ) be a function defined as W 3 (a 2 ) max c 3 ln c 3 + λ 3 [( + r)a 2 c 3 ] (2.) W 3 (a 2 ) is simply the optimum value of utility in period 3. The first order condition is From the budget constraint, we have c 3 : c 3 = λ 3. (2.2) c 3 = ( + r)a 2. (2.3) By envelope condition, we have dw 3 (a 2 ) da 2 = λ 3 ( + r) = a 2. (2.4) After solving for optimal consumption in the third period, we step back to the second period. We choose c 2 and a 2 to maximize the following function W 2 (a ) max c 2, a 2 ln c 2 + βw 3 (a 2 ) + λ 2 [( + r)a c 2 a 2 ] (2.5) W 2 (a ) is the optimum value of the objective function. Here we are choosing c 2 & a 2 in order to maximize the utility of the second period plus the discounted optimum value 3
of utility in period 3 subject to the second period budget constraint. Notice that (2.5) involves two periods. The first order conditions are c 2 : c 2 = λ 2. (2.6) a 2 : β dw 3(a 2 ) da 2 = λ 2. (2.7) Using (2.4), (2.6), (2.7), and the second period budget constraint, we have From (2.8), we have β a 2 = ( + r)a a 2. (2.8) By envelope condition, we have a 2 = c 2 = β( + r) + β a (2.9) ( + r) + β a. (2.0) dw 2 (a ) da = λ 2 ( + r) = + β a. (2.) Now we step back to period and solve for optimal c and a. Define W (a 0 ) as W (a 0 ) max c, a ln c + βw 2 (a ) + λ [( + r)a 0 c a ] (2.2) (2.2) has interpretation similar to (2.5). Notice that the objective function in (2.2) involves W 2 (a ) but not W 3 (a 2 ), since W 2 (a ) already incorporates W 3 (a 2 ) (see 2.5). The first order conditions are c : c = λ. (2.3) a : β dw 2(a ) da = λ. (2.4) Using (2.), (2.3), (2.4), and the first period budget constraint, we have From (2.5), we get β( + β) a = ( + r)a 0 a. (2.5) a = β( + β)( + r) + β( + β) a 0. (2.6) 4
As you can see, the solution for a is exactly the same as we found earlier (.6). Using (2.9) and budget constraints, we can find out the value of other choice variables. Let us recapitulate the steps involved in the second method. Basically, we reformulated the optimization problem (.) by breaking it down in three smaller optimization problems as follows: where where W (a 0 ) max c, a ln c + βw 2 (a ) + λ [( + r)a 0 c a ] W 2 (a ) max c 2, a 2 ln c 2 + βw 3 (a 2 ) + λ 2 [( + r)a c 2 a 2 ] W 3 (a 2 ) max c 3 ln c 3 + λ 3 [( + r)a 2 c 3 ]. Then we solved this problem backward. We started with the last period and solved for c 3 & dw 3(a 2 ) da 2. Then we stepped back and solved for c 2, a 2 & dw 2(a ) da. In the final step, we solved for c & a. As you can see, at each step we are choosing only current period choice variables subject to the current period budget constraint, which reduces the dimension of the system of simultaneous equations that needs to be solved at any step. In the Langrangian method, we were choosing choice variables for all the periods at the same time. It is this feature of the method of dynamic programming, which makes it quite suitable for solving DGE models. Numerically, it is much easier to invert 0 by 0 matrix 0 times rather than invert 00 by 00 matrix one time. Let us now discuss some of the elements of the method of dynamic programming. Functions such as W 3 (a 2 ), W 2 (a ) & W (a 0 ) are called value functions. They are nothing but indirect utility functions. The value function W t (a t ) is a function of a t, which the utility maximizer at time t takes as given. Such variables are known as state variables (a 0 at time, a at time 2, and so on). These variables describe the state of the system at a particular time and provide inter-temporal linkages. The variables which utility maximizer chooses at time t are called control or choice or decision variables (c, a at time, c 2, a 2 at time 2, and c 3 at time 3). Notice that the control variables at time t can be state variables at time t +. We distinguish between control and state variables on the basis of what variables the decision maker takes as given and what he can choose at the time of making decision. At each stage of the solution process, the first order conditions together with the budget constraint and envelope condition allow us to solve for choice variables as a function of state variables. Such functions are called policy functions (2.3, 2.9, 2.0, 2.6 etc.). Finally, we have budget constraints which express next period value of state variable as a function of current state and choice variables (e.g a 2 = ( + r)a c ). Such functions are known as laws of motion, which describe how state variables evolve over time. Let us look at the method of dynamic programming in a somewhat more formal way. Let F t (X t, U t ) be the period objective function where X t is a vector of state variables and 5
U t is the vector of control variables at time t. Suppose that the optimization problem given to us is as follows: subject to the laws of motion W = max U t T β t F t (X t, U t ) (2.7) t= x t+ = m tx (X t, U t ) x t X t & t (2.8) given X 0 & X T. We can recast this optimization problem as subject to the laws of motion W t (X t ) = max U t F t (X t, U t ) + βw t+ (X t+ ) (2.9) x t+ = m tx (X t, U t ) x t X t (2.20) Equations such as (2.9) are known as Bellman Equations. Since W t (X t ) is a function of another value function W t+ (X t+ ), it is also known as functional equation. Bellman equation is said to satisfy the principle of optimality. The idea behind the principle of optimality is as follows. Suppose we solve the optimality problem for time t onwards given the state variables at time t and derive optimal policy functions (U t = G t (X t ), U t+ = G t+ (X t+ ),... U t+i = G t+i (X t+i ),..., U T = G T (X T )). These policy functions provide optimal trajectories of choice variables as functions of state variables. In other words, we derive the optimal plan of actions over time as a function of state variables. Now suppose we solve for optimal plan of action from time t + i, with i onwards and derive optimal policy functions (U t+i = G t+i (X t+i),..., U T = G T (X T )). Then the principle of optimality is satisfied iff G t+i (X t+i ) = G t+i (X t+i), i. The implication of the principle of optimality is that the optimal plan of action (solution) is time consistent. We compute the optimal path from the beginning of the plan period and start moving along it. After a while, we stop and recalculate the optimal path from the current period onwards. The principle of optimality tells us that the solution of the new problem will be the remainder of the original optimal plan. Hence, the decision maker will not be tempted to change his mind or deviate from the original plan as time passes. In the parlance of game theory, the optimal plan is sub-game perfect. So far we have not asked the question, whether all optimization problems can be recast as dynamic programming problems? Answer is NO! Only under certain conditions, we can solve the optimization problem through the method of dynamic programming. The main conditions are as follows: () The objective function must be time-additive and separable, and (2) the period objective function F t (X t, U t ) and laws of motion x t+ = m t (X t, U t ) must be the functions of current state or control variables or both. They cannot be the functions of past or future state or control variables. 6
Let us recapitulate the steps involved in solving dynamic programming problem. () Start at the last period and solve for control variables as a function of state variable. Derive envelope condition dw T (X T ) dx T. (2) Step one period back and solve subject to the laws of motion W T (X T ) = max U T F T (X T, U T ) + βw T (X T ) (2.2) x T = m T x (X T, U T ) x T X T (2.22) Derive policy functions and solve for envelope condition. condition satisfies One can show that envelope dw t (X t ) dx t = df t(x t, U t ) dx t + β dm tx(x t, U t ) dx t dw t+ (X t+ ) dx t+ x t X T & t. (2.23) The envelope condition is also known as Benveniste-Scheinkman condition. (3) Keep on repeating step 2 till t =. Example 3 Let us solve the problem of optimal consumption and saving using the method of dynamic programming. subject to max c t,k t 3 β t ln c t (2.24) t= given k 0 > 0 and k 3 = 0. We can recast the above problem as k t = k α t c t (2.25) W t (k t ) = max c t,k t ln c t + βw t+ (k t ) + λ t [k α t c t k t ], t =..3. (2.26) Here k t is the state variable, which decision maker takes as given at time t. We begin with the last period t = 3. We have The solution is W 3 (k 2 ) = max c 3 ln c 3 + λ 3 [k α 2 c 3 ]. (2.27) 7
c 3 = k α 2 (2.28) which is our policy function (choice variable as a function of state variable). From envelope condition, we get Now we go back to period 2. The period 2 problem is First order conditions are dw 3 (k 2 ) dk 2 = α k 2 (2.29) W 2 (k ) = max c 2,k 2 ln c 2 + βw 3 (k 2 ) + λ 2 [k α c 2 k 2 ]. (2.30) c 2 : c 2 = λ 2 (2.3) k 2 : β dw 3(k 2 ) dk 2 = λ 2. (2.32) From (2.29), (2.3), (2.32), and the second period budget constraint, we have (2.33) implies that k α k 2 = αβ k 2. (2.33) and From envelope condition, we have k 2 = αβ + αβ kα (2.34) c 2 = kα + αβ. (2.35) (2.34) and (2.36) imply that dw 2 (k ) dk = βα2 k α. (2.36) k 2 dw 2 (k ) dk Now we go back to period. The period problem is First order conditions are = α( + αβ) k. (2.37) W (k 0 ) = max c,k ln c + βw 2 (k ) + λ [k α 0 c k ]. (2.38) 8
c : c = λ (2.39) k : β dw 2(k ) dk = λ. (2.40) From (2.37), (2.39), (2.40), and the first period budget constraint, we have From (2.4), we have k α 0 k = αβ( + αβ) k. (2.4) k = αβ( + αβ) + αβ( + αβ) kα 0. (2.42) Once we have solved for k, we can solve for other choice variables. Exercise: Solve the above problem using the Langrangian method. 3. Uncertainty So far we have dealt with certainty case. Now we introduce uncertainty. We will see that the method of dynamic programming can be readily applied to solve optimization problems with uncertainty. Before we extend our analysis to multi-period economies with uncertainty, it is useful to refresh our memories about joint probability distribution. Suppose x and y are two random variables, where x and y can take values {, 2}. Suppose their joint probability distribution is given as follows: x 2 Table Joint Probability Table y Joint Probability Total 0.4 0. 0.5 2 0. 0.4 0.5 Total 0.5 0.5 The marginal distribution of x is given by entries in the last row. For instance, P rob (x = ) = 0.5 = P (x = 2). similarly, the marginal distribution of y is given by the entries in the last column. Using these marginal distributions, one can derive unconditional average and variance. The unconditional average of x is given by 9
P (x = ) + 2 P (x = 2) =.5. One can similarly derive unconditional moments of y. The conditional probability distribution is derived by using joint and marginal probability distributions. The probability that x = given (conditional on) y = is given by Similarly, P (x = y = ) = P (x =, y = ) P (y = ) = 0.4 0.5 = 0.8. P (x = 2 y = ) = P (x = 2, y = ) P (y = ) = 0. 0.5 = 0.2. Notice that the conditional probability distribution adds up to one. Corresponding to the conditional probability distribution, one can calculate conditional average and variance. For instance, the average of x conditional on y = is given by E(x y = ) = P (x = y = ) + 2 P (x = 2 y = ) =.2 which is lower than the unconditional average of x. Exercise: Find out conditional distribution of x when y = 2. Derive the conditional mean and variance. Two random variables x and y are said to be independent if P (x = x i, y = y i ) = P (x = x i )P (y = y i ). In this case, the conditional and unconditional (marginal) expectations coincide. From the probabilities given one can work out covariance or correlation. In the case two random variables are independent, their covariance or correlation is zero. In the previous example, x and y are not independent. Exercise: What is the covariance between x and y in the previous example? Exercise: Construct an example where x and y are independent. Suppose now that x is the income in period and y is the income in period 2. If period income and period 2 income are not independent (in the above example actually they are positively correlated), then knowing about period income helps us in improving our forecast about period 2 income. In what follows, we will make extensive use of conditional expectation and variance. We will denote conditional expectation operator as E t to show that expectation is made conditional on information available at the beginning of period t. 0
Example 4 Let us introduce uncertainty in example 3. Suppose the production function is given by A t k α t, where A t is a non-independent random variable. Suppose that A t can take two values A l, A h in any period t. Also suppose that the value of A t is realized at the beginning of period t, before the decision maker chooses consumption and investment for that period. The optimization problem is subject to max E c t,k t 3 β t ln c t (3.) t= c t + k t = A t k α t (3.2) given k 0 > 0 and k 3 = 0. We can recast this optimization problem as W t (k t, A t ) = max c t,k t ln c t + βe t W t+ (k t, A t+ ) + λ t [ At k α t c t k t ] (3.3) Notice that there are two state variables k t and A t, since the state of technology is revealed to the decision maker before he makes choices for time t. As a convention, if a random variable is i.i.d, it is not treated as a state variable, as knowledge of it is not informative about its future values. The expectation operator E t denotes the fact that the expectation is conditional on information available at the beginning of period t (in this case value of k t and A t ). As usual, we start with the last period. The optimization problem is Since economy lasts for three periods, we have W 3 (k 2, A 3 ) = max c 3 ln c 3 + λ 3 [A 3 k α 2 c 3 ] (3.4) c 3 = A 3 k α 2. (3.5) c 3 can take two values depending on whether A 3 is A l or A h. From envelope condition, we have Now we go back to period 2. The period 2 problem is First order conditions are dw 3 (k 2, A 3 ) dk 2 = α k 2 (3.6) W 2 (k, A 2 ) = max c 2,k 2 ln c 2 + βe 2 W 3 (k 2, A 3 ) + λ 2 [A 2 k α c 2 k 2 ]. (3.7)
c 2 : k 2 : E 2 ( dw3 (k 2, A 3 ) dk 2 c 2 = λ 2 (3.8) ) = λ 2. (3.9) From (3.6), (3.8), (3.9), and the second period budget constraint, we have A 2 k α k 2 The solution for k 2 conditional on A 2 and k is = αβ k 2. (3.0) and k 2 = αβ + αβ A 2k α (3.) c 2 = A 2k α + αβ. (3.2) Since A 2 can take two values A l, A h, there will be two values of k 2 and c 2 for a given k. From envelope condition, we have (3.) and (3.3) imply that dw 2 (k, A 2 ) dk = βα2 A 2 k α. (3.3) k 2 dw 2 (k, A 2 ) dk Now we go back to period. The period problem is First order conditions are = α( + αβ) k. (3.4) W (k 0, A ) = max c,k ln c + βe W 2 (k, A 2 ) + λ [A k α 0 c k ]. (3.5) c : k : E ( dw2 (k, A 2 ) dk c = λ (3.6) ) = λ. (3.7) From (3.4), (3.6), (3.7), and the first period budget constraint, we have From (3.8), we have A k α 0 k = αβ( + αβ) k. (3.8) 2
k = αβ( + αβ) + αβ( + αβ) A k0 α. (3.9) Once we have solved for k, we can solve for other choice variables. Notice again that k can take two values depending on whether A is equal to A l or A h. Suppose that A = A l, then k l = Given k l, k 2 can take two values in the second period αβ( + αβ) + αβ( + αβ) A lk0 α. (3.20) and k 2l = αβ + αβ A lk α l (3.2) k 2h = αβ + αβ A hk α l. (3.22) Similarly, there will be two values of consumption in period 2. Corresponding to each value of investment in period two, there will be two values of consumption in period 3, c 3 (see 3.5). Thus from the perspective of the first period, there will be four values of c 3. Exercise: Solve the above example using the Langrangian method. 4. Asset Pricing In ECON 50, we learned about asset pricing in two-period economies. Here, we extend the analysis to more than two periods (finite). With multi-period economies, the methodology to price assets remains the same as in two period economies. The price of an asset equates the marginal cost (in utility terms) to its expected marginal benefit (in utility terms). Suppose that an asset pays off at time t + i with i and its payoff is y t+i (suppose resale value is 0), which is a random variable. Then its price at time t, q t satisfies q t u (c t ) = β i E t [u (c t+i )y t+i ]. (4.) The left hand side is the marginal cost of buying the asset in utility terms and the right hand is the expected marginal benefit. The asset pays off y t+i in period t + i, which is converted in utility terms by multiplying it with the marginal utility of consumption at t + i. To make it comparable to time t utility, we multiply it by β i. (4.) can be rewritten as [ u q t = β i ] (c t+i )y t+i E t u. (4.2) (c t ) 3
Long and Short Bonds Let q L be the price of a bond today, which pays unit in period 3 (long or twoperiod bond), and q S be the price of one period bond, which pays unit next period. Let u(c) = ln c. Then, [ ] q L = β 2 c E (4.3) c 3 [ ] q S c = βe c 2 (4.4) From (4.3) and (4.4), we can derive rates of interest on long and short bonds. The gross return on long bond satisfies Similarly, the gross return on short bond satisfies ( + r L ) 2 = q L = β 2 E [ c c 3 ]. (4.5) + r S = q S = βe [ c c 2 ]. (4.6) The pattern of returns on long and short bonds are known as term structure. The plot of term structure over maturity is also known as yield curve. The term structure or yield curve embodies the forecasts of future consumption growth. In general, yield curve slopes up reflecting growth. Downward sloping yield curve often forecasts a recession. What is the relationship between the prices of short and long bonds? We turn to covariance decomposition. [ ] q L = β 2 c c 2 E (4.7) c 2 c 3 which implies ( q L = q S E q2 S βc + Cov, βc ) 2 c 2 c 3 (4.8) where q2 S is the second period price of one period bond. If we ignore the covariance term, then in terms of returns (4.8) can be written as ( ) 2 + r L = + r S E + r2 S. (4.9) Taking logarithms, utilizing the fact that ln( + r) r, and ignoring Jensen s inequality we get 4
r L rs + E r2 S. (4.0) 2 (4.0) suggests that the long run bond yield is approximately equal to the arithmetic mean of the current and expected short bond yields. This is called expectation hypothesis of the term structure. Exercise: Find out the prices of short and long term nominal bonds and their relationship. Exercise: What is the price of bond which pays unit of good in period T in all states? Forward Prices Suppose in period, you sign a contract, which requires you to pay f in period 2 in exchange for a payoff of in period 3. How do we value this contract? Notice that the price of contract, which is to be paid in period 2, is agreed in period. Then the expected marginal cost of the contract in period is βe u (c 2 )f. The expected benefit of the contract is β 2 E u (c 3 ). Since the price equates the expected marginal cost with expected marginal benefit of the asset, we have Share f = βe u (c 3 ) E u (c 2 ) = ql q S (4.) Exercise: Show that the expectation hypothesis implies that forward rates are equal to the expected future short rates. Exercise: Find the relationship among forward, long, and short rates of return. Also use covariance decomposition to define risk-premium on forward prices. Suppose that a share pays a stream of dividend d i for i [t, T ]. The resale value of the share at time T is zero. Then the price of the share at time t is given by References: T t q ST = E t β i u (c t+i )d t+i. (4.2) i= Sargent, Thomas J., 987, Dynamic Macroeconomic Theory, Cambridge, Harvard University press. 5
Additional Exercises (.) Time Inconsistency: Suppose the representative agent maximizes following utility subject to ln(c c 2 c 3 ) + β ln(c 2 c 3 ) + β 2 ln c 3 a t = a t c t, t =, 2, 3 given a 0, and a 3 = 0. Compute the consumption plan from the perspective of period. Now maximize the following utility subject to β ln(c 2 c 3 ) + β 2 ln c 3 a t = a t c t, t = 2, 3 and a computed in the first part. Find the optimal c 2 and c 3. Compare your results from the original plan. Has the consumer changed his mind? How and why? Does Bellman equation hold? We will discuss some other examples of time-inconsistency later in the course. (2.) Imagine a three period model with utility function: ln c + βe ln c 2 + β 2 ln c 3. Let β = 0.98. Suppose that c =.02 but c 2 and c 3 can take two values.02 and.08. Also suppose that in the next period the value stays the same with probability 0.8 and changes to other value with probability 0.2. (a.) Write down the joint probability distribution of c 2 and c 3. What is the expected value of c 3 conditional on c 2 =.02? (b.) What is the expected value of c 3 conditional on c =.02? (c.) Find the values of q S, q L, and f. (d.) Find the values of r S and r L. Also find E r2 S, the forecast of next-period s short interest rate (Hint: Find possible values of q2 S first). 6