Quasi-Convex Stochastic Dynamic Programming

Transcription

1 Quasi-Convex Stochastic Dynamic Programming John R. Birge University of Chicago Booth School of Business JRBirge SIAM FM12, MSP, 10 July

2 General Theme Many dynamic optimization problems dealing with commodity storage management have a value function with quasi-convex (or quasi- concave) properties Taking advantage of the structure u can help overcome curse of dimensionality An approximation algorithm can repeatedly identify sub-level sets to obtain convergence JRBirge SIAM FM12, MSP, 10 July

3 Outline Motivation Basic properties Level-set approximation algorithm Computational examples Conclusions and next steps JRBirge SIAM FM12, MSP, 10 July

4 Motivation Commodity storage characterized by inventory (x) and price (p): px For a one-stage objective x 0 andp 0, V(p,x)=px is quasi-concave: JRBirge SIAM FM12, MSP, 10 July

5 Challenges Quasi-convexity is not generally additive Many DP objectives are additive but others are not, e.g., risk-sensitive objective: Log[ E(exp(θ v t (x t,p t )) ] When does the property persist in the value function across time? How to take advantage of the property when present? JRBirge SIAM FM12, MSP, 10 July

6 Simple Example: Commodity Management Objective (with action u, capacity C, discount factor δ): V t (x t,p t )= max -xt u C-x p + E[V +u,p t t u δ t+1 (x t t+1 )] Quasi-concave at end of horizon (T) Induction possible? Assume: (p t+1 /p t ) ~ F (fixed stationary) ti JRBirge SIAM FM12, MSP, 10 July

7 Quasi-Concavity Result for V t Basic induction: V t is then quasi-concave if V t (x, p) min{v t (x 1,p 1 ),V t (x 2,p 2 )}, (1) for any such combination of points, (x 1, p 1 ) and (x 2, p 2 ). Without loss of generality, suppose x 1 <x 2 and let and Suppose and x = x 1 + = x 2, (2) p = Up 1 = Dp 2. (3) V t (x 1,p 1 )= u 1 p 1 + E p1 V t+1 (x 1 + u 1,p 0 ), (4) V t (x 2,p 2 )= u 2 p 2 + E p2 V t+1 (x 2 + u 2,p 0 ). (5) JRBirge SIAM FM12, MSP, 10 July

8 Quasi-concavity of Value Function Suppose policy σ 1 starting from (x, p) at stage that sets u 1 = u 1 + at stage t and then follows policy π 1 (ω) (optimal along all p t (ω) =w t (ω)p 1 at time t; σ 2 starting from (x, p) withu 2 = u 2 at t and then π 2 (ω) -optimalfor p t (ω) = v t (ω)p 2 ; σ be the policy that randomly chooses σ 1 or σ 2 from state (x, p) V t (x, p) V σ = 1 ( (u p 0 t,σ (x, p) 2 (u 1 + )Up 1 + UE p1 V t+1 (x 1 + u 1, p ) (u 2 )Dp 2 + DE p2 V t+1 (x 2 + u 2,p 0 )) = ( Up( p Dp ) + (UV t(x 1, p 1 ) + DV t t( (x 2, p 2 )) (U + D)min(V t(x 1,p 1 ),V t (x 2,p 2 )) p = p 1 + p 2 + 2p 1p 2 min(v t (x 1,p 1 ),V t (x 2,p 2 )), 4p 1 p 2 min{v t (x 1,p 1 ),V t (x 2,p 2 )} using p = Up 1 = Dp 2, U = p 1+p 2 2p 1, D = p 1+p 2 2p 2,andthat2p 1 p 2 p p 2 2. JRBirge SIAM FM12, MSP, 10 July

9 Analysis and Extensions of Result Extends to proportional transaction costs directly Key assumption: Price process increments are independent of the price level (as in a log-normal/gbm model) Extends to other price processes as well: Distributions such that V t,π1 (x 1,Up 1 )=U 0 V t (x 1,p 1 ) and V t = t,π1 2(x 2, Dp 2 ) D V t (x 1, s 1 ) where U + D 2. JRBirge SIAM FM12, MSP, 10 July

10 Extended Processes Ornstein-Uhlenbeck: d log p t = θ(μ log p t )dt + σdw t, (1) where θ, μ, andσ are scalars; W t is a standard Brownian motion. The price p t given initial value p is then p t =exp(logpe θt + μ(1 e θt )+ Z t 0 σe θ(q t) dw q ). (2) Let p = Up 1 = Dp 2, p i t be spot price at t given spot price p i at 0, i =1, 2, then p 1 U e θt = U 0 2 D e θt = D 0 t p t p t, p t p t p t, (3) the condition, U 0 + D 0 2holdsforsufficiently small θ and sufficiently large time intervals. JRBirge SIAM FM12, MSP, 10 July

11 Algorithm Motivation Assume: Quasi-concave (-convex) structure Basic idea: approximate the level sets of the value function at each stage Use the level-set approximation in stage t+1 to construct an approximation at t. Use outer linearization i i to construct each successive approximation JRBirge SIAM FM12, MSP, 10 July

12 Algorithms Quasi-concave Dynamic Program Method(QDPM) 1. Set U t (x, p) for all t. LetU T (x, p) =V T (x, p). Set t = T REPEAT: (a) Randomly select (x t,p t ); (b) Find Ût(x U = t t, p t ) max u A(xt,p t ) v t (u, x t, p t )+ P z Z t+1 (u,x t,s t ) π t+1(z, a, x t, p t )U t+1 (z); (c) Perform Bound Update; (d) Choose t. 3. UNTIL U 1 (x 1,p 1 )isstable. Bound Update Construct upper bound on the value function using outer linearization: U t (x, p) min(u t (x, p), Ût(x t,p t )) (1) ˆ for all (x, p) such that U t (x t,p t ) T (x x t,p p t )) 0. JRBirge SIAM FM12, MSP, 10 July

13 Bound Update Û t (x t,p t ) Ût(x t,p t )(x x t,p p t )) 0 U t (x, p) Û(x t,p t ) JRBirge SIAM FM12, MSP, 10 July

14 Examples Setup: Lognormal prices (log-mean: 1, logst.dev.: 0.4) Transaction cost: 5% proportional Note: bound update must achieve sufficient accuracy to also maintain quasi-concavity Use of 1000, 500, and 250 random samples per period JRBirge SIAM FM12, MSP, 10 July

15 Approximations at T-1 JRBirge SIAM FM12, MSP, 10 July

16 Upper Bounds at T-5 JRBirge SIAM FM12, MSP, 10 July

17 Observations from Example For the T-1-stage results: Close to original for N T-1 =1000 More differences for N T-1 =500 and N T-1 =250 increasingly less consistent in shape with fewer samples. For the T-5-stage results: Similar il pattern more error less consistent level sets with N T-5 =250 JRBirge SIAM FM12, MSP, 10 July \

18 Conclusions Quasi-convexity may persist in some in some commodity inventory control problems Possible resolutions with level set approximations Limitations: Accuracy needed in the approximation to preserve quasi-concavity Unclear results for higher-dimensions JRBirge SIAM FM12, MSP, 10 July

19 Future Work Find other structures (such as risk-sensitive objective) with quasi-convex structure Identify methods to maintain the property in the approximation Extend results for multiple dimensions JRBirge SIAM FM12, MSP, 10 July

20 Thank you! JRBirge SIAM FM12, MSP, 10 July