DASC: A DECOMPOSITION ALGORITHM FOR MULTISTAGE STOCHASTIC PROGRAMS WITH STRONGLY CONVEX COST FUNCTIONS

Transcription

1 DASC: A DECOMPOSITION ALGORITHM FOR MULTISTAGE STOCHASTIC PROGRAMS WITH STRONGLY CONVEX COST FUNCTIONS Vincent Guigues School of Applied Mathematics, FGV Praia de Botafogo, Rio de Janeiro, Brazil vguigues@fgv.br Abstract. We introduce DASC, a decomposition method akin to Stochastic Dual Dynamic Programming SDDP which solves some multistage stochastic optimization problems having strongly convex cost functions. Similarly to SDDP, DASC approximates cost-to-go functions by a maximum of lower bounding functions called cuts. However, contrary to SDDP where cuts are affine functions, the cuts computed with DASC are quadratic functions. We also prove the convergence of DASC. Keywords: Strongly convex value function and Monte-Carlo sampling and Stochastic Programming and SDDP. AMS subject classifications: 90C15, 90C Introduction Stochastic Dual Dynamic Programming SDDP, introduced in [13], is a sampling-based extension of the Nested Decomposition algorithm [1] which builds policies for some multistage stochastic optimization problems. It has been used to solve many real-life problems and several extensions of the method have been considered such as DOASA [15], CUPPS [3], ReSA [11], AND [2], and more recently risk-averse [8], [9], [14], [5], [16], [17], [12] or inexact [7] variants. SDDP builds approximations for the cost-to-go functions which take the form of a maximum of affine functions called cuts. We propose an extension of this algorithm called DASC, which is a Decomposition Algorithm for multistage stochastic programs having Strongly Convex cost functions. Similarly to SDDP, at each iteration the algorithm computes in a forward pass a sequence of trial points which are used in a backward pass to build lower bounding functions called cuts. However, contrary to SDDP where cuts are affine functions, the cuts computed with DASC are quadratic functions and therefore the cost-to-go functions are approximated by a maximum of quadratic functions. The outline of the study is as follows. In Section 2, we give in Proposition 2.3 a simple condition ensuring that the value function of a convex optimization problem is strongly convex. In Section 3, we introduce the class of optimization problems to which DASC applies and the necessary assumptions. DASC algorithm, which is based on Proposition 2.3, is given in Section 4, while convergence of the algorithm is shown in Section Strong convexity of the value function Let be a norm on R m and let f : X R be a function defined on a convex subset X R m. Definition 2.1 Strongly convex functions. f is strongly convex on X R m with constant of strong convexity α > 0 with respect to norm iff for all 0 t 1, x, y X. ftx + 1 ty tfx + 1 tfy αt1 t y x 2, 2 We have the following equivalent characterization of strongly convex functions: Proposition 2.2. Let X R m be a convex set. Function f : X R is strongly convex on X with constant of strong convexity α > 0 with respect to norm iff 2.1 fy fx + s T y x + α 2 y x 2, x, y X, s fx. 1

2 Let X R m and Y R n be two nonempty convex sets. Let A be a p n real matrix, let B be a p m real matrix, let f : Y X R, and let g : Y X R q. For b R p, we define the value function { inf fy, x 2.2 Qx = y Sx := {y Y, Ay + Bx = b, gy, x 0}. DASC algorithm is based on Proposition 2.3 below giving conditions ensuring that Q is strongly convex: Proposition 2.3. Consider value function Q given by 2.2. Assume that i X, Y are nonempty and convex sets such that X domq and Y is closed, ii f, g are lower semicontinuous and the components g i of g are convex functions. If additionally f is strongly convex on Y X with constant of strong convexity α with respect to norm on R m+n, then Q is strongly convex on X with constant of strong convexity α with respect to norm on R m. Proof. Take x 1, x 2 X. Since X domq the sets Sx 1 and Sx 2 are nonempty. Our assumptions imply that there are y 1 Sx 1 and y 2 Sx 2 such that Qx 1 = fy 1, x 1 and Qx 2 = fy 2, x 2. Then for every 0 t 1, by convexity arguments we have that ty ty 2 Stx tx 2 and therefore Qtx tx 2 fty ty 2, tx tx 2 tfy 1, x tfy 2, x αt1 t y 2, x 2 y 1, x 1 2 tqx tqx αt1 t x 2 x 1 2, which completes the proof. 3. Problem formulation and assumptions We consider multistage stochastic optimization problems of the form T inf E 3.3 ξ2,...,ξ x T [ f t x t ξ 1, ξ 2,..., ξ t, x t 1 ξ 1, ξ 2,..., ξ t 1, ξ t ] 1,...,x T t=1 x t ξ 1, ξ 2,..., ξ t X t x t 1 ξ 1, ξ 2,..., ξ t 1, ξ t a.s., x t F t -measurable, t = 1,..., T, where x 0 is given, ξ 1 is deterministic, ξ t T t=2 is a stochastic process, F t is the sigma-algebra F t := σξ j, j t, and X t x t 1, ξ t, t = 1,..., T, can be of two types: S1 X t x t 1, ξ t = {x t R n : x t X t : x t 0, A t x t + B t x t 1 = b t } in this case, for short, we say that X t is of type S1; S2 X t x t 1, ξ t = {x t R n : x t X t, g t x t, x t 1, ξ t 0, A t x t + B t x t 1 = b t }. In this case, for short, we say that X t is of type S2. For both kinds of constraints, ξ t contains in particular the random elements in matrices A t, B t, and vector b t. Note that a mix of these types of constraints is allowed: for instance we can have X 1 of type S1 and X 2 of type S2. We make the following assumption on ξ t : H0 ξ t is interstage independent and for t = 2,..., T, ξ t is a random vector taking values in R K with a discrete distribution and a finite support Θ t = {ξ t1,..., ξ tm } with p ti = Pξ t = ξ ti > 0, i = 1,..., M, while ξ 1 is deterministic. 1 We will denote by A tj, B tj, and b tj the realizations of respectively A t, B t, and b t in ξ tj. For this problem, we can write Dynamic Programming equations: assuming that ξ 1 is deterministic, the first stage problem is { infx1 R 3.4 Q 1 x 0 = n F 1x 1, x 0, ξ 1 := f 1 x 1, x 0, ξ 1 + Q 2 x 1 x 1 X 1 x 0, ξ 1 for x 0 given and for t = 2,..., T, Q t x t 1 = E ξt [Q t x t 1, ξ t ] with { infxt R 3.5 Q t x t 1, ξ t = n F tx t, x t 1, ξ t := f t x t, x t 1, ξ t + Q t+1 x t x t X t x t 1, ξ t, 1 To alleviate notation and without loss of generality, we have assumed that the number M of possible realizations of ξt, the size K of ξ t, and n of x t do not depend on t. 2

3 with the convention that Q T +1 is null. We set X 0 = {x 0 } and make the following assumptions H1 on the problem data: for t = 1,..., T, H1-a for every x t, x t 1 R n the function f t x t, x t 1, is measurable and for every j = 1,..., M, the function f t,, ξ tj is strongly convex on X t X t 1 with constant of strong convexity α tj > 0 with respect to norm 2 ; H1-b X t is nonempty, convex, and compact; H1-c there exists ε t > 0 such that for every j = 1,..., M, for every x t 1 X εt t 1, the set X tx t 1, ξ tj rix t is nonempty. If X t is of type S2 we additionally assume that: H1-d for t = 1,..., T, there exists ε t > 0 such that for every j = 1,..., M, each component g ti,, ξ tj, i = 1,..., p, of the function g t,, ξ tj is convex on X t X εt t 1 ; H1-e for t = 2,..., T, j = 1,..., M, there exists x tjt 1, x tjt X t 1 rix t such that A tj x tjt + B tj x tjt 1 = b tj, and x tjt 1, x tjt ri{g t,, ξ tj 0}. Remark 3.1. For a problem of form 3.3 where the strong convexity assumption of functions f t,, ξ tj fails to hold, if for every t, j function f t,, ξ tj is convex and if the columns of matrix A tj B tj are independant then we may reformulate the problem pushing and penalizing the linear coupling constraints in the objective, ending up with the strongly convex cost function f t,, ξ tj +ρ t A tj x t +B tj x t 1 b tj 2 2 in variables x t, x t 1 for stage t realization ξ tj for some well chosen penalization ρ t > DASC Algorithm Due to Assumption H0, the M T 1 realizations of ξ t T t=1 form a scenario tree of depth T + 1 where the root node n 0 associated to a stage 0 with decision x 0 taken at that node has one child node n 1 associated to the first stage with ξ 1 deterministic. We denote by N the set of nodes, by Nodest the set of nodes for stage t and for a node n of the tree, we define: Cn: the set of children nodes the empty set for the leaves; x n : a decision taken at that node; p n : the transition probability from the parent node of n to n; ξ n : the realization of process ξ t at node n 2 : for a node n of stage t, this realization ξ n contains in particular the realizations b n of b t, A n of A t, and B n of B t ; ξ [n] : the history of the realizations of process ξ t from the first stage node n 1 to node n: for a node n of stage t, the i-th component of ξ [n] is ξ P t i n for i = 1,..., t, where P : N N is the function associating to a node its parent node the empty set for the root node. Similary to SDDP, at iteration k, trial points x k n are computed in a forward pass for all nodes n of the scenario tree replacing recourse functions Q t+1 by the approximations Q k 1 t+1 available at the beginning of this iteration. In a backward pass, we then select a set of nodes n k 1, n k 2,..., n k T with nk 1 = n 1, and for t 2, n k t a node of stage t, child of node n k t 1 corresponding to a sample ξ 1 k, ξ 2 k,..., ξ T k of ξ 1, ξ 2,..., ξ T. For t = 2,..., T, a cut 4.6 Ct k x t 1 = θt k + βt k, x t 1 x k n + α t k t 1 2 x t 1 x k n 2 k 2 t 1 2 The same notation ξindex is used to denote the realization of the process at node Index of the scenario tree and the value of the process ξ t for stage Index. The context will allow us to know which concept is being referred to. In particular, letters n and m will only be used to refer to nodes while t will be used to refer to stages. 3

4 is computed for Q t at x k where n k t α t = M p tj α tj, j=1 and where the computation of coefficients θ k t, β k t is given below. We show in Section 5 that cut C k t is a lower bounding function for Q t. Contrary to SDDP where cuts are affine functions our cuts are quadratic functions. In the end of iteration k, we obtain the lower approximations Q k t of Q t, t = 2,..., T + 1, given by Q k t x t 1 = max 1 l k Cl t x t 1, which take the form of a maximum of quadratic functions. The detailed steps of the DASC algorithm are given below. DASC, Step 1: Initialization. For t = 2,..., T, take as initial approximations Q 0 t. Set x 1 n 0 = x 0, set the iteration count k to 1, and Q 0 T DASC, Step 2: Forward pass. The forward pass performs the following computations: For t = 1,..., T, For every node n of stage t 1, For every child node m of node n, compute an optimal solution x k m of 4.8 Q k 1 t x k n, ξ m = where x k n 0 = x 0. End For End For End For { inf x m Ft k 1 x m, x k n, ξ m := f t x m, x k n, ξ m + Q k 1 t+1 x m x m X t x k n, ξ m, DASC, Step 3: Backward pass. We select a set of nodes n k 1, n k 2,..., n k T with nk t a node of stage t n k 1 = n 1 and for t 2, n k t a child node of n k t 1 corresponding to a sample ξ 1 k, ξ 2 k,..., ξ T k of ξ 1, ξ 2,..., ξ T. Set θt k +1 = α T +1 = 0 and βt k +1 = 0 which defines Ck T For t = T,..., 2, For every child node m of n = n k t 1 Compute an optimal solution x Bk m 4.9 Q k t xk n, ξ m = of { inf Ft k x m, x k x m n, ξ m := f t x m, x k n, ξ m + Q k t+1x m x m X t x k n, ξ m. For the problem above, if X t is of type S1 we define the Lagrangian Lx m, λ, µ = Ft k x m, x k n, ξ m + λ T A m x m + B m x k n b m and take optimal Lagrange multipliers λ k m. If X t is of type S2 we define the Lagrangian Lx m, λ, µ = Ft k x m, x k n, ξ m + λ T A m x m + B m x k n b m + µ T g t x m, x k n, ξ m and take optimal Lagrange multipliers λ k m, µ k m. If X t is of type S1, denoting by SG ftx Bk m,,ξm x k n a subgradient of convex function f t x Bk m,, ξ m at x k n, we compute θt km = Q k t xk n, ξ m and β km = SG ftx Bk m,,ξm x k n + B T mλ k m. If X t is of type S2 denoting by SG gtix Bk m,,ξm x k n a subgradient of convex function g ti x Bk m,, ξ m at x k n we compute θt km = Q k t xk n, ξ m and β km = SG ftx Bk m,,ξm x k n + B T mλ k m + 4 p i=1 µ k misg gtix Bk m,,ξm x k n.

5 End For The new cut Ct k is obtained computing 4.10 θt k = p m θt km and βt k = End For DASC, Step 4: Do k k + 1 and go to Step 2. p m β km. Remark 4.1. In DASC, decisions are computed at every iteration for all the nodes of the scenario tree in the forward pass. However, in practice, sampling will be used in the forward pass to compute at iteration k decisions only for the nodes n k 1,..., n k T and their children nodes. The variant of DASC written above is convenient for the convergence analysis of the method, presented in the next section. From this convergence analysis, it is possible to show the convergence of the variant of DASC which uses sampling in the forward pass see also Remark 5.4 in [7] and Remark 4.3 in [10]. 5. Convergence analysis In Theorem 5.2 below we show the convergence of DASC making the following additional assumption: H2 The samples in the backward passes are independent: ξ k 2,..., ξ k T is a realization of ξk = ξ k 2,..., ξ k T ξ 2,..., ξ T and ξ 1, ξ 2,..., are independent. We will make use of the following lemma: Lemma 5.1. Let Assumptions H0 and H1 hold. Then for t = 2,..., T + 1, function Q t is convex and Lipschitz continuous on X t 1. Proof. The proof is analogue to the proofs of Lemma 3.2 in [6] and Lemma 2.2 in [4]. Theorem 5.2. Consider the sequences of stochastic decisions x k n and of recourse functions Q k t generated by DASC. Let Assumptions H0, H1 and H2 hold. Then i almost surely, for t = 2,..., T + 1, the following holds: Ht : n Nodest 1, lim k + Q tx k n Q k t x k n = 0. ii Almost surely, the limit of the sequence F1 k 1 x k n 1, x 0, ξ 1 k of the approximate first stage optimal values and of the sequence Q k 1 x 0, ξ 1 k is the optimal value Q 1 x 0 of 3.3. Let Ω = Θ 2... Θ T be the sample space of all possible sequences of scenarios equipped with the product P of the corresponding probability measures. Define on Ω the random variable x = x 1,..., x T as follows. For ω Ω, consider the corresponding sequence of decisions x k nω n N k 1 computed by DASC. Take any accumulation point x nω n N of this sequence. If Z t is the set of F t -measurable functions, define x 1ω,..., x T ω taking x t ω : Z t R n given by x t ωξ 1,..., ξ t = x mω where m is given by ξ [m] = ξ 1,..., ξ t for t = 1,..., T. Then Px 1,..., x T is an optimal solution to 3.3 = 1. Proof. Let us prove i. We first check by induction on k and backward induction on t that for all k 0, for all t = 2,..., T + 1, for any node n of stage t 1 and decision x n taken at that node we have 5.11 Q t x n C k t x n, almost surely. For any fixed k, relation 5.11 holds for t = T + 1 and if it holds until iteration k for t + 1 with t {2,..., T }, we deduce that for any node n of stage t 1 and decision x n taken at that node we have Q t+1 x n Q k t+1x n, Q t x n, ξ m Q k t x n, ξ m for any child node m of n. Now note that function x m, x n Q k t+1x m is convex as a maximum of convex functions and recalling that x m, x n f t x m, x n, ξ m is strongly convex with constant of strong convexity α tm, the function x m, x n f t x m, x n, ξ m + Q k t+1x m is also strongly convex with the same parameter of strong convexity. Using Proposition 2.3, it follows that Q k t, ξ m is strongly convex with constant of strong convexity α tm. 5

6 Using Lemma 2.1 in [6] we have that β km Q k t, ξ mx k n k t 1. Recalling characterization 2.1 of strongly convex functions see Proposition 2.2, we get for any x n X t 1 : Q k t x n, ξ m Q k t xk, ξ n k m + β km, x n x k + t 1 nt 1 αtm k 2 x n x k nt 1 2 k 2 and therefore for any node n of stage t 1 and decision x n taken at that node we have Q t x n = p m Q t x n, ξ m 5.12 p m Q k t x n, ξ m p m Q k t xk n, ξ k m + β km, x n x k t 1 n + α tm k t 1 2 x n x k n 2 k 2 t 1 = θ k t + β k t, x n x k n k t 1 + αt 2 x n x k n k t = C k t x n. This completes the induction step and shows 5.11 for every t, k. Let Ω 1 be the event on the sample space Ω of sequences of scenarios such that every scenario is sampled an infinite number of times. Due to H2, this event has probability one. Take an arbitrary realization ω of DASC in Ω 1. To simplify notation we will use x k n, Q k t, θt k, βt k instead of x k nω, Q k t ω, θt k ω, βt k ω. We want to show that Ht, t = 2,..., T + 1, hold for that realization. The proof is by backward induction on t. For t = T + 1, Ht holds by definition of Q T +1, Q k T +1. Now assume that Ht + 1 holds for some t {2,..., T }. We want to show that Ht holds. Take an arbitrary node n Nodest 1. For this node we define S n = {k 1 : n k t 1 = n} the set of iterations such that the sampled scenario passes through node n. Observe that S n is infinite because the realization of DASC is in Ω 1. We first show that lim Q t x k n Q k t x k n = 0. k +,k S n For k S n, we have n k t 1 = n, i.e., x k n = x k n k t 1, which implies, using 5.11, that 5.13 Q t x k n Q k t x k n C k t x k n = θ k t = p m θ km t = by definition of Ct k and θt k. It follows that for any k S n we have 0 Q t x k n Q k t x k n p m Q t x k n, ξ m Q k t xk n, ξ m 5.14 = = = p m Q k t xk n, ξ m p m Q t x k n, ξ m Q k 1 t x k n, ξ m p m Q t x k n, ξ m Ft k 1 x k m, x k n, ξ m p m Q t x k n, ξ m f t x k m, x k n, ξ m Q k 1 t+1 xk m p m Q t x k n, ξ m F t x k m, x k n, ξ m + Q t+1 x k m Q k 1 t+1 xk m p m Q t+1 x k m Q k 1 t+1 xk m, where for the last inequality we have used the definition of Q t and the fact that x k m X t x k n, ξ m. Next, recall that Q t+1 is convex; by Lemma 5.1 functions Q k t+1 k are Lipschitz continuous; and for all k 1 we have Q k t+1 Q k+1 t+1 Q t+1 on compact set X t. Therefore, the induction hypothesis lim Q t+1x k m Q k t+1x k m = 0 k + 6

7 implies, using Lemma A.1 in [4], that 5.15 lim k + Q t+1x k m Q k 1 t+1 xk m = 0. Plugging 5.15 into 5.14 we obtain 5.16 lim k +,k S n Q t x k n Q k t x k n = 0. It remains to show that 5.17 lim k +,k / S n Q t x k n Q k t x k n = 0. The relation above can be proved using Lemma 5.4 in [10] which can be applied since A relation 5.16 holds convergence was shown for the iterations in S n, B the sequence Q k t k is monotone, i.e., Q k t Q k 1 t for all k 1, C Assumption H2 holds, and D ξt 1 k is independent on x j n, j = 1,..., k, Q j t, j = 1,..., k 1. 3 Therefore, we have shown i. ii can be proved as Theorem 5.3-ii in [7] using i. Acknowledgments The author s research was partially supported by an FGV grant, CNPq grant /2016-9, and FAPERJ grant E-26/ /2014. References [1] J.R. Birge. Decomposition and partitioning methods for multistage stochastic linear programs. Oper. Res., 33: , [2] J.R. Birge and C. J. Donohue. The Abridged Nested Decomposition Method for Multistage Stochastic Linear Programs with Relatively Complete Recourse. Algorithmic of Operations Research, 1:20 30, [3] Z.L. Chen and W.B. Powell. Convergent Cutting-Plane and Partial-Sampling Algorithm for Multistage Stochastic Linear Programs with Recourse. J. Optim. Theory Appl., 102: , [4] P. Girardeau, V. Leclere, and A.B. Philpott. On the convergence of decomposition methods for multistage stochastic convex programs. Mathematics of Operations Research, 40: , [5] V. Guigues. SDDP for some interstage dependent risk-averse problems and application to hydro-thermal planning. Computational Optimization and Applications, 57: , [6] V. Guigues. Convergence analysis of sampling-based decomposition methods for risk-averse multistage stochastic convex programs. SIAM Journal on Optimization, 26: , [7] V. Guigues. Inexact decomposition methods for solving deterministic and stochastic convex dynamic programming equations. arxiv, [8] V. Guigues and W. Römisch. Sampling-based decomposition methods for multistage stochastic programs based on extended polyhedral risk measures. SIAM J. Optim., 22: , [9] V. Guigues and W. Römisch. SDDP for multistage stochastic linear programs based on spectral risk measures. Operations Research Letters, 40: , [10] V. Guigues, W. Tekaya, and M. Lejeune. Regularized decomposition methods for deterministic and stochastic convex optimization and application to portfolio selection with direct transaction and market impact costs. Optimization OnLine, [11] M. Hindsberger and A. B. Philpott. Resa: A method for solving multi-stage stochastic linear programs. SPIX Stochastic Programming Symposium, [12] V. Kozmik and D.P. Morton. Evaluating policies in risk-averse multi-stage stochastic programming. Mathematical Programming, 152: , [13] M.V.F. Pereira and L.M.V.G Pinto. Multi-stage stochastic optimization applied to energy planning. Math. Program., 52: , [14] A. Philpott and V. de Matos. Dynamic sampling algorithms for multi-stage stochastic programs with risk aversion. European Journal of Operational Research, 218: , [15] A. B. Philpott and Z. Guan. On the convergence of stochastic dual dynamic programming and related methods. Oper. Res. Lett., 36: , [16] A. Shapiro. Analysis of stochastic dual dynamic programming method. European Journal of Operational Research, 209:63 72, Lemma 5.4 in [10] is similar to the end of the proof of Theorem 4.1 in [6] and uses the Strong Law of Large Numbers. This lemma itself applies the ideas of the end of the convergence proof of SDDP given in [4], which was given with a different more general sampling scheme in the backward pass. 7

8 [17] A. Shapiro, D. Dentcheva, and A. Ruszczyński. Lectures on Stochastic Programming: Modeling and Theory. SIAM, Philadelphia,