Risk aversion in multi-stage stochastic programming: a modeling and algorithmic perspective

Risk aversion in multi-stage stochastic programming: a modeling and algorithmic perspective Tito Homem-de-Mello School of Business Universidad Adolfo Ibañez, Santiago, Chile Joint work with Bernardo Pagnoncelli (Universidad Adolfo Ibañez) ICSP XIII Bergamo, July 2013 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 1 / 31

Motivation The problem There has been active research recently in the literature regarding the introduction of risk into multi-stage stochastic programs. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 2 / 31

Motivation The problem There has been active research recently in the literature regarding the introduction of risk into multi-stage stochastic programs. There are of course multiple ways of doing that, and no definitive formulation. Our goal is to understand some formulations and discuss pros and cons. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 2 / 31

Motivation The problem There has been active research recently in the literature regarding the introduction of risk into multi-stage stochastic programs. There are of course multiple ways of doing that, and no definitive formulation. Our goal is to understand some formulations and discuss pros and cons. We then focus on a specific formulation that has some interesting properties, and show some numerical results. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 2 / 31

The problem The problem We consider multi-stage stochastic programs of the form where: min R (f 1 (x 1, ξ 1 ) +... + f T (x T, ξ T )) (1) x 1,...,x T s.t. x t X t ( x[t 1], ξ [t] ), t = 1,..., T ξ t, t = 1,..., T is the uncertainty observed just before stage t; f t (x t, ξ t ) corresponds to the cost of decision x t given the observed uncertainty at that stage; x [t] denotes the vector x 1,..., x t, and similarly for ξ [t] ; X t ( x[t 1], ξ [t] ) denotes the feasibility set in stage t; R is a multi-period risk measure. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 3 / 31

The problem Multi-period risk measures In our context, multi-period risk measures are those applied to real-valued functions of the stochastic process {ξ t } T t=1. To simplify notation, let Z t := f t (x t, ξ t ), Z := Z 1 +... + Z T. Examples: R is an unconditional risk global measure ρ applied to the random variable Z for instance, R(Z) = ρ(z) = CVaR α (Z). R measures the risk in each stage, i.e., R(Z) = Z 1 + ρ 2 (Z 2 ) +... + ρ T (Z T ) where each ρ i is an unconditional risk measure such as ρ i = CVaR αi. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 4 / 31

The problem Multi-period risk measures In our context, multi-period risk measures are those applied to real-valued functions of the stochastic process {ξ t } T t=1. To simplify notation, let Z t := f t (x t, ξ t ), Z := Z 1 +... + Z T. Examples: R is an unconditional risk global measure ρ applied to the random variable Z for instance, R(Z) = ρ(z) = CVaR α (Z). R measures the risk in each stage, i.e., R(Z) = Z 1 + ρ 2 (Z 2 ) +... + ρ T (Z T ) where each ρ i is an unconditional risk measure such as ρ i = CVaR αi. R is a nested risk measure given by R(Z) = ρ 2 ρ ξ [2] 3... ρ ξ [T 1] T (Z), where the notation indicates that each ρ ξ [t 1] t is a conditional one-period risk measure, defined in terms of the history ξ 2,..., ξ t 1. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 4 / 31

The problem Coherent risk measures We assume that all unconditional risk measures ρ that appear in the definition of R are coherent, i.e., 1) ρ(x + c) = ρ(x ) + c. 2) X Y w.p.1 ρ(x ) ρ(y ). 3) ρ(λx ) = λρ(x ) for λ 0. 4) ρ(x + Y ) ρ(x ) + ρ(y ). (Artzner et al. 1999, Ruszczynski and Shapiro 2006) Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 5 / 31

Consistency Consistency of risk measures Several notions of consistency have been proposed in the literature (Shapiro 2009, Ruszczynski 2010, Carpentier et al. 2012). Our notion is similar to that in Shapiro (2009) and Carpentier et al. (2012) in that it defines consistency in terms of an underlying optimization problem. To understand the definitions, we introduce the following notation: Let ˆx := [ˆx τ : τ = 1,..., T ] be an optimal solution of (1). Let u be an arbitrary time period such that 1 < u T. Given a realization ˆξ 1,..., ˆξ u, let S ṷ denote the set of optimal x,ˆξ solutions of (1) that coincide with ˆx along the path given by that realization. Note that we can view an element of S ṷ x,ˆξ as [ x τ : τ = u,..., T ], where x τ is a function of ξ u+1,..., ξ τ. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 6 / 31

Consistency Consistency of risk measures Definition R is consistent for problems of the form (1) if there exists an element of S ṷ which is also an optimal solution of the subproblem solved at time u x,ˆξ given the history ˆx [u 1], ˆξ [u]. In words... Consistency means that if you solve the problem at time 1, some optimal solution will also be optimal at time u. Inconsistency means that if you solve the problem at time 1, no optimal solution will also be optimal at time u. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 7 / 31

Examples Example: A 3-stage inventory problem Assume you are a retailer who sells one product and needs to decide now how much inventory to buy, at price c. There will be two selling opportunities: in the second stage the product can be sold at price s 2 > c and in the third stage the product can be sold for s 3 > s 2. At the end of the horizon unsold units are discarded. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 8 / 31

Problem formulation Examples min R ( ) cx 1 s 2 x 2 s 3 x 3 s.t. x 2 D 2 x 2 x 1 x 3 D 3 x 3 x 1 x 2 x 1, x 2, x 3 0, Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 9 / 31

Problem instance Examples In our example, c = 2, s 2 = 3, s 3 = 10, and demand is given by a binary tree. 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 10 / 31

Examples Case 1: unconditional global R Suppose we use R(Z) = CVaR 0.8 (cx 1 s 2 x 2 s 3 x 3 ) Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 11 / 31

Examples Case 1: unconditional global R Suppose we use Optimal solution: R(Z) = CVaR 0.8 (cx 1 s 2 x 2 s 3 x 3 ) 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 11 / 31

Checking consistency Examples We consider now the problem given by the subtree: 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 12 / 31

Checking consistency Examples Optimal solution: 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 13 / 31

Checking consistency Examples We see that the original solution is not optimal for the subproblem. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 14 / 31

Checking consistency Examples We see that the original solution is not optimal for the subproblem. However, the following solution is optimal for the original problem and for the subproblem; thus, the chosen R is consistent. 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 14 / 31

Case 2: stage-wise R Examples Suppose we use R(Z) = cx 1 + CVaR 0.8 ( s 2 x 2 ) + CVaR 0.8 ( s 3 x 3 ) Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 15 / 31

Case 2: stage-wise R Examples Suppose we use Optimal solution: R(Z) = cx 1 + CVaR 0.8 ( s 2 x 2 ) + CVaR 0.8 ( s 3 x 3 ) 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 15 / 31

Checking consistency Examples Subtree optimal solution. No optimal solution for the original problem is optimal for the subproblem; thus, the chosen R is inconsistent. 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 16 / 31

Consistent risk measures Consistent risk measures The result below is similar to others in the literature. Theorem Consider the multi-period risk measure R given by R(Z) = ρ 2 ρ ξ [2] 3... ρ ξ [T 1] T (Z), where the notation indicates that each ρ ξ [t 1] t is a conditional risk measure, defined in terms of the history ξ 2,..., ξ t 1. Then, R is consistent for problem (1). The following corollary is almost immediate: Corollary The risk-neutral measure R(Z) = E[Z] is consistent for problem (1). The worst-case risk measure R(Z) = ess sup (Z) is consistent for problem (1). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 17 / 31

Nested CVaR Consistent risk measures Several authors have studied the case of nested risk measures where each ρ ξ [t 1] t is defined as a conditional CVaR, i.e., { ρ ξ [t 1] t (Z t ) := CVaR ξ [t 1] α t [Z t ] = min η t + 1 E [ } ] (Z t η t ) + ξ η t R 1 α [t 1]. t There are several advantages to this approach: Consistency is ensured due to the nested form; consequently, Bellman-type algorithms for dynamic programming can be employed. The formulation of CVaR as linear stochastic optimization problem allows for seamless integration into problem (1) and consequently relatively easy adaptation of existing methods developed for risk-neutral problems. A prominent example is the Stochastic Dual Dynamic Programming algorithm, which has been used in the risk-averse setting by Guigues and Römisch (2012), Guigues and Sagastizábal (2012), Philpott and de Matos (2012), Shapiro et al. (2012), Kozmik and Morton (2013). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 18 / 31

Consistent risk measures Nested CVaR (cont.) There are however some disadvantages to the nested CVaR approach: Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 19 / 31

Consistent risk measures Nested CVaR (cont.) There are however some disadvantages to the nested CVaR approach: It is difficult to evaluate the objective function directly, which precludes the calculation of simple upper bounds for the optimal value based on a feasible solution. Such upper bounds are important for the implementation of stopping criteria, for instance in the SDDP algorithm. Philpott, de Matos and Finardi (2013) and Kozmik and Morton (2013) have proposed some ways around that problem. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 19 / 31

Consistent risk measures Nested CVaR (cont.) There are however some disadvantages to the nested CVaR approach: It is difficult to evaluate the objective function directly, which precludes the calculation of simple upper bounds for the optimal value based on a feasible solution. Such upper bounds are important for the implementation of stopping criteria, for instance in the SDDP algorithm. Philpott, de Matos and Finardi (2013) and Kozmik and Morton (2013) have proposed some ways around that problem. Nested risk measures can have a somewhat unexpected behavior (which we call lack of structural monotonicity ), as we shall see next. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 19 / 31

Consistent risk measures Back to example Recall the solution of the inventory problem with c = 2, s 2 = 3, s 3 = 10. R(Z) = CVaR 0.8 (cx 1 s 2 x 2 s 3 x 3 ) 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 20 / 31

Changing costs Consistent risk measures Suppose we change the costs to c = 2, s 2 = 8, s 3 = 10. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 21 / 31

Changing costs Consistent risk measures Suppose we change the costs to c = 2, s 2 = 8, s 3 = 10. We see that, even though the selling price goes up, we order and sell less! 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 21 / 31

The E-CVaR risk measure An alternative risk measure One alternative that can be used to address the above issues is the following multi-period risk measure: E-CVaR(Z) := Z 1 + T t=2 [ ] E ξ[t 1] CVaR ξ [t 1] α t (Z t ) The idea is to measure the risk in each period, but compute the expectation instead of carrying it backward as in the nested case. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 22 / 31

The E-CVaR risk measure An alternative risk measure One alternative that can be used to address the above issues is the following multi-period risk measure: E-CVaR(Z) := Z 1 + T t=2 [ ] E ξ[t 1] CVaR ξ [t 1] α t (Z t ) The idea is to measure the risk in each period, but compute the expectation instead of carrying it backward as in the nested case. This risk measure was proposed by Pflug and Ruszczynski (2005). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 22 / 31

The E-CVaR risk measure An alternative risk measure One alternative that can be used to address the above issues is the following multi-period risk measure: E-CVaR(Z) := Z 1 + T t=2 [ ] E ξ[t 1] CVaR ξ [t 1] α t (Z t ) The idea is to measure the risk in each period, but compute the expectation instead of carrying it backward as in the nested case. This risk measure was proposed by Pflug and Ruszczynski (2005). Can also be viewed as particular case of polyhedral risk measures (Eichhorn and Römisch 2005). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 22 / 31

The E-CVaR risk measure Back to the example Solution of the inventory problem with E-CVaR, with worst-case in each stage. ] R(Z) = cx 1 + CVaR 0.8 ( s 2 x 2 ) + E D2 [CVaR D 2 0.8 ( s 3x 3 ) 0.5 0.5 0.5 0.5 0.5 0.5 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 23 / 31

The E-CVaR risk measure An attractive property At first sight it appears that E-CVaR could be inconsistent since it does not have an apparent nested formulation. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 24 / 31

The E-CVaR risk measure An attractive property At first sight it appears that E-CVaR could be inconsistent since it does not have an apparent nested formulation. However, the result below shows that this is not the case: Theorem Problem (1) with R = E-CVaR can be written as a risk-neutral multi-stage problem with some extra variables and constraints. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 24 / 31

The E-CVaR risk measure An attractive property At first sight it appears that E-CVaR could be inconsistent since it does not have an apparent nested formulation. However, the result below shows that this is not the case: Theorem Problem (1) with R = E-CVaR can be written as a risk-neutral multi-stage problem with some extra variables and constraints. The proof follows from the optimization formulation of CVaR ξ [t 1]; the extra variables are the η t. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 24 / 31

The E-CVaR risk measure An attractive property At first sight it appears that E-CVaR could be inconsistent since it does not have an apparent nested formulation. However, the result below shows that this is not the case: Theorem Problem (1) with R = E-CVaR can be written as a risk-neutral multi-stage problem with some extra variables and constraints. The proof follows from the optimization formulation of CVaR ξ [t 1]; the extra variables are the η t. See also Eichhorn and Römisch (2005). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 24 / 31

Comments The E-CVaR risk measure The equivalence with a risk-neutral multi-stage problem allows for the use of standard methods developed for that type of problem. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 25 / 31

Comments The E-CVaR risk measure The equivalence with a risk-neutral multi-stage problem allows for the use of standard methods developed for that type of problem. In particular, the calculation of upper bounds (e.g., in SDDP) is not an issue. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 25 / 31

Comments The E-CVaR risk measure The equivalence with a risk-neutral multi-stage problem allows for the use of standard methods developed for that type of problem. In particular, the calculation of upper bounds (e.g., in SDDP) is not an issue. Moreover, the E-CVaR risk measure does not suffer from the lack of structural monotonicity discussed earlier since risk is measured in a stage-wise fashion. Indeed, the solution does not change when we increase s 2. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 25 / 31

Numerical illustration A numerical example We consider the Dutch pension fund problem described in Haneveld, Streutker and van der Vlerk (2010). The fund sponsor has to maintain the ratio between assets and liabilities above some pre-specified threshold at every time period. To achieve this goal, three sources of income can be used: Returns from the asset portfolio (stocks, bonds, real estate, cash) Regular contributions made by fund participants Remedial contributions done by the company, which are money injections intended to keep the fund solvent. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 26 / 31

Numerical illustration A numerical example We consider the Dutch pension fund problem described in Haneveld, Streutker and van der Vlerk (2010). The fund sponsor has to maintain the ratio between assets and liabilities above some pre-specified threshold at every time period. To achieve this goal, three sources of income can be used: Returns from the asset portfolio (stocks, bonds, real estate, cash) Regular contributions made by fund participants Remedial contributions done by the company, which are money injections intended to keep the fund solvent. The objective function is to minimize the expected value of the sum, for all stages, of remedial contributions and contribution rates for the participants, while keeping the funding ratio above a threshold. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 26 / 31

Results Numerical illustration The problem instance has 4 stages; uncertainty (liabilities, contributions, returns) is modeled with a scenario tree with 10 stages per node. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 27 / 31

Results Numerical illustration The problem instance has 4 stages; uncertainty (liabilities, contributions, returns) is modeled with a scenario tree with 10 stages per node. We solved the problem using the E-CVaR risk measure; implementation was done in SLP-IOR (Kall and Mayer 1996) using the equivalent risk-neutral formulation. The table shows the first-stage portfolio allocations, participant contribution rate and remedial contributions. α stock bond real est. cash rate remedial (0,0,0) 7184 4789 3991 0.18 0 (0,0,1) 7395 4790 3782 0.21 0 (0,1,0) 8202 4790 2975 0.21 0 (1,0,0) 4866 8379 2975 0.21 265 (1,1,1) 7468 5189 3570 0.21 260 (.5,.5,.5) 6361 5609 3990 0.21 0 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 27 / 31

Numerical illustration Second-stage variables The graphs show the cdfs of the second-stage variables. 1 Stocks + Real estate 1 Bonds 0.8 0.8 0.6 0.6 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0 0.8 0.9 1 1.1 1.2 1.3 1.4 x 10 4 0 5000 5200 5400 5600 5800 6000 6200 6400 6600 1 Cash 1 Remedial contribution 0.8 0.8 0.6 0.6 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0 0 500 1000 1500 2000 2500 3000 3500 4000 0 0 500 1000 1500 2000 2500 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 28 / 31

Numerical illustration Third-stage variables The graphs show the cdfs of the third-stage variables. 1 Stocks + Real estate 1 Bonds 0.8 0.8 0.6 0.6 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0 0.6 0.8 1 1.2 1.4 1.6 1.8 2 x 10 4 0 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 x 10 4 1 Cash 1 Remedial contribution 0.8 0.8 0.6 0.6 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0.4 0.2 (1,0,0) (1,1,1) (0,0,1) (0,0,0) 0 0 1000 2000 3000 4000 5000 6000 0 0 1000 2000 3000 4000 5000 6000 Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 29 / 31

Summary Conclusions While there are multiple ways to incorporate risk into multi-stage stochastic programs, much of the (numerical) literature focuses on nested CVaR formulations. We have discussed an alternative approach, which we call E-CVaR, that addresses some issues that appear with nested formulations. An attractive feature of the E-CVaR formulation is that is can be represented as a risk-neutral problem on a lifted space, so standard algorithms can be used. Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 30 / 31

Summary Conclusions While there are multiple ways to incorporate risk into multi-stage stochastic programs, much of the (numerical) literature focuses on nested CVaR formulations. We have discussed an alternative approach, which we call E-CVaR, that addresses some issues that appear with nested formulations. An attractive feature of the E-CVaR formulation is that is can be represented as a risk-neutral problem on a lifted space, so standard algorithms can be used. Moreover, since the risk measures are applied in a stage-wise fashion, it is easier to see what is being controlled (e.g., via the value of α t ). Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 30 / 31

Acknowledgment Conclusions This research has been supported by Fondecyt project 1120244, Chile. THANK YOU! Homem-de-Mello & Pagnoncelli (UAI) Risk-averse multi-stage SP ICSP 2013 31 / 31