Risk-Averse Decision Making and Control

Transcription

1 Marek Petrik University of New Hampshire Mohammad Ghavamzadeh Adobe Research February 4, 2017

2 Introduction to Risk Averse Modeling Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

3 Introduction to Risk Averse Modeling Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

4 Introduction to Risk Averse Modeling Risk Aversion Risk (Wikipedia): Risk is the potential of gaining or losing something of value.... Uncertainty is a potential, unpredictable, and uncontrollable outcome; risk is a consequence of action taken in spite of uncertainty. Risk aversion (Wikipedia):... risk aversion is the behavior of humans, when exposed to uncertainty, to attempt to reduce that uncertainty.... Tutorial: Modern methods for risk-averse decision making

5 Introduction to Risk Averse Modeling Desire for Risk Aversion Empirical evidence: 1. People buy insurance 2. Diversifying financial portfolios 3. Experimental results Other reasons: Reduce contingency planning

6 Introduction to Risk Averse Modeling Where Risk Aversion Matters Financial portfolios Heath-care decisions Agriculture Public infrastructure Self-driving cars?

7 Introduction to Risk Averse Modeling When Risks Are Ignored... Housing bubble leads to a financial collapse Seawalls overflow in a tsunami

8 Introduction to Risk Averse Modeling Need to Quantify Risk Mitigating risk is expensive, how much is it worth?

9 Introduction to Risk Averse Modeling Need to Quantify Risk Mitigating risk is expensive, how much is it worth? Expected utility theory: E[u(X)] = E[utility(X)]

10 Introduction to Risk Averse Modeling Need to Quantify Risk Mitigating risk is expensive, how much is it worth? Expected utility theory: E[u(X)] = E[utility(X)] Exponential utility function (Bernoulli functions): u(x) = 1 e a x a Utility Function Reward ($)

11 Introduction to Risk Averse Modeling Example: Buying Car Insurance Car value: $ Insurance options Option Deductible Cost X 1 $ $0 X 2 $2 000 $112 X 3 $100 $322

12 Introduction to Risk Averse Modeling Example: Buying Car Insurance Car value: $ Insurance options Option Deductible Cost X 1 $ $0 X 2 $2 000 $112 X 3 $100 $322 Expected utility: Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422

13 Introduction to Risk Averse Modeling Example: Buying Car Insurance Car value: $ Insurance options Option Deductible Cost X 1 $ $0 X 2 $2 000 $112 X 3 $100 $322 Expected utility: Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422

14 Introduction to Risk Averse Modeling Example: Buying Car Insurance Car value: $ Insurance options Option Deductible Cost X 1 $ $0 X 2 $2 000 $112 X 3 $100 $322 Expected utility: Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422

15 Introduction to Risk Averse Modeling Example: Buying Car Insurance Car value: $ Insurance options Option Deductible Cost X 1 $ $0 X 2 $2 000 $112 X 3 $100 $322 Expected utility: Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $ $ $ Risk-neutral choice: no insurance

16 Introduction to Risk Averse Modeling Risk Averse Utility Functions Exponential utility function u(x) = 1 exp( 10 6 (x )) 10 6 X 1 no insurance X 2 high deductible insurance Event P X 1 u(x 1 ) X 2 u(x 2 ) No accident 92% $ $ Minor accident 7.5% $ $ Major accident 0.5% $ $ E $ $

17 Introduction to Risk Averse Modeling Risk Averse Utility Functions Exponential utility function u(x) = 1 exp( 10 6 (x )) 10 6 X 1 no insurance X 2 high deductible insurance Event P X 1 u(x 1 ) X 2 u(x 2 ) No accident 92% $ $ Minor accident 7.5% $ $ Major accident 0.5% $ $ E $ $ Prefer insurance, but difficult to interpret and elicit

18 Introduction to Risk Averse Modeling Drawbacks of Expected Utility Theory (Schoemaker 1980) 1. Does not explain human behavior 2. Difficult to elicit utilities 3. Complicates optimization (Friedman et al. 2014)

19 Introduction to Risk Averse Modeling Major Alternatives for Measuring Risk 1. Markowitz portfolios: Penalize dispersion risk [ ] min Var c i X i c 0 [ i ] s.t. E c i X i = µ, c i = 1 i Limited modeling capability and also penalizes upside i

20 Introduction to Risk Averse Modeling Major Alternatives for Measuring Risk 1. Markowitz portfolios: Penalize dispersion risk [ ] min Var c i X i c 0 [ i ] s.t. E c i X i = µ, c i = 1 i Limited modeling capability and also penalizes upside i 2. Risk measures: (Artzner et al. 1999) Value at risk (V@R) Conditional value at risk (CV@R) Coherent measures of risk

21 Introduction to Risk Averse Modeling Coherent Measures of Risk Topic of this tutorial Alternative to expected utility theory

22 Introduction to Risk Averse Modeling Coherent Measures of Risk Topic of this tutorial Alternative to expected utility theory + Flexible modeling framework

23 Introduction to Risk Averse Modeling Coherent Measures of Risk Topic of this tutorial Alternative to expected utility theory + Flexible modeling framework + Convenient to use with optimization and decision making

24 Introduction to Risk Averse Modeling Coherent Measures of Risk Topic of this tutorial Alternative to expected utility theory + Flexible modeling framework + Convenient to use with optimization and decision making + Easier to elicit than utilities

25 Introduction to Risk Averse Modeling Coherent Measures of Risk Topic of this tutorial Alternative to expected utility theory + Flexible modeling framework + Convenient to use with optimization and decision making + Easier to elicit than utilities Difficulties in sequential decision making

26 (Average) Value at Risk Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

27 (Average) Value at Risk Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

28 (Average) Value at Risk Risk Measure Risk measure: function ρ that maps random variable to a real number

29 (Average) Value at Risk Risk Measure Risk measure: function ρ that maps random variable to a real number Expectation is a risk measure ρ(x) = E[X] = ω Ω X(ω)P (ω) Risk neutral

30 (Average) Value at Risk Risk Measure Risk measure: function ρ that maps random variable to a real number Expectation is a risk measure ρ(x) = E[X] = ω Ω X(ω)P (ω) Risk neutral Worst-case is a risk measure ρ(x) = min[x] = min ω Ω X(ω) Very risk averse

31 (Average) Value at Risk Value at Risk ρ(x) = V@R α (X) = sup { t : P[X t] < α } Rewards smaller than V@R α (X) with probability at most α Example α values: α = 0.5 Median

32 (Average) Value at Risk Value at Risk ρ(x) = V@R α (X) = sup { t : P[X t] < α } Rewards smaller than V@R α (X) with probability at most α Example α values: α = 0.5 Median α = 0.3 More conservative

33 (Average) Value at Risk Value at Risk ρ(x) = V@R α (X) = sup { t : P[X t] < α } Rewards smaller than V@R α (X) with probability at most α Example α values: α = 0.5 Median α = 0.3 More conservative α = 0.05 Conservative

34 (Average) Value at Risk Value at Risk ρ(x) = V@R α (X) = sup { t : P[X t] < α } Rewards smaller than V@R α (X) with probability at most α Example α values: α = 0.5 Median α = 0.3 More conservative α = 0.05 Conservative α = 0 Worst case

35 (Average) Value at Risk Example 1: Cumulative Distribution Function 0.05 (X) = CDF Reward ($)

36 (Average) Value at Risk Example 2: Cumulative Distribution Function 0.3 (X) = CDF Reward ($)

37 (Average) Value at Risk Car Insurance And 25% 1.0 Event P X 1 No accident 92% $0 Minor accident 7.5% $2 500 Major accident 0.5% $ CDF Reward ($) V@R α (X) = sup { t : P[X t] < α } α = 0.25 t P[X t] α $ $ $

38 (Average) Value at Risk Car Insurance And 8% 1.0 Event P X 1 No accident 92% $0 Minor accident 7.5% $2 500 Major accident 0.5% $ CDF Reward ($) V@R α (X) = sup { t : P[X t] < α } α = t P[X t] α $ $

39 (Average) Value at Risk Car Insurance And X 1 : no insurance (high risk) X 2 : high deductible insurance (medium risk) X 3 : low deductible insurance (low risk) Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330

40 (Average) Value at Risk Car Insurance And X 1 : no insurance (high risk) X 2 : high deductible insurance (medium risk) X 3 : low deductible insurance (low risk) Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330 V@R 0.25 $0 $112 $322

41 (Average) Value at Risk Car Insurance And X 1 : no insurance (high risk) X 2 : high deductible insurance (medium risk) X 3 : low deductible insurance (low risk) Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330 V@R 0.25 $0 $112 $322 V@R 0.05 $2 500 $2 112 $422

42 (Average) Value at Risk Properties of + Preserves affine transformations: α (τ X + c) = τ V@R α (X) + c + Simple and intuitive to model and understand + Compelling meaning in finance Ignores heavy tails Not convex

43 (Average) Value at Risk Properties of + Preserves affine transformations: α (τ X + c) = τ V@R α (X) + c + Simple and intuitive to model and understand + Compelling meaning in finance Ignores heavy tails Not convex Coherent measures of risk: Preserve V@R positives and improve negatives (Artzner et al. 1999)

44 (Average) Value at Risk Average Value at Risk AKA Conditional Value at Risk and Expected Shortfall Popular coherent risk measure ρ Simple definition for atomless distributions: [ ] CV@R α (X) = E X X V@R α (X) Recall: V@R α (X) = sup { t : P[X t] < α } Convex extension of V@R (Rockafellar and Uryasev 2000)

45 (Average) Value at Risk vs Cumulative Distribution Function 0.3 (X) = 0.5 CV@R 0.3 (X) = CDF CDF Reward ($) Reward ($)

46 (Average) Value at Risk vs Heavy Tails A more expensive car? Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $ Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $

47 (Average) Value at Risk vs Heavy Tails A more expensive car? Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $ V@R 0.05 $2 500 Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $ V@R 0.05 $2 500

48 (Average) Value at Risk vs Heavy Tails A more expensive car? Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $ V@R 0.05 $2 500 CV@R 0.05 $3 250 Event P X 1 No accident 92% $0 Minor acc. 7.5% $2 500 Major acc. 0.5% $ V@R 0.05 $2 500 CV@R 0.05 $

49 (Average) Value at Risk Heavy Tails and Financial Crisis

50 (Average) Value at Risk vs Continuity Risk ($) V@R AV@R Risk level α

51 (Average) Value at Risk Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

52 Coherent Measures of Risk Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

53 Coherent Measures of Risk Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

54 Coherent Measures of Risk Coherent Measures of Risk Generalize to allow more general models Framework introduced in (Artzner et al. 1999) Coherence: Requirements for risk measure ρ to satisfy Our treatment based on (Shapiro, Dentcheva, and Ruszczynski 2009) and (Follmer and Schied 2011)

55 Coherent Measures of Risk Coherence Requirements of Risk Measures 1. Convexity: (really concavity for maximization!) ρ(t X + (1 t) Y ) t ρ(x) + (1 t) ρ(y ) 2. Monotonicity: If X Y, then ρ(x) ρ(y ) 3. Translation equivariance: For a constant a: ρ(x + a) = ρ(x) + a 4. Positive homogeneity: For t > 0, then: ρ(t X) = t ρ(x)

56 Coherent Measures of Risk Convexity Why: Diversification should decrease risk (and it helps with optimization) ρ(t X + (1 t) Y ) t ρ(x) + (1 t) ρ(y )

57 Coherent Measures of Risk Convexity Why: Diversification should decrease risk (and it helps with optimization) ρ(t X + (1 t) Y ) t ρ(x) + (1 t) ρ(y ) 1 Event P X 1 X 2 2 X X 2 No accident 92% $0 $112 $56 Minor accident 7.5% $2 500 $2 112 $2 306 Major accident 0.5% $ $2 112 $6 056 CV@R $238 $272 $240

58 Coherent Measures of Risk Convexity Why: Diversification should decrease risk (and it helps with optimization) ρ(t X + (1 t) Y ) t ρ(x) + (1 t) ρ(y ) 1 Event P X 1 X 2 2 X X 2 No accident 92% $0 $112 $56 Minor accident 7.5% $2 500 $2 112 $2 306 Major accident 0.5% $ $2 112 $6 056 CV@R $238 $272 $ = 255

59 Coherent Measures of Risk Monotonicity Why: Do not prefer an outcome that is always worse If X Y, then ρ(x) ρ(y )

60 Coherent Measures of Risk Monotonicity Why: Do not prefer an outcome that is always worse If X Y, then ρ(x) ρ(y ) X 2 : Insurance with deductible of $ Event P X 1 X 2 No accident 92% $0 $112 Minor accident 7.5% $2 500 $2 500 Major accident 0.5% $ $ ρ $238 $320

61 Coherent Measures of Risk Monotonicity Why: Do not prefer an outcome that is always worse If X Y, then ρ(x) ρ(y ) X 2 : Insurance with deductible of $ Event P X 1 X 2 No accident 92% $0 $112 Minor accident 7.5% $2 500 $2 500 Major accident 0.5% $ $ ρ $238 $320 $320 $238

62 Coherent Measures of Risk Translation equivariance Why: Risk is measured in the same units as the reward ρ(x + a) = ρ(x) + a

63 Coherent Measures of Risk Translation equivariance Why: Risk is measured in the same units as the reward ρ(x + a) = ρ(x) + a More expensive insurance by $100 Event P X 2 X 2 No accident 92% $112 $212 Minor accident 7.5% $2 112 $2 212 Major accident 0.5% $2 112 $2 212 ρ $272 $372

64 Coherent Measures of Risk Translation equivariance Why: Risk is measured in the same units as the reward ρ(x + a) = ρ(x) + a More expensive insurance by $100 Event P X 2 X 2 No accident 92% $112 $212 Minor accident 7.5% $2 112 $2 212 Major accident 0.5% $2 112 $2 212 ρ $272 $372 $372 = $272 $100

65 Coherent Measures of Risk Positive homogeneity Why: Risk is measured in the same units as the reward ρ(t X) = t ρ(x)

66 Coherent Measures of Risk Positive homogeneity Why: Risk is measured in the same units as the reward ρ(t X) = t ρ(x) What if the prices are in e: $1 = e0.94 Event P X 2 X 2 No accident 92% $112 e105 Minor accident 7.5% $2 112 e1 985 Major accident 0.5% $2 112 e1 985 ρ $272 e256

67 Coherent Measures of Risk Positive homogeneity Why: Risk is measured in the same units as the reward ρ(t X) = t ρ(x) What if the prices are in e: $1 = e0.94 Event P X 2 X 2 No accident 92% $112 e105 Minor accident 7.5% $2 112 e1 985 Major accident 0.5% $2 112 e1 985 ρ $272 e256 $272 = e256

68 Coherent Measures of Risk Convex Risk Measures Weaker definition than coherent risk measures 1. Convexity: 2. Monotonicity: ρ(t X + (1 t) Y ) t ρ(x) + (1 t) ρ(y ) If X Y, then ρ(x) ρ(y ) 3. Translation equivariance: For a constant a: 4. Positive homogeneity ρ(x + a) = ρ(x) + a

69 Coherent Measures of Risk Additional Property: Law Invariance Value of risk measure is independent of the names of the events Consider a coin flip Event P X Y Heads 1/2 1 0 Tails 1/2 0 1 Require that ρ(x) = ρ(y ); violated by some coherent risk measures Distortion risk measures: coherence & law invariance & comonotonicity

70 Coherent Measures of Risk Simple Coherent Measures of Risk Expectation: ρ(x) = E[X] = ω Ω X(ω)P (ω) 1. Convexity: E[X] is linear 2. Monotonicity: E[X] E[Y ] if X Y 3. Translation equivariance: E[X + a] = E[X] + a 4. Positive homogeneity: E[t X] = t E[X] for t > 0

71 Coherent Measures of Risk Simple Coherent Measures of Risk Expectation: ρ(x) = E[X] = ω Ω X(ω)P (ω) 1. Convexity: E[X] is linear 2. Monotonicity: E[X] E[Y ] if X Y 3. Translation equivariance: E[X + a] = E[X] + a 4. Positive homogeneity: E[t X] = t E[X] for t > 0 Worst case: ρ(x) = min[x] = min ω Ω X(ω) 1. Convexity: min[x] is convex 2. Monotonicity: min[x] min[y ] if X Y 3. Translation equivariance: min[x + a] = min[x] + a 4. Positive homogeneity: min[t X] = t E[X] for t > 0

72 Coherent Measures of Risk for Discrete Distributions Simple definition is not coherent [ ] CV@R α (X) = E X X V@R α (X) Violates convexity when distribution has atoms (discrete distributions)

73 Coherent Measures of Risk for Discrete Distributions Simple definition is not coherent [ ] CV@R α (X) = E X X V@R α (X) Violates convexity when distribution has atoms (discrete distributions) Coherent definition of CV@R: { CV@R α (X) = sup t + 1 } t α E[X t] t = V@R α (X) when the distribution is atom-less

74 Coherent Measures of Risk for Discrete Distributions Simple definition is not coherent [ ] CV@R α (X) = E X X V@R α (X) Violates convexity when distribution has atoms (discrete distributions) Coherent definition of CV@R: { CV@R α (X) = sup t + 1 } t α E[X t] t = V@R α (X) when the distribution is atom-less Definitions the same for continuous distributions

75 Coherent Measures of Risk Computing Discrete distributions: Solve a linear program max t,y t + 1 α p y s.t. y X t, y 0 Continuous distributions: Closed form for many (Nadarajah, Zhang, and Chan 2014; Andreev, Kanto, and Malo 2005)

76 Coherent Measures of Risk Car Insurance and X 1 no insurance X 2 high deductible insurance X 3 low deductible insurance Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330

77 Coherent Measures of Risk Car Insurance and X 1 no insurance X 2 high deductible insurance X 3 low deductible insurance Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330 V@R 0.25 $0 $112 $322 CV@R 0.25 $950 $752 $354

78 Coherent Measures of Risk Car Insurance and X 1 no insurance X 2 high deductible insurance X 3 low deductible insurance Event P X 1 X 2 X 3 No accident 92% $0 $112 $322 Minor accident 7.5% $2 500 $2 112 $422 Major accident 0.5% $ $2 112 $422 E $238 $272 $330 V@R 0.25 $0 $112 $322 CV@R 0.25 $950 $752 $354 V@R 0.05 $2 500 $2 112 $422 CV@R 0.05 $3 250 $2 112 $422

79 Coherent Measures of Risk Robust Representation of Coherent Risk Measures Important representation for analysis and optimization For any coherent risk measure ρ: ρ(x) = min E [ ] ξ X = inf ξ A ξ A ξ X

80 Coherent Measures of Risk Robust Representation of Coherent Risk Measures Important representation for analysis and optimization For any coherent risk measure ρ: ρ(x) = min E [ ] ξ X = inf ξ A ξ A ξ X A is a set of measures such that is: 1. convex 2. bounded 3. closed

81 Coherent Measures of Risk Robust Representation of Coherent Risk Measures Important representation for analysis and optimization For any coherent risk measure ρ: ρ(x) = min E [ ] ξ X = inf ξ A ξ A ξ X A is a set of measures such that is: 1. convex 2. bounded 3. closed Proof: Double convex conjugate Convex conjugate: Fenchel Moreau theorem: ρ (y) = sup x y ρ(x) x ρ (x) = ρ(x)

82 Coherent Measures of Risk Robust Set for { α (X) = sup t + 1 } t α E[X t] Robust representation: ρ(x) = inf E [ ] ξ X ξ A Robust set for probability distribution P : { A = ξ 0 ξ 1 } α P, 1 ξ = 1

83 Coherent Measures of Risk Robust Set for Robust representation: ρ(x) = min E [ ] ξ X ξ A { A = ξ 0 ξ 1 } α P, 1 ξ = 1 Random variable: X = [10, 5, 2] Probability distribution: p = [ 1 /3, 1 /3, 1 /3] CV@R 1/2 (X) = min ξ 0 10 ξ ξ ξ 3 ξ i 1 α p i = 1 1/2 1/3 = 2 3 ξ 1 + ξ 2 + ξ 3 = 1

84 Coherent Measures of Risk Other Coherent Risk Measures 1. Combination of expectation and 2. Entropic risk measure 3. Coherent entropic risk measure (convex, incoherent) 4. Risk measures from utility functions 5....

85 Coherent Measures of Risk Convex Combination of Expectation and ignores the mean return Risk-averse solutions bad in expectation Practical trade-off: Combine mean and risk ρ(x) = c E[X] + (1 c) CV@R α (X)

86 Coherent Measures of Risk Entropic Risk Measure Convex risk measure ρ(x) = 1 /τ ln E [ e τ X] τ > 0

87 Coherent Measures of Risk Entropic Risk Measure ρ(x) = 1 /τ ln E [ e τ X] τ > 0 Convex risk measure Incoherent (violates translation invariance) No robust representation

88 Coherent Measures of Risk Entropic Risk Measure Convex risk measure ρ(x) = 1 /τ ln E [ e τ X] τ > 0 Incoherent (violates translation invariance) No robust representation Coherent entropic risk measure: (Föllmer and Knispel 2011) { } ρ(x) = max E ξ [X] KL(ξ P ) c, 1 ξ = 1 ξ 0

89 Coherent Measures of Risk Risk Measure From Utility Function Concave utility function u( ) Construct a coherent risk measure from g?

90 Coherent Measures of Risk Risk Measure From Utility Function Concave utility function u( ) Construct a coherent risk measure from g? Direct construction: ρ(x) = E[u(X)] Not coherent or convex

91 Coherent Measures of Risk Risk Measure From Utility Function Concave utility function u( ) Construct a coherent risk measure from g? Direct construction: Not coherent or convex ρ(x) = E[u(X)] Optimized Certainty Equivalent (Ben-Tal and Teboulle 2007) ρ(x) = sup (t + E[g(X t)]) t

92 Coherent Measures of Risk Optimized Certainty Equivalent ρ(x) = sup (t + E[g(X t)]) t How much consume now given uncertain future

93 Coherent Measures of Risk Optimized Certainty Equivalent ρ(x) = sup (t + E[g(X t)]) t How much consume now given uncertain future Convex risk measure for any concave u Coherent risk measure for pos. homogeneous u

94 Coherent Measures of Risk Optimized Certainty Equivalent ρ(x) = sup (t + E[g(X t)]) t How much consume now given uncertain future Convex risk measure for any concave u Coherent risk measure for pos. homogeneous u Exponential u: OCE = entropic risk measure Piecewise linear u: OCE = CV@R

95 Coherent Measures of Risk Recommended References Lectures on Stochastic Programming: Modeling and Theory (Shapiro, Dentcheva, and Ruszczynski 2014) Stochastic Finance: An Introduction in Discrete Time (Follmer and Schied 2011)

96 Coherent Measures of Risk Remainder of Tutorial: Multistage Optimization How to apply risk measures when optimizing over multiple time steps Results in machine learning and reinforcement learning Time or dynamic consistency in multiple time steps

97 Coherent Measures of Risk Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

98 Risk Measures in Reinforcement Learning Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

99 Risk Measures in Reinforcement Learning Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

100 Risk Measures in Reinforcement Learning Please see the other slide deck

101 Risk Measures in Reinforcement Learning Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

102 Time consistency of in reinforcement learning Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

103 Time consistency of in reinforcement learning Schedule 9:00 9:20 Introduction to risk-averse modeling 9:20 9:40 Value at Risk and Average Value at Risk 9:40 9:50 Break 9:50 10:30 Coherent Measures of Risk: Properties and methods 10:30 11:00 Coffee break 11:00 12:30 Risk-averse reinforcement learning 12:30 12:40 Break 12:40 12:55 Time consistent measures of risk

104 Time consistency of in reinforcement learning Example: Driving Test Discount Option 1: Plain Insurance Cost: $9.00 No deductible Certain expected outcome: E[X 1 ] = 9.00 ρ(x 1 ) = E[X 1 ] = 9.00

105 Time consistency of in reinforcement learning Example: Driving Test Discount Option 1: Plain Insurance Cost: $9.00 No deductible Certain expected outcome: E[X 1 ] = 9.00 ρ(x 1 ) = E[X 1 ] = 9.00 Option 2: Custom Insurance Take a safety exam Pass with probability 1 /2 OK [P = 2/3]: +$5.00 Not [P = 2/3]: $20.00 Fail with probability 1 /2 OK [P = 2/3]: $5.00 Not [P = 2/3]: $10.00

106 Time consistency of in reinforcement learning Example: Driving Test Discount Option 1: Plain Insurance Cost: $9.00 No deductible Certain expected outcome: E[X 1 ] = 9.00 ρ(x 1 ) = E[X 1 ] = 9.00 Option 2: Custom Insurance Take a safety exam Pass with probability 1 /2 OK [P = 2/3]: +$5.00 Not [P = 2/3]: $20.00 Fail with probability 1 /2 OK [P = 2/3]: $5.00 Not [P = 2/3]: $10.00 Risk measure: ρ = CV@R 2/3

107 Time consistency of in reinforcement learning Risk Measure of Option 2 OK 5 1/3 Fail P = 1 /2 1/3 1/3 OK 5 Risk measure: ρ(x 2 ) = CV@R 2/3 (X 2 ) Take test Pass P = 1 /2 1/3 1/3 1/3 NOK 10 OK 5 OK 5 P X 2 1/6 5 1/6 5 1/6 10 1/6 5 1/6 5 1/6 20 NOK 20

108 Time consistency of in reinforcement learning Risk Measure of Option 2 1/3 OK 5 Risk measure: ρ(x 2 ) = CV@R 2/3 (X 2 ) Fail P = 1 /2 Take test Pass P = 1 /2 1/3 1/3 1/3 OK 5 NOK 10 OK 5 P X 2 1/6 5 1/6 5 1/6 10 1/6 5 1/6 5 1/6 20 1/3 1/3 OK 5 NOK ρ(x 2 ) = = 4 = 10.0 < 9.0 = ρ(x 1 )

109 Time consistency of in reinforcement learning Risk Measure of Option 2 1/3 OK 5 Risk measure: ρ(x 2 ) = CV@R 2/3 (X 2 ) Fail P = 1 /2 Take test Pass P = 1 /2 1/3 1/3 1/3 1/3 1/3 OK 5 NOK 10 OK 5 OK 5 P X 2 1/6 5 1/6 5 1/6 10 1/6 5 1/6 5 1/6 20 NOK 20 ρ(x 2 ) < ρ(x 1 ) Prefer option 1

110 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 NOK 20 1/3 Pass test 1/3 OK 5 1/3 OK 5 P 1/3 1/3 1/3 X

111 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 NOK 20 1/3 Pass test 1/3 OK 5 1/3 OK 5 P 1/3 1/3 1/3 X ρ(x 2 Pass) = = If pass, prefer option 2

112 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 NOK 10 1/3 Fail test 1/3 OK 5 1/3 OK 5 P 1/3 1/3 1/3 X

113 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 NOK 10 1/3 Fail test 1/3 OK 5 1/3 OK 5 P 1/3 1/3 1/3 X

114 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 NOK 10 1/3 Fail test 1/3 OK 5 1/3 OK 5 P 1/3 1/3 1/3 X ρ(x 2 Fail) = If fail, prefer option 2 = 7.5

115 Time consistency of in reinforcement learning Optimal Solution of Subproblems Recall we prefer option 1: ρ(x 1 ) = 9 1/3 Pass test 1/3 1/3 1/3 Fail test 1/3 1/3 NOK 20 OK 5 OK 5 NOK 10 OK 5 OK 5 P 1/3 1/3 1/3 X P 1/3 1/3 1/3 X ρ(x 2 Pass) = = If pass, prefer option 2 ρ(x 2 Fail) = = If fail, prefer option 2 Time inconsistent behavior (Roorda, Schumacher, and Engwerda 2005; Iancu, Petrik, and Subramanian 2015)

116 Time consistency of in reinforcement learning Time Consistent Risk Measures Filtration (scenario tree) of rewards with T levels: X 1, X 2, X 3,..., X T Dynamic risk measure at time t: ρ t (X t + + X T )

117 Time consistency of in reinforcement learning Time Consistent Risk Measures Filtration (scenario tree) of rewards with T levels: X 1, X 2, X 3,..., X T Dynamic risk measure at time t: ρ t (X t + + X T ) Time consistent: if for all X, Y (also dynamic consistent) ρ t+1 (X t + ) ρ t+1 (Y t + ) ρ t (X t + ) ρ t (Y t + )

118 Time consistency of in reinforcement learning Time Consistent Risk Measures Filtration (scenario tree) of rewards with T levels: X 1, X 2, X 3,..., X T Dynamic risk measure at time t: ρ t (X t + + X T ) Time consistent: if for all X, Y (also dynamic consistent) ρ t+1 (X t + ) ρ t+1 (Y t + ) ρ t (X t + ) ρ t (Y t + ) Similar to subproblem optimality in programming optimality

119 Time consistency of in reinforcement learning Time Consistency via Iterated Risk Mappings Time consistent risk measures must be composed of iterated risk mappings (Roorda, Schumacher, and Engwerda 2005): µ 1, µ 2,..., µ t Dynamic risk measure: ρ t (X t + + X T ) = µ t (X t + µ t+1 (X t+1 + µ t+2 (x t+3 + ))) Each µ t : a coherent risk measure applied on subtree of filtration

120 Time consistency of in reinforcement learning Time Consistency via Iterated Risk Mappings Time consistent risk measures must be composed of iterated risk mappings (Roorda, Schumacher, and Engwerda 2005): µ 1, µ 2,..., µ t Dynamic risk measure: ρ t (X t + + X T ) = µ t (X t + µ t+1 (X t+1 + µ t+2 (x t+3 + ))) Each µ t : a coherent risk measure applied on subtree of filtration Markov risk measures for MDPs (Ruszczynski 2010)

121 Time consistency of in reinforcement learning Computing Time Consistent Risk Measure NOK 20 1/3 Pass test 1/3 OK 5 1/3 OK 5 ρ(x 2 Pass) = = 7.5

122 Time consistency of in reinforcement learning Computing Time Consistent Risk Measure NOK 10 1/3 Fail test 1/3 OK 5 1/3 OK 5 ρ(x 2 Fail) = = 7.5

123 Time consistency of in reinforcement learning Computing Time Consistent Risk Measure Pass 7.5 1/2 Take test 1/2 Fail 7.5 ρ(x 2 ) = ρ( 7.5) = 7.5 > 9

124 Time consistency of in reinforcement learning Computing Time Consistent Risk Measure 1/3 Pass test 1/3 1/3 1/3 Fail test 1/3 1/3 NOK 20 OK 5 OK 5 NOK 10 OK 5 OK 5 ρ(x 2 Pass) = = 7.5 ρ(x 2 Fail) = = 7.5 Pass 7.5 1/2 Take test 1/2 Fail 7.5 ρ(x 2 ) = ρ( 7.5) = 7.5 > 9 Consistently prefer option 1 throughout the execution

125 Time consistency of in reinforcement learning Approximating Inconsistent Risk Measures Time consistent risk measures are difficult to specify Approximate an inconsistent risk measure by a consistent one? Best lower bound: e.g. what is the best α 1, α 2 such that CV@R α1 (CV@R α2 (X)) CV@R α (X) for all X Best upper bound: e.g. what is the best α 1, α 2 such that CV@R α1 (CV@R α2 (X)) CV@R α (X) for all X (Iancu, Petrik, and Subramanian 2015)

126 Time consistency of in reinforcement learning Best Time Consistent Bounds Compare robust sets of consistent and inconsistent measures Main insight: need to compare down-monotone closures of robust sets

127 Time consistency of in reinforcement learning Time Consistent Bounds: Main Results Lower consistent bound: Uniformly tightest bound can be constructed in polynomial time Method: rectangularization Upper consistent bound: NP hard to even evaluate how tight the approximation is Approximation can be tighter than the lower bound

128 Time consistency of in reinforcement learning Planning with Time Consistent Risk Measures Stochastic dual dynamic programming (Shapiro 2012) Applied in reinforcement learning (Petrik and Subramanian 2012) Only entropic dynamically consistent risk measures are law invariant (Kupper and Schachermayer 2006)

129 Summary Outline Introduction to Risk Averse Modeling (Average) Value at Risk Coherent Measures of Risk Risk Measures in Reinforcement Learning Time consistency of in reinforcement learning Summary

130 Summary Risk Measures: Many Other Topics 1. Elicitation of risk measures 2. Estimation of risk measure from samples 3. Relationship to acceptance sets 4. Relationship to robust optimization

131 Summary Take Home Messages Coherent risk measures are a convenient and established risk aversion framework

132 Summary Take Home Messages Coherent risk measures are a convenient and established risk aversion framework Computations with coherent risk measure are more efficient than with utility functions

133 Summary Take Home Messages Coherent risk measures are a convenient and established risk aversion framework Computations with coherent risk measure are more efficient than with utility functions Risk measures (V@R, CV@R) are more intuitive than utility functions

134 Summary Take Home Messages Coherent risk measures are a convenient and established risk aversion framework Computations with coherent risk measure are more efficient than with utility functions Risk measures (V@R, CV@R) are more intuitive than utility functions Time consistency is important in dynamic settings, but can be difficult to achieve (open research problems)

135 Summary Take Home Messages Coherent risk measures are a convenient and established risk aversion framework Computations with coherent risk measure are more efficient than with utility functions Risk measures (V@R, CV@R) are more intuitive than utility functions Time consistency is important in dynamic settings, but can be difficult to achieve (open research problems) Risk measures are making inroads in reinforcement learning and artificial intelligence

136 Summary Thank you!!

137 Summary Bibliography I Andreev, Andriy, Antti Kanto, and Pekka Malo (2005). Closed-Form Calculation of Cvar. In: Sweedish School of Economics. Artzner, Philippe et al. (1999). Coherent Measures of Risk. In: Mathematical Finance 9, pp issn: doi: / Ben-Tal, Aharon and Marc Teboulle (2007). An Old-New Concept of Convex Risk Measures: The Optimized Certainty Equivalent. In: Mathematical Finance 17, pp url:

138 Summary Bibliography II Föllmer, Hans and Thomas Knispel (2011). Entropic Risk Measures: Coherence Vs. Convexity, Model Ambiguity and Robust Large Deviations. In: Stochastics and Dynamics 11.02n03, pp issn: doi: /S url: S Follmer, Hans and Alexander Schied (2011). Stochastic Finance: An Introduction in Discrete Time. 3rd. Walter de Gruyter. Friedman, Daniel et al. (2014). Risky Curves: On the Empirical Failure of Expected Utility.

139 Summary Bibliography III Iancu, Dan A, Marek Petrik, and Dharmashankar Subramanian (2015). Tight Approximations of Dynamic Risk Measures. In: Mathematics of Operations Research 40.3, pp issn: X. doi: /moor arxiv: url: abs/ /moor Kupper, Michael and Walter Schachermayer (2006). Representation results for law invariant time consistent functions. In: Mathematics and Financial Economics 16.2, pp url:

140 Summary Bibliography IV Nadarajah, Saralees, Bo Zhang, and Stephen Chan (2014). Estimation methods for expected shortfall. In: Quantitative Finance 14.2, pp issn: doi: / url: Petrik, Marek and Dharmashankar Subramanian (2012). An approximate solution method for large risk-averse Markov decision processes. In: Uncertainty in Artificial Intelligence (UAI). url: Rockafellar, R. Tyrrell and S. Uryasev (2000). Optimization of conditional value-at-risk. In: Journal of Risk 2, pp

141 Summary Bibliography V Roorda, Berend, Hans Schumacher, and Jacob Engwerda (2005). Coherent acceptability measures in multiperiod models. In: Mathematical Finance, pp url: Ruszczynski, Andrzej (2010). Risk-averse dynamic programming for Markov decision processes. In: Mathematical Programming B 125.2, pp issn: doi: /s url: Schoemaker, P.J.H. (1980). Experiments on Decisions under Risk: The Expected Utility Hypothesis.

142 Summary Bibliography VI Shapiro, A., D. Dentcheva, and A. Ruszczynski (2014). Lectures on stochastic programming: modeling and theory. P isbn: X. doi: Shapiro, Alexander (2012). Minimax and risk averse multistage stochastic programming. In: European Journal of Operational Research 219.3, pp issn: doi: /j.ejor url: Shapiro, Alexander, Darinka Dentcheva, and Andrzej Ruszczynski (2009). Lectures on Stochastic Programming. SIAM. isbn: