Abstract Number: Risk-Reward Analysis in Stochastic Dynamic Programming

Transcription

1 Absrac Number: Risk-Rewar Analysis in Sochasic Dynamic Programming Preeam Basu Assisan Professor, Operaions Managemen Group Inian Insiue of Managemen Calcua Diamon Harbor Roa, Kolkaa , Inia el.: Suresh K. Nair Professor, Operaions an Informaion Managemen School of Business, Universiy of Connecicu, Sorrs, C suresh.nair@business.uconn.eu; el.: Absrac: Sochasic ynamic programming moels are exensively use for sequenial ecision making when oucomes are uncerain. hese moels have been wiely applie in ifferen business conexs such as invenory conrol, capaciy expansion, cash managemen, ec. he objecive in hese moels is o euce opimal policies base on expece rewar crieria. However, in many cases, managers are concerne abou he risks or he variabiliy associae wih a se of policies an no jus he expece rewar. Consiering risk an rewar simulaneously in a sochasic ynamic seing is a cumbersome ask an ofen ifficul o implemen for pracical purposes. Here we evelop heurisics ha sysemaically rack he variance an he average rewar for a se of policies, which are hen uilize o consruc efficien froniers. We apply our heurisics o he invenory conrol moel. Our heurisics perform creiably in proviing efficien risk-rewar curves. (Keywors: Sochasic Dynamic Programming, Risk-Rewar froniers, Mean-Variance raeoffs, Heurisics) 1

2 1: Inroucion Sochasic ynamic programming moels have been exensively use in solving problems ha have uncerainy as well as emporal aspecs. hese moels provie sequenial opimal ecisions where presen acions are aken in consieraion of fuure oucomes. he moels are characerize by ecision epochs, saes, acions, rewars an probabiliy funcions. A a specifie poin in ime, which is referre o as a ecision epoch, he ecision maker observes he sae of he sysem an base on ha chooses an acion. he acion prouces an immeiae rewar an he sysem moves o a ifferen sae in he nex ime-perio accoring o a probabiliy isribuion. Sochasic ynamic programming moels fin wie applicaions amongs various problems ha arise in ifferen secions of an organizaion such as invenory conrol, capaciy planning, averising expeniures, cash managemen o name jus a few. he usual opimizaion crieria for sochasic ynamic programming moels are o maximize he expece value of he limiing average rewar per uni ime or he sum of iscoune rewars in a finie or infinie-horizon seing. he focus of majoriy of hese moels is on opimizaion of average rewar crieria. However, using expece oal rewar crieria may yiel opimal policies ha are unaccepable o a risk-sensiive ecision maker. In many cases, managers are concerne abou he risks or he variabiliy associae wih a se of policies an no jus he expece rewar. Insea of ienifying one policy o achieve he sae objecive, managers are ofen inerese in obaining se of policies a ifferen levels of risk sensiiviy. hen base on heir risk-olerance level ecisions coul be aken ha provie he maximum expece rewars. Risk-rewar rae-offs are an essenial componen of many managerial ecisions. Variance of he oucomes abou he expece value is a wiely use measure of risk in porfolio heory. Invesors use variance o measure he risk of a porfolio of socks. he basic iea is ha variance is a measure of volailiy an he more a sock s reurns vary from he sock's average reurn, he more volaile is he sock. Porfolios of financial insrumens are chosen o minimize he variance of he reurns subjec o a level of expece reurn or vice versa o maximize expece reurn subjec o a level of variance of he reurn. his paraigm was firs inrouce by Markowiz s (1959) mean-variance analysis for which conribuion he was honore wih he Nobel Prize in Economics. In many operaional ecisions also managers are inerese in he mean-variance rae-offs. hey o no wan o focus on a single crierion raher hey wan o euce a se of policies ha give hem opimal soluions a ifferen levels of risk-olerance. In sar-up operaions i is exremely imporan o ascerain he risks associae wih a se of policies be is invenory sraegies or averising venures an no jus he expece reurn. Decisions such as capial invesmens or capaciy expansion ha have long-erm repercussions can be erimenal if he focus is only on he expece reurns. Risk-sensiive opimizaion in sochasic ynamic programming seing has been subjecs of some limie research. Markov Decision Process (MDP) moels are one of he mos wiely use sequenial ecision moels where he se of available acions, rewars an 2

3 ransiion probabiliies epen only on he curren sae an no on saes occupie in he pas. MDP moels have foun wie applicaions as hey can efficienly moel mos realisic sequenial ecision-making problems. he usual opimizaion crieria for MDPs are o maximize he expece value of he sum of iscoune rewars. here is a small sream of research ha employs risk-sensiive opimizaion crieria in MDPs. Whie (1974) use Lagrange mulipliers o inclue probabilisic consrains on variances in he opimizaion of mean performance. Howar an Maheson (1972) consiere he maximizaion of cerain equivalen rewar generae by MDP wih consan risk sensiiviy. Kawai (1987) analyze ranomize policies ha minimize he variance of he rewar in a ransiion among he policies ha lea o a cerain level of expece rewar for uniscoune MDPs. Filar e al. (1989) suie mean-variance rae-offs in MDPs wih boh expece limiing average an iscoune oal reurn crieria. hey formulae appropriae nonlinear programs in he space of sae-acion frequencies o analyze he corresponing variance penalize MDP moels. Baykal-Gursoy an Ross (1992) consiere ime-average MDPs wih finie sae an acion spaces an inrouce wo efiniions of variabiliy, viz., expece ime-average variabiliy an ime-average expece variabiliy. he auhors use hese crieria o penalize he variance in a sream of rewars. Sobel (1994) analyze uniscoune MDPs o generae Pareo opima in he sense of high mean an low variance of he saionary probabiliy isribuion of he rewar. Whie (1988) provies a survey on risk-sensiive opimizaion in finie MDPs. In he sream of lieraure iscusse above he variance-penalize MDPs are formulae as mahemaical programs wih linear consrains an nonlinear objecive funcion. However, as Filar e al. poin ou hese formulaions lea o formiable mahemaical ifficulies an are ifficul o implemen for pracical purposes. Here we come up wih a simple easy-o-use heurisic o consruc near-opimal riskrewar curves. Using our heurisic managers can come up wih policies ha lea o maximum average rewar a a paricular level of risk. he heurisic can be easily applie o any finiehorizon sochasic ynamic programming moel an gives a manager an useful ool wih which he can figure ou a se of ecisions ha maximizes profi a a cerain level of risk. In a sanar finie-horizon ynamic programming problem, informaion abou he expece rewar of a se of ecisions is carrie along an is use in solving he problem by backwar inucion algorihm. In our heurisic, we carry informaion abou he variance of a se of ecisions in aiion o expece rewars an his informaion abou he variance is uilize in selecing he opimal policies. Using our heurisic a risk-rewar curve can be easily rawn which can be use o ascerain risk exposure of a se of acions. Once he managemen ienifies he risk exposure of a se of policies hey can choose heir acions base on he level of risk hey are willing o accep. 2: Sochasic Dynamic Programming Formulaion Here we consier a finie-horizon sochasic ynamic programming moel wih ime-perios. A each ime-perio he sysem occupies a sae enoe by s. Le he se of possible sysem saes be S an he allowable acions a any sae be A s. A each ime-perio he ecision- 3

4 maker chooses an acion a( a A s ) an base on ha receives a rewar r ( s, a) an he sysem sae moves o he nex ime-perio accoring o he probabiliy isribuion p(. s, a ). Le he erminal rewar be given by r () s an be he iscoun facor. he following recursive funcional equaions specify he moel: Max r (, ) (, ) f () s = s a p j s a f1( j) aa s (1) he bounary coniion is given by: f () s = r () s (2) In mos of he sochasic ynamic programming moels he objecive is o maximize he expece sum of rewars. As we have poine ou before, in many cases, managers are inerese in no only he expece reurn bu also on he risk or he variabiliy of he reurns. Normally in a finie-horizon iscree-ime ynamic programming problem, he informaion abou he average rewar for a se of opimal acions is carrie a each sae. his informaion is hen use o compue he average rewar a a subsequen sae. Backwar inucion is an efficien soluion mehoology for hese kins of ynamic programming problems. I is he process of reasoning backwars in ime, from he en of a problem o eermine a sequence of opimal acions. I procees by firs choosing he opimal acion a he las ecision epoch or he bounary of he problem. hen using his informaion, he opimal acion a he secon-o-las ime of ecision is eermine. his process coninues backwars unil one has eermine he bes acion for every possible siuaion a every poin in ime. In our heurisics, we use he backwar inucion algorihm o solve he ynamic programming moel an in aiion o carrying informaion abou he average rewar, we also keep rack of he variance of a se of policies a each sae. his variance informaion is uilize o obain risk-rewar rae-offs. Nex, we explain he wo heurisics in eail ha provie us wih an easy-o-implemen mechanism o consruc risk-rewar curves. 3: Heurisic for Risk-Rewar Curves Here we evelop a heurisic uilizing he variance informaion a each sae for he possible acions. In he firs heurisic, which we call Riskrackr_MaxEfficiency, he focus is on ienifying soluions ha have high values of he raio of expece rewar o variance. 3.1: he Riskrackr_MaxEfficiency Heurisic In he Riskrackr_MaxEfficiency heurisic our focus is on ienifying soluions ha have high values of he raio of expece rewar o variance an hen picking he soluions ha have high expece rewar values. he average rewar a any sae, s, for a se of policies is capure by he funcional value f () s. Here we nee aiional informaion abou he risk associae wih 4

5 hose se of policies. We efine risk as he variance of he possible oucomes a any sae. We enoe he variance of he possible oucomes for a se of acion a, a any sae s, by (). a () s is given as follows: a s 2 2 a ( s) p( j s, a) v 1( j) p( j s, a)( r ( s, a) f1( j)) 1 1 p( j s, a)( r ( s, a) f ( j)) ( s) (3) where, 1 ( s ) r ( s, a ) p ( j s, a ) f ( j ) he variance a any sae s, is enoe by v () s an i is he variance corresponing o he opimal acion * a. In he Riskrackr_MaxEfficiency heurisic, we use he backwar inucion algorihm o solve he ynamic programming moel an in aiion o carrying informaion abou he average rewar, we also keep rack of he variance of a se of policies a each sae. A, i.e., a he bounary, he variance a any sae is zero since no acions are aken. hen a any preceing ecision epoch ( 1, 2,...,0) he variance a any sae for each of he possible acions is calculae base on equaion (3). his variance informaion is hen use o evaluae he opimal acion a each sae. Nex, we explain how we eermine he opimal acion * a a each ecision epoch. A each sae of he problem, we evelop a raio base on average rewar o variance for each of he possible acions. We call his raio crieria an enoe i by Ka () s. K () s a 1 r ( s, ) p( j s, a) f ( j) () s a (4) We hen solve he ynamic programming moel repeaely for j 1,1 y,1 2 y,..., X where j is he collecion of acions ha give he op j crieria values from he se of all possible acions a any sae enoe by A. Here X is he oal number of elemens in he acion se an y is he sep size use in incremening j. A any sae we rank all he acions escening orer 5

6 base on he crieria value using (4). We hen pick he op j acions. hen ou of he op j acions, we pick he acion ha gives he maximum rewar. We firs solve he moel for j 1, i.e., a each sae we only consier he acions ha give he highes crieria value an hen from hose acions pick he one ha gives he maximum average rewar. As we solve he moels repeaely for ifferen values of j saring from j 1, we keep rack of he average rewar earne in each sae an sore ha informaion in [] G s. For a subsequen value of j,( j j y, j 2 y,...), we check ha he average rewar earne a each sae is a leas as large as ha has been earne for a lower value of j. he average rewar an variance for he opimal acions are sore in () for a paricular value of, j f0 () s an v0 () f s an v () s respecively. A 0, s gives he average rewar an variance for a sequence of opimal acions execue a each of he ime perios base on he op j crieria values. We sore he average rewar an variance value for each j in E 1 ( j) an Var 1 ( j ) respecively. We coninue he above process unil j X. E 1 ( j) an Var 1 ( j ) j 1,1 y,..., X, obaine from he Riskrackr_MaxEfficiency heurisic is o consruc he efficien fronier. Solving he sochasic ynamic programming moel repeaely for j 1,1 y,1 2 y,..., X using he Riskrackr_MaxEffciency heurisic gives us a se of policies ha consier boh he expece rewar an he associae variance. Since a each sae we pick acions ha lea o highes value of he raio of average rewar o variance, we are in urn consiering he acions ha are he mos efficien a each sae. hen we are picking he acion ha has he highes average rewar. herefore, ou of all he possible acions we are picking he ones ha are he mos efficien, i.e., he ones ha give he highes average rewar wih respec o variance. Hence as we increase j, we are increasing he risk exposure of he moel an a a higher risk-exposure we are again picking he acion ha has he maximum average rewar. his process in urn leas o oucomes saring from very low risk-exposure an increasing o he maximum riskexposure. Nex, we apply Riskrackr_MaxEfficiency o one of he mos wiely use applicaions of sochasic ynamic programming moeling, namely, he invenory conrol moel o check he performance of our heurisics in consrucing efficien risk- rewar curves. We o so by comparing he risk-rewar fronier obaine by our heurisic agains he risk-rewar soluions from all enumerae oucomes of he invenory moels. 3.2: Applicaion Riskrackr_MaxEfficiency: Invenory Conrol Moel Sochasic ynamic programming has been exensively use in invenory conrol moels. hese moels are wiely applicable in a variey of business problems such as eermining orer-sizes in a supply chain se-up an even for managing cash balances. Nex we presen a simplifie version of ha moel. 6

7 Each ime perio he manager looks a he curren invenory or sock-on-han an ecies wheher or no o orer aiional sock from a supplier. We look a a finie-horizon moel wih ime-perios. he sae of he sysem a any ime-perio is given by z which enoes he sock-on-han. We assume eman for he prouc a each ime-perio,, is ranom an inepenen an follows a probabiliy isribuion p ( ). Le q be he unis of sock orere a each ime-perio an ˆq be he maximum possible orer-size. he uni selling price is given by s an he uni purchase cos of he prouc is u an he uni holing cos is h.he salvage value for uni unsol invenory is given by. f () z is he maximal ne presen value of being in sae, z, when opimal acions are aken in each ime perio from perio following recursive funcional equaions specify he moel: o. he f () z = Max p( )( s uq hz f1( z q)) q (5) where Minimum (, z) an z Maximum (0, z ) he bounary coniion is given by: f () z = z (6) Nex we explain he seps in applying he Riskrackr_MaxEfficiency o he invenory conrol moel (5)-(6). A each sae of he sysem, we nee o calculae he average rewar an he variance for each orer-size. he average rewar a any sae, z, is capure by he funcional value f () z. he variance of he possible oucomes for any orer-size q, is given by () which is calculae as follows using equaion (3): q z q 1 1 ( z) p( ) v ( z q) p( )( s uq hz f ( z q)) 1 1 p( )( s uq hz f ( z q)) ( z) 2 (7) where, 1 ( z ) s uq hz p ( ) f ( z q ) A each sae of he problem, we also nee o calculae he crieria value for any orersize which is enoe i by K () z an is obaine using equaion (4) as follows: q 7

8 K () z q 1 p( )( s uq hz f ( z q)) () z q (8) We solve he invenory conrol moel repeaely for j 1,1 y,1 2 y,..., qˆ where j is he collecion of acions ha give he op j crieria values from he se of all possible orer-sizes a any sae. Afer ranking he orer-sizes in a escening orer base on he crieria value using (8) we pick he op j acions. hen ou of he op j acions, we pick he acion ha gives he maximum rewar. As we solve he moels repeaely for ifferen values of j saring from j 1, we keep rack of he average rewar earne in each sae an sore ha informaion in [] G z. For a subsequen value of j,( j j y, j 2 y,...), we check ha he average rewar earne a each sae is a leas as large as ha has been earne for a lower value of j. he average rewar an variance for he opimal acions are sore in () for a paricular value of, j f0 () z an v0 () f z an v () z respecively. A 0, z gives he average rewar an variance for a sequence of opimal acions execue a each of he ime perios base on he op j crieria values. We sore he average rewar an variance value for each j in E 1 ( j) an Var 1 ( j ) respecively. We coninue he above process unil j qˆ. Now, we nee o check he performance of our heurisic in consrucing efficien riskrewar curves. We o so by comparing he risk-rewar fronier obaine by our heurisic agains he risk-rewar associae wih all enumerae oucomes of he invenory conrol moel. For he performance evaluaion, we picke a 6-perio problem wih maximum orer-size equal o 20 in each sae. Here he uni cos u $3an he uni selling price s $10. We assume ha he salvage value for unsol invenory a he bounary is zero. We compare he performance of our heurisic agains he efficien fronier of he enumerae soluions for ifferen eman isribuions. We use iscreize Bea isribuion o generae ifferen kins of eman isribuion. able 1 presens he performance evaluaion resuls of he efficien fronier consruce by Riskrackr_MaxEfficiency compare o he enumerae efficien fronier for various eman isribuions. Here we look a he Mean Percenage Deviaion of he Riskrackr_MaxEfficiency efficien fronier from he enumerae efficien fronier. We also analyze he Hi-Rae of he Riskrackr_MaxEfficiency efficien fronier. Hi-Rae of he Riskrackr_MaxEfficiency heurisic is efine as he percenage of risk-rewar soluions wih eviaions from he enumerae efficien fronier. We calculae he rewar a various levels of risk along he wo efficien froniers an compue he Mean Percenage Deviaion an Hi-Rae of he Riskrackr_MaxEfficiency heurisic a <1%, <2%, <3%,<5% an <10% eviaions from he enumerae efficien fronier. 8

9 able 1: Performance of he Heurisics Efficien Froniers agains Enumerae Efficien Froniers for he Invenory Conrol Moel Hi Rae (% occurrence of soluions wih Bea Parameers Shape of he Mean eviaions from enumerae eff fronier) Densiy %Deviaion Funcion a <1% <2% <3% <5% <10% 5 10 Righ Skewe 0.00% 100.0% 100.0% 100.0% 100.0% 100.0% 1 1 Uniform 0.02% 99.2% 99.2% 100.0% 100.0% 100.0% 5 5 Bell-Shape 0.02% 99.2% 99.6% 99.6% 100.0% 100.0% 10 5 Lef Skewe 0.04% 99.2% 99.2% 100.0% 100.0% 100.0% We fin ha on an average he Mean Percenage Deviaion beween he heurisics an he enumerae efficien froniers were less han 0.04% an he Hi-Rae of he heurisics efficien fronier is 100% a he 5% level. Base on he above accuracy measures we can conclue ha our heurisics performs creiably in consrucing efficien froniers. 4: Conclusions an Fuure Research Direcions In his paper, we come up wih a simple easy-o-use heurisic o consruc near-opimal riskrewar curves in sochasic ynamic programming moels. In many cases, managers are concerne abou he risks or he variabiliy associae wih a se of policies an no jus he expece rewar. Insea of ienifying one policy o achieve he sae objecive, managers are ofen inerese in obaining se of policies a ifferen levels of risk sensiiviy. Using our heurisic a risk-rewar curve can be easily rawn which can be use o ascerain risk exposure of a se of acions. he heurisic can be easily applie o any finie-horizon sochasic ynamic programming moel. he heurisic gives a manager a useful ool wih which he can figure ou a se of ecisions ha maximizes reurn a a cerain level of risk. Here we have applie our heurisic o a finie-horizon sochasic ynamic programming moel, namely, he invenory conrol moel. here are oher imporan problems ha fall uner he umbrella of opimal sopping rule moels such as equipmen replacemen, opions exercising an secreary problems. Coming up wih efficien heurisics ha can be use o consruc risk-rewar curves for hese moels coul be an ineresing fuure research eneavor. Risk-sensiive opimizaion in sochasic ynamic seing is an imporan area ha has seen very limie research. In he fuure, we coul also look a eveloping heurisics ha provie riskrewar soluions for infinie horizon sochasic ynamic programming moels. 9

10 References: Baykal-Gursoy, M., K. W. Ross Variabiliy Sensiive Markov Decision Processes, Mahemaics of Operaions Research, 17(3), Howar, R. A., J. E. Maheson Risk-Sensiive Markov Decision Processes, Managemen Science, 18(7), Chen, F., A. Feergruen Mean-Variance Analysis of Basic Invenory Moels, Working Paper, Columbia Universiy, New York. Chen, X., M. Sim, D. Simchi-Levi, P. Sun Risk Aversion in Invenory Managemen, Operaions Research, 55(5), Filar, J. A., L. C. M. Kallenberg, H. M. Lee Variance-Penalize Markov Decision Processes, Mahemaics of Operaions Research, 14(1), Heyman, D., M.Sobel Sochasic Moels in Operaions Research, Vol II: sochasic Opimizaion, Mcgraw-Hill, New York. Kawai, H A Variance Minimizaion Problem for a Markov Decision Process, European Journal of Operaional Research, 31(1), Markowiz, H Porfolio Selecion: Efficien Diversificaion of Invesmen, Cowls Founaion Monograph 16, Yale Universiy Press, New Haven. Marinez-e-Albeniz, V., D. Simchi-Levi Mean-Variance rae-offs in Supply Conracs, Naval Research Logisics, 53(7), Sobel,M he Variance of Discoune Markov Decision Processes, Journal of Applie Probabiliy, 19(4), Sobel,M Mean-Variance raeoffs in an Uniscoune MDP, Operaions Research, 42(1), Whie, D. J Mean, variance, an probabilisic crieria in finie Markov ecision processes: A review, Journal of Opimizaion heory an Applicaions, 56(1), Wu, C., Y. Lin Minimizing Risk Moels in Markov Decision Processes wih Policies Depening on arge Values, Journal of Mahemaical Analysis an Applicaions, 231(1),