Multi-Time-Scale Decision Making for Strategic Agent Interactions

Transcription

1 Proceedings of he 2010 Indusrial Engineering Research Conference A. Johnson and J. Miller eds. Muli-Time-Scale Decision Making for Sraegic Agen Ineracions Chrisian Wernz Virginia Tech Blacksburg VA Abhiji Deshmukh Texas A&M Universiy College Saion TX Absrac We model a muli-ime-scale sysem in which wo hierarchically ineracing agens make decisions under uncerainy. The higher level agen makes decisions less frequenly han is lower level counerpar. We combine Markov decision processes and game heory o model he sraegic muli-ime-scale agen ineracion. In our analysis opimal decision sraegies and informaion needs for subordinae agens are deermined. Furhermore we calculae a ime-invarian incenive level ha encourages cooperaive behavior by subordinae agens in all periods. Resuls are represened as analyic soluions providing insighs on he influence of curren and fuure daa on agens decisions. Keywords Decision heory muliscale decision making Markov decision processes game heory incenive sysems 1. Inroducion We model a hierarchically ineracion beween wo agens agen (supremal) and agen (infimal). Agen works for agen i.e. agen s performance affecs he success of is superior. To moivae good work and hereby increase he likelihood of success for iself agen offers agen a share of is reward. Provided ha he incenive is sufficienly large agen will selec an acion ha suppors agen in reaching is preferred oucome. Wihou his incenive paymen agen would have chosen an alernaive acion ha saisfied agen s self ineres bu would have hindered agen in aaining is goal. Good decision oucomes (saes) in he curren period lead o higher chances of success for boh agens in fuure periods i.e. high reward saes in he curren period resul in higher ransiion probabiliies o high reward saes in he nex period. A characerisic of muli-ime-scale sysems [1-3] is ha higher level agens make decision less frequenly han lower level agens. We incorporae he muli-ime-scale propery by having agen commi o one ime-invarian incenive level and all fuure acions in he firs period of he decision horizon. An example ha illusraes he model is provided in he following paragraphs. We consider he decision-making problem of a producion planner and a raw maerial buyer who is he planner s immediae subordinae. The planner is represened by agen and he buyer by agen in our model. The main goal of he planner is o fulfill cusomer orders on ime. The planner conrols a machine which ransforms raw maerials ino he final produc for he cusomer. Wih he seing of he machine he planner influences producion ime and raw maerial consumpion. The planner can choose beween a faser bu less reliable machine seing and a slower bu more reliable seup. The less reliable seup resuls in more scrap pars hus requiring more raw maerials o offse he addiional requiremens. Ye even including rejecs he fas seing sill has a higher effecive oupu rae han he slower one. Hence he chances of meeing he cusomer s deadline are higher when choosing he fas seing. Furhermore he planner s reward only depends on wheher producion is compleed prior o he cusomer s deadline; oher operaional goals such as maerial consumpion machine ime and usage have only a small negligible effec on he planner s reward. Producion and failure rae of boh machine seings are probabilisic so ha meeing he deadline or being delayed is possible wih boh machine seings however wih differen likelihoods.

2 Wernz Deshmukh The buyer has o decide on he quaniy of he raw maerials ordered which will lead o an eiher high or low invenory sae. The raw maerials serve as inpus o he machine under he planner s conrol. The buyer s decision srucure is similar o ha of he planner because decisions and consequenial saes are linked probabilisically. The decision for a high (low) order quaniy resuls wih greaer probabiliy in a high (low) receiving order bu boh saes are possible given any one decision. The buyer s cos or (negaive) reward is he incurred maerial handling and invenory cos. In addiion he reward of he buyer is affeced by he planner s performance. The planner gives a share of is reward o he buyer. This share is deermined by he planner and chosen such ha he buyer prefers and decides o ake he cooperaive acion which is placing a large order. The process repeas for every period. The ransiion probabiliies o he nex sae depend on he acion bu also on he prior saes. An on ime producion in a prior period makes on ime producion in he fuure more likely. The same holds for he invenory saes: high invenory saes in he curren period mos likely resul in high invenory saes in he nex period. Decisions a higher levels in an organizaion are ypically made less frequenly. This fac is recognized in our model by having he planner commi o one reward share percenage for he enire planning horizon. The planner canno adap he incenive in each period and is hus ime invarian. The planner also commis o all fuure machine seings in he firs period. To make an opimal decision in a muli-period decision problem agens mus consider he enire ime horizon and collec necessary informaion abou curren and fuure periods. The complexiy of he agen s decision problem furher increases wih muliple inerdependen decision makers ha influence each oher s rewards and probabiliies of decision oucomes. Muliscale decision heory [4 5] fuse he emporal and organizaional dimensions of muliperiod muli-agen sysems. A muliscale decision making model can deermine opimal courses of acion for agens ineracing over muliple emporal and organizaional scales. We exend a one-period hierarchical agen ineracion model [6] o a muli-ime-scale model. Previously Wernz and Deshmukh [7] had exended he one-period model o muliple periods however wihou incorporaing muli-ime-scale properies. 2. Model and Noaion In his secion we inroduce he noaion and provide he mahemaical deails of he muli-ime-scale model. Agens make decisions a he beginning of a period; his poin in ime is referred o as he decision epoch T. A every decision epoch 1... N 1 a. Decisions are no made a he final decision each agen carries ou an acion epoch N. Depending on is curren sae and decision each agen moves o a sae s 1 S wih probabiliy p s 1 s a. Each agen receives a reward ha depends on he sae a he nex decision epoch r s 1. This process repeas for every period 1... N 1. The acion spaces for agens and are denoed by A a1 a2 A a1 a2 and heir sae spaces by S s1 s2 S s1 s2 period each agen has a disinc se of acions and saes. The iniial rewards for agen and for 1... N 1 are r s In every r s2 1 2 (1) r s1 1 1 r s (2) The iniial sae-dependen ransiion probabiliies for agen are p s s a i i p s s a 1 1 i 2 i.2 p s2 1 si a1 1 i.11 (3) p s2 1 si a2 i.2 (4) wih i N 1 and 0 i. m 1 for m 12. The ransiion probabiliies for agen are denoed equivalenly wih superscrip insead of. The final ransiion probabiliy of agen is affeced by he sae agen moves o. We model his influence using an addiive influence funcion f : p s s a s p s s a f s s (5) final k 1 i m l 1 k 1 i m k 1 l 1

3 Wernz Deshmukh for i j k l m N 1. We choose he influence funcion o be a consan and define i as if c k l f sk 1 sl 1 wih c 0. (6) c if k l Consan c is referred o as he change coefficien. Since probabiliies canno be negaive nor exceed uniy p final 0 1 mus hold. The meaning and impac of he chosen change coefficien srucure is as follows: for 2... N sae s1 increases he probabiliy of sae s1 and consequenially reduces he probabiliy of sae s 2. The probabiliies change in opposie direcion for sae s 2 : sae s2 becomes more likely and sae s1 less likely. This effec on ransiion probabiliies applies o siuaions where agen s sae suppors or hinders agen reaching a specific sae. The final rewards of agen is affeced by agen s sae dependen reward of which agen receives a proporional share b 1 N (read: 1 hrough N). The share coefficien b1 N is idenical for all periods hus he index. The final reward in period for agen is r s s r s b r s b. (7) final k 1 l 1 l 1 1 N k 1 l 1 N k Agen s iniial reward is reduced by he reward share given o agen resuling in is final reward Figure 1 provides a graphical summary of he model. 1 r s b (8) final k 1 1 N k Saes Acions Transiions Agen s11 s21 s12 s22 s13 s23 Influence on rewards and ransiion probabiliies Agen s11 s21 s12 s22 s13 s23 Time periods =1 =2 Figure 1: Schemaic model represenaion The following furher assumpions on he model s daa are made: (9) 1 m n. (10) 2 Inequaliies in (9) express ha agen prefers sae s 1 1 over s2 1 and agen reversely prefers s 2 1 over s1 1 a leas iniially. Expression (10) saes ha an acion is linked o he sae wih he same index; in oher words here is a corresponding acion for every sae which is he mos likely consequence of he respecive acion. This resricion circumvens redundan cases in he analysis bu does no limi he generaliy of he model.

4 Wernz Deshmukh 3. Analysis Agens make heir decision wihou knowing he oher agen s decision. We assume ha agens are raional and risk neural. Consequenially we can analyze he agen ineracion using game heory and deermine Nash equilibria ha maximize agens rewards aking ino accoun he oher agen s acions. The expeced reward for agen for a given period is 2 2 final i j m n final k 1 l 1 l 1 j n final k 1 i l 1 m E r s s a a r s s p s s a p s s s a. (11) k 1 l1 The expeced reward for agen can be calculaed similarly. Agens seek o maximize heir expeced cumulaive rewards r final(1) i.e. he expeced value of he sum of all rfinal for 1... N 1 given agens iniial saes in 1. The cumulaive reward rfinal( ) for agen from period unil he end of he decision horizon is N 1 r s s r s s for 1... N 1. (12) final( ) i1 j1 final k 1 l 1 1 To deermine he expeced cumulaive reward we apply he backward inducion principle [8 9]. For each period we recursively deermine: wih i j m n final final i j m n E r s s a a E r s s a a 2 2 k 1 l 1 final 1 p s s s s a a E r s s k 1 l 1 i j m n k 1 l p s s s s a a p s s a p s s s a k l i j m n l j n final k i l m and E r sk N sl N 0 for i j m n N 1. (14) final N (13) To derive E r s s final i j from final i j m n E r s s a a agens deermine heir Nash equilibrium acion pair am an. To find he Nash equilibrium we deermine he condiion under which agen swiches from is iniially preferred acion o he cooperaive acion. Thus we evaluae final( ) i j 1 1 final( ) i j 1 2 E r s s a a E r s s a a (15) for i j 1 2. For he las epoch N 1 a which a decision is made solving (15) resuls in 2 N 1 1 N 1 [ N 1] 2cN 1 1 N 1 2 N 1 b. (16) We are inroducing an auxiliary share coefficien b [ ]. This share coefficien does no represen he ime invarian share coefficien b1 N agen chooses o encourage cooperaive acions in all periods. For he remainder of he paper we use a more compac noaion for he model s daa: (17) (18) Coninuing wih he backward inducion and evaluaing (13) and (15) for N 2 we ge 1 N 1 N 1 N 2 b[ N 2]. (19) 2 c c N 1 N 1 N 1 N 2 N 1 N 1 N 2 For N 3 he ransiion funcion coninues o furher increase in size. Thus we presen numeraor and denominaor separaely: 1 numn 3 b[ N 3] (20) 2 denn 3 where

5 Wernz Deshmukh num (21) N 3 N 1 N 1 N 2 N 2 N 2 N 3 denn 3 cn 3 N 1 N 1 N 2 N 2 N 2 N 3 N 2 cn 1 N 1 N 1 cn 2 N 1 N 1 N 2. (22) A recursive formula can be derived ha deermines he agens auxiliary share coefficiens b [ ]. The recursive formula describes how numeraor and denominaor evolve as he inducion process proceeds backwards in ime. The auxiliary share coefficiens are 1 num b[ ] for 1... N 1 (23) 2 den where num (24) num N 1 N 1 num for 1... N 2 ; (25) 1 1 den c (26) N 1 N 1 N 1 den den c N N N for 1... N 2. (27) In a final sep agen chooses b1 N by combining all inervals described in (23). Agen will offer an incenive o agen ha induces cooperaive behavior. Condiion (23) describes how large his incenive has o be and agen who has o pay for he incenive will choose he smalles possible value. We denoe wih b [ ] he values ha solve (23) as equaliies. Share coefficiens b [ ] represen agen s cos minimal parameers in period. Among he values of b [ ] agen chooses he period wih larges coefficien o ensure ha agen will choose he cooperaive acion a1 in every period: b max b. (28) 1 N 1... N 1 [ ] We assume ha he paricipaion condiion for agen is saisfied i.e. he daa of he model are such ha cooperaion in every period is preferred by agen over he non-cooperaive soluion. Wihou he assumpion abou agen s paricipaion condiion resul (28) does no necessarily hold. Agen could ge a higher expeced reward by choosing a share coefficien smaller han he maximum. A smaller share coefficien leads o periods in which agen chooses he non-cooperaive acions. The opimal soluion could be a share coefficien value ha induces cooperaive behavior in some periods and non-cooperaive behavior in ohers. Deermining he opimal share coefficien wihou he assumpion abou he paricipaion condiion is lef for fuure research. In conclusion agen commis o opimal share coefficien b 1N and chooses acions a1 for 1... N 1. Agen makes his choice in he firs period and informs agen of is selecion of share coefficien Agen will respond by choosing he cooperaive acions a1 in every epoch 1... N. 4. Conclusions For he described influence and incenive srucures we showed ha agens can deermine incenives and opimal decision sraegies for a muli-period ime horizon via closed-form analyical soluion. In a given period all fuure daa is necessary o make an opimal decision however he daa can be communicaed in aggregae form since only reward and ransiion probabiliy differences play a role. Fuure research will exend he wo-agen model o a muliorganizaional-scale sysem. The resuls inform managers how o design incenive and influence srucures as well as daa managemen sysems. b 1 N. References 1. Jacobson M. N. Shimkin and A. Shwarz Markov Decision Processes wih Slow Scale Periodic Decisions. Mahemaics of Operaions Research (4): p

6 Wernz Deshmukh 2. Hauskrech M. e al. Hierarchical Soluion of Markov Decision Processes Using Macro-Acions. in Proceedings of he Foureenh Conference on Uncerainy in Arificial Inelligence Universiy of Wisconsin Business School Madison WI. 3. Chang H.S. e al. Muliime Scale Markov Decision Processes. IEEE Transacions on Auomaic Conrol (6): p Wernz C. and A. Deshmukh Muliscale Decision-Making: Bridging Organizaional Scales in Sysems wih Disribued Decision-Makers. European Journal of Operaional Research (3): p Wernz C. Muliscale Decision Making: Bridging Temporal and Organizaional Scales in Hierarchical Sysems in Mechanical and Indusrial Engineering Disseraion Universiy of Massachuses Amhers. 6. Wernz C. and A. Deshmukh Decision Sraegies and Design of Agen Ineracions in Hierarchical Manufacuring Sysems. Journal of Manufacuring Sysems (2): p Wernz C. and A. Deshmukh. An Incenive-Based Muli-Period Decision Model for Hierarchical Sysems. in Proceedings of he 3rd Annual Conference of he Indian Subconinen Decision Sciences Insiue Region (ISDSI) Hyderabad India. 8. Bellman R.E. Dynamic Programming Princenon NJ: Princeon Universiy Press. 9. Puerman M.L. Markov Decision Processes: Discree Sochasic Dynamic Programming New York: Wiley.