CS 188 Introduction to Artificial Intelligence Fall 2018 Note 4

Transcription

1 CS 188 Introduction to Artificil Intelligence Fll 2018 Note 4 These lecture notes re hevily bsed on notes originlly written by Nikhil Shrm. Non-Deterministic Serch Picture runner, coming to the end of his first ever mrthon. Though it seems likely he will complete the rce nd clim the ccompnying everlsting glory, it s by no mens gurnteed. He my pss out from exhustion or misstep nd slip nd fll, trgiclly breking both of his legs. Even more unlikely, literlly erth-shttering erthquke my spontneously occur, swllowing up the runner mere inches before he crosses the finish line. Such possibilities dd degree of uncertinty to the runner s ctions, nd it s this uncertinty tht will be the subject of the following discussion. In the first note, we tlked bout trditionl serch problems nd how to solve them; then, in the third note, we chnged our model to ccount for dversries nd other gents in the world tht influenced our pth to gol sttes. Now, we ll chnge our model gin to ccount for nother influencing fctor the dynmics of world itself. The environment in which n gent is plced my subject the gent s ctions to being nondeterministic, which mens tht there re multiple possible successor sttes tht cn result from n ction tken in some stte. This is, in fct, the cse in mny crd gmes such s poker or blckjck, where there exists n inherent uncertinty from the rndomness of crd deling. Such problems where the world poses degree of uncertinty re known s nondeterministic serch problems, nd cn be solved with models known s Mrkov decision processes, or MDPs. Mrkov Decision Processes A Mrkov Decision Process is defined by severl properties: A set of sttes S. Sttes in MDPs re represented in the sme wy s sttes in trditionl serch problems. A set of ctions A. Actions in MDPs re lso represented in the sme wy s in trditionl serch problems. A strt stte. Possibly one or more terminl sttes. Possibly discount fctor γ. We ll cover discount fctors shortly. A trnsition function T (s,,s ). Since we hve introduced the possibility of nondeterministic ctions, we need wy to delinete the likelihood of the possible outcomes fter tking ny given ction from ny given stte. The trnsition function for MDP does exctly this - it s probbility function which represents the probbility tht n gent tking n ction A from stte s S ends up in stte s S. CS 188, Fll 2018, Note 4 1

2 A rewrd function R(s,,s ). Typiclly, MDPs re modeled with smll "living" rewrds t ech step to rewrd n gent s survivl, long with lrge rewrds for rriving t terminl stte. Rewrds my be positive or negtive depending on whether or not they benefit the gent in question, nd the gent s objective is nturlly to cquire the mximum rewrd possible before rriving t some terminl stte. Constructing MDP for sitution is quite similr to constructing stte-spce grph for serch problem, with couple dditionl cvets. Consider the motivting exmple of rcecr: There re three possible sttes, S = {cool, wrm, overheted}, nd two possible ctions A = {slow, f st}. Just like in stte-spce grph, ech of the three sttes is represented by node, with edges representing ctions. Overheted is terminl stte, since once rcecr gent rrives t this stte, it cn no longer perform ny ctions for further rewrds (it s sink stte in the MDP nd hs no outgoing edges). Notbly, for nondeterministic ctions, there re multiple edges representing the sme ction from the sme stte with differing successor sttes. Ech edge is nnotted not only with the ction it represents, but lso trnsition probbility nd corresponding rewrd. These re summrized below: Trnsition Function: T (s,,s ) T (cool,slow,cool) = 1 T (wrm,slow,cool) = 0.5 T (wrm,slow,wrm) = 0.5 T (cool, f st,cool) = 0.5 T (cool, f st,wrm) = 0.5 T (wrm, f st,overheted) = 1 Rewrd Function: R(s,,s ) R(cool,slow,cool) = 1 R(wrm,slow,cool) = 1 R(wrm,slow,wrm) = 1 R(cool, f st,cool) = 2 R(cool, f st,wrm) = 2 R(wrm, f st, overheted) = 10 We represent the movement of n gent through different MDP sttes over time with discrete timesteps, defining s t S nd t A s the stte in which n gent exists nd the ction which n gent tkes t timestep t respectively. An gent strts in stte s 0 t timestep 0, nd tkes n ction t every timestep. The movement of n gent through MDP cn thus be modeled s follows: s s1 2 s2 3 s3... Additionlly, knowing tht n gent s gol is to mximize it s rewrd cross ll timesteps, we cn correspondingly express this mthemticlly s mximiztion of the following utility function: U([s 0, 0,s 1, 1,s 2,...]) = R(s 0, 0,s 1 ) + R(s 1, 1,s 2 ) + R(s 2, 2,s 3 ) +... CS 188, Fll 2018, Note 4 2

3 Mrkov decision processes, like stte-spce grphs, cn be unrveled into serch trees. Uncertinty is modeled in these serch trees with q-sttes, lso known s ction sttes, essentilly identicl to expectimx chnce nodes. This is fitting choice, s q-sttes use probbilities to model the uncertinty tht the environment will lnd n gent in given stte just s expectimx chnce nodes use probbilities to model the uncertinty tht dversril gents will lnd our gent in given stte through the move these gents select. The q-stte represented by hving tken ction from stte s is notted s the tuple (s,). Observe the unrveled serch tree for our rcecr, truncted to depth-2: The green nodes represent q-sttes, where n ction hs been tken from stte but hs yet to be resolved into successor stte. It s importnt to understnd tht gents spend zero timesteps in q-sttes, nd tht they re simply construct creted for ese of representtion nd development of MDP lgorithms. Finite Horizons nd Discounting There is n inherent problem with our rcecr MDP - we hven t plced ny time constrints on the number of timesteps for which rcecr cn tke ctions nd collect rewrds. With our current formultion, it could routinely choose = slow t every timestep forever, sfely nd effectively obtining infinite rewrd without ny risk of overheting. This is prevented by the introduction of finite horizons nd/or discount fctors. An MDP enforcing finite horizon is simple - it essentilly defines "lifetime" for gents, which gives them some set number of timesteps n to ccrue s much rewrd s they cn before being utomticlly terminted. We ll return to this concept shortly. Discount fctors re slightly more complicted, nd re introduced to model n exponentil decy in the vlue of rewrds over time. Concretely, with discount fctor of γ, tking ction t from stte s t t timestep t nd ending up in stte s t+1 results in rewrd of γ t R(s t, t,s t+1 ) insted of just R(s t, t,s t+1 ). Now, insted of mximizing the dditive utility U([s 0, 0,s 1, 1,s 2,...]) = R(s 0, 0,s 1 ) + R(s 1, 1,s 2 ) + R(s 2, 2,s 3 ) +... we ttempt to mximize discounted utility U([s 0, 0,s 1, 1,s 2,...]) = R(s 0, 0,s 1 ) + γr(s 1, 1,s 2 ) + γ 2 R(s 2, 2,s 3 ) +... Noting tht the bove definition of discounted utility function looks dngerously close to geometric series with rtio γ, we cn prove tht it s gurnteed to be finite-vlued s long s the constrint γ < 1 CS 188, Fll 2018, Note 4 3

4 (where n denotes the bsolute vlue opertor) is met through the following logic: U([s 0,s 1,s 2,...]) = R(s 0, 0,s 1 ) + γr(s 1, 1,s 2 ) + γ 2 R(s 2, 2,s 3 ) +... = t=0 γ t R(s t, t,s t+1 ) t=0 γ t R mx = R mx 1 γ where R mx is the mximum possible rewrd ttinble t ny given timestep in the MDP. Typiclly, γ is selected strictly from the rnge 0 < γ < 1 since vlues vlues in the rnge 1 < γ 0 re simply not meningful in most rel-world situtions - negtive vlue for γ mens the rewrd for stte s would flip-flop between positive nd negtive vlues t lternting timesteps. Mrkoviness Mrkov decision processes re "mrkovin" in the sense tht they stisfy the Mrkov property, or memoryless property, which sttes tht the future nd the pst re conditionlly independent, given the present. Intuitively, this mens tht, if we know the present stte, knowing the pst doesn t give us ny more informtion bout the future. To express this mthemticlly, consider n gent tht hs visited sttes s 0,s 1,...,s t fter tking ctions 0, 1,..., t 1 in some MDP, nd hs just tken ction t. The probbility tht this gent then rrives t stte s t+1 given their history of previous sttes visited nd ctions tken cn be written s follows: P(S t+1 = s t+1 S t = s t,a t = t,s t 1 = s t 1,A t 1 = t 1,...,S 0 = s 0 ) where ech S t denotes the rndom vrible representing our gent s stte nd A t denotes the rndom vrible representing the ction our gent tkes t time t. The Mrkov property sttes tht the bove probbility cn be simplified s follows: P(S t+1 = s t+1 S t = s t,a t = t,s t 1 = s t 1,A t 1 = t 1,...,S 0 = s 0 ) = P(S t+1 = s t+1 S t = s t,a t = t ) which is "memoryless" in the sense tht the probbility of rriving in stte s t time t + 1 depends only on the stte s nd ction tken t time t, not on ny erlier sttes or ctions. In fct, it is these memoryless probbilities which re encoded by the trnsition function: T (s,,s ) = P(s s,). Solving Mrkov Decision Processes Recll tht in deterministic, non-dversril serch, solving serch problem mens finding n optiml pln to rrive t gol stte. Solving Mrkov decision process, on the other hnd, mens finding n optiml policy π : S A, function mpping ech stte s S to n ction A. An explicit policy π defines reflex gent - given stte s, n gent t s implementing π will select = π(s) s the pproprite ction to mke without considering future consequences of its ctions. An optiml policy is one tht if followed by the implementing gent, will yield the mximum expected totl rewrd or utility. Consider the following MDP with S = {,b,c,d,e}, A = {Est,West,Exit} (with Exit being vlid ction only in sttes nd e nd yielding rewrds of 10 nd 1 respectively), discount fctor γ = 0.1, nd deterministic trnsitions: CS 188, Fll 2018, Note 4 4

5 Two potentil policies for this MDP re s follows: () Policy 1 (b) Policy 2 With some investigtion, it s not hrd to determine tht Policy 2 is optiml. Following the policy until mking ction = Exit yields the following rewrds for ech strt stte: Strt Stte Rewrd 10 b 1 c 0.1 d 0.1 e 1 We ll now lern how to solve such MDPs (nd much more complex ones!) lgorithmiclly using the Bellmn eqution for Mrkov decision processes. The Bellmn Eqution In order to tlk bout the Bellmn eqution for MDPs, we must first introduce two new mthemticl quntities: The optiml vlue of stte s, V (s) the optiml vlue of s is the expected vlue of the utility n optimlly-behving gent tht strts in s will receive, over the rest of the gent s lifetime. The optiml vlue of q-stte (s,), Q (s,) - the optiml vlue of (s,) is the expected vlue of the utility n gent receives fter strting in s, tking, nd cting optimlly henceforth. Using these two new quntities nd the other MDP quntities discussed erlier, the Bellmn eqution is defined s follows: V (s) = mx T (s,,s )[R(s,,s ) + γv (s )] s Before we begin interpreting wht this mens, let s lso define the eqution for the optiml vlue of q-stte (more commonly known s n optiml q-vlue): Q (s,) = s T (s,,s )[R(s,,s ) + γv (s )] Note tht this second definition llows us to reexpress the Bellmn eqution s V (s) = mxq (s,) which is drmticlly simpler quntity. The Bellmn eqution is n exmple of dynmic progrmming eqution, n eqution tht decomposes problem into smller subproblems vi n inherent recursive structure. We cn see this inherent recursion in the eqution for the q-vlue of stte, in the term CS 188, Fll 2018, Note 4 5

6 [R(s,,s ) + γv (s )]. This term represents the totl utility n gent receives by first tking from s nd rriving t s nd then cting optimlly henceforth. The immedite rewrd from the ction tken, R(s,,s ), is dded to the optiml rewrd ttinble from s, V (s ), which is discounted by γ to ccount for the pssge of the timestep in tking. Though in most cses there exists vst number of possible sequences of sttes nd ctions from s to some terminl stte, ll this detil is bstrcted wy nd encpsulted in single recursive vlue, V (s ). We cn now tke nother step outwrds nd consider the full eqution for q-vlue. Knowing [R(s,,s ) + γv (s )] represents the utility ttined by cting optimlly fter rriving in stte s from q-stte (s,), it becomes evident tht the quntity s T (s,,s )[R(s,,s ) + γv (s )] is simply weighted sum of utilities, with ech utility weighted by its probbility of occurrence. This is definitionlly the expected utility of cting optimlly from q-stte (s, ) onwrds! This completes our nlysis nd gives us enough insight to interpret the full Bellmn eqution - the optiml vlue of stte, V (s), is simply the mximum expected utility over ll possible ctions from s. Computing mximum expected utility for stte s is essentilly the sme s running expectimx - we first compute the expected utility from ech q-stte (s, ) (equivlent to computing the vlue of chnce nodes), then compute the mximum over these nodes to compute the mximum expected utility (equivlent to computing the vlue of mximizer node). One finl note on the Bellmn eqution its usge is s condition for optimlity. In other words, if we cn somehow determine vlue V (s) for every stte s S such tht the Bellmn eqution holds true for ech of these sttes, we cn conclude tht these vlues re the optiml vlues for their respective sttes. Indeed, stisfying this condition implies s S, V (s) = V (s). Vlue Itertion Now tht we hve frmework to test for optimlity of the vlues of sttes in MDP, the nturl follow-up question to sk is how to ctully compute these optiml vlues. To nswer this question, we need timelimited vlues (the nturl result of enforcing finite horizons). The time-limited vlue for stte s with time-limit of k timesteps is denoted V k (s), nd represents the mximum expected utility ttinble from s given tht the Mrkov decision process under considertion termintes in k timesteps. Equivlently, this is wht depth-k expectimx run on the serch tree for MDP returns. Vlue itertion is dynmic progrmming lgorithm tht uses n itertively longer time limit to compute time-limited vlues until convergence (tht is, until the V vlues re the sme for ech stte s they were in the pst itertion: s,v k+1 (s) = V k (s)). It opertes s follows: 1. s S, initilize V 0 (s) = 0. This should be intuitive, since setting time limit of 0 timesteps mens no ctions cn be tken before termintion, nd so no rewrds cn be cquired. 2. Repet the following updte rule until convergence: s S, V k+1 (s) mx T (s,,s )[R(s,,s ) + γv k (s )] s At itertion k of vlue itertion, we use the time-limited vlues for with limit k for ech stte to generte the time-limited vlues with limit (k + 1). In essence, we use computed solutions to subproblems (ll the V k (s)) to itertively build up solutions to lrger subproblems (ll the V k+1 (s)); this is wht mkes vlue itertion dynmic progrmming lgorithm. CS 188, Fll 2018, Note 4 6

7 Note tht though the Bellmn eqution looks essentilly identicl in construction to the updte rule bove, they re not the sme. The Bellmn eqution gives condition for optimlity, while the updte rule gives method to itertively updte vlues until convergence. When convergence is reched, the Bellmn eqution will hold for every stte: s S, V k (s) = V k+1 (s) = V (s). Let s see few updtes of vlue itertion in prctice by revisiting our rcecr MDP from erlier, introducing discount fctor of γ = 0.5: We begin vlue itertion by initiliztion of ll V 0 (s) = 0: cool wrm overheted V In our first round of updtes, we cn compute s S, V 1 (s) s follows: V 1 (cool) = mx{1 [ ], 0.5 [ ] [ ]} = mx{1, 2} = 2 V 1 (wrm) = mx{0.5 [ ] [ ], 1 [ ]} = mx{1, 10} = 1 V 1 (overheted) = mx{} = 0 cool wrm overheted V V Similrly, we cn repet the procedure to compute second round of updtes with our newfound vlues for CS 188, Fll 2018, Note 4 7

8 V 1 (s) to compute V 2 (s). V 2 (cool) = mx{1 [ ], 0.5 [ ] [ ]} = mx{2, 2.75} = 2.75 V 2 (wrm) = mx{0.5 [ ] [ ], 1 [ ]} = mx{1.75, 10} = 1.75 V 2 (overheted) = mx{} = 0 cool wrm overheted V V V It s worthwhile to observe tht V (s) for ny terminl stte must be 0, since no ctions cn ever be tken from ny terminl stte to rep ny rewrds. Policy Extrction Recll tht our ultimte gol in solving MDP is to determine n optiml policy. This cn be done once ll optiml vlues for sttes re determined using method clled policy extrction. The intuition behind policy extrction is very simple: if you re in stte s, you should tke the ction which yields the mximum expected utility. Not surprisingly, is the ction which tkes us to the q-stte with mximum q-vlue, llowing for forml definition of the optiml policy: s S, π (s) = rgmx Q (s,) = rgmx T (s,,s )[R(s,,s ) + γv (s )] s It s useful to keep in mind for performnce resons tht it s better for policy extrction to hve the optiml q-vlues of sttes, in which cse single rgmx opertion is ll tht is required to determine the optiml ction from stte. Storing only ech V (s) mens tht we must recompute ll necessry q-vlues with the Bellmn eqution before pplying rgmx, equivlent to performing depth-1 expectimx. Policy Itertion Vlue itertion cn be quite slow. At ech itertion, we must updte the vlues of ll S sttes (where n refers to the crdinlity opertor), ech of which requires itertion over ll A ctions s we compute the q-vlue for ech ction. The computtion of ech of these q-vlues, in turn, requires itertion over ech of the S sttes gin, leding to poor runtime of O( S 2 A ). Additionlly, when ll we wnt to determine is the optiml policy for the MDP, vlue itertion tends to do lot of overcomputtion since the policy s computed by policy extrction generlly converges significntly fster thn the vlues themselves. The fix for these flws is to use policy itertion s n lterntive, n lgorithm tht mintins the optimlity of vlue itertion while providing significnt performnce gins. Policy itertion opertes s follows: 1. Define n initil policy. This cn be rbitrry, but policy itertion will converge fster the closer the initil policy is to the eventul optiml policy. CS 188, Fll 2018, Note 4 8

9 2. Repet the following until convergence: Evlute the current policy with policy evlution. For policy π, policy evlution mens computing V π (s) for ll sttes s, where V π (s) is expected utility of strting in stte s when following π: V π (s) = s T (s,π(s),s )[R(s,π(s),s ) + γv π (s )] Define the policy t itertion i of policy itertion s π i. Since we re fixing single ction for ech stte, we no longer need the mx opertor which effectively leves us with system of S equtions generted by the bove rule. Ech V π i (s) cn then be computed by simply solving this system. Alterntively, we cn lso compute V π i (s) by using the following updte rule until convergence, just like in vlue itertion: V π i k+1 (s) s T (s,π i (s),s )[R(s,π i (s),s ) + γv π i k (s )] However, this second method is typiclly slower in prctice. Once we ve evluted the current policy, use policy improvement to generte better policy. Policy improvement uses policy extrction on the vlues of sttes generted by policy evlution to generte this new nd improved policy: π i+1 (s) = rgmx s T (s,,s )[R(s,,s ) + γv π i (s )] If π i+1 = π i, the lgorithm hs converged, nd we cn conclude tht π i+1 = π i = π. Let s run through our rcecr exmple one lst time (getting tired of it yet?) to see if we get the sme policy using policy itertion s we did with vlue itertion. Recll tht we were using discount fctor of γ = 0.5. We strt with n initil policy of Alwys go slow: cool wrm overheted π 0 slow slow Becuse terminl sttes hve no outgoing ctions, no policy cn ssign vlue to one. Hence, it s resonble to disregrd the stte overheted from considertion s we hve done, nd simply ssign i, V π i (s) = 0 for CS 188, Fll 2018, Note 4 9

10 ny terminl stte s. The next step is to run round of policy evlution on π 0 : V π 0 (cool) = 1 [ V π 0 (cool)] V π 0 (wrm) = 0.5 [ V π 0 (cool)] [ V π 0 (wrm)] Solving this system of equtions for V π 0(cool) nd V π 0(wrm) yields: We cn now run policy extrction with these vlues: cool wrm overheted V π π 1 (cool) = rgmx{slow : 1 [ ], f st : 0.5 [ ] [ ]} = rgmx{slow : 2, f st : 3} = f st π 1 (wrm) = rgmx{slow : 0.5 [ ] [ ], f st : 1 [ ]} = rgmx{slow : 3, f st : 10} = slow Running policy itertion for second round yields π 2 (cool) = f st nd π 2 (wrm) = slow. Since this is the sme policy s π 1, we cn conclude tht π 1 = π 2 = π. Verify this for prctice! cool wrm π 0 slow slow π 1 f st slow π 2 f st slow This exmple shows the true power of policy itertion: with only two itertions, we ve lredy rrived t the optiml policy for our rcecr MDP! This is more thn we cn sy for when we rn vlue itertion on the sme MDP, which ws still severl itertions from convergence fter the two updtes we performed. Summry The mteril presented bove hs much opportunity for confusion. We covered vlue itertion, policy itertion, policy extrction, nd policy evlution, ll of which look similr, using the Bellmn eqution with subtle vrition. Below is summry of when to use ech lgorithm: Vlue itertion: Used for computing the optiml vlues of sttes, by itertive updtes until convergence. Policy evlution: Used for computing the vlues of sttes under specific policy. Policy extrction: Used for determining policy given some stte vlue function. If the stte vlues re optiml, this policy will be optiml. This method is used fter running vlue itertion, to compute n optiml policy from the optiml stte vlues; or s subroutine in policy itertion, to compute the best policy for the currently estimted stte vlues. CS 188, Fll 2018, Note 4 10

11 Policy itertion: A technique tht encpsultes both policy evlution nd policy extrction nd is used for itertive convergence to n optiml policy. It tends to outperform vlue itertion, by virtue of the fct tht policies usully converge much fster thn the vlues of sttes. CS 188, Fll 2018, Note 4 11