arxiv: v1 [cs.lg] 23 Jan 2019

Transcription

1 Robust temporl difference lerning for criticl domins rxiv: v1 [cs.lg] 23 Jn 2019 Richrd Klim University of Liverpool, UK Michel Kisers Centrum Wiskunde & Informtic, NL Jnury 24, 2019 Abstrct Dn Bloembergen Centrum Wiskunde & Informtic, NL Krl Tuyls DeepMind & University of Liverpool, UK We present new Q-function opertor for temporl difference (TD) lerning methods tht explicitly encodes robustness ginst significnt rre events (SRE) in criticl domins. The opertor, which we cll the κ-opertor, llows to lern sfe policy in model-bsed fshion without ctully observing the SRE. We introduce single- nd multi-gent robust TD methods using the opertor κ. We prove convergence of the opertor to the optiml sfe Q-function with respect to the model using the theory of Generlized Mrkov Decision Processes. In ddition we prove convergence to the optiml Q-function of the originl MDP given tht the probbility of SREs vnishes. Empiricl evlutions demonstrte the superior performnce of κ-bsed TD methods both in the erly lerning phse s well s in the finl converged stge. In ddition we show robustness of the proposed method to smll model errors, s well s its pplicbility in multi-gent context. Keywords: reinforcement lerning; robust lerning; multi-gent lerning 1 Introduction Mny criticl systems exhibit globl system dynmics tht re highly sensitive to the locl performnce of individul components. This holds for exmple for (ir) trffic nd trnsport networks, communiction networks, security systems, nd (smrt) power grids [5, 11, 14, 22]. In ech cse, the filure of or mlicious ttck on smll set of nodes my led to knock-on effects tht cn potentilly destbilise the whole system. Moreover, innovtions in criticl systems my introduce dditionl vulnerbilities to such ttcks: e.g., in smrt grids communiction chnnels re needed for distributed intelligent energy mngement strtegies, while simultneously forming potentil trget tht could compromise sfety [32]. Our reserch is motivted precisely by the need for sfety in these criticl systems, which cn be chieved by building in robustness ginst rre but significnt devitions cused by one or more system components filing or being compromised in n ttck. In this rticle we present new pproch for lerning policies in such systems tht re sfe nd robust ginst chosen scenrio of potentil ttcks or filures. We ccomplish this by introducing This pper will pper in the proceedings of AAMAS 19. 1

2 new Q-function opertor, which we cll the κ-opertor, tht encodes robustness into the bootstrpping updte of trditionl temporl difference (TD) lerning methods. In prticulr, we design the opertor to encode the possibility of significnt rre events (SREs) without requiring the lerning gent to observe such events in trining. Although the κ-opertor is model-bsed with respect to these SREs, it cn be combined with ny TD method nd cn thus still be model-free with respect to the environment dynmics. We prove convergence of our method to the optiml sfe Q-function with respect to the model using the theory of Generlized Mrkov Decision Processes. In ddition we prove convergence to the optiml Q-function of the originl MDP given tht the probbility of SREs vnishes. Empiricl evlutions demonstrte the superior performnce of κ-bsed TD methods both in the erly lerning phse s well s in the finl converged stge. In ddition we show robustness of the proposed method to smll model errors, s well s its pplicbility to multi-gent joint-ction lerning. The reminder of this rticle is structured s follows: The next section embeds our pproch in relted work, followed in Section 3 by n introduction of bckground concepts. Section 4 introduces the new TD opertor κ, for which we subsequently prove convergence. Section 6 provides empiricl results, nd Section 7 concludes. 2 Relted Work The im to find robust policies is relevnt to multiple reserch res, including security gmes, robust control/lerning, sfe reinforcement lerning nd multi-gent reinforcement lerning. The domin of security gmes hs expnded in recent yers with mny rel-world pplictions in criticl domins [18, 21], where the min pproch hs been computing exct solutions nd deriving strong theoreticl gurntees, mostly using equilibri concepts such s Nsh nd Stckelberg equilibri [12, 15]. On the one hnd, our work dopts the informtion symmetry ssumption often used in Stckelberg Security gmes [12], providing the model of ttck types for the leder, nd llowing leder-strtegy-informed best response strtegies by ttckers. On the other, we bse our pproch on reinforcement lerning from interctions with the environment, thus we do not need to know the system model. Until now there hs been substntilly less work on reinforcement lerning for security gmes thn on gme-theoretic pproches, exceptions being for exmple Run et l. [20] nd Klim et l. [10] who use reinforcement lerning in the context of ptrolling nd illegl rhino poching problems, respectively. Similr to security gmes, control theory strts with model of the system to be controlled (the plnt), nd for the purpose of robust control ssumes set of possible plnts s n explicit model of uncertinty, seeking to design policy tht stbilises ll these plnts [33]. A slightly weker ssumption is mde in relted work tht ssumes control over the number of observtions for significnt rre events (SREs), performing updtes by smpling [3]. It introduces policy grdient vrint to improve lerning in presence of SREs, using proposl distribution tht controls dt from which it lerns nd importnce smpling to djust updtes. In contrst, our work ssumes tht the model of this system is not known priori, nd policy needs to be lerned by intercting with it, s in robust lerning. While erly work on robust reinforcement lerning focused on lerning within prmeterised cceptble policies [24], lter work trnsferred the objective of mximising tolerble disturbnces from control theory to reinforcement lerning [16]. Our work is similr to the therein defined Actor- 2

3 disturber-critic, but we replce its model of minimx simultneous ctions with stochstic trnsitions between multiple controllers (one being in control t ny time) with rbitrry objectives for ech controller. Similrly, our pproch hs commonlities with the multi-gent reinforcement lerning lgorithm Minimx-Q [13] for zero-sum gmes, which ssumes minimistion over the opponent ction spce. However, in contrst, we define n ttck to minimise over our own ction spce, nd thus lern (but not enct) simultneously our optiml policy nd the (rre) ttcks it is susceptible to. We further cover not only minimising dversries but lso rndom filures or ny other policy encoding other dversries gends (see Section 4.1). While on-policy lerning lgorithms hve been shown to perform better thn clssicl Q-lerning in perturbed environment [23], nd cn thus in some sense be considered sfer (ginst mistkes or explortion), our method combines with both on- nd off-policy lerning, nd provides robustness ginst chosen trget. 3 Bckground This work belongs to the field of Reinforcement lerning (RL) [27], nd mkes use of the core concept of Mrkov Decision Process (MDP). An MDP is formlly defined by tuple (S, A, R, P ), where S is finite set of sttes, A is finite set of ctions, R(s, ) r R is rewrd function for given stte s S nd n ction A nd P (s s, ) is trnsition function giving probbility of reching stte s fter tking ction in stte s. In this work we lso consider multi-gent setting, which uses the formultion of the Stochstic gme, which is generliztion of MDP to multiple gents nd is defined by tuple (n, S, A 1... A n, R 1... R n, P ), extending the MDP, where n is the number of gents, A i is the ction spce of gent i. The joint ction spce is A = A 1... A n, nd joint ction is = ( 1, 2,..., n ). 1 R i (s, i, i ) r i is the rewrd function of gent i for given stte s nd joint ction, nd P (s s, ) is the stte trnsition function. The min gol of RL is finding n optiml policy for given MDP. One of the most common methods is Temporl difference (TD) lerning, which is one-step bootstrpping method bsed on Bellmn style equtions. Of crucil importnce in the TD lerning is the definition of the TD error, describing the difference between lredy lernt vlue nd new informtion bout the vlue obtined from intercting with the environment. TD error, s the difference between the trget nd the current vlue, prescribes the type of djustment we mke to the lredy lernt vlue. In this work we focus on modifying the trget, which hs the stndrd form of r + γv (s ), where γ is the discount fctor nd V (s ) is the vlue of the next stte s. The trget cn be induced by the behviour policy in which cse we re tlking bout on-policy methods or by something else (e.g., mximiztion over the ction spce) in which cse we rrive to off-policy type of lerning. For further explntion of common RL concepts used in this pper we refer the reder to Sutton nd Brto [27]. 4 The Robust TD Opertor κ We now present our robust TD opertor κ. Before we formlly define the opertor, we give n intuitive exmple. Suppose Q-lerning gent needs to lern sfe policy ginst potentil 1 We use the common shorthnd i to denote the joint ction of ll gents except gent i, i.e., i = ( 1,..., i 1, i+1,..., n). 3

4 mlicious dversry who could, with some probbility κ, tke over control in the next stte. 2 The vlue of the next stte s t+1, thus, depends on who is in control 3 : if the gent is in control, she cn choose n optiml ction tht mximizes expected return; or if the dversry is in control he might, in the worst cse, im to minimize the expected return. This cn be cptured by the following modified TD error ( δ t = r t+1 + γ (1 κ) mx ) Q(s t+1, ) + κ min Q(s t+1, ) Q(s t, t ), where we ssume tht the gent hs knowledge of (or cn estimte) the probbility κ. In the following we first present forml, generl model of the opertor κ, by modifying the trget in the clssicl Bellmn style vlue function. We then present prcticl implementtions of TD(κ) methods tht use this opertor for both single- nd multi-gent settings, bsed on the clssicl on- nd off-policy TD lerning lgorithms (Expected) SARSA nd Q-lerning. 4.1 Forml Model We consider set of m possible control policies C = {σ 1,..., σ m }. At ech time step, one of these policies is in control (nd thus decides on the next ction) with some probbility p(σ i s) tht my depend on the stte s. The set C nd probbility function p( ) re ssumed to be (pproximtely) known by the gent. In our new TD methods, the vlue of the next stte s then becomes function of both, the stte nd the function p( ), which we cpture in our proposed opertor κ, s V κ (s ). Note tht the set C includes the focl policy π tht we seek to optimise in fce of (possibly dversril) lterntive controllers. Such externl control policies cn represent for exmple mlicious ttcker, iming to minimize the expected return, or ny rbitrry dynmics, such s rndom filures, represented by, e.g., uniformly rndom policy. Bsed on prior ssumption bout the nture of σ we wnt to optimise the focl policy π without necessrily observing ctul ttcks or filures. This mens lerning our robust policy π right from the strt. We define σ in terms of our own Q-vlue function, for exmple n ttcker tht is minimising our expected return. Thus we need to lern only one Q-vlue function Q π. This is similr to the stndrd ssumption in Stckelberg gmes tht the ttcker is ble to fully observe our pst ctions nd thus cn enct the informed best response. We define the Q-vlue function updte for our policy π bsed on stndrd Bellmn eqution nd given the opertor κ s [ ] Q π (s, ) Q π (s, ) + α r + γv κ (s ) Q π (s, ). (1) }{{} trget Note tht where in the stndrd Bellmn eqution we would hve V π (s) = π(s, )Qπ (s, ), in our cse we hve V κ (s) = p(σ s) σ(s, )Q π (s, ), (2) σ C 2 We use the symbol κ to denote the proposed TD opertor nd the symbol κ for the prmeter of probbility of ttck, both re different versions of the letter kpp. 3 Note tht while token of control could be included in the stte (doubling its size), our pproch rther directly pplies model-bsed bootstrp updtes. This mkes it explicit tht the robustness trget is chosen prmeter of the opertor, nd mkes it possible to lern robust strtegies before observing SREs, or when lerning does not occur during SREs due to compromistion. 4

5 computed s weighted sum over ll possible control policies σ C. Note tht we cn lern Q π without ctully experiencing ny ttck or mlfunction, bsed only on prior ssumptions bout the possible control policies s cptured by the opertor κ. We refer to this trget modifiction s the opertor κ becuse it closely resembles the Bellmn optimlity opertor T, which is defined s T [ V (s) = mx R(s, )+ s P (s s, )γv (s ) ]. Thus, we cn then formlly define the κ optimlity opertor Tκ by substituting the vlue function V ( ) with V κ ( ). In the following we present severl κ-versions of clssicl TD methods. For simplicity we ssume scenrio in which we hve only single dversril externl policy σ tht ims to minimize our vlue, nd thus C = {π, σ}. Note however tht our model is generl, nd would work for ny C nd p( ). 4.2 Exmples of TD(κ) Methods We first present single-gent κ-bsed lerning methods by building on the stndrd TD methods Q-lerning nd Expected SARSA. Then we present two-gent joint-ction lerning pproches. Although generliztion to n gents is reltively strightforwrd, we choose to focus solely on the single- nd two-gent cse in this pper for clrity of exposition. In ech cse, we consider the setting in which either the focl gent, with policy π, is in control, or the externl dversry with policy σ iming to minimize return. We further simplify the model by mking the control policy probbility function p( ) stte-independent, reducing it to probbility vector Single-Agent Methods Before we present the lgorithms, it is importnt to note tht we need to distinguish the trget nd behviour policies. The κ-opertor is defined on the trget (see Eq. (1)), while the behviour policy is used only for selecting ctions. We ssume n ɛ-greedy behviour policy throughout. In off-policy Q(κ), the trget policy is the greedy policy π(s) = rg mx Q(s, ) tht mximizes expected return. The dversril policy on the other hnd ims to minimize the return, i.e., σ(s) = rg min Q(s, ). Assuming probbility of ttck of κ s before, we hve p(π) = (1 κ) nd p(σ) = κ. Thus, Eq. (2) becomes V κ (s) = (1 κ) mx Q(s, ) + κ min Q(s, ). For on-policy Expected SARSA(κ) the trget is the (expecttion over the) focl policy π, while the dversril policy σ remins the sme s before. Thus, we hve V κ (s) = (1 κ)e π [ Q(s, ) ] + κ min Q(s, ) = (1 κ) π( s)q(s, ) + κ min Q(s, ) Multi-Agent Methods We move from single-gent setting to scenrio in which multiple gents interct. For ske of exposition we only present two-gent cse, which we further exmine in the reminder of this pper. 5

6 We ssume two gents with different ction spces, A 1 nd A 2, but n identicl rewrd function nd thus shred joint ction Q-vlue function Q : S A 1 A 2 R. Moreover, we ssume full communiction during the lerning phse, llowing the gents to tke ech other s policies into ccount when selecting the next ction. 4 Our lgorithms re therefore bsed on the joint-ction lerning (JAL) prdigm [4]. We further ssume tht only one gent cn be ttcked t ech time step. 5 For multi-gent Q(κ) we cn write Eq. (2) for ech individul gent s V κ (s) = (1 κ) mx mx Q(s, 1, 2 ) A 1 A 2 + κ 2 + κ 2 min mx Q(s, 1, 2 ) A 1 A 2 min mx Q(s, 1, 2 ) A 2 A 1 with 1 A 1 nd 2 A 2, representing the scenrio in which no ttck hppens with probbility (1 κ), nd ech gent is ttcked individully with probbility κ /2. 6 Anlogously, we cn define Eq. (2) for multi-gent Expected SARSA(κ) s V κ (s) = (1 κ) E 1 π 1, 2 π 2 [ Q(s, 1, 2 ) ] + κ 2 + κ 2 min A 1 E 2 π 2 [ Q(s, 1, 2 ) ] min A 2 E 1 π 1 [ Q(s, 1, 2 ) ] where we now compute n expecttion over the ctul policy of the gents tht re not ttcked, while the ttcker is still minimizing. 5 Theoreticl Anlysis In this section we nlyze theoreticl properties of the proposed κ-methods. We strt by relting the different lgorithms to ech other in the limit of their respective prmeters. Then we proceed to show convergence of both Q(κ) nd Expected SARSA(κ) to two different fixed points: (i) to the optiml vlue function Q of the originl MDP in the limit where κ 0; nd (ii) to the optiml robust vlue function Q κ of the MDP tht is generlized w.r.t. κ for constnt prmeter κ. Note tht optimlity in this sense is purely induced by the relevnt opertor. In (i) this is the stndrd Bellmn optimlity which mximizes the expected discounted return of the MDP. However, in (ii) we derive optimlity in the context of Generlized MDPs [29], where optiml simply mens the fixed point of given opertor, which cn tke mny forms. Before proceeding with the convergence proofs, Figure 1 summrizes some reltionships between the lgorithms in terms of their trgets, in the limit of their respective prmeters: As is known, Expected SARSA, SARSA, nd Q-lerning become identicl in the limit of greedy policy [27, 31]. Furthermore, the updte trgets of our κ-methods pproch the updte trgets of the stndrd TD methods on which they re bsed s κ 0. Finlly, Expected SARSA(κ) nd Q(κ) shre lst. 4 A common prctice in coopertive multi-gent lerning settings, see e.g., [7, 26]. 5 Although relxing this ssumption is strightforwrd, we opt to keep it for clrity. 6 Note the order of the min mx, which follows the Stckelberg ssumption of n ll-knowing ttcker who moves 6

7 Expected SARSA(κ) κ 0 ɛ 0 κ 0 Q(κ) Expected SARSA ɛ 0 Q-lerning ɛ 0 ɛ 0 SARSA Figure 1: The reltionship between the lerning trgets of different lgorithms in the limits of their prmeters. On-policy methods re in green, off-policy methods in ornge. the sme reltionship s their originl versions, nd thus Expected SARSA(κ) pproches Q(κ) s ɛ 0. Note tht the lgorithms equivlence in the limit does not hold in the trnsient phse of the lerning process, nd hence in prctice they my converge on different pths nd to different policies tht shre the sme vlue function. For comprehensive understnding of the lgorithms introduced in Section 4.2, the following sections provide proofs for both convergence of κ methods for κ 0, s well s their convergence when κ stys constnt. While we focus on the dversril trgets considered in Section 4.2, previous proof of convergence under persistent explortion [29] cn be interpreted s model of rndom filures with fixed kpp. 5.1 Convergence to the Optiml Q There exist severl proofs of convergence for the temporl difference lgorithms Q-lerning [9, 30], SARSA [23], nd Expected SARSA [31]. Ech of these proofs hinges on linking the studied lgorithm to stochstic process, nd then using convergence results from stochstic pproximtion theory [6, 19]. These proofs re bsed on the following lemm, presented s Theorem 1 in Jkkol et l. [9] nd s Lemm 1 in Singh et l. [23]. These differ in the third condition, which describes the contrction mpping of the opertor. The contrction property used for the Q-lerning proof [9] hs the form E{F t ( ) P t } γ t, where γ [0, 1). We show the lemm s it ws used for the SARSA proof provided by Singh et l. [23], who show tht the contrction property does not need to be strict; strict contrction is required to hold only symptoticlly. Lemm 5.1. Consider stochstic process (α t, t, F t ), t 0, where α t, t, F t : X R stisfy the equtions t+1 (x) = ( 1 α t (x) ) t (x) + α t (x)f t (x), x X, t = 0, 1, 2,... Let P t be sequence of incresing σ-fields such tht α 0 nd 0 re P 0 -mesurble nd α t, t nd F t 1 re P t -mesurble, t = 1, 2,.... Then, t converges to zero with probbility one (w.p.1) under the following ssumptions: 1. the set X is finite, 2. 0 α t (x t ) 1, t α t(x t ) =, t α2 t (x t ) < w.p.1, 3. E{F t ( ) P t } γ t + c t, where γ [0, 1) nd c t converges to zero w.p.1, 4. V r{f t (x t ) P t } K(1 + t ) 2, where K is some constnt, 7

8 where denotes mximum norm. The proof continues by relting Lemm 5.1 to the temporl difference lgorithm, following the sme resoning s Vn Seijen et l. [31] in their convergence proof for Expected SARSA. We define X = S A, P t = {Q 0, s 0, 0, r 0, α 0, s 1, 1,..., s t, t }, x t = (s t, t ), which represents the pst t step t nd α t (x t ) = α t (s t, t ) is lerning rte for stte s t nd ction t. To show the convergence of Q to the optiml fixed point Q we set t (x t ) = Q t (s t, t ) Q (s t, t ), therefore when t converges to zero, then the Q vlues converge to Q. The mximum norm cn be expressed s mximizing over sttes nd ctions s t = mx s mx Q t (s, ) Q (s, ). We follow the resoning of Theorem 1 from Vn Seijen et l. [31], where we repet the conditions (1), (2) nd (4) nd modify the condition (3) for the κ methods s: Theorem 5.2. Q(κ) nd Expected SARSA(κ) s defined in Section using the respective vlue function V κ, defined by Q t+1 (s t, t ) = (1 α t (s t, t ))Q t (s t, t ) + α t (s t, t )[r t + γv κ t (s t+1 )] converge to the optiml Q function Q (s, ) if: 1. the stte spce S nd ction spce A re finite, 2. α t (s t, t ) (0, 1), t α t(s t, t ) = nd t α2 t (s t, t ) < w.p.1, (3) κ converges to zero w.p.1, (3) for Expected SARSA(κ) the policy is greedy in the limit with infinite explortion (GLIE ssumption), 4. the rewrd function is bounded. Proof. Convergence of Q(κ): To prove convergence of Q(κ) we hve to show tht the conditions from Lemm 5.1 hold. Conditions (1), (2) nd (4) of Theorem 5.2 correspond to conditions (1), (2) nd (4) of Lemm 5.1 [31]. We now need to show tht the contrction property holds s well, using condition (3) of Theorem 5.2. Adpting the proof of Vn Seijen et l. [31], we set F t (x) = F t (s, ) = r t (s, ) + γvt κ (s ) Q (s, ) to show tht F t (s, ) is contrction mpping, i.e., condition (3) in Lemm 5.1. For Q(κ) we write: F t = r t + γ ( (1 κ) mx Q t (s t+1, ) + κ min Q t (s t+1, ) ) Q (s t, t ). We wnt to show tht E{F t } γ t + c t to prove the convergence of Q(κ) to the optiml vlue Q. E{F t } = E{r t + γ ( (1 κ) mx Q t (s t+1, ) + κ min Q t (s t+1, ) ) Q (s t, t )} E{r t + γ mx Q t (s t+1, ) Q (s t, t )} + γ E{κ min γ mx s γ mx s γ t + γκ mx s mx κ min min Q t (s t+1, ) κ mx Q t (s, ) mx Q t (s, ) κ mx Q t (s t+1, )} Q (s, ) + Q t (s, ) Q t (s, ) mx Q t (s, ), 8

9 where the first inequlity follows from stndrd lgebr nd the fct tht splitting the mximum norm yields t lest s lrge number, the second inequlity follows from the definition of Q 7 nd the mximl difference in vlues over ll sttes being t lest s lrge s difference between vlues given in stte s t+1, nd the third inequlity follows from the definition of t bove. We cn see tht if we set c t = γκ mx s min Q t (s, ) mx Q t (s, ), then for κ 0 we get c t converging to zero w.p.1, thus proving convergence of Q(κ). Proof. Convergence of Expected SARSA(κ): Similrly s in the proof of Q(κ) we need to show tht the contrction property holds s well, this time using conditions (3) nd (3) of Theorem 5.2. We first define: F t = r t + γ ( (1 κ) π t ( s t+1 )Q t (s t+1, ) + κ min Q t (s t+1, ) ) Q (s t, t ) nd then show the following: E{F t } = E{r t + γ ( (1 κ) π t ( s t+1 )Q t (s t+1, ) + κ min Q t (s t+1, ) ) Q (s t, t )} E{r t + γ mx Q t (s t+1, ) Q (s t, t )} + γ E{(1 κ) γ mx s mx γ mx (1 κ) s π t ( s t+1 )Q t (s t+1, ) + κ min Q t (s, ) mx Q (s, ) + π t ( s)q t (s, ) + κ min Q t (s t+1, ) mx Q t (s t+1, )} Q t (s, ) mx Q t (s, ), where the inequlities use the sme opertions s bove in the proof of Q(κ). If we set c t = γ mx s (1 κ) π t( s)q t (s, ) + κ min Q t (s, ) mx Q t (s, ) nd ssume tht the policy is greedy in the limit with infinite explortion (GLIE ssumption) nd prmeter κ 0 w.p.1 (conditions (3) nd (3)), it follows tht c t converges to zero w.p.1, thereby proving tht Expected SARSA(κ) converges to optiml fixed point Q. 5.2 Convergence to the Robust Q κ In this section we show convergence to the robust vlue function Q κ which is optiml w.r.t. the opertor κ. The min difference with the proof of Theorem 5.2 is tht here we do not require κ 0 but insted ssume it remins constnt over time. We bse our resoning on the theory of Generlized MDPs [29]. A Generlized MDP is defined using opertor-bsed nottion s ( ) (R + γv ) (s) = mx P (s ( s, ) R(s, ) + γv (s ) ), s where the opertor defines how n optiml gent chooses her ctions (in the clssic Bellmn eqution this denotes mximiztion) nd opertor defines how the vlue of the current stte is updted by the vlue of the next stte (in the clssic Bellmn eqution this denotes probbility 7 Recll tht we set out in this section to show convergence to the sme optiml Q-vlue s clssicl Q-lerning Q (s t, ) = r t + γ mx Q (s t+1, ), even if we do so by our new opertor. 9

10 weighted verge over the trnsition function). These opertors cn be chosen to model vrious different scenrios. The generlized Bellmn eqution cn now be written s V = (R + γv ). The min result of Szepesvri nd Littmn [29] is tht if nd re non-expnsions, then there is unique optiml solution to which the generlized Bellmn eqution converges, given certin ssumptions. For 0 γ < 1 nd non-expnsion properties of nd we get contrction mpping of the Bellmn opertor T defined s T V = (R + γv ). Then, the opertor T hs unique fixed point by the Bnch fixed-point theorem [25]. Building on the stochstic pproximtion theory results (s we lso used in the Section 5.1), Szepesvri nd Littmn [29] show the following: Lemm 5.3. Generlized Q-lerning with opertor using Bellmn opertor T t (Q, Q)(s, ) = {( 1 αt (s, ) ) Q (s, ) + α t (s, ) ( r t + γ( Q)(s t) ) if s = s t, = t Q (s, ) otherwise converges to the optiml Q function w.p.1, if 1. s t is rndomly selected ccording to the probbility distribution defined by P (s t, t, ), 2. α t (s t, t ) (0, 1), t α t(s t, t ) = nd t α2 t (s t, t ) < w.p.1, 3. is non-expnsion, 4. the rewrd function is bounded. We bse our convergence proofs for Q(κ) nd Expected SARSA(κ) on the insights of Szepesvri nd Littmn [29] given in Lemm 5.3. Theorem 5.4. Q(κ) nd Expected SARSA(κ) s defined in Section converge to the robust Q function Q κ for ny fixed κ. Proof. Convergence of Q(κ) to Q κ: To prove convergence of Q(κ) we follow the proof of Generlized Q-lerning in Lemm 5.3. The only condition we need to gurntee is the non-expnsion property of the opertor in the vlue function updte, which for Q(κ) is weighted verge of the opertors min nd mx. We write the opertor for Q(κ) s κ nd define it s ( κ Q)(s, ) = (1 κ) mx Q(s, ) + κ min Q(s, ). In Appendix B of Szepesvri nd Littmn [29], Theorem 9 sttes tht ny liner combintion of non-expnsion opertors is lso non-expnsion opertor. Moreover Theorem 8 sttes tht the summry opertors mx nd min re lso non-expnsions. Therefore, κ is non-expnsion s well, thus proving the convergence of Q(κ) to the robust fixed point Q κ induced by the opertor κ. Proof. Convergence of Expected SARSA(κ) to Q κ: We bse our convergence proof of Expected SARSA(κ) gin on the work of Szepesvri nd Littmn [29], this time on their insights regrding 10

11 persistent explortion (Section 4.5 in their pper). They show tht Generlized Q-lerning with ɛ- greedy ction selection converges, for fixed ɛ, in the Generlized MDP. Following similr resoning, we define the opertor for Expected SARSA(κ) with fixed ɛ s ( κ Q)(s, ) = (1 κ) ( ɛ 1 A Q(s, ) + (1 ɛ) mx Q(s, ) ) + κ min Q(s, ). Agin, from repeted ppliction of Theorems 8 nd 9 in Appendix B of Szepesvri nd Littmn [29] it follows tht κ is non-expnsion s well. Therefore, by Lemm 5.3, Expected SARSA(κ) converges to Q κ for fixed explortion ɛ. It remins n open question whether Expected SARSA(κ) lso converges for decresing ɛ, e.g., under the GLIE ssumption, even though we conjecture tht it might. 5.3 Convergence in the Multi-Agent Cse We now prove convergence of the coopertive multi-gent vrint of the κ methods presented in Section This proof builds on the theory of Generlised MDPs, similr to the proofs presented in Section 5.2. Therefore this proof lso ssumes fixed probbility of ttck κ. In ddition, we mke use of the ssumption tht gents cn communicte freely in the lerning phse, nd thus receive identicl informtion nd cn build common joint-ction Q-tble. Theorem 5.5. Multi-gent Q(κ) nd Expected SARSA(κ) s defined in Section converge to the robust Q function Q κ for ny fixed κ. Proof. The κ opertor for our multi-gent versions of Q(κ) nd Expected SARSA(κ) consists of nested combintion of different components, in prticulr mx Q(s, ), min Q(s, ), nd πɛ (s, )Q(s, ) where π ɛ is the ɛ-greedy policy. By Theorem 8 of Szepesvri nd Littmn [29], mx nd min re non-expnsions. By Theorem 9 of [29], liner combintions of non-expnsion opertors re lso non-expnsion opertors. Finlly, by Theorem 10 of [29], products of non-expnsion opertors re lso non-expnsion opertors. Therefore, lso mx mx, mx min, nd min mx re nonexpnsion opertors, s re liner combintions of those compounds. Similrly, πɛ (s, )Q(s, ) for fixed ɛ cn be written s liner combintion of summry opertors, which by Theorems 8 nd 9 of Szepesvri nd Littmn [29] is non-expnsion. Therefore, the κ opertor used in both multi-gent Q(κ) nd Expected SARSA(κ) is non-expnsion. Thus, by Lemm 5.3, Q(κ) nd Expected SARSA(κ) converge to Q κ for fixed κ, nd in the cse of Expected SARSA(κ), for fixed ɛ. 6 Experiments nd Results In this section we evlute temporl difference methods with the proposed opertor κ; off-policy type of lerning Q(κ) nd on-policy type of lerning Expected SARSA(κ). We experiment with the clssicl Cliff wlking scenrio for the single-gent cse nd the multi-gent Puddle world scenrio. Both these domins contin some criticl sttes, cliff nd puddle respectively, which render very high negtive rewrd for the gent(s) in cse of stepping into them. These criticl sttes represent the significnt rre events (SREs). We compre our methods with clssic temporl difference methods 11

12 Figure 2: Cliff Wlking: The gent needs to get from the strt [S] to the gol [G], voiding the cliff (grey tiles) Q-lerning, pth = Q( ) for = 0.01, pth = Q( ) for = 0.1, pth = Figure 3: The Puddle world: Q(κ) lerns sfer pth with incresing κ. Puddles re drk blue, the rrows show the optiml ctions on the lerned pth, nd the hetmp shows the number of visits to ech stte (, blue is none). like SARSA, Q-lerning nd Expected SARSA. In ll the experiments we consider n undiscounted (γ=1), episodic scenrio. Cliff wlking: single-gent The Cliff wlking experiment s shown in Figure 2 is clssicl scenrio proposed in Sutton nd Brto [27] nd used in mny other ppers ever since (e.g., [31]). The gent needs to get from the strt stte [S] to the gol stte [G], while voiding stepping into the cliff, otherwise rendering rewrd of -100 nd sending him bck to the strt. For every move which does not led into the cliff the gent receives rewrd of -1. Puddle world: multi-gent The puddle world environment is grid world with puddles which need to be voided by the joint-lerning gents. The two gents jointly control the movement of single robot in this puddle world, with ech controlling either direction up, down or left, right. Agent 1 cn tke the ctions {sty, move down, move up} nd gent 2 cn choose {sty, move left, move right, move right by 2 }, thus their ction spces re different, further complicting the lerning process compred to the single-gent scenrio. The joint ction is the combintion of the two selected ctions. We ssume rewrd of -1 for every move nd -100 for stepping into puddle (returning to the strt node). The gents hve to move together from the strt node t the top left corner to the gol t the bottom right corner. Figure 3 shows the policy lerned by our proposed lgorithm Q(κ) for the two joint-lerning gents. Note how sfer pth (longer, voiding the puddles) is lerned with incresing prmeter κ. For κ = 0 our lgorithm degenertes to Q-lerning (left pnel). 12

13 Cliff wlking: deterministic environment Cliff wlking: 10 % stochstic environment Cliff wlking: 10 % ttck while trined 50 verge return SARSA Q-lerning Q( ) Exp. SARSA Exp. SARSA( ) lerning rte Puddle world: deterministic environment lerning rte Puddle world: 10 % stochstic environment lerning rte Puddle world: 10 % ttck while trined verge return lerning rte lerning rte lerning rte Figure 4: Cliff wlking (single-gent) in first row nd Puddle world (multi-gent) in second row. Deterministic environment (first column), 10 % stochstic environment (second column) nd 10 % ttck while trining (third column). ɛ-greedy policy with fixed ɛ = 0.1. Erly performnce - dshed lines (100 episodes), converged performnce - solid lines (100, 000 episodes). 6.1 Performnce We replicte the experiment of Vn Seijen et l. [31] on the cliff wlking domin, which compres our κ methods with Q-lerning, SARSA nd Expected SARSA. In line with previous results we show (i) n erly performnce, which is verge return over first 100 trining episodes nd (ii) converged performnce, which is verge return over 100, 000 episodes. Figure 4 shows this methodology on 3 different settings; (i) deterministic environment, where ech ction chosen by the policy is executed with certinty, (ii) n environment with 10% stochsticity, where there is rndom ction tken with probbility 10% insted of the chosen ction nd (iii) n environment with 10% probbility of ttck, where there is n dversril ction tken with probbility 10% insted of the chosen ction, s discussed before we define n ttck s n ction tht minimizes the Q-vlue function in the given stte. The 10% stochsticity in the environment cn be seen s rndom filure of the system. Furthermore, we lso show the sme set of experiments for the Puddle world in the second row of tht figure. The erly performnce experiments (100 episodes) re verged over 300 trils nd the converged performnce (10, 000 episodes) experiments re verged over 10 trils. We lso 13

14 show the 95% confidence intervls. Similrly to Vn Seijen et l. [31] we fix the explortion rte to ɛ = 0.1. For the κ methods we fix κ = 0.1, further below we lso experiment with different settings of the prmeter κ. Note tht the y-xis, showing the verge return, is the sme in ech row for better comprison. The x-xis shows different lerning rtes α. We cn see how the verge return decreses with more complex scenrios, from deterministic, over to stochstic, to one with ttcks. We cn observe how the κ methods re superior to the other bselines in the erly performnce experiments, they re especilly good in the ttck cse, which is the scenrio the κ methods re the most suitble for. In the converged performnce experiments the κ methods bet Q-lerning nd SARSA nd re similr or better compred to Expected SARSA. 6.2 Different Levels of Probbility of Attck In this section we investigte how the methods behve under different levels of ttck, defined by the probbility of ttck per stte. We consider n ttck on trined (converged) methods, thus we first trin ech method for 100, 000 episodes (in deterministic environment) nd then we test it on 50, 000 trils with given probbility of ttck per stte. We verge the results over 10 trils nd provide 95% confidence intervls. Note, tht this is different methodology of testing the methods ginst n dversril ttck compred to the experiments in Figure 4, where we considered ttcks while trining. This experiment shows the strength of the κ methods for different levels of ttcks. We ssume the probbility of ttck to be known here nd thus we set the prmeter κ to be equl to tht probbility, which is the mening of the prmeter κ s described before. In other words, prmeter κ prescribes how much sfely we wnt to ct. We consider very rre ttcks (0.001 probbility of ttck in ech stte) to more frequent ttcks (0.2 probbility of ttck in ech stte) s shown in Figure 5. For better visulistion we use logrithmic xes. We trin ll the methods with lerning rte α = 0.1 nd fixed explortion rte ɛ = 0.1. We lso experimented with different lerning rtes nd conclude tht ll the methods, except SARSA, re quite stble for different lerning rtes, which cn lso be seen in the deterministic experiments in Figure 4. However, SARSA is very unstble for different lerning rtes (lso see Figure 4), lerns different pths for different lerning rtes nd does not converge fst enough or not t ll. This cn be prtly explined by higher vrince of SARSA [31]. This instbility of SARSA is demonstrted in our experiments by the wide confidence intervls in Figure 5. We test the different levels of probbility of ttck on the Cliff wlking experiment in the left pnel of Figure 5, where we cn see tht the κ methods compre fvourbly to the other bselines, however in some prts they give similr performnce s Expected SARSA or SARSA. The Cliff wlking experiment hs limited expressiveness for testing the methods due to limited number of possible sfe pths with low costs (see Figure 2), which is the reson for the κ methods to show only similr performnce compred to the bselines nd thus not being ble to show the full potentil of the κ methods. However, the Puddle world is more expressive, becuse there re severl possible pths differing in level of sfety nd cost. The bigger solution spce of the Puddle world is lso induced by the 2 cooperting gents, ech hving their own ction spce. Therefore, on the right pnel of Figure 5 we show the Puddle world experiment for different levels of probbility of ttck. Here, we cn clerly see the κ methods outperform the bselines, especilly Q(κ) is superior over the whole rnge of considered probbilities of ttck. Note tht Q(κ) lerns sfer pth even for very rre ttcks (0.001 probbility of ttck), which lso shows in Figure 3, where for 0.01 probbility of ttck Q(κ) lerns sfer pth with the sme cost compred to Q-lerning, which lerns pth of the sme distnce but much closer to the puddles. 14

15 14 Cliff wlking, different levels of ttck 5 Puddle world, different levels of ttck 20 7 verge return verge return SARSA Q-lerning Q( ) Exp. SARSA Exp. SARSA( ) probbility of ttck SARSA Q-lerning Q( ) Exp. SARSA Exp. SARSA( ) probbility of ttck Figure 5: Vrying probbility of ttck: Cliff wlking (left), Puddle world (right), trined 100k, test 50k, α = 0.1, ɛ = Robustness Anlysis Here we test the robustness of the proposed lgorithms with the opertor κ by no longer ssuming the probbility of ttck to be known nd thus it is not possible to correctly set the κ prmeter for Q(κ) nd Expected SARSA(κ). Note tht in our previous experiments we set the prmeter κ to be equl to the ctul probbility of ttck. In Figure 6 we show the performnce of our lgorithms for rnge of ctul ttck probbilities (y-xis) while lerning using fixed prmeter κ = 0.1. Note tht we no longer use logrithmic scles s in Figure 5 nd do not consider very rre ttcks, only probbilities of ttck round 0.1. One cn see tht even for the cses where κ is not equl to the ctul probbility of ttck the proposed κ lgorithms still outperform the bselines in most cses, especilly for Q-lerning nd Expected SARSA. In the Cliff wlking experiment (the left pnel in Figure 6) we get similr performnce of the κ methods s SARSA, however SARSA is quite unstble s discussed before nd s one cn see by the width of the confidence intervl. In the Puddle world experiment in the right pnel of Figure 6 one cn see the superior performnce of κ methods, where they bet ll the bselines even for fixed prmeter κ. Thus, we experimentlly confirmed tht even when we do not know the probbility of ttck ccurtely we cn lern better strtegy by using the κ methods. 7 Discussion nd Conclusion We presented new opertor κ for temporl difference lerning, which improves robustness of the lerning process ginst potentil ttcks or perturbtions in control. We proved convergence of Q(κ) nd Expected SARSA(κ) to (i) the optiml vlue function Q of the originl MDP in the limit where κ 0; nd (ii) the optiml robust vlue function Q κ of the MDP tht is generlized w.r.t. κ for constnt prmeter κ. In the ltter cse we lso proved convergence of coopertive joint-ction lerning version of our methods. Our complementry empiricl results demonstrte tht the proposed κ-methods indeed provide 15

16 20 30 Cliff wlking, fixed = 0.1 SARSA Q-lerning Q( ) Exp. SARSA Exp. SARSA( ) 6 8 Puddle world, fixed = 0.1 SARSA Q-lerning Q( ) Exp. SARSA Exp. SARSA( ) verge return verge return probbility of ttck probbility of ttck Figure 6: Robustness nlysis: Cliff wlking (left), Puddle world (right), trined 100k, test 50k, α = 0.1, ɛ = 0.1, κ = 0.1. robustness ginst chosen scenrio of potentil ttcks nd filures in both single- nd multi-gent settings. Although our method ssumes tht model of such ttcks nd filures is known to the gent, we further demonstrte tht our methods re robust ginst smll model errors. Moreover, we show tht even in bsence of ttcks or filures, our method lerns policy tht is robust in generl ginst environment stochsticity, in prticulr in the erly stges of lerning. There re severl interesting directions for future work. One possibility would be extending the control spce, llowing for more gents being ttcked or mlfunctioning with different intensity. We defined the probbility of control depending on the stte, but more prmeters could be introduced. Such extensions would nrrow the relity gp nd would llow for lerning more complex policies, where we believe our pproch could prove even more competitive. Furthermore, the trget of dversril policies could be lerned from experience, where ides from opponent modelling could be used (e.g., DPIQN [8]). Our proposed opertor κ cn be closely linked or even combined with some recent stte-ofthe-rt reinforcement lerning methods. Considering multi-step updte, the opertor could be combined with Retrce(λ) [17], which would potentilly speed up convergence. Another promising extension of our model would be to combine it with Q(σ) [1] to llow for mixed multi-step updtes. Note tht the prmeter σ in this lgorithm cn lso be time- or stte-dependent similrly to the potentil extensions of the control in our model. This would llow to lern robust policies ginst more complex controls such s multi-step ttcks. Another interesting extension long this line would be to model the control trnsition similr to the options frmework [2, 28], in which cse the lternte control policies could be seen s mlicious options over which the gent hs no control, with potentilly complex initition sets nd termintion conditions. Such extensions would further increse the flexibility of our proposed opertor, mking it pplicble to wide rnge of rel-world scenrios. 16

17 Acknowledgement This project hs received funding in the frmework of the joint progrmming inititive ERA-Net Smrt Energy Systems focus inititive Smrt Grids Plus, with support from the Europen Union s Horizon 2020 reserch nd innovtion progrmme under grnt greement No References [1] Kristopher De Asis, J. Hernndez-Grci, G. Hollnd, nd Richrd S. Sutton. Multi-step reinforcement lerning: A unifying lgorithm. In AAAI Conference on Artificil Intelligence, [2] Pierre-Luc Bcon, Jen Hrb, nd Doin Precup. The option-critic rchitecture. In Proceedings of Assocition for the Advncement of Artificil Intelligence Conference (AAAI), pges , [3] Kmil Andrzej Ciosek nd Shimon Whiteson. OFFER: Off-environment reinforcement lerning. In Proceedings of Assocition for the Advncement of Artificil Intelligence Conference (AAAI), [4] Croline Clus nd Crig Boutilier. The dynmics of reinforcement lerning in coopertive multigent systems. AAAI/IAAI, pges , [5] Flviu Cristin, Bob Dncey, nd Jon Dehn. Fult-tolernce in ir trffic control systems. ACM Trnsctions on Computer Systems (TOCS), 14(3): , [6] Aryeh Dvoretzky. On stochstic pproximtion. In Proceedings of the Third Berkeley Symposium on Mthemticl Sttistics nd Probbility, Volume 1: Contributions to the Theory of Sttistics, pges University of Cliforni Press, [7] Jkob Foerster, Gregory Frquhr, Trintfyllos Afours, Nnts Nrdelli, nd Shimon Whiteson. Counterfctul multi-gent policy grdients. rxiv preprint rxiv: , [8] Zhng-Wei Hong, Shih-Yng Su, Tzu-Yun Shnn, Yi-Hsing Chng, nd Chun-Yi Lee. A deep policy inference q-network for multi-gent systems. In Proceedings of the 17th Interntionl Conference on Autonomous Agents nd MultiAgent Systems, pges , [9] Tommi Jkkol, Michel I Jordn, nd Stinder P Singh. Convergence of stochstic itertive dynmic progrmming lgorithms. In Advnces in neurl informtion processing systems, pges , [10] Richrd Klim, Krl Tuyls, nd Frns Oliehoek. Mrkov Security Gmes: Lerning in Sptil Security Problems. NIPS Workshop on Lerning, Inference nd Control of Multi-Agent Systems, pges 1 8, [11] John C Knight. Sfety criticl systems: chllenges nd directions. In Proceedings of the 24th Interntionl Conference on Softwre Engineering, pges ACM,

18 [12] Dmytro Korzhyk, Zhengyu Yin, Christopher Kiekintveld, Vincent Conitzer, nd Milind Tmbe. Stckelberg vs. Nsh in Security Gmes: An Extended Investigtion of Interchngebility, Equivlence, nd Uniqueness. Journl of Artificil Intelligence Reserch, 41: , [13] Michel L. Littmn. Mrkov gmes s frmework for multi-gent reinforcement lerning. Technicl report, Brown University, [14] Jing Liu, Yng Xio, Shuhui Li, Wei Ling, nd CL Philip Chen. Cyber security nd privcy issues in smrt grids. IEEE Communictions Surveys & Tutorils, 14(4): , [15] Jin Lou, Andrew M Smith, nd Yevgeniy Vorobeychik. Multidefender security gmes. IEEE Intelligent Systems, 32(1):50 60, [16] Jun Morimoto nd Kenji Doy. Robust reinforcement lerning. Neurl computtion, 17(2): , [17] Rémi Munos, Tom Stepleton, Ann Hrutyunyn, nd Mrc Bellemre. Sfe nd efficient offpolicy reinforcement lerning. In Advnces in Neurl Informtion Processing Systems (NIPS), pges , [18] Jmes Pit, Mnish Jin, Jnusz Mrecki, Fernndo Ordonez, Christopher Portwy, Milind Tmbe, Crig Western, Prveen Pruchuri, nd Srit Krus. Deployed ARMOR Protection: The Appliction of Gme Theoretic Model for Security t the Los Angeles Interntionl Airport. In Interntionl Joint Conference on Autonomous Agents nd Multigent Systems (AAMAS), volume 3, pges , [19] Herbert Robbins nd Sutton Monro. A stochstic pproximtion method. The Annls of Mthemticl Sttistics, 22(3): , [20] Sui Run, Cndr Meirin, Feili Yu, Krishn R Pttipti, nd Robert L Popp. Ptrolling in stochstic environment. Technicl report, Electricl nd Computer Engineering Deprtment, University of Connecticut, Storrs, [21] Eric Shieh, Bo An, Rong Yng, Milind Tmbe, Crig Bldwin, Joseph DiRenzo, Ben Mule, nd Grrett Meyer. PROTECT: A Deployed Gme Theoretic System to Protect the Ports of the United Sttes. Interntionl Conference on Autonomous Agents nd Multigent Systems (AAMAS), 1:13 20, [22] Mrtin L Shoomn. Relibility of computer systems nd networks: fult tolernce, nlysis, nd design. John Wiley & Sons, [23] Stinder Singh, Tommi Jkkol, Michel L Littmn, nd Csb Szepesvári. Convergence results for single-step on-policy reinforcement-lerning lgorithms. Mchine lerning, 38(3): , [24] Stinder P Singh, Andrew G Brto, Roderic Grupen, nd Christopher Connolly. Robust reinforcement lerning in motion plnning. In Advnces in neurl informtion processing systems (NIPS), pges , [25] D. R. Smrt. Fixed point theorems. Cmbridge University Press, Cmbridge,

19 [26] Peter Sunehg, Guy Lever, Audruns Gruslys, Wojciech Mrin Czrnecki, Vinicius Zmbldi, Mx Jderberg, Mrc Lnctot, Nicols Sonnert, Joel Z Leibo, Krl Tuyls, nd Thore Grpel. Vlue-decomposition networks for coopertive multi-gent lerning. rxiv preprint rxiv: , [27] Richrd S. Sutton nd Andrew G. Brto. Reinforcement lerning: An introduction. The MIT Press, Cmbridge, MA, [28] Richrd S Sutton, Doin Precup, nd Stinder Singh. Between mdps nd semi-mdps: A frmework for temporl bstrction in reinforcement lerning. Artificil intelligence, 112(1-2): , [29] Csb Szepesvri nd Michel L Littmn. Generlized mrkov decision processes: Dynmicprogrmming nd reinforcement-lerning lgorithms. Technicl report, Brown University, [30] John N Tsitsiklis. Asynchronous stochstic pproximtion nd q-lerning. Mchine lerning, 16(3): , [31] Hrm vn Seijen, Hdo vn Hsselt, Shimon Whiteson, nd Mrco Wiering. A theoreticl nd empiricl nlysis of Expected Srs. In IEEE Symposium on Adptive Dynmic Progrmming nd Reinforcement Lerning, ADPRL 2009, pges , [32] Ye Yn, Yi Qin, Hmid Shrif, nd Dvid Tipper. A survey on smrt grid communiction infrstructures: Motivtions, requirements nd chllenges. IEEE communictions surveys & tutorils, 15(1):5 20, [33] Kemin Zhou nd John Comstock Doyle. Essentils of robust control, volume 104. Prentice hll, Upper Sddle River, NJ,