Repeated Games against Budgeted Adversaries

Transcription

1 Repeated Games aganst Budgeted Adversares Jacob Abernethy Dvson of Computer Scence UC Berkeley Manfred K. Warmuth Department of Computer Scence UC Santa Cruz Abstract We study repeated zero-sum games aganst an adversary on a budget. Gven that an adversary has some constrant on the sequence of actons that he plays, we consder what ought to be the player s best mxed strategy wth knowledge of ths budget. We show that, for a general class of normal-form games, the mnmax strategy s ndeed effcently computable and reles on a random playout technque. We gve three dverse applcatons of ths new algorthmc template: a cost-senstve Hedge settng, a partcular problem n Metrcal Task Systems, and the desgn of combnatoral predcton markets. Introducton How can we reasonably expect to learn gven possbly adversaral data? Overcomng ths obstacle has been one of the major successes of the Onlne Learnng framework or, more generally, the so-called compettve analyss of algorthms: rather than measure an algorthm only by the cost t ncurs, consder ths cost relatve to an optmal comparator algorthm whch has knowledge of the data n advance. A classc example s the so-called experts settng : assume we must predct a sequence of bnary outcomes and we are gven access to a set of experts, each of whch reveals ther own predcton for each outcome. After each round we learn the true outcome and, hence, whch experts predcted correctly or ncorrectly. The expert settng s based around a smple assumpton, that whle some experts predctons may be adversaral, we have an a pror belef that there s at least one good expert whose predctons wll be reasonably accurate. Under ths relatvely weak good-expert assumpton, one can construct algorthms that have qute strong loss guarantees. Another way to nterpret ths sequental predcton model s to treat t as a repeated two-player zero-sum game aganst an adversary on a budget; that s, the adversary s sequence of actons s restrcted n that play ceases once the adversary exceeds the budget. In the experts settng, the assumpton there s a good expert can be renterpreted as a nature shall not let the best expert err too frequently, perhaps more than some fxed number of tmes. In the present paper, we develop a general framework for repeated game-playng aganst an adversary on a budget, and we provde a smple randomzed strategy for the learner/player for a partcular class of these games. The proposed algorthms are based on a technque, whch we refer to as a random playout, that has become a very popular heurstc for solvng games wth massvely-large state spaces. Roughly speakng, a random playout n an extensve-form game s a way to measure the lkely outcome at a gven state by fnshng the game randomly from ths state. Random playouts, often known smply as Monte Carlo methods, have become partcularly popular for solvng the game of Go [5], whch has led to much follow-up work for general games [2, ]. The Budgeted Adversary game we consder also nvolves exponentally large state spaces, yet we acheve effcency usng these random playouts. The key result of ths paper s that the proposed random playout s not smply a good heurstc, t s ndeed mnmax optmal for the games we consder. Supported by a Yahoo! PhD Fellowshp and NSF grant Supported by NSF grant IIS

2 Abernethy et al [] was the frst to use a random playout strategy to optmally solve an adversaral learnng problem, namely for the case of the so-called Hedge Settng ntroduced by Freund and Schapre [0]. Indeed, ther model can be nterpreted as a partcular specal case of a Budgeted Adversary problem. The generalzed framework that we gve n the frst half of the paper, however, has a much larger range of applcatons. We gve three such examples, descrbed brefly below. More detals are gven n the second half of the paper. Cost-senstve Hedge Settng. In the standard Hedge settng, t s assumed that each expert suffers a cost n [0, ] on each round. But a surprsngly-overlooked case s when the cost ranges dffer, where expert may suffer per-round cost n [0, c ] for some fxed c > 0. The vanlla approach, to use a generc bound of max c, s extremely loose, and we know of no better bounds for ths case. Our results provde the optmal strategy for ths cost-senstve Hedge settng. Metrcal Task Systems (MTS). The MTS problem s decson/learnng problem smlar to the Hedge Settng above but wth an added dffculty: the learner s requred to pay the cost of movng through a gven metrc space. Fndng even a near-optmal generc algorthm has remaned elusve for some tme, wth recent encouragng progress made n one specal case [2], for the so-called weghted-star metrc. Our results provde a smple mnmax optmal algorthm for ths problem. Combnatoral Predcton Market Desgn: There has been a great deal of work n desgnng socalled predcton markets, where bettors may purchase contracts that pay off when the outcome of a future event s correctly guessed. One mportant goal of such markets s to mnmze the potental rsk of the market maker who sells the contracts and pays the wnnng bettors. Another goal s to desgn combnatoral markets, that s where the outcome space mght be complex. The latter has proven qute challengng, and there are few postve results wthn ths area. We show how to translate the market-desgn problem nto a Budgeted Adversary problem, and from here how to ncorporate certan knds of combnatoral outcomes. 2 Prelmnares Notaton: We shall wrte [n] for the set {, 2,..., n}, and [n] to be the set of all fnte-length sequences of elements of [n]. We wll use the greek symbols ρ and σ to denote such sequences 2... T, where t [n]. We let denote the empty sequence. When we have defned some T -length sequence ρ = 2... T, we may wrte ρ t to refer to the t-length prefx of ρ, namely ρ t = 2... t, and clearly t T. We wll generally use w to refer to a dstrbuton n n, the n-smplex, where w denotes the th coordnate of w. We use the symbol e to denote the th bass vector n n dmensons, namely a vector wth a n the th coordnate, and 0 s elsewhere. We shall use [ ] to denote the ndcator functon, where [predcate] s f predcate s true, and 0 f t s false. It may be that predcate s a random varable, n whch case [predcate] s a random varable as well. 2. The Settng: Budgeted Adversary Games We wll now descrbe the generc sequental decson problem, where a problem nstance s characterzed by the followng trple: an n n loss matrx M, a monotonc cost functon cost : [n] R+, and a cost budget k. A cost functon s monotonc as long as t satsfes the relaton cost(ρσ) cost(ρσ) for all ρ, σ [n] and all [n]. Play proceeds as follows:. On each round t, the player chooses a dstrbuton w t n over hs acton space. 2. An outcome t [n] s chosen by Nature (potentally an adversary). 3. The player suffers w t Me t. 4. The game proceeds untl the frst round n whch the budget s spent,.e. the round T when cost( 2... T ) k < cost( 2... T T ). The goal of the Player s to choose each w t n order to mnmze the total cost of ths repeated game on all sequences of outcomes. Note, mportantly, that the player can learn from the past, and hence would lke an effcently computable functon w : [n] n, where on round t the player s gven ρ t = (... t ) and sets w t w(ρ t ). We can defne the worst-case cost of an algorthm 2

3 w : [n] n by ts performance aganst a worst-case sequence, that s WorstCaseLoss(w; M, cost, k) := max ρ = 2... [n] cost(ρ T ) k < cost(ρ T ) w(ρ t ) Me t. Note that above T s a parameter chosen accordng to ρ and the budget. We can also defne the mnmax loss, whch s defned by choosng the w( ) whch mnmzes WorstCaseLoss( ). Specfcally, MnmaxLoss(M, cost, k) := mn w:[n] n t= max ρ = 2... [n] cost(ρ T ) k < cost(ρ T ) w(ρ t ) Me t. In the next secton, we descrbe the optmal algorthm for a restrcted class of M. That s, we obtan the mappng w whch optmzes WorstCaseLoss(w; M, cost, k). 3 The Algorthm We wll start by assumng that M s a nonnegatve dagonal matrx, that s M = dag(c, c 2,..., c n ), and c > 0 for all. Wth these values c, defne the dstrbuton q n wth q := Pj /c. /cj Gven a current state ρ, the algorthm wll rely heavly on our ablty to compute the followng functon Φ( ). For any ρ [n] such that cost(ρ) > k, defne Φ(ρ) := 0. Otherwse, let [ ] Φ(ρ) := /c E [cost(ρ... t ) k] t: t q Notce, ths s the expected length of a random process. Of course, we must mpose the natural condton that the length of ths process has a fnte expectaton. Also, snce we assume that the cost ncreases, t s reasonable to requre that the dstrbuton over the length,.e. mn{t : cost(ρ... t ) > k}, has an exponentally decayng tal. Under these weak condtons, the followng m-tral Monte Carlo method wll provde a hgh probablty estmate to error wthn O(m /2 ). Algorthm Effcent Estmaton of Φ(ρ) for =... m do Sample: nfnte random sequence σ := 2... where Pr( t = ) = q Let: T = max{t : cost(ρσ t ) k} end forp m = Return T m Notce that the nfnte sequence σ does not have to be fully generated. Instead, we can contnue to sample the sequence and smply stop when the condton cost(ρσ t ) k s reached. We can now defne our algorthm n terms of Φ( ). Algorthm 2 Player s optmal strategy Input: state ρ Compute: Φ(ρ), Φ(ρ, ), Φ(ρ, 2),..., Φ(ρ, n) Let: set w(ρ) wth values w (ρ) = Φ(ρ) Φ(ρ,) c t=0 t= 4 Mnmax Optmalty Now we prove that Algorthm 2 s both legal and mnmax optmal. Lemma 4.. The vector w(ρ) computed n Algorthm 2 s always a vald dstrbuton. 3

4 Proof. It must frst be establshed that w (ρ) 0 for all and ρ. Ths, however, follows because we assume that the functon cost() s monotonc, whch mples that cost(ρσ) cost(ρσ) and hence cost(ρσ) k = cost(ρσ) k, and hence [cost(ρσ) k] [cost(ρσ) k]. Takng the expected dfference of the nfnte sum of these two ndcators leads to Φ(ρ) Φ(ρ) 0, whch mples w (ρ) 0 as desred. We must also show that w (ρ) =. We clam that the followng recurrence relaton holds for the functon Φ(ρ) whenever cost(ρ) k: Φ(ρ) = /c + q Φ(ρ), for any ρ s.t. cost(ρ) < k. }{{} }{{} frst step remanng steps Ths s clear from notcng that Φ s an expected random walk length, wth transton probabltes defned by q, and scaled by the constant ( /c ). Hence, w (ρ) = ( ) Φ(ρ) Φ(ρ) = /c Φ(ρ) Φ(ρ) c c ( ) ( = /c /c + ) q Φ(ρ) Φ(ρ) = c where the last equalty holds because q = Pj /c. /cj Theorem 4.. For M = dag(c,..., c n ), Algorthm 2 s mnmax optmal for the Budgeted Adversary problem. Furthermore, Φ( ) = MnmaxLoss(M, cost, k). Proof. Frst we prove an upper bound. Notce that, for an sequence ρ = T, the total cost of Algorthm 2 wll be w(ρ t ) Me t = t= w t (ρ t )c t = t= and hence the total cost of the algorthm s always bounded by Φ( ). t= Φ(ρ t ) Φ(ρ t ) c t c t = Φ( ) Φ(ρ T ) Φ( ) On the other hand, we clam that Φ( ) can always be acheved by an adversary for any algorthm w ( ). Construct a sequence ρ as follows. Gven that ρ t has been constructed so far, select any coordnate t [n] for whch w t (ρ t ) w t (ρ t ), that s, where the the algorthm w places at least as much weght on t as the proposed algorthm w we defned n Algorthm 2. Ths must always be possble because both w(ρ t ) and w (ρ t ) are dstrbutons and nether can fully domnate the other. Set ρ t ρ t. Contnue constructng ρ untl the budget s reached,.e. cost(ρ) > k. Now, let us check the loss of w on ths sequence ρ: w (ρ t ) Me t = t= w t (ρ t )c t t= w t (ρ t )c t = Φ( ) Φ(ρ) = Φ( ) Hence, an adversary can acheve at least Φ( ) loss for any algorthm w. 4. Extensons For smplcty of exposton, we proved Theorem 4. under a somewhat lmted scope: only for dagonal matrces M, known budget k and cost(). But wth some work, these restrctons can be lfted. We sketch a few extensons of the result, although we omt the detals due to lack of space. Frst, the concept of a cost() functon and a budget k s not entrely necessary. Indeed, we can redefne the Budgeted Adversary game n terms of an arbtrary stoppng crteron δ : [n] {0, }, where δ(ρ) = 0 s equvalent to the budget has been exceeded. The only requrement s that δ() s monotonc, whch s naturally defned as δ(ρσ) = = δ(ρσ) = for all ρ, σ [n] and all [n]. Ths alternatve budget nterpretaton lets us consder the sequence ρ as a path through t= 4

5 a game tree. At a gven node ρ t of the tree, the adversary s acton t+ determnes whch branch to follow. As soon as δ(ρ t ) = 0 we have reached a termnal node of ths tree. Second, we need not assume that the budget k, or even the generalzed stoppng crteron δ(), s known n advance. Instead, we can work wth the followng generalzaton: the stoppng crteron δ s drawn from a known pror λ and gven to the adversary before the start of the game. The resultng optmal algorthm depends smply on estmatng a new verson of Φ(ρ). Φ(ρ) s now redefned as both an expectaton over a random σ and a random δ drawn from the posteror of λ, that s where we condton on the event δ(ρ) =. Thrd, Theorem 4. can be extended to a more general class of M, namely nverse-nonnegatve matrces, where M s nvertble and M has all nonnegatve entres. (In all the examples we gve we need only dagonal M, but we sketch ths generalzaton for completeness). If we let n be the vector of n ones, then defne D = dag (M n ), whch s a nonnegatve dagonal matrx. Also let N = DM and notce that the rows of N are the normalzed rows of M. We can use Algorthm 2 wth the dagonal matrx D, and attan dstrbuton w (ρ) for any ρ. To obtan an algorthm for the matrx M (not D), we smply let w(ρ) = (w (ρ) N), whch s guaranteed to be a dstrbuton. The loss of w s dentcal to w snce w(ρ) M = w (ρ) D by constructon. Fourth, we have only dscussed mnmzng loss aganst a budgeted adversary. But all the results can be extended easly to the case where the player s nstead maxmzng gan (and the adversary s mnmzng). A partcularly surprsng result s that the mnmax strategy s dentcal n ether case; that s, the the recursve defnton of w (ρ) s the same whether the player s maxmzng or mnmzng. However, the termnaton condton mght change dependng on whether we are mnmzng or maxmzng. For example n the expert settng, the game stops when all experts have cost larger than k versus at least one expert has gan at least k. Therefore for the same budget sze k, the mnmax value of the gan verson s typcally smaller than the value of the loss verson. Smplfed Notaton. For many examples, ncludng two that we consder below, recordng the entre sequence ρ s unnecessary the only relevant nformaton s the number of tmes each occurs n ρ and not where t occurs. Ths s the case precsely when the functon cost(ρ) s unchanged up to permutatons of ρ. In such stuatons, we can consder a smaller state space, whch records the counts of each n the sequence ρ. We wll use the notaton s N n, where s t = e e t for the sequence ρ t = 2... t. 5 The Cost-Senstve Hedge Settng A straghtforward applcaton of Budgeted Adversary games s the Hedge settng ntroduced by Freund and Schapre [0], a verson of the aforementoned experts settng. The mnmax algorthm for ths specal case was already thoroughly developed by Abernethy et al []. We descrbe an nterestng extenson that can be acheved usng our technques whch has not yet been solved. The Hedge game goes as follows. A learner must predct a sequence of dstrbutons w t n, and receve a sequence of loss vectors l t {0, } n. The total loss to the learner s t w t l t, and the game ceases only once the best expert has more than k errors,.e. mn t l t, > k. The learner wants to mnmze hs total loss. The natural way to transform the Hedge game nto a Budgeted Adversary problem s as follows. We ll let s be the state, defned as the vector of cumulatve losses of all the experts. [ ] M =... cost(s) = mn s w t l t = wt Me t t t The proposed reducton almost works, except for one key ssue: ths only allows cost vectors of the form l t = Me t = e t, snce by defnton Nature chooses columns of M. However, as shown n Abernethy et al, ths s not a problem. Lemma 5. (Lemma and Theorem 2 of []). In the Hedge game, the worst case adversary always chooses l t {e,..., e n }. The standard and more well-known, although non-mnmax, algorthm for the Hedge settng [0] uses a smple modfcaton of the Weghted Majorty Algorthm [4], and s descrbed smply by 5

6 settng w (s) = exp( ηs) Pj exp( ηsj). Wth the approprate tunng of η, t s possble to bound the total loss of ths algorthm by k + 2k ln n + ln n, whch s known to be roughly optmal n the lmt. Abernethy et al [] provde the mnmax optmal algorthm, but state the bound n terms of an expected length of a random walk. Ths s essentally equvalent to our descrpton of the mnmax cost n terms of Φ( ). A sgnfcant drawback of the Hedge result, however, s that t requres the losses to be unformly bounded n [0, ], that s l t [0, ] n. Ideally, we would lke an algorthm and a bound that can handle non-unform cost ranges,.e. where expert suffers loss n some range [0, c ]. The l t, [0, ] assumpton s fundamental to the Hedge analyss, and we see no smple way of modfyng t to acheve a tght bound. The smplest trck, whch s just to take c max := max c, leads to a bound of the form k + 2c max k ln n + c max ln n whch we know to be very loose. Intutvely, ths s because only a sngle rsky expert, wth a large c, should not affect the bound sgnfcantly. In our Budgeted Adversary framework, ths case can be dealt wth trvally: lettng M = dag(c,..., c n ) and cost(s) = mn s c gves us mmedately an optmal algorthm that, by Theorem 4., we know to be mnmax optmal. Accordng to the same theorem, the mnmax loss bound s smply Φ( ) whch, unfortunately, s n terms of a random walk length. We do not know how to obtan a closed form estmate of ths expresson, and we leave ths as an ntrgung open queston. 6 Metrcal Task Systems A classc problem from the Onlne Algorthms communty s known as Metrcal Task Systems (MTS), whch we now descrbe. A player (decson-maker, algorthm, etc.) s presented wth a fnte metrc space and on each of a sequence of rounds wll occupy a sngle state (or pont) wthn ths metrc space. At the begnnng of each round the player s presented wth a cost vector, descrbng the cost of occupyng each pont n the metrc space. The player has the opton to reman at the hs present state and pay ths states assocated cost, or he can decde to swtch to another pont n the metrc and pay the cost of the new state. In the latter case, however, the player must also pay the swtchng cost whch s exactly the metrc dstance between the two ponts. The MTS problem s a useful abstracton for a number of problems; among these s job-schedulng. An algorthm would lke to determne on whch machne, across a large network, t should process a job. At any gven tme pont, the algorthm observes the number of avalable cycles on each machne, and can choose to mgrate the job to another machne. Of course, f the subsequent machne s a great dstance, then the algorthm also pays the travel tme of the job mgraton through the network. Notce that, were we gven a sequence of cost vectors n advance, we could compute the optmal path of the algorthm that mnmzed total cost. Indeed, ths s effcently solved by dynamc programmng, and we wll refer to ths as the optmal offlne cost, or just the offlne cost. What we would lke s an algorthm that performs well relatve to the offlne cost wthout knowledge of the sequence of cost vectors. The standard measure of performance for an onlne algorthm s the compettve rato, whch s the rato of cost of the onlne algorthm to the optmal offlne cost. For all the results dscussed below, we assume that the onlne algorthm can mantan a randomzed state a dstrbuton over the metrc and pays the expected cost accordng to ths random choce (Randomzed algorthms tend to exhbt much better compettve ratos than determnstc algorthms). When the metrc s unform,.e. where all pars of ponts are at unt dstance, t s known that the compettve rato s O(log n), where n s the number of ponts n the metrc; ths was shown by Borodn, Lnal and Saks who ntroduced the problem [4]. For general metrc spaces, Bartal et al acheved a compettve rato of O(log 6 n) [3], and ths was mproved to O(log 2 n) by Fat and Mendel [9]. The latter two technques, however, rely on a scheme of randomly approxmatng the metrc space wth a herarchcal tree metrc, addng a (lkely-unnecessary) multplcatve cost factor of log n. It s wdely beleved that the mnmax compettve rato s O(log n) n general, but ths gap has remaned elusve for at least 0 years. The most sgnfcant progress towards O(log n) s the 2007 work of Bansal et al [2] who acheved such a rato for the case of weghted-star metrcs. A weghted star s a metrc such that each pont has a fxed dstance d from some center state, and travelng between any state and j requres 6

7 gong through the center, hence ncurrng a swtchng cost of d + d j. For weghted-star metrcs, Bansal et al managed to justfy two smplfcatons whch are qute useful:. We can assume that the cost vector s of the form 0,...,,..., 0 ; that s, all state receve 0 cost, except some state whch receves an nfnte cost. 2. When the onlne algorthm s currently mantanng a dstrbuton w over the metrc, and an nfnte cost occurs at state, we can assume that algorthm ncurs exactly 2d w, exactly the cost of havng w probablty weght enter and leave from the center. Bansal et al provde an effcent algorthm for ths settng usng prmal-dual technques developed for solvng lnear programs. Wth the methods developed n the present paper, however, we can gve the mnmax optmal onlne algorthm under the above smplfcatons. Notce that the adversary s now choosng a sequence of states, 2, 3... [n] at whch to assgn an nfnte cost. If we let ρ = , then the onlne algorthm s job s to choose a sequence of dstrbutons w(ρ t ), and pays 2d t+ w t+ (ρ t ) at each step. In the end, the onlne algorthm s cost s compared to the offlne MTS cost of ρ, whch we wll call cost(ρ). Assume 2 we know the cost of the offlne n advance, say t s k, and let us defne M = dag(2d,..., 2d n ). Then the player s job s to select an algorthm w whch mnmzes max w(ρ t ) Me t. ρ = (,..., T ) t= cost(ρ) k As we have shown, Algorthm 2 s( mnmax optmal for ths settng. The compettve rato of ths algorthm s precsely lm sup k k MnmaxLoss(M, cost, k)). Notce the convenent trck here: by boundng a pror the cost of the offlne at k, we can smply magne playng ths repeated game untl the budget k s acheved. Then the compettve rato s just the worst-case loss over the offlne cost, k. On the downsde, we don t know of any easy way to bound the worst-case loss Φ( ). 7 Combnatoral Informaton Markets We now consder the desgn of so-called cost-functon-based nformaton markets, a popular type of predcton market. Ths work s well-developed by Chen and Pennock [7], wth much useful dscusson by Chen and Vaughn [8]. We refer the reader to the latter work, whch provdes a very clear pcture of the nce relatonshp between onlne learnng and the desgn of nformaton markets. In the smplest settng, a predcton market s a mechansm for sellng n types of contract, where a contract of type corresponds to some potental future outcome, say event wll occur. The standard assumpton s that the set of possble outcomes are mutually exclusve, so only one of the n events wll occur for example, a pendng electon wth n competng canddates and one eventual wnner. When a bettor purchases a contract of type, the manager of the market, or market maker, promses to pay out $ f the outcome s and $0 otherwse. A popular research queston n recent years s how to desgn such predcton markets when the outcome has a combnatoral structure. An electon mght produce a complex outcome lke a group of canddates wnnng, and a bettor may desre to bet on a complex predcate, such as none of the wnnng canddates wll be from my state. Ths queston s explored n Hanson [3], although wthout much dscusson of the relevant computatonal ssues. The computatonal aspects of combnatoral nformaton markets are addressed n Chen et al [6], who provde a partcular hardness result regardng computaton of certan prce functons, as well as a postve result for an alternatve type of combnatoral market. In the present secton, we propose a new technque for desgnng combnatoral markets usng the technques lad out n the present work. In ths type of nformaton market, the task of a market maker s to choose a prce for each of the n contracts, but where the prces may be set adaptvely accordng to the present demand. Let s N n denote the current volume, where s s the number of contracts sold of type. In a costfuncton-based market, these prces are set accordng to a gven convex cost functon C(s) whch Precsely, they clam that t should be upper-bounded by 4d. We omt the detals regardng ths ssue, but t only contrbutes a multplcatve factor of 2 to the compettve rato. 2 Even when we do not know the offlne cost n advance, standard doublng trcks allow you to guess ths value and ncrease the guess as the game proceeds. For space, we omt these detals. 7

8 represents a potental on the demand. It s assumed that C( ) satsfes the relaton C(s + α) = C(s) + α for all s, and α > 0 and 2 C > 0. A typcal example of such a cost functon s C(s) = s 2 b log n = exp(s /b) where b s a parameter (see Chen and Pennock for further dscusson [7]); t s easy to check ths functon satsfes the desred propertes. Gven the current volume s, the prce of contract s set at C(s + e ) C(s). Ths prcng scheme has the advantage that the total money earned n ths market s easy to compute: t s exactly C(s) regardless of the order n whch the contracts were purchased. A dsadvantage of ths market, however, s that the posted prces (typcally) sum to greater than $! A prmary goal of an nformaton market s to ncentvze bettors to reveal ther prvate knowledge of the outcome of an event. If a gven bettor beleves the true dstrbuton of the outcome to be q n, he wll have an ncentve to purchase any contract for whch the current prce p s smaller than q, thus provdng postve expected reward (relatve to hs predcted dstrbuton). Usng ths cost-functon scheme, t s possble that q < C(s + e ) C(s) for all and hence a bettor wll have no ncentve to bet. We propose nstead an alternatve market mechansm that avods ths dffculty: for every gven volume state s, the market maker wll advertse a prce vector w(s) n. If a contract of type s purchased, the state proceeds to s + e, and the market maker earns w (s). If a sequence of contracts 2... s purchased, the market maker s total earnng s t w(e e t ) e t. On the other hand, f the fnal demand s s, n the worst case the market maker may have to payout a total of max s dollars. If we assume the market maker has a fxed budget k on the max number of contracts he s wllng to sell, and wants to maxmze the total earned money from sellng contracts subject to ths constrant, then we have 3 exactly a Budgeted Adversary problem: let M be the dentty and let cost(s) := max s. Ths looks qute smlar to the Budgeted Adversary reducton n the Hedge Settng descrbed above, whch s perhaps not too surprsng gven the strong connectons dscovered n Chen and Vaughn [8] between learnng wth experts and market desgn. But ths reducton gves us addtonal power: we now have a natural way to desgn combnatoral predcton markets. We sketch one such example, but we note that many more can be worked out also. Assume we are n a settng where we have n electon canddates, but some subset of sze m < n wll become the wnners, and any such subset s possble. In ths case, we can magne a market maker sellng a contract of type wth the followng promse: f canddate s n the wnnng subset, the payout s /m and 0 otherwse. For smlar reasons as above, the market maker should sell contracts at prces p where p =. If we assume that market maker has a budget constrant of k for the fnal payout, then we can handle ths new settng wthn the Budgeted Adversary framework by smply modfyng the cost functon approprately: cost(s) = s max U [n], U =m m. U Ths soluton looks qute smple, so what dd we gan? The beneft of our Budgeted Adversary framework s that we can handle arbtrary monotonc budget constrants, and the combnatoral nature of ths problem can be encoded wthn the budget. We showed ths for the case of subset bettng, but t can be appled to a wde range of settngs wth combnatoral outcomes. 8 Open problem We have provded a very general framework for solvng repeated zero-sum games aganst a budgeted adversary. Unfortunately, the generalty of these results only go as far as games wth payoff matrces that are nverse-nonnegatve. For one-shot games, of course, Von Neumann s mnmax theorem leads us to an effcent algorthm,.e. lnear programmng, whch can handle any payoff matrx, and we would hope ths s achevable here. We thus pose the followng open queston: Is there an effcent algorthm for solvng Budgeted Adversary games for arbtrary matrces M? 3 The careful reader may notce that ths modfed model may lead to a problem not present n the costfuncton based markets: an arbtrage opportunty for the bettors. Ths ssue can be dealt wth by ncludng a suffcent transacton fee per contract, but we omt these detals due to space constrants. 8

9 References [] J. Abernethy, M. K. Warmuth, and J. Yelln. Optmal strateges from random walks. In Proceedngs of the 2st Annual Conference on Learnng Theory (COLT 08), pages , July [2] Nkhl Bansal, Nv Buchbnder, and Joseph (Seff) Naor. A Prmal-Dual randomzed algorthm for weghted pagng. In Proceedngs of the 48th Annual IEEE Symposum on Foundatons of Computer Scence, pages IEEE Computer Socety, [3] Y. Bartal, A. Blum, C. Burch, and A. Tomkns. A polylog (n)-compettve algorthm for metrcal task systems. In Proceedngs of the twenty-nnth annual ACM symposum on Theory of computng, page 779, 997. [4] A. Borodn, N. Lnal, and M. E Saks. An optmal on-lne algorthm for metrcal task system. Journal of the ACM (JACM), 39(4):745763, 992. [5] B. Brügmann. Monte carlo go. Master s Thess, Unpublshed, 993. [6] Y. Chen, L. Fortnow, N. Lambert, D. M Pennock, and J. Wortman. Complexty of combnatoral market makers. In Proceedngs of the ACM Conference on Electronc Commerce (EC), [7] Y. Chen and D. M Pennock. A utlty framework for bounded-loss market makers. In Proceedngs of the 23rd Conference on Uncertanty n Artfcal Intellgence, page 4956, [8] Y. Chen and J. W Vaughan. A new understandng of predcton markets va No-Regret learnng. Arxv preprnt arxv: , 200. [9] A. Fat and M. Mendel. Better algorthms for unfar metrcal task systems and applcatons. In Proceedngs of the thrty-second annual ACM symposum on Theory of computng, page , [0] Yoav Freund and Robert E. Schapre. A decson-theoretc generalzaton of on-lne learnng and an applcaton to Boostng. J. Comput. Syst. Sc., 55():9 39, 997. Specal Issue for EuroCOLT 95. [] S. Gelly and D. Slver. Combnng onlne and offlne knowledge n UCT. In Proceedngs of the 24th nternatonal conference on Machne learnng, page 280, [2] S. Gelly, Y. Wang, R. Munos, and O. Teytaud. Modfcaton of UCT wth patterns n Monte- Carlo go [3] R. Hanson. Combnatoral nformaton market desgn. Informaton Systems Fronters, 5():079, [4] N. Lttlestone and M. K. Warmuth. The Weghted Majorty algorthm. Inform. Comput., 08(2):22 26, 994. Prelmnary verson n FOCS 89. 9