Game theory for. Leonardo Badia.

Transcription

1 Game theory for information engineering Leonardo Badia

2 Zero-sum games A special class of games, easier to solve

3 Zero-sum We speak of zero-sum game if u i (s) = -u -i (s). player A player B L C R T -9,9 8,-8-5,5 M -2,2 6,-6 2,-2 D -1,1 3,-3 4,-4 The odd/even game and rock/paper/scissors are zero-sum games.

4 Minimax Theorem (1) Consider a zero-sum game G. (1) G has a NE iff maxmin i = minmax i for each i (2) All NEs yield the same payoff ( = maxmin i ) (3) All NEs have the form (s i*,s -i*), where s i* is a security strategy player B player A L C R T -9, 9 8, -8-5, 5 M -2, 2 6, -6 2, -2 D -1, 1 3, -3 4, -4 for player A: maxmin = -1 minmax = -1 (L,D) is a NE, u A = -1

5 Remarks Since the game is zero-sum, it is sufficient to check the maxmin = minmax condition for one player only. It also holds maxmin i = minmax -i minmax i = maxmin -i The common value of maxmin 1 = minmax 1 is called the value of the game. A joint security strategy (if any), i.e., a NE, is called a saddle point of the game.

6 Remarks The bi-matrix for this special kind of games can be represented with a regular matrix (utility of player -i is implicit). The proof of the theorem is due to von Neumann (1928) and makes use of linear programming (constrained optimization). The criterion of minmaximizing the utility has been widely employed in artificial intelligence applications: e.g., chess, which is a zero-sum (although sequential) game.

7 Case closed? In general, finding Nash Equilibria is tricky; but in the special case of zero-sum games we have a nice criterion (necessary+sufficient). However, it does not seem to be useful if the maxmin=minmax condition fails. In the odd/even game, maxmin = -4 < minmax = 4 So? Odd Even , 4 4, , -4-4, 4

8 Mixed strategies Simple games get fuzzy

9 Missing outcome Expand the Odd/Even game to find its outcome Odd Even 0 0 ½ , -4, 4 4 0, 0 4, -4 4, -4 ½ 1 0, 0 4, -4 0, 0 0, 0-4, 4 1 4, -4 0, 0-4, 4 It seems that (½,½) is a NE. Let formalize this.

10 Mixed strategies If A is a non-empty discrete set, a probability distribution over A is a function p : A [0,1], such that x A p(x) = 1 The set of possible p.d. s over A is denoted as ΔA For a normal form game (S 1,,S n ; u 1,,u n ), a mixed strategy for player i is a probability distribution m i over set S i. That is, i chooses strategies in S i = (s i,1,,s i,n ) with probabilities (m i (s i,1 ),, m i (s i,n ) ).

11 Payoff The utility function u i can be extended to a real function over ΔS 1 ΔS 2 ΔS n If players choose mixed strategies (m 1,,m n ), compute player i s payoff by weighing on m i s. u i (m 1,,m n ) = s S m 1 (s 1 ) m 2 (s 2 ) m n (s n ) u i (s) In other words: fix a global strategy s compute its probability weigh the utility of s on this probability and sum

12 Intuition Consider the odd/even game and assume Odd decides to play 0 with probability q, while Even plays 0 with probability r. Consequently 1 is played by Odd and Even with probability 1-q and 1-r, respectively. Odd 0 (prob r ) Even the table shows a single global strategies s = (q, r ) 1 (prob 1-r ) 0 (prob q) -4qr, 4qr 4q(1-r), -4q(1-r) 1 (prob 1-q) 4(1-q)r, -4(1-q)r -4(1-q)(1-r), 4(1-q)(1-r)

13 Intuition In other words, we revise the game so that each player can choose not only either 0 or 1, but also a value between them: q for Odd, r for Even. Odd s payoff is -16qr +8q +8r -4 = -4(2q-1)(2r-1) Odd 0 (prob r ) Even 1 (prob 1-r ) 0 (prob q) -4qr, 4qr 4q(1-r), -4q(1-r) 1 (prob 1-q) 4(1-q)r, -4(1-q)r -4(1-q)(1-r), 4(1-q)(1-r)

14 Intuition In other words, we revise the game so that each player can choose not only either 0 or 1, but also a value between them: q for Odd, r for Even. Odd s payoff is -16qr +8q +8r -4 = -4(2q-1)(2r-1) 0 Even 0 r 1 Odd q -16qr +8q +8r -4, 16qr -8q -8r +4 1

15 Pure strategies Given a mixed strategy m i ΔS i we define the support of m i as {s i S i : m i (s i ) > 0 } Each strategy s i S i (an element of S i ) can be identified with the mixed strategy p (which is an element of ΔS i ) such that p(s i ) = 1 Hence, p(s i ) = 0 if s i s i and also support(p)={s i } Thereafter, we identify p with s i. In this context, s i is called a pure strategy. Previous definitions of dominance and NE only refer to the pure strategy case.

16 Strict/weak dominance Consider game G = {S 1,,S n ; u 1,,u n }. If m i, m i S i, m i strictly dominates m i if u i (m i,m -i ) > u i (m i,m -i ) for every m -i We say that m i weakly dominates m i if u i (m i,m -i ) u i (m i,m -i ) for every m -i u i (m i,m -i ) > u i (m i,m -i ) for some m -i Note: there are infinitely (and continuously) many m -i in the set: S 1 S i-1 S i+1 S n

17 Strict/weak dominance However, it is possible to prove that: If m i, m i S i, m i strictly dominates m i if u i (m i,s -i ) > u i (m i,s -i ) for every s -i S -i Similarly, m i weakly dominates m i if u i (m i,s -i ) u i (m i,s -i ) for every s -i S -i u i (m i,s -i ) > u i (m i,s -i ) for some s -i S -i That is, we can limit our search to pure strategies of the opponents.

18 Nash equilibrium Consider game G = {S 1,,S n ; u 1,,u n }. A joint mixed strategy m S 1 S n is said to be a Nash equilibrium if for all i : u i (m) > u i (m i i,m -i ) for every m i i S i This reprise the same concept of NE in pure strategies: no player has an incentive to change his/her move (which is a mixed strategy now)

19 back to Example 3 In the Odd/Even game, the payoff for Odd is -4(2q - 1) (2r - 1), the opposite for Even. If q = ½, or r = ½, both players have payoff 0. If q = r = ½ no player has incentive to change. Odd Even 0 ½ 1 0 0, 0 0, 0 ½ 0, 0 0, 0 0, 0 0, 0 0, 0 0, 0 1 0, 0 Nash equilibrium

20 back to Example 3 As an exercise, prove that (½, ½) is the only Nash Equilibrium of the Odd/Even game. How to proceed. Consider three cases, where the payoff of player Odd is <0, >0, =0 but joint strategy is not (½, ½). Show that in each case there is a player (who?) having an incentive in changing strategy. None of this is a NE. (½, ½) is the only one.

21 Using mixed strategies and introducing the Nash theorem

22 IES vs mixed strategies pl layer A player B L C R T 7, 4 5, 0 8, 1 D 6, 0 3, 4 9, 1 Nash equilibrium R is not dominated by L or C. But mixed strategy m = ½L + ½C gets payoff 2 regardless of A s move. Pure strategy R is strictly dominated by m and can be eliminated. Further eliminations are possible.

23 IES vs mixed strategies Similar theorems to the pure strategy case hold for IES in mixed strategies (IESM). Theorem. Nash equilibria survive IESM. Theorem. The order of IESM is irrelevant. Note. Strict (not weak) dominance must be used. A weakly dominated strategy can be a NE, or belong to the support of a NE.

24 Characterization Theorem. Take a game G = {S 1,,S n ; u 1,,u n } and a joint mixed strategy m for game G. The following statements are equivalent: (1) Joint mixed strategy m is a Nash equilibrium. (2) For each i: u i (m) = u i (s i, m -i ) for every s i support(m i ) u i (m) u i (s i, m -i ) for every s i support(m i ) Corollary. If a pure strategy is a NE, it is such also as a mixed strategy.

25 back to Example 5 R Brian S Ann R 2, 1 0, 0 S 0, 0 1, 2 This game had two pure NEs: (R,R) and (S,S). We show now that there is also a mixed NE. Ann (or Brian) plays R with probabilities q (or r). A mixed strategy is uniquely identified by (q,r). Ann s payoff is u A (q,r) = 2qr + (1-q)(1-r) Brian s is u B (q,r) = qr + 2(1-q)(1-r)

26 back to Example 5 Assume (a,b) is a mixed NE. Note: support (a) = support (b) = {R,S}. Pure strategies R/S correspond with q (or r) being 0/1. Due to the Theorem, u A (a,b) = u A (0,b) = u A (1,b) Now, use: u A (q,r) = 2qr + (1-q)(1-r) 2ab + (1-a)(1-b) = 1-b = 2b Solution: b = ⅓. Similarly, u A (a,0) = u A (a,1) Solution: a = ⅔.

27 Nash theorem (intro) The reasoning we used to find the third (mixed) NE of the Battle of Sexes is more general. Every two-player games with two strategies has a NE in mixed strategies. This is easy to prove and is part of the more general Nash theorem. Theorem (Nash 1950). Every game with finite S i s has at least one Nash equilibrium (possibly involving mixed strategies).

28 Critics of mixed strategy Mixed strategies are important for Nash Theorem. However, many authors criticize mixed strategies as probabilities. In the end, players take a pure strategy. Possible alternative interpretations. Large numbers. Probability q means: if we play the game M times, a pure strategy is chosen qm times (note: each of the M times is one-shot memoryless) Fuzzy values. Actions are not simply white/black. Beliefs. The probability q reflects the uncertainty that my opponent has about my choice (which is pure).

29 NE as best responses Using beliefs, we can speak of best response to an opponent s (mixed) strategy. Intuition. F Bea G Art U 6, 1 0, 4 D 2, 5 4, 0 Bea ignores what Art will play, so she assumes he will play U with probability q. Similarly, Art thinks Bea will play F with probability r.

30 NE as best responses F Bea G Art U 6, 1 0, 4 D 2, 5 4, 0 E.g., if Bea is known for always playing F (r =1), Art s best response is to play U (q =1). In general? It holds: u A (D,r ) = 2r + 4(1-r ), u A (U,r ) = 6r U is actually Art s best response as long as r > ½, else it is D. If r = ½ they are equivalent. Denote Art s best response with q*(r).

31 NE as best responses q 1 q*(r ) 0 ½ 1 r For Bea: u B (q,f ) = q + 5(1-q ), u B (q,g ) = 4q Thus, Bea s best response r*(q) is step-wise r*(q) = 1 if q <⅝, r*(q) = 0 if q >⅝

32 NE as best responses q 1 ⅝ (mixed) Nash equilibrium q*(r ) r*(q ) 0 ½ 1 r Joint For Bea: strategy u B (q,f m ) = (q q =½, + 5(1-q r =⅝) ), uis B (q,g a NE. ) = 4q NE Thus, are Bea s points best were response the choice r*(q) of is each step-wise player is the best r*(q) response = 1 if q <⅝, to the other r*(q) player s = 0 if q choice. >⅝

33 Existence of NE Clearly, the existence of at least one NE is guaranteed by topological reasons. There may be more NEs (e.g. Battle of Sexes). R Brian S Ann R 2, 1 0, 0 S 0, 0 1, 2 u A (R,r ) = 2r, u A (S,r ) = 1 r, q*(r ) = 1 - h(r -⅓) u B (q,r) = q, u B (q,s) = 2(1- q), r*(q ) = 1 - h(q -⅔)

34 Existence of NE q 1 (pure) Nash equilibria ⅔ q*(r ) (mixed) Nash equilibrium r*(q ) 0 ⅓ 1 Anyway, q*(r) must intersect r*(q) at least once. The Nash theorem generalizes this reasoning. r

35 The Nash theorem Consider game G = {S 1,,S n ; u 1,,u n }. Define: best i : ΔS 1 ΔS i-1 ΔS i+1 ΔS n ΔS i best i (m -i ) = {m i ΔS i : u i (m i, m -i ) is maximal } Then define best : ΔS ΔS as best(m) = best 1 (m -1 ) best n (m -n ) That is, best i (m -i ) is the set of best responses of user i to moves m -i by other players. Aggregating them, we obtain best. m is a NE if m best(m) Note. best i (m -i ) is always non-empty and always contains at least a pure strategy.

36 The Nash theorem Lemma (Kakutani Theorem). Let A be a compact and convex subset of R n. If F : A A is s.t.: For all x A, F(x) is non-empty and convex Let {x i } and {y i } be sequences converging to x and y, respectively. If y i F(x i ) then y F(x). Then there exists x* A such that x* F(x*). Nash theorem. It s nothing but Kakutani theorem applied to previously defined function best.

37 Mixed maxmin/minmax the extensions to mixed strategies

38 Mixed security strategy Consider a two- player game ( i vs -i ), and take f i :ΔS i R as f i (m i ) = min m-i ΔS -i u i (m i,m -i ) Any mixed strategy m i * maximizing f i (m i ) is a mixed security strategy for i. This max, i.e. max mi ΔS i min m-i Δ S -i u i (m i,m -i ) is the maxmin im or the mixed security payoff of i. A mixed security strategy is the conservative mixed strategy guaranteeing the highest payoff for i in case of the worst mixed strategy by -i.

39 Mixed minmax Also if F i :ΔS -i R is F i (m -i ) = max mi ΔS i u i (m i,m -i ) min m-i ΔS -i F i (m -i ) = min m-i ΔS -i max mi ΔS i u i (m i,m -i ) is the minmax for i in mixed strategy, minmax im. If i could move after -i, there is a mixed strategy which guarantees i to achieve at least minmax m i. Note 1. It can be shown that f i (m i ) can be found minimizing u i (m i,s -i ), i.e., using pure strategies only. Equally, F i (m -i ) can be defined maximizing u i (s i,m -i ) Note 2. maxmin im and minmax im always exist. This is due to payoff u i (m i,m -i ) being continuous.

40 maxmin m vs minmax m We already know from pure minmax: (1) For every player i, maxmin im minmax i m (2) If joint mixed strategy m is a Nash equilibrium, then for every player i, minmax im u i (m) Jim S Joe C T 3,- 0,- M 1,- 2,- (only Jim s payoffs are shown) Jim: maxmin = 1, minmax = 2 Jim can increase his maxmin if he plays ¼ T + ¾ M. maxmin m = 1.5 For Jim, the worst strategy Joe can play is ⅓ S + ⅔ C, minmax m = 1.5 i i

41 maxmin m vs minmax m Art Bea F G U 6, 1 0, 4 D 2, 5 4, maxmin m ¼ ½ 1 q Art s mixed strategies are uniquely described by q. f A (q) = min sb {F,G} u A (q,s B ) = min { u A (q,f), u A (q,g) } = = min { 6q+2(1-q), 4(1-q) } = min { 2+4q, 4-4q }

42 maxmin m vs minmax m Check yourself that minmax Am is also 3. So it is verified that maxmin i maxmin im minmax im minmax i As an exercise, do the same check for Bea. Note that we found a Nash equilibrium at (⅝, ½), so Art s payoff at NE is 3.75 > 3.

43 back to Example 3 Odd 0 Even 1 0-4, 4 4, , -4-4, 4 Also for this game (which is zero-sum) maxmin = -4 < maxmin im = minmax im = 0 < minmax = 4 Condition maxmin im = minmax im seems to hold. 0 was the payoff at the (mixed) NE for this zerosum game. Remember the minimax theorem?

44 Minimax Theorem (2) (1) For every player i, maxmin im = minmax i m (2) If G is a zero-sum game, all Nash equilibria in mixed strategies are security strategies for player i and yield a payoff to player i equal to maxmin m i Note. In zero-sum games maxmin 1 m = -minmax 2 m All Nash equilibria are equivalent (same payoff) maxmin 1 m is called the value of the game.

45 Linear Programming The search of minmax solutions (i.e., NEs) of a zero-sum game is a nice application of LP. Player 1 has pure strategies {A 1, A 2,, A L }. A mixed strategy a = { a j } is a linear combination a 1 A a L A L Player 2 has pure strategies {B 1, B 2,, B M }. A mixed strategy b = { b j } is a linear combination b 1 B b M B M Note. We only need u = u 1 as u 2 = - u 1

46 Linear Programming a j 0, j a j = 1 j a j u(a j B k ) W ( k) maximize W W must be maximized. W cannot be increased, when some constraints become active. These constraint describe the support of player 2 s b. The a j s are a probability distribution M constraints The payoff of (a, B k ). We check a against M pure strategies only In general, this finds a mixed minimax strategy for player 1.

47 Linear Programming Since maxmin im = minmax i m a j 0, j a j = 1 j a j u(a j B k ) W ( k) maximize W minmax version b j 0, j b j = 1 j b j u(a k B j ) W ( k) minimize W maxmin version The two problems yield the same solution. Note. This formulation can be made for every problem, but solution is not always guaranteed. Zero-sum games are special in that u 2 = - u 1

48 How to solve minmax LP problems can be solved by using well known techniques (see Optimization courses). Polynomial-time techniques exist. Simplex method is widely used (CPLEX, lpsolve). Even though (worst-case) exponential, it is often fast in practice. Meta-heuristic techniques (Genetic Algorithms, Tabu search): sometimes even faster, but they do not guarantee to find the solution.