Game Theory (part-2) Week 8, Lecture 8

Transcription

1 CS 8/68 Knowledge-Based Agents Game Theory (part-) Week 8, Lecture 8 In addition to our own examples, examples and some slides come from various lecture notes available online, including: ) Andrew Moore (CMU) ) an Neill (CMU) ) X. Liu (CMU) Tonight ynamic games, complete and imperfect information The Tragedy of the Commons epeated Games Finite Infinite Iterated Prisoner s ilemma Bayesian Games (incomplete information) Evolutionary Game Theory ynamic games of complete and imperfect information Imperfect information A player may not know exactly Who has made What choices when she has an opportunity to make a choice. Example: player makes her choice after player does. Player needs to make her decision without knowing what player has made. Perfect information and imperfect information A dynamic game in which every information set contains exactly one node is called a game of perfect information. A dynamic game in which some information sets contain more than one node is called a game of imperfect information. Imperfect information: illustration Each of the two players has a penny. Player first chooses whether to show the ead or the Tail. Then player chooses to show ead or Tail without knowing player s choice, Both players know the following rules: If two pennies match (both heads or both tails) then player wins player s penny. Otherwise, player wins player s penny. Player T -,, - Player T Player T, - -, Information set An information set for a player is a collection of nodes satisfying: the player has the move at every node in the information set, and when the play of the game reaches a node in the information set, the player with the move does not know which node in the information set has (or has not) been reached. All the nodes in an information set belong to the same player The player must have the same set of feasible actions at each node in the information set. 6

2 Information set: illustration Information set: illustration,, Player L L Player L L L Player L L,,,,,,,,,,,,,, an information set for player containing three nodes two information sets for player each containing a single node an information set for player containing a single node 7 All the nodes in an information set belong to the same player Player E,, Player C F,, G,, This is not a correct information set Player,, 8 Information set: illustration The player must have the same set of feasible actions at each node in the information set. Player E, Player C F, G,, An information set cannot contains these two nodes K Player, epresent a static game as a game tree: illustration Prisoners dilemma Prisoner Mum Mum Prisoner Fink Mum Fink Prisoner Fink 9,,,, Example: mutually assured destruction Two superpowers, and, have engaged in a provocative incident. The timing is as follows. The game starts with superpower s choice either ignore the incident ( I ), resulting in the payoffs (, ), or to escalate the situation ( E ). Following escalation by superpower, superpower can back down ( B ), causing it to lose face and result in the payoffs (, -), or it can choose to proceed to an atomic confrontation situation ( A ). Upon this choice, the two superpowers play the following simultaneous move game. They can either retreat ( ) or choose to doomsday ( ) in which the world is destroyed. If both choose to retreat then they suffer a small loss and payoffs are (-., -.). If either chooses doomsday then the world is destroyed and payoffs are (-K, -K), where K is very large number. Example: mutually assured destruction I E, B A, - -., -. -K, -K -K, -K -K, -K

3 A strategy for a player is a complete plan of actions. It specifies a feasible action for the player in every contingency in which the player might be called on to act. It specifies what the player does at each of her information sets Strategy and payoff a strategy for player : Player T -,, - Player T a strategy for player : T Player T, - -, Player s payoff is and player s payoff is - if player plays and player plays T Strategy and payoff: illustration, a strategy for player : A,, if player plays E, written as A I, - B E a strategy for player : E, and if player plays A, written as E A -., -. -K, -K -K, -K -K, -K Nash equilibrium in a dynamic game We can also use normal-form to represent a dynamic game The set of Nash equilibria in a dynamic game of complete information is the set of Nash equilibria of its normal-form ow to find the Nash equilibria in a dynamic game of complete information Construct the normal-form of the dynamic game of complete information Find the Nash equilibria in the normal-form emove unreasonable Nash equilibrium Subgame perfect Nash equilibrium is a refinement of Nash equilibrium It can rule out unreasonable Nash equilibria or incredible threats We first need to define subgame 6 Subgame Subgame: illustration A subgame of a dynamic game tree bens at a singleton information set (an information set contains a single node), and includes all the nodes and edges following the singleton information set, and does not cut any information set; that is, if a node of an information set belongs to this subgame then all the nodes of the information set also belong to the subgame., I, - B E A a subgame a subgame Not a subgame -., -. -K, -K -K, -K -K, -K 7 8

4 Subgame-perfect Nash equilibrium A Nash equilibrium of a dynamic game is subgame-perfect if the stratees of the Nash equilibrium constitute or induce a Nash equilibrium in every subgame of the game. Subgame-perfect Nash equilibrium is a Nash equilibrium. 9 Find subgame perfect Nash equilibria: backward induction, Starting with those smallest subgames Then move backward until the root is reached One subgameperfect Nash equilibrium ( I, A ) I, - B E A -., -. -K, -K a subgame a subgame -K, -K -K, -K Find subgame perfect Nash equilibria: backward induction, Starting with those smallest subgames Then move backward until the root is reached Another subgameperfect Nash equilibrium ( E, B ) I, - B E A -., -. -K, -K a subgame a subgame -K, -K -K, -K TE TAGEY OF TE Commons YE OLE COMMONS You graze goats on the commons to eventually fatten up and sell The more goats you graze the less well fed they are And so the less money you get when you sell them Selling Price per Goat Commons Facts ow many goats would one rational farmer choose to graze? What would the farmer earn? What about a group of n individual farmers? Price = 6 G 6 6 G= number of goats n farmers i th farmer has an infinite space of stratees g i [, 6 ] An outcome of ( g, g, g, g n ) will pay how much to the i th farmer?

5 n farmers i th farmer has an infinite space of stratees g i [, 6 ] An outcome of ( g, g, g, g n ) will pay how much to the i th farmer? g i 6 g j n j= Let s Assume a pure Nash Equilibrium exists. Call it ( g, g, Λ gn ) What can wesay about g? Payoff to farmer i,assuming = argmax the other playersplay ( ) g, g, Λ, +, Λ gn For NotationalConvenience, write G i = g j j i TEN g = argmax i What? 6 Let s Assume a pure Nash Equilibrium exists. Call it ( g, g, Λ g ) What can wesay about g i TEN argmax the other playersplay ( g, g, Λ g, g, Λ g ) For NotationalConvenience, g i = = i argmax n write G = j i g j g? Payoff to farmer i,assuming i i+ n * [ 6 G i ] Let s Assume a pure Nash Equilibrium exists. Call it ( g, g, Λ g ) * * * What can wesay about g? g 6 * i G i = Payoff to farmer i,assuming therefore = argmax the other playersplay * * 6 G i g ( g, g,, +, gn ) i Λ Λ = * * For NotationalConvenience, 6 G i TEN g i = i argmax n write G = j i g j * [ 6 G i ] g* i must satisfy 7 8 We have n linear equations in n unknowns g * = - /( g *+g *+ g n *) g * = - /(g *+ g *+ g n *) g * = - /(g *+g *+ g * g n *) : : : g n * = - /(g *+ g n- *) Clearly all the g i * s are the same Write g*=g *= g n * Solution to g*= /(n-)g* is: g*= 7 n+ 9 Consequences At the Nash Equilibrium a rational farmer grazes 7 goats. n+ ow many goats in general will be grazed? Trivial algebra ves: 6-6 goats total being grazed n+ ow much profit per farmer? [as n --> infinity, #goats --> 6] ow much if the farmers could all cooperate? *sqrt() = 8. n n (n+) /.6 if farmers.6 if farmers

6 The Tragedy The farmers act rationally and earn.6 cents each. But if they d all just got together and decided one goat each they d have got.6 cents each. Is there a bug in Game Theory? in the Farmers? in Nash? INTEMISSION oes the Tragedy of the Commons matter to us when we re building intelligent machines? Maybe repeated play means we can learn to cooperate?? epeated game A repeated game is a dynamic game of complete information in which a (simultaneous-move) game is played at least twice, and the previous plays are observed before the next play. We will find out the behavior of the players in a repeated game. epeated Games with Implausible Threats Takeo and andy are stuck in an elevator Takeo has a $ bill andy has a stick of dynamite andy says Give me $ or I ll blow us both up. o Nothing andy Takeo: - andy : ves andy the money Explode Takeo: andy : What should Takeo do????? Takeo Takeo: andy : keeps money o Nothing andy Explode Takeo: - 7 andy - 7 : Using the formalism of epeated Games With Implausible Threats, Takeo should Not ve the money to andy andy Takeo Assumes andy is ational -Step Prisoner s ilemma GAME Player B GAME (Played with knowledge of outcome of GAME ) Player B T: T: - 7 : : - 7 epeated Games At this node, andy will choose the left branch Suppose you have a game which you are going to play a finite number of times. What should you do? Player A C -, - C, -9 Idea Is Idea correct? -9, -6, -6 Player A C C -, -, -9-9, -6, -6 Player A has four pure stratees C then C C then then C then 6 itto for B 6

7 Important Theoretical esult: Assuming Implausible Threats, if the game G has a unique N.E. (s *, s n *) then the new game of repeating G T times, and adding payouts, has a unique N.E. of repeatedly choosing the orinal N.E. (s *, s n *) in every game. If you re about to play prisoner s dilemma times, you should defect times. AT 7 Finitely repeated game A finitely repeated game is a dynamic game of complete information in which a (simultaneous-move) game is played a finite number of times, and the previous plays are observed before the next play. The finitely repeated game has a unique subgame perfect Nash equilibrium if the stage game (the simultaneous-move game) has a unique Nash equilibrium. The Nash equilibrium of the stage game is played in every stage. 8 Infinitely repeated game A infinitely repeated game is a dynamic game of complete information in which a (simultaneous-move) game called the stage game is played infinitely, and the outcomes of all previous plays are observed before the next play. Precisely, the simultaneous-move game is played at stage,,,..., t-, t, t+,... The outcomes of all previous t- stages are observed before the play at the t th stage. Each player discounts her payoff by a factor δ, where < δ <. A player s payoff in the repeated game is the present value of the player s payoffs from the Present value efinition: Given a discount factor δ, the present value of an infinite sequence of payoffs π π, π,,... is, π t π t t= π + δπ + δ π + δ π +... = δ stage games. 9 Example : The present value of an infinite sequence of payoffs,,,... ( π t =, for all t) is. δ Example : The present value of an infinite sequence of payoffs,,,,,...( in every odd stage, in every even stage) is δ +. δ δ Infinitely repeated game: example The following simultaneous-move game is repeated infinitely The outcomes of all previous plays are observed before the next play bens Each player s payoff for the infinitely repeated game is present value of the payoffs received at all stages. Question: what is the subgame perfect Nash equilibrium? Player L L,, Player,, Example: subgame L L L (, ) (, ) (, ) (, ) Every subgame of an infinitely repeated game is identical to the game as a whole. 7

8 Example: strategy A strategy for a player is a complete plan. It can depend on the history of the play. A strategy for player i: play L i at every stage (or at each of her information sets) Another strategy called a trigger strategy for player i: play i at stage, and at the t th stage, if the outcome of each of all t- previous stages is (, ) then play i ; otherwise, play L i. Example: subgame perfect Nash equilibrium Check whether there is a subgame perfect Nash equilibrium in which player i plays L i at every stage (or at each of her information sets). This can be done by the following two steps. Step : check whether the combination of stratees is a Nash equilibrium of the infinitely repeated game. If player plays L at every stage, the best response for player is to play L at every stage. If player plays L at every stage, the best response for player is to play L at every stage. ence, it is a Nash equilibrium of the infinitely repeated game. Example: subgame perfect Nash equilibrium Step : check whether the Nash equilibrium of the infinitely repeated game induces a Nash equilibrium in every subgame of the infinitely repeated game. ecall that every subgame of the infinitely repeated game is identical to the infinitely repeated game as a whole Obviously, it induces a Nash equilibrium in every subgame ence, it is a subgame perfect Nash equilibrium. Example: subgame L L L (, ) (, ) (, ) (, ) L L L L L L L L L L L L TO INFINITY 6 Trigger strategy trigger strategy for player i: play i at stage, and at the t th stage, if the outcomes of all t- previous stages are (, ) then play i ; otherwise, play L i. Check whether there is a subgame perfect Nash equilibrium in which each player plays the trigger strategy. This can be done by the following two steps. Step : check whether the combination of the trigger stratees is a Nash equilibrium of the infinitely repeated game Step : if yes, check whether the Nash equilibrium induces a Nash equilibrium in every subgame 7 Trigger strategy: step Stage : (, ) Stage : (, ) Stage t-: (, ) Stage t: (, L ) Stage t+: (L, L ) Stage t+: (L, L ) Suppose that player plays the trigger strategy. Can player be better-off if she deviates from the trigger strategy at stage t? If she continues to play the trigger strategy at stage t and after, then she will get a sequence of payoffs,,,... (from stage t to stage + ). iscounting these payoffs to stage t ves us + δ + δ + δ +... = δ If she deviates from the trigger strategy at stage t then she will trigger noncooperation. Player will play L after stage t forever. Player ' best response is L. So player will get a sequence of payoffs,,,... (from stage t to stage + ). iscounting these payoffs to stage t ves us δ + δ + δ + δ +... = + δ 8 8

9 Trigger strategy: step cont d Trigger strategy: step Stage : (, ) Stage : (, ) Stage t-: (, ) Stage t: (, L ) Stage t+: (L, L ) Stage t+: (L, L ) δ + δ δ δ ence, if δ, player cannot be better off if she deviates from the trigger strategy. This implies that if player plays the trigger strategy the player 's best response is the trigger strategy for δ. By symmetry, if player plays the trigger strategy then player 's best response is the trigger strategy. ence, there is a Nash equilibrium in which both players play the trigger strategy if δ. 9 Stage : (, ) Stage : (, ) Stage t-: (, ) Stage t: (, ) Stage t+: (, ) Stage t+: (, ) Step : check whether the Nash equilibrium induces a Nash equilibrium in every subgame of the infinitely repeated game. ecall that every subgame of the infinitely repeated game is identical to the infinitely repeated game as a whole Trigger strategy: step cont d We have two classes of subgames: subgame following a history in which the stage outcomes are all (, ) subgame following a history in which at least one stage outcome is not (, ) The Nash equilibrium of the infinitely repeated game induces a Nash equilibrium in which each player still plays trigger strategy for the first class of subgames The Nash equilibrium of the infinitely repeated game induces a Nash equilibrium in which (L, L ) is played forever for the second class of subgames. The Iterated Prisoner s ilemma When a Prisoner s ilemma interaction is iterated over a large number of rounds, cooperation can become a rational option. Axelrod (98) and many others have argued that cooperation can evolve by reciprocal altruism: I ll cooperate with you if you cooperate with me. For example, the Tit for Tat (TFT) strategy: cooperate iff opponent cooperated on the previous round. TFT (and variants) performed very well in Axelrod s experiments, sparking a huge literature on cooperation via reciprocal altruism. P and IP model a huge number of situations: business agreements, animal behavior, nuclear arms race, environmental conservation, Tragedy of the Commons see Axelrod s book for more details. For infinitely repeated games, or for finitely repeated games with an unknown number of moves and sufficiently small probability of stopping, TFT is a symmetric Nash equilibrium. But there are lots of other NE: for example, always defect! Individual rationality Many possible Nash equilibria in Iterated Prisoner s ilemma: problem of equilibrium selection. More generally, question of individual rationality: what should an individual rational player do to maximize his payoff, ven no prior knowledge of the opponent? Choose best strategy according to some performance measure but which one? Axelrod (98): round-robin tournament. First choose a set of stratees, then the best strategy is the one with highest average payoff vs. all stratees in that set. (Tit For Tat won!) Problem: performance is very dependent on set of stratees under consideration (e.g. if many stratees are unconditional cooperators, exploiters will win). Bayesian Games So far, have considered games of complete information. Now let s look at games with incomplete information. 9

10 Static (or simultaneous-move) games of INCOMPLETE information Payoffs are no longer common knowledge Incomplete information means that At least one player is uncertain about some other player s payoff function. Static games of incomplete information are also called static Bayesian games Prisoners dilemma of complete information Two suspects held in separate cells are charged with a major crime. owever, there is not enough evidence. Both suspects are told the following policy: If neither confesses then both will be convicted of a minor offense and sentenced to one month in jail. If both confess then both will be sentenced to jail for six months. If one confesses but the other does not, then the confessor will be released but the other will be sentenced to jail for nine months. Prisoner Cooperate Confess Prisoner Cooperate Confess -, - -9,, -9-6, -6 6 Prisoners dilemma of incomplete information Prisoner is always rational (selfish). Prisoner can be rational (selfish) or altruistic, depending on whether he is happy or not. If he is altruistic then he prefers to cooperate and he thinks that confess is equivalent to additional four months in jail. Prisoner can not know exactly whether prisoner is rational or altruistic, but he believes that prisoner is rational with probability.8, and altruistic with probability.. Payoffs if prisoner is altruistic Prisoner Cooperate Confess Cooperate -, -, -9 Prisoner Confess -9, - -6, - 7 Prisoners dilemma of incomplete information cont d Given prisoner s belief on prisoner, what strategy should prison choose? What strategy should prisoner choose if he is rational or altruistic? Payoffs if prisoner is rational Prisoner Cooperate Confess Payoffs if prisoner is altruistic Prisoner Cooperate Confess Prisoner Cooperate Confess -, - -9,, -9-6, -6 Prisoner Cooperate Confess -, - -9, -, -9-6, - 8 Prisoners dilemma of incomplete information cont d Solution: Prisoner chooses to confess, ven his belief on prisoner Prisoner chooses to confess if he is rational, and cooperate if he is altruistic This can be written as (Confess, (Confess if rational, Cooperate if altruistic)) Confess is prisoner s best response to prisoner s choice (Confess if rational, Cooperate if altruistic). (Confess if rational, Cooperate if altruistic) is prisoner s best response to prisoner s Confess A Nash equilibrium called Bayesian Nash equilibrium 9 Battle of the sexes At the separate workplaces, Chris and Pat must choose to attend either an opera or a prize fight in the evening. Both Chris and Pat know the following: Both would like to spend the evening together. But Chris prefers the opera. Pat prefers the prize fight. Chris,, Pat,, 6

11 Battle of the sexes with incomplete information Now Pat s preference depends on whether he is happy. If he is happy then his preference is the same. If he is unhappy then he prefers to spend the evening by himself and his preference is shown in the following table. Chris cannot figure out whether Pat is happy or not. But Chris believes that Pat is happy with probability. and unhappy with probability. Battle of the sexes with incomplete information ow to find a solution? Payoffs if Pat is happy with probability. Chris,, Pat,, Payoffs if Pat is unhappy Chris,, Pat,, 6 Payoffs if Pat is unhappy with probability. Chris,, Pat,, 6 Battle of the sexes with incomplete information Best response If Chris chooses opera then Pat s best response: opera if he is happy, and prize fight if he is unhappy Suppose that Pat chooses opera if he is happy, and prize fight if he is unhappy. What is Chris best response? If Chris chooses opera then she get a payoff if Pat is happy, or if Pat is unhappy. er expected payoff is.+.= If Chris chooses prize fight then she get a payoff if Pat is happy, or if Pat is unhappy. er expected payoff is.+.=. Since >., Chris best response is opera A Bayesian Nash equilibrium: (opera, (opera if happy and prize fight if unhappy)) 6 Battle of the sexes with incomplete information Best response If Chris chooses prize fight then Pat s best response: prize fight if he is happy, and opera if he is unhappy Suppose that Pat chooses prize fight if he is happy, and opera if he is unhappy. What is Chris best response? If Chris chooses opera then she get a payoff if Pat is happy, or if Pat is unhappy. er expected payoff is.+.= If Chris chooses prize fight then she get a payoff if Pat is happy, or if Pat is unhappy. er expected payoff is.+.=. Since >., Chris best response is opera (prize fight, (prize fight if happy and opera if unhappy)) is not a Bayesian Nash equilibrium. 6 Bayesian games in the real world: an introduction to auctions Let s assume that the seller wants to sell an object worth nothing to him: anything the seller is paid is pure profit. There are N buyers, each with a value v i for the object. v i are drawn uniformly at random (i.i.d.) from [,], and this is common knowledge. Each buyer knows his own value v i, but not the other buyers values v j (j i). First price sealed bid auction Each buyer i writes down their bid b i simultaneously; no buyer gets to see another s bid. The buyer with the highest bid pays the seller b i and gets the object. What should a buyer bid, in terms of his value v i? (Why not bid b i = v i?) First, a couple obvious properties (proof omitted). By symmetry, each buyer must have the same function b i (v i ) = b * (v i ) at the Bayes-Nash equilibrium. Also, b * (v i ) must be monotonically increasing with v i. 6 66

12 First price sealed bid () b * (v i ) = arg max b E[Profit if playing b] = arg max b (profit if b wins) P(b wins) = arg max b (v i - b) P(all b * (v j ) < b) = arg max b (v i -b) P(b * (v) < b) N-, where v ~ U[,]. Since b * is monotonically increasing, this equals: arg max b (v i - b) P(v < (b*) - (b)) N- = arg max b (v i - b) ((b*) - (b)) N- = arg max b (v i - b) (f(b)) N-, where function f = (b * ) -. First price sealed bid () Setting the first derivative equal to zero: (v i - b) (f(b)) N- b = (v i - b) (N-) (f(b)) N- f '(b) - (f(b)) N- = (v i - b) (N-) f '(b) - f(b) = Since b = b * (v i ), v i = f(b). (f(b) - b) (N-) f '(b) - f(b) = f '(b) = f(b) / ((N-)(f(b)-b)) First price sealed bid () Solving the differential equation: f(b) = (N / (N-)) b v i = (N / (N-)) b b = ((N-) / N) v i b * (v i ) = ( - /N) v i. Thus, in a first price sealed bid auction, each bidder should bid ( - /N) times Also called a Vickrey auction! Second price sealed bid auction Each buyer i writes down their bid b i simultaneously; no buyer gets to see another s bid. The buyer with the highest bid gets the object, but pays the seller only the amount b' of the second highest bid. What should a buyer bid, in terms of his value v i? his value! 69 7 Second price sealed bid () If a bidder with value v i bids b i, and the highest bid among the other bidders is b', his utility is v i -b', if b i > b', and otherwise. If b' < v i, any bid b i > b' is optimal. If b' > v i, any bid b i < b' is optimal. If b' = v i, any bid is optimal. Thus the dominant strategy for a bidder with valuation v i is to bid b i = v i. ow to compute the seller s profit? In the first price sealed bid auction, the seller s profit is (N-) / N times the highest v i. In the second price sealed bid auction, the seller s profit is the second highest v i. A little useful information from statistics: E[kth order statistic] = k / (N+). Var[kth order statistic] = k (N-k+) / (N+)(N+) Expected profit from second price sealed bid: (N-) / (N+). Expected profit from first price sealed bid: ((N-) / N) (N / (N+)) = (N-) / (N+). Given N i.i.d. values drawn from U[,] Seller s expected profit is the same in either case! 7 7

13 Oral auctions English auction: the standard auction with ascending price; the item is sold when all but one buyer drop out. ominant strategy: remain in the bidding until price is equal to v i, then drop out. Equivalent to nd price sealed bid (Why?). utch auction: descending price; the item is sold when a buyer agrees to buy for that price. Going going gone! Will anyone buy at $? Equivalent to st price sealed bid (Why?) Mine! 7 Evolutionary game theory A performance measure should be justified by a model: i.e. what sort of interactions are assumed between stratees? Evolutionary game theory assumes that the payoff for a game is a measure of its value toward evolutionary survival. Evolution occurs by natural selection: stratees which earn higher average payoffs are more likely to survive and reproduce, and less fit stratees die off. Samuelson (): Evolutionary game theory covers a wide variety of models. The common theme is a dynamic process describing how players adapt their behavior over the course of repeated plays of a game. The dynamic process can be biolocal evolution (i.e. the players evolve better stratees) or individual processes of learning/imitation (i.e. the players choose stratees that they observe to be better!) 7 eplicator dynamics Compute how the proportion of a population playing a ven strategy changes over time, assuming rate of reproduction is proportional to average payoff received. The share of agents playing a strategy grows at a rate equal to the difference between the average payoff of that strategy and the average payoff of the entire population. dx i / dt = x i (w i - w avg ). More generally, increase must be monotonically increasing with w i (i.e. rate of reproduction higher for stratees receiving higher average payoffs). This could describe an actual biolocal scenario where natural selection determines reproductive fitness. Or it could describe a model of learning/imitation in which players are able to switch from worse to better stratees. 7 Short term vs. long term dynamics eplicator dynamics describe the short term evolution of a system: ven a starting state (proportions of stratees in the population), the population will evolve over time by natural selection. Short term dynamics will eventually converge to a stable state : this can be: a single strategy, mixture of stratees, or a cyclic or even chaotic fluctuation in population shares. The long term evolution of a system is how the system moves between different short term stable states, i.e. when some change (mutation, migration, environmental fluctuation) knocks it from one state into another. Most important stability concept in long term dynamics: 76 evolutionarily stable strategy (Maynard Smith). Evolutionary stability () Assume that a homogeneous population (everyone playing the same strategy) is invaded by a small population of mutants playing an alternative strategy. If the average payoff of the common strategy is greater than the average payoff of the mutants, natural selection will eliminate the mutants. Otherwise, the mutants will be able to invade, and possibly to take over the population. Maynard Smith s criteria (98): A strategy X can invade a strategy Y if w(x Y) > w(y Y), or if w(x Y) = w(y Y) and also w(x X) > w(y X). Also, if w(x Y) = w(y Y) and w(x X) = w(y X), strategy X can mix with strategy Y, and evolutionary drift occurs. Evolutionary stability () If no other strategy A can invade or mix with strategy X, then X is an evolutionarily stable strategy (ESS): once it is established in a population it cannot be displaced by any single mutant. This is the case if, for all alternative stratees A X, w(a X) w(x X), and if w(a X)=w(X X) then w(x A)>w(A A). ESS is a refinement of (symmetric) Nash equilibrium. An ESS is at least a weak best response to itself, thus ESS NE! All strict symmetric NE are ESS. A weak symmetric NE is an ESS if it passes the stability condition that it earns a higher payoff when facing any alternative best response than does the alternative itself. w(x Y) = average payoff to strategy X vs. opponent playing strategy Y 77 78

14 The Nowak-Sigmund model A simple model of long-term evolution based on Maynard Smith s invasion criteria: Assume an initial large homogeneous population of some (randomly selected) strategy Y. Each round, select a mutant strain X at random from the strategy space. Then if X invades Y according to the Maynard Smith criteria, X takes over the population, otherwise the initial population of Y will continue. ESS are stable points of the Nowak-Sigmund model. Once the population has evolved to an ESS, no other strategy can invade, so the population will play the ESS strategy forever Under what conditions is an ESS really stable? Assumptions of ESS The population evolves (in the short term) according to some payoff-monotone selection dynamics. The population size is infinite. Only a single type of mutant strategy can attempt to invade the population at a time. Mutations are rare: the population evolves to a shortterm stable state after each mutation, before the next mutation occurs. Mutations have small impact: the proportion of a mutant strategy in the combined population is neglible ESS for finite populations X is an infinite population (Maynard Smith) ESS if: For all alternative stratees A X, w(a X) w(x X). If w(a X) = w(x X), then w(x A) > w(a A). Schaffer (988) demonstrated that a strategy meeting these criteria can, in fact, be invaded if the population is finite. The reason: a player cannot play himself in a contest. In a population of N players with M mutants, the mutants will play against a mutant (M-)/(N-) of the time, while players of the common strategy will play against a mutant M/(N-) of the time. Thus a mutant will invade if (M-)w(A A)+(N-M)w(A X) > Mw(X A) + (N-M-)w(X X). X is a finite population (Schaffer) ESS if a population of size ESS for finite populations () oes the finite population ESS converge to the infinite population ESS as population size N? Surprisingly, the answer is NO! Neill () defines a large population ESS as a strategy which cannot be invaded by any finite number of mutants M, as long as the population size N is sufficiently large. The large population ESS is not equivalent to the infinite population ESS; see examples to right. A large population ESS X must satisfy: For all A X, w(a X) w(x X). If w(a X) = w(x X), then w(x A) w(a A) and w(x A) > w(a X). N cannot be invaded by M mutants of any type, for ven N X is ESS for large populations, 8 8 and M. For infinite population, this was but not for infinite populations. If w(a X) = w(x X), then w(x A) > w(a A). X A X A vs. X vs. A - X is ESS for infinite populations, but not for large populations. vs. X vs. A