Finite Additivity in Dubins-Savage Gambling and Stochastic Games. Bill Sudderth University of Minnesota

Transcription

1 Finite Additivity in Dubins-Savage Gambling and Stochastic Games Bill Sudderth University of Minnesota This talk is based on joint work with Lester Dubins, David Heath, Ashok Maitra, and Roger Purves. 1

2 References I will survey some measure theoretic aspects of 50 years of research in Dubins-Savage gambling and its connection to stochastic games. Most references are left to the end. The basic reference is of course: L. E. Dubins and L. J. Savage (1965). How to Gamble If You Must: Inequalities for Stochastic Processes. McGraw-Hill. (Dover edition in 2014.) 2

3 A Dubins-Savage gambling problem: (S, Γ, g) S - the state space Γ - the gambling house - for each x S, Γ(x) is a nonempty set of possible distributions for the next state. A player starts at some x 0 S and chooses σ 0 Γ(x 0 ). The next state X 1 has distribution σ 0. Given X 1 = x 1, the player chooses σ 1 (x 1 ) Γ(x 1 ) as the conditional distribution of X 2. Given X 1 = x 1, X 2 = x 2, the player choses σ 2 (x 1, x 2 ) Γ(x 2 ) as the conditional distribution of X 3. And so on. The sequence σ = (σ 0, σ 1,...) is a strategy. g : S R - the payoff function. The player seeks to choose σ to maximize the expected value E σ g(x 1, x 2,...). 3

4 A finitely additive formulation S is an arbitrary nonempty set. For each x S, Γ(x) is a nonempty set of gambles on S. (A gamble on a set S is a finitely additive probability measure defined on all subsets of S.) The payoff function g : S R is bounded and F-measurable where F is the sigma-field generated by the open subsets of S when S has the discrete topology and S has the product topology. The definition of E σ g is in three steps. 4

5 Step 1. The definition of E σ g = g dp σ for a space of functions g : S R Dubins and Savage recast the conditioning formula E[Y ] = E[E[Y X]] as E σ g = E σ[x1 ] (gx 1) σ 0 (dx 1 ). where σ[x 1 ] is the conditional strategy that follows σ after the first stage and gx 1 is the section of g such that (gx 1 )(x 2, x 3,...) = g(x 1, x 2, x 3,...). This formula together with the requirement that E σ c = c for constants c determines E σ on a linear space of functions g that includes the indicators of all clopen subsets of S when S has the discrete topology and S has the product topology. 5

6 Step 2. The Lebesgue-like extension to open sets For open subsets O S, let P σ (O) = sup{p σ (K) : K O, K clopen}. Dubins (1974) showed P σ is then finitely additive on the lattice of open sets with a unique extension, also written as P σ, to the algebra generated by the open sets. Non-uniqueness: There may be other finitely additive extensions to the open sets, but this one has many good properties. 6

7 Step 3. A further extension by squeezing Let A be the collection of all A S such that inf{p σ (O) : A O, O open} = sup{p σ (C) : C A, C closed}. Write P σ (A) for this common value. Then P σ is finitely additive on A and A contains the sigma-field F generated by the open sets. The sigma-field F includes all the sets usually considered in countably additive probability. The expectation E σ g is well-defined for all bounded, F-measurable functions g : S R. The classical limit theorems such as the strong law of large numbers and the martingale convergence theorem generalize to this finitely additive setting (Karandikar, 1982, 1988). 7

8 Examples of payoff functions Let r : S R be a bounded daily reward function. Discounted: g(x 1, x 2,...) = n=1 β n 1 r(x n ) where 0 < β < 1. Long run average: g(x 1, x 2,...) = lim sup m ( 1 m mn=1 r(x n )). These are special cases of: Limsup: u (x 1, x 2,...) = lim sup n u(x 1,..., x n ) Here u is a bounded real-valued function defined for all finite sequencies (x 1,..., x n ) of states. 8

9 The optimal reward function Γg For x S, Γg(x) = sup E σ [g] where the sup is over all strategies σ available at x in Γ. Main problems: Calculate Γg and find optimal or nearly optimal strategies. 9

10 A regularity property of the optimal reward Let B F and σ be a strategy. Then, by the definition of P σ, P σ (B) = inf{p σ (O) B O, O open}. The optimal reward has the same property. Write ΓB(x) = Γ1 B (x) = sup{p σ (B) σ at x}. Theorem 1. For x S, ΓB(x) = inf{γo(x) B O, O open}, The equality can be rewritten as: sup inf σ at P σ(o) = inf sup P σ (O) x O B O B σ at x 10

11 The one-day operator T For bounded functions f : S R and x S, define. T f(x) = sup{ f(x 1 ) γ(dx 1 ) γ Γ(x)} For an open set O, the optimal reward function ΓO can be calculated by a transfinite form of backward induction. The algorithm for ΓO is completely determined by T. So, by the regularity property, the value of ΓB is also determined by T for all B F. 11

12 An example. Let E S and let O = n [x n E]. Assume that δ(x) Γ(x) for every x S. Define functions U ξ : S R for ordinals ξ by setting U 0 = 1 E and, for ξ > 0, U ξ = T U ξ 1, sup η<ξ U η, if ξ is a successor if not. Then ΓO = U ξ where ξ is the least ordinal such that T U ξ = U ξ. For S finite, ξ = ω and ΓO = lim n U n. 12

13 Another regularity property Let g : S R be bounded and F-measurable. Theorem 2. For x S, Γg(x) = sup{γu (x) u g}. Here u (x 1, x 2,...) = lim sup n u(x 1,..., x n ) is the limsup payoff and u ranges over all bounded, real-valued u. There is a transfinite algorithm for calculating the functions Γu. The algorithm is completely determined by the one-day operator T. So Γg is also determined by T. 13

14 A measurable, countably additive formulation (Strauch, 1967) 1. The state space S is a nonempty Borel subset of a Polish space. 2. For every x S, Γ M (x) is a nonempty set of countably additive probability measures defined on the sigma-field B(S) of Borel subsets of S. 3. Γ M (x) varies measurably in x in the sense that the set {(x, γ) γ Γ M (x)} is a Borel subset of S C(S) where C(S) is the set of countably additive probability measures on B(S) with its usual topology. 4. The payoff function g : S R is bounded and Borel measurable when S is given its product Borel sigma-field. 14

15 Measurable strategies A strategy σ = (σ 0, σ 1,...) is measurable and available at x in Γ M if σ 0 Γ M (x) and, for all n 1 and all (x 1,..., x n ) S n, σ n (x 1,..., x n ) Γ M (x n ) and σ n is a universally measurable function from S n to C(S). A measurable strategy σ determines a countably additive probability measure Pσ M on the Borel subsets of S with expectation operator Eσ M. The optimal reward for a measurable problem with payoff function g is, for x S, Mg(x) = sup E M σ g where the sup is over all measurable σ at x. The function Mg need not be Borel, but is universally measurable. 15

16 A regularity property of M Let g : S R be bounded and Borel measurable. Theorem 3. For x S, Mg(x) = sup{mu (x) u g}, where u (x 1, x 2,...) = lim sup n u(x 1,..., x n ) and u ranges over all bounded, real-valued, Borel measurable u. As in the finitely additive case, there is a transfinite algorithm for calculating the functions Mu.

17 Finitely additive extensions of measurable problems Every countably additive measure γ M available in a measurable house Γ M can be extended to a finitely additive additive measure γ defined on all subsets of S. (The measure γ is typically not unique.) Suppose every γ M is extended in this way and let Γ be the resulting finitely additive house. Question: related? How are the optimal reward functions M g and Γg 16

18 Theorem 4. Suppose Γ M is a measurable gambling house defined on the Borel state space S and that Γ is a finitely additive extension of Γ M. If g : S R is bounded and Borel measurable, then Γg = Mg. So finitely additive gambling theory can be viewed as a generalization of the measurable, countably additive theory. Also a player in a measurable house cannot improve his payoff by using a nonmeasurable strategy. The proof of Theorem 4 uses the regularity properties of the operators Γ and M. 17

19 A two-person, zero-sum game: G = (A, B, f). A, B - nonempty sets of actions for players 1 and 2. f : A B R - a bounded, real-valued payoff function. Player 1 chooses an action a A and simultaneously player 2 chooses b B; player 2 pays player 1 the amount f(a, b). The players may choose their actions independently and at random using probability distributions µ and ν on A and B. Player 1 seeks to maximize and player 2 to minimize the expected payoff E µ,ν f. 18

20 The game G = (A, B, f) has a value if sup inf µ ν E µ,νf = inf ν sup µ E µ,νf. Theorem 5.(von Neumann, 1928) If A and B are finite, then the game has a value. If A and B are countably infinite, and probability distributions are required to be countably additive, then the game may not have a value. Example. (Wald) Let A = B = {1, 2,...} and let f(a, b) = 1 or 1 according as a b or a < b. 19

21 Games with finitely additive mixed actions Recall that a gamble on a set is a finitely additive probability measure defined on all subsets. Theorem 6. Suppose 1. A and B are nonempty sets, 2. the distributions µ and ν vary over all gambles on A and B, respectively, 3. f : A B R is a bounded function, A 4. the expectation E µ,ν f = B integral in a given order. Then the game (A, B, f) has a value. f(a, b)µ(da)ν(db) is a double See Marinacci (1997) and Flesch, Vermeulen, and Zseleva (2017) for more on finitely additive values. 20

22 A coherence lemma. Let X be the Banach space of all bounded functions g : B R with the supremum norm. Suppose F is a convex subset of X such that inf g 0 for all g F. (There is no sure win in F.) Then there is a gamble ν on B such that g dν 0 for all g F. Proof. Let N = {g X inf g > 0}. Then N is convex with a nonempty interior and N F =. By a separation theorem, there exists a linear functional x X such that xg 0, for g F and xg 0 for g N. Normalize x so that x(1) = 1 and take ν = x. 21

23 Proof of Theorem 6. To show sup inf µ ν f(a, b)µ(da)ν(db) = inf ν sup µ note that is always true. To prove, suppose sup µ inf ν E µ,ν f < r for some r R. Then So ( µ)( ν) ( ( µ)( b) ( f(a, b)µ(da)ν(db) < r). f(a, b)µ(da) < r). f(a, b)µ(da)ν(db), Thus every function g µ (b) = f(a, b)µ(da) r has inf b g µ (b) 0. By the lemma, there exists ν such that f(a, b)µ(da)ν (db) r = g µ dν 0 for all µ. So inf ν sup µ f(a, b)µ(da)ν(db) sup[ µ f(a, b)µ(da)ν (db)] r. 22

24 Finitely Additive Nash Equilibria (NE) Two-person, nonzero-sum Game: G = (A, B, f 1, f 2 ). Payoffs: E µ,ν f i = A B f i(a, b)µ(da)ν(db) to player i for i = 1, 2. NE: µ, ν where µ is optimal for player 1 vs ν and ν is optimal for player 2 vs µ. An example of Flesch and Predtetchinski (2018) shows a NE need not exist. Whether there always exist ɛ-ne is an interesting question. 23

25 Two-Person Zero-Sum Stochastic Game - Finitely Additive Six ingredients: S, A, B, p, g. S - state space A, B - action sets for players 1 and 2 p( x, a, b) - law of motion - a gamble on S for every (x, a, b) S A B g - payoff function from player 2 to player 1 - a bounded function from (A B S) to R that is measurable for the sigma-field G generated by the open subsets of (A B S) when A, B, S have their discrete topologies and the product has the product topology 24

26 Play of the game Play begins at some state x 0 S, player 1 chooses a 0 A using a gamble on A, player 2 chooses b 0 B using a gamble on B. The next state x 1 has distribution p( x 0, a 0, b 0 ) and play continues from x 1. At the next stage, player 1 chooses a 1 A using a gamble on A, player 2 chooses b 1 B using a gamble on B. The next state x 2 has distribution p( x 1, a 1, b 1 ). As play continues the players generate an infinite sequence (a 0, b 0, x 1, a 1, b 1, x 2,...) (A B S) and player 2 pays player 1 the amount g(a 0, b 0, x 1, a 1, b 1, x 2,...). 25

27 Strategies for the players Let Z = A B S. A strategy π for player 1 is a sequence π 0, π 1,... such that π 0 is a gamble on A, and for n 1 and (z 1,..., z n ) Z n, π n (z 1,..., z n ) is a gamble on A. A strategy ρ for player 2 is defined similarly. Strategies π and ρ together with the law of motion p determine a Dubins-Savage strategy σ = σ 0, σ 1... on Z. To define σ associate to x S and gambles µ on A and ν on B the gamble m = m(x, µ, ν) on Z where m(d(a, b, x 1 ))) = p(dx 1 x, a, b)µ(da)ν(db). For the initial state x 0, let σ 0 = m(x 0, π 0, ρ 0 ), σ n (z 1,..., z n ) = m(x n, σ n (z 1,..., z n ), ρ n (z 1,..., z n )). 26

28 Existence of the value Let S, A, B, p, g be a finitely additive stochastic game. Write E x,π,ρ [g] for the expected reward determined by an initial state x and strategies π and ρ. The game has a value at state x if sup π inf ρ E x,π,ρ[g] = inf ρ sup π E x,π,ρ[g]. This quantity is then the value and is written V g(x). Theorem 7. If the payoff function g is bounded and G-measurable, then the finitely additive stochastic game has a value V g(x) for every initial state x. This cannot be true for a general countably additive setting even if the payoff function g depends only on a 0 and b 0. This follows from Wald s example. 27

29 A regularity property of V Let g : Z R be bounded and G-measurable. Theorem 8. For x S, V g(x) = sup{v u (x) u g}, where u (z 1, z 2,...) = lim sup n u(z 1,..., z n ) and u ranges over all bounded, real-valued, G-measurable u. Also there is a transfinite algorithm for calculating the value functions V u. 28

30 Martin s Theorem: The determinacy of Blackwell Games D. A. Martin (1998) proved Theorem 7 in a countably additive setting with a countable state space S and finite action sets A and B. The proof of Theorem 7 adapts Martin s proof to the general finitely additive setting. Measurability problems arise if we try to adapt Martin s proof to a Borel measurable setting. Question. Is there a countably additive, Borel measurable analogue of Theorem 7? 29

31 Measurable Stochastic Games 1. S, A, B are nonempty Borel subsets of Polish spaces. 2. B is compact. 3. p( x, a, b) is a regular conditional distribution on S given S A B. 4. For every Borel subset E of S and (x, a) S A, p(e x, a, ) is continuous on B. 5. The payoff function g : S R is a bounded Borel measurable function of (x 1, x 2,...). It does not depend on the actions. 30

32 A theorem and a possibility Theorem 9. A measurable stochastic game with a limsup payoff u (x 1, x 2,...) = lim sup n u(x 1, x 2,..., x n ) has a value V u (x) for each initial state x, if u is a bounded Borel measurable function defined for all finite sequences of states. Also, there is a transfinite algorithm for calculating the value. Possibility. Perhaps every measurable stochastic game satisfying conditions 1 through 5 has a value V g(x) for each initial state x, and V g(x) = sup{v u (x) u g}. 31

33 Some references 1. R. Chen (1976/7). On almost sure convergence in a finitely additive setting. Z. Wahrsch. Verw. Gebiete B. de Finetti (1937). La prévision: ses lois logiques, ses sources subjectives. Inst. H. Poincaré L. E. Dubins (1974). On Lebesgue-like extensions of finitely additive measures. Ann. Probab L. E. Dubins and L. J. Savage (1965). How to Gamble If You Must: Inequalities for Stochastic Processes. McGraw- Hill. (Dover edition in 2014.) 32

34 5. J. Flesch, D. Vermeulen, and A. Zseleva (2017). Zero-sum games with charges. Games and Econ. Behavior J. Flesch and A. Predtetchinski (2018). A counterexample. Preprint. 7. R. L. Karandikar (1982). A general principle for limit theorems in finitely additive probability. Trans. Amer. Math. Soc R. L. Karandikar (1988). A general principle for limit theorems in finitely additive probability, the dependent case. Journal of Multivariate Analysis

35 9. A. Maitra, R. Purves, and W. Sudderth (1991). A capacitability theorem in finitely additive gambling. Rendiconti di Matematica e delle sue applicazioni A. Maitra, R. Purves, and W. Sudderth (1992). A capacitability theorem in measurable gambling theory. Trans. Amer. Math. Soc A. Maitra and W. Sudderth (1993). Borel stochastic games with limsup payoff. Ann. Probab A. Maitra and W. Sudderth (1996). Discrete Gambling and Stochastic Games. Springer.

36 13. A. Maitra and W. Sudderth (1998). Finitely additive stochastic games with Borel measurable payoffs. International Journal of Game Theory M. Marinacci (1997). Finitely additive and epsilon Nash equilibria. Int. J. Game Theory D. A. Martin (1998). The determinacy of Blackwell games. Journal of Symbolic Logic A. S. Nowak (1985). Universally measurable strategies in zero-sum stochastic games. Ann. Probab R. Purves and W. Sudderth (1976). Some finitely additive probability. Ann. Probab

37 18. R. Purves and W. Sudderth (2010). Big vee: the story of a function, an algorithm, and three mathematical worlds. Sankhya, Series A R. Strauch (1967). Measurable gambling houses. Trans. Amer. Math. Soc