Game Theory Preliminaries: Playing and Solving Games

Transcription

1 Game Theory Preliminaries: Playing and Solving Games Zero-sum games with erfect information R&N 6 Definitions Game evaluation Otimal solutions Minimax Non-deterministic games (first take)

2 Tyes of Games (informal) Deterministic Chance Perfect Information Imerfect Information Chess, Checkers Go Battleshi Backgammon, Monooly Bridge, Poker, Scrabble, wargames Tyes of Games (informal) Deterministic Chance Perfect Information Chess, Checkers Go Backgammon, Monooly Imerfect Information Battleshi Bridge, Poker, Scrabble, wargames Note: This initial material uses the common definition of what a game is. More interesting is the generalization of the theory to scenarios that are far more useful to a wide range of decision making roblems. Stay tuned. 2

3 Definitions Two-layer game: Player A and B. Player A starts. Deterministic: None of the moves/states are subject to chance (no random draws). Perfect information: Both layers see all the states and decisions. Each decision is made sequentially. Zero-sum: Player s A gain is exactly equal to layer B s loss. One of the layer s must win or there is a draw (both gains are equal). Examle Initially a stack of ennies stands between two layers Each layer divides one of the current stacks into two unequal stacks. The game ends when every stack contains one or two ennies The first layer who cannot lay loses A B 3

4 7 A s turn 6, 5, 2 4, 3 B s turn 5,, 4, 2, 3, 2, 2 3, 3, A s turn 4,,, 3, 2,, 3,,,, 2, 2,,, A Loses 2,,,,, B Loses 2, 2, 2, B Loses B s turn A s turn B s turn Search Problem States: Board configuration + next layer to move Successor: List of states that can be reached from the current state through of legal moves Terminal state: States at which the games ends Payoff/Utility: Numerical value assigned to each terminal state. Examle: U(s) = for A win, for B win, 0 for draw Game value: The value of a terminal that will be reached assuming otimal strategies from both layers (minimax value) Search: Find move that maximizes game value from current state 4

5 U = U = 2, 2, 2, U = 2, 2,,, 2,,,,, Otimal (minimax) Strategies Search the game tree such that: A s turn to move find the move that yields maximum ayoff from the corresonding subtree This is the move most favorable to A B s turn to move find the move that yields minimum ayoff (best for B) from the corresonding subtree This is the move most favorable to B 5

6 Minimax Minimax (s) If s is terminal Return U(s) If next move is A ( s' ) Return max Minimax s' Succs( s) Else min Minimax s' Return s' Succs( s) ( ) A 3 = max(3,2,2) B 3 = min(3,2,8)

7 Minimax Proerties Comlete: If finite game Otimal: If oonent lays otimally Essentially DFS Efficiency: αβ runing Use heuristic evaluation functions to cut off search early Examle: Weighted sum of number of ieces (material value of state) Sto search based on cutoff test (e.g., maximum deth) Choice of Value? Absolute game value is different in the two cases Minimax solution is the same Only the relative ordering of values matters, not the absolute values ordinal utility values True only for deterministic games Evaluation functions can be any function that reserves the ordering of the utility values 7

8 Non-Deterministic Games A Non-Deterministic Games Chance B 8

9 Chance Non-Deterministic Games Includes states where neither layer makes a A choice. A random decision is made (e.g., rolling dice) Use exected value of successors at chance nodes: s' Succs( s) ( s') MiniMax( s') B Non-Deterministic Minimax Minimax (s) If s is terminal Return U(s) max Minimax If next move is A: Return If next move is B Return min s' Succs( s) Minimax s' Succs( s) ( s' ) ( s' ) If chance node Return s' Succs( s) ( s' ) Minimax( s' ) 9

10 Choice of Utility Values Different utility values may yield radically different result even though the order is the same Absolute utility values do matter Utility should be roortional to actual ayoff, it is not sufficient to follow the same order Think of choosing between 2 lotteries with same odds but radically different ayoff distributions Imlication: Evaluation functions must be linear ositive functions of utility Kind of obvious but imortant consideration for later develoments 0

11 Definitions Game evaluation Otimal solutions Minimax Non-deterministic games Matrix Form of Games R&N Chater 6 R&N Section 7.6

12 Assumtions so far: Two-layer game: Player A and B. Perfect information: Both layers see all the states and decisions. Each decision is made sequentially. Zero-sum: Player s A gain is exactly equal to layer B s loss. We are going to eliminate these constraints. We will eliminate first the assumtion of erfect information leading to far more realistic models. Some more game-theoretic definitions Matrix games Minimax results for erfect information games Minimax results for hidden information games Player A L R Player B L 2 3 R L R Player A 4 L +4 Extensive form of game: Reresent the game by a tree 2

13 A ure strategy for a layer defines the move that the layer would make for every ossible state that the layer would see. L L R 2 3 R L R L 4 +4 Pure strategies for A: Strategy I: ( L,4 L) Strategy II: ( L,4 R) Strategy III: ( R,4 L) Strategy IV: ( R,4 R) Pure strategies for B: Strategy I: (2 L,3 L) Strategy II: (2 L,3 R) Strategy III: (2 R,3 L) Strategy IV: (2 R,3 R) L 4 L R L R 2 3 R L R +4 In general: If N states and B moves, how many ure strategies exist? 3

14 Matrix form of games Pure strategies for A: Strategy I: ( L,4 L) Strategy II: ( L,4 R) Strategy III: ( R,4 L) Strategy IV: ( R,4 R) I II III IV I +4 II +4 III Pure strategies for B: Strategy I: (2 L,3 L) Strategy II: (2 L,3 R) Strategy III: (2 R,3 L) Strategy IV: (2 R,3 R) IV L 4 L R +4 L R 2 3 R L R Pure strategies for Player A Pure strategies for Player B I II III IV I +4 II +4 III IV Player A s ayoff if game is layed with strategy I by Player A and strategy III by Player B Matrix normal form of games: The table contains the ayoffs for all the ossible combinations of ure strategies for Player A and Player B The table characterizes the game comletely, there is no need for any additional information about rules, etc. Although, in many cases, the number of ure strategies may be too large for the table to be reresented exlicitly, the matrix reresentation is the basic reresentation that is used for deriving fundamental roerties of games. 4

15 Max value of all the rows Minimax Matrix version I II III IV I II III IV Min value across each row Max Min M( i, j) Rows i Columns j Minimax Matrix version For each strategy (each row of the game matrix), Player A should assume that Player B will use the otimal strategy given Player A s strategy (the strategy with the minimum value in the row of the matrix). Therefore the best value that Player can achieve is the maximum over all the rows of the minimum values across each of the rows: Max Rows i Min Columns j Max value = game value = M( i, j) The corresonding ure strategy is the otimal solution for this game It is the otimal strategy for A assuming that B lays otimally. I II III IV I +4 II +4 III IV Min value across each row 5

16 Max value across each column I II III IV I II III IV Min of all the columns Min Max M( i, j) Columns j Rows i Minimax or Maximin? But we could have used the oosite argument: For each strategy (each column of the game matrix), Player B should assume that Player A will use the otimal strategy given Player B s strategy (the strategy with the maximum value in the column of the matrix): Min Columns j Max M( i, j) Rows i Therefore the best value that Player B can achieve is the minimum over all the columns of the maximum values across each of the columns Problem: Do we get to the same result?? Is there always a solution? Max value across each column I II III IV I II III IV Min value = game value = 6

17 Max value = game value = I II III IV Min value across each row Max Rows i I +4 II +4 Min Columns j Note that we find the same value and same strategies in both cases. Is that always the case? III IV M( i, j) Max value across each column I II III IV Min Columns j I II III IV Min value = game value = Max M( i, j) Rows i Minimax vs. Maximin Fundamental Theorem I (von Neumann): For a two-layer, zero-sum game with erfect information: There always exists an otimal ure strategy for each layer Minimax = Maximin Note: This is a game-theoretic formalization of the minimax search algorithm that we studied earlier. 7

18 Games with Hidden Information R&N Chater 6 R&N Section 7.6 Another (Seemingly Simle) Game The two Players A and B each have a coin They show each other their coin, choosing to show either head or tail. If they both choose head Player B ays Player A $2 If they both choose tail Player B ays Player A $ If they choose different sides Player A ays Player B $ 8

19 Side Note about all toy examles If you find this kind of toy examle annoying, it models a large number of real-life situations. For examle: Player A is a business owner (e.g., a restaurant, a lant..) and Player B is an insector. The insector icks a day to conduct the insection; the owner icks a day to hide the bad stuff. Player B wins if the days are different; Player A wins if the days are the same. This class of roblems can be reduced to the coin game (with different ayoff distributions, erhas). Extensive Form Player A H T Player B H T H T 9

20 Extensive Form Player A H Player B H T H T Problem: Since the moves are simultaneous, Player B does not know which move Player A chose This is no longer a game with refect information we have to deal with hidden information T Player B H T Player A H T 20

21 Matrix Normal Form Player A H T Player B H T It is no longer the case that maximin = minimax (easy to verify: vs. ) Therefore: It aears that there is no ure strategy solution In fact, in general, none of the ure strategies are solutions to a zero-sum game with hidden information Why no Pure Strategy Solutions? Player A H T Player B H T Intuition: If Player A considers move H, he has to assume that Player B will choose the worst-case move (for A), which is move T Therefore Player A should try move T instead, but then he has to assume that Player B will choose the worst-case move (for A), which is move H. Therefore Player A should consider move H, but then he has to assume that Player B will choose the worst-case move (for A), which is move T. Therefore Player A should try move T instead, but then he has to assume that Player B will choose the worst-case move (for A), which is move H.» Therefore Player A should consider move H, but then he has to assume that Player B will choose the worst-case move (for A), which is move T.». 2

22 H T H T Using Random Strategies Suose that, instead of choosing a fixed ure strategy, Player A chooses randomly strategy H with robability, and strategy T with robability -. If Player B chooses move H, the exected ayoff for Player A is: ( + 2) + ( ) ( ) = 3 If Player B chooses move T, the exected ayoff for Player A is: ( ) + ( ) ( + ) = 2 + So, the worst case is when Player B chooses a strategy that minimizes the ayoff between the 2 cases: min( 3, 2 + ) Player A should then adjust the robability so that its ayoff is maximized (note the similarity with the standard maximin rocedure described earlier): max min(3, 2 + ) Exceted ayoff for Player A 2 0 Grahical Solution Exected ayoff if Player B chooses T 2 Exected ayoff if Player B chooses H 3 22

23 Exceted ayoff for Player A 2 0 Grahical Solution Exected ayoff if Player B chooses T 2 Exected ayoff if Player B chooses H 3 No matter what strategy Player B follows (choosing a move at random with rob. q for H), the resulting ayoffs will still be between the two lines corresonding to B s ure strategies 2 Exceted ayoff for Player A 0 2/5 23

24 Exceted ayoff for Player A 2 0 2/5 Otimal choice of : * = arg max min(3, 2 + ) = 2 / 5 Exected ayoff: max min(3, 2 + ) = / 5 min( 3, 3 + 2) Mixed Strategies It is no longer ossible to find an otimal ure strategy for Player A. We need to change the roblem a bit: We assume that Player A chooses a ure strategy randomly at the beginning of the game. In that scenario, Player A selects one ure strategy robability and the other one with robability -. This strategy of choosing ure strategies randomly is called a mixed strategy for Player A and is entirely defined by the robability. Question: We know that we cannot find an otimal ure strategy for Player A, but can we find an otimal mixed strategy? Answer: Yes! The result that we derived for the simle examle holds for general games. It yields a rocedure for finding the otimal mixed strategy for zero-sum games. 24

25 Minimax with Mixed Strategies Theorem II (von Neumann): For a two-layer, zero-sum game with hidden information: There always exists an otimal mixed strategy with value max min( m + ( ) m2, m2 + ( ) m22) Where the matrix form of the game is: m m 2 m 2 m 22 Note: This is a direct generalization of the minimax result to mixed strategies. Minimax with Mixed Strategies Theorem II (von Neumann): For a two-layer, zero-sum game with hidden information: There always exists an otimal mixed strategy In addition, just like for games with erfect information, it does not matter in which order we look at the layers, minimax is the same as maximin max min( m min max( q m q + ( ) m + ( q) m, m, q m + ( + ( q) m Note: This is a direct generalization of the minimax result to mixed strategies ) m ) ) = 25

26 0 max 0 max max 0 Recie for 2x2 games max max max min( m + ( ) m2, m2 + ( ) m22) Since the two functions of are linear, the maximum is attained either for: = 0 = The intersection of the two lines, if it occurs for between 0 and 26

27 General Case: NxM Games We have illustrated the roblem on 2x2 games (2 strategies for each of Player A and Player B) The result generalizes to NxM games, although it is more difficult to comute A mixed strategy is a vector of robabilities (summing to!) = (,.., N ). i is the robability with which strategy i will be chosen by Player A. The otimal strategy is found by solving the roblem: max min i i j = i i m ij This is solved by using Linear Programming General Case: NxM Games We have illustrated the roblem on 2x2 games (2 strategies for each of Player A and Player B) The result generalizes to NxM games, although it is more difficult to comute A mixed strategy is a vector of robabilities (summing to!) = (,.., N ). i is the robability with which strategy i will be chosen by Player A. The otimal strategy is found by solving the roblem: max min i i j = i i m ij Exected ayoff for Player A if Player B chooses ure strategy number j and Player A chooses ure strategy i with rob. i 27

28 Grahical Illustration: 2xM Game m j + ( ) m2 j max min( m j + ( ) m2 j j ) 0 min( m j + ( ) m2 j j ) Discussion The criterion for selecting the otimal mixed strategy is the average ayoff that Player A would receive over many runs of the game. It may seem strange to use random choices of ure strategies as mixed strategies and to search for otimal mixed strategies. In fact, it formalizes what haens in common situations. For examle, in oker, if Player A follows a single ure strategy (taking the same action every time a articular configuration of cards is dealt), Player B can guess and resond to that strategy and lower Player A s ayoffs. The right thing to do is for Player A to change randomly the way each configuration is handled, according to some olicy. A good layer would use a good olicy. The game theory results formalize the need for things like bluffing in games with hidden information. The theory assumes rational layers Roughly seaking, the layers make decision based on increasing their resective ayoffs (utility values, references,..). 28

29 Another Examle: Sort of Poker Players A and B lay with two tyes of cards: Red and Black Player A is dealt one card at random (50% rob. of being Red) If the card is red, Player A may resign and loses $20 Or Player A may hold B may then resign A wins $0 B may see A loses $40 if the card is Red A wins $30 otherwise Modified version of an examle from Andrew Moore Prob. = 0.5 Prob. = 0.5 hold -20 resign resign hold see resign see

30 The game is nondeterministic because of the initial random choice of cards = 0.5 hold Prob. = 0.5 hold Hidden information: Player B cannot know which of these 2 states it s in -20 resign resign see resign see Player B Player A Resign Hold Resign See Generate the matrix form of the game (be careful: It s not a deterministic game) Find the otimal mixed strategy Find the exected ayoff for Player A 30

31 Summary Matrix form of games Minimax rocedure and theorem for games with erfect information Always a ure strategy solution Minimax rocedure and theorem for games with hidden information Always a mixed strategy solution Procedure for solving 2x2 games with hidden information Understanding of how the roblem is formalized for NxM games (actually solving them requires linear rogramming tools which will not be covered here) Imortant: These results aly only to zero-sum games. This is still a severe restrictions as most realistic decision-making roblems cannot be modeled as zerosum games This restriction will be eliminated next! 3