Two-player games overview

1. Two-player games overview • Computer programs which play 2-player games • game-playing as search • with the complication of an opponent • General principles of game-playing and search • evaluation functions • minimax principle • alpha-beta-pruning • heuristic techniques

2. Status of Game-Playing Systems • in chess, checkers, backgammon, Othello, etc, computers routinely defeat leading world players • Applications? • think of “nature” as an opponent • economics, war-gaming, medical drug treatment

3. Games of strategy • Deterministic rules (or deterministic rules plus probabilistic rules – these are games that combine strategy and luck, e.g. bridge, backgammon, blackjack) • Moves are alternately made by two players A and B. • rules define how configurations change • A subset F of configurations is identified as final. • typically F is partitioned into three sets: T, A and B. • T is tie, A (B) is win for player A (B) • Goal is to develop a strategy for one player to win. (computer plays for that player)

4. Chess Rating Scale

5. Two-Player Games with Complete Trees • We can use search algorithms to write “intelligent” programs that play games against a human opponent. • Just consider this extremely simple (and not very exciting) game: • At the beginning of the game, there are seven coins on a table. • Player 1 makes the first move, then player 2, then player 1 again, and so on. • One move consists of removing 1, 2, or 3 coins. • The player who makes the last move wins.

6. Two-Player Games with Complete Trees • Let us assume that the computer has the first move. Then, the game can be described as a series of decisions, where the first decision is made by the computer, the second one by the human, the third one by the computer, and so on, until all coins are gone. • The computer wants to make decisions that guarantee its victory, against every possible opponent. • The underlying assumption is that the opponent always finds the optimal move.

7. Game Tree Representation Computer Moves S • New aspect to search problem • there’s an opponent we cannot control • how can we handle this? Opponent Moves Computer Moves Possible Goal State lower in Tree (winning situation for computer) G

8. Game Trees

9. Game Trees

10. An optimal procedure: The Min-Max method • Designed to find the optimal strategy for Max and find best move: 1. Generate the whole game tree to leaves 2. Apply utility (payoff) function to leaves 3. Back-up values from leaves toward the root: • a Max node computes the max of its child values • a Min node computes the Min of its child values 4. When value reaches the root: choose max value and the corresponding move. • However:It is impossible to develop the whole search tree, instead develop part of the tree and evaluate promise of leaves using a static evaluation function.

11. Complexity of Game Playing • Suppose the entire tree is explored. (depth d, branching factor b) • What is the time for search be in this case? • worst case, it will be O(bd) • Chess: • b ~ 35 (average branching factor) • d ~ 100 (depth of game tree for typical game) • bd ~ 35100 ~10154 nodes!! • Tic-Tac-Toe • ~5 legal moves, total of 9 moves • 59 = 1,953,125 • 9! = 362,880 (Computer goes first) • 8! = 40,320 (Computer goes second) • well-known games can produce enormous search trees

12. Static (Heuristic) Evaluation Functions • An Evaluation Function: • estimates how good the current board configuration is for a player. • Typically, one figures how good it is for the player, and how good it is for the opponent, and subtracts the opponents score from the players • Othello: Number of white pieces - Number of black pieces • Chess: Value of all white pieces - Value of all black pieces • Typical values from -infinity (loss) to +infinity (win) or [-1, +1]. • If the board evaluation is X for a player, it’s -X for the opponent

13. Two-Player Games • We need to define a static evaluation function e(p) that tells the computer how favorable the current game position p is from its perspective. • In other words, e(p) will assume large values if a position is likely to result in a win for the computer, and low values if it predicts its defeat. • In any given situation, the computer will make a move that guarantees a maximum value for e(p) after a certain number of moves. • For this purpose, we can use the Minimax procedure with a specific maximum search depth (ply-depth k for k moves of each player).

14. e(p) for tic-tac toe • e(p) = 8 – 8 = 0 • e(p) = 6 – 2 = 4 • e(p) = 2 – 2 = 0 • e(p) =  • e(p) = - 

15. General Minimax Procedure on a Game Tree For each move: 1. expand the game tree as far as possible 2. assign state evaluations at each open node 3. propagate upwards the minimax choices if the parent is a Min node (opponent) propagate up the minimum value of the children if the parent is a Max node (computer) propagate up the maximum value of the children

16. Minimax Principle • “Assume the worst” • say each configuration has an evaluation number • high numbers favor the player (the computer) • so we want to choose moves which maximize evaluation • low numbers favor the opponent • so they will choose moves which minimize evaluation • Minimax Principle • you (the computer) assume that the opponent will choose the minimizing move next (after your move) • so you now choose the best move under this assumption • i.e., the maximum (highest-value) option considering both your move and the opponent’s optimal move. • we can extend this argument more than 2 moves ahead: we can search ahead as far as we can afford.

17. Backup Values

20. Games of chance • Backgammon is a two player game with uncertainty. • Players roll dice to determine what moves to make. • White has just rolled 5 and 6 and had four legal moves: • 5-10, 5-11 • 5-11, 19-24 • 5-10, 10-16 • 5-11, 11-16 • Such games are good for exploring decision making in adversarial problems involving skill and luck.

21. Backgammon start direction of move

22. Backgammon

23. Game trees with chance nodes • Chance nodes (shown as circles) represent the dice rolls. • Each chance node has 21 distinct children with a probability associated with each. • We can use minimax to compute the values for the MAX and MIN nodes. • Use expected values for chance nodes. • For chance nodes over a max node, as in C, we compute: • epectimax(C) = Sumi(P(di) * maxvalue(i)) • For chance nodes over a min node compute: expectimin(C) = Sumi(P(di) * minvalue(i))

24. Meaning of the evaluation function A1 is best move A2 is best move 2 outcomes with prob {.9, .1} • Dealing with probabilities and expected values means we have to be careful about the “meaning” of values returned by the static evaluator. • Note that a “relative-order preserving” change of the values would not change the decision of minimax, but could change the decision with chance nodes. • Linear transformations are ok

25. Pruning with Alpha/Beta Backup Values

26. Alpha Beta Procedure • Idea: • Do Depth first search to generate partial game tree, • Give static evaluation function to leaves, • compute bound on internal nodes. • Alpha, Beta bounds: • Alpha value for Max node means that Max real value is at least alpha. • Beta for Min node means that Min can guarantee a value below Beta. • Computation: • Alpha of a Max node is the maximum value of its seen children. • Beta of a Min node is the lowest value seen of its child node .

27. When to Prune • Pruning • Below a Min node whose beta value is lower than or equal to the alpha value of its ancestors. • Below a Max node having an alpha value greater than or equal to the beta value of any of its Min nodes ancestors.

28. The Alpha-Beta Procedure • Now let us specify how to prune the Minimax tree in the case of a static evaluation function. • Use two variables alpha (associated with MAX nodes) and beta (associated with MIN nodes). • These variables contain the best (highest or lowest, resp.) e(p) value at a node p that has been found so far. • Notice that alphacan never decrease, and beta cannever increase.

29. The Alpha-Beta Procedure • There are two rules for terminating search: • Search can be stopped below any MIN node having a beta value less than or equal to the alpha value of any of its MAX ancestors. • Search can be stopped below any MAX node having an alpha value greater than or equal to the beta value of any of its MIN ancestors. • Alpha-beta pruning thus expresses a relation between nodes at level n and level n+2 under which entire subtrees rooted at level n+1 can be eliminated from consideration.

30. Alpha-beta procedure [Adapted from J.Pearl]

31. The Alpha-Beta Procedure Example: max min max min

32. The Alpha-Beta Procedure Example: max min max min  = 4 4

33. The Alpha-Beta Procedure Example: max min max min  = 4 5 4

34. The Alpha-Beta Procedure Example: max min max  = 3 min  = 3 5 4 3

35. The Alpha-Beta Procedure Example: max min max  = 3 min  = 3  = 1 5 4 3 1

36. The Alpha-Beta Procedure Example: max min  = 3 max  = 3 min  = 3  = 1  = 8 5 4 3 1 8

37. The Alpha-Beta Procedure Example: max min  = 3 max  = 3 min  = 3  = 1  = 6 5 6 4 3 1 8

38. The Alpha-Beta Procedure Example: max min  = 3 max  = 3  = 6 min  = 3  = 1  = 6 5 6 4 3 1 8 7

39. The Alpha-Beta Procedure Example:  = 3 max min  = 3 max  = 3  = 6 min  = 3  = 1  = 6 5 6 4 3 1 8 7

40. The Alpha-Beta Procedure Example:  = 3 max Propagated from grandparent – no values below 3 can influence MAX’s decision any more. min  = 3 max  = 3  = 6  = 3 min  = 3  = 1  = 6  = 2 5 6 2 4 3 1 8 7

41. The Alpha-Beta Procedure Example:  = 3 max min  = 3 max  = 3  = 6  = 3 min  = 3  = 1  = 6  = 2  = 5 5 6 2 4 3 1 8 7 5

42. The Alpha-Beta Procedure Example:  = 3 max min  = 3 max  = 3  = 6  = 3 min  = 3  = 1  = 6  = 2  = 4 5 6 4 2 4 3 1 8 7 5

43. The Alpha-Beta Procedure Example:  = 3 max min  = 3  = 4 max  = 3  = 6  = 4 min  = 3  = 1  = 6  = 2  = 4 5 6 4 2 4 3 1 8 7 5 4

44. The Alpha-Beta Procedure Example:  = 3 max min  = 3  = 4 max  = 3  = 6  = 4 min  = 3  = 1  = 6  = 2  = 4  = 6 5 6 4 2 4 3 1 8 7 5 4 6

45. The Alpha-Beta Procedure Example:  = 3 max min  = 3  = 4 max  = 3  = 6  = 4 min  = 3  = 1  = 6  = 2  = 4  = 6 5 6 4 2 4 3 1 8 7 5 4 7 6

46. The Alpha-Beta Procedure Example:  = 4 max min  = 3  = 4 max  = 3  = 6  = 4  = 6 min  = 3  = 1  = 6  = 2  = 4  = 6 5 6 4 2 4 3 1 8 7 5 4 7 6 7

47. The Alpha-Beta Procedure Example:  = 4 max Done! min  = 3  = 4 max  = 3  = 6  = 4  = 6 min  = 3  = 1  = 6  = 2  = 4  = 6 5 6 4 2 4 3 1 8 7 5 4 7 6 7

48. The Alpha-Beta Procedure • Can we estimate the benefit of the alpha-beta method? • Suppose that there is a game that always allows a player to choose among b different moves, and we want to look d moves ahead. • Then our search tree has bdleaves. • Therefore, if we do not use alpha-beta pruning, we would have to apply the static evaluation function Nd = bd times.

49. The Alpha-Beta Procedure • Of course, the efficiency gain by the alpha-beta method always depends on the rules and the current configuration of the game. • However, if we assume that new children of a node are explored in a particular order - those nodes p are explored first that will yield maximum values e(p) at depth d for MAX and minimum values for MIN - the number of nodes to be evaluated is:

50. The Alpha-Beta Procedure • Therefore, the actual number Nd can range from about 2bd/2 (best case) to bd (worst case). • This means that in the best case the alpha-beta technique enables us to look ahead almost twice as far as without it in the same amount of time. • In order to get close to the best case, we can compute e(p) immediately for every new node that we expand and use this value as an estimate for the Minimax value that the node will receive after expanding its successors until depth d. • We can then use these estimates to expand the most likely candidates first (greatest e(p) for MAX, smallest for MIN).