Neural Network Implementation of Poker AI

1. Neural Network Implementation of Poker AI Christopher Kramer

2. Outline of Information • The Challenge • Application, problem to be solved, motivation • Why create a poker machine with ANNE? • The Flop • The hypothesis • Can a Poker AI run using only an ANNE? • The Turn • The approach and methodology • How is this machine created? • The River • Experimentation • How should it be tested? • The Showdown • Conclusions and future work • Did it work?

3. The Challenge • The approach focused on creating an AI capable of playing poker at a reasonable level of play using Neural Networks. • The difficulty: • Incomplete knowledge • Multiple competing agents • Risk management • Opponent modeling • Deception • Unreliable information • Creating solutions to address these difficulties can be applied to other problems with similar domains. NOISE!

4. The Fold (Hypothesis) • The strengths of computers: • Quick computation • Large memory • Purely logical • Using a series of machine learning techniques including neural networks, it is predicted that a machine will be able to play at a moderate level while ignoring typical features of poker programs such as statistical analysis and pot odds.

5. The Turn (Approach) • Problem split into a number of stages

6. Pre-Flop • Hardcoded • Action chosen based off of known values of starting hands The Turn (continued) • Information gathered from percentage of games won with starting hand from poker databases • All hands in top 30th percentile were analyzed for characteristics which were hardcoded into if statements • All hands below threshold were folded. • Those with characteristics in top 5th percentile had a higher random chance of being raised

7. The Turn (continued) • Post-Flop • Neural Net • 5 card input • Inputs the unique id of each card in hand and play • Hidden nodes utilizing sigmoid function • Output using sigmoid function between -1 and 1 • Output values above .3 indicated an action to raise • Below -.3 fold • Else, Call

8. The Turn (continued)

9. The Turn (continued) • River • Neural Net • 6 card input • Inputs the unique id of each card in hand and play • Hidden nodes utilizing sigmoid function • Output using sigmoid function between -1 and 1 • Output values above .3 indicated an action to raise • Below -.25 fold • Else, Call

10. The Turn (continued) • Showdown • Cards analyzed for poker hands such as Flush, Fold, etc. and high card of each returned. • Input nodes for each poker hand. Value is the value of the highest card in hand. • Hidden nodes utilizing sigmoid function • Output using sigmoid function between -1 and 1 • Output values above .5 indicated an action to raise • Below -.2 fold • Else, Call • Neural Net • Player hand analyzed and given as set of input values

11. The Turn (opponent modeling) • Opponent modeling generalized to fit into previous Neural networks • Opponents action modeling taken into account by Neural nets due to opponents actions changing the weights of neural network to account for their actions. • 2 Factors • Player’s calls • For each raise by opponent, have a higher required confidence to raise again • Players treatment of poker hand values • As player actions change, the confidence values of the Neural nets should change to reflect the modeling of the player

12. The River (experimentation) • Experiment was developed to overcome short term failure due to bad luck • 1000 dollars given to each player. $10 bet . • Engines initial weights were determined experimentally for each ANNE • Weights were assigned randomly and run for 10000 generations against a set of random poker hands and card tables. Winning gave an expected value of 1, loss -1, and draw 0. • Network was then checked against 1000 random validation hands. • A degree of accuracy was required for each ANNE • Required accuracy lowered as amount of information known decreases. • Once Neural Network was fit level of confidence its results were accepted as the Engines default weights. • Weights continue training during the course of game to profile player. • Tested against a number of human opponents based on skill level to measure success.

13. The River (experiment results) • Beginners • AI won 82% of games against 50 opponents • Moderate (play at least 1 game a month) • AI won 20% of games against 20 opponents • Advanced (play at least 1 game a week) • AI won 1 game out of 10 against 2 opponents. • Random (never folded) • AI won 63.2% of games out of 1000. • 100% won against Random opponent who could also fold

14. The Showdown (Conclusion) • The complete reliance on generalized Neural Networks for a system with extreme noise is not a viable option for higher levels of play. • While it did not shown significant advantage over moderate and advanced opponents, it did well against beginning opponents who knew how to play poker but did not do so often. • So much noise…. • While a degree of noise was expected, generalized poker domain’s noise was is in excess of 10-40% (approx.) • This led to extreme cases of overlearning regardless of an extremely low learning rate. • While this was partially handled using confidence, the amount of folds late in the game due to incorrect early predictions led to the machine still failing to moderate opponents. • Was still shown to do well against a random opponent and beginners.

15. The Showdown (future research) • There are numerous areas for improvement • Current systems adjusts slower than humans • Possible GA implementation instead of Neural Network to avoid direct noise issues. • Statistical analysis and pot odds would help as inputs for neural network.

16. Questions?