Evolution: Games, dynamics and algorithms

1. Evolution: Games, dynamics and algorithms Karen Page Bioinformatics Unit Dept. of Computer Science, UCL

2. Evolution • Darwinian evolution is based on three fundamental principles: reproduction, mutation and selection • Concepts like fitness and natural selection are best defined in terms of mathematical equations • We show how many of the existing frameworks for the mathematical description of evolution may be derived from a single unifying framework

3. Summary of what will be discussed • Games, evolutionary game theory • Key frameworks of evolutionary dynamics • Deriving a unifying framework • An application to Fisher’s Fundamental Theorem • Relationship with Genetic Algorithms

4. What is game theory? • Formal way to analyse interactions between agents who behave strategically • Mathematics of decision making in conflict situations • Usual to assume players are “rational” • Widely applied to the study of economics, warfare, politics, animal behaviour, sociology, business, ecology and evolutionary biology

5. Assumptions of game theory • The game consists of an interaction between two or more players • Each player can decide between two or more well-defined strategies • For each set of specified choices, each player gets a given score (payoff)

6. The Prisoners’ Dilemma • Probably most studied of all games • Not enough evidence to convict two suspects of armed robbery, enough for theft of getaway car • Both confess (4 years each), both stay quiet (2 years each), one tells (0 years) the other doesn’t (5 years) • Stay quiet= cooperate (C) ; confess = defect (D) • Payoff to player 1: R is REWARD for mutual cooperation =3 SSUCKER’s payoff =0 TTEMPTATION to defect =5 PPUNISHMENT for mutual defection=1 with T>R>P>S

7. The problem of cooperation • What ever player 2 does, player 1 does better by defecting: • Classical game theory both players D • Shame because they’d do better by both cooperating • Cooperation is a very general problem in biology • Everyone benefits from being in cooperative group, but each can do better by exploiting cooperative efforts of others

8. Trade wars and cartels • Import tariffs - Should countries remove them? • Price fixing- why not cheat?

9. Repeated games • In many situations, typically players interact repeatedly- repeated Prisoners Dilemma • Strategies can involve memory, use reciprocity • Tit-for-tat • Pavlov

10. Game theory and a computer tournament • Game theory says it is rational to defect in single game or fixed number of rounds • Axelrod’s tournament- double victory for Tit-for-Tat

11. Evolutionary Game Theory So how can cooperation be explained?

12. Evolutionary games • John Maynard Smith- evolution of animal behaviour • Behaviour shaped by trial and error- adaptation through natural selection or individual learning • Players no longer have to be ‘rational’: follow instincts, procedures, habits rather than computing best strategy. • Games played in a population. Scores are summed. Strategies which do well against the population on average propagate. • Phenotypic approach to evolution • Frequency-dependent selection

13. Simple evolutionary game simulations • Everyone starts with a random strategy • Everyone population plays game against everyone else • The payoffs are added up • The total payoff determines the number of offspring (Selection) • Offspring inherit approximately the strategy of their parents (Mutation) • [Note similarity to genetic algorithms.] • [Nash equilibrium in a population setting- no other strategy can invade]

14. Evolution in the Prisoners’ Dilemma • Standard evolutionary game (random interactions)  all Defect • Modifications- spatial games: Interactions no longer random, but with spatial neighbours: • Sum scores. Player with highest score of 9 shaded takes square (territory, food, mates) in next generation • Some degree of cooperation evolves!

15. Simulations of the spatial Prisoners Dilemma 75 generations Winner-takes-all selection No mutation Red=d(d last) Blue=c(c last) Yellow=d(c last) Green=c(d last)

16. Conclusions on Evolutionary Games • Game theory can be applied to studying animal and human behaviour (economics - evolutionary biology). • Often traditional game theory’s assumption of ‘rationality’ fails to describe human/ animal behaviour • Instead of working out the optimal strategy, assume that strategies are shaped by trial and error by a process of natural selection or learning. This can be modelled by evolutionary game theory. • Space can matter

17. Evolutionary dynamics

18. General framework Quasispecies equation replicator-mutator Price equation Replicator-mutator equation Price equation Lotka-Volterra equation Game dynamical equation replicator Price equation Adaptive dynamics

19. The replicator equation • Replicator equation describes evolution of frequencies of phenotypes within a population with fitness-proportionate selection • Eg. game theory, replicators like “Game of Life” • Frequency of type i is and fitness of type i is then

20. The equivalence with Lotka Volterra equations • Lotka Volterra systems of ecology describe the numbers of animals (eg. fish) of different species and are of the form: where is the abundance of species i, its fitness and there are n species in total. • Often these interacting species oscillate in abundance. • There is a precise equivalence with the replicator system for (n+1) types given by the substitution

21. Replicator equation with mutation and quasispecies • Suppose there are errors in replicating. The probability of type j mutating to type i is . • We obtain a replicator equation with mutation: • The equivalent with numbers rather than frequencies of types is • When the fitnesses do not depend on frequencies, this is the quasispecies eqn. (Probably the case in most GAs?)

22. Quasispecies equation • Describes molecular evolution (Eigen) • N biochemical sequences • Biochemical species i has frequency yi • Replication at rate fi is error-prone - mutation to type j at rate qij

23. Adaptive dynamics framework • Game consists of a continuous space of strategies (eg.) • Population is assumed to be homogeneous- all players adopt same strategy • Mutation generates variant strategies very close to the resident strategy • If a mutant beats the resident players it takes over otherwise it is rejected • Adaptive dynamics illustrates the nature of evolutionary stable strategies

24. Adaptive dynamics equations • Strategies are described by continuous parameters : • Expected score of mutant against S is given by E(S’,S) • The adaptive dynamics flow in the direction which maximises the score:

25. We can derive Price’s equation from replicator-mutator equation • Price’s equation from population genetics describes any type of selection. • Suppose an individual of type i, frequency , has some trait p of value • , so using the replicator equation with mutation we obtain • This applies when the values of are const. • [p is the expected mutational change in p.]

26. Price’s equation mutation selection

27. Price’s equation gives rise to adaptive dynamics • If we assume that the mutation is localised and symmetrical then we can neglect the second term in Price’s eqn. • Assume population is almost homogeneous and fitness is differentiable then we can Taylor expand the fitness, obtaining • cf. adaptive dynamics:

28. General framework Quasispecies equation replicator-mutator Price equation Replicator-mutator equation Price equation Lotka-Volterra equation Game dynamical equation replicator Price equation Adaptive dynamics

29. Fisher’s fundamental theorem • Suppose fitnesses of genotypes constant. Can consider f as the trait p and obtain (for symmetric mutation): • Fisher’s fundamental theorem of NS • In general, fitnesses of genotypes depend on environment. In game theory context, depend on the frequencies of other genotypes. Fisher’s theorem doesn’t apply- eg. PD

30. Generalized version • We can use Price’s equation to obtain a generalized version of Fisher’s fundamental theorem: where • This applies when the s depend linearly on the frequencies of genotypes- normally the case in evolutionary game theory.

31. Fisher’s theorem and GAs • In most GAs, fitnesses of particular solutions (chromosomes) probably fixed and so (except for the complication of recombination) Fisher’s theorem should hold: • So for a GA with fitness-proportionate selection, no recombination and fixed fitness for a given solution, the average fitness of the population of solutions increases until there is no diversity left in the fitnesses.

32. Conclusions on unifying evolutionary dynamics • Unifying framework • Different frameworks for different problems. • We derive from Price’s equation a generalized version of Fisher’s Fundamental Theorem of Natural Selection. • The Price – replicator framework can also be applied to discrete time formulations and to formulations with sexual reproduction.

33. Relationship to GAs

34. Evolutionary games and genetic algorithms • Two-way interaction: 1) So far discussed computer simulations of evolutionary processes, eg. evolution of animal behaviour 2) Evolutionary computation, eg. genetic algorithms = computer science based on theory of biological evolution • Evolutionary games very like genetic algorithms- but 1) Population size is usually quite large and may be few phenotypes: space well searched but not v. efficient. 2) Usually no recombination 3) Fitnesses depend on interactions

35. Genetic Algorithms • Evolutionary models are computer algorithms which use evolutionary methods of optimisation to solve practical problems (cf. finding stable strategies in games rather than working out ‘rational’ solution)- eg. Evolutionary programming, genetic algorithms • Evolutionary operations involved in genetic algorithms: selection, mutation, recombination:

36. How evolutionary dynamics relates to GAs • GAs evolve by selection and mutation  their dynamics can be (to some extent) described by the replicator equation with mutation (cf. unifying framework). • The replicator equation describes fitness-proportionate selection. • Ficici, Melnik and Pollack (2000) - effects of different types of selection (eg. truncation) on the dynamics of the Hawk-Dove game + relevance for evolutionary algorithms. Can lead to different dynamics. • Must also consider the effects of recombination.

37. Incorporating recombination into the replicator framework • Do this by assuming that rjk;i = probability that when parent chromosome of type j combines with parent chromosome of type k, an offspring of type i is formed. • No mutation, recombination after replication: • [NB discrete-time version]

38. Adding in mutation • Add in mutation. Assume, as before, is probability type i mutates to form type j ( large). Assume this happens after recombination. • What we had before was • What we have now is

39. The diversity of the population and adaptive dynamics • From Fisher’s theorem, see that no diversity of fitness in population  no further increase in average fitness. • However, because the variation in the parameters of the your system has become very small (population convergence), does not mean no further evolution. • In the case of small variation, we can apply the adaptive dynamics framework which shows how the average values of traits (parameters) will change in time

40. Relationship: evolutionary games & GAs - Conclusions • Often evolution leads in the long run to ‘optimal’ solutions, like Nash equilibria. • Ability of evolutionary processes to seek out optimal strategies has been exploited in computer science by the development of genetic algorithms and evolutionary computation for problem solving. • Comparing with the use of computer simulations to study biological evolution, we see that there is a two-way interaction between biological evolutionary theory and computer science.

41. Relationship to GAs- Conclusions • Frameworks of evolutionary dynamics can be applied to GAs by modifying them to include recombination. • Which framework is most informative depends on the individual problem, but we have shown they are equivalent. • Eg. can look at detailed dynamics using the replicator-mutator framework • Or we can look at a “converged” population using the adaptive dynamics framework. • Looking further at the relationship between GAs and evolutionary dynamics could yield new solutions/ techniques for both.

42. Acknowledgements • Martin Nowak (IAS, Princeton) • Terry Leaves (BNP Paribas, London) • Karl Sigmund (Univ. Vienna) • Steven Frank (Univ. California, Irvine) • Peter Bentley (UCL) • Christoph Hauert (Univ. British Columbia) http://www.univie.ac.at/virtuallabs/Spatial2x2Games/ • Anargyros Sarafopoulos (Univ. Bournemouth) • Bernard Buxton (UCL)