Evolutionary Computation: introduction Dirk Thierens Universiteit Utrecht The Netherlands Dirk Thierens (Universiteit Utrecht) EC Introduction 1 / 44 What? Evolutionary Computation Evolutionary Computation = Population-based, stochastic search algorithms based on the mechanisms of natural evolution Evolution viewed as search algorithm or problem solver. Natural evolution only used as metaphor for designing computational systems ($ evolutionary biology). Dirk Thierens (Universiteit Utrecht) EC Introduction 2 / 44
Evolutionary Computation Why? Example: NASA ST5 spacecraft antenna This complicated shape was found by an evolutionary computer design program to create the best radiation pattern. Dirk Thierens (Universiteit Utrecht) EC Introduction 3 / 44 Evolutionary Computation Darwinian process characteristics 5 Key requirements of a Darwinian system 1 Structures: Information 2 Structures are copied 3 Copies partially vary from the original: Inheritance 4 Structures are competing for a limited resource: Selection 5 Relative reproductive success depends on the environment: Survival of the Fittest Dirk Thierens (Universiteit Utrecht) EC Introduction 4 / 44
Evolutionary Computation Darwinian process characteristics in EC 1 Structures ) e.g. binary strings, real-valued vectors, programs,... 2 Structures are copied ) selection algorithm: e.g. tournament selection,... 3 Copies partially vary from the original ) mutation & crossover operators 4 Structures are competing for a limited resource ) selecting fixed sized parent pool 5 Reproductive success depends on environment ) user defined fitness function Dirk Thierens (Universiteit Utrecht) EC Introduction 5 / 44 Evolutionary Computation Dirk Thierens (Universiteit Utrecht) EC Introduction 6 / 44
Genetic Algorithm GAs: structures = discrete representations. Neo-Darwinism...AUUCGCCAAU... ) organism Genetic Algorithm...0101001111... ) F(...0101001111...) 2< User: String representation and function F GA: string manipulation selection: copy better strings variation: generate new strings Dirk Thierens (Universiteit Utrecht) EC Introduction 7 / 44 Selection methods Genetic Algorithm Proportionate Selection Probability individual i selected: f i P fj (f i : fitness ind. i ; j: 1! N) Expected number of copies of ind. i : f i P fj.n = f i f (t) (N: population size) Disadvantages: 1 An initial solution might be a lot better than the rest by chance and will be selected too often ) loss of diversity! 2 When population starts to convergence to similar solutions the selection pressure disappears. Dirk Thierens (Universiteit Utrecht) EC Introduction 8 / 44
Genetic Algorithm Ranked based selection methods Selection based on relative fitness rather than absolute fitness values. Truncation selection Sort population based on fitness. Select the top %. Copy each selected solution 100 times. Tournament selection Randomly select (with replacement) Select the best of these Hold N tournaments. solutions. solutions. (N = population size) Dirk Thierens (Universiteit Utrecht) EC Introduction 9 / 44 Variation methods Genetic Algorithm Mutation {1111111111} ) {1111111011} Crossover 2-point crossover: 1111111111 0000000000 1111111111 uniform crossover: 0000000000 ) ) 1111000011 0000111100 1001110101 0110001010 Dirk Thierens (Universiteit Utrecht) EC Introduction 10 / 44
Toy example Genetic Algorithm x [0, 31] : f (x) =x 2 binary integer representation: x i {0, 1} x = x 1 2 4 + x 2 2 3 + x 3 2 2 + x 4 2 1 + x 5 2 0 Initial Random Population: 10010 : 18 2 = 324 01100 : 12 2 = 144 01001 : 9 2 = 81 10100 : 20 2 = 400 01000 : 8 2 = 64 00111 : 7 2 = 49 population mean fitness f (0) = 177 Dirk Thierens (Universiteit Utrecht) EC Introduction 11 / 44 Genetic Algorithm Generation 1: tournament selection ( = 2), 1-point crossover, mutation Parents Fitness Offspring Fitness 100!10 324 10100 400 101!00 400 10111 529 01!000 64 00010 4 10!010 324 10010 324 0110!0 144 11100 784 1010!0 400 10000 256 Parent population mean fitness f (1) =383 Dirk Thierens (Universiteit Utrecht) EC Introduction 12 / 44
Genetic Algorithm Generation 3: Parents Fitness Offspring Fitness 1!1111 961 11110 900 1!1100 784 11011 729 110!00 576 11110 900 111!10 900 11101 841 1101!1 729 11111 961 1100!1 625 01001 81 Parent population mean fitness f (3) =762 Dirk Thierens (Universiteit Utrecht) EC Introduction 13 / 44 Permutation Representation Traveling salesman problem Find the shortest route while visiting all cities exactly once. Dirk Thierens (Universiteit Utrecht) EC Introduction 14 / 44
Permutation Representation Permutation problems Solutions are permutations of a set of elements. Tour representation: list of cities? I p1 = 1 2 3 4 5 6 7 8 I p2 = 4 6 2 1 7 8 5 3 I simple crossover ) illegal tours I c1 = 1 23 1 7853 I c2 = 4 6 2 4 5678 Need suitable representations and genetic operators for permutation problems. Dirk Thierens (Universiteit Utrecht) EC Introduction 15 / 44 Mutation Permutation Representation Swap mutation parent A B C D E F G H I swap -? - -? - - - - child A E C D B F G H I Randomly select and swap 2 elements. Dirk Thierens (Universiteit Utrecht) EC Introduction 16 / 44
Permutation Representation Crossover Cycle crossover parent 1 A B C D E F G H I parent 2 f e d a b c h i g cycles 1 2 1 1 2 1 3 3 3 child A e C D b F h i g Mark and randomly recombine elements in a cycle set. Dirk Thierens (Universiteit Utrecht) EC Introduction 17 / 44 Permutation Representation Permutation problems Permutation problems have different characteristics: adjacency relative order absolute order Cycle crossover emphasizes absolute position above adjacency or relative order. TSP problems: adjacency only aspect that counts. Dirk Thierens (Universiteit Utrecht) EC Introduction 18 / 44
Edge recombination Permutation Representation Edge map: Table of all cities with their direct neighbors in both parents. edge map: Two parent tours [ABCDEF] & [BDCAEF] city A B C D E F edges BFCE ACDF BD A C EB DF A AE B : common edge of both parents Dirk Thierens (Universiteit Utrecht) EC Introduction 19 / 44 Permutation Representation Edge recombination algorithm 1 Choose initial city from one parent. 2 Remove current city from edge map. 3 If current city has remaining edges: goto step 4. else goto step 5. 4 Choose from current city a common edge, if none present choose the city with fewest remaining edges (ties broken randomly). 5 If still remaining cities, choose one with fewest remaining edges. Dirk Thierens (Universiteit Utrecht) EC Introduction 20 / 44
Permutation Representation 1 random choice ) B 2 next candidates: A C D F choose from C D F (same edge number) ) D 3 next candidates: C E (C is a common edge) ) C 4 next candidate: A ) A 5 next candidates: F E (same edge number) ) F 6 next candidate: E ) E Resulting tour = [BDCAFE]. Dirk Thierens (Universiteit Utrecht) EC Introduction 21 / 44 Permutation Representation Properties Parent tours: [ABCDEF] &[BDCAEF]. Child tour = [BDCAFE]. Common edges in the parents are inherited: CD and EF. Most edges in the child tour are inherited from (at least) one of the parent tours. Dirk Thierens (Universiteit Utrecht) EC Introduction 22 / 44
Permutation Representation Fitness correlation coefficients What variation operator is most suitable for a given problem? Genetic operators should preserve useful fitness characteristics between parents and offspring = inheritance. Calculate the fitness correlation coefficient to quantify this. Operator fitness correlation coefficient op : op = cov(f p, F c ) (F p ) (F c ) Dirk Thierens (Universiteit Utrecht) EC Introduction 23 / 44 Permutation Representation Fitness correlation coefficients K-ary operator: generate n sets of k parents. ) {f (p g1 ), f (p g2 ),...,f (p gn } Apply operator to each set to create children. ) {f (c g1 ), f (c g2 ),...,f (c gn } F p : mean fitness of the parents F c : mean fitness of the children (F p ) = standard deviation of fitness parents (F c ) = standard deviation of fitness children cov(f p, F c )= P n (f (p gi ) F p )(f (c gi ) F c ) i=1 n covariance between fitness parents and fitness children Dirk Thierens (Universiteit Utrecht) EC Introduction 24 / 44
Permutation Representation Traveling Salesman problem: crossover operators For TSP edge crossover (EX) gives better results than cycle crossover (CX). This performance difference is reflected in the crossover correlation coefficients cross : EX 0.90 CX 0.57 Dirk Thierens (Universiteit Utrecht) EC Introduction 25 / 44 Dirk Thierens (Universiteit Utrecht) EC Introduction 26 / 44
Evolving a Checkers Player Can we build intelligent systems to learn to play checkers? No expert knowledge provided to the learning system. Programs simply have to play against themselves, and figure out how to play. Evolutionary computation feasible approach? Dirk Thierens (Universiteit Utrecht) EC Introduction 27 / 44 Game playing Board representation Move search: minimax algorithm Traditional game playing programs: board evaluation functions are extensively knowledge based 1 weighted feature function 2 opening games 3 end games table look-up ) they do not learn by themselves! Here: Evolving Neural Networks to Play Checkers Dirk Thierens (Universiteit Utrecht) EC Introduction 28 / 44
Board representation Output 2 [ 1...+ 1] -1: loss positions +1: win positions! closer to +1 ) better evaluations Input: vector of 32 possible positions, 5 possible values 1 - K : king opponent 2-1 : checker opponent 3 0 : empty 4 + 1 : checker self 5 + K : king self K 2 [1...3] : exact value evolved Dirk Thierens (Universiteit Utrecht) EC Introduction 29 / 44 Mini-Max algorithm Dirk Thierens (Universiteit Utrecht) EC Introduction 30 / 44
Game lookahead The further we can lookahead the better. Computational restrictions: number of possible board positions grows very fast with increasing number of lookahead levels. Deep Blue when defeating chess champion Garry Kasparov made 200 million chess board evaluations per second! Here only lookahead search of 2 moves each side when evolving. When testing against players on Internet: lookahead search of 3 moves each side. Dirk Thierens (Universiteit Utrecht) EC Introduction 31 / 44 Board Evaluation Evaluation function represented by an artificial neural network Dirk Thierens (Universiteit Utrecht) EC Introduction 32 / 44
Neural Network Architecture Input Layer: 32 inputs First hidden layer: 40 neurons Second hidden layer: 10 neurons Direct input-output connections with weight 1.0 Total number of neural network weights (incl. bias term) = (32 + 1) x 40 + (40 + 1) x 10 + (10 + 1) x 1 = 1741 Need to determine values for the 1741 weights + Use evolutionary algorithm to co-evolve the weights Dirk Thierens (Universiteit Utrecht) EC Introduction 33 / 44 Evolutionary search for network weights Mutate neural network weights by adding a small random, Gaussian distributed number to each weight. Each weight has its own Gaussian distribution (different widths or standard deviations). The width or variance of each Gaussian distribution also evolves by mutation. For each neural network NN i all the N w (= 1741) neural network weights w i (j) (j = 1...N w ) gets associated with the corresponding standard deviation i (j) (j = 1...N w ) of the Gaussian mutation distribution (mean value is always zero). Dirk Thierens (Universiteit Utrecht) EC Introduction 34 / 44
Self-adaptive Mutation First mutate the N w widths of the Gaussian distributions, then mutate the N w weights (j = 1...N w ): Self-adaptive mutation of the mutation step-size: 0 i (j) = i(j) exp( RandNorm j(0,1) p2 p N w ) Mutation of the neural network weights: w 0 i (j) =w i(j)+ 0 i (j)randnorm j(0, 1) Dirk Thierens (Universiteit Utrecht) EC Introduction 35 / 44 Evolutionary Cycle Dirk Thierens (Universiteit Utrecht) EC Introduction 36 / 44
Fitness evaluation 15 parents + 15 offspring neural networks Each NN competes against 5 randomly chosen NN Score: win : + 1; draw : 0; loss : -2 Fitness: sum of scores Dirk Thierens (Universiteit Utrecht) EC Introduction 37 / 44 Darwinian system? 1 Structures?! neural networks 2 Structures are copied?! 15 parents ) 15 offspring 3 Copies partially vary from the original! weights Gaussian mutated 4 Structures are competing for a limited resource! fixed population size each generation 5 Reproductive success depends on environment! winning strategies survive Dirk Thierens (Universiteit Utrecht) EC Introduction 38 / 44
Experiment total of 250 generations evolved 15 neural networks each generation ) 15 x 250 = 3750 neural networks created fitness evaluation: 15 parents + 15 offspring: each competing against 5 others ) 30 x 5 x 250 = 37500 games played Dirk Thierens (Universiteit Utrecht) EC Introduction 39 / 44 Checkers Rating Dirk Thierens (Universiteit Utrecht) EC Introduction 40 / 44
Evolved neural network rating: 1902 Dirk Thierens (Universiteit Utrecht) EC Introduction 41 / 44 1 draw against player rated 2207, ie. master level, ranked 18 out of 40000 listed players Dirk Thierens (Universiteit Utrecht) EC Introduction 42 / 44
Discussion Chinook: opening games, end games table look-up, hand-crafted evaluation function Deep Blue: 200 million board evaluations per second vs. 3500 here Payoff: summed score over 5 games! no immediate feedback about winning or losing a single game Input to neural networks do not give spatial information: only 1 x 32 vector of {-K,-1,0,1,K} Allen Newell: It is extremely doubtful whether there is enough information in win, lose, or draw when referred to the whole play of the game to permit any learning at all over available time scales Dirk Thierens (Universiteit Utrecht) EC Introduction 43 / 44 Conclusion Building intelligent systems by evolutionary computing. Learn to play checkers at high level by competing against themselves. No tedious domain knowledge extraction. Learning by evolution: feasible approach. Dirk Thierens (Universiteit Utrecht) EC Introduction 44 / 44