Méta-heuristiques pour l optimisation Differential Evolution (DE) Particle Swarm Optimization (PSO) Alain Dutech Equipe MAIA - LORIA - INRIA Nancy, France Web : http://maia.loria.fr Mail : Alain.Dutech@loria.fr 3 mai 2010
Context : Neuro-Sciences Question : influence/role of pre-frontal-cortex in reinforcement-like learning in rats? (Context of Operant Conditionning, work with Alain Marchand, Bordeaux). Real Data PFC-lesion/normal SARSA State : w/o memory Policy Fitting Optimize Softmax-Policy Likelyhood Likelyhood Page 2
Global Optimization Search space S x = (α, γ, β) T Objective function to minimize f is unknown f : S R f : policy likelyhood (vector? traj?) sampled. Global Optimization non-convex objective function. unconstrained state space. Meta-heuristics : Simulated Annealing, Tabu Search, Ant Colony Optimization, Evolutionary Algorithm (Differential Evolution), Particle Swarm Optimization, Variable Neighborhood Search, etc. Ceci n est pas exhaustif! Page 3
Differential Evolution (DE) [Storn and Price, 1995] D-dimensionnal parameter vector x = (x 0,, x D ) T (1) Population of N parameter vectors at generation g Evolution of one parameter vector 1. mutation : 2. cross-over : X [g] = { x i [g]}, i = 0,, N (2) x i [g + 1] = m i = f (X [g]) (3) { m j i for some j x j i [g] otherwise (4) Page 4
DE canonical : uninformed evolution Mutation : r 1, r 2 and r 3 randomly different in [0,, N] Cross-Over : n and L randomly chosen x i [g + 1] = v i = x r1 [g] + F.( x r2 [g] x r3 [g]) (5) { v j i for j = n D,, n + L 1 D x j i [g] otherwise (6) Page 5
DE canonical : uninformed mutation x 2 min x i [g] x r3 [g] x r2 [g] x r1 [g] ( x r2 [g] x r3 [g]) x r1 [g] + F.( x r2 [g] x r3 [g]) v i x 1 v i = x r1 [g] + F.( x r2 [g] x r3 [g]) Page 6
DE canonical : informed mutation x 2 ( x best [g] x i [g]) x i [g] x r3 [g] x r2 [g] x best [g] v i x r1 [g] min x i [g] + λ.( x best [g] x i [g]) +F.( x r2 [g] x r3 [g]) ( x r2 [g] x r3 [g]) x 1 v i = x i [g] + λ.( x best [g] x i [g]) + F.( x r2 [g] x r3 [g]) Page 7
DE canonical : cross-over x i [g] v i = f (X [g]) x i [g + 1] j = 0 1 2 3 4 5 6 j = 0 1 2 3 4 5 6 j = 0 1 2 3 4 5 6 (n = 2, L = 3) Page 8
Particle Swarm Optimisation (PSO) [Kennedy and Eberhart, 1995] D-dimensionnal parameter vector and velocity x = (x 0,, x D ) T (7) v = (v 0,, v D ) T (8) Population of N parameter vectors and velocity at generation g Initialisation Evolution of one parameter vector 1. velocity : 2. position : (9) X [g] = {( x i [g], v i [g])}, i = 0,, N (10) v i [g + 1] = f (X [g]) (11) x i [g + 1] = x i [g] + v i [g + 1] (12) Page 9
PSO canonical Velocity : ϕ 1 and ϕ 2 are accelaration constants R 1 and R 2 are uniformly chosen in [0, 1] x ibest best past position of i x i,nbest best position in neighbors of i (past??) v i [g + 1] = v i [g] + ϕ 1.R 1.( x ibest [g] x i [g]) + ϕ 2.R 2.( x i,nbest [g] x i [g]) (13) Position : x i [g + 1] = x i [g] + v i [g + 1] (14) Page 10
Constraints Enforce constraints on particles Projection into constrained space Periodic Search Space [Zhang et al., 2004] Page 11
Neighbors What is x i,nbest [g]? x k [g] ɛ x i [g] x j [g] Sphere, radius ɛ Page 12
Neighbors What is x i,nbest [g]? x i [g] Small world Page 13
Neighbors What is x i,nbest [g]? x i [g] Small world with random added links [Saxena and Vora, 2008] Page 14
DE+PSO Hybrid algorithm [Zhang et al., 2009] Initialisation Population of N particles. Only one particle in ɛ-radius sphere. Update best & neighbors nb iter multiple of I? Y DE N PSO Special x i == x i,nbest N nb iter nb max? Y End Page 15
DE in hybrid algorithm Mutation : j randomly different in [0,, N] x ibest best past position of i x i,nbest best position in neighbors of i (past??) v i = x i,nbest [g] + F.( x ibest [g] x jbest [g]) (15) Cross-Over : D i randomly chosen[0,, N], C R in [0, 1] x i [g + 1] = { v j i if j = D i or with proba C r x j i [g] otherwise (16) Page 16
PSO in hybrid algorithm Canonical version of PSO (small world + 2 rnd links). Prevent beeing stuck with null-velocity when ( x i == x i,nbest ) x i [g + 1] = x i,nbest [g] + δ. rand (17) v i [g + 1] = x i [g + 1] x i [g] (18) Page 17
B2 problem One global minimum of 0 at (0, 0) B2(x 1, x 2) = x 2 1 + 2x 2 2 0.3 cos(3πx1) 0.4 cos(4πx2) + 0.7 Page 18 100 < x j < 100,j = 1, 2
ES problem One global minimum of 1 at (π, π) (π, π) Es(x 1, x 2) = cos(x 1) cos(x 2). exp( ((x 1 x 2) 2 + (x 2 π) 2 )) Page 19
GP problem One global minimum of 3 at (0, 1) GP(x 1, x 2) = [1 + (x 1 + x 2 + 1) 2.(19 14x 1 + 3x 2 1 14x2 + 6x1x2 + 3x 2 2 )] 2 < x j < 2, j = 1, 2 [30 + (2x 1 3x 2) 2.(18 32x 1 + 12x 2 1 + 48x2 36x1x2 + 27x 2 2 )] Page 20
SH problem 760 local minima, 18 global minima of 186.73. 5 5 SH(x 1, x 2) = j cos[(j + 1)x 1 + j] j cos[(j + 1)x 2 + j] j=1 j=1 10 < x j < 10, j = 1, 2 Page 21
Références I Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. In Proc. of the IEEE Int. Conf. on Neural Networks (ICNN 95), volume 4, pages 1942 1948. Saxena, A. and Vora, M. (2008). Novel approach for the use of small world theory in particle swarm optimization. In 16th Int. Conf. on Advanced Computing and Communications (ADCOM 2008), pages 363 366. Storn, R. and Price, K. (1995). Differential evolution - A simple and efficient adaptive scheme for global optimization over continuous spaces. Technical Report TR-95-012, International Computer Science Institute. Page 22
Références II Zhang, C., Ning, J., Lu, S., Ouyang, D., and Ding, T. (2009). A novel hybrid differential evolution and particle swarm optimization algorithm for unconstrained optimization. Operations Research Letters, 37 :117 122. Zhang, W.-J., Xie, X.-F., and Bi, D.-C. (2004). Handling boundary constraints for numerical optimization by particle swarm flying in periodic search space. In Congress on Evolutionary Computation (CEC), pages 2307 2311, Prtland, OR. Page 23