A Mathematical Modelling Technique for the Analysis of the Dynamics of a Simple Continuous EDA

Transcription

1 A Mthemtcl Modellng Technque for the Anlyss of the Dynmcs of Smple Contnuous EDA Bo Yun, Student Member, IEEE, nd Mrcus Gllgher, Member, IEEE Abstrct Ths pper presents some ntl ttempts to mthemtclly model the dynmcs of contnuous Estmton of Dstrbuton Algorthm (EDA) bsed on Gussn dstrbutons. Cse studes re conducted on both unmodl nd multmodl problems to hghlght the effectveness of the proposed technque nd eplore some fundmentl ssues of the EDA. Wth some generl ssumptons, we cn show tht, for one-dmensonl unmodl problems nd wth the (µ, λ) scheme: (). The convergence behvour of the EDA s ndependent of the test functon ecept ts generl shpe; (). When strtng fr wy from the globl optmum, the EDA my get stuck; (3). Gven certn selecton pressure, there s unque prmeter vlue tht could help the EDA cheve desrble performnce; for one-dmensonl multmodl problems: (). The EDA could get stuck wth the (µ, λ) scheme; (). The EDA wll never get stuck wth the (µ+λ) scheme. I. INTRODUCTION Estmton of Dstrbuton Algorthms (EDAs) [8] refer to reltvely new prdgm of Evolutonry Algorthms (EAs), whch s bsed on sttstcl Mchne Lernng technques nsted of conventonl genetc opertors such s crossover nd mutton. The mor dvntge of EDAs s tht they cn eplctly buld probblty densty functon bsed on the dstrbuton of promsng ndvduls nd utlze the dependence nformton to conduct optmzton effcently, whch s prtculr mportnt upon the presence of strong epstss mong vrbles. Usully, EAs re evluted on the emprcl bss whle only lmted progresses hve been mde n the theoretcl spect, especlly compred to the lrge number of lgorthms n the lterture[]. Ths s mnly due to ther comple dynmcs nd mssve prllel behvour s well s the lck of pproprte modellng technques. In trdtonl EAs, new ndvduls re generted by drectly mnpultng current ndvduls through vrous stochstc genetc opertors. Consequently, t my be not strghtforwrd to precsely estmte the new populton nd thus predct the detled evoluton process n ech generton. Fortuntely, ths ssue my be less chllengng n EDAs becuse the serchng process s solely drven by hghlevel sttstcl model nd ll ndvduls re generted by smplng from ths model. In order to descrbe the behvour Bo Yun s wth the School of Informton Technology nd Electrcl Engneerng, The Unversty of Queenslnd, Brsbne, QLD 407, Austrl (phone: ; eml: boyun@tee.uq.edu.u) Mrcus Gllgher s wth the School of Informton Technology nd Electrcl Engneerng, The Unversty of Queenslnd, Brsbne, QLD 407, Austrl (phone: ; eml: mrcusg@tee.uq.edu.u) of n EDA, only the model needs to be estmted, whch cn often be fully specfed by few prmeters. Ths clrty n the mechnsm of EDAs mkes t possble to conduct more detled theoretcl nlyss[, 4, 7, 9-, 3]. In ths pper, generl technque s proposed to model the behvour of contnuous EDA bsed on Gussn dstrbutons nd Truncton Selecton, whch s med t precsely predctng the Gussn model n ech generton specfed by the men nd stndrd devton prmeters. The mportnce of ths technque s tht t cn be used to theoretclly nvestgte some fundmentl ssues such s the nfluence of vrous lgorthm fctors nd the problem structure. As to the prevous relted work tht we re wre of, González et. l nd Grhl et. l present some theoretcl results for the UMDA c lgorthm, whch s equl to the EDA used n ths pper n one dmensonl spces [5, 6]. However, our work s dfferent n severl spects. For emple, we consder not only unmodl but lso multmodl problems on whch some nterestng dynmcs could be observed. Furthermore, n ddton to the (µ, λ) scheme, the more comple (µ+λ) scheme s lso tken nto ccount (.e., n both cses µ=λ), whch hs not been ttempted before. II. ESTIMATION OF DISTRIBUTION ALGORITHMS TABLE I THE GENERAL FRAMEWORK OF THE GAUSSIAN EDA Intlze nd evlute the strtng populton P Whle stoppng crter not met Select the top ndvduls P sel Ft Gussn model θ to P sel Smple set of ndvduls P' from θ Evlute new ndvduls n P' Use P' s the new populton or Select the top ndvduls from P U P' End Whle The generl mechnsm of EDAs s n tertve process of evolvng sttstcl model such s Gussn models, Gussn Mture models nd Gussn Networks, whch specfes the dstrbuton of the promsng ndvduls n the current populton. The mor motvton s tht, wth the gudnce of the sttstcl model, EDAs re epected to be ble to eplctly cpture the dependence reltonshp mong Dgstuhl Semnr Proceedngs 0606 Theory of Evolutonry Algorthms

2 problem prmeters nd my cheve fster convergence speed when comple dependence reltonshp does est n the problem of nterest. In the contnuous EDA used n ths pper (Tble ), porton of top ndvduls n the current populton P re selected usng Truncton Selecton t ech generton nd multvrte Gussn model θ wth full covrnce structure s then bult usng the mmum lkelhood estmte. All new ndvduls P' re generted by smplng from θ nd the new populton s creted through ether the (µ, λ) scheme (.e., replce ll old ndvduls) or the (µ+λ) scheme (.e., choose the top ndvduls from the unon of old nd new ndvduls). Note tht P nd P' re of the sme sze n ths frmework (.e., µ=λ). III. MODELLING UNIMODAL PROBLEMS Fg.. A typcl one-dmensonl unmodl problem The D unmodl problems consdered n ths secton hve the generl structure lke the qudrtc functon y= s shown n Fgure. The only ssumpton s tht ech problem s symmetrc wth regrd to ts globl mnmum, whch s plced t the orgn wthout loss of generlty (.e., there re no upper nd lower boundres). For the EDA, n nfntely lrge populton sze s ssumed so tht new ndvduls hve ectly the sme sttstcs s the Gussn model from whch they re smpled. However, s wll be shown n the eperments, moderte populton sze s often suffcent to produce close mtch between emprcl nd theoretcl results. Furthermore, t s ssumed tht the ntl populton s generted from Gussn (µ 0, 0 ). Durng evoluton, the Gussn model wll be evolved towrds the orgn wth chngng men nd stndrd devton vlues. A demo of the dynmcs of the EDA s shown n Fgure. Fg.. A demo of the dynmcs of the EDA on the unmodl problem Wth the (µ, λ) scheme, the Gussn model to be bult n the th generton s completely determned by selected ndvduls n the th populton smpled from the Gussn model bult n the (-) th generton. As result, there s smple recursve reltonshp mong the models n dfferent genertons. Another mportnt feture s tht the selected ndvduls n ny generton re restrcted wthn contnuous rnge [-α, α] s shown n Fgure wth the vlue of α dependng on the model prmeters s well s the selecton pressure (0<<). Ths s becuse Truncton Selecton pples determnstc ftness threshold nd only ndvduls strctly better thn t could be chosen. The specfc vlue of α n the (+) th generton cn be clculted by solvng the followng equton: ( µ ) e d = () π The menng of Eq. s tht the cumultve densty of the Gussn dstrbuton wthn [-α, α] should be equl to, whch s the porton of ndvduls to be selected. Although t my be dffcult to come up wth n nlytcl soluton to Eq., t s esy to see tht the vlue of the left hnd sde ncreses from zero monotonclly s α +, whch mkes t possble to use smple lne serchng method to fnd the pproprte α vlue t desred ccurcy level. Once the vlue of α s vlble, the men prmeter µ + nd stndrd devton prmeter + re determned by the sttstcs of the ndvduls wthn [-α, α]: ( µ ) + = e π µ d () ( µ ) e π + = ( µ ) + Now we hve enough tools to descrbe the EDA s behvor on unmodl problems. It should be ponted out tht, n the bove three equtons, there s no utlzton of ny specfc nformton of test functons. Ths shows tht the convergence behvor of the EDA n ths stuton s ndependent of the problems ecept ther generl structure (.e., strctly unmodl, symmetrc wth regrd to the orgn). In the net, some cse studes re to be conducted to formlly nvestgte the dynmcs of the EDA wth regrd to vrous lgorthm fctors. Fgures 3&4 shows the stndrd devtons nd the men vlues of the EDA wth µ 0 = -0 nd =0.3 where the three lnes n ech plot represent results wth 0 =0.,.0 nd 5.0 respectvely. A few nterestng thngs tht cn be observed from the bove cse study re summrzed s below: When the men of the Gussn model ws fr from the globl optmum (.e., severl stndrd devtons wy from the orgn), the trectores of the stndrd devton vlues were ppromtely logrthmclly lner, whch mens tht the stndrd devtons of the Gussn model reduced eponentlly. d (3)

3 The grdent of the trectory ws ndependent of 0 (.e., ll stndrd devton trectores were prllel to ech other). The EDA could stll get stuck on unmodl problems even wth n nfnte populton, especlly wth smll ntl stndrd devtons s shown by the flt curves n Fgure 4. Fg. 5. The trectores of the stndrd devton prmeter of the Gussn model on unmodl problems wth dfferent selecton pressures Fg. 3. The trectores of the stndrd devton prmeter of the Gussn model on unmodl problems wth dfferent ntl 0 vlues Fg. 6. The trectores of the men prmeter of the Gussn model on unmodl problems wth dfferent selecton pressures Fg. 4. The trectores of the men prmeter of the Gussn model on unmodl problems wth dfferent ntl 0 vlues In order to demonstrte the nfluence of the selecton pressure on the dynmcs of the EDA, the stndrd devtons nd the men vlues of the EDA wth µ 0 = -0 nd 0 =.0 re shown n Fgures 5&6 where the three lnes represent results wth selecton pressure =0., 0.3 nd 0.5 respectvely. It s cler tht s the selecton pressure ncresed, the convergence speed of the EDA lso ncresed ccordngly (.e., the slope of the trectory of the stndrd devton vlues becme steeper). Agn, no stsfctory performnce could be cheved s the EDA quckly got stuck somewhere fr from the optmum. Note tht wth wek selecton pressure (.e., =0.5), the EDA converged reltvely slowly but only mde lmted progress before gettng stuck. Ths s becuse the lrger the porton of ndvduls to be selected, the smller the dstnce between µ nd µ +. In the etreme stuton where =.0, the EDA wll smply keep buldng new model bsed on the populton generted from the prevous model, whch wll result n sttonry sttstcl model nd no progress could be epected. Certnly, n prctcl stutons where the populton szes re lmted, the EDA s epected to present some knd of rndom wlkng behvour. In the bove two cse studes, the EDA ws never ble to converge to the globl optmum nd, s suggested by our prevous reserch[], t s necessry to eplctly mntn the populton dversty to prevent the EDA from convergng too quckly. A smple pproch s to ntroduce new lgorthm prmeter γ, whch s used to mplfy the orgnl stndrd devton prmeter (.e., ' = γ ). Certnly, for γ=.0, the EDA wll be kept unchnged. Fgures 7 & 8 shows the stndrd devtons nd the men vlues of the EDA wth µ 0 = -0, 0 =.0 nd =0.3 where the three lnes n ech plot represent results wth γ =.0,.5 nd.0 respectvely. It s evdent tht ths new prmeter could drmtclly chnge the dynmcs of the EDA. Wth γ>.0, the stndrd devtons were often orders of mgntude lrger thn n the orgnl EDA. In the mentme, the men of the Gussn model could lso consstently move towrds the globl optmum wth γ=.0, showng some sgnfcntly mproved performnce. However, t does not necessrly men tht such lrge γ vlues should be used n prctce where the populton sze s lmted. The ssue s tht the stndrd devton my ncrese durng evoluton nd become qute lrge, whch mkes the EDA close to dong rndom serch. Snce the dstnce between the current model nd the globl optmum s typclly not known n optmzton, hvng the stndrd devton ncrese or decrese my ll seem to be dngerous. As result, good strtegy s to keep the current stndrd devton vlue unchnged. Recll tht, when the EDA s fr from the globl optmum, ts convergence trectory s only nfluenced by nd s ndependent of the ntl stndrd devton. So, for dfferent vlues, there re correspondng 3

4 optmum γ vlues (.e., γ*= / +). For emple, the γ* vlues for =0., 0.3 & 0.5 re ppromtely.435,.9443 nd.6589 respectvely. Fg. 9. The trectory of the stndrd devton prmeter of the Gussn model on unmodl problems wth dversty mntennce (γ*=.9443, =0.3) Fg. 7. The trectores of the stndrd devton prmeter of the Gussn model on unmodl problems wth dversty mntennce Fg. 0. The trectory of the men prmeter of the Gussn model on unmodl problems wth dversty mntennce (γ*=.9443, =0.3) Fg. 8. The trectores of the men prmeter of the Gussn model on unmodl problems wth dversty mntennce In order to demonstrte the effect of the optmum γ vlues, we nvestgted both emprclly nd theoretclly the EDA wth γ*=.9943 n ccordnce wth =0.3. The EDA strted wth µ 0 =-0 nd 0 =.0 nd, for emprcl studes, the populton sze ws 5000 nd results were verged over 5 ndependent trls. Fgure 9 shows tht the stndrd devton of the EDA ws lmost unchnged wthn the frst 6 genertons due to the γ* vlue n use. After tht, when the EDA ws qute close to the globl optmum s shown n Fgure 0, t strted convergng very quckly nd chnged ts behvor to locl serchng. Ths s becuse when t s fr from the globl optmum, the problem could be regrded s beng monotonous nd ts behvor s ppromtely consstent. However, the grdent of the convergence trectory of the EDA gets sgnfcntly lrger when t s close to the globl optmum nd the orgnl γ* vlue s no longer lrge enough to keep the stndrd devton unchnged. Ths phenomenon could be roughly eplned by the fct tht when the Gussn model s very close to the optmum, selected ndvduls tend to dstrbute wthn much smller rnge thn when the Gussn model s fr from the optmum. In fct, ths property s very useful s t could dpt the EDA s behvor dependng on ts current stuton (.e., ths s n mportnt dvntge over hll-clmbng lgorthm wth fed step sze). At lst, the emprcl results mrked by + hd good mtch gnst the theoretcl nlyss, whch verfed the soundness of our methods. IV. MODELLING MULTIMODAL PROBLEMS In ths secton, we wll pply the modellng technques n more generl wy. Frstly, the problems to be solved hve globl optmum nd locl optmum. Secondly, n ddton to the reltvely smple (µ, λ) scheme, the (µ+λ) scheme s lso consdered, whch choose the best ndvduls from the unon of both old nd new popultons. For the followng nlyss nd eperments, test problems (.e., to be mmzed) re constructed by two Gussn functons due to ther smoothness nd symmetrc shpe (.e., ths hs nothng to do wth the Gussn model of the EDA). The ftness vlue of s determned by the Gussn functon tht gves t the lrger vlue. Ech Gussn functon s governed by ts µ nd whle ω s used to dust ts heght. { G ( ), G ( )} F ( ) = m (4) G ( ) = ω wth ( µ ) e ω > π Two emples re shown n Fgures & wth dfferent prmeter vlues. When there re two or more peks n the problem, t s not lwys suffcent to descrbe the rnge of selected ndvduls by sngle ntervl becuse they my come from dfferent peks. In Fgure, when the worst ftness vlue (selecton threshold) ndcted by lne A s greter thn the vlue of the locl optmum t =0., ll selected ndvduls wll come from the pek correspondng 0 4

5 to the globl optmum t =-0.3 nd bounded wthn [, ], whch s smlr to the unmodl cse. On the other hnd, when the threshold s lower, s ndcted by lne B, the selected ndvduls wll be from both peks nd two ntervls [, ] nd [ 3, 4 ] re requred. Furthermore, when the two peks re very close, t s possble tht only one ntervl s needed, s shown n Fgure. Fg.. A multmodl problem wth two seprte optm Once the ntervls re determned, the new model prmeters re clculted n smlr mnner s n the unmodl cse (.e., =, ] U [, ] ): [ 3 4 = d + P( ) µ (6) + = ( µ + ) P( ) d (7) The mor dfference between the (µ, λ) scheme nd the (µ+λ) scheme s tht the ltter my keep old ndvduls of hgh qulty to the net generton. Tht s to sy, the th populton s not solely smpled from the sttstcl model bult n the (-) th generton. Insted, t cn be decomposed nto subset of ndvduls generted by the ntl Gussn model (.e., the ntl populton) nd - subsets of ndvduls generted from the prevous - models respectvely. Followng ths rule, t s esy to work out the new equtons wth the smlr menng s Eqs. 5-7: k= = P( k) d P( k) d (8) 3 µ + = P( k ) d (9) k = 0 Fg.. A multmodl problem wth two close optm Smlrly, the ntl populton s smpled from Gussn (µ 0, 0 ). In generl, wth the (µ, λ) scheme, the two ntervls t the (+) th generton represented by [, ] nd [ 3, 4 ] could be clculted by solvng the followng equton (.e., solved by lne serchng n prctce): 4 ( ) d + P( ) d = P P (5) wth 3 ( µ ) ( ) = e π subect to G ( ) = G( ) = G ( 3) = G ( 4) In stutons lke the threshold A n Fgure, the ddtonl condton wll reduce to G ( )=G ( ), ssumng G () corresponds to the globl optmum. Furthermore, n stutons lke Fgure, the two ntervls [, ] nd [ 3, 4 ] stsfyng the ddtonl condton my overlp wth ech other. Consequently, the two ntervls must be merged together. In both cses 3 nd 4 re set to some dentcl vlues, representng n empty ntervl. + = ( µ + ) P( k) d (0) k= 0 A test problem ws constructed ccordng to Eq. 4 wth the globl optmum creted by Gussn (-0.5, 0.05) s well s locl optmum creted by Gussn (0.5, 0.), whch ws scled up by fctor of 3.5 to mke t compettve optmum (.e., lrge bsn s well s 87.5% of the ftness vlue of the globl optmum). The ntl Gussn model ws set s µ 0 =0 nd 0 =0.5. Snce the ntl Gussn model ws centred n the mddle between the two optm, t s lkely tht, t lest t the begnnng of evoluton, qute lrge porton of selected ndvduls mght come from the pek correspondng to the locl optmum due to ts sgnfcntly lrger bsn sze. From ths pont of vew, ths problem s deceptve nd t would be nterestng to see how the EDA could hndle ths dffculty wth the two selecton schemes. The dynmcs of the EDA wth the (µ+λ) scheme s shown n Fgures 3&4 (.e., =0.3). Compred to the monotonous behvour observed n the unmodl cse, the behvour of the EDA s much more complcted on multmodl problems. For emple, n the frst fve genertons, the stndrd devton vlues were shrnkng nd, n the mentme, the men of the Gussn ws movng towrds the locl optmum. Ths ndctes tht the EDA ws beng msled by the lrge porton of selected ndvduls from the locl optmum. However, fter tht pont, the EDA moved bck nd consstently converged towrds the globl optmum, correctng ts prevous msudgement. 5

6 Fg. 3. The trectory of the stndrd devton prmeter of the Gussn model on the multmodl test problem wth the (µ+λ) scheme It should be ponted out tht the EDA wth the (µ+λ) scheme wll never get stuck t the locl optmum wth n nfntely lrge populton provded tht every pont n the serch spce hs the chnce to be smpled bsed on the sttstcl model n use, even wth smll probbltes. Ths s becuse the (µ+λ) scheme could lwys memorze good ndvduls tht hve been found nd those ndvduls from the bsn correspondng to the globl optmum (.e., t s gurnteed tht some of them would est n the ntl populton) could not only prevent the EDA from convergng to the locl optmum nfntely but lso grdully updte the model towrds the globl optmum. By contrst, there could be qute dfferent story wth the (µ, λ) scheme n whch ll old ndvduls re dscrded. As long s ndvduls from the locl optmum strt domntng the populton, the model wll be updted towrds the locl optmum nd the chnce of gettng new ndvduls from the globl optmum wll decrese, further reducng ts nfluence n the populton. In Fgure 4, the stndrd devton quckly dropped to nerly zero fter fve genertons whle, s shown n Fgure 5, the men got stuck t 0.5 where the locl optmum ws locted. Agn, n both cses, emprcl results (.e., populton sze = 0,000 nd 5 ndependent trls) hd good mtch gnst the theoretcl nlyss. Fg. 4. The trectory of the men prmeter of the Gussn model on the multmodl test problem wth the (µ+λ) scheme Fg. 5. The trectory of the stndrd devton prmeter of the Gussn model on the multmodl problem wth the (µ, λ) scheme Fg. 6. The trectory of the men prmeter of the Gussn model on the multmodl problem wth the (µ, λ) scheme V. CONCLUSIONS A mthemtcl modellng technque s proposed under the ssumpton of nfnte populton sze to nvestgte the dynmcs of contnuous EDA wth Truncton Selecton nd Gussn models. The mportnce of ths technque s tht t provdes prncpled method to nlyse the behvour of the EDA under the nfluence of dfferent lgorthm / problem fctors. In the most generl stuton, the frst step s to dentfy the re where the currently best ndvduls re dstrbuted. Typclly, ths re s specfed by the contours of the ftness functon. Wth the (µ, λ) scheme, ths re represented by should stsfy the followng condton where P(X, µ, ) s the densty functon of multvrte Gussn dstrbutons: P( X, µ, dx = () ) The men vlue of the new Gussn model n the th dmenson could then be clculted by: +, = X P( X, µ, ) dx µ () The stndrd devton of the new Gussn model n the th dmenson s gven by: +, = ( X µ +, ) P( X, µ, ) dx (3) Fnlly, the covrnce between the th nd the kth vrbles s gven by: +, k = ( X µ +, )( Xk µ +, k) P( X, µ, ) dx (4) 6

7 In the cse studes, ths technque s ppled to D unmodl nd multmodl problems nd some nterestng propertes of the EDA hve been found some of whch re unknown n prevous reserch. For D unmodl problems wth the (µ, λ) scheme: The dynmcs of the EDA s ndependent of the test functon ecept of ts generl shpe, whch s lso true for the (µ+λ) scheme. When strtng fr wy from the globl optmum, the EDA my get stuck. Gven certn selecton pressure, there s unque γ* tht could help the EDA cheve desrble performnce by mntnng the dversty. For D multmodl problems: The EDA could get stuck wth the (µ, λ) scheme. The EDA wll never get stuck wth the (µ+λ) scheme, whch s lso true for unmodl problems for obvous resons. The bove conclusons lso hold for hgh dmensonl problems. Furthermore, the proposed method could be esly dpted to model other spects of the evoluton such s the porton of ndvduls on ech optmum or the men ftness vlue of the populton. As to hgher dmensonl problems, some prelmnry work hs been successfully conducted on D problems wth nd wthout dependences mong prmeters. However, mor chllenge my come from specfyng the boundres of selected ndvduls nd clcultng multple ntegrls over severl vrbles. As result, t seems tht some further ssumptons my be requred to keep the complety of such theoretcl nlyss t resonble level. At lst, we hve lso notced the connecton between the theoretcl nlyss of EDAs nd the estng work on Evoluton Strteges [3], whch my help these two res beneft from ech other. ACKNOWLEDGMENT Ths work ws supported by n Austrln Postgrdute Awrd grnted to Bo Yun. REFERENCES [] T. Bck, D. B. Fogel, nd Z. Mchlewcz, "Hndbook of Evolutonry Computton," IOP Publshng Ltd nd Oford Unversty Press, 997. [] A. Berny, "Selecton nd Renforcement Lernng for Combntorl Optmzton," In Prllel Problem Solvng from Nture VI, 000. [3] H.-G. Beyer, The theory of evoluton strteges: Sprnger, 00. [4] C. González, J. A. Lozno, nd P. Lrrñg, "Anlyzng the PBIL lgorthm by mens of dscrete dynmcl systems," Comple Systems, vol. (4), pp , 000. [5] C. González, J. A. Lozno, nd P. Lrrñg, "Mthemtcl Modellng of UMDAc lgorthm wth tournment selecton. Behvour on lner nd qudrtc functons," Interntonl Journl of Appromte Resonng, vol. 3, pp , 000. [6] J. Grhl, S. Mnner, nd F. Rothluf, "Behvour of UMDAc wth Truncton Selecton on Monotonous Functons," In Congress on Evolutonry Computton 005, 005. [7] M. Hohfeld nd G. Rudolph, "Towrds Theory of Populton- Bsed Incrementl Lernng," In IEEE Interntonl Conference on Evolutonry Computton (ICEC 97), 997. [8] P. Lrrñg nd J. A. Lozno, "Estmton of Dstrbuton Algorthms: A New Tool for Evolutonry Computton," Kluwer Acdemc Publshers, 00. [9] H. Mühlenben, "The Equton for Response to Selecton nd Its Use for Predcton," Evolutonry Computton, vol. 5 (3), pp , 997. [0] H. Mühlenben nd T. Mhng, "Convergence Theory nd Applctons of the Fctorzed Dstrbuton Algorthm," Journl of Computng nd Informton Technology, vol. 7 (), pp. 9-3, 999. [] J. L. Shpro, "Sclng of Probblty-bsed Optmzton Algorthms," In Advnces n Neurl Informton Processng Systems 5 (NIPS00), 003. [] B. Yun nd M. Gllgher, "On the Importnce of Dversty Mntennce n Estmton of Dstrbuton Algorthms," In the 005 Genetc nd Evolutonry Computton Conference, 005. [3] Q. Zhng nd H. Mühlenben, "On the convergence of clss of estmton of dstrbuton lgorthms," IEEE Trnsctons on Evolutonry Computton, vol. 8 (), pp. 7-36,