TECNICHE DI DIAGNOSI AUTOMATICA DEI GUASTI. Silvio Simani References

Transcription

1 TECNICHE DI DIAGNOSI AUTOMATICA DEI GUASTI Re Neural per l Idenfcazone d Ssem non Lnear e Paern Recognon slvo.sman@unfe. References Texbook suggesed: Neural Neworks for Idenfcaon, Predcon, and Conrol, by Duc Truong Pham and Xng Lu. Sprnger Verlag; December ISBN: Nonlnear Idenfcaon and Conrol: A Neural Nework Approach, by G. P. Lu. Sprnger Verlag; Ocober ISBN: Fuzzy Modelng for Conrol, by Rober Babuska. Sprnger; 1s edon May 1, 1998 ISBN-10: , ISBN-13: /05/2011 2

2 Course Overvew 1. Inroducon. Course nroducon. Inroducon o neural nework. Issues n neural nework 2. Smple neural nework. Percepron. Adalne 3. Mullayer Percepron. Bascs 4. Radal bass neworks: overvew 5. Fuzzy Sysems: overvew 6. Applcaon examples 01/05/ Machne Learnng Improve auomacally wh experence Imang human learnng Human learnng Fas recognon and classfcaon of complex classes of objecs and conceps and fas adapaon Example: neural neworks Some echnques assume sascal source Selec a sascal model o model he source Oher echnques are based on reasonng or nducve nference e.g. Decson ree 01/05/2011 4

3 Machne Learnng Defnon A compuer program s sad o learn from experence E wh respec o some class of asks T and performance measure P, f s performance a asks n T, as measured by P, mproves wh experence. 01/05/ Examples of Learnng Problems Example 1: handwrng recognon: T: recognzng and classfyng handwren words whn mages. P: percenage of words correcly classfed. E: a daabase of handwren words wh gven classfcaon. Example 2: learn o play checkers: T: play checkers. P: percenage of games won n a ournamen. E: opporuny o play agans self war games. 01/05/2011 6

4 Type of Tranng Experence Drec or ndrec? Drec: board sae -> correc move Indrec: cred assgnmen problem degree of cred or blame for each move o he fnal oucome of wn or loss Teacher or no? Teacher selecs board saes and provde correc moves or Learner can selec board saes Is ranng experence represenave of performance goal? Tranng playng agans self Performance evaluaed playng agans world champon 01/05/ Issues n Machne Learnng Wha algorhms can approxmae funcons well and when? How does he number of ranng examples nfluence accuracy? How does he complexy of hypohess represenaon mpac? How does nosy daa nfluence accuracy? How do you reduce a learnng problem o a se of funcon approxmaon? 01/05/2011 8

5 Summary Machne learnng s useful for daa mnng, poorly undersood doman face recognon and programs ha mus dynamcally adap. Draws from many dverse dscplnes. Learnng problem needs well-specfed ask, performance merc and ranng experence. Involve searchng space of possble hypoheses. Dfferen learnng mehods search dfferen hypohess space, such as numercal funcons, neural neworks, decson rees, symbolc rules. 01/05/ Inroducon o Neural Neworks 01/05/

6 Bran neurons processors On average connecons 01/05/ Arfcal Neuron bas j ne = j w j y j + b 01/05/

7 Arfcal Neuron Inpu/Oupu Sgnal may be. Real value. Unpolar {0, 1}. Bpolar {-1, +1}. Wegh : w j srengh of connecon. Noe ha w j refers o he wegh from un j o un no he oher way round. 01/05/ Arfcal Neuron The bas b s a consan ha can be wren as w 0 y 0 wh y 0 = b and w 0 = 1 such ha n ne = w y j = 0 The funcon f s he un s acvaon funcon. In he smples case, f s he deny funcon, and he un s oupu s jus s ne npu. Ths s called a lnear un. Oher acvaon funcons are : sep funcon, sgmod funcon and Gaussan funcon. 01/05/ j j

8 Acvaon Funcons Ideny funcon Bnary Sep funcon Bpolar Sep funcon y 2 x μ 1 2 2σ x = e 2πσ Sgmod funcon Bpolar Sgmod funcon Gaussan funcon 01/05/ Arfcal Neural Neworks ANN Inpu vecor wegh Sgnal roung Acvaon funcon wegh Acvaon funcon Oupu vecor 01/05/

9 When Should ANN Soluon Be Consdered? The soluon o he problem canno be explcly descrbed by an algorhm, a se of equaons, or a se of rules. There s some evdence ha an npu-oupu mappng exss beween a se of npu and oupu varables. There should be a large amoun of daa avalable o ran he nework. 01/05/ Problems Tha Can Lead o Poor Performance? The nework has o dsngush beween very smlar cases wh a very hgh degree of accuracy. The ran daa does no represen he ranges of cases ha he nework wll encouner n pracce. The nework has a several hundred npus. The man dscrmnang facors are no presen n he avalable daa. E.g. Tryng o assess he loan applcaon whou havng knowledge of he applcan's salares. The nework s requred o mplemen a very complex funcon. 01/05/

10 Applcaons of Arfcal Neural Neworks Manufacurng : faul dagnoss, fraud deecon. Realng : fraud deecon, forecasng, daa mnng. Fnance : fraud deecon, forecasng, daa mnng. Engneerng : faul dagnoss, sgnal/mage processng. Producon : faul dagnoss, forecasng. Sales & markeng : forecasng, daa mnng. 01/05/ Daa Pre-processng Neural neworks very rarely operae on he raw daa. An nal pre-processng sage s essenal. Some examples are as follows: Feaure exracon of mages: for example, he analyss of x-rays requres pre-processng o exrac feaures whch may be of neres whn a specfed regon. Represenng npu varables wh numbers. For example "+1" s he person s marred, "0" f dvorced, and "-1" f sngle. Anoher example s represenng he pxels of an mage: 255 = brgh whe, 0 = black. To ensure he generalzaon capably of a neural nework, he daa should be encoded n form whch allows for nerpolaon. 01/05/

11 Daa Pre-processng Caegorcal Varable A caegorcal varable s a varable ha can belong o one of a number of dscree caegores. For example, red, green, blue. Caegorcal varables are usually encoded usng 1 ou-of n codng. e.g. for hree colors, red = 1 0 0, green =0 1 0 Blue = If we used red = 1, green = 2, blue = 3, hen hs ype of encodng mposes an orderng on he values of he varables whch does no exs. 01/05/ Daa Pre-processng CONTINUOUS VARIABLES A connuous varable can be drecly appled o a neural nework. However, f he dynamc range of npu varables are no approxmaely he same, s beer o normalze all npu varables of he neural nework. 01/05/

12 Smple Neural Neworks Smple Percepron 01/05/ Oulnes The Percepron Lnearly separable problem Nework srucure Percepron learnng rule Convergence of Percepron 01/05/

13 THE PERCEPTRON The percepron was a smple model of ANN nroduced by Rosenbla of MIT n he 1960 wh he dea of learnng. Percepron s desgned o accomplsh a smple paern recognon ask: afer learnng wh real value ranng daa { x, d, =1,2,, p} where d = 1 or -1 For a new sgnal paern x+1, he percepron s capable of ellng you o whch class he new sgnal belongs x+1 percepron = 1 or 1 01/05/ Percepron Lnear Threshold Un LTU x 1 w 1 x 0 =1 w 0 =b ox= { 1 f Σ =0 n w x >0-1 oherwse x 2. w 2 w n Σ x= Σ =0n w x o x n 01/05/

14 01/05/ = = = + = m m x w f b x w f y 0 1 where f s he hard lmer funcon.e. + > + = = = m m b x w f b x w f y We can always rea he bas b as anoher wegh wh npus equal 1 Mahemacally he Percepron s 01/05/ = + = m b x w Why s he nework capable of solvng lnearly separable problem? > + = m b wx 0 1 < + = m b w x

15 Learnng rule An algorhm o updae he weghs w so ha fnally he npu paerns le on boh sdes of he lne decded by he percepron Le be he me, a = 0, we have + w 0 x = 0 01/05/ d w + = f 1 f Percepron learnng rule In Mah Where η s he learnng rae >0, +1 f x>0 sgnx = 1 f x<=0, x n x n class class = w + η [ d sgn w x ] x NB : d s he same as d and x as x 01/05/ hard lmer funcon

16 In words: If he classfcaon s rgh, do no updae he weghs If he classfcaon s no correc, updae he wegh owards he oppose drecon so ha he oupu move close o he rgh drecons. 01/05/ Percepron convergence heorem Rosenbla, 1962 Le he subses of ranng vecors be lnearly separable. Then afer fne seps of learnng we have lm w = w whch correcly separae he samples. The dea of proof s ha o consder w+1-w - w-w whch s a decrease funcon of 01/05/

17 Summary of Percepron learnng Varables and parameers x = m+1 dm. npu vecors a me = b, x 1, x 2,..., x m w = m+1 dm. wegh vecors = 1, w 1,..., w m b = bas y = acual response η = learnng rae parameer, a +ve consan < 1 d = desred response 01/05/ Summary of Percepron learnng Daa { x, d, =1,,p} Presen he daa o he nework once a pon could be cyclc : x1, d1, x2, d2,, xp, dp, xp+1, dp+1, or randomly Hence we mx me wh here 01/05/

18 Summary of Percepron learnng algorhm 1. Inalsaon Se w0=0. Then perform he followng compuaon for me sep =1,2, Acvaon A me sep, acvae he percepron by applyng npu vecor x and desred response d 3. Compuaon of acual response Compue he acual response of he percepron y = sgn w x where sgn s he sgn funcon 4. Adapaon of wegh vecor Updae he wegh vecor of he percepron w+1 = w+ η [ d - y ] x 5. Connuaon 01/05/ Quesons reman Where or when o sop? By mnmzng he generalzaon error For ranng daa {x, d, =1, p} How o defne ranng error afer seps of learnng? E= p =1 [d-sgnw. x]2 01/05/

19 We nex urn o ADALINE learnng, from whch we can undersand he learnng rule, and more general he Back-Propagaon BP learnng 01/05/ Smple Neural Nework ADALINE Learnng 01/05/

20 Oulnes ADALINE Graden descendng learnng Modes of ranng 01/05/ Unhappy Over Percepron Tranng When a percepron gves he rgh answer, no learnng akes place Anyhng below he hreshold s nerpreed as no, even s jus below he hreshold. I mgh be beer o ran he neuron based on how far below he hreshold s. 01/05/

21 ADALINE ADALINE s an acronym for ADApve LINear Elemen or ADApve LInear NEuron developed by Bernard Wdrow and Marcan Hoff There are several varaons of Adalne. One has hreshold same as percepron and anoher jus a bare lnear funcon. The Adalne learnng rule s also known as he leasmean-squares LMS rule, he dela rule, or he Wdrow- Hoff rule. I s a ranng rule ha mnmzes he oupu error usng approxmae graden descen mehod. 01/05/ Replace he sep funcon n he percepron wh a connuous dfferenable funcon f, e.g he smples s lnear funcon Wh or whou he hreshold, he Adalne s raned based on he oupu of he funcon f raher han he fnal oupu. +/Σ f x Adalne 01/05/

22 Afer each ranng paern x s presened, he correcon o apply o he weghs s proporonal o he error. E, = ½ [ d fw x ] 2 =1,...,p N.B. If f s a lnear funcon fw x = w x Summng ogeher, our purpose s o fnd w whch mnmzes E = E, 01/05/ General Approach graden descen mehod To fnd g w+1 = w+g Ew so ha w auomacally ends o he global mnmum of Ew. w+1 = w- E wη see fgure below 01/05/

23 Graden drecon s he drecon of uphll for example, n he Fgure, a poson 0.4, he graden s uphll F s E, consder one dm case Fw F Graden drecon F /05/ w In graden descen algorhm, we have w+1 = w F w ητ herefore he ball goes downhll snce F w s downhll drecon Fw Graden drecon w 01/05/ w

24 In graden descen algorhm, we have w+1 = w F w ητ herefore he ball goes downhll snce F w s downhll drecon Fw Graden drecon w+1 w 01/05/ Gradually he ball wll sop a a local mnma where he graden s zero Fw Graden drecon w+k w 01/05/

25 In words Graden mehod could be hough of as a ball rollng down from a hll: he ball wll roll down and fnally sop a he valley Thus, he weghs are adjused by w j +1 = w j +η Σ [d - fw x ] x j f Ths corresponds o graden descen on he quadrac error surface E When f =1, we have he percepron learnng rule we have n general f >0 n neural neworks. The ball moves n he rgh drecon. 01/05/ Two ypes of nework ranng: Sequenal mode on-lne, sochasc, or per-paern : Weghs updaed afer each paern s presened Percepron s n hs class Bach mode off-lne or per-epoch : Weghs updaed afer all paerns are presened 01/05/

26 Comparson Percepron and Graden Descen Rules Percepron learnng rule guaraneed o succeed f Tranng examples are lnearly separable Suffcenly small learnng rae η Lnear un ranng rule uses graden descen guaraneed o converge o hypohess wh mnmum squared error gven suffcenly small learnng rae η Even when ranng daa conans nose Even when ranng daa no separable by hyperplanes 01/05/ Summary Percepron W+1= W+η [ d - sgn w. x] x Adalne Graden descen mehod W+1= W+η [ d - fw. x] x f 01/05/

27 Mul-Layer Percepron MLP Idea: Cred assgnmen problem Problem of assgnng cred or blame o ndvdual elemens nvolvng n formng overall response of a learnng sysem hdden uns In neural neworks, problem relaes o dvdng whch weghs should be alered, by how much and n whch drecon. 01/05/ x 1 Example: Three-layer neworks x 2 Inpu Oupu x n Sgnal roung Inpu layer Hdden layer Oupu layer 01/05/

28 Properes of archecure No connecons whn a layer No drec connecons beween npu and oupu layers Fully conneced beween layers Ofen more han 2 layers Number of oupu uns need no equal number of npu uns Number of hdden uns per layer can be more or less han npu or oupu uns Each un s a percepron m y = f w x + b j j j = 1 01/05/ BP Back Propagaon + 01/05/

29 MulLayer Percepron I Back Propagang Learnng 01/05/ BP learnng algorhm Soluon o cred assgnmen problem n MLP Rumelhar, Hnon and Wllams 1986 BP has wo phases: Forward pass phase: compues funconal sgnal, feedforward propagaon of npu paern sgnals hrough nework Backward pass phase: compues error sgnal, propagaon of error dfference beween acual and desred oupu values backwards hrough nework sarng a oupu uns 01/05/

30 BP Learnng for Smples MLP O Task : Daa {I, d} o mnmze E = d - o 2 /2 = [d - fwy ] 2 /2 = [d - fwfwi ] 2 /2 W y w Error funcon a he oupu un Wegh a me s w and W, nend o fnd he wegh w and W a me +1 Where y = fwi, oupu of he npu un I 2 layers example 01/05/ Forward pass phase Suppose ha we have w, W of me O For gven npu I, we can calculae and y = fwi o = f W y = f W f w I W y w Error funcon of oupu un wll be E = d - o 2 /2 I 2 layers example 01/05/

31 Backward Pass Phase O de W + 1 = W η dw de df = W η df dw = W + η d o f ' W y y W y w I E = d - o 2 /2 o = f W y 01/05/ W + 1 Backward pass phase de = W η df = W + η d o f ' W y y = W + ηδy de = W η dw df dw W y I w O where Δ = d-o f 01/05/

32 01/05/ I I w f W w dw dy W y W f o d w dw dy dy de w dw de w w ' ' 1 Δ + = + = = = + η η η η I w W y O Backward pass phase o= f W y = f W f w I 01/05/ Summary wegh updaes are local oupu un npu un 1 1 y W W I w w j k kj kj j j j Δ = + = + η ηδ Δ = = + k kj k j j j j I W ne f I w w ' 1 η ηδ ' 1 y Ne f O d y W W j k k k j k kj kj = Δ = + η η Once wegh changes are compued for all uns, weghs are updaed a same me bas ncluded as weghs here We now compue he dervave of he acvaon funcon f. npu un oupu un

33 Acvaon Funcons o compue j and kwe need o fnd he dervave of acvaon funcon f o fnd dervave he acvaon funcon mus be smooh Sgmodal logsc funcon-common n MLP f δ ne Δ 1 = 1+ exp kne where k s a posve consan. The sgmodal funcon gves value n range of 0 o 1 Inpu-oupu funcon of a neuron rae codng assumpon 01/05/ Shape of sgmodal funcon Noe: when ne = 0, f = /05/

34 Shape of sgmodal funcon dervave Dervave of sgmodal funcon has max a x= 0, s symmerc abou hs pon fallng o zero as sgmodal approaches exreme values 01/05/ Reurnng o local error gradens n BP algorhm we have for oupu uns Δ = d = d O f ' Ne O ko 1 O For npu uns we have δ = ky = f ' ne 1 y k k Δ Δ k k W W k k Snce degree of wegh change s proporonal o dervave of acvaon funcon, wegh changes wll be greaes when uns receves md-range funconal sgnal han a exremes 01/05/

35 η Summary of BP learnng algorhm Se learnng rae η Se nal wegh values ncl.. bases: w, W Loop unl soppng crera sasfed: presen npu paern o NN npus compue funconal sgnal for npu uns compue funconal sgnal for oupu uns presen Targe response o oupu uns compue error sgnal for oupu uns compue error sgnal for npu uns updae all weghs a same me ncremen n o n+1 and selec nex I and d end loop 01/05/ Nework ranng: Tranng se shown repeaedly unl soppng crera are me Each full presenaon of all paerns = epoch Randomse order of ranng paerns presened for each epoch n order o avod correlaon beween consecuve ranng pars beng learn order effecs Two ypes of nework ranng: Sequenal mode on-lne, sochasc, or per-paern Weghs updaed afer each paern s presened Bach mode off-lne or per -epoch 01/05/

36 Advanages and dsadvanages of dfferen modes Sequenal mode: Less sorage for each weghed connecon Random order of presenaon and updang per paern means search of wegh space s sochasc-reducng rsk of local mnma able o ake advanage of any redundancy n ranng se.e. same paern occurs more han once n ranng se, esp. for large ranng ses Smpler o mplemen Bach mode: Faser learnng han sequenal mode 01/05/ MulLayer Percepron II Dynamcs of MulLayer Percepron

37 Summary of Nework Tranng Forward phase: I, w, ne, y, W, Ne, O Backward phase: Oupu un W kj + 1 W = η d O k kj k = η Δ k f ' Ne k y j y j Inpu un w j = η f + 1 w j = ηδ j I ' ne j Δ k W kj I k 01/05/ Nework ranng: Tranng se shown repeaedly unl soppng crera are me. Possble convergence crera are Eucldean norm of he graden vecor reaches a suffcenly small denoed as θ. When he absolue rae of change n he average squared error per epoch s suffcenly small denoed as θ. Valdaon for generalzaon performance : sop when generalzaon reachng he peak llusrae n hs lecure 01/05/

38 Goals of Neural Nework Tranng To gve he correc oupu for npu ranng vecor Learnng To gve good responses o new unseen npu paerns Generalzaon 01/05/ Tranng and Tesng Problems Suck neurons: Degree of wegh change s proporonal o dervave of acvaon funcon, wegh changes wll be greaes when uns receves md-range funconal sgnal han a exremes neuron. To avod suck neurons weghs nalzaon should gve oupus of all neurons approxmae 0.5 Insuffcen number of ranng paerns: In hs case, he ranng paerns wll be learn nsead of he underlyng relaonshp beween npus and oupu,.e. nework jus memorzng he paerns. Too few hdden neurons: nework wll no produce a good model of he problem. Over-fng: he ranng paerns wll be learn nsead of he underlyng funcon beween npus and oupu because of oo many of hdden neurons. Ths means ha he nework wll have a poor generalzaon capably. 01/05/

39 Dynamcs of BP learnng Am s o mnmse an error funcon over all ranng paerns by adapng weghs n MLP Recallng he ypcal error funcon s he mean squared error as follows E= 1 2 p k = 1 d k O k 2 The dea s o reduce E o global mnmum pon. 01/05/ Dynamcs of BP learnng In sngle layer percepron wh lnear acvaon funcons, he error funcon s smple, descrbed by a smooh parabolc surface wh a sngle mnmum 01/05/

40 Dynamcs of BP learnng MLP wh non-lnear acvaon funcons have complex error surfaces e.g. plaeaus, long valleys ec. wh no sngle mnmum For complex error surfaces he problem s learnng rae mus keep small o preven dvergence. Addng momenum erm s a smple approach dealng wh hs problem. 01/05/ Momenum Reducng problems of nsably whle ncreasng he rae of convergence Addng erm o wegh updae equaon can effecvely holds as exponenally wegh hsory of prevous weghs changed Modfed wegh updae equaon s w n + 1 w n = ηδ n y n j j j + α [ w n w n 1] j j + 01/05/

41 Effec of momenum erm If wegh changes end o have same sgn, momenum erm ncreases and graden decrease speed up convergence on shallow graden If wegh changes end have opposng sgns, momenum erm decreases and graden descen slows o reduce oscllaons sablzes Can help escape beng rapped n local mnma 01/05/ Selecng Inal Wegh Values Choce of nal wegh values s mporan as hs decdes sarng poson n wegh space. Tha s, how far away from global mnmum Am s o selec wegh values whch produce mdrange funcon sgnals Selec wegh values randomly from unform probably dsrbuon Normalse wegh values so number of weghed connecons per un produces mdrange funcon sgnal 01/05/

42 Convergence of Backprop Avod local mnumum wh fas convergence: Add momenum Sochasc graden descen Tran mulple nes wh dfferen nal weghs Naure of convergence Inalze weghs near zero or nal neworks near-lnear Increasngly non-lnear funcons possble as ranng progresses 01/05/ Use of Avalable Daa Se for Tranng The avalable daa se s normally spl no hree ses as follows: Tranng se use o updae he weghs. Paerns n hs se are repeaedly n random order. The wegh updae equaon are appled afer a ceran number of paerns. Valdaon se use o decde when o sop ranng only by monorng he error. Tes se Use o es he performance of he neural nework. I should no be used as par of he neural nework developmen cycle. 01/05/

43 Earler Soppng - Good Generalzaon Runnng oo many epochs may overran he nework and resul n overfng and perform poorly n generalzaon. Keep a hold-ou valdaon se and es accuracy afer every epoch. Manan weghs for bes performng nework on he valdaon se and sop ranng when error ncreases ncreases beyond hs. error No. of epochs Valdaon se Tranng se 01/05/ Model Selecon by Cross-valdaon Too few hdden uns preven he nework from learnng adequaely fng he daa and learnng he concep more han wo layer neworks. Too many hdden uns leads o overfng. Smlar cross-valdaon mehods can be used o deermne an approprae number of hdden uns by usng he opmal es error o selec he model wh opmal number of hdden layers and nodes. Valdaon se error Tranng se No. of epochs 01/05/

44 Radal Bass Funcons Radal Bass Funcons Overvew Radal-bass funcon RBF neworks RBF = radal-bass funcon: a funcon whch depends only on he radal dsance from a pon XOR problem quadracally separable 01/05/

45 Radal-bass funcon RBF neworks So RBFs are funcons akng he form φ x x where φ s a non-lnear acvaon funcon, x s he npu and x s he h poson, prooype, bass or cenre vecor. The dea s ha pons near he cenres wll have smlar oupus.e. f x ~ x hen f x ~ f x snce hey should have smlar properes. The smples s he lnear RBF : φx = x x 01/05/ Typcal RBFs nclude a Mulquadrcs φ 2 r = r + c 2 1 / 2 for some c>0 b Inverse mulquadrcs φ r 2 = r + c 2 1 / 2 for some c>0 c Gaussan 2 r φ r = exp 2 2σ for some σ >0 01/05/

46 nonlocalzed funcons localzed funcons 01/05/ Idea s o use a weghed sum of he oupus from he bass funcons o represen he daa. Thus ceners can be hough of as prooypes of npu daa. * * * * * * O 1 MLP vs RBF dsrbued local

47 Sarng pon: exac nerpolaon Each npu paern x mus be mapped ono a arge value d 01/05/ Tha s, gven a se of N vecors x and a correspondng se of N real numbers, d he arges, fnd a funcon F ha sasfes he nerpolaon condon: F x = d for =1,...,N or more exacly fnd: sasfyng: N F x = wφ x x j j= 1 N j F x = w φ x x d j j = j= 1 01/05/

48 Inpu y 1 y 2 Sngle-layer neworks φ 1 y=φ 1 y-x 1 w j Σ d Oupu y p Inpu layer : φ Ν y=φ N y-x N oupu = Σ w φ y -x adjusable parameers are weghs w j number of npu uns number of daa pons Form of he bass funcons decded n advance 01/05/ To summarze: For a gven daa se conanng N pons x,d, =1,,N Choose a RBF funcon φ Calculae φx j x Solve he lnear equaon Φ W = D Ge he unque soluon Done Lke MLP s, RBFNs can be shown o be able o approxmae any funcon o arbrary accuracy usng an arbrarly large numbers of bass funcons. Unlke MLP s, however, hey have he propery of bes approxmaon.e. here exss an RBFN wh mnmum approxmaon error. 01/05/

49 Large σ = 1 01/05/ Small σ = /05/

50 Problems wh exac nerpolaon can produce poor generalsaon performance as only daa pons consran mappng Bshop1995 example Overfng problem Underlyng funcon fx= sne2π x sampled randomly for 30 pons added Gaussan nose o each daa pon 30 daa pons 30 hdden RBF uns fs all daa pons bu creaes oscllaons due added nose and unconsraned beween daa pons 01/05/ All Daa Pons 5 Bass funcons 01/05/

51 To f an RBF o every daa pon s very neffcen due o he compuaonal cos of marx nverson and s very bad for generalzaon so: Use less RBF s han daa pons I.e. M<N Therefore don necessarly have RBFs cenred a daa pons Can nclude bas erms Can have Gaussan wh general covarance marces bu here s a rade-off beween complexy and he number of parameers o be found eg for d rbfs we have: 01/05/ Fuzzy Modellng and Idenfcaon Fuzzy Cluserng wh Applcaon o Daa-Drven Modellng

52 Inroducon The ably o cluser daa conceps, percepons, ec. essenal feaure of human nellgence. A cluser s a se of objecs ha are more smlar o each oher han o objecs from oher clusers. Applcaons of cluserng echnques n paern recognon and mage processng. Some machne-learnng echnques are based on he noon of smlary decson rees, case-based reasonng Non-lnear regresson and black-box modellng can be based on he paronng daa no clusers. 01/05/ Secon Oulne Basc conceps n cluserng daa se paron marx dsance measures Cluserng algorhms fuzzy c-means Gusafson Kessel Applcaon examples sysem denfcaon and modellng dagnoss 01/05/

53 Examples of Clusers 01/05/ Problem Formulaon Gven s a se of daa n R n and he esmaed number of clusers o look for a dffcul problem, more on hs laer. Fnd he paronng of he daa no subses clusers, such ha samples whn a subse are more smlar o each oher han o samples from oher subses. Smlary s mahemacally formulaed by usng a dsance measure.e., a dssmlary funcon. Usually, each cluser wll have a prooype and he dsance s measured from hs prooype. 01/05/

54 Dsance Measure 01/05/ Dsance Measures Eucldean norm: d 2 z j, v = z j v T z j v Inner-produc norm: d 2 A z j, v = z j v T A z j v Many oher possbles... 01/05/

55 Generalzed Prooypes Varees 01/05/ Correspondng Dsance Measures 01/05/

56 Mahemacal Formulaon of Cluserng Gven he daa: z T n [ z z,, z ] R, k = 1, N k = 1k, 2k nk, Fnd: μ he paron 11 U = marx: μc1 and he cluser prooype cenres: 01/05/ μ μ 1k ck μ μ 1N cn n { v, v2,, v } R V =, v 1 c Fuzzy Cluserng: an Opmsaon Approach Objecve funcon leas-squares creron: subjec o consrans: 01/05/

57 Fuzzy c-means Algorhm Repea: 1. Compue cluser prooypes means: 2. Calculae dsances: 3. Updae paron marx: unl = 1,, c. k = 1,, N 01/05/ Falure o Dscover Non- Sphercal Clusers 01/05/

58 Adapve Dsance Measure Inner-produc norm: norm-nducng marx covarance marx 01/05/ Inner-Produc Norm 01/05/

59 Gusafson Kessel Algorhm Repea: 1. Compue cluser prooypes means: 2. Compue covarance marces: 3. Compue dsances: 4. Compue paron marx: unl 01/05/ Clusers of Dfferen Shape and Orenaon 01/05/

60 Number of Clusers Valdy measures Fuzzy hypervolume: Average whn-cluser dsance: Xe Ben ndex... 01/05/ Valdy Measures: Example Daa over 4 clusers 01/05/

61 Valdy Measures 01/05/ Number of Clusers 01/05/

62 Daa-Drven Black-Box Modellng Lnear model for lnear sysems only, lmed n use Neural nework black box, unrelable exrapolaon Rule-based model more ransparen, grey-box 01/05/ Exracon of Rules by Fuzzy Cluserng 01/05/

63 Exracon of Rules by Fuzzy Cluserng 01/05/ Example: Non-lnear Auoregressve Sysem NARX 01/05/

64 Srucure Selecon and Daa Preparaon 1. Choose model order p 2. Form paern marx Z o be clusered 01/05/ Cluserng Resuls 01/05/

65 Rules Obaned 01/05/ Idenfcaon of Pressure Dynamcs 01/05/

66 01/05/ Concludng Remarks Opmsaon approach o cluserng effecve for merc e.g., real-valued daa accurae resuls for small o medum complexy problems for large problems, convergence o local opma, slow Many oher echnques agglomerave mehods herarchcal splng mehods graph-heorec mehods Varey of applcaons 01/05/

67 Applcaon Examples Neural Neworks for Non-lnear Idenfcaon Nonlnear Sysem Idenfcaon Targe funcon: y p k+1 = f. Idenfed funcon: y NET k+1 = F. Esmaon error: ek+1 01/05/

68 Nonlnear Sysem Idenfcaon Neural nework npu generaon Pm 01/05/ Nonlnear Sysem Idenfcaon Neural nework arge Tm Neural nework response angle & velocy 01/05/

69 Malab NNool GUI Graphcal User Inerface 01/05/