Artificial Neural Networks

Transcription

1 Atfcal Neual Netwoks Ronald J. Wllams CSG22, Spng 27 Bans ~ neuons of > 2 types, ~ 4 synapses, -ms cycle tme Sgnals ae nosy spke tans of electcal potental Synaptc stength beleved to ncease o decease wth use ( leanng?) Atfcal Neual Netwoks: Slde 2

2 A Neuon Atfcal Neual Netwoks: Slde 3 x Standad ANN Neuon o Unt Bas Input x = Bas Weght w Fo leanng o fo handdesgnng, weghts w j ae adjustable paametes Extenal Input x 2 w 2 x n... w w n s Squashng Functon g Output y n s = wjx j= y = g( s) j Atfcal Neual Netwoks: Slde 4 2

3 Use had-lmtng squashng functon g( s) = Lnea Theshold Unt Smple Pecepton Unt Theshold Logc Unt f f s> s Boolean ntepetaton: false, tue g(s) s Atfcal Neual Netwoks: Slde 5 Note that n w x > w x > w x = w j j j= j= n j j Thus an equvalent fomulaton s to take the appopate weghted sum nvolvng only the tue (extenal) nputs and compae t aganst the theshold w The use of a bas nput of and a coespondng bas weght s a mathematcal devce to allow us to teat the theshold as just anothe weght Atfcal Neual Netwoks: Slde 6 3

4 Implementng Boolean Functons -.5 x x OR x 2 x x - NOT x x x AND x 2 x 2 Atfcal Neual Netwoks: Slde 7 Implementng Boolean Functons (cont.) x k-.5 x n... At-least-k-out-of-n gate Genealzes AND, OR Challenge: Wte a Boolean expesson fo ths Anothe challenge: Constuct a decson tee fo ths Atfcal Neual Netwoks: Slde 8 4

5 Geometc Intepetaton Defne and x = ( x, x2, K, x w = w, w K, w n ( 2, n ) ) I.e., hee the bas nput and bas weght ae not ncluded Then the output of the unt s detemned by the sgn of n j= wj x j = w x + w so the sepaato between the y= and y= egons of the nput space conssts of all ponts x fo whch w x + w = Atfcal Neual Netwoks: Slde 9 Geometc Intepetaton (cont.) Ths sepaato s a hypeplane n n-dmensonal space wth nomal vecto w and whose dstance to the ogn s / w x 2 w w Thus the functons ealzable by a smple pecepton unt ae called lnealy sepaable sepaato x Atfcal Neual Netwoks: Slde 5

6 Boolean examples x 2 x 2 x OR x 2 w = w 2 = w = -.5 x x + x 2 =.5 x AND x 2 w = w 2 = w = -.5 x x + x 2 =.5 Atfcal Neual Netwoks: Slde Boolean examples (cont.) x 2 x -x 2 =.5 x x AND NOT x 2 w = w 2 = - w = -.5 Atfcal Neual Netwoks: Slde 2 6

7 x 2 x XOR x 2 x But... Not lnealy sepaable XOR and ts negaton ae the only Boolean functons of two aguments that ae not lnealy sepaable Howeve, fo lage and lage n, the numbe of lnealy sepaable Boolean functons gows much moe slowly than the numbe of possble Boolean functons Atfcal Neual Netwoks: Slde 3 Implementng XOR wth smple pecepton unts Input Output x x AND NOT x 2 x 2 x 2 AND NOT x OR gate Suffces to use one ntemedate stage of smple pecepton unts Appoach genealzes to any Boolean functon: wte t n DNF, use one ntemedate unt fo each dsjunct, then use an OR gate fo output Poves that any Boolean functon s ealzable by a netwok of smple pecepton unts Atfcal Neual Netwoks: Slde 4 7

8 What about leanng? Stat wth tanng data {(x, d )}, whee each nput/desed output pa s ndexed by =,..., R and x = (, x, x2, K, xn ) epesents the nput (ths tme augmented by the bas nput x =) Each d s of couse ethe o The objectve s to fnd a weght vecto w = ( w, w, w2, K, wn ) such that y = g( w x ) agees wth d fo each, whee g s the had-lmtng theshold functon Atfcal Neual Netwoks: Slde 5 Pecepton algothm w (any ntal values ok) epeat fo = to R w w + η( d y ) untl no eos x η > s the leanng ate It can be taken to be when nputs ae and In that case, body of nne loop s: f actual output too small, add nput vecto to weght vecto f actual output too lage, subtact nput vecto fom weght vecto else don t change weghts Atfcal Neual Netwoks: Slde 6 8

9 Pecepton algothm (cont.) Easy to check that ths moves weghts geedly n coect decton fo the cuent tanng example Convegence theoem: Fo any lnealy sepaable tanng data, the algothm conveges to a soluton (as long as the leanng ate s sutably small). But f the data s not lnealy sepaable, the weghts loop ndefntely. Atfcal Neual Netwoks: Slde 7 Multlaye Netwoks Ths algothm has been known snce ~96 (Rosenblatt) But the most nteestng functons we mght want to lean ae not necessaly lnealy sepaable Dlemma faced by ANN eseaches between ~96 and ~985: fo geate expessveness, need multlaye netwoks of these lnea theshold unts only known easonable algothm was fo snglelaye netwoks (.e., one laye of weghts) Atfcal Neual Netwoks: Slde 8 9

10 Multlaye Netwoks (cont.) Input Hdden Output How should we tan these weghts? We know how to tan these weghts, assumng the othes ae fxed Atfcal Neual Netwoks: Slde 9 Leanng n multlaye nets basc dea One geneal way to appoach any leanng poblem: expess the leanng objectve n tems of a functon to optmze seach the hypothess space fo a hypothess gvng the optmal value Appled to a supevsed leanng task: fo each possble hypothess, defne a measue of ts oveall eo on the tanng data smplest way: defne ths eo measue fo each tanng example and then defne the oveall eo measue as the sum of these Atfcal Neual Netwoks: Slde 2

11 Expanded notaton: necessay snce usng multple unts x = th unt w x w x 2 w 2 x n... w n s g s y n y = w j= = g( s ) j x j Atfcal Neual Netwoks: Slde 2 Leanng n multlaye nets Defne the eo on the th tanng example to be E ( d y 2 OutputUnts = ) 2 d whee and ae the desed and actual outputs, espectvely, of the th unt fo tanng example. Ths s a functon of the netwok weghts snce s. Then defne the oveall eo to be E y = E y Atfcal Neual Netwoks: Slde 22

12 Gadent Descent Eo Weght space s N- dmensonal, whee N s the total numbe of weghts n the netwok weght space th E Gadent W E s a vecto whose α component s, whee w s a weght n the netwok. w α α E Gadent descent: ncement each w α by Δw α = η w α Atfcal Neual Netwoks: Slde 23 Oh, oh..., a poblem Fo a netwok of lnea theshold unts, the gadent s zeo eveywhee t exsts (whch s almost eveywhee) The eo functon has a teace shape flat eveywhee wth occasonal clffs So gadent descent useless n ths case Now ntoduce a tck... Atfcal Neual Netwoks: Slde 24 2

13 Sgmod squashng functon Instead of the had-lmtng theshold functon of the smple pecepton unt, use a smooth appoxmaton to t g(s ) s Commonly used: g s = + e ( ) s Logstc functon Atfcal Neual Netwoks: Slde 25 Soft lnea sepaaton Atfcal Neual Netwoks: Slde 26 3

14 Fo any netwok of such sgmod unts, the netwok output s a smooth functon of ts nput. Thus so s the eo functon. But how do we compute the necessay gadent? It would be panful to wte down an explct expesson fo the netwok output (o the eo) as a functon of the netwok nput and the weghts. Then magne tyng to dffeentate t. To the escue: the chan ule Atfcal Neual Netwoks: Slde 27 The eo backpopagaton algothm x = th unt w x w x 2 w 2... w n s δ g x n Can ntepet δ and ε as senstvtes Fo tanng example, defne and E ε = y E δ = s y ε Fo smplcty, we hencefoth suppess the supescpt except on E Atfcal Neual Netwoks: Slde 28 4

15 Snce t follows that fo any weght. Devaton of backpop E E w = j w j Now we focus on how to compute. E E = w j E w j Atfcal Neual Netwoks: Slde 29 Snce we see that Futhemoe, Devaton of backpop (cont.) s = w E w δ j j j x j E = s E s w E so all that emans s to compute fo any unt. j = δ x dy = = = ε s y ds ε j g ( s ) Atfcal Neual Netwoks: Slde 3 5

16 Devaton of backpop (cont.) Fo each output unt, E ε = = ( d y y 2 k k OutputUnts y k 2 ) = d y What about hdden unts? Fo each hdden unt, let Downsteam() = all unts to whch that unt dectly sends ts output. Note that fom the pont of vew of each unt k n Downsteam(), the output y of unt s the nput x of unt k (.e, the sgnal on the nput wth weght w k ). Atfcal Neual Netwoks: Slde 3 Devaton of backpop (cont.) Thus fo hdden unt, E ε = y = k Downsteam( ) E s k s y k = δ k k Downsteam( ) w k usng the fact that s = w k j kj x j so s y k = s x k = w k Atfcal Neual Netwoks: Slde 32 6

17 Backpop summay Ths gves a ecusve fomulaton of how all the elevant ntemedate quanttes ae computed. To do the computaton teatvely, stat at the output unts, computng the appopate ε and δ values thee, then poceed though the netwok backwads untl all unts have the necessay δ values. It s moe common to fomulate ths wthout explctly dentfyng ε, although dong t ou way moe clealy demonstates the geneal stage-wse oganzaton of ths computaton. Hee s the moe common δ-only fomulaton of backpop: Atfcal Neual Netwoks: Slde 33 Backpop algothm sngle step Basc sngle fowad/backwad computaton fo a gven nput/desed output pa:. Place the nput vecto at the nput nodes and popagate fowad 2. At each output node, compute 3. At each hdden node, compute δ = g ( s ) 4. Fo each weght compute w j k k k Downsteam( ) δ = g ( s )( d y ) w δ x j δ Atfcal Neual Netwoks: Slde 34 7

18 Devatve of squashng functon If the squashng functon s the logstc functon g( s ) = + e the devatve has the convenent fom g ( s ) = g( s )( g( s )) = y ( y ) s Execse: Pove ths Anothe popula choce of squashng functon s tanh, whch takes values n the ange (-,) athe than (,) eques pluggng a dffeent g nto the algothm Atfcal Neual Netwoks: Slde 35 The full backpop algothm Intalze weghts to small andom values Repeat untl satsfed Fo each tanng example Do one fowad and backwad pass to compute fo each adjustable weght Batch veson: accumulate these values ove the tanng set, then do w j w Incemental veson: nsde nne loop do w j j w + η j w j δ x + ηδ x j j δ x j Atfcal Neual Netwoks: Slde 36 8

19 Remaks Batch veson epesents tue gadent descent Incemental veson only an appoxmaton, but often conveges faste n pactce Many vaatons: Momentum essentally smooths successve weght changes Dffeent values of η fo dffeent unts, o as functon of tme, o adapted based on stll othe consdeatons Use of second-ode technques o appoxmatons to them Dawbacks May take many teatons to convege May convege to suboptmal local mnma Leaned netwok may be had to ntepet n humanundestandable tems Atfcal Neual Netwoks: Slde 37 Remaks (cont.) Gadent-based cedt assgnment make changes to all paametes whee such changes would contbute some benefcal effect sze of change popotonal to senstvty make lage changes to paametes to whch benefcal outcome most senstve Atfcal Neual Netwoks: Slde 38 9

20 Lnea unts Sometmes useful to take g = dentty functon,.e., no squashng Appopate fo output unts f the ange of the functon to be leaned not bounded But f all unts ae lnea, multlaye netwoks ae no moe expessve than sngle-laye netwoks Can you see why? In a sngle-laye netwok of lnea unts, backpop also known as LMS o Wdow-Hoff ule wdely used n adaptve contol, sgnal-pocessng, etc. Atfcal Neual Netwoks: Slde 39 Pactcal consdeatons Useful squashng functons only appoach the exteme values asymptotcally E.g., logstc functon can neve actually attan values of o Wth such output unts, tanng to unattanable output values would neve temnate Instead, n pactce use ethe a dead zone: e.g., tan to tagets of and but consde any output wthn a toleance of, say,. to be coect tagets of, say,. and.9 n place of and, espectvely Atfcal Neual Netwoks: Slde 4 2

21 Neual net epesentatons Have to encode all possble nput and output as Eucldean vectos What f nput o output s dscete (e.g., symbolc)? If exactly two possble values, one natual encodng would be to use fo one of these and fo the othe Altenatve encodng that woks fo any fnte numbe of values: use a sepaate node fo each value and set exactly one node to and all othes to called -out-of-n o ado button encodng But f the values have a natual topology (e.g., fall on an odnal scale), mght make sense to use an encodng that captues ths Atfcal Neual Netwoks: Slde 4 Repesentaton example Consde Outlook = Sunny, Ovecast, o Ran -out-of-3 encodng: Sunny Uses 3 nput nodes Ovecast Ran Teatng Ovecast as halfway between Sunny and Ran: Sunny. Uses nput node Ovecast.5 Ran. Such choces help detemne the undelyng nductve bas Atfcal Neual Netwoks: Slde 42 2

22 Othe consdeatons Avodng ovefttng ealy stoppng explct penalty tems weght decay Incopoatng po knowledge enfocng nvaances though weght shang lmtng connectvty lettng some of the nput epesent moe complex pecomputed featues ntalzng the netwok accodng to a best guess, then lettng backpop fne-tune the weghts settng some weghts by hand and keepng them fxed Atfcal Neual Netwoks: Slde 43 Avodng ovefttng by ealy stoppng % coect Tanng Set Valdaton Set Epochs Atfcal Neual Netwoks: Slde 44 22

23 Expessveness Any contnuous functon can be appoxmated abtay closely ove a bounded egon by a two-laye netwok wth sgmod squashng functons n the hdden laye and lnea unts n the output laye (gven enough hdden unts) Atfcal Neual Netwoks: Slde 45 Inductve bas When weghts ae close to zeo, behavo s appoxmately lnea Keepng weghts nea zeo gves a pefeence bas towad lnea functons Atfcal Neual Netwoks: Slde 46 23

24 Wde vaety of applcatons Speech ecognton Autonomous dvng Handwtten dgt ecognton Cedt appoval But may be had to tanslate netwok behavo nto moe explct, easly-undestood ules Backgammon Etc. Geneally appopate fo poblems whee the fnal answe depends heavly on combnatons of many nput featues Decson tees mght be bette when decsons depend on only a small subset of the nput featues Atfcal Neual Netwoks: Slde 47 ALVINN: autonomous vehcle Atfcal Neual Netwoks: Slde 48 24

25 Handwtten dgt ecognton Atfcal Neual Netwoks: Slde 49 Accuacy on test dgts 3-neaest-neghbo 2.4% eo 4-3- unt MLP.6% eo LeNet: unt MLP.9% lmted connectvty to enfoce localty constants weght shang to ceate tanslaton-nvaant featues (leaned) Atfcal Neual Netwoks: Slde 5 25

26 Extenson: Recuent netwoks Wth feedback connectons, atfcal neual netwoks can exhbt nteestng tempoal behavos oscllatons convegence to fxed ponts appoxmate fnte-state machne behavo An extenson of backpop (backpop-though-tme) can be used to tan these behavos Atfcal Neual Netwoks: Slde 5 Leanng to be a FSM Consde the followng nput/output behavo tme Input C B D A D B C B B A C A D B C D... Output... Atfcal Neual Netwoks: Slde 52 26

27 Leanng to be a FSM Consde the followng nput/output behavo tme Input C B D A D B C B B A C A D B C D... Output... If nput == A enabled tue Else If nput == B If enabled output enabled false Else output Else output Bus dve poblem Atfcal Neual Netwoks: Slde 53 Fnte state machne A,C,D/ Mealy Machne A/ Enabled B/ Dsabled B,C,D/ Atfcal Neual Netwoks: Slde 54 27

28 Neual net mplementaton -5 Flp-flop + A B AND gate C + D -5 Weght values appopate fo standad logstc squashng functon -step delays not explctly shown Gadent descent can lean ths fom a steam of I/O examples Atfcal Neual Netwoks: Slde 55 Summay Most bans have lots of neuons, so maybe the knds of computng that bans ae good at ae best accomplshed by lage netwoks of smple computng unts (lnea theshold unts?) One-laye netwoks nsuffcently expessve Multlaye netwoks ae suffcently expessve and can be taned by gadent descent,.e., eo backpopagaton Some geneal-pupose ways to look at leanng Fomulaton as an optmzaton poblem Gadent seach when appopate Vaous technques fo ncopoatng po knowledge and fo avodng ovefttng Many applcatons Even some tempoal behavos can be taned by backpopagaton-lke gadent descent algothms Atfcal Neual Netwoks: Slde 56 28