Regression and Classification with Neural Networks

Transcription

1 Note to other teachers and users of these sldes. Andre ould be delhted f you found ths source materal useful n vn your on lectures. Feel free to use these sldes verbatm, or to modfy them to ft your on needs. PoerPont ornals are avalable. If you mae use of a snfcant porton of these sldes n your on lecture, please nclude ths messae, or the follon ln to the source repostory of Andre s tutorals: Comments and correctons ratefully receved. eresson and Classfcaton th Neural Netors Andre W. Moore Professor School of Computer Scence Carnee Mellon Unversty.cs.cmu.edu/~am am@cs.cmu.edu Copyrht 00, 003, Andre W. Moore Sep 5th, 00 Lnear eresson DATASET nputs outputs y y. y 3 y 4.9 y 5 3. Lnear reresson assumes that the epected value of the output ven an nput, E[y ], s lnear. Smplest case: Out( for some unnon. Gven the data, e can estmate. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde

2 -parameter lnear reresson Assume that the data s formed by y + nose here the nose snals are ndependent the nose has a normal dstrbuton th mean 0 and unnon varance σ P(y, has a normal dstrbuton th mean varance σ Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 3 Bayesan Lnear eresson P(y, Normal (mean, var σ We have a set of dataponts (,y (,y ( n,y n hch are EVIDENCE about. We ant to nfer from the data. P(,, 3, n, y, y y n You can use BAYES rule to or out a posteror dstrbuton for ven the data. Or you could do Mamum Lelhood Estmaton Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 4

3 Mamum lelhood estmaton of Ass the queston: For hch value of s ths data most lely to have happened? <> For hat s P(y, y y n,, 3, n, mamzed? <> For hat s n P( y, mamzed? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 5 For hat s n P ( y, For hat s ep( ( For hat s y For hat s n y σ n σ n mamzed? mamzed? mamzed? ( y mnmzed? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 6

4 Lnear eresson The mamum lelhood s the one that mnmzes sumof-squares of resduals Ε E( ( y ( y + ( We ant to mnmze a quadratc functon of. y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 7 Easy to sho the sum of squares s mnmzed hen Lnear eresson y The mamum lelhood model s Out We can use t for predcton ( Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 8

5 Easy to sho the sum of squares s mnmzed hen Lnear eresson y The mamum lelhood model s Out ( p( Note: In Bayesan stats you d have ended up th a prob dst of And predctons ould have ven a prob dst of epected output We can use t for predcton Often useful to no your confdence. Ma lelhood can ve some nds of confdence too. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 9 Multvarate eresson What f the nputs are vectors? d nput eample Dataset has form y y 3 y 3.: : ẋ y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 0

6 Multvarate eresson Wrte matr X and Y thus: M M... m m m y y y M y (there are dataponts. Each nput has m components The lnear reresson model assumes a vector such that Out( T [] + [] +. m [D] The ma. lelhood s (X T X - (X T Y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde Multvarate eresson Wrte matr X and Y thus: M M... m m m y y y M y IMPOTANT EXECISE: (there are dataponts. Each nput has POVE m components IT!!!!! The lnear reresson model assumes a vector such that Out( T [] + [] +. m [D] The ma. lelhood s (X T X - (X T Y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde

7 Multvarate eresson (con t The ma. lelhood s (X T X - (X T Y X T X s an m m matr:, th elt s X T Y s an m-element vector: th elt y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 3 What about a constant term? We may epect lnear data that does not o throuh the orn. Statstcans and Neural Net Fols all aree on a smple obvous hac. Can you uess?? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 4

8 The constant term The trc s to create a fae nput X 0 that alays taes the value X 3 5 X Y Before: Y X + X has to be a poor model X 0 X 3 5 X Y After: Y 0 X 0 + X + X 0 + X + X has a fne constant term In ths eample, You should be able to see the MLE 0, and by nspecton Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 5 eresson th varyn nose Suppose you no the varance of the nose that as added to each datapont. ½ 3 y ½ 3 Assume σ 4 /4 4 /4 y ~ y3 y y y0 N( σ σ σ σ/ 0 3, σ σ/ What s the MLE estmate of? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 6

9 MLE estmaton th varyn nose arma lo p( y, y,..., y armn such that,,...,, σ, σ,..., σ, Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 7 ( y σ ( y 0 σ y σ σ Assumn..d. and then plun n equaton for Gaussan and smplfyn. Settn dll/d equal to zero Trval alebra Ths s Wehted eresson We are asn to mnmze the ehted sum of squares armn ( y σ y3 y y y0 σ σ σ σ/ 0 3 σ/ here eht for th datapont s σ Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 8

10 Wehted Multvarate eresson The ma. lelhood s (WX T WX - (WX T WY (WX T WX s an m m matr:, th elt s (WX T WY s an m-element vector: th elt σ y σ Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 9 Non-lnear eresson Suppose you no that y s related to a functon of n such a ay that the predcted values have a non-lnear dependence on, e.: ½ 3 3 Assume y ½ y3 y y y0 y ~ N( + 0 3, σ What s the MLE estmate of? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 0

11 Non-lnear MLE estmaton arma lo p( y, y,..., y,,..., armn such that ( y + y + + 0, σ, Assumn..d. and then plun n equaton for Gaussan and smplfyn. Settn dll/d equal to zero Copyrht 00, 003, Andre W. Moore Neural Netors: Slde Non-lnear MLE estmaton arma lo p( y, y,..., y,,..., armn such that ( y + y + +, σ, Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 0 Assumn..d. and then plun n equaton for Gaussan and smplfyn. Settn dll/d equal to zero We re don the alebrac tolet So uess hat e do?

12 Non-lnear MLE estmaton arma Common (but not only approach: Numercal Solutons: Lne Search lo p( y, y,..., y,,..., armn Smulated Annealn Gradent Descent Conuate Gradent such that Levenber Marquart Neton s Method Also, specal purpose statstcaloptmzaton-specfc trcs such as E.M. (See Gaussan Mtures lecture for ntroducton ( y + y + +, σ, Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 3 0 Assumn..d. and then plun n equaton for Gaussan and smplfyn. Settn dll/d equal to zero We re don the alebrac tolet So uess hat e do? GADIENT DESCENT f(: Suppose e have a scalar functon We ant to fnd a local mnmum. Assume our current eht s GADIENT DESCENT ULE: η f ( η s called the LEANING ATE. A small postve number, e.. η 0.05 Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 4

13 GADIENT DESCENT f(: Suppose e have a scalar functon We ant to fnd a local mnmum. Assume our current eht s GADIENT DESCENT ULE: η f ( ecall Andre s favorte default value for anythn η s called the LEANING ATE. A small postve number, e.. η 0.05 QUESTION: Justfy the Gradent Descent ule Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 5 Gradent Descent n m Dmensons Gven f( : m f ( GADIENT DESCENT ULE: Equvalently m f M ( f ( f ( - η ponts n drecton of steepest ascent. s the radent n that drecton -η f f ( (.here s the th eht ust le a lnear feedbac system Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 6

14 What s all ths ot to do th Neural Nets, then, eh?? For supervsed learnn, neural nets are also models th vectors of parameters n them. They are no called ehts. As before, e ant to compute the ehts to mnmze sumof-squared resduals. Whch turns out, under Gaussan..d nose assumpton to be ma. lelhood. Instead of eplctly solvn for ma. lelhood ehts, e use GADIENT DESCENT to SEACH for them. Why? you as, a querulous epresson n your eyes. Aha!! I reply: We ll see later. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 7 Lnear Perceptrons They are multvarate lnear models: Out( T And trann conssts of mnmzn sum-of-squared resduals by radent descent. Ε ( Out ( y Τ ( y QUESTION: Derve the perceptron trann rule. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 8

15 Lnear Perceptron Trann ule E T ( y Gradent descent tells us e should update thusly f e sh to mnmze E: E - η E So hat s? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 9 Lnear Perceptron Trann ule E T ( y Gradent descent tells us e should update thusly f e sh to mnmze E: E - η E So hat s? E Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 30 T ( y T T ( y ( y T δ δ m δ δ y here T

16 Lnear Perceptron Trann ule E T ( y Gradent descent tells us e should update thusly f e sh to mnmze E: E - η here E δ + η Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 3 δ We frequently nelect the (meann e halve the learnn rate The Batch perceptron alorthm andomly ntalze ehts m Get your dataset (append s to the nputs f you don t ant to o throuh the orn. 3 for to δ : y Τ 4 for to m + η 5 f δ stops mprovn then stop. Else loop bac to 3. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 3 δ

17 δ y Τ + ηδ A ULE KNOWN BY MANY NAMES The LMS ule The delta rule The Wdro Hoff rule Classcal condtonn The adalne rule Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 33 If data s volumnous and arrves fast Input-output pars (,y come streamn n very qucly. THEN Don t bother remembern old ones. Just eep usn ne ones. observe (,y δ y Τ + η δ Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 34

18 Gradent Descent vs Matr Inverson for Lnear Perceptrons GD Advantaes (MI dsadvantaes: Bolocally plausble Wth very very many attrbutes each teraton costs only O(m. If feer than m teratons needed e ve beaten Matr Inverson More easly parallelzable (or mplementable n etare? GD Dsadvantaes (MI advantaes: It s moronc It s essentally a slo mplementaton of a ay to buld the XTX matr and then solve a set of lnear equatons If m s small t s especally outaeous. If m s lare then the drect matr nverson method ets fddly but not mpossble f you ant to be effcent. Hard to choose a ood learnn rate Matr nverson taes predctable tme. You can t be sure hen radent descent ll stop. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 35 Gradent Descent vs Matr Inverson for Lnear Perceptrons GD Advantaes (MI dsadvantaes: Bolocally plausble Wth very very many attrbutes each teraton costs only O(m. If feer than m teratons needed e ve beaten Matr Inverson More easly parallelzable (or mplementable n etare? GD Dsadvantaes (MI advantaes: It s moronc It s essentally a slo mplementaton of a ay to buld the XTX matr and then solve a set of lnear equatons If m s small t s especally outaeous. If m s lare then the drect matr nverson method ets fddly but not mpossble f you ant to be effcent. Hard to choose a ood learnn rate Matr nverson taes predctable tme. You can t be sure hen radent descent ll stop. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 36

19 Gradent Descent vs Matr Inverson for Lnear Perceptrons GD Advantaes (MI dsadvantaes: Bolocally plausble Wth very very many attrbutes each teraton costs only O(m. If feer than m teratons needed e ve beaten Matr Inverson More easly parallelzable (or mplementable But e ll n etare? GD Dsadvantaes (MI soon advantaes: see that It s moronc GD It s essentally a slo mplementaton has an mportant of a ay to etra buld the XTX matr and then solve a set of lnear equatons trc up ts sleeve If m s small t s especally outaeous. If m s lare then the drect matr nverson method ets fddly but not mpossble f you ant to be effcent. Hard to choose a ood learnn rate Matr nverson taes predctable tme. You can t be sure hen radent descent ll stop. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 37 Perceptrons for Classfcaton What f all outputs are 0 s or s? or We can do a lnear ft. Our predcton s 0 f out( / f out(>/ WHAT S THE BIG POBLEM WITH THIS??? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 38

20 Perceptrons for Classfcaton What f all outputs are 0 s or s? or We can do a lnear ft. Blue Out( Our predcton s 0 f out( ½ f out(>½ WHAT S THE BIG POBLEM WITH THIS??? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 39 Perceptrons for Classfcaton What f all outputs are 0 s or s? or We can do a lnear ft. Our predcton s 0 f out( ½ f out(>½ Blue Out( Green Classfcaton Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 40

21 Classfcaton th Perceptrons I Τ (. Don t mnmze y Mnmze number of msclassfcatons nstead. [Assume outputs are + & -, not + & 0] y ( Τ ound here ound( - f <0 f 0 The radent descent rule can be chaned to: f (,y correctly classed, don t chane f ronly predcted as f ronly predcted as - ( - + NOTE: CUTE & NON OBVIOUS WHY THIS WOKS!! Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 4 Classfcaton th Perceptrons II: Smod Functons Least squares ft useless Ths ft ould classfy much better. But not a least squares ft. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 4

22 Classfcaton th Perceptrons II: Smod Functons Least squares ft useless SOLUTION: Instead of Out( T We ll use Out( ( T ( ( here : 0, squashn functon s a Ths ft ould classfy much better. But not a least squares ft. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 43 ( h + ep( h The Smod Note that f you rotate ths curve throuh 80 o centered on (0,/ you et the same curve..e. (h-(-h Can you prove ths? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 44

23 ( h + ep( h The Smod No e choose to mnmze Τ [ y Out( ] [ y ( ] Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 45 Lnear Perceptron Classfcaton eons X X We ll use the model Out( ( T (, ( Whch reon of above daram classfed th +, and hch th 0?? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 46

24 Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 47 Gradent descent th smod on a perceptron ( ( ( ( ( ( ( ( ( ( ( ( Ε Ε y y y y e e e e e e e e e net Out( here net net ' Out( ' so Because: ' notce Frst, δ δ ( + δ η m y δ The smod perceptron update rule: here Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 48 Other Thns about Perceptrons Invented and popularzed by osenblatt (96 Even th smod nonlnearty, correct converence s uaranteed Stable behavor for overconstraned and underconstraned problems

25 Perceptrons and Boolean Functons If nputs are all 0 s and s and outputs are all 0 s and s Can learn the functon X X Can learn the functon. X Can learn any conuncton of lterals, e.. ~ ~ X QUESTION: WHY? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 49 Perceptrons and Boolean Functons Can learn any dsuncton of lterals e.. ~ ~ Can learn maorty functon f(, n f n/ s or more are 0 f less than n/ s are What about the eclusve or functon? f(, ( ~ (~ Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 50

26 Multlayer Netors The class of functons representable by perceptrons s lmted Out( Τ ( Use a der representaton! Out( W Ths s a nonlnear functon Of a lnear combnaton Of non lnear functons Of lnear combnatons of nputs Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 5 A -HIDDEN LAYE NET N INPUTS N HIDDEN 3 3 v v N INS N INS N HID Out W v 3 v 3 N INS Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 5 3 3

27 OTHE NEUAL NETS 3 -Hdden layers + Constant Term JUMP CONNECTIONS NINS N HID Out 0 + Wv Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 53 Bacpropaaton Out( W Fnd a set of ehts to mnmze ( y Out( by radent descent. { W },{ } That s t! t! That s the the bacpropaaton alorthm. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 54

28 Bacpropaaton Converence Converence to a lobal mnmum s not uaranteed. In practce, ths s not a problem, apparently. Tean to fnd the rht number of hdden unts, or a useful learnn rate η, s more hassle, apparently. IMPLEMENTING BACKPOP: Dfferentate Monster sum-square resdual Wrte don the Gradent Descent ule It turns out to be easer & computatonally effcent to use lots of local varables th names le h o v net etc Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 55 Choosn the learnn rate Ths s a subtle art. Too small: can tae days nstead of mnutes to convere Too lare: dveres (MSE ets larer and larer hle the ehts ncrease and usually oscllate Sometmes the ust rht value s hard to fnd. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 56

29 Learnn-rate problems From J. Hertz, A. Kroh, and. G. Palmer. Introducton to the Theory of Neural Computaton. Addson-Wesley, 994. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 57 Improvn Smple Gradent Descent Momentum Don t ust chane ehts accordn to the current datapont. e-use chanes from earler teratons. Let (t eht chanes at tme t. Let Ε η Instead e use Momentum damps oscllatons. A hac? Well, maybe. be the chane e ould mae th reular radent descent. Ε t + t + ( t + η + α ( t ( ( ( t momentum parameter Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 58

30 Momentum llustraton Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 59 Improvn Smple Gradent Descent Neton s method E( + h E( + h T E + h T E h + O( h 3 If e nelect the O(h 3 terms, ths s a quadratc form Quadratc form fun facts: If y c + b T -/ T A And f A s SPD Then opt A - b s the value of that mamzes y Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 60

31 Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 6 Improvn Smple Gradent Descent Neton s method ( ( ( 3 h h h h h O E E E E T T If e nelect the O(h 3 terms, ths s a quadratc form E E Ths should send us drectly to the lobal mnmum f the functon s truly quadratc. And t mht et us close f t s locally quadratcsh Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 6 Improvn Smple Gradent Descent Neton s method ( ( ( 3 h h h h h O E E E E T T If e nelect the O(h 3 terms, ths s a quadratc form E E Ths should send us drectly to the lobal mnmum f the functon s truly quadratc. And t mht et us close f t s locally quadratcsh BUT (and t s a b but That second dervatve matr can be epensve and fddly to compute. If e re not already n the quadratc bol, e ll o nuts.

32 Improvn Smple Gradent Descent Conuate Gradent Another method hch attempts to eplot the local quadratc bol assumpton But does so hle only needn to use and not E E It s also more stable than Neton s method f the local quadratc bol assumpton s volated. It s complcated, outsde our scope, but t often ors ell. More detals n Numercal ecpes n C. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 63 BEST GENEALIZATION Intutvely, you ant to use the smallest, smplest net that seems to ft the data. HOW TO FOMALIZE THIS INTUITION?. Don t. Just use ntuton. Bayesan Methods Get t ht 3. Statstcal Analyss eplans hat s on on 4. Cross-valdaton Dscussed n the net lecture Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 64

33 What You Should Kno Ho to mplement multvarate Leastsquares lnear reresson. Dervaton of least squares as ma. lelhood estmator of lnear coeffcents The eneral radent descent rule Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 65 What You Should Kno Perceptrons Lnear output, least squares Smod output, least squares Multlayer nets The dea behnd bac prop Aareness of better mnmzaton methods Generalzaton. What t means. Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 66

34 To Dscuss: APPLICATIONS What can non-lnear reresson be useful for? What can neural nets (used as non-lnear reressors be useful for? What are the advantaes of N. Nets for nonlnear reresson? What are the dsadvantaes? Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 67 Other Uses of Neural Nets Tme seres th recurrent nets Unsupervsed learnn (clustern prncpal components and non-lnear versons thereof Combnatoral optmzaton th Hopfeld nets, Boltzmann Machnes Evaluaton functon learnn (n renforcement learnn Copyrht 00, 003, Andre W. Moore Neural Netors: Slde 68