CIS603 - Artificial Intelligence. Logistic regression. (some material adopted from notes by M. Hauskrecht) CIS603 - AI. Supervised learning

Transcription

1 CIS63 - Artfcal Itellgece Logstc regresso Vasleos Megalookoomou some materal adopted from otes b M. Hauskrecht Supervsed learg Data: D { d d.. d} a set of eamples d < > s put vector ad s desred output gve b a teacher Obectve: lear the mappg f : X Y s.t. f for all.. To tpes of problems: Regresso: Y s cotuous Eample: eargs product orders compa stock prce Classfcato: Y s dscrete Eample: temperature heart rate dsease No: BINARY classfcato problems

2 Bar classfcato To classes Y {} Our goal s to lear to classf correctl to tpes of eamples Class labeled as Class labeled as We ould lke to lear f : X {} Frst step: e eed to devse a model of the fucto f Isprato: euro erve cells Neuro euro erve cell ad ts actvtes

3 Neuro-based bar classfcato model z k Threshold fucto Bar classfcato Istead of learg the mappg to dscrete values f : X {} It s easer to lear a probablstc fucto f : X [] here f descrbes the probablt of a class gve p Trasformato to dscrete class values: If p / the choose Else choose Logstc regresso model uses a probablstc fucto 3

4 Logstc regresso: Logstc regresso p g z g k here are parameters of the models ad gz s a logstc fucto g z / + e z k Bas term Iput vector p k k z Logstc fucto Logstc fucto fucto g z z + e also referred to as sgmod fucto replaces threshold fucto th smooth stchg takes a real umber ad outputs the umber the terval []

5 Logstc regresso - Decso boudar Logstc regresso model defes a lear decso boudar Eample: classes crosses ad crcles Decso boudar Bar classfcato - Error To classes Y {} Our goal s to classf correctl as ma eamples as possble Zero-oe error fucto Error f f Error e ould lke to mmze: E Error The error s mmzed f e choose: f p > p otherse We costruct a probablstc verso of the error fucto based o the lkelhood of the data L D P D Iverse optmzato problem Error D L D 5

6 6 Lkelhood of data We at eghts that mamze the lkelhood of data Trck: mamze the log-lkelhood of data stead Ratoal: The optmal eghts are the same for both the lkelhood ad the log-lkelhood Logstc regresso: parameter learg D l log log D P D L k o g z g p + + d J ole log log > < d here Logstc regresso: parameter estmato log log ole d J D l + ole z g d J Log lkelhood O-le compoet of the log-lkelhood Dervatves of the ole error compoet terms of eghts log log ole d J + ole z g d J

7 Logstc regresso. Ole gradet. We at to fd the set of parameters optmzg the loglkelhood of data or mmzg the error O-le learg update for eght J ole d * α [ J ole d * ] +th update for the logstc regresso ad d < k + u + α + g + u u > + + α + g + k u u u α - aealed learg rate depeds o the umber of updates The same eas update rule as used the lear regresso!!! Ole logstc regresso algorthm Ole-logstc-regresso D umber of teratos talze eghts k for :: umber of teratos do select a data pot d<> from D set α / update eghts parallel + α[ p ] ed for retur eghts + α[ p ] 7

8 Ole algorthm. Eample. Ole algorthm. Eample. 8

9 Ole algorthm. Eample. Lmtatos of basc lear uts Lear regresso Logstc regresso z p k k k k Fucto lear puts Lear decso boudar 9

10 Logstc regresso - Decso boudar Logstc regresso model defes a lear decso boudar Eample: classes crosses ad crcles Decso boudar Lear decso boudar Eample he logstc regresso model s ot optmal but ot that bad 3 Decso boudar

11 Whe logstc regresso fals? Eample hch the logstc regresso model fals Lmtatos of logstc regresso. part fucto - o lear decso boudar

12 Etesos of smple lear uts Replace puts to lear uts th feature bass fuctos to model oleartes f φ m + φ - a arbtrar fucto of φ φ k φ m 3 The same trck ca be doe for the logstc regresso Eteso of smple lear uts Eample: Fttg of a polomal of degree m Data pots: pars of < > Feature fuctos: φ Fucto to lear: f O le update for <> par + m + α f + α f

13 Mult-laered eural etorks Alteratve a to troduce oleartes to regresso/classfcato models Idea: Cascade several smple eural models based o logstc regresso. Much lke euro coectos. 3