Pattern Recognition 2015 Linear Models for Classification

Transcription

1 Pttern Recognition 2015 Liner Models for Clssifiction Ad Feelders Universiteit Utrecht Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 1 / 55

2 Clssifiction Prolems We re concerned with the prolems of 1 Allocting n oject to clss, on the sis of numer of vriles tht descrie the oject. 2 Estimting the proility tht prticulr oject elongs to specific clss. Interconnected, since lloction is usully sed on the estimted proilities. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 2 / 55

3 Exmples of Clssifiction Prolems Churn: is customer going to leve for competitor? SPAM filter: e-mil messge is SPAM or not? Medicl dignosis: does ptient hve rest cncer? Hndwritten digit recognition. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 3 / 55

4 Clssifiction Prolems In this kind of clssifiction prolem there is trget vrile t tht ssumes vlues in n unordered discrete set. An importnt specil cse is when there re only two clsses, in which cse we usully choose t {0, 1}. The gol of clssifiction procedure is to predict the trget vlue (clss lel) given set of input vlues x = {x 1,..., x D } mesured on the sme oject. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 4 / 55

5 Clssifiction Prolems At prticulr point x the vlue of t is not uniquely determined. It cn ssume oth its vlues with respective proilities tht depend on the loction of the point x in the input spce. We write p(c 1 x) = 1 p(c 2 x) = y(x). The gol of clssifiction procedure is to produce n estimte of y(x) t every input point. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 5 / 55

6 Two types of pproches to clssifiction Discrimintive Models ( regression ; section 4.3). Genertive Models ( density estimtion ; section 4.2). Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 6 / 55

7 Discrimintive Models Discrimintive methods only model the conditionl distriution of t given x. The proility distriution of x itself is not modeled. For the inry clssifiction prolem: y(x) = p(c 1 x) = f (x, w) where f (x, w) is some deterministic function of x. Note tht the pproch to regression we discussed follows the sme strtegy. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 7 / 55

8 Discrimintive Models Exmples of discrimintive clssifiction methods: Liner proility model Logistic regression Feed-forwrd neurl networks... Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 8 / 55

9 Genertive Models An lterntive prdigm for estimting y(x) is sed on density estimtion. Here Byes theorem y(x) = p(c 1 x) = p(c 1 )p(x C 1 ) p(c 1 )p(x C 1 ) + p(c 2 )P(x C 2 ) is pplied where p(x C k ) re the clss conditionl proility density functions nd p(c k ) re the unconditionl ( prior ) proilities of ech clss. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 9 / 55

10 Genertive Models Exmples of density estimtion sed clssifiction methods: Liner/Qudrtic Discriminnt Anlysis, Nive Byes clssifier,... Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 10 / 55

11 Discrimintive Models: liner proility model Consider the liner regression model t = w x + ε t {0, 1} By ssumption E[ε x] = 0, so we hve E[t x] = w x But E[t x] = 1 p(t = 1 x) + 0 p(t = 0 x) = p(t = 1 x) p(c 1 x) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 11 / 55

12 Liner response function Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 12 / 55

13 Logistic regression Logistic response function E[t x] = ew x 1 + e w x or (divide numertor nd denomintor y e w x ) E[t x] = e w x = (1 + e w x ) 1 (4.59 nd 4.87) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 13 / 55

14 Logistic Response Function Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 14 / 55

15 Lineriztion: the logit trnsformtion ln p(c 1 x) 1 p(c 1 x) (1 + e w x ) 1 = ln 1 (1 + e w x ) 1 1 = ln (1 + e w x ) 1 = ln 1 e w x = ln e w x = w x In the second step, we divided the numertor nd the denomintor y (1 + e w x ) 1. The rtio p(c 1 x)/(1 p(c 1 x)) is clled the odds. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 15 / 55

16 Liner Seprtion Assign to clss C 1 if p(c 1 x) > p(c 2 x), i.e. if p(c 1 x) 1 p(c 1 x) > 1 This is true if { } p(c1 x) ln > 0 1 p(c 1 x) So ssign to clss C 1 if w x > 0, nd to clss C 2 otherwise. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 16 / 55

17 Mximum Likelihood Estimtion t = 1 if heds, t = 0 if tils. µ = p(t = 1). One coin flip p(t) = µ t (1 µ) 1 t Note tht p(1) = µ, p(0) = 1 µ s required. Sequence of N independent coin flips p(t) = p(t 1, t 2,..., t N ) = N µ tn (1 µ) 1 tn which defines the likelihood function when viewed s function of µ. n=1 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 17 / 55

18 Mximum Likelihood Estimtion In sequence of 10 coin flips we oserve t = (1, 0, 1, 1, 0, 1, 1, 1, 1, 0). The corresponding likelihood function is p(t µ) = µ (1 µ) µ µ (1 µ) µ µ µ µ (1 µ) = µ 7 (1 µ) 3 The corresponding loglikelihood function is ln p(t µ) = ln(µ 7 (1 µ) 3 ) = 7 ln µ + 3 ln(1 µ) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 18 / 55

19 Computing the mximum To determine the mximum we tke the derivtive nd equte it to zero d ln p(t µ) dµ = 7 µ 3 1 µ = 0 which yields mximum likelihood estimte µ ML = 0.7. This is just the reltive frequency of heds in the smple. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 19 / 55

20 Loglikelihood function for t = (1, 0, 1, 1, 0, 1, 1, 1, 1, 0) loglikelihood Ad Feelders ( Universiteit Utrecht ) Pttern Recognition mu 20 / 55

21 ML estimtion for logistic regression Now proility of success depends on x n : y n = p(c 1 x n ) = (1 + e w x n ) 1 1 y n = p(c 2 x n ) = (1 + e w x n ) 1 we cn represent its proility distriution s follows p(t n ) = y tn n (1 y n ) 1 tn t n {0, 1}; n = 1,..., N Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 21 / 55

22 ML estimtion for logistic regression Exmple n x n t n p(t n ) (1 + e w 0+8w 1 ) (1 + e w 0+12w 1 ) (1 + e w 0 15w 1 ) (1 + e w 0 10w 1 ) 1 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 22 / 55

23 LR: likelihood function Since the t n oservtions re independent: p(t w) = N p(t n ) = n=1 Or, tking minus the nturl log: ln p(t w) = ln = N n=1 N n=1 y tn n (1 y n ) 1 tn y tn n (1 y n ) 1 tn (4.89) N {t n ln y n + (1 t n ) ln(1 y n )} (4.90) n=1 This is clled the cross-entropy error function. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 23 / 55

24 LR: error function Since for the logistic regression model we get E(w) = N n=1 y n = (1 + e w x n ) 1 1 y n = (1 + e w x n ) 1 { } t n ln(1 + e w x n ) + (1 t n ) ln(1 + e w x n ) Non-liner function of the prmeters. No closed form solution. Likelihood function glolly concve. Estimte with e.g. itertive re-weighted lest squres (section 4.3.3). Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 24 / 55

25 Fitted Response Function Sustitute mximum likelihood estimtes into the response function to otin the fitted response function ˆp(C 1 x) = ew ML x 1 + e w ML x Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 25 / 55

26 Exmple: Progrmming Assignment Model the proility of successfully completing progrmming ssignment. Explntory vrile: progrmming experience. We find w 0 = nd w 1 = , so ˆp(C 1 x n ) = 14 months of progrmming experience: e xn 1 + e xn ˆp(C 1 x = 14) = e (14) e (14) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 26 / 55

27 Exmple: Progrmming Assignment month.exp success fitted month.exp success fitted Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 27 / 55

28 Alloction Rule Proility of the clsses is equl when Solving for x we get x x = 0 Alloction Rule: x 19: ssign to clss C 1 (t = 1) x < 19: ssign to clss C 2 (t = 0) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 28 / 55

29 Progrmming Assignment: Confusion Mtrix Cross tle of oserved nd predicted clss lel: Row: oserved, Column: predicted Error rte: 6/25= Defult (predict mjority clss): 11/25=0.44 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 29 / 55

30 How to in R > prog.logreg <- glm(succes month.exp, dt=prog.dt, fmily=inomil) > summry(prog.logreg) Coefficients: Estimte Std. Error z vlue Pr(> z ) (Intercept) * month.exp * --- Numer of Fisher Scoring itertions: 4 > tle(prog.dt$succes, s.numeric(prog.logreg$fitted > 0.5)) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 30 / 55

31 Exmple: Conn s syndrome Two possile cuses: () Benign tumor (denom) of the drenl cortex. () More diffuse ffection of the drenl glnds (ilterl hyperplsi). Pre-opertive dignosis on sis of 1 Sodium concentrtion (mmol/l) 2 CO 2 concentrtion (mmol/l) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 31 / 55

32 Conn s syndrome: the dt =1, =0 sodium co2 cuse sodium co2 cuse Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 32 / 55

33 Conn s Syndrome: Plot of Dt sodium co2 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 33 / 55

34 Mximum Likelihood Estimtion The mximum likelihood estimtes re: w 0 = w 1 = w 2 = Assign to group if nd to group otherwise sodium 0.76 CO 2 > 0 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 34 / 55

35 Conn s Syndrome: Alloction Rule sodium co2 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 35 / 55

36 How to in R # plot dt points > plot(conn.dt[,1],conn.dt[,2], pch=c(rep("",20), rep("",10)), col=c(rep(4,20), rep(2,10)), cex=1.5, xl="sodium", yl="co2") # drw decision oundry > line( / , / ,lwd=2) co sodium Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 36 / 55

37 Conn s Syndrome: Confusion Mtrix Cross tle of oserved nd predicted clss lel: Row: oserved, Column: predicted Error rte: 5/30=1/6 Defult: 1/ Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 37 / 55

38 Conn s Syndrome: Line with lower empiricl error sodium co Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 38 / 55

39 Likelihood nd Error Rte Likelihood mximiztion is not the sme s error rte minimiztion! n t n ˆp 1 (t n = 1) ˆp 2 (t n = 1) Which model hs the lower error-rte? Which one the higher likelihood? Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 39 / 55

40 Qudrtic Model Coefficient Vlue (Intercept) sodium CO sodium CO sodium CO Cross tle of oserved (row) nd predicted clss lel: Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 40 / 55

41 Conn s Syndrome: Qudrtic Specifiction sodium co Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 41 / 55

42 Non-inry clsses in logistic regression Recll the logistic regression model ssumption for inry clss vrile t {0, 1}: p(t = 1 x) = exp(w x) 1 + exp(w x) from which it follows tht p(t = 0 x) = exp(w x) since p(t = 1 x) + p(t = 0 x) = 1 Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 42 / 55

43 Non-inry clsses in logistic regression We cn generlize this model to non-inry clss vrile t {0, 1,..., K 1} (where K is the numer of clsses) s follows p(t = k x) = exp(w k x) K 1 j=0 exp(w j x) (4.104 nd 4.105) where we now hve weight vector w k for ech clss. This is clled the multinomil logit model or multi-clss logistic regression (section 4.3.4). Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 43 / 55

44 Multi-clss logistic regression We cn rrive t this model in the following steps: 1 Assume tht p(t = k x) is function of the liner comintion w k x. 2 To ensure tht the proilities re non-negtive, tke the exponentil exp(w k x). 3 To mke sure the proilities sum to 1, we divide exp(w k x) y K 1 j=0 exp(w j x): p(t = k x) = exp(w k x) K 1 j=0 exp(w j x) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 44 / 55

45 Identifiction Restriction Note tht exp((w k + d) x) K 1 j=0 exp((w j + d) x) = exp(d x) exp(wk exp(d x) x) K 1 j=0 exp(w j x) so dding vector d to ech of the vectors w j, j = 0,..., K 1 would yield the sme fitted proilities. To identify the model, we put w 0 = 0. Verify tht inry logistic regression is specil cse of the multinomil logit model, with K = 2, since exp(w0 x) = exp(0) = 1. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 45 / 55

46 Interprettion We hve ln { } p(t = k x) = (w k w l ) x, p(t = l x) which llows us to interpret w k,i w l,i s follows: for unit increse in x i, the log-odds of clss k versus clss l is expected to chnge y w k,i w l,i units, holding ll the other vriles constnt. Since w 0 = 0, w k,i is the effect of x i on the log-odds of clss k reltive to clss 0: for unit increse in x i, the log-odds of clss k versus clss 0 is expected to chnge y w k,i units, holding ll the other vriles constnt. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 46 / 55

47 Multinomil Logit in R # lod trining dt > optdigits.trin <- red.csv("d:/pttern Recognition/Dtsets/optdigits-tr.txt", heder=f) # convert clss lel to fctor > optdigits.trin[,65] <- s.fctor(optdigits.trin[,65]) # sme for test dt > optdigits.test <- red.csv("d:/pttern Recognition/Dtsets/optdigits-tes.txt", heder=f) > optdigits.test[,65] <- s.fctor(optdigits.test[,65]) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 47 / 55

48 Multinomil Logit in R # lod nnet lirry > lirry(nnet) # fit multinomil logistic regression model # column 1 nd 40 re not used (lwys 0) > optdigits.multinom <- multinom(v65., dt = optdigits.trin[,-c(1,40)], mxit = 1000) # weights: 640 (567 vrile) initil vlue converged # predict clss lel on trining dt > optdigits.multinom.pred <- predict(optdigits.multinom, optdigits.trin[,-c(1,40,65)],type="clss") Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 48 / 55

49 Multinomil Logit in R # mke confusion mtrix: true lel vs. predicted lel > tle(optdigits.trin[,65],optdigits.multinom.pred) optdigits.multinom.pred Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 49 / 55

50 Multinomil Logit in R # predict clss lel on test dt > optdigits.multinom.test.pred <- predict(optdigits.multinom, optdigits.test[,-c(1,40,65)],type="clss") > tle(optdigits.test[,65],optdigits.multinom.test.pred) optdigits.multinom.test.pred Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 50 / 55

51 Multinomil Logit in R # mke confusion mtrix for predictions on test dt > confmt <- tle(optdigits.test[,65], optdigits.multinom.test.pred) # use it to compute ccurcy on test dt > sum(dig(confmt))/sum(confmt) [1] The ccurcy on the test smple is out 89%. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 51 / 55

52 Multinomil Logit with LASSO With ordinry multinomil logit the ccurcy on the trining set ws 100%, ut on the test set only 89%. Mye we re overfitting. Apply regulriztion (LASSO). # lod glmnet lirry M 1 Ẽ(w) = E(w) + λ w i i=1 > lirry(glmnet) # pply 10-fold cross-vlidtion with different vlues of lmd > optdigits.lsso.cv <- cv.glmnet(s.mtrix(optdigits.trin[,-c(1,40,65)]), optdigits.trin[,65],fmily="multinomil", type.mesure="clss") # plot lmd gins misclssifiction error > plot(optdigits.lsso.cv) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 52 / 55

53 Plot of lmd ginst misclssifiction error Misclssifiction Error log(lmd) Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 53 / 55

54 Results of Cross-Vlidtion Best vlue is λ The cross vlidtion misclssifiction error is out 3% for this vlue of λ. On verge, only 27 out of 62 coefficients re non-zero. Sprse solution. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 54 / 55

55 Prediction on Test Set # predict clss lel on test set using the est cv model > optdigits.lsso.cv.pred <- predict(optdigits.lsso.cv, s.mtrix(optdigits.test[,-c(1,40,65)]),type="clss") # mke the confusion mtrix > optdigits.lsso.cv.confmt <- tle(optdigits.test[,65],optdigits.lsso.cv.pred) # compute the ccurcy on the test set > sum(dig(optdigits.lsso.cv.confmt))/ sum(optdigits.lsso.cv.confmt) [1] We hve improved the ccurcy from 89% to 95% y using regulriztion. Ad Feelders ( Universiteit Utrecht ) Pttern Recognition 55 / 55