Online Learning from Experts: Minimax Regret

Transcription

1 E0 370 tatstcal Learnng Theory Lecture 2 Nov 24, 20) Onlne Learnng from Experts: Mn Regret Lecturer: hvan garwal crbe: Nkhl Vdhan Introducton In the last three lectures we have been dscussng the onlne learnng algorthms where we receve the nstance x t and then ts label y t for t =,..., T. pecfcally n the last lecture we talked about onlne learnng from experts and onlne predcton. We saw many algorthms lke Halvng algorthm, Weghted Majorty WM) algorthm and lastly Weghted Majorty Contnuous WMC) algorthm. We also saw bounds on the cumulatve loss ncurred by these algorthms. Today, we wll focus on onlne predcton. For the WMC algorthm the settng s: we have N experts who predct the outcome label) n [0,], then we combne these predctons usng a weghted average of these. Then we receve the true label and ncur some loss absolute loss n our settng), then we make an update to the weght vectors based on the loss. The ntuton s that the hgher the loss ncurred by an expert, the more drastcally we reduce ts weght. For the WMC algorthm, we proved that: [W MCη)] lnn) e η + η mn [ξ ] e η where the symbols have ther meanngs as explaned n the last lecture. lso f the number of examples T s known n advance, then we can choose the WMC parameter η s.t. [W MCη )] mn [ξ ] + 2T lnn) + lnn) 2 Mn Regret Lets defne the regret of an algorthm w.r.t. the set of experts ξ = {ξ,..., ξ N } on the sample = x, y ),..., x T, y T )) as: R l ξ,[] = L l [] mn L l [ξ ].e. the dfference between the total loss of your algorthm and the total loss of the best expert. Thus f we had chosen the best expert always for our predcton then our regret would be zero; ths makes sense as n onlne settng absolute loss s nsgnfcant as the adversary may always choose the label other than what we predcted. The goal of the predctor s to mnmze the regret whle the goal of adversary s to mze t. In general, we can formalze t as a game where at each tral you receve expert s predctons, then you make your own predcton and fnally the true label s revealed by the adversary. In ths settng we want to get close to the best possble loss mnmum regret). Thus we want to choose an algorthm that mnmzes the total regret, however we do not have any control over the set of expert predctons and the sample as adversary can choose a set of experts or sample whch gves a larger regret over the algorthm. Here, we can defne the mn value regret) of the game as: V l = mn ξ, Rl ξ,[] It s the worst case guarantee on the regret over all the sequences of labels and expert predctons. From the last lecture we know that: 2T lnn) + lnn) natural queston here s can we bound the mn regret for other loss functon? Is ths the best possble mn regret?

2 2 Onlne Learnng from Experts: Mn Regret 3 Mn regret for varous loss functons In ths secton we wll look at upper bound on mn regret for some other loss functons whch we have seen before. We wll focus on bnary outcomes y t {0, } wth experts predctng n [0,]; ξ t [0, ] and ŷ t [0, ]. ξ t can be seen as probablty of th expert predctng the label as. Defne the loss functon l : {0, } [0, ] [0, ) Lets us also defne l 0 ŷ) = l0, ŷ) and l ŷ) = l, ŷ). 3. quared Loss For squared loss ly, ŷ) = y ŷ) 2, t can be easly seen that l 0 ŷ) = ŷ 2 and l ŷ) = ŷ) 2. Let us defne some new quanttes assumng l 0 ) and l ) beng twce dfferentable): l ŷ) = l 0ŷ) l ŷ) l ŷ) l 0ŷ) R l ŷ) = l 0ŷ) l ŷ) 2 l ŷ) l 0ŷ) 2 l ŷ) c l = sup R l ŷ) 0<ŷ< Theorem 3.. Haussler et al., 998) Let l be such that l 0 0) = l ) = 0. Let l 0 ) be strctly ncreasng n 0,) and l ) be strctly decreasng n 0,). Let l 0 ) and l ) be three tmes dfferentable n 0,). If l ŷ) > 0 n 0,) and c l <, then: V l = ΘlnN)) pecfcally, we have V l c l lnn) and V l c l o)) lnn) For the squared loss: l 0ŷ) = 2ŷ, l ŷ) = 2ŷ ), l 0ŷ) = 2, l ŷ) = 2 Thus, l ŷ) = 4 > 0, and R l ŷ) = 2ŷ ŷ). Clearly, sup 0<ŷ< R l ŷ) = /2 <. Therefore, for squared loss we have, V lsq 2 lnn) 3.2 Logarthmc Loss Ths type of loss functon haven t been dscussed before n the lectures. Let us defne t frst. { lnŷ) f y = l log y, ŷ) = ln ŷ) f y = 0 For l log loss, l 0 ŷ) = ln ŷ) and l ŷ) = ln ŷ). mple mathematcs wll lead us to: Thus, we get l 0ŷ) = ŷ, l ŷ) = ŷ, l 0ŷ) = ŷ) 2, l ŷ) = ŷ 2 l ŷ) = Therefore for the logarthmc loss we get, ŷ) 2 ŷ 2 and subsequently R lŷ) =, c l = V l log lnn)

3 Onlne Learnng from Experts: Mn Regret bsolute Loss For the absolute loss we have l abs y, ŷ) = y ŷ, here t s easy to see that l 0 ŷ) = ŷ and l ŷ) = ŷ, for ŷ 0, ). lso l 0ŷ) =, l ŷ) =, l 0ŷ) = 0, l ŷ) = 0 nd therefore we get l ŷ) = 0 c l =. Thus, the above theorem doesn t apply n the absolute loss settng. Thus for the absolute loss we have the result of WMC algorthm from the last lecture whch gves an upper bound on the regret. Clearly for the squared and logarthmc loss we saw a much tghter bound; then a natural queston arses: can we do better for the absolute loss too? It turns out that we can only mprove the constants n the WMC regret bound for the absolute error case. Theorem 3.2. Let l be s.t. l 0 ) s strctly ncreasng n 0,) whle l ) s strctly decreasng n 0,) and both beng three tmes dfferentable n 0,). Then, If l ŷ) = 0 for some ŷ 0, ), then V = Ω T /6 ) lnn) T ) If l ŷ) < 0 or a < b, s.t. l ŷ) = 0 ŷ [a, b], then V = Ω lnn) Note. Clearly, l abs falls n the second category of the above theorem. T ) T ) = O lnn). Thus, = Θ lnn) []. nce, we have already seen that 4 Vovk s lgorthm, 990 Ths algorthm s very smlar to the Weghted Majorty algorthm. The dfference les n the update,.e. how do we combne the predctons from the experts to make our own predctons. lgorthm Vovk Parameters: c, η > 0 Intal weght vector w = [N] Loss functon l : {0, } [0, ] [0, ) For t =,..., T : Receve expert predctons ξ, t..., ξn t )[0, ] Compute y = c ln for y = 0,. wt e ηly,ξt ) wt Predct any ŷ satsfyng: ly, ŷ t ) y y {0, }. If such a ŷ, then the algorthm fals. Receve true label y t {0, } Incur loss ly t, ŷ) Update [N] : w t+ w t e ηlyt,ξ t ) nalyss of Vovk s lgorthm uppose that the algorthm doesn t fals. Defne U t = c lnw t ), where W t = N = wt. Then for each tral t, ly t, ŷ t ) U t+ U t t y ummng over t =,..., T, Now, L l [Vovkc, η)] U T + U T U T + = c lnw T + ) c lnw T + ), W T + w T +

4 4 Onlne Learnng from Experts: Mn Regret lso, U = c lnn) Thus, ) L l [Vovkc, η)] c lnn) + η mn L l [ξ ] Note. The above s true for all for whch the algorthm wth parameters c, η) doesn t fal. Thus gven the loss functon of nterest, f one can fnd the parameters c and η such that the algorthm never fals, then we get the above bound for all possble sequence. The proof of above nvolves takng c = c l, η = /c l wth l beng c, η)-realzable. howng n general that any loss functon satsfyng these condtons s realzable w.r.t. these values nvolves a sgnfcant amount of work. Defnton. Defne l to be c, η)-realzable f, ξ; Vovkc, η) doesn t fal. Now, we wll see what condtons are suffcent for the Vovk s algorthm not to fal. We wll verfy t for some loss functons. Logarthmc Loss Run Vovk algorthm wth c = c l and η = /c l. For the l log loss we have, c =, η =. Defne p t = w t N j= wt j to be the normalzed verson of the weght vector. ) ) 0 = c ln p t e ηl0ξt ) = ln p t e ln ξt ) = ln p t ξ t) We bascally need: whch s same as requrng: ) ) = c ln p t e ηlξt ) = ln p t e lnξt ) = ln p t ξ t) l 0 ŷ t ) ln p t ξ t) l ŷ t ) ln p t ξ t) l 0 ln pt ξ t)) ŷ t ŷ p t ξ t ln pt ξ t)) ŷ t ŷ p t ξ t l Thus there s only one value ŷ t = p t ξ t satsfyng t. Exercse. Work out the condtons for the Vovk s algorthm to be c, η)-realzable n the squared loss settng. bsolute Loss We start wth some η > 0 and c abs η) = 2. 2 ln +e η For the above choce of c, l abs turns out to be realzable for any η > 0. The bound that we get here s [Vovkη, c absη))] lnn) + η mn [ξ ] 2 2 ln +e η ) The above holds for any η > 0. The bound seems qute smlar to the one n Weghted Majorty algorthm except for the denomnator term. nce, t holds for all η, f we choose η carefully, we can get a bound of the form [Vovkη, c abs η ))] T lnn)+lnn)/2. For the absolute loss the best possble bound happens to be T/2) lnn + ) + lnn + )/2, whch s obtaned when we add an addtonal arbtrary expert ξ N+, who always predct the nverse of one of the experts say ξ ). Here, at least one of the two experts ξ and ξ N+ ) must have loss at most T/2 tmes.

5 Onlne Learnng from Experts: Mn Regret 5 5 Lower Bound on Mn Regret for bsolute Loss From earler secton we know that, = mn ξ, [] mn ) [ξ ] mn ξ, Rl abs ξ, [] Clam. For any and any ξ,..., ξ N [0, ] T some label sequence for whch the regret follows: [0,] T R l abs ξ, [] T 2 E [0,] T [mn ] [ξ ] Proof. t each tral the algorthm s predctng some value ŷ t. The label for the tral s unformly ether 0 or,.e. w.p. 2 the label s 0 and w.p. 2 t s. Thus, [ ] [ E [0,] T = 2 ŷt 0) + ] 2 ŷt ) T = T/2 The clam follows by observng that, [ ] R l abs [0,] T ξ, [] E R l abs [0,] T ξ, [] Now, = mn mn mn mn ξ { R l abs ξ {0,} T ) N E ξ {0,} T ) N ξ, [] } { } R l abs ξ, [] [ T [ 2 E [0,] T [ T 2 E ξ {0,} T ) N, [0,] T [ mn mn ] ] [ξ ] ] ] [ξ ] 6 Next Lecture In the next lecture we wll study the connectons of the onlne learnng wth the batch learnng and also see some methods to transform the onlne learnng problem nto a batch learnng problem. References [] Ncolo Cesa-Banch, Yoav Freund, Davd P. Helmbold, Davd Haussler, Robert E. chapre and Manfred K. Warmuth. How to use expert advce. Journal of the CM, 443): , 997. [2] Davd Haussler, Jyrk Kvnen and Manfred K. Warmuth. equental predcton of ndvdual sequences under general loss functons. IEEE Transactons on Informaton Theory, 445): , 998.