Stochastic approximation vis-a-vis online learning for big data analytics

Transcription

1 Sochasic approximaion vis-a-vis online learning for big daa analyics Konsaninos Slavakis, Seung-Jun Kim, Gonzalo Maeos, and Georgios B. Giannakis July 30, 2014 We live in an era of daa deluge, where daa ranslae o knowledge and can hus conribue in various direcions if harnessed and processed inelligenly. here is no doub ha signal processing (SP) is of uermos relevance o imely big daa applicaions such as real-ime medical imaging, smar ciies, nework sae visualizaion and anomaly deecion (e.g., in he power grid and he Inerne), healh informaics for personalized reamen, senimen analysis from online social media, web-based adverising, recommendaion sysems, sensor-empowered srucural healh monioring, and e-commerce fraud deecion, jus o name a few. Accordingly, abundan chances unfold o SP researchers and praciioners for fundamenal conribuions in big daa heory and pracice. Wih such big blessings however, come big challenges. he sheer volume and dimensionaliy of daa make i ofen impossible o run analyics and radiional bach inferenial mehods on sandalone processing unis. Wih regards o scalabiliy, online daa processing is well moivaed as he compuaional complexiy of joinly processing he enire daa-se as a bach is prohibiive. Furhermore, here are many applicaions in which daa hemselves are made available in a sreaming fashion, meaning ha smaller chunks of daa are acquired sequenially in ime, e.g., nodes of a large nework ransmiing small blocks of daa o a cenral uni coninuously and incoherenly in ime. As informaion sources unceasingly produce daa in real ime, analyics mus ofen be performed on-he-fly, ypically wihou a chance o revisi previous daa. In addiion, ofenimes big daa asks are subjec o sringen ime consrains, so ha a high-qualiy answer obained slowly via bach echniques can be less useful han a medium-qualiy answer ha is obained fas in an online fashion. RELEVANCE In his conex, his lecure noe presens recen advances in online learning for big daa analyics. I is demonsraed ha many of hese approaches, mosly developed wihin he machine learning discipline, have srong ies wih workhorse saisical SP ools such as sochasic approximaion (SA) and sochasic gradien (SG) algorihms. Imporan differences and novel aspecs are highlighed as well. A key message conveyed is ha e.g., Robbins-Monro s and Widrow s seminal works on SA, ha go back half a cenury, can play insrumenal roles in modern online learning asks for big daa analyics. Consequenly, ample opporuniies arise for he SP communiy o conribue in his growing and inherenly cross-disciplinary field, spanning muliple areas across science and engineering. Work in his paper was suppored by he NSF grans EARS , EAGER , and he MURI Gran AFOSR FA

2 he remainder of his lecure noe, which also serves as a supplemen o [1], is organized as follows. Basic principles of SA are reviewed firs, followed by a couple of examples. Sandard performance merics of SA algorihms are hen oulined, accompanied by a recen wis on performance analysis hrough convex analyic argumens. Sequenial schemes and daa skeching or sampling wih eminen poenial for big daa analyics are also delineaed. Finally, online learning approaches based on he powerful online convex opimizaion (OCO) framework are reviewed, where he links and differences vis-a-vis SA are highlighed. PREREQUISIES he required background includes basics of linear algebra, probabiliy heory, convex analysis, and sochasic opimizaion. SOCHASIC APPROXIMAION BASICS Consider he prooypical saisical learning problem in he realm of sochasic opimizaion (SO) [2, 3] where given a loss funcion f, one aims a minimizing he expeced loss E y { f (w; y)}, possibly augmened wih a complexiy-conrolling convex regularizer r(w), wih respec o (w.r..) a deerminisic parameer (weigh) vecor w W. An example of r(w) is he recenly popular sparsiy-promoing l 1 -norm of he p 1 vecor w where r(w) = w 1 := p i=1 w i. Expecaion E y { } is aken w.r.. he ypically unknown probabiliy disribuion of daa y describing, e.g., inpu-response pairs in a supervised learning seing, and W denoes a subse of some Euclidean space, inroduced here o cover general cases where consrains are imposed on w. In lieu of he aforemenioned disribuional informaion, given raining daa {y } =1 one can insead op for solving he empirical risk minimizaion problem min w W 1 f (w; y ) + r(w) (1) =1 which is an approximaion of is ensemble counerpar, namely min w W [E y { f (w; y)} + r(w)]. Beyond a purely learning paradigm, one should appreciae he generaliy offered by (1), since i can subsume, e.g., (consrained) maximum-likelihood problems wih f idenified as he log-likelihood funcion and daa assumed saisically independen. In big daa seings, can be huge, poenially infinie in a real-ime paradigm where idenifies ime insances of daa acquisiion. Moreover, he search space W can be excessively high-dimensional wih complex srucure. his observaion jusifies he inclusion of a regularizer in (1) o effecively reduce he dimensionaliy and/or size of W and yield parsimonious models ha are inerpreable and have saisfacory predicive performance. Unsurprisingly, here has been growing ineres over he las decade in devising scalable and fas online algorihms for big daa learning asks such as (1). he main premise of SO is cenered around solving he minimizaion ask [cf. (1)] min w R p [ϕ(w) := E y{ f (w; y)}] (2) wihou having E y { } available; see e.g., [3]. (Compared o (1) and is ensemble version, boh W and he regularizer r have been dropped here for breviy.) Key feaures presen in SO algorihms 2

3 are: (i) he daa comprise a sequence of eiher dependen vecors wih (asympoically) vanishing covariance, or, independen idenically disribued (i.i.d.) realizaions {y } =1 of y; and, (ii) given (w, y ), here is a means of obaining an unbiased sochasic gradien esimae f (w; y ), so ha E y { f (w; y )} = ϕ(w). For ϕ smooh, minimizing ϕ in (2) amouns o searching for a zero of Φ(w) := ϕ(w), i.e., a w 0 such ha (s..) Φ(w 0 ) = 0 [3]. he classical Newon-Raphson (N-R) algorihm provides he means o achieve his goal. For w scalar and wih denoing differeniaion, he sequence generaed by he recursion w k+1 := w k Φ(w k )/Φ (w k ) = w k ϕ (w k )/ϕ (w k ) converges under mild condiions o a roo of Φ(w), and hus o a minimizer of ϕ(w). An illusraion of he N-R ieraion can be seen in Figure 1: Newon-Raphson mehod for finding a Fig. 1. Saring from w 1 and using he derivaives {Φ (w k )} + k=1 w 0 s.. Φ(w 0 ) = 0. in he N-R ieraion, he resulan updaes {w k } + k=2 gradually approach w 0, where Φ(w 0 ) = 0. Such a simple recursion can be readily exended o he p 1 vecor case as w k+1 := w k H 1 ϕ (w k ) ϕ(w k ), where now H ϕ (w k ) sands for he p p Hessian marix of ϕ a w k wih (i, j)h enry 2 ϕ(w k )/( w i w j ). Clearly, he N-R algorihm canno be applied if E y { } is no available; e.g., if he probabiliy densiy funcion (pdf) of y is unknown, or, when compuing E y { } enails cumbersome inegraion over high-dimensional domains. o alleviae his burden, SA hrough he celebraed Robbins-Monro algorihm relies on he sequence of realizaions {y } and ingeniously uses he insananeous f (w ; y ) insead of he ensemble ϕ(w k ) (indexes have been changed from k o, for ime-adapive operaion). Wih µ denoing he sep-size, SA generaes he online (or sochasic) gradien descen (OGD) ieraion w +1 = w µ f (w ; y ) (3) which learns expecaions on-he-fly. his poin is beer illusraed by he following example. Online averaging as SA: he soluion of min w E y { w y 2 2 /2} is clearly w 0 = E y {y}. Following he SA raionale, consider f (w; y ) := w y 2 2 /2. he OGD ieraion is w +1 = w µ (w y ), and if w 1 := 0 as well as µ := 1/, simple mahemaical inducion yields w +1 = (1/) τ=1 y τ, which in accordance wih he law of large numbers converges o w 0 = E y {y} as + [3]. Several well-known adapive signal processing and online learning algorihms sem from OGD. LMS as SA: Consider for insance scalar d and vecor x processes which comprise he raining daa colleced in y := [d, x ], and le f (w; y ) := (d w x ) 2 /2, where sands for ransposiion. I can be readily verified ha f (w; y ) = (w x d )x, and applicaion of OGD yields w +1 = w µ (w x d )x, which is nohing bu he celebraed leas mean-squares (LMS) algorihm [3]. RLS as SA: he OGD class can be furher broadened by allowing marix sep-sizes {M } insead of scalar ones {µ } o obain w +1 = w M f (w ; y ). o highligh he poenial of his exension, consider (joinly) wide sense saionary {d, x } =1, wih C xx := E x {x x }, as well as r dx := E d,x {d x }. I urns ou ha he soluion of min w E d,x {(d w x ) 2 } is he linear minimum mean-square error esimaor w 0 = C 1 xx r dx. However, wihou knowing C xx one relies on he sample average esimae 3

4 ĈC := (1/) τ=1 x τ x 1 τ, and on OGD wih M := (1/)ĈC o obain w +1 = w 1 ĈC 1 +1 = ĈC [ĈC 1 x (w x d ) (4a) ] ĈC x +1 x+1ĉc /( + x+1ĉc x +1 ) (4b) where he marix inversion lemma is applied o carry ou efficienly he inversion in (4b). Recursions (4) comprise he well-known recursive leas-squares (RLS) algorihm [3]. PERFORMANCE OF SA ALGORIHMS Based on he samples {y }, SA algorihms produce esimaes {w } ha allow for esimaion, racking, and ou-of-sample inference asks, such as predicion. Performance analysis of SA schemes has leveraged advances in maringale and ordinary differenial equaion heories o esablish, e.g., in he saionary case, convergence of {w } o a ime-invarian w 0 in probabiliy, or wih probabiliy one, or in he mean-square sense [3]. In his saionary seing, convergence of OGD requires sep-sizes seleced o diminish wih a cerain rae. Specifically, {µ } mus saisfy (i) µ 0; (ii) lim µ = 0; and, (iii) =1 µ = +. Clearly, (i)-(iii) are saisfied for µ := 1/, which vanishes as +, bu no oo fas so ha (iii) enables {w } o reach asympoically he desired w 0. Deparing from he sandard roue of SA convergence analysis [3], recen resuls ake advanage of convexiy if i is presen in he objecive funcion. Specifically for convex coss, he OGD recursion (3) generalizes o: w +1 = P W [w µ f (w ; y )], where P W (w) := arg min w W w w 2 sands for he projecion mapping ono a closed and convex consrain se W. For ϕ differeniable and srongly convex wih index c > 0, i holds ha ϕ(w ) ϕ(w) + (w w) ϕ(w) + (c/2) w w 2 2, for all (w, w). Wih sep-sizes seleced as µ := µ/ wih µ > 1/(2c), and for bounded sochasic gradiens as in sup E w y{ f (w; y) 2 2 }, i can be verified ha he error E y{ w w }, where w 0 = arg min w W E y { f (w; y)}, saisfies he following finie-sample bound [2] E y { w w 0 2 2} Q(µ), wih Q(µ) := max { µ 2 2 /(2µc 1), w 1 w 0 2}. If in addiion ϕ is L-Lipschiz coninuous, i.e., ϕ(w) ϕ(w ) 2 L w w 2, w, w, hen a similar finie-sample bound holds also for he sequence of funcion values {ϕ(w )} [2] E y {ϕ(w ) ϕ(w 0 )} LQ(µ) 2 where expecaion is aken over {w } which involves sochasic gradiens. Performance analysis of SA algorihms deals wih convergence of {w }, whereas he online convex opimizaion framework oulined in a subsequen secion sars from (1), invokes less or no assumpions on he underlying pdfs, and assers convergence of he coss { f (w ; y )}, raher han primal variables. Recenly, SA was combined wih he alernaing direcion mehod of mulipliers (ADMM) which is aracive for off-line opimizaion of composie coss [4]. he resulan SA-ADMM solver [5] is suiable for online opimizaion of composie coss such as min w W [E y { f (w; y)} + r(w)], in a fully disribued fashion an operaional mode ha is highly desirable for big daa applicaions. 4

5 SEQUENIAL OPIMIZAION AND DAA SKECHING he imporance of sequenial opimizaion along wih he aracive operaion of random sampling (a.k.a. skeching) big daa will be illusraed in his subsecion in he conex of he familiar LS ask: [ ] 1 min w R p 2 X w d 2 2 = 1 1 =1 2 (x w d ) 2 (5) where X := [x 1,..., x ] denoes he p marix which gahers all available regressor or inpu vecors, and d := [d 1,...,d ] he 1 vecor of desired oupus (responses). Alhough irrelevan o he minimizaion in (5), he normalizaion wih is included o draw connecions wih (1). In his sense, he loss funcion becomes f (w; y ) = (x w d ) 2 /2, wih y := [d, x ], and is gradien f ( ; y ) is Lipschiz coninuous wih consan L = x 2 2. Differen from he previous discussion, here is fixed, and online means processing {d, x } =1 sequenially. Searching for a soluion w 0 of (5) requires eigen-decomposiion of X X, which incurs complexiy O( p 2 ). Alernaively, he sandard gradien descen recursion w k+1 = w k µ k (X X w k X d) enails O(p 2 ) compuaions per ieraion k. Boh cases are prohibiive in big daa seings where he number of samples,, is massive and/or he daa dimensionaliy, p, can be huge. o surmoun hese obsacles, solving for w 0 can rely on sub-sampling (a.k.a. skeching o obain a subse of) he rows of X, along wih he corresponding enries of d, o reduce complexiy w.r.., while visiing hem in a sequenial fashion ha scales linearly wih p. Kaczmarz s algorihm, a special case of he projecions ono convex ses (POCS) mehod [6], produces a sequence of esimaes {w k } o solve (5). For an arbirary iniial esimae w 1, he kh ieraion of Kaczmarz s algorihm selecs a row (k) of X, ogeher wih he corresponding enry d (k), and projecs he curren esimae w k ono he se of all minimizers H (k) := {w x(k) w = d (k) } of f (w; y (k) ), which is nohing bu a hyperplane (a closed and convex se). Hence, he (k + 1)s esimae is w k+1 := P H(k) (w k ) = w k x (k) w k d (k) x (k) 2 x (k) 2 (6) where P H(k) sands for he projecion mapping ono H (k). Noice here ha he complexiy of compuing P H(k) (w k ) scales linearly wih p. If every (d, x ) is visied infiniely ofen, hen under several condiions (6) converges o a soluion Figure 2: Kaczmarz s algorihm for hree hyperplanes {H } =1 3 wih non-empy inersecion {w 0 } = =1 3 H. Row (hyperplane) selecion affecs convergence rae; {w k } which alernaes beween H 1 and H 2 approaches w 0 faser han { w k } which is generaed via H 2,H 3. of (5) [6]. Visiing each (d, x ) a large number of imes is prohibiive wih big daa since can be excessively large. In conras, poor selecion of rows can slow down convergence; see Fig. 2. Neverheless, randomly drawing rows wih equal probabiliies has been shown empirically o accelerae convergence relaive o cyclic revisis of rows [7]. 5

6 Acceleraing SG via non-uniform sampling: In he noiseless case (X w = d), randomly drawing rows in proporion o heir Lipschiz consans L is known o provide finie-sample bounds of he form [7] E R { w k w 0 2 2} [ 1 κ(x) 2] k w1 w where κ(x) sands for he condiion number of X, and E R { } denoes expecaion w.r.. he disribuion over which {d, x } are seleced. he previous non-uniform sampling scheme yields beer convergence raes han hose resuling from uniform skeching [7]. More informaion on (non-)uniform skeching and is applicaion o SG descen mehods can be found in [8, 9]. LEARNING VIA ONLINE CONVEX OPIMIZAION Recenly, online learning approaches based on online convex opimizaion (OCO) framework have araced significan aenion, as hey do no require elaborae saisical models for daa and ye can provide robus performance guaranees. his is rue even under an adversarial seup, where he daa sequence {y } may be generaed sraegically in reacion o he learner s ieraes {w }, as in he humans-in-he-loop applicaions such as he web adverising opimizaion. he OCO framework can be viewed as a muli-round game beween a player (learner) and Figure 3: OCO as a muli-round game. an adversary [10]. In he conex of he learning formulaion in (1), he learner plays an acion w W in round, where W is assumed o be closed and convex. Based on he acion w ha he player ook, he adversary provides some feedback informaion F, manifesed in he daa (feaure) vecor y, based on which a convex loss funcion L : W R {+ } is consruced, such as L (w) := f (w; y ) + r(w). he learner hen suffers he loss a w, namely, L (w ). he overall process is depiced in Fig. 3. he learner s goal is o minimize he so-ermed regre R( ) over rounds, defined as R( ) := =1 L (w ) min w W =1 L (w) (7) which capures how much worse he learner performed cumulaively, compared o he case where a single bes acion is chosen wih he knowledge of he enire sequence of cos funcions {L } =1 in hindsigh. In paricular, OCO aims a producing a sequence {w }, which gives rise o sublinear regre, ha is he one wih R( )/ 0 as grows. Key quesion now for he learner is how o pick w in each round. OCO ALGORIHMS AND PERFORMANCE An imporan class of algorihms ha can achieve he desired sublinear regre bound is based on he online mirror descen (OMD) ieraion [11]. In a nushell, he mehod minimizes a firs-order 6

7 approximaion of L a he curren ierae w, while encouraging he search in he viciniy of w. Specifically, OMD compues he nex round ierae w +1 as w +1 = arg min w W (w w ) L (w ) + 1 µ D ψ(w, w ) (8) where L (w ) is a (sub)gradien of L a w, µ > 0 a learning rae parameer, and D ψ (w, v) is he Bregman divergence associaed wih a coninuously differeniable and srongly convex ψ, defined as D ψ (w, v) := ψ(w) ψ(v) (w v) ψ(v). (9) In he special case of using ψ(w) := w 2 2 /2, he corresponding D ψ(w, v) = w v 2 2 /2, and he OMD updae in (8) boils down o OGD [10], esablishing an immediae link beween OCO and SA. In general, a judicious choice of ψ can capure he srucure of he search space W, leading o an efficien updae formula for w. For example, when W is he probabiliy simplex, i.e., W := {w w i 0, i w i = 1}, seing ψ(w) := i w i logw i in (8) (9) yields he exponeniaed gradien algorihm, which obviaes he need o explicily impose he probabiliy simplex consrains [10]. COMID algorihm: While he OMD updae provides a compuaionally aracive soluion o (1), he linearizaion involved ofen defeas one of he purposes of he regularizer r, which is o promoe a priori known srucure in he soluion. For example, seing r(w) proporional o he l 1 -norm of w encourages sparsiy in w. o properly capure such a benefi, one has o respec he composie srucure of L, which decomposes ino he daa-dependen par f (w) := f (w; y ) and he invarian par r(w) [12, 13]. In paricular, he composie objecive mirror descen (COMID) algorihm relies on [12] w n+1 = arg min w W (w w ) f (w ) + r(w) + 1 µ D ψ(w, w ) (10) where i is seen ha he regularizer is no linearized. Boh COMID and OMD (which is a special case of COMID) can aain sublinear regre bounds. Specifically, R( ) = O( ) in general, and he bound becomes O(log ) when L is srongly convex [10, 12]. SA vis-a-vis OCO: Compared o he SA approaches, he OCO framework does no require sochasic models. his is a salien deparure from ypical SA seups, since he regre bounds are guaraneed even for {y } ha may have been generaed adversarially, i.e., wih y arbirary correlaed o pas acions {w τ } τ and pas daa {y τ } τ<. On he oher hand, he bounds perain o convergence of he sequence of coss raher han he ieraes {w } hemselves. Noneheless, building upon he flexibiliy offered by OCO, cerain limied feedback learning asks are feasible as elaboraed nex, where ineresingly, he SA ideas prove insrumenal once again. ONLINE LEARNING WIH BANDI FEEDBACK he bandi se-up of OCO refers o he case where he feedback F from he adversary does no explicily reveal he cos funcion L ( ), bu only he sample cos L (w ) due o acion w ; refer also o Fig. 3. For example, w may represen he adverising budge allocaed o differen media channels, and L (w ) he corresponding overall cos (e.g., he oal adverising expense minus he resuling 7

8 Figure 4: SA/SO vis-a-vis OCO: Feaures and implicaions. income). In his case, i may be difficul o know he explici form of L, bu L (w ) can be easily observed. he idea of bandi OCO is o esimae he necessary gradien using SA in he conex of OGD. Specifically, a key observaion is ha if one can evaluae a funcion f : R p R a w perurbed by a small δ v, where δ > 0 and v is uniformly disribued on he surface of a uni sphere, hen p δ f (w + δ v)v offers an unbiased esimae of he gradien a w of a locally smoohed version of f [14]. hus, plugging his noisy gradien direcly ino he OGD updae in he spiri of SA, one can sill esablish a sublinear regre bound. However, he bes bound found in [14] is O( 3/4 ), slower han he O( )-bound for he full informaion case, illusraing he price o pay for he lack of informaion. LESSONS LEARNED AND FUURE AVENUES his lecure noe offered a shor exposiion of recen advances in online learning for big daa analyics, highlighing heir differences and many similariies wih prominen saisical SP ools such as sochasic approximaion (SA) and sochasic opimizaion (SO) mehods. I was demonsraed ha he seminal Robbins-Monro algorihm, he workhorse behind several classical SP ools such as he LMS and RLS algorihms, carries rich poenial for solving large-scale learning asks under low compuaional budge. I was also explained ha sequenial or online learning schemes ogeher wih random sampling or daa skeching mehods are expeced o play a principal role in solving large-scale opimizaion asks. A shor descripion of he online convex opimizaion (OCO) framework revealed is flexibiliy on he variey of opimizaion asks ha can be accommodaed, including scenarios where daa are provided in an adversarial fashion, or wih limied feedback. Ye, such a flexibiliy comes a a price; OCO-based saisical analysis refers mosly o bounds of he regre cos. Based on he common ground beween OCO and SA, OCO can only benefi from he rich heoreical armory of SA, e.g., maringale heory, where resuls perain also o convergence of he primal (random) variables of he opimizaion ask a hand. Vice versa, SA can also profi from he powerful oolbox of convex analysis, he engine behind OCO, for esablishing srong analyical claims in he big daa conex. In closing, Fig. 4 depics he unique and complemenary srenghs SA, SO, and OCO offer o online learning, as well as adapive SP heory and big daa applicaions. Auhors: Konsaninos Slavakis (kslavaki@umn.edu) is a Research Assoc. Professor in he Dep. of Elecrical & Compuer Engineering and Digial echnology Cener, Univ. of Minnesoa, MN, USA; Seung-Jun Kim (sjkim@umbc.edu) is an Assis. Professor in he Dep. of Compuer Science 8

9 & Elecrical Engineering, Univ. of Maryland, Balimore Couny, MD, USA; Gonzalo Maeos is an Assis. Professor in he Dep. of Elecrical & Compuer Engineering, Univ. of Rocheser, Rocheser, NY, USA; Georgios B. Giannakis is a Professor in he Dep. of Elecrical & Compuer Engineering and Direcor of he Digial echnology Cener, Univ. of Minnesoa, Minneapolis, MN, USA. References [1] K. Slavakis, G. B. Giannakis, and G. Maeos, Modeling and opimizaion for big daa analyics, IEEE Signal Process. Magaz., vol. 31, Sep [2] A. Nemirovski, A. Judiski, G. Lan, and A. Shapiro, Robus sochasic approximaion approach o sochasic programming, SIAM J. Opim., vol. 19, no. 4, pp , [3] H. J. Kushner and G. G. Yin, Sochasic Approximaion Algorihms and Applicaions. New York: Springer, [4] D. P. Bersekas and J. N. sisiklis, Parallel and Disribued Compuaion: Numerical Mehods, 2nd ed. Ahena Scienific, [5] I. D. Schizas, G. Maeos, and G. B. Giannakis, Disribued LMS for consensus-based innework adapive processing, IEEE rans. Signal Process., vol. 57, no. 6, pp , [6] H. H. Bauschke and J. M. Borwein, On projecion algorihms for solving convex feasibiliy problems, SIAM Review, vol. 38, no. 3, pp , Sep [7]. Srohmer and R. Vershynin, A randomized Kaczmarz algorihm wih exponenial convergence, J. Fourier Anal. Appl., vol. 15, no. 2, pp , [8] D. Needell, N. Srebro, and R. Ward, Sochasic gradien descen and he randomized Kaczmarz algorihm, ArXiv e-prins, Feb [Online]. Available: arxiv: v2 [9] A. Nedić and D. P. Bersekas, Incremenal subgradien mehods for nondiffereniable opimizaion, SIAM J. Opim., vol. 12, pp , [10] S. Shalev-Shwarz, Online learning and online convex opimizaion, Foundaions and rends in Machine Learning, vol. 4, no. 2, pp , Mar [11] A. Beck and M. eboulle, Mirror descen and nonlinear projeced subgradien mehods for convex opimizaion, Operaional Research Leers, vol. 31, pp , [12] J. C. Duchi, S. Shalev-Shwarz, Y. Singer, and A. ewari, Composie objecive mirror descen, in Proc. Inl. Conf. Learning heory, Haifa: Israel, June [13] L. Xiao, Dual averaging mehods for regularized sochasic learning and online opimizaion, J. Machine Learning Research, vol. 11, pp , Oc [14] A. D. Flaxman, A.. Kalai, and H. B. McMahan, Online convex opimizaion in he bandi seing: Gradien descen wihou a gradien, in Proc. ACM-SIAM Symp. Discree Algorihms, Vancouver, Jan. 2005, pp