Sequential Learning for Optimal Monitoring of Multi-channel Wireless Networks

Transcription

1 Sequetial Learig for Optimal Moitorig of Multi-chael Wireless Networs Pallavi Arora Csaba Szepesvári Rog Zheg Departmet of Computer Sciece Departmet of Computig Sciece Departmet of Computer Sciece Uiversity of Housto Uiversity of Alberta Uiversity of Housto Housto, TX 7704, USA Edmoto, Alberta, Caada Housto, Tx 7704, USA Abstract We cosider the problem of optimally assigig p siffers to K chaels to moitor the trasmissio activities i a multi-chael wireless etwor The activity of users is iitially uow to the siffers ad is to be leared alog with chael assigmet decisios while maximizig the beifits of this assigmet, resultig i the fudametal trade-off betwee exploratio versus exploitatio We formulate it as the liear partial moitorig problem, a super-class of multi-armed badits As the umber of arms siffer-chael assigmets is expoetial, ovel techiques are called for, to allow efficiet learig We use the liear badit model to capture the depedecy amogst the arms ad develop two policies that tae advatage of this depedecy Both policies ejoy logarithmic regret boud of timeslots with a term that is sub-liear i the umber of arms I INTRODUCTION Deploymet ad maagemet of wireless devices ad etwors are ofte hampered by the poor visibility of PHY ad MAC characteristics, ad complex iteractios at various layers of the protocol stac both iside a maaged etwor ad across multiple admiistrative domais The ca you hear me Verizo wireless TV commercial is a vivid demostratio of the shortage of real-time owledge that cellular providers have regardig the coditio of operatioal etwors Accurate ad timely estimates of etwor coditios ad performace characteristics ca yield to better performace i a umber of applicatios, icludig the followig: Networ resource maagemet: Wireless service providers ad etwor admiistrators eed to determie the coverage of their ow etwors ad mae critical decisios such as dimesioig ad allocatio of etwor resources Wireless advisory: Idividual devices ca better adapt their operatioal parameters eg, chaels, sub-carriers, hoppig sequeces, trasmissio power levels, etc for co-existece ad better performace Trouble shootig ad diagosis: Availability of crosslayer iformatio of the operatioal etwor ca help etwor admiistrators to determie the root causes of service outage or performace degradatio as well as idetify malicious behavior ad itrusio Passive moitorig is a techique where a dedicated set of hardware devices, called siffers, are used to moitor activities i wireless etwors These devices capture trasmissios of wireless devices or activities of iterferece sources i their viciity, ad store pacet level or PHY layer iformatio i trace files, which ca be aalyzed distributively or at a cetral locatio A caoical moitorig applicatio has three compoets: 1 siffer hardware, siffer coordiator ad data collector, ad 3 data processor ad mier Sice most, if ot all ifrastructure etwors utilize multiple cotiguous or o-cotiguous chaels or bads 1, a importat issue is to determie which set of frequecy bads each siffer should operate o to maximize the total amout of iformatio gathered This is called the siffer-chael assigmet problem or chael assigmet problem for short It is a challegig problem for two reasos First, moitorig resources are limited, ad thus it is ifeasible to moitor all chaels at all locatios at all times Secod, itelliget chael assigmet requires the owledge of usage patters, ie, the lielihood of occurrece of iterestig evets These are of course ot ow a priori A iterestig trade-off arises betwee assigig siffers to chaels ow to be the busiest based o curret owledge, versus explorig chaels that are udersampled Siffer-chael assigmet with o prior owledge of user activity is closely related to the multiarmed badit problem MAB [1] I a MAB, a gambler must decide which arm of N o-idetical slot machies to play i a sequece of trials so as to maximize his payoff I the sifferchael assigmet problem, each of the p siffers must be assiged to oe of the K o-idetical chaels to moitor so as to maximize the total iformatio gathered The umber of choices arms available i a roud is thus N = K p I this wor we assume that the payoff is proportioal to the umber of distict users detected For simplicity, we assume that a user s activity i a give chael ca be described with a sequece of IID Beroulli radom variables However, as opposed to the stadard MAB problem, the observatio upo a sigle assigmet is ot oly the reward associated with the assigmet, but also the activity patters observed at each moitored chael Note that the observed patter may have correlated compoets, eg whe two siffers observe the trasmissio of the same set of users A policy for siffer-chael assigmet determies at ay poit i time the assigmet to be chose based o past 1 A chael ca be a sigle frequecy bad, a code i a CDMA system, or a hoppig sequece i a frequecy-hoppig system

2 iformatio The efficiecy of differet policies is measured i terms of their associated regret, which is defied as the differece betwee the expected payoff gaied by a geie a uattaiable ideal who always uses the optimal statioary siffer-chael assigmet, ad that obtaied by the give policy The regret achieved by a policy ca be evaluated i terms of its growth over time ad how it scales with respect to the various problem parameters A aive approach to the chael assigmet problem would be to treat each sifferchael combiatio as a arm actio, ad lear the statistics of each arm idividually With p siffers ad K chaels i the etwor, the statistics of a total of K p arm-payoffs eeds to be leared Direct applicatio of ow approaches to MAB eg, UCB[], ɛ-greedy[3] results i a regret boud liear i the umber of arms K p I this paper, we formulate the optimal chael assigmet problem as a multi-aget multi-arm partial iformatio problem with liearly parameterized payoff Our proposed policies are cetralized ad slotted i ature, amely, a fusio ceter collects the iformatio from each siffer i each slot ad mae decisio regardig the chael assigmet for the ext slot Utilizig the depedecy amog arms, we reduce the uow parameter space to K p We devise two order-optimal policies both havig a regret boud that grows logarithmically with respect to time with a associated costat that grows sub-liearly i the umber of arms The ey improvemet compared to the aive approach comes from the cocept of spaer arms, ie, a small collectio of arms which provide iformatio about all parameters The policies ad regret bouds are derived for geeral correlatio structures amog the siffers, ad remai valid for special cases where the siffer s observatios are idetical or idepedet I both cases methods exist to idetify the best arm to play i each slot which are liear i the umber of arms ad expoetial i the umber of siffers The rest of the paper is orgaized as follows I Sectio II, related wor o wireless moitorig ad sequetial learig is summarized We preset the problem formulatio i Sectio III Details ad aalysis of the two policies are provided i Sectio IV ad Sectio V, respectively Simulatio results are preseted i Sectio VI, followed by coclusio ad a list of future wor i Sectio VII II RELATED WORK Wireless moitorig is a active area of research that has received much attetio from several perspectives There has bee much wor doe o wireless moitorig from a systemlevel viewpoit, i a attempt to desig complete systems, ad address the iteractios amog the compoets of such systems [4], [5], [6], [8], [9] The authors of these wors have argued both qualitatively ad quatitatively the eed for moitorig o the wireless side To determie the optimal allocatio of moitorig resources to maximize captured iformatio remais, i [10], where Shi ad Bagchi cosider the selectio of moitorig odes ad their associated chaels for moitorig wireless mesh etwors The optimal moitorig is formulated as a maximum coverage problem with group budget costraits, which was previously studied by Cheuri ad Kumar i [11] I [1], we itroduced a quality of moitorig QoM metric defied by the expected umber of active users moitored, ad ivestigated the problem of maximizig QoM by judiciously assigig siffers to chaels based o owledge of user activities i a multi-chael wireless etwor Two capture models are cosidered The first oe, called the user-cetric model assumes frame-level capturig capability of siffers such that the activities of differet users ca be distiguished The secod oe, called the siffer-cetric model utilizes biary chael iformatio olyactive or ot at a siffer The above wors assume that certai statistics regardig the users activity are give [10], [11] or ca be iferred [1] Whe such statistics are ot ow a priori, sequetial learig is eeded Sequetial decisio maig i presece of ucertaity, faces the fudametal tradeoff betwee exploratio ad exploitatio O oe had, it is desirable to put siffers to the chaels where most activities have bee observed ad thus more iformatio is liely to be gathered exploitatio O the other had, explorig the chaels that are uder-sampled helps to reduce ucertaity ad thus avoid beig misled by imprecise iformatio Such tradeoffs are vividly illustrated by the famous multi-armed badit problem MAB A large volume of wor has bee bee devoted to desigig good strategies for variatios of the MAB problem ad to the uderstadig of the theoretical limits of such procedures, amog which, just to ame a few, Lai ad Robbi [] established logarithmic upper ad lower bouds for det stochastic arms with parametric payoff distributios; Agrawal [13] cosidered a class of sample-mea based policies for the same settig; Auer et al aalyzed upper cofidece boud UCB based ad ɛ-greedy policies for o-parametric stochastic badit problems [3] Recetly, badit problems with liear parameterized payoff are cosidered i [15], [7] Regret miimizatio uder partial moitorig is ivestigated i [16], where the player i a repeated game, istead of observig the actio chose by the oppoet i each game roud, receives a feedbac geerated by the combied choice of the two players Recogizig the coectio betwee the MAB ad spectrum access i cogitive radio etwors, Lai et al applied the UCB1 algorithm [3] to sigle user-chael selectio i [17], ad later exteded it to cosider Marovia payoffs ad for the case of multiple users i [18] Liu ad Zhao [19] formulated the problem of secodary user chael selectio as a decetralized multi-armed badit problem, ad preseted a policy that achieves asymptotically logarithmic regret i time Aadumar [0] proposed two policies for distributed learig ad access with order-optimal cogitive system throughput uder self play I additio to learig the chael availability, the secod users also lear the other users strategies ad the umber of total users i the system through chael feedbac Existig wor applyig MAB i the cogitive radio cotext assumes idetical chael view with the exceptio of Gai et al [1] However, the model cosidered i this wor, i fact

3 3 maes the implicit assumptio that all secodary users are colocated if there are multiple users o the chael, the we assume that, due to iterferece, at most oe of the coflictig users gets reward Sice co-located secodary users liely observe idetical primary user activities, a cotradictio arises to the claim of allowig the reward process o the same chael to be differet [1] I cotrast to existig wor, we cosider a model where siffers are i geeral cofiguratio ad may observe differet sets of users i the same chael This ecompasses models whe either siffers are co-located or whe they are sufficietly far apart The algorithms ad aalytical bouds devised are directly applicable to the specialized cases Admittedly, due to its geerality, the model suffers from a higher computatio ad storage complexity Ufortuately, this uavoidable as a result of the NP-hardess of the omial resource allocatio problem whe all statistics are ow as show i Sectio III III PROBLEM FORMULATION Cosider p siffers moitorig user activities i K chaels A user u operates i oe of K chaels, cu K = {1,, K} Let p u deote the trasmissio probability of user u We represet the relatioship betwee users ad siffers usig a udirected bi-partite graph G = S, U, E, where S = {1,, p} is the set of siffer odes ad U is the set of users A edge e = s, u exists betwee siffer s S ad user u U if u is withi the receptio rage of siffer s If trasmissios from a user caot be captured by ay siffer, the user is excluded from G For every vertex v S U, we let Nv deote vertex v s eighbors i G For users, their eighbors are siffers, ad vice versa We assume that oe siffer ca observe oe user at a time This is cosistet with may existig multiple access mechaisms icludig FDMA, TDMA At ay poit i time, a siffer ca oly observe trasmissios over a sigle chael We will cosider chael assigmets of siffers to chaels, = 1,, p, where 1 i K Let K = { : S {1,, K} p } be the set of all possible assigmets The set of users a siffer s ca observe is give by Ns { u : cu = s } A Optimal chael assigmet i the omial form We first cosider the formulatio of the optimal sifferchael assigmet where the graph G ad the user-activity probabilities p u ; u U are both ow The discussio serves two purposes First, optimal chael assigmet with ucertaity is iheretly harder tha that without ucertaity Therefore, determiig the complexity of the later provides a baselie uderstadig of the computatioal aspect of the former problem Secod, as will become clearer, each istace of the decisio problem alog the sequetial learig process ca i fact be cast as the optimal chael assigmet with ow parameters, where the ow parameters i this case are i fact the best estimates of these parameters plus some margis due to isufficiet samples The objective of optimal chael assigmet is to maximize the expected umber of active users moitored Let MAX- EFFORT-COVER MEC deote the problem of fidig the largest weight set of users that ca be moitored by a set of siffers, where each siffer ca moitor oe of a set of chaels Note that i MEC, the weights ca i fact be ay o-egative values ad are ot limited to [0, 1] The MEC problem ca be cast as the followig iteger program IP: max u U p uy u st K =1 z s, 1 s S y u s Nu z s,cu u U 1 y u, z s, {0, 1} u, s, Each siffer is associated with a set of biary decisio variables, z s, = 1 if the siffer is assiged to chael ; 0, otherwise Further, y u is a biary variable but ot a decisio variable idicatig whether or ot user u is moitored, ad p u is the weight associated with user u We have prove that MEC is NP-hard i [1]: Theorem 1 Theorem 1[1]: The MEC problem is NP-hard with respect to the umber of siffers, eve for K = I other words, the computatioal complexity for a geie to mae the optimal choice with the owledge of all users activity grows at least expoetially with respect to the umber of siffers, uless P = NP However, whe the graphs G have some specific structure, there may exist efficiet algorithms For example, whe G is restricted to be a complete bipartite graph, it ca be show that MEC reduces to maximum matchig i a trasformed bipartite graph, which ca be solved i polyomial time B Liear badit for optimal chael assigmet with ucertaity Now, we tur to the optimal chael assigmet whe there is ucertaity i both G ad p u s We first defie the structure of istataeous feedbac ad payoff of each siffer Let U i t be a oegative, iteger-valued radom variable which deotes the idex of the user whose activity siffer i ca observe i chael at time t, or which taes the value of zero if there is o activity i the chose chael For simplicity, we assume that Ut = U i t; 1 i p, 1 K is a sequece of IID radom variables The istataeous feedbac observatios received uder the joit actio t = 1,, p is Y t = U 1,, p 1, 1 t, U, t,, U p,p t Note that the idicator I {Ui1, i1 t=u i, i t==u is,is t>0} is a fuctio of Y 1,, p t ad hece ca be tae as part of the observatio Y 1,, pt, defied as the collectio [ ] I {Ui1, t=u i1 i, t==u i is,is t>0}; 1 s p, 1 i 1 < < i s p Note that spatial multiplexig is allowed such that multiple users ca be active at the same time i oe chael as log as they are sufficietly far apart geographically However, we assume oe user ca be observed by oe siffer at a

4 4 time This is cosistet with may existig multiple access mechaisms icludig FDMA, TDMA As i Sectio III-A, the payoff upo selectig the joit actio is the umber of distict users observed That is, the joit payoff for selectig chaels = 1,,, p is X t = {U 1,1 t,, U p,p t} I {U1,1 t=0,,u p,p t=0} = p i=1 I {U 1,i t>0} p i,j=1 I {U i,i t=u j,j t>0}i {i= j,i j} 1 p I {U1,1 t=u, t==u p,p t>0} I {1= == p} 3 The expected payoff for chaels = 1,,, p is give by, E [X t] = p i=1 P U 1, i t > 1 p i,j=1 P U i,i t = U j,j t > 0 I {i= j,i j} 1 p P U 1,1 t = = U p,p t > 0 I {1= == p} Defie a uow vector θ with the followig elemets: P U i, > 0, 1 i p, 1 K, P U i1, = U i, > 0, 1 i 1 < i p, 1 K, P U 1, = U, = = U p, > 0, 1 K 5 We itroduce the arm features, φ R M as 6, where M = K p 1 Note that the jth arm feature φ,j is uiquely determied by the arm = 1,,, p Let M = { i : 1 i M, φ,i 0 } be the set of ozero compoets of feature vector φ ad let M = M To this ed, we ca rewrite the expected payoff as a liear fuctio of the arm feature φ, 4 E [X t] = θ T φ, 7 where T deotes traspositio Kowig θ suffices to play optimally: A arm with maximal payoff is give by = argmax K θ φ here, ad i what follows, for the sae of simplicity, we assume that there is a uique optimal arm A reasoable way to estimate the parameter vector θ is to eep a ruig average for the compoets of θ If at time t the aget chose t K the the curret estimate, ˆθt 1, ca be updated by ˆθ i t = ˆθ i t Y i t N i t ˆθ i t 1 I {i Mt }, N i t = N i t 1 + I {i Mt } 8 Here N i 0 = 0, ˆθ i 0 = 0 Thus, N i t couts the umber of times data for compoet i was observed up to time t Example 1 Co-located siffers: Whe the siffers are co-located or are deployed at close proximity, their observatios are idetical Therefore, Ut will be such that if i = j the U i,i t = U j,j t The, the expected payoff is maximized by puttig differet siffers to differet chaels, ie, i j, 1 i < j p It ca be proved that it is strictly better to put differet siffers to differet chaels I this case it suffices to estimate P U i > 0, ie, a total of K p parameters The problem the becomes essetially the multi-armed badit problem with multiple plays cosidered i a umber of previous wors [3], [19], [0] Example Idepedet siffers: The opposite case is whe U i,i t U j,j t wheever i j ad whe oe of U i,i t ad U j,j t is ozero I words, all siffers are guarateed to observe distict users eg, they are far away from oe aother The, I {Ui1, i1 =U i, i ==U is,is >0} = 0, s p, 1 i 1 <,, < i s p Therefore, the umber of parameters are reduced to K p ad each siffer ca decide idepedetly which chael to moitor Thus the, problem reduces to p idepedet K-arm badit problems I practice, siffers are deployed distributedly Their observatios are typically correlated but o-idetical This motives us to cosider the optimal chael assigmet i geeral cofiguratios A optimal moitorig policy π determies a sequece of actios i K over time such that the expected regret is miimized: [ { } ] R π = E max A φt θ φ T t θ t=1 Here, t deotes the joit actio selected at time t C Relatioships betwee p u ad θ Theorem states the oe-to-oe mappig betwee p u ad θ with G properly defied As such, we ca apply optimizatio solutios to 1 to determie the best arm to play at each istace Theorem : Let p be the vector deotig the user-activity probabilities Uder some mild o-limitig coditios, there exists a full ra matrix A such that log1 θ = log1 p A Proof: See Appedix A D Spaers Sice some arms reveal iformatio about other arms, it might be possible to idetify a restricted set E K, which might be much smaller tha K, so that playig oly arms i E gives sufficiet iformatio to idetify the optimal arm A sufficiet coditio for this is that E M = {1,, M} This coditio esures that by choosig a appropriate arm i E ay compoet of Xt ca be observed, which is clearly sufficiet to idetify θ Sice exploratio is geerally costly, the set E is ideally chose to be small I the moitorig problem E ca be chose to be E = {,, : 1 K }, ie all the siffers assiged to the same chael to cover p 1 parameters, whose cardiality is K K p = K The set E is called a spaig set or a spaer ad its elemets are called spaer arms Cloc sychroizatio amog siffers ca be achieved olie or offlie usig methods such as i []

5 5 I {1=i}, if 1 i K; I {=i l K}, if l K + 1 i l + 1 K; φ,i = I {1= =i p K}, if p K + 1 i p + 1 K; 1 p I {1= == p=i K p }, if K p + 1 i K p 1 6 IV AN UPPER CONFIDENCE BOUND UCB-BASED POLICY The first policy that we cosider is similar to UCB1 [3] with the differece that i the iitializatio stage, we oly play each of the spaers oce Formally, the algorithm first plays each arm i E oce ad the at time t E + 1 chooses where t = argmax V t 1, E V t 1 = ˆµ t 1 + ˆµ t 1 = ˆθt 1 φ i M ρ log t N i t 1, After playig t ad observig Y i t; i M t the parameter estimate is updated usig 8 The, the process is repeated Theorem 3: Choose ay ρ that satisfies ρ > 1/199 The, there exists a costat C > 0 which may deped o ρ such that for all 1, the expected regret of UCB1 satisfies R UCB1 4M max max : >0 M ρ log + C, where max = max The exact depedece of C o the problem parameters ca be extracted from the proof I particular, C scales liearly with K Proof: The proof is similar to the origial proof that give by Auer et al[3], with some elemets borrowed from the aalysis techique of Audibert et al[4] see also, [5] ad Gai et al [1] We start by itroducig the ecessary otatio We deote by T the umber of times arm is chose up to time icludig time : T = t=1 I {t=} We let µ = max µ, = µ µ The, it is easy see that E [ ] R UCB1 = E [T ] max E [ : >0 T ] Our goal is to develop a boud o E [ : >0 T ] which scales liearly with M rather that with K Let It = argmi j Mt N j t 1 ties ca be broe, say, i favor of the smallest idex, Z i t = I {t,it=i}, T i t = T i t 1 + Z i t 3 Note that T = 3 We are usig the assumptio that there is a uique optimal arm Note that this is assumed just for the sae of simplicity ad the proof, at the price of a more complicated presetatio, wors without it T i i, sice exactly oe of the couters is icremeted o both sides whe a suboptimal arm is chose Thus, it suffices to boud T i Therefore pic ay idex 1 i M ad let u be a iteger to be chose later We have Z i t = Z i ti { Tit 1>u} + Z i ti { Tit 1 u} Sice t=1 Z iti { Tit 1 u} u + 1, it suffices to deal with the first term, which we boud as follows: Z i ti { Tit 1>u} I {Vt t 1>µ, T it 1>u,It=i} + I {V t 1 µ } Thus, [ ] E Ti u P V t t 1 > µ, T i t 1 > u, It = i + t=1 P V t 1 µ t=1 We will ow show that both sums are fiite, provided that u is sufficietly large The summad of the first sum is bouded as follows: def p 1t = P V t t 1 > µ, T i t 1 > u, It = i { P ˆµ t t 1 > µ t + t c t,t 1, } T i t 1 > u, It = i where c,t 1 = ρ log t W t 1 Now, ρ log t i M 1 N it 1 c,t 1 = W t 1 1 ρ log t N i t 1 i M We claim that uder the coditio that TIt t 1 > u the largest value W t t 1 ca tae is bouded from above by M t / u To see this ote that T i t 1 N i t 1 holds for ay i ad t, because N i is always icremeted whe T i is icremeted Further, sice It = argmi j Mt N j t 1, N It t 1 N j t 1 holds for ay j M t Thus, for arbitrary j M t, u < T It t 1 N It t 1 N j t 1 The claim the follows from the defiitio of W t t 1 def =

6 6 Hece, t c t,t 1 t u i M t M t 1 ρ log t N i t 1 Further, t u M t ρ log t ρ log holds for 1 t if u max : >0 M ρ log The, t c t,t 1 ρ log W t t 1 ad thus p 1t P ˆµ t t 1 > µ t + ρ log W t t 1 M 4 log exp 199ρ log, where the last iequality follows from the uio boud ad Lemma 6, which is preseted i Appedix B Thus, t=1 p 1t M 4 log 1 199ρ Hece, if ρ is such that 199ρ > 1, we get that t=1 p 1t = o1 ote that p 1t does deped o through u, although this depedece was chose ot to be show i the otatio Usig Lemma 6 agai, we get that p t = P V t 1 µ M 4 log t 199ρ Hece, t=1 p t M 4 log 199ρ 1 1, agai, uder the assumptio that 199ρ > 1 Puttig together the iequalities obtaied we get the desired result Note that i liear badit problems there exist similar regret bouds, see [6], [7] The problem depedet boud developed i [7] taes the form M / mi log 3, ie, it is i geeral icomparable to our boud: Our boud scales better as a fuctio of ad M whe max M is small However, the scalig of our boud as a fuctio of mi = mi : >0 is worse I geeral, oe expects the algorithm preseted here to perform better tha the oes developed for the liear badit problem sice those algorithms do ot tae advatage of the potetially richer feedbac However, this remais to be prove Our result is more directly comparable to that of Gai et al [1] I fact, their problem is a special case of the problem studied here whe we allow arbitrary φ { 1, 0, 1} M The scalig behavior of our boud for their problem is essetially the same as a fuctio of ad mi after boudig max : >0 M / by max M / mi but our boud scales better as a fuctio of max M their boud scales with max M 3, whereas ours scales with max M Note that i the proof o attempt was made to optimize the costats The major issue with this algorithm is that apart from the iitializatio phase i its exploratio it does ot tae full advatage of the correlatios betwee the payoffs of the arms, at least whe it is explorig Oe idea to overcome the algorithm s potetial isesitivity to the correlatio structure is to modify the algorithm so that the arms i E are explored with uiform probabilities i the explicit exploratio steps, ie, whe t argmax ˆµ t 1 We cojecture that this algorithm ideed overcomes the above metioed hadicap, ie, its regret would scale with E ad ot with the umber of the parameters I the ext sectio we explore a similar idea i the cotext of a simpler algorithm, ε-greedy V AN ɛ-greedy ALGORITHM The policy cosidered here is a variat of ε-greedy The stadard ε-greedy algorithm for badit problems chooses with probability ε uiformly at radom some arm ie, it explores with probability ε ad it chooses the arm with the highest estimated payoff otherwise Whe ε is appropriately scheduled basically, oe eeds ε = ε = c/ with a appropriately selected costat c > 0 this policy ca also achieve a logarithmically bouded expected regret just lie UCB1 [3] Sice i our case the arms are correlated ad whe a arm is chose oe receives some additioal iformatio i additio to the payoffs, oe may restrict the set of arms explored to a spaer E We expect that performace will improve if E K sice the oe pays less for the exploratio steps Formally, the algorithm wors as follows: Choose a spaer E K ad a sequece ε t ; t 1, ε t [0, 1] I the iitializatio phase explore each arm i E oce ad iitialize the parameter estimates ˆθ based o the iformatio received After the exploratio phase, at time t E, the arm to be played is decided by first drawig a radom umber U t from the uiform distributio over [0, 1] If U t ε t the t is chose uiformly at radom from E Otherwise, t = argmax ˆµ t 1, where ˆµ t 1 = ˆθt 1 φ After playig t ad observig the feedbac, the parameters are updated usig 8 The ext theorem gives a boud o the regret of this policy: Theorem 4: Let { ɛ = mi 1, c }, > E, 9 where c > 0 is a tuig parameter The, assumig that c > mi10 E, 4 E d, where d = mi : >0, the expected regret of ε-greedy satisfies E [ R ε greedy ] c log O1 10 From the poit of view of miimizig the leadig term, the best choice of c is mi10 E, 4 E d With such a choice, we see that the leadig term of regret scales liearly with E, ad ot with K This is the mai differece betwee the boud i this theorem ad i the previous result This ca be a major advatage whe E K eg, i the moitorig problem The disadvatage of this algorithm is that i practice tuig c might be difficult, sice, typically, d is uow Oe remedy the is to replace c with a slowly growig sequece c eg, log log c = log log, ie, use ε = mi1, This would result i a regret that grows i the order of c log, but the proof of this result is omitted for brevity Proof: The proof follows the steps of the proof i [3] with some modificatios ad slight improvemets We will use the otatio itroduced i the proof of Theorem 3

7 7 Without the loss of geerality, we may assume that ε = 0 if E ote that the algorithm does ot deped o the values of ε 1,, ε E ad this assumptio allows us to shorte the proof Clearly, it suffices to boud E [T ] For this purpose we will boud P =, where is ay suboptimal actio For > E, the probability of choosig is bouded by P = ɛ I { E} E + 1 ɛ P ˆµ 1 ˆµ 1 We have P ˆµ 1 ˆµ 1 P + P ˆµ 1 µ + ˆµ 1 µ We boud the first term as follows: Defie δ = M The, P ˆµ 1 µ + ˆθ i 1φ,i θ i φ,i + δ i M P 1 Pic i M Defie x 0 = 1 E t=1 ɛ t By Lemma 7, P ˆθ i 1φ,i θ i φ,i + δ P N i 1 x 0 + δ exp x 0 δ Let us ow boud the first term of the righthad side Let e E be such that i M e Let Ni R be the umber of times e was selected up to time i a exploratio step: Ni R = t=1 I {t= e,u t ε t} Clearly, Ni R 1 N i 1 Hece, P N i 1 x 0 P Ni R 1 x 0 Furthermore, E [ Ni R 1] = 1 1 t=1 ɛ t = x 0, ad Var[Ni R 1 1] E Berstei s iequality for details see [3], we have E 1 t=1 ɛ t = x 0 Therefore, by P Ni R 1 x 0 e x 0/5 11 Sice x 0 = 1 1 E t=1 ɛ t c E log, we have P N R i 1 x 0 e x 0/5 Thus, P ˆµ 1 µ + M c 10 E c 10 E + i M δ cδ 4 E 1 The same boud holds for P ˆµ 1 µ Therefore, combiig the iequalities obtaied so far, we get P = c I { E} E + M c 10 E + i M 4 δ cδ 4 E Now, E [ ] R ε greedy E + t= E +1 P = E + c log + M c t=1 t 10 E + 4M : >0 t=1 t cδ 4 E If holds for ay suboptimal the the sum of the last two terms over t = 1,, becomes fiite This fiishes the proof of the result c > mi10 E, 4 E δ VI NUMERICAL RESULTS We have implemeted the proposed UCB ad ɛ-greedy algorithms, ad a aive extesio of the UCB scheme proposed by Gai et al [1] i Matlab I extedig [1] to deal with correlated arms, sice the dimesio of a arm is the umber of o-zero elemets i the arm-features, we used V t 1 = ˆµ t 1 + M M + 1 log t mi M N i t 1, ˆµ t 1 = ˆθt 1 φ as the idex for the UCB scheme i [1] I the simulatios, we vary the umber of siffers p = {1,,, 5}, ad the umber of chaels K {1,,, 8} Each chael has oe user associated to it The users are active with probability [01 08] respectively i chael [1 8] The adjacecy matrix G of all 5 siffers is give by where g i,j = 0 idicates that the user j is out of the receptio rage of siffer i θ ca be obtaied from the adjacecy matrix ad the user active probability due to Theorem From Figure 1, we see that for all schemes the regret teds to flatte out over time We used the exact value of the parameter d for ɛ-greedy algorithm Amog the three schemes, ɛ-greedy has the fastest covergece followed by the proposed UCB as secod, ad the UCB scheme of Gai et al [1] This is because ɛ-greedy utilizes the spaers durig the exploratio phases ad ca gai most iformatio regardig the uow parameters ad it also avoids usig cofidece bouds i maig decisios I cotrast, both UCB policies update their cofidece bouds quite coservatively, ad thus exhibit slow covergece Similar observatios ca be made from Figure 1bc showig the regret after 5000 time slots VII CONCLUSION AND FUTURE WORK I this paper, we cosidered the problem of optimally assigig p siffers to K chaels to moitor the trasmissio activities i a multi-chael wireless etwor Two policies were proposed that lear sequetially the user activities while maig chael assigmet decisios Both policies were show to achieve logarithmic regret i the umber of time slots with a term sub-liear i cardiality of the actio space

8 UCB[1] Proposed UCB ε greedy Regret vs time UCB[1] Proposed UCB ε greedy Regret vs Number of chaels UCB[1] Proposed UCB ε greedy Regret vs Number of siffers Regret 1500 regret 3000 Regret Time slots x Number of chaels Number of siffers a Regret vs time p =, K = 5 b Regret vs # of cha p = 3, t = 5000 c Regret vs siffers K = 5, t = 5000 Fig 1: Compariso of regrets of three schemes The geeralizatio of our theorems to the followig cases is trivial: i X i t is sub-gaussia with ow tail behavior eg, X i t are bouded with ow bouds, ii φ R M Other possible future wor icludes extesio to ostatioarity eviromets, which could be doe, eg, alog the lie of wor of [8], the cosideratio of a adversarial settig [16], [9], ad/or switchig costs [30] APPENDIX A EQUIVALENCE BETWEEN THE TWO MODELS I this sectio, we establish the equivalece betwee the two models to describe user activities I the first model, the restrictio of which siffer ca observe which users is modeled as a bi-partite graph G = S, U, E, ad the user activity is ecoded by a vector p = p u u U I the secod model, a K p -dimesio vector θ is defied with the followig elemets: P U i, > 0, 1 i p, 1 K, P U i1, = U i, > 0, 1 i 1 < i p, 1 K, P U 1, = U, = = U p, > 0, 1 K Whe the set of siffers that ca observe u ad v are idetical i the same chael, amely, Nu = Nv, we treat u ad v as a sigle user I aother word, i the first model, we oly cosider distict users, each coectig to a differet set of siffers over a specific chael Note that the total umber of distict users is at most K p Let us cosider users o chael without loss of geerality Each user u is represeted by y u, a vector of legth p, where y u i = 1 if u Ni, i = 1,, p Deote a partial order betwee two biary vectors y u y v if y u y v, ad l, st, y u l = 1, y v l = 1 Let p = [p 1, p,, p 1], ie, the vector of p probabilities of the idividual user beig active i chael Costruct a matrix A as follows A u,v = 1 if y v y u Clearly, with proper permutatio, A is a upper diagoal matrix with all 1 etries at the diagoal Furthermore, log1 θ = log1 p A Sice A is of full ra, we have log1 p = log1 θ A 1 Now let A be the th diagoal bloc of A ad p = [p 1, p,, p ] We have log1 θ = log1 p A ad log1 p = log1 θ A 1 APPENDIX B TAIL PROBABILITY BOUNDS The followig lemma geeralizes Hoeffdig s iequality to sums with a radom umber of terms The lemma i the form preseted here ca be foud as Theorem 18 of [8] a similar statemet, geeralizig Berstei s iequality ca be extracted from [4] Lemma 5: Let F t ; t 0 be a filtratio Let X t ; t 1 be a iid sequece taig values i some iterval of legth B Let ε t {0, 1} be a biary sequece Assume that X t is F t -measurable ad ε t is F t 1 -measurable t 1 Let N = t=1 ε t, X = t=1 ε tx t /N The, for ay 1, η > 0, 1 P X > E [X 1 ] + z, N 1 N log 1 log1 + η exp z B η 16 I particular, whe η = 03, P X > E [X 1 ] + z, N 1 N 4 l exp 199z Now, we cosider a multi-dimesioal geeralizatio of this result: Lemma 6: Let F t ; t 0 be a filtratio Let X t ; t 1 be a iid sequece taig values i R M such that X ti, the i th compoet of X t, taes values i some iterval of legth B Defie µ = M i=1 E [X 1i] Let ε t {0, 1} M be a M-dimesioal biary sequece Assume that X t is F t - measurable ad ε t is F t 1 -measurable t 1 Let N i = t=1 ε ti, X i = N 1 i The, for ay 1, P X > µ + z M i=1 B t=1 ε tix ti ad X = M i=1 X i 1, N 1,, N M 1 N i M 4 l exp 199z B Proof: Let p deote the probability to be bouded ad let µ i = E [X 1i ] The, p M i=1 P 1 X i > µ i + z N i, N i 1 The result the

9 9 follows by applyig Lemma 5 to each of the M terms o the right-had side The ext result ca be extracted from [3] with a slight improvemet The settig is similar to that of Lemma 5 with the deviatio from the mea as a determiistic umber Lemma 7: Let F t ; t 0 be a filtratio Let X t ; t 1 be a iid sequece taig values i some iterval of legth 1 Let ε t {0, 1} be a biary sequece Assume that X t is F t -measurable ad ε t is F t 1 -measurable t 1 Let N = t=1 ε t, X = t=1 ε tx t /N The, for ay 1, x > 0, z > 0, P X > E [X 1 ] + z Proof: We have P X > E [X 1 ] + z P N < x Now, P N < x + z exp x z + P P N x, X > E [X 1 ] + z = N x, X > E [X 1 ] + z s= x P N = s, X > E [X 1 ] + z Let S = t=1 ε tx t Defie τs as the first time whe s values of X are observed: τs = mi { t 1 : N t = s } Further, let S 1 = S τ1, S = S τ, Note that S has exactly terms ad S is a F -adapted martigale, where F = F τ 1 the so-called the optioal sippig process Now, X = S /N = S N /N Hece, P N = s, X > E [X 1 ] + z = P N = s, S N /N > E [X 1 ] + z = P N = s, S s /s > E [X 1 ] + z P S s /s > E [X 1 ] + z By the Hoeffdig-Azuma iequality, P S s /s > E [X 1 ] + z exp s z / Usig s=u e κu κ 1 e κu, which holds for ay iteger u ad κ > 0, we obtai the desired result REFERENCES [1] Robbis H, Some aspects of the sequetial desig of experimets, Bulleti of the America Mathematical Society, vol 58, pp , 195 [] T L Lai ad H Robbis, Asymptotically efficiet adaptive allocatio rules, Advaces i Applied Mathematics, vol 6, o 1, pp 4, 1985 [3] Peter Auer, Nicolò Cesa-Biachi, ad Paul Fischer, Fiite-time aalysis of the multiarmed badit problem, Mach Lear, vol 47, o -3, pp 35 56, 00 [4] Aad Balachadra, Geoffrey M Voeler, Paramvir Bahl, ad P Veat Raga, Characterizig user behavior ad etwor performace i a public wireless LAN, SIGMETRICS Perform Eval Rev, vol 30, o 1, pp , 00 [5] Trista Hederso, David Kotz, ad Ilya Abyzov, The chagig usage of a mature campus-wide wireless etwor, i Mobicom, 004, pp [6] Jihwag Yeo, Moustafa Youssef, ad Asho Agrawala, A framewor for wireless LAN moitorig ad its applicatios, i WiSe 04: Proceedigs of the 3rd ACM worshop o Wireless security, 004, pp [7] Jihwag Yeo, Moustafa Youssef, Trista Hederso, ad Asho Agrawala, A accurate techique for measurig the wireless side of wireless etwors, i the 005 worshop o Wireless traffic measuremets ad modelig, 005, pp [8] Maya Rodrig, Charles Reis, Ratul Mahaja, David Wetherall, ad Joh Zahorja, Measuremet-based characterizatio of 8011 i a hotspot settig, i Proceedigs of the 005 ACM SIGCOMM worshop o Experimetal approaches to wireless etwor desig ad aalysis, 005, pp 5 10 [9] Yu-Chug Cheg, Joh Bellardo, Péter Beö, Alex C Soere, Geoffrey M Voeler, ad Stefa Savage, Jigsaw: solvig the puzzle of eterprise 8011 aalysis, i SIGCOMM, 006 [10] Dog-Hoo Shi ad Saurabh Bagchi, Optimal moitorig i multichael multi-radio wireless mesh etwors, i MobiHoc, 009, pp 9 38 [11] Chadra Cheuri ad Amit Kumar, Maximum coverage problem with group budget costraits ad applicatios, i APPROX, 004, pp 7 83 [1] Aru Chhetri, Huy Nguye, Gabriel Scalosub, ad Rog Zheg, O quality of moitorig for multi-chael wireless ifrastructure etwors, i The ACM Iteratioal Symposium o Mobile Ad Hoc Networig ad Computig MobiHoc, 010 [13] Rajeev Agrawal, Sample mea based idex policies with olog regret for the multi-armed badit problem, Advaces i Applied Probability, 1995 [14] P Auer, N Cesa-Biachi, Y Freud, ad RE Schapire, Gamblig i a rigged casio: The adversarial multi-armed badit problem, Foudatios of Computer Sciece, Aual IEEE Symposium o, vol 0, pp 3, 1995 [15] Paat Rusmevichietog ad Joh N Tsitsilis, Liearly parameterized badits, Math Oper Res, vol 35, o, pp , 010 [16] N Cesa-Biachi, G Lugosi, ad G Stoltz, Regret miimizatio uder partial moitorig, Mathematics of Operatios Research, vol 31, pp , 006 [17] Lifeg Lai, Hesham El Gamal, Hai Jiag, ad H Vicet Poor, Cogitive medium access: Exploratio, exploitatio ad competitio, CoRR, vol abs/ , 007 [18] Lifeg Lai, Hai Jiag, ad HV Poor, Medium access i cogitive radio etwors: A competitive multi-armed badit framewor, i Sigals, Systems ad Computers, 008 4d Asilomar Coferece o, , pp [19] Keqi Liu ad Qig Zhao, Decetralized multi-armed badit with multiple distributed players, CoRR, vol abs/ , 009, iformal publicatio [0] A Aadumar, N Michael, ad AK Tag, Opportuistic Spectrum Access with Multiple Users: Learig uder Competitio, i Proc of IEEE INFOCOM, Sa Deigo, USA, Mar 010 [1] Yi Gai, B Krishamachari, ad R Jai, Learig multiuser chael allocatios i cogitive radio etwors: A combiatorial multi-armed badit formulatio, i New Frotiers i Dyamic Spectrum, 010 IEEE Symposium o, , pp 1 9 [] Jeremy Elso, Lewis Girod, ad Deborah Estri, Fie-graied etwor time sychroizatio usig referece broadcasts, SIGOPS Oper Syst Rev, vol 36, o SI, pp , 00 [3] V Aatharam, P Varaiya, ad J Walrad, Asymptotically efficiet allocatio rules for the multiarmed badit problem with multiple playspart i: Iid rewards, Automatic Cotrol, IEEE Trasactios o, vol 3, o 11, pp , ov 1987 [4] J-Y Audibert, R Muos, ad Cs Szepesvári, Tuig badit algorithms i stochastic eviromets, i Algorithmic Learig Theory - 18th Iteratioal Coferece, ALT , pp , Spriger [5] J-Y Audibert, R Muos, ad Cs Szepesvári, Exploratio-exploitatio tradeoff usig variace estimates i multi-armed badits, Theoretical Computer Sciece, vol 410, o 19, pp , 009 [6] P Auer, Usig upper cofidece bouds for olie learig, i Proceedigs of the 41th Aual Symposium o Foudatios of Computer Sciece 000, pp 70 93, IEEE Computer Society [7] V Dai, TP Hayes, ad SM Kaade, Stochastic liear optimizatio uder badit feedbac, i COLT-008, 008, pp [8] A Garivier ad E Moulies, O upper-cofidece boud policies for o-statioary badit problems, Tech Rep, LTCI, Dec 008

10 [9] N Cesa-Biachi ad G Lugosi, Combiatorial badits, i COLT 009, S Dasgupta ad A Klivas, Eds, 009 [30] R Agrawal, MV Hedge, ad D Teeetzis, Asymptotically efficiet adaptive allocatio rules for the multiarmed badit problem with switchig cost, IEEE Trasactios o Automatic Cotrol, vol 33, o 10, pp ,