Learning the Best K-th Channel for QoS Provisioning in Cognitive Networks

Transcription

1 Learnng the Best K-th Channel for QoS Provsonng n Cogntve Networks Anonymous Author(s) Afflaton Address emal Abstract A learnng strategy for dstrbuted channel selecton n Cogntve Rado networks s proposed. The goal of the learnng s qualty of servce (QoS) provsonng by whch competng secondary users cooperatvely converge to ther rank-optmal channels whle channel avalablty statstcs are ntally unknown. By ths convergence, collsons reaches zero snce users eventually work on ther own channels. The proposed learnng strategy, k th MAB, s nspred from the Mult- Armed Bandt problem but t converges to the k th best arm. The rank-optmal channel for each user\player s dentfed based on the user s QoS demands. We beleve that under ths learnng and allocaton polcy, cogntve users get servces proportonal to ther QoS level snce evaluaton results represent order optmalty n terms of average throughput. 1 Introducton Due to extensve need for wreless spectrum and the neffcency n utlzng t, Cogntve Rado(CR) technology s emerged to allow unlcensed\secondary users(su) for opportunstc access to the spectrum when lcensed\prmary users(pu) are not actve. To take advantage of the possble empty spaces n the spectrum, SUs sense a part of the spectrum and use t for transmsson f t s found free. Thus, t s crucal for SUs to make optmal decsons about whch part of the spectrum to sense at dfferent tmes. Ths gves rse to the trade-off between exploraton: sensng new channels n the hope of obtanng better avalablty and explotaton: ensurng successful transmsson n the current tme. When there are multple SUs, there s a competton among SUs to access the channel wth the best avalablty. Hence collson s lkely snce there s no explct nformaton exchange among SUs about ther observatons and channel selecton strategy. Moreover, SUs demand dverse levels of qualty of servce (QoS) requrements proportonal to ther traffc mportance. To provson these requrements, SUs should cooperatvely fnd ther own unque rank-optmal channels and work on them. The goal of ths paper s proposng a learnng strategy for channel selecton by whch SUs estmate the rank of channels wth respect to ther avalabltes through sensng samples. Ths strategy helps SU- wth rank k to allocate tself to an orthogonal channel wth k th hghest avalablty, n a dstrbuted manner. In ths regard, Mult-Armed Bandt (MAB) problem for fndng the best channel s revewed n Secton 2. k th MAB channel-selecton strategy for fndng the k th best arm, s proposed n Secton 3. Performance evaluaton and concluson of the paper are covered n Secton 4 and 5 respectvely. 1

2 Mult-Armed Bandt (MAB) problem MAB problem formulzes explotaton\exploaraton trade-off for choosng the best arm by selectng one out of M possble arms n each tral t 1,..., T. For the chosen arm n tral t, reward x (t) s drawn from some fxed but unknown dstrbutons D 1, D 2,..., D M whle the rewards for other arms excludng,.e. {1,..., M}\, are not revealed. The approprate strategy for the MAB problem, pursues the goal of maxmzng the total reward up to the observaton perod T,.e. T t=1 x(t) where the upper expected total reward s obtaned by the best dstrbuton D. The dfference between ths upper bound and the acheved total reward s defned as regret. The exploraton\explotaton trade-off s reflected on one hand by the necessty for tryng all arms and on the other hand by the regret suffered by tryng a non-optmal arm. Too lttle exploraton mght make a sub-optmal alternatve look better than the optmal one because of random fluctuatons whle too much exploraton prevents the algorthm from playng the optmal often enough whch also result n a large regret. 2.1 Upper confdence bound (UCB) algorthm Upper Confdence Bound (UCB) algorthm for solvng the MAB problem, chooses arm (t) n tral t as: (t) = arg max M ( x (t) + ζlog(t) ) n (t) UCB calculates weght of arm based on x (t) +σ (t) when ths arm has dstrbutons D and expected reward R. The frst term, x (t), s the current average reward whch s an estmate for the true expected reward R. And the second term, σ (t) true and average rewards fall n wth hgh probablty,.e. x (t), corresponds to the confdence nterval that both the σ (t) R x (t) + σ (t). Wth s large UCB, we may say that a tral s an explotaton tral f an alternatve s chosen snce x (t) and that s an exploraton tral f σ (t) s large. Snce σ (t) decreases rapdly wth each choce of arm, the number of exploraton tral s lmted. Thus the use of UCB automatcally trades off between exploraton and explotatons. An mproved verson, UCB-V, consders the effect of the emprcal varance, s proposed n [1] and estmates the best arm n tral t as followng where ζ and c are constant coeffcents: arg max M ( x(t) + ( x(t) 3 MAB problem and K-th best arm 3.1 System model ( x (t) ) 2 )ζlog(t) n (t) + c.log(t) ) n (t) We assume that tme s slotted and a tme-slot on channel s occuped by PUs wth Bernoull dstrbuton wth parameter µ,.e. W B(µ ). There s a set of U cogntve users grabbng free tme-slots from M ndependent and orthogonal channels on the premse of not nterferng the operaton of lcensed PUs. Also, cogntve user has a pror nformaton about ts own unque rank, k, among the rest of U-1 users. Here, users should learn channel mean avalabltes, µ, n a dstrbuted manner and converge to an approprate channel whle on one hand they do not exchange nformaton on ther decsons and observatons and on the other hand they mplement the preallocated rankngs. Note that we use two terms of tme-slot and tral nterchangeably. Thus, the optmal channel selecton strategy for a SU- s the one that narrows operaton of user on the channel wth k th hghest mean avalablty. At the begnnng of tme-slot t, user selects a channel, e.g. channel j, and keeps the hstory of ts selectons on T,j. User then senses the selected channel j to fnd f PU has occuped ths slot or not and keeps the hstory of sensng results regardng to channel j n X,j. User approxmate the 2

3 mean avalablty of channel j as ˆµ j = X,j T,j where T,j ndcates the number of tmes that channel j s selected by user so far. In tme slot t+1, channel selecton strategy of user explots prevous observatons as the form of ˆµ j, j 1,..., M to pck a channel for sensng. Note that although X,j = 0 ndcates that ths slot s free of PU transmsson but t does not guarantee that user j s the sole transmtter n ths slot. In fact, collson s lkely snce multple users may select a common channel. However, a proper learnng strategy eventually confnes the operaton of user on the channel wth k th hghest mean avalablty and consequently collson probablty reaches zero n the course of tme. To estmate the achevable throughput, user receves an acknowledgement on whether ts transmsson on channel j was successful(z,j = 1) or not (Z,j = 0). 3.2 Greedy dstrbuted learnng under pre-allocaton (ρ P RE ) Authors n [2] have proposed a dstrbuted learnng under pre-allocaton, ρ P RE, as a modfed verson of the ɛ greedy strategy for fndng the K-th best channel. Ther general dea s that a SU should do a lot of experments by selectng dfferent channels to estmate ther avalablty ratos and eventually settles down n the approprate one. In ρ P RE, SU- wth rank k, selects a unformly random channel wth probablty ɛ n = mn(1, β/n) and selects the channel wth k th hghest sample mean wth probablty 1 ɛ n. It means that, there s a fnte probablty ɛ n for user to not select the channel accordng to ts rank and nstead fnds an opportunty to explore other channels to fnd better estmaton about ther sample means. The value of β defnes the trade-off between explotaton and exploraton and so the effcency of ρ P RE s hghly senstve to the approprate choce of β. In the next secton we propose a new approach, k th MAB to solve the problem of fndng the k th best channel whch s more effcent than ρ P RE as evaluated n Secton UCB for ordrng (k th MAB) In ths secton, a decentralzed polcy called k th MAB s constructed by whch a cogntve user wth rank k fnds the best k channels n order, and converges to the one wth k th hghest mean avalablty. Note that the hgher the value of µ s, {1,..., M}, the more avalable a channel s. Wthout loss of generalty, from now we suppose that user has the th hghest rank and channel j has the j th hghest µ. Wth ths assumpton, user 1 and user 2 want to converge to channel 1 and 2 respectvely. The basc dea s that user selects the best channels n a hypothetcal frame structure conssts of tme-slots. The formal explanaton of ths polcy s summarzed n n Table 1. As an example, consder a case of three cogntve users,.e. U=3, n whch SU-1 and SU-3 have the hghest and lowest prortes respectvely. For SU-1, the problem s smplfed as the common MAB problem n whch a player wants to fnd the best arm (channel). Thus, SU-1 always apples UCB-V to effcently learn and select the best channel. SU-2, works n frames of two tme-slots snce t wants to fnd the best two channels. For ths, n odd tme-slot of each frame, t apples UCB-V to fnd the best channel. In order to fnd the second best channel n an even tme-slot of the frame, t apples UCB-V polcy to the remanng M-1 channels after removng the channel consdered as the best one n the odd tme-slot. SU-3 wants to fnd the best three channels and fnally converges to channel 3. For ths, t works n a hypothetcal frame of three tme-slots and consders fndng the best channel n the frst tme-slot. Then t apples UCB-V to M-1 channels remaned from the frst tme-slot, to fnd the second best channel. Fnally, t estmates the thrd best channel by applyng UCB-V polcy to the lst of M-1 channels resulted from the case that the second best channel s estmated. At the end of the thrd tme-slot, the current frame of SU-3 s completed and ths user resumes ts channel selecton pattern from the next frame. Snce the goal of user s convergence to the th best channel, t swtches to channel j wth ˆµ j > ˆµ only wth probablty P swtch B(mn(1, 5.0 t )). Ths allows user to smoothly converge to the th best channel. For example, SU-2 tres to fnd the best channel n odd tme-slots only wth probablty P swtch. Smlarly, n trals t, t%3!= 0, SU-3 estmates the best and the second-best channels wth probablty P swtch and estmates the thrd best one wth probablty 1 P swtch. 3

4 Int: Table 1: k th MAB for user wth k th hghest rank Slectng each channel j, j 1,..., M once and updates X,j and T,js. Set K sub-sequences wth ˆµ j = X,j T,j. At tral t = 0,..., T : Calculates j = t % k and P swtch B(mn(1, 5.0 t )) f P swtch = 0 : Applyng UCB-V to the k th sub-sequence, Selectng the best channel, say h, and updates µ h. else f P swtch = 1 : f j = 0 : Applyng UCB-V to the k th sub-sequence. Selectng the best channel, say h, and updates µ h, else f j = 1 : Applyng UCB-V to the frst sub-sequence, Selectng the best channel, say h, and updates µ h, Updatng the second sub-sequence wth frst sub-sequence\ h else f j = 2 : Applyng UCB-V to the second sub-sequence, Selectng the best channel, say h, and updates µ h, Updatng the thrd sub-sequence wth second sub-sequence\ h... else f j = (k-1) : Applyng UCB-V to the k 1 th sub-sequence, Selectng the best channel, say h, and updates µ h, Updatng the K th sub-sequence wth the (k 1) th sub-sequence\ h 4 Performance evaluaton We present smulatons for comparng effcency of two dscussed schemes ρ P RE and k th MAB. A set of M=5 orthogonal channels wth mean avalabltes characterzed by the followng Bernoull dstrbutons s avabale, CH1 B(.8), CH2 B(.6), CH3 B(.4), CH4 B(.2), CH5 B(.1). Note that n a presence of a centralzed arbtrator, SU- wth rank k s centrally assgned to work on the k th best channel. Ths assgnment of SUs to ther rank-optmal orthogonal channels makes collson occurrence s unlkely and guarantees the optmum throughput. However, wthout such an optmal allocaton, users may not choose ther desred channels and face wth collson. For SU-, three crtera are ntroduced whch represent how well learnng polces work n comparson to the optmal case: 1) regret T = T.µ h M j=1 Z,j ndcates how much throughput s lost up to an observaton perod T where h s the k th best channel wth mean avalablty µ h. 2) rank opt = T,h M j=1 T,j gves an estmate about effcency of the learnng strategy n boundng the operaton of user on ts desred channel h. M j=1 3) T hroughput = Z,j M estmates the percentage of channel selectons that lead to a successful packet transmsson. Under the presence of a centralzed arbtrator, the average throughput for j=1 T,j SU- would be µ h. 4

5 (a) (b) (c) (d) (e) (f) (g) (h) () Fgure 1: {(a)-(d)-(g)} s normalzed regret, regrett log(t ), vs. T slots, {(b)-(e)-(h)} s rank opt vs. T slots, {(c)- (f)-()} s T hroughput vs. T slots. Fg. 1 and Fg. 2 represent the results for k th MAB, ρ P β=200 RE RE and ρp β=1000 where two and three compettve SUs,.e. U=2 and U=3, exst. SUs compete on a set of M=5 channels where SU-1 has the hghest rank and ts desred channel s CH1 wth 80% mean avalablty and SU-3 has the lowest rank and ts desred channel s CH3 wth µ = 40%. To make the comparson easer, a vertcal lne corresponds to 0.9% of the fnal value, s added to each graph. Worthwhle to menton that performance of ρ P RE s evaluated under varous values of β. It s emprcally estmated that β = 200 gves the best results for rank opt and T hroughput. Therefore, results of k th MAB are compared to the best emprcal confguraton of ρ P RE. To emphasze that performance of ρ P RE drectly hnges on the confguraton parameter β, results related to β = 1000 are also provded. In fgures 1 and 2: Subfgures (a),(d) and (g) ndcate how fast an appled learnng strategy converges to the desred channel. Under k th MAB, after tral 2000 the desred channel s selected wth probablty hgher than 80% whle for ρ P β=200 RE, smlar stuaton happens after 4000 trals. Subfgures (b),(e) and (h) represent normalzed regret up to tme-slot T as regrett log(t ). Comparson of the results justfes that at each arbtrary tral T, SU-, {1, 2, 3} suffers the least regret when t works based on k th MAB. Subfgures (c),(f) and () represent that under k th MAB learnng polcy, T hroughput reaches 0.9% of ts hghest value around tral T=2500 whle smlar results are obtaned around tral T=4000 for the case of ρ P β=200 RE. 5

6 (a) (b) (c) (d) (e) (f) (g) (h) () Fgure 2: {(a)-(d)-(g)} s normalzed regret, regrett log(t ), vs. T slots, {(b)-(e)-(h)} s rank opt vs. T slots, {(c)- (f)-()} s T hroughput vs. T slots. 5 Concluson In ths paper, we desgn a dstrbuted learnng polcy by whch SUs estmate channel statstcs and cooperatvely converge to ther rank-optmal channels. Under ths onlne learnng strategy, acheved throughput for each SU would be proportonal to the level of ts QoS requrements. Smulaton results represent that convergence rate to the desred channel s hgh and SUs get rank-based average throughput. We plan to extend ths work for the non-greedy case n whch SUs may have less traffc than channel avalablty of ther desred channels. Thus, SUs can mprove ther average throughput by capturng leftover of channels wth hgher ranks. References [1] Auer, P., Cesa-Banch, N., & Fscher, P. (2002) Fnte-tme Analyss of the Multarmed Bandt Problem.Journal of Machne Learnng. [2] Anandkumar, A., Mchael, N. & Tang, A. (2010) Opportunstc Spectrum Access wth Multple Users: Learnng under Competton. Proceedngs of IEEE INFOCOM. [3] Lu, K. & Zhao, Q. (2010) Dstrbuted learnng n cogntve rado networks: Mult-armed bandt wth dstrbuted multple players. Proceedngs of IEEE Internatonal Conference on Acoustcs Speech and Sgnal Processng. 6