Slow Fading Channel Selection: A Restless Multi-Armed Bandit Formulation

Transcription

1 Slow Fadng Channel Selecton: A Restless Mult-Armed Bandt Formulaton Konstantn Avrachenkov INRIA, Maestro Team BP95, Sopha Antpols, France Emal: k.avrachenkov@sopha.nra.fr Laura Cottatellucc, Lorenzo Magg Eurecom Moble Communcatons Department BP193, F Sopha Antpols, France Emal: {laura.cottatellucc,lorenzo.magg}@eurecom.fr Abstract We deal wth a mult-access wreless network n whch transmtters dynamcally select a frequency band to communcate on. The slow fadng channel attenuatons follow an autoregressve model. In the sngle user case, we formulate ths selecton problem as a restless mult-armed bandt problem and we propose two strateges to dynamcally select a band at each tme slot. Our objectve s to maxmze the SNR n the long run. Each of these strateges s close to the optmal strategy n dfferent regmes. In the general case wth several users, we formulate the problem as a stochastc game wth uncountable state space, where the objectve s the SINR. Then we propose two strateges to approxmate the best response polcy for one user when the other users strategy s fxed. I. INTRODUCTION Next generaton of wreless networks s expected to be characterzed by a hgh decentralzaton/dstrbuton of control functons among nodes to support self-organzng and selfhealng capabltes. Network devces shall be able to montor and sense the surroundngs, learn from ther montorng and smartly and dynamcally allocate resources. Ths perspectve scenaro s attractng a consderable amount of research efforts to develop learnng technques able to optmze the tradeoff between exploraton and explotaton of envronment and resources. A relevant class of learnng algorthms s the Mult- Armed Bandt (MAB) one. In the classc MAB problem there exst several arms that offer a reward when pulled (n analogy wth gamblng on bandts n casnos). Each arm s assocated wth a Markov process, and the reward of an arm s a functon of ts state. Gttns provded n [1] a dynamc allocaton procedure, then dubbed Gttns ndex, whch s optmal f the arms that are not pulled do not evolve over tme. The more general case when the arms that are not pulled keep evolvng n tme s known as Restless MAB. It was proven by Papadmtrou and Tstskls n [2] that restless MAB are PSPACE-hard n general. In [3], Whttle proposed to adopt a heurstc Lagrangan relaxaton to extend the Gttns ndex to the restless case, whch s asymptotcally optmal under certan lmtng regme [4]. In ths work, we consder a wreless network where transmtters can select a frequency band from a shared pool to communcate on. The evoluton of the slow fadng channel attenuaton assocated to each frequency band and each transmtter s a random process that can be well approxmated by an autoregressve process [5]. We assume that all such random processes are ndependent of each other. The goal of each transmtter s to maxmze ts average Sgnal to Interference and Nose Rato (SINR) n the long run. To get nsght nto ths problem, frst we focus on a sngle transmtter system to nvestgate the exploraton-explotaton trade-off for the randomness ntroduced n the system by the autoregressve channel attenuatons. Then, we consder the mult-transmtter case where the problem s further complcated by the randomness ntroduced by the autonomous band selectons of multple transmtters. For the sngle termnal case, the problem of dynamc frequency allocaton for SNR maxmzaton can be modeled as a restless Mult Armed Bandt (MAB) snce the transmtter only knows the nstantaneous attenuatons on the bands utlzed n the past and they evolve also when not utlzed. To the best of the authors knowledge, there are no avalable results on the MAB problem for autoregressve processes. We propose two heurstc frequency allocaton strateges, one called myopc and the other randomzed. When the AR processes possess smlar autocorrelaton functons, we suggest to use the myopc strategy. Instead, when there s one AR process havng a much hgher autocorrelaton, we suggest to use the randomzed strategy. In the scenaro wth multple transmtters the problem s formulated as a stochastc game wth uncountable state space. We focus on a two-user system and we assume that user 1 s oblvous of the presence of user 2 and follows a plan sngle-user myopc approach. Then we propose two strateges for user 2 to approxmate ts best response aganst user 1 s strategy. Agan, one strategy s myopc and the other s randomzed, wth respect to the SINR objectve functon. A lexcal remark. We say that we sample a frequency band when we utlze t for the communcaton n a certan tme slot. II. MODEL In Secton III we consder one transmtters and one recever, whle n Secton IV we deal wth a model wth two transmtters. Tme s dvded nto slots and, at the begnnng of a tme slot, each transmtter (or user) selects a frequency band, out of a pool of M dfferent ones, to transmt. At the recever, a sngle-user decoder per transmtter s deployed. In the two-transmtter case, when a both users access the same frequency band at tme slot t, they nterfere wth each other, and the SINR (Sgnal to Interference plus Nose Rato) for user j = 1, 2 at tme t s SINR,j [t] = P j h,j [t] 2 N 0 + q j P q h,q [t] 2

2 where P j s the transmt power of user j, h,j [t] C s the -th channel coeffcent of user j at tme t and N 0 s the varance of the addtve whte Gaussan nose at the recever. When only one user s present, the SINR defnton bols down to the classc SNR. For smplcty of notaton, henceforth we wll denote the channel attenuaton coeffcent h,j [t] 2 as g,j [t]. Let us descrbe now our channel model. In [5] t s shown that, under slow fadng condtons, the SNR (Sgnal to Nose Rato) of ndoor wreless channels can be well approxmated by an autoregressve (AR) model. Ths means that, under such condtons, we can model the channel attenuatons as g,j [t] = p,j k=1 ϕ (k),j g,j[t k] + c,j + ϵ,j [t] where ϕ,j R, {ϵ,j [t]} t s an..d. Gaussan process wth zero mean and varance σ,j 2, c,j > 0, and p,j s the order of the model. Moreover, all the channels consdered are ndependent of each other,.e. ϵ 1,j 1 [t] s ndependent of ϵ 2,j 2 [t] when ether 1 2 or j 1 j 2. We assume the AR process to be wde sense statonary (WSS),.e. the roots of the polynomal z p p k=1 ϕ(k),j zp k must le nsde the unt crcle. III. SINGLE USER: MDP FORMULATION In ths secton we consder the sngle user case. In order to smplfy the notaton, we drop the user ndex. In our study we consder an AR(1) channel attenuaton model,.e. g [t] = ϕ g [t 1] + c + ϵ [t] For ϕ < 1, the process s WSS, and the (uncondtoned) expected value of channel attenuaton g [t] at any tme nstant t can be expressed as m = E(g [t]) = c t. 1 ϕ Therefore we can say that E(SNR [t]) = P m /N 0, for all t. Straghtforward computatons show that the autocovarance functon of the channel attenuaton can be wrtten as E ( (g [t] m )(g [t n] m ) ) = ϕ n σ 2 1 ϕ 2. (1) We now llustrate the two fundamental assumptons of ths paper. Frst, the coeffcents ϕ and σ are known by the transmtter, whch mght have estmated them durng a tranng phase. Second, the transmtter, at tme t, only knows the nstantaneous attenuatons of the frequency bands utlzed up to tme t 1. Indeed, we assume that the recever estmates g,j and broadcast ths nformaton on the channel along wth an dentfer for the transmtter and the frequency band. The goal of the user s to dynamcally swtch among the channels at each tme slot n order to maxmze the expected average SNR over an nfnte horzon. Equvalently, t wants to maxmze the expected average over tme of channel attenuatons, denoted by r(π): T 1 1 } max π {r(π) = lm E π (g π(t) [t]) T T t=0 (2) where π s a dynamc samplng strateges over the channels 1,..., M. The reader should notce that a channel samplng strategy π at tme t may depend on the whole hstory of the observed channels and of the samplng decsons up to tme t. Ths class also ncludes statc strateges, that choose one channel once for all. Intutvely, when there exsts a channel wth much lower uncondtoned expected attenuaton,.e. m m k for all k, a statc selecton of the channel s the nearly optmal strategy, snce wth hgh probablty g [t] > g k [t] for all k for almost all t. In ths secton, we want to study how to dynamcally select the band on whch to transmt when, a pror, all of them are nearly equvalent,.e. there exsts m m k, for all k. At each tme slot, there s always one channel better than the others, hence we wsh to track dynamcally the evoluton over tme of the best channel. Intutvely, the samplng choce at each nstant has to be a trade-off between exploraton and explotaton. To gve a hnt, the most natural polcy, that we wll call myopc, at each tme step t ams at maxmzng the expected value of SNR[t], gven all the prevous channel observatons. On the other hand, the statcal nformaton about channels that are not used becomes more and more obsolete, therefore n some cases t mght be better to explore dfferent channels wth a randomzed strategy. We can formulate the optmzaton problem (2) as a restless Mult Armed Bandt problem (MAB for short), n whch a user at each tme nstant t selects an arm (here, frequency band) whch gves a reward (here, the SNR) and all the arms, ncludng the ones that have not been selected, evolve accordng to a certan stochastc process (here, an autoregressve process). More specfcally, we can descrbe the decson problem at hand as a Markov Decson Process (MDP) wth an uncountable set of states S or, equvalently, as a Partally Observable MDP. Let us descrbe t n detal. At tme t, we call n (t) the number of steps ago n whch channel has been last used. The attenuaton of channel at tme t condtoned on ts last observaton s a Gaussan r.v., and we denote ts mean and varance as µ (t) and ν (t), respectvely: µ (t) = E ( g [t] g [t n (t)] ) = ϕ n (t) 1 ϕ n (t) g [t n (t)] + c (3) 1 ϕ ν (t) = Var ( g [t] g [t n (t)] ) = σ 2 1 ϕ 2n (t) 1 ϕ 2 where g [t n (t)] s the attenuaton of channel durng ts last utlzaton. At tme step t, thanks to the Markov property of the AR(1) process, the whole statstcal nformaton about channel s hence contaned n (µ (t), ν (t)). We observe that µ R, whle ν s bounded between [σ 2; σ2 /(1 ϕ2 )]. The decson on whch channel to utlze at tme t hnges on the set S[t]: S[t] = {µ 1 (t), ν 1 (t), µ 2 (t), ν 2 (t),..., µ M (t), ν M (t)}. (5) By utlzng the MDP jargon [6], we call by S[t] the state of the decson problem at tme t. The state space S s the uncountable collecton of all the possble states. In each state S S, a set of actons A = {1, 2,..., M} s avalable to (4)

3 the transmtter, whch represents the collecton of channels that can be selected at tme slot t. If channel s selected, then we map the reward for the user n state S[t] to the expected channel attenuaton at tme t condtoned on the last observaton of channel tself,.e. µ (t). The state of the system at tme t + 1 evolves stochastcally, accordng to the followng Markovan rule. If channel s selected at tme t, then at tme t + 1, µ (t + 1) = ϕ Y + c, where Y N ( µ (t), ν (t) ) ν (t + 1) = σ 2. Instead f channel s not selected at tme t, µ (t + 1) = ϕ µ (t) + c ν (t + 1) = ϕ 2 ν (t) + σ 2. A. Heurstc algorthms The theory of MDP allows us to clam that there exsts an optmal statonary strategy π O for the problem (2). Unfortunately, the computaton of π O turns out to be a dffcult task. Indeed the soluton to a Markov Decson Problem wth uncountable state can only be approxmated by means of dscretzaton algorthms [6], and even n ths case the curse of dmensonalty entals that the sze of the dscretzed state space ncreases exponentally wth the number of arms. A dfferent approach would be to compute the Whttle ndex [3] of each channel, but ths approach s not guaranteed to be optmal. Hence, t becomes crucal to devse a smple polcy whose performance s reasonably close to the optmal r(π O ). In the followng we propose the most natural statonary strategy one can thnk of,.e. the myopc polcy π M that ams at maxmzng the nstantaneous expected SNR n each state. Such a polcy does not take nto account that the statstcs of the channel that have not been selected for a long perod mght become too stale. Frst, we need to ntalze the algorthm, and we choose to sample the coeffcent of each channel once. Algorthm 1. Myopc polcy π M. For 0 t M 1 select channel t,.e. π M (S[t]) = t + 1. For t M, π M (S[t]) = argmax µ (t). {1,...,M} We ntend to compare the performance of the myopc polcy wth a more sophstcated one, that we call randomzed strategy and s nspred by the Thompson samplng strategy for Bayesan Mult Armed Bandt problems [7]. We suggests to draw, n each state S[t], one realzaton of the random varable ξ = g [t] g [t n (t)], for each channel = 1,..., M. Then, the arm correspondng to the hghest realzaton of ξ s chosen. Ths procedure does not always follow the myopc rule, but wth a certan probablty explore the arms that, though possessng a lower µ, mght be optmal snce ther last observaton s too stale. Algorthm 2. Randomzed polcy π R. For 0 t M 1 select channel t,.e. π R (S[t]) = t + 1. For t M, draw a realzaton of the Gaussan varable ξ N (µ (t), ν ()) for all = 1,..., M. Select π R (S[t]) = argmax ξ. =1,...,M B. Smulatons In ths secton we show the results of some smulatons, gvng a hnt about the performance of the myopc and the randomzed polces, descrbed respectvely n Algorthm 1 and 2. Gven a statonary polcy π, we want to assess ts average reward r(π). We compare the myopc and randomzed polces wth ) the optmal polcy π O, approxmated by means of a state dscretzaton technque [6], wth ) the upper bound for the performance of any strategy, computed by selectng the channel wth the hghest coeffcent g at each tme step: π U (t) = argmax g [t], t 0 (6) =1,...,M and wth ) the statc polcy π S, that selects off-lne the arm wth the hghest expected value, and no longer swtches to other channels,.e. π S [t] = argmax m, t 0. =1,...,M Of course, the strategy π U s not applcable, snce t s not causal. In theory, ts performance s achevable only when the channels are determnstc hence perfectly predctable,.e. σ = 0 for all = 1,..., M. We now show the performance of the fve polces under scrutny, the myopc π M, the randomzed π R, the statc π S, the optmal π O, and the upper bound polcy π U, under dfferent channel condtons. Frst, we consder 3 arms, where arms 2,3 are statstcally equvalent, and ϕ 2 = ϕ 3 = 0.3, σ2 2 = σ3 2 = 1, and m 2 = m 3 = 8. Arm 1 has the same coeffcents ϕ 1 = 0.3, σ1 2 = 1 as arms 2,3. In Fgure 1 we show the performance of the fve polces when m 1 vares wthn [7; 9]. We see that, under these condtons, the myopc polcy outperforms the randomzed one snce the latter wastes too much tme n explorng arms that are not optmal. As ntuton confrms, the statc polcy π S performs as well as the myopc π M when arm 1 has the hghest expected value m 1 > m 2 = m = 3. Instead, for m 1 < m 2 = m 3, dynamcally swtchng between the arms 2,3 s benefcal wth respect to statcally selectng one of the two. As we see n Fgure 1, when all the arms are characterzed by the same uncondtoned expectaton,.e. m = 8, for = 1, 2, 3, the statc polcy π S s outperformed by both the myopc and the randomzed strateges. It s ndeed better to swtch among the channels to attempt to track the best nstantaneous channel at each tme nstant, based on the prevous observatons. Remarkably, the performance of the myopc polcy π M s close to the optmal π O. Hence, we evaluate our algorthms n a dfferent scenaro, n whch the value m s are the same for all the channels, but there exsts one channel (say, 1) whose autocovarance functon (1) decays consderably more slowly than the others. It s clear from Fgure 2 that there are lapses n whch channel 1 s by far the best, and some others n whch ts channel coeffcent g 1 plummets below the others. From Fgure 2 we observe that the myopc strategy often fals to track channel 1 when t s the best. The reason s qute ntutve: durng the lapse n whch channel 1 s the worst one, the myopc strategy does not choose t, then ts last observaton become obsolete, and consequently the predcton µ 1 (t) tends to m 1 = 10. Thus, t s hghly

4 r(π) r(π U ) upper bound r(π O ) optmal r(π M ) myopc r(π R ) randomzed r(π S ) statc arm m 1 Fgure 1. Performance of myopc and randomzed algorthm wth 3 arms (channels). Arms 2 and 3 are statstcally equvalents, wth ϕ 2 = ϕ 3 = 0.3, σ2 2 = σ2 3 = 1, and m 2 = m 3 = 8. Arm 1 has the same ϕ 1 = 0.3, σ1 2 = 1 as arms 2,3, whle the performance of the proposed algorthms are assessed when m 1 vares wthn [7; 9]. h[t] arm 1 arm 2 arm 3 π R randomzed π M myopc t Fgure 2. Channel (or arm) selecton when c = 10 for all, ϕ 1 = 0.9, ϕ 2 = ϕ 3 = 0.3, σ1 2 = 1.5, σ2 2 = σ2 3 = 0.5. The randomzed strategy succeeds n trackng the frst channel wth hgher autocorrelaton, when t s the best one. probable (and ths probablty ncreases wth M) that one of the other, suboptmal, channels, havng a fresher observaton, offers a hgher predcton. It easly follows that, for ts nherent features, the randomzed polcy s more sutable to such knd of stuatons, as results n Fgure 3 confrm. We consdered 3 arms (frequency bands). Arms 2 and 3 are statstcally equvalents, wth ϕ 2 = ϕ 3 = 0.3, σ 2 2 = σ 2 3 = 1, c 2 = c 3 = 10. Arm 1 has the same coeffcents c 1 = 10, σ 2 1 = 1 as arms 2,3, whle the performance of the proposed algorthms are assessed when the coeffcent ϕ 1 vares wthn [0.3; 0.98]. As we ntutvely explaned before, when the coeffcent ϕ 1 s suffcently hgh,.e. ϕ > 0.85, the randomzed strategy outperforms the myopc one. Notably, the myopc polcy s quas-optmal for ϕ 1 < 0.6, whle the the randomzed one s nearly optmal for ϕ 1 > 0.9. IV. MULTI USER: STOCHASTIC GAME FORMULATION In ths secton we dscuss the more general scenaro descrbed n Secton II, n whch two transmtters dynamcally select one among M channels at each tme slot. If some users choose the same channel n one tme slot, they nterfere wth each other. Therefore, the objectve functon for each user s ts SINR, and no longer ts SNR. Snce n the sngle user case the decson process can be descrbed as an MDP, then the scenaro wth two users can be formalzed as a stochastc game, also called compettve MDP [8], wth uncountable state space. In general, a stochastc game s an MDP n whch the nstantaneous rewards for each player and the transton probabltes among the states are controlled by the jont actons of the players n each state. In our case, the set of channels h 1,j,..., h M,j for player j evolve ndependently from the ones avalable to any other player k j, and the acton space for each player s stll A = {1,..., M},.e. the channel ndces to be selected at each slot. Therefore, we are allowed to formulate the game as a stochastc game n whch each user j controls ts own Markov chan on the state space S j. As n the sngle user case S j s the set of all the possble states (5). Formally, the state space of the stochastc game at hand s the Cartesan product S = S 1 S 2. Let us denote by π j a samplng strategy for user j and by π j the one for the other users. Possbly, π j, π j are randomzed polces. We defne the nstantaneous reward for user j n state S [t] S as the expected reward E ( SINR j [t] S [t], π, π ). Thus, the nteracton on the players occurs only on the nstantaneous rewards ganed n each state, through the SINR expresson. Thus we can say that our model s a rewardcoupled stochastc game. Ths model s very smlar wth the one dealt wth n [9], except that here the state space s uncountable and there are no constrants on the rewards. A. Heurstc Best Response We now propose a heurstc best response polcy for user 2. Suppose that user 1 s oblvous of the presence of user 2 and performs a myopc polcy π1 M to maxmze the expected average of channel attenuatons over tme, as n the sngle user case. On the other hand, user 2 knows the parameters of the channels, the current state, and the strategy of user 1. Thus, user 2 stll faces an MDP wth uncountable states, whch s equvalent to the stochastc game descrbed before, when user 1 fxes ts own statonary strategy. Let us gve an nsght on a possble strategy for user 2. Assume that, for user 2, channel 1 presents at tme t the hghest coeffcent g 1,2[t], but the expected SINR guaranteed by channel 2 wth suboptmal attenuaton s hgher, snce the nterference s much weaker. Then, t s n general not clear what user 2 should do. A myopc soluton would suggest to swtch to the free channel 2, but on the other hand, n such a way the nformaton about channel 1 becomes stale, and moreover channel 1 tself mght become free n a near future. Then, n analogy wth the sngle player case, we propose two strateges, one myopc and one randomzed, to approxmate the best response for user 2

5 r(π U ) upper bound r(π O ) optmal r(π M ) myopc r(π R ) randomzed r(π S ) statc arm upper bound best response r 2 (π 1 M,π 2 MS ) SINR myopc r 2 (π 1 M,π 2 RS ) SINR randomzed r(π) r(π) φ φ 1,2 Fgure 3. Performance of myopc and randomzed algorthm wth 3 arms (frequency bands). Arms 2 and 3 are statstcally equvalents, wth ϕ 2 = ϕ 3 = 0.3, σ2 2 = σ2 3 = 1, c 2 = c 3 = 10. Arm 1 has the same coeffcents c 1 = 10, σ1 2 = 1 as arms 2,3. ϕ 1 vares wthn [0.3; 0.98]. aganst a myopc polcy π1 M that user 1 mplements regardless of user 2 s behavor. We suppose the algorthms are ntalzed by samplng each channel once. Algorthm 3. SINR myopc polcy π MS 2 for user 2, aganst myopc polcy π M 1 for user 1. π2 MS (S [t], π1 M ) = argmax E(SINR,2 [t] S [t], π1 M {1,...,M}, ). Algorthm 4. Randomzed polcy π2 RS for user 2, aganst myopc polcy π1 M for user 1. Draw a realzaton of the random varable ξ = SINR,2 [t] (S [t], π1 M, ), for all = 1,..., M. Select π2 RS (S [t], π1 M ) = argmax ξ. =1,...,M About the performance of polces π MS, π RS, we can do smlar consderatons to the one made for the myopc and randomzed algorthms n the sngle user case. Let us explan the results llustrated n Fgure 4. We consdered 2 users and 2 channels. The nose varance s N 0 = 1 and P 1 = P 2 = 1. The channels for user 1 are almost determnstc,.e. σ1,1 2 = σ2,1 2 = 0.1 and ϕ 1,1 = ϕ 2,1 = 0.3, m 1,1 = 2, m 2,1 = 0.5. Thus user 1, that s unaware of the presence of user 2 and adopts a myopc polcy π1 M, selects channel 1 almost always. For user 2, σ1,2 2 = 0.8, σ2,2 2 = 0.4, m 1,2 = 8, m 2,2 = 3, ϕ 2,2 = 0.3. Hence, a statc strategy for user 2 would suggest not to collde and to select channel 2. Anyway, sometmes t s benefcal for user 2 to select channel 1 when ths s good enough. Indeed, for values of ϕ 1,2 approachng 1, the autocorrelaton of channel 1 for user 2 ncreases, and the randomzed polcy π RS succeeds n trackng channel 1 n the tme slots n whch ts coeffcent g s large enough to overwhelm the nterference caused by user 1. V. CONCLUSIONS We proposed two strateges to dynamcally select one out of a pool of M slow fadng channels, modeled as autoregressve Fgure 4. Best response strategy of user 2 aganst a myopc polcy for user 1. For user 1, σ1,1 2 = σ2 2,1 = 0.1 and ϕ 1,1 = ϕ 2,1 = 0.3, m 1,1 = 2, m 2,1 = 0.5. For user 2, σ1,2 2 = 0.8, σ2 2,2 = 0.4, m 1,2 = 8, m 2,2 = 3, ϕ 2,2 = 0.3. ϕ 1,2 vares wthn [0.8; 0.98]. r 2 (π1 M, π 2) s the expected long run average SINR for user 2 when user 1 adopts strategy π1 M. processes of order 1. The decson process s modeled as a restless bandt, or equvalently as a Markov Decson Process. The myopc channel selecton strategy s nearly optmal when the channels are smlarly correlated. Instead we suggest to adopt a randomzed strategy when one channel shows hgher autocorrelaton. When two users are present, they nterfere wth each other, and we model the compettve learnng process as a stochastc game. We fnally propose two ways to approxmate a best response selecton strategy for the transmtters. Acknowledgments: Ths research was supported by Agence Natonale de la Recherche, wth reference ANR-09- VERS-001, and Orange France Telecom Grant on Content- Centrc Networkng. We would lke to thank Alexey Punovsky for very helpful dscusson. REFERENCES [1] J. C. Gttns, R. Weber, and K. D. Glazebrook, Mult-armed bandt allocaton ndces. Wley Onlne Lbrary, 1989, vol. 25. [2] C. H. Papadmtrou and J. N. Tstskls, The complexty of optmal queueng network control, Mathematcs of Operatons Research, vol. 24, [3] P. Whttle, Restless bandts: Actvty allocaton n a changng world, Journal of appled probablty, pp , [4] R. Weber and G. Wess, On an ndex polcy for restless bandts, Journal of Appled Probablty, pp , [5] R. Aguero, M. Garca, and L. Mufoz, BEAR: A bursty error autoregressve model for ndoor wreless envronments, n Personal, Indoor and Moble Rado Communcatons, PIMRC IEEE 18th Internatonal Symposum on. IEEE, 2007, pp [6] N. Bäuerle and U. Reder, Markov Decson Processes wth applcatons to fnance. Sprnger Verlag, [7] W. Thompson, On the lkelhood that one unknown probablty exceeds another n vew of the evdence of two samples, Bometrka, vol. 25, no. 3/4, pp , [8] J. Flar and K. Vreze, Compettve Markov decson processes. Sprnger Verlag, [9] E. Altman, K. Avrachenkov, N. Bonneau, M. Debbah, R. El-Azouz, D. Sadoc Menasche, Constraned cost-coupled stochastc games wth ndependent state processes, Operatons Research Letters, vol. 36, no. 2, pp , 2008.