Online Advertisement, Optimization and Stochastic Networks

Transcription

1 Onlne Advertsement, Optmzaton and Stochastc Networks Bo (Rambo) Tan and R. Srkant Department of Electrcal and Computer Engneerng Unversty of Illnos at Urbana-Champagn Urbana, IL, USA 1 arxv: v6 [cs.ds] 7 Sep 2012 Abstract In ths paper, we propose a stochastc model to descrbe how search servce provders charge clent companes based on users queres for the keywords related to these companes ads by usng certan advertsement assgnment strateges. We formulate an optmzaton problem to maxmze the long-term average revenue for the servce provder under each clent s long-term average budget constrant, and desgn an onlne algorthm whch captures the stochastc propertes of users queres and clck-through behavors. We solve the optmzaton problem by makng connectons to schedulng problems n wreless networks, queueng theory and stochastc networks. Unlke pror models, we do not assume that the number of query arrvals s known. Due to the stochastc nature of the arrval process consdered here, ether temporary free servce,.e., servce above the specfed budget (whch we call overdraft ) or under-utlzaton of the budget (whch we call underdraft ) s unavodable. We prove that our onlne algorthm can acheve a revenue that s wthn O(ǫ) of the optmal revenue whle ensurng that the overdraft or underdraft s O(1/ǫ), where ǫ can be arbtrarly small. Wth a vew towards practce, we can show that one can always operate strctly under the budget. In addton, we extend our results to a clck-through rate maxmzaton model, and also show how our algorthm can be modfed to handle non-statonary query arrval processes and clents wth short-term contracts. Our algorthm also allows us to quantfy the effect of errors n clck-through rate estmaton on the 1+ acheved revenue. We show that we lose at most fracton of the revenue f s the relatve error n clck-through rate estmaton. We also show that n the long run, an expected overdraft level of Ω(log(1/ǫ)) s unavodable (a unversal lower bound) under any statonary ad assgnment algorthm whch acheves a long-term average revenue wthn O(ǫ) of the offlne optmum. I. INTRODUCTION Provdng onlne advertsng servces has been the major source of revenue for search servce provders such as Google, Yahoo and Mcrosoft. When an Internet user queres a keyword, alongsde the search results, the search engne may also dsplay advertsements from some companes whch provde servces or goods related to ths keyword. These companes pay the search servce provders for postng ther ads wth a specfed amount of prce for each ad on a pay-per-mpresson or pay-per-clck bass. We call them clents n the followng text. Maxmzng the revenue obtaned from ther clents s the key objectve of search servce provders. Research whch targets ths objectve has followed two major drectons. One s based on aucton theory, n whch the goal s to desgn mechansms n favour of the servce provder, and much of the research n ths drecton consders statc bds (e.g. [13]; see [10] for a survey), whle dynamc models such the one n [22] are stll emergng. The other s from the perspectve of onlne resource allocaton wthout consderng the mpact of the servce provder s mechansms on the clents bds, and the man focus of ths knd of research s on desgnng an onlne algorthm whch posts specfc ads n response to each search query arrvng onlne, n order to September 10, 2012 DRAFT

2 2 acheve a hgh compettve rato wth respect to the offlne optmal revenue. Our work follows the second drecton. Our model s as follows: Onlne Advertsng Model: Assume that queres for keyword q arrve to the search engne accordng to a stochastc process at rate ν q queres per tme slot, where we have assumed that tme s dscrete and a tme slot s our smallest dscrete tme unt. In response to each query arrval, the search engne may dsplay ads from some clents on the webpage. There are L dfferent places (e.g., top, bottom, left, rght, etc.) on a webpage where ads could be dsplayed. We wll call these places webpage slots. When clent s ad s dsplayed n webpage slot s when keyword q s quered, there s a probablty wth whch the user who s vewng the page (the one who generated the query) wll clck on the ad. Ths probablty, called the clck-through rate, s denoted by c qs. A clent specfes the amount of money ( bd ) that t s wllng to pay to the search servce provder when a user clcks on ts ad related to a specfc query. We use r q to denote ths per-clck payment from clent for ts ad related to a query for keyword q. Addtonally, clent also specfes an average budget b whch s the maxmum amount that t s wllng to pay per budgetng cycle on average, where a budgetng cycle equals to N tme slots (we have ntroduced the noton of a budgetng cycle snce the tme-scale over whch queres arrve may be dfferent than the tme-scales over whch budgets may be settled). The problem faced by the search servce provder s then to assgn advertsements to webpage slots, n response to each query, so that ts long-term average revenue s maxmzed. Based on the above model, we desgn an onlne algorthm whch acheves a long-term average revenue wthn O(ǫ) of the offlne optmal revenue, where ǫ can be chosen arbtrarly small, ndcatng the near-optmalty of our onlne algorthm. Before enterng nto the detals, n the next two subsectons we wll frst survey the related lterature, hghlght the man contrbutons of our work, and dscuss the dfferences between our model and prevous ones. A. Related Work We wll only survey the onlne resource allocaton models here, and not the aucton models. The onlne ads model n pror lterature manly nclude two types, namely AdWords (AW) and Dsplay Ads (DA), of whch the dfference les n the constraned resource of each clent. In the AW model, the resource s the clent s budget, whle n the DA model, the resource s the maxmum number of mpressons agreed on by the clent and the servce provder. Correspondngly, after each resource allocaton step, the resource of a clent whose ad s posted, s reduced by the bd value 1 n the AW model, or 1 mpresson n the DA model. Both of them belong to a general class of packng lnear programs formulated n [8]. Most of the pror onlne algorthms for solvng the AW and DA model respect the hard constrant on the clent s resources. One excepton s [9], where the authors argue that free dsposal of resources makes the DA model more tractable (but not necessary for the AW model). Mehta et al. [20] modeled the onlne ads problem as a generalzaton of an onlne matchng problem [16] on a bpartte graph of queres and clents. Later n [5], Buchbnder et al. showed that matchng clents to webpage slots (whether t s a sngle slot or multple slots) can be solved 1 Ths refers to the pay-per-mpresson scheme. Wth a pay-per-clck scheme, the reducton only happens f the ad s clcked.

3 3 as a maxmum-weghted matchng problem. Followng [5], a number of other onlne algorthms usng the maxmum-weghted bpartte matchng dea have been proposed n [19], [9], [6] and [8]. The algorthms n [15] and [20], whch were earler than [5], can also be regarded as maxmum-weghted matchng solutons on ths bpartte graph of clents and webpage slots. In [15], the b-matchng problem (related to the onlne ads context, bds are trvally 0 or 1 and budgets are all b) s solved by an 1 1/e compettve algorthm as b and the weghts are the remanng budgets of those clents nterested n the newly arrved query (.e., the bd equals 1). For the onlne ads problem n whch bds and budgets can have general and dfferent values, [20] (ts longer verson s [21]) uses the dscounted bds as the weghts correspondng to each clent. The dscount factor s calculated by a functon ψ(x) = 1 e x 1, of whch the nput x s the fracton of a clent s budget that has been consumed. Ther algorthm s also 1 1/e compettve, under an assumpton that bds are small compared to budgets. By takng advantage of estmated numbers of query arrvals for each keyword wthn a gven perod and modfyng the dscount factor n [20], Mahdan et al. [19] desgned a class of algorthms whch acheve a consderably better compettve rato wth accurate estmates whle stll guarantee a reasonably good compettve rato wth naccurate estmates, also assumng small bds. The algorthms n [5], [9], [6], [8] and [1], all use a prmal-dual framework to compute a maxmum-weghted matchng at each teraton, n whch the dual varables (correspondng to each clent) are used to determne the weghts. The two 1 1/e compettve algorthms n [5] and [9] update the dual varables dynamcally n ther prmal-dual type algorthms every tme a decson s made. Specfcally, each dual varable n [5], whch mplctly tracks the fracton of budget that has been spent by the correspondng clent, grows durng each teraton at a rate parameterzed by the fracton of the bd for the ncomng query n ths clent s total budget, whle [9] uses an exponentally weghted average of the up-to-date n() most valuable mpressons 2 assgned to clent as a new dual varable wth respect to ths clent. On the other hand, the three dual type learnng-based algorthms n [6], [8] and [1] acheve a compettve rato of 1 O(ǫ) based on a random-order arrval model (rather than the adversaral model n most of the earler work), assumng small bds and knowledge of the total number of queres. The man dfference between them s that [6] and [8] use an ntal ǫ fracton of queres to learn the optmal dual varables (wth respect to ths tranng set), whle the algorthm n [1] repeats the learnng process over geometrcally growng ntervals. Addtonally, the small bds condton n [1] s slghtly weaker than the condton n [6] and [8]. B. Our Contrbutons and Comparson to Pror Work As n pror work (especally [5] and [9]), our soluton reles on a prmal-dual framework to solve a maxmum-weghted matchng problem on a bpartte graph of clents and webpage slots, wth dynamcally updated dual varables whch contrbute to the weghts on the edges of the bpartte graph. However, unlke pror work, we are able to obtan a revenue whch s O(ǫ) close to the optmal revenue usng a purely adaptve algorthm wthout the need for the knowledge of the number of query arrvals over a tme perod or the average arrval rates. Our soluton s related to schedulng problems n wreless networks. In partcular, we use the optmzaton decomposton deas n [11], the stochastc performance bounds n [18] and the modelng of delay-senstve flows n [14]. Borrowng from that lterature, we ntroduce the concept of an overdraft queue. The overdraft queue measures the amount by whch the 2 In the DA model n [9], n() s defned as the maxmum number of mpressons agreed for clent. After allowng free dsposal, only the current n() most valuable mpressons assgned to clent wll be consdered.

4 4 provded servce temporarly exceeds the budget specfed by a clent. In makng the connecton to wreless networks, we defne somethng called the per-clent revenue regon, whch s related to the concept of capacty regon n queueng networks (see [11], [18]). In our context, t characterzes the revenue extractable from each clent as a functon of all the clents budgets. Our onlne algorthm exhbts a trade-off between the revenue obtaned by the servce provder and the level of overdrafts. We can further modfy our onlne algorthm so that clents can always operate strctly under ther budgets. Fnally, our algorthm and analyss naturally allow us to assess the mpact of clck-through rate estmaton on the servce provders revenue. We are able to show that our onlne algorthm acheves an overdraft level of O(1/ǫ). So a natural queston s whether ths bound s tght. We show that the overdraft for any algorthm must be Ω(log(1/ǫ)). Whle there s a gap between the upper and lower bounds, together they mply that the overdraft must ncrease when ǫ goes to zero. Ths work s related to [3], [25], [26], [24] and [12] n the context of communcaton networks. See Secton IV for a detaled survey. Besdes the revenue maxmzaton model, we also study another onlne ads model n whch the objectve s to maxmze the average overall clck-through rate, subject to a mnmum mpresson requrement for each clent. We also show that our results can be naturally extended to handle non-statonary query arrval processes and clents whch have short-term contracts wth the servce provder.. Lke the algorthm n [1], our algorthm can also be generalzed to a wder class of lnear programs wthn dfferent applcaton contexts, where the coeffcents n the objectve functon and constrants are not necessarly nonnegatve. There are two ponts of departure n our algorthm compared to exstng models: the frst one s that we assume a purely stochastc model n whch the query arrval rates are unknown. Thus, there s no need to know the number of arrvals n a tme perod as n pror models, and ths s even true for non-statonary query arrval processes. The other s that we assume an average budget rather a fxed budget over a tme horzon. Ths allows us to better model permanent clents (e.g., bg companes who do not stop advertsng) and who do not provde a fxed tme-horzon budget. Clents who advertse for a lmted amount of tme can also be handled well snce the algorthm s naturally adaptve. A mnor dfference wth respect to pror models s that our model assumes that tme s slotted. Ths can be easly modfed to assume that query arrvals can occur at any tme accordng to some contnuous-tme stochastc process. The only dfference s that our analyss would then nvolve contnuous-tme Lyapunov drft nstead of the dscrete-tme drft used n ths paper. From a theoretcal pont of vew, our analyss s dfferent from pror work whch uses compettve ratos: our model and soluton s smlar n sprt to stochastc approxmaton [4] where gradents (here the gradent of the dual objectve) are known only wth stochastc perturbatons. Ths pont of vew s essental to model stochastc traffc wth unknown statstcs. Instead of the 1 O(ǫ) compettve rato n pror work, we show that our algorthm acheves a revenue whch s wthn O(ǫ) of the optmal revenue. The O(ǫ) penalty arses due to the stochastc nature of our model. However, we do not requre assumptons such as knowledge of the total number of queres n a gven perod [19], [6], [8], [1], or nformaton of keyword frequences [19]. 3 3 It should be mentoned that another common assumpton small bds (or large budgets, large offlne optmal value ) used n [15], [20], [19], [9], [6] and [8] s not essentally dfferent from our long-term assumpton.

5 5 C. Organzaton of the Paper The rest of the paper s organzed as follows: In Secton II, we formulate an optmzaton problem nvolvng long-term averages. In Secton III, we start consderng the stochastc verson of our model and propose an onlne algorthm, whch also ntroduces the concept of overdraft queue. Performance analyss of ths onlne algorthm, whch ncludes the near-optmalty of the long-term revenue and an upper bound on the overdraft level, wll also be done n Secton III. The last two subsectons of Secton III present two extensons, namely the decsons based on estmated clck-through rates and the underdraft mechansm. In Secton IV, we derve a unversal lower bound on the expected overdraft level under any statonary algorthms for onlne advertsng. The second onlne ads model clck-through rate maxmzaton problem wth ts related extensons, algorthm desgn and analyss s gven n Secton V. Secton VI concludes the whole paper. Compared to an earler verson of ths paper whch appeared n [28], we gve a more detaled lterature survey n Subsecton I-A, all the proofs for the lemmas, theorems and corollares n Secton III (we only stated these results wthout proofs n [28] due to page lmts), and full dscussons on the underdraft mechansm n Subsecton III-F. Sectons IV and V are completely new. II. AN OPTIMIZATION PROBLEM INVOLVING LONG-TERM AVERAGES Based on the model descrbed n Secton I, we frst pose the revenue maxmzaton problem as an optmzaton problem nvolvng long-term averages. For ths purpose, we defne an assgnment of clents to webpage slots as a matrx M of whch the (,s) th element s defned as follows: { 1, f clent s assgned to webpage slot s M s = 0, else. The matrx M has to satsfy some practcal constrants. Frst, a webpage slot can be assgned to only one clent and vse versa. Furthermore, the assgnment of clents to certan webpage slots may be prohbted for certan queres. For example, t may not make sense to advertse chocolates when someone s searchng for nformaton about treatments for dabetes. These constrants can be abstracted as follows: For the quered keyword q, the set of assgnment matrces have to belong to some set M q. We also let be the probablty of choosng matrx M when the quered keyword s q. The optmzaton problem s then gven by subject to N q max p R(p) = q ν q ν q M s c qs r q (1),s M s c qs r q b, ; (2) s 0 1, q, M M q ; (3) 1, q. (4) In the above formulaton, the objectve (1) s the average revenue per tme slot and constrant (2) expresses the fact that the average payment over a budgetng cycle should not exceed the average budget. The optmzaton s a lnear program and f all the problem parameters are known, n

6 6 prncple, t can be solved offlne, returnng probabltes { } whch can be used by a servce provder to maxmze ts revenue. However, such an offlne soluton s not desrable for at least two reasons: Beng a statc approach, t does not use any feedback about the current state of the system. For example, the fact that the emprcal average payment of a clent has severely exceeded ts average budget would have no mpact on the subsequent assgnment strategy. Snce the formulaton and hence, the soluton, only cares about long-term budget constrant satsfacton, severe overdraft or underdraft of the budget can occur over long perods of tme. The offlne soluton s a functon of the query arrval rates {ν q }. Thus, a change n the arrval rates would requre a recomputaton of the soluton. In vew of these lmtatons of the offlne soluton, we propose an onlne soluton whch adaptvely assgns clent advertsements to webpage slots to maxmze the revenue. As we wll see, the onlne soluton does use feedback about the overdraft (or underdraft) level n future decsons, and does not requre knowledge of {ν q }. III. ONLINE ALGORITHM AND PERFORMANCE ANALYSIS A. A Dual Gradent Descent Soluton To get some nsght nto a possble adaptve soluton to the problem, we frst perform a dual decomposton whch suggests a gradent soluton. However, a drect gradent soluton wll not take nto the account the stochastc nature of the problem and wll also requre knowledge of the query arrval rates {ν q }. We wll address these ssues n the followng subsectons, usng technques that, to the best of our knowledge, have not been used n pror lterature on the onlne advertsng problem. We append the constrant (2) to the objectve (1) usng Lagrange multplers δ 0 to obtan a partal Lagrangan functon L(p,δ)= ν q M s c qs r q δ ν q M s c qs r b N q,s q s = q ν q,s M s c qs r q (1 δ )+ δ b N, subject to constrants (3) and (4). The dual functon s D(δ) = max ν q M s c qs r q (1 δ )+ p q,s subject to constrants (3) and (4). Note that the maxmzaton part n the dual functon can be decomposed nto ndependent maxmzaton problems wth regard to each quered keyword q,.e., for all q, max M s c qs r q (1 δ ) = max M s c qs r q (1 δ ), {, },s where t s easy to see that each maxmzaton s solved by a determnstc soluton. Ths suggests the followng prmal-dual algorthm to teratvely solve the orgnal optmzaton problem (1): at,s δ b N,

7 7 step k, q, ˆM (q,k) arg max,s [ (, δ (k +1) = δ (k)+ǫ N q M s c qs r q (1 δ (k)); ) ] ν q [ ˆM (q,k)] s c qs r q b +, where ǫ > 0 s a fxed step-sze parameter, and [x] + = x f x 0 or [x] + = 0 otherwse. Furthermore, defnng ˆQ (k) δ (k)/ǫ, the above teratve algorthm becomes ( ) 1 q, ˆM (q,k) arg max M s c qs r q ǫ ˆQ (k) ;, ˆQ (k +1) = [ˆQ (k)+ ˆλ ] +, (k) b,s s where ˆλ (k) N q ν q [ ˆM (q,k)] s c qs r q. (5) s Note that ˆQ (k) can be nterpreted as a queue whch has ˆλ (k) arrvals and b departures at step k. Although ths algorthm already uses the feedback provded by { ˆQ(k)} (or {δ(k)}) about the state of the system, t s stll usng a pror nformaton about the arrval rates of queres n {ˆλ(k)}, hence not really onlne. However, t motvates us to ncorporate a queueng system wth stochastc arrvals nto the real onlne algorthm, whch wll be descrbed n the next subsecton. B. Stochastc Model, Onlne Algorthm, and Overdraft Queue In practce, a search servce provder may not have a pror nformaton about the query arrval rates {ν q }, and generally, query arrvals durng each tme slot are stochastc rather than constant. Let tme slots be ndexed by t Z + {0}. We specfy our detaled statstcal assumptons as follows: Query arrvals: Assume that a tme slot s short enough so that query arrvals n each tme slot can be modeled as a Bernoull random varable wth occurrence probablty ν. The probablty that an arrved query s for keyword q s assumed to be ϑ q and q ϑ q = 1. Let q(t) represent the ndex of the keyword quered n tme slot t, such that q(t) = q w.p. ν q = νϑ q for all q (ndexed by postve ntegers) and q(t) = 0 w.p. 1 ν, whch accounts for the case that no query arrves. Budget spendng: We lmt the values of budget spent n each budgetng cycle to be ntegers. To match the average budget b (when t s not an nteger), the budget of clent n budgetng cyclek s assumed to be a random varable b(k) whch equals b w.p. and b otherwse, such that E[ b(k)] = b +(1 ) b = b,.e., = b b = b b b b. For the trval case that b s already an nteger, we let = 1. Clck-through behavors: In tme slot t, after a query for keyword q arrves, f the ad of clent s posted on webpage slot s n response to ths query, then whether ths ad wll be clcked s modeled as a Bernoull random varable c qs (t) wth occurrence probablty c qs. We now want to mplement the above teratve algorthm onlne based on ths stochastc model. Accordng to defnton (5), ˆλ ncludes average query arrvals and clck-through choces wthn

8 8 N tme slots (.e., one budgetng cycle). Thus, each teraton step n the onlne algorthm should correspond to a budgetng cycle. For convenence, we defne u(k) { q(t), c(t) for kn t kn +N 1} as a collecton of random varables descrbng user behavors (ncludng stochastc query arrvals and clck-through choces) n budgetng cycle k. The onlne algorthm s then descrbed as follows: Onlne Algorthm: (n each budgetng cycle k 0) In each tme slot t [kn,kn +N 1], f q(t) > 0, choose the assgnment matrx ( ) 1 M (t, q(t),q(k)) arg max M s c q(t)s r q(t) ǫ Q (k). (6) M M q(t) At the end of budgetng cycle k, for each clent, update [ +, Q (k +1) = Q (k)+a (k,q(k),u(k)) b (k)] (7) where A (k,q(k),u(k)) kn+n 1,s [ M (t, q(t),q(k))] s c q(t)s (t) r q(t). (8) t=kn s Here, A (k,q(k),u(k)) represents the revenue obtaned by the servce provder from clent durng budgetng cycle k, and recall that b (k) s a random varable whch takes nteger values whose mean s equal to the average budget per budgetng cycle. In ths algorthm, clent s assocated wth a vrtual queue Q (mantaned at the search servce provder). Durng budgetng cycle k, the amount of money clent s charged by the search servce provder A (k,q(k),u(k)) s the arrval to ths queue, and the average budget per budgetng cycle b s the departure from ths queue. Note that f ths queue s postve, t means that the total value of the real servce already provded to the clent has temporarly exceeded the clent s budget,.e., free servce has been provded temporarly. Hence, we call ths queue the overdraft queue. There are two dfferent tme scales here. The faster one s a tme slot, the smallest tme unt used to capture user behavors (ncludng stochastc query arrvals and clck-through choces) and execute ad-postng strateges. The slower one s a budgetng cycle (equal to N tme slots), at the end of whch the overdraft queues are updated based on the revenue obtaned over the whole budgetng cycle. We make the followng assumptons on the above stochastc model: { q(t)} are..d. across tme slots t; { c qs (t)} are ndependent across q,, s, and t; each varable n { q(t)} and each varable n { c qs (t)} are mutually ndependent. In fact, the model can be generalzed to allow for query arrvals correlated over tme and across keywords, and other smlar correlatons nsde the clck-through choces or between these two stochastc processes. Such models would only make the stochastc analyss more cumbersome, but the man results wll contnue to hold under these more general models. In order to guarantee that the Markov chan whch we wll defne later s both rreducble and aperodc, we further assume that the probablty of whether there s an arrval n a tme slot ν (0,1). We also assume that r q for all q and can only take nteger values. Together wth the fact that b(k) takes nteger values, {Q(k)} becomes a dscrete-tme nteger-valued queue. Note that assumng nteger values s only for ease of analyss, but not necessary.

9 9 C. An Upper Bound on the Overdraft Accordng to the ad assgnment step (6), f at the begnnng of budgetng cycle k, Q (k) > 1/ǫ, then for ths budgetng cycle, the th row of M (t,q,q(k)) s always a zero vector,.e., the servce provder wll not post the ads of clent untl Q (k) falls below 1/ǫ. Snce by assumpton the number of query arrvals per tme slot s upper bounded, for any budgetng cycle k, one can bound the transent length of each overdraft queue as below: Q (k) 1 ǫ +N argmax q,s {r qc qs } b,. Therefore, Q (k) O(1/ǫ) for all, and stablty s not an ssue for these upper bounded queues. It further mples that ths onlne algorthm satsfes the budget constrants n the long run,.e., for all clent, lm E K [ 1 K K 1 k=0 A (k,q(k),u(k)) ] b (9) must hold. It should be mentoned that n [12], through usng the LIFO queueng dscplne, the authors show an O((log(1/ǫ)) 2 ) bound on the averaged watng tme encountered by most of the packets, whch s tghter than the bound O(1/ǫ) under the FIFO queueng dscplne (see e.g. [11]; our above result also fts ths bound). Whle the length of a FIFO queue s proportonal to the arrval rate accordng to Lttle s law [2], the length of a LIFO queue n [12] s stll O(1/ǫ), even f t s occuped by very old packets whch only accounts for a neglgble fracton O(ǫ log(1/ǫ) ) of all the packets that have arrved. Unlke n a communcaton network where watng tme s usually the man concern and droppng a small fracton of old packets does almost no hurt to many onlne applcatons, what clents of onlne advertsng servce care about s how much they have pad beyond ther budgets, whch s measured by the overdraft queue n our model. D. Near-Optmalty of the Onlne Algorthm We now show that, n the long term, the proposed onlne algorthm acheves a revenue that s close to the optmal revenue R(p ) (where p s the soluton to the optmzaton problem (1)). We start wth the followng lemma: Lemma 1: Consder the Lyapunov functon V(Q) = 1 2 Q2. For any ǫ > 0, and each tme perod k, E[V(Q(k +1)) Q(k) = Q] V(Q) N ( R(p ) ǫ R( p (k,q)) ) +B 1 B 2 Q. Here, B 1 1 ( (N(N 1)L 2 +NL)(argmax 2 {c qsr q }) 2 q,,s + ) b 2 (b b )+ b 2 (1 b + b ), (10) where L s the number of webpage slots; B 2 mn {b N q ν q p qm s M s c qs r q }; (11)

10 10 and p (k,q) { p qm (k,q), q,m M q} where p qm (k,q) equals 1 f M = M (t,q,q) for kn t kn +N 1 (.e., the optmal matrx n the maxmzaton step (6)) and 0 otherwse. The proof s gven n Appendx A. Now we are ready to present one of the major theorems n ths paper, ndcatng that the long-term average revenue acheved by our onlne algorthm s wthn O(ǫ) of the maxmum revenue obtaned by the offlne optmal soluton. The proof s gven n Appendx B. Theorem 1: For any ǫ > 0, 0 lm K E [ ] R(p ) 1 K 1 R(k) KN k=0 B 1ǫ N for some constant B 1 > 0 (defned n (10) n Lemma 1), where R(k) A (k,q(k),u(k)). s defned as the revenue obtaned durng budgetng cycle k. Remark 1: If we choose a very small ǫ, the matchng n (6) behaves lke a greedy soluton untl the queue lengths grows comparably large. Ths ndcates a tradeoff between how close to the long-term optmal revenue the algorthm can acheve and the actual convergence tme. Addtonally, supposng that {r q } and {b } are both measured n another scale wth a factor α, e.g., usng cents nstead of dollars (α = 100), and assumng that α s unknown, t can be shown that the O(ǫ) convergence bound wll also be scaled by α f we measure the revenue n the orgnal scale. To change the algorthm nto a scale-free verson, {r q } and {b } should be dvded by a common benchmark value, e.g., the largest budget specfed by all the ntally exstng clents. Snce the benchmark value s also mplctly multpled by α f measured n another scale, the scalng factor wll be canceled n the normalzed {r q } and {b } and no longer affect the convergence bound. E. Impact of Clck-Through Rate Estmaton In our onlne algorthm, the decson of pckng an optmal ad assgnment matrx n (6) n response to each query s based on the true clck-through rates c. In realty, an estmate ĉ based on hstorcal clck-through behavors s used,.e., n response to each query for keyword q, whch arrves n tme slot t [kn,kn +N 1], we choose the assgnment matrx ( ) 1 M (t, q(t),q(k)) arg max M s ĉ q(t)s r q(t) ǫ Q (k). (12) M M q(t) We then have the followng corollary n addton to Theorem 1 n Subsecton III-D: Corollary 1: Assume that the estmated clck-through rates ĉ [c(1 ),c(1 + )] wth some (0, 1). Under our onlne algorthm wth estmated clck-through rates, Q(k) s stll postve recurrent. Then, for any ǫ > 0, lm E K [ 1 KN K 1 k=0 R(k) ],s ( ) 1 1+ R(p ) B 1ǫ N, for some constant B 1 > 0 (defned n equaton (10) n Lemma 1).

11 11 Provng ths needs some mnor changes to the proof of Lemma 1 and Theorem 1, whch wll be shown n Appendx C. Remark 2: Corollary 1 tells us that for small ǫ, the long-term average revenue acheved by our onlne algorthm wth estmated clck-through rates wll be at least ( 1 1+ ) of the offlne optmal revenue. F. Underdraft: Stayng under the Budget In the prevous sectons, we allowed the provson of temporary free servce to clents, whch we call overdraft. If ths s not desrable for some reason, the algorthm can be modfed to have non-postve overdraft. We do ths by allowng the queue lengths to become negatve, but not postve. The practcal meanng of negatve queue lengths s to allow each clent to accumulate a certan volume of credts f the current budget s under-utlzed and use these credts to offset future possble overdrafts. We call ths negatve queue length underdraft. Correspondng to ths mechansm, we modfy our onlne algorthm as follows: n response to each query for keyword q, whch arrves n tme slot t [kn,kn +N 1], choose the assgnment matrx M (t, q(t),q(k)) arg max M M q(t) and at the end of budgetng cycle k, for each clent, update,s M s c q(t)s r q(t) (Γ Q (k)), Q (k +1) = max{q (k)+a (k,q(k),u(k)) b (k), C }, where Γ denotes a customzed throttlng threshold (not necessarly 1/ǫ) and C denotes the maxmum allowable credt volume for clent. Recall that A (k,q(k),u(k)) s defned n equaton (8). We can bound each overdraft queue as below: Q (k) Γ +N argmax q,s {r qc qs } b,,k. Thus, f our objectve s to elmnate overdrafts (.e., Q (k) 0 for all k), we can set [ ] Γ := b N argmax {r qc qs },, (13) q,s where n contrary to [x] +, [x] takes the non-postve part of x,.e., [x] = x f x 0 or [x] = 0 otherwse. We further let C := 1 ǫ Γ,, so that after convertng Q (k) to be nonnegatve by usng Q (k) = Q (k)+c for all, everythng s transformed back to the orgnal onlne algorthm except that each Q (k) s replaced by Q (k), hence we can stll show that the revenue acheved by ths modfed verson of onlne algorthm s wthn O(ǫ) of the optmal revenue. It mght seem counter-ntutve that by lettng ǫ go to zero, we can ncur potentally large underdrafts (under-utlzaton of the budget) and yet are able to acheve maxmum revenue. Ths s not a contradcton: for each fxed ǫ, n the long term, the average servce provded to each clent s close to the average budget. The O(1/ǫ) s a fxed amount by whch the total budget

12 12 w(k) w(k) ε = 0.01 clent 1 clent k ε = clent 1 clent k Fg. 1: Temporary unfarness n servce up to any tme T s under-utlzed, and, after dvded by T, t goes to zero when T approaches nfnty. We note that whle an underdraft does not seem to sgnfcantly hurt ether the clent, who actually benefts from an underdraft, or the servce provder, whose long-run average revenue s stll dmnshed only by O(ǫ), large values of the underdraft may result n temporary unfarness n the system. 4 If, for example, a clent accumulates a large underdraft compared to the other clents, then t may receve prorty over other clents for large perods of tme. To llustrate ths, we consder an example wth two clents and one quered keyword. Assume that Γ < 0 for = 1,2, and at tme slot k 0, Q 1 (k 0 ) = Γ 1 and Q 2 (k 0 ) = C 2 (ths occurs wth a postve probablty due to the ergodcty of the Markov chan {Q(k)} proved before). We smulate the sample paths of the weghts n the maxmzaton step (32) wth the followng settng: budgets b 1 = b 2 = 0.6, clck-through rates c 1 = c 2 = 0.5, revenue-per-clck r 1 = r 2 = 1; the number of query arrvals per tme slot equals 2 w.p. 0.5 and 0 otherwse; a budgetng cycle equals to one tme slot (N = 1) for smplcty. The results for both ǫ = 0.01 and ǫ = (k = 0 corresponds to k 0 here) are shown n Fgure 1. Clent 2 keeps gettng servces untl the weghts of both clents reaches the same level, and the smaller ǫ s, the longer the unfar servng perod lasts. It should be mentoned that ths underdraft dea can be used under any upper-bounded query arrval model, not restrcted n the Bernoull arrval model consdered n ths paper. IV. A UNIVERSAL LOWER BOUND ON THE EXPECTED OVERDRAFT LEVEL We want to show that n the long run, an expected overdraft level ofω(log(1/ǫ)) s unavodable under any statonary ad assgnment algorthm whch acheves a long-term average revenue wthn O(ǫ) of the offlne optmum, when the queue length s only allowed to be nonnegatve. An ad assgnment algorthm s defned as a strategy whch uses matrx M (t,q) M q for ad 4 Note that ths temporary unfarness s not an artfact of the underdraft mechansm. In fact, t occurs once a sample path enters a state where some clents have huge dfferences from others n ther correspondng queue lengths, whch can also happen under the orgnal algorthm. We are just usng the underdraft scheme to llustrate ths phenomenon.

13 13 assgnment when a query for keyword q arrves at each tme slot t. Durng each budgetng cycle k, the revenue obtaned from clent under algorthm s defned as A (k) kn+n 1 t=kn [M (t, q(t))] s c q(t)s (t) r q(t). (14) s We then defne average revenue obtaned from clent per budgetng cycle as λ E[A (k)] n the steady state. The long-term average revenue (per tme slot) s thus R = λ /N, and the overdraft level of clent evolves as Q (k +1) = [Q (k)+a (k) b (k)] +. (15) Note that our onlne algorthm s one partcular, whch makes the decson based on the current overdraft levels of all clents. To seek a unversal lower bound on expected overdraft level n the long run (here, equvalent to steady state), we only have to consder those algorthms such that Q E[Q (k)] < for all. To categorze these stable algorthms, we defne per-clent revenue regon, smlar to the concept of capacty regon n the context of queueng networks: Defnton 1 ( Per-Clent Revenue Regon ): { } C λ ={λ } 0: s.t. λ E[A (k)] b,, gven fxed parameters {r q }, {b }, {c qs }, N and statstcal propertes of q(t) and { c qs (t)}. The offlne optmal average revenue s then equal to max λ C λ /N, whch s denoted as R. Note that f the query arrval rates per budgetng cycle are too low, the average revenue drawn from some clent wll never ht ts specfed budget, no matter whch algorthm s.t. λ C you pck (.e., s.t. no feasble soluton p can make constrant (2) for ths tght). The system resources (here, budgets) are underutlzed and t s not so mportant to consder the tradeoff between revenue and overdraft. To avod ths, we can assume a relatvely large N (.e., the number of tme slots n one budgetng cycle) such that { } N max b q ν qr q max M M q c qs(,m), (16) where M q M q s defned as a set of ad assgnment matrces, of whch the th row has a 1, and s(,m) n c qs(,m) refers to the column n M where that 1 stays. Ths guarantees that for each, there exsts an algorthm such that λ C and λ = b. The reason s that In the followng text, we wll assume the above condton for N. A. One Keyword, One Clent and One Webpage Slot We start wth the smplest model: one keyword, one clent and one webpage slot (hence we omt all the subscrpts n the correspondng notatons). Under condton (16), the offlne maxmum average revenue s trvally b/n.

14 14 Theorem 2: Gven a small ǫ > 0, f an algorthm leads to E[A (k)] b ǫ n the steady state, then log(1/ǫ) Q 2(1 log(ϕp + )) 1, where we assume that and P + Pr( b(k) > 0) > 0. ϕ Pr(no query arrval n a budgetng cycle) > 0, Note that ths result works for any query arrval and budget spendng model satsfyng the above two stated assumptons, and not only restrcted to the model we descrbed n Subsecton III-B. In the proof below, we generally wrte b(k) as a random varable whch can possbly take all nonnegatve nteger values. Proof: We gnore the superscrpt for brevty. The dynamcs of the queue s rewrtten as Q(k +1) = Q(k)+A(k) ˆb(k), where the actual departure process s defned as { b(k) f Q(k)+A(k) b(k) 0; ˆb(k) (17) Q(k) + A(k) otherwse. Let p Pr(ˆb(k) = ) and q Pr( b(k) = ) n the steady state. Note that b ǫ E[A(k)] = E[ˆb(k)] = Pr(ˆb(k) ) = Pr(ˆb(k) 1)+ Pr(ˆb(k) ) (a) (1 p 0 )+ = q 0 p 0 +b, =1 Pr( b(k) ) = 1 p 0 + =2 ( b Pr( b(k) 1) =2 ) = 1 p 0 +b (1 q 0 ) where (a) holds because Pr(ˆb(k) ) Pr( b(k) ) for all 0. Thus, p 0 q 0 +ǫ. Snce Pr(ˆb(k) = 0) = Pr( b(k) = 0)+Pr(ˆb(k) = 0, b(k) 1), we have p 0 = q 0 + p 0, where p 0 Pr(ˆb(k) = 0, b(k) 1). Therefore, p 0 ǫ. (18) Next, we are lookng for a lower bound on p 0 n relaton to Q. Lettng P + Pr( b(k) > 0) (whch s surely a postve constant snce b > 0), we then have ( n 1 n 1 ) n p 0 = Pr(ˆb(k) = 0, b(k) > 0) (a) Pr {ˆb(k) = 0, b(k) > 0} k=0 k=0 (b) Pr(Q(0) n 1; A(k) = 0, b(k) > 0, 0 k n 1) n 1 = Pr(Q(0) n 1) Pr(A(k) = 0) Pr( b(k) > 0) k=0 = (ϕp + ) n Pr(Q(0) n 1) (c) (ϕp + ) n( 1 Q/n ), (19)

15 15 where (a) holds accordng to the unon bound, (b) holds snce the event on the RHS mples the one on the LHS, and (c) holds due to the Markov nequalty. If we pck n := 2 Q [2 Q,2 ( Q+1 ) ], nequalty (19) further mples that p 0 (ϕp +) n 2n (e) e n(1 log(ϕp +)) e 2( Q+1)(1 log(ϕp + )), (20) where (e) holds because 1 2x e x for all x > 0. Combnng nequaltes (18) and (20) then completes the proof. In the related lterature, [3] comes up wth an Ω(1/ ǫ) bound for a set of algorthms under some admssblty condtons, whle [25] provdes an Ω(log(1/ǫ)) bound for more general algorthms. Our proof uses the followng deas nspred by [25]: f the throughput s lower bounded by a number close to the average potental departure rate, then the probablty of zero actual departures gven nonzero potental departures must be upper bounded by a small number; further, f the average queue length s gven, then the probablty of httng zero must be upper bounded because otherwse, the queue length would become small. However, we cannot drectly use the expresson for the lower bound n [25] snce t mposes certan strct convexty assumptons whch do not apply to our model where the objectve s lnear. So we have provded a very smple dervaton of the lower bound on the queue length for our specfc model. Addtonally, our Ω(log(1/ǫ)) bound based on a lnear objectve functon can be extended to the mult-queue case (n Subsecton IV-B). The Ω(1/ ǫ) bound n [3] has been extended to the mult-queue case n [24] but stll under strct convexty assumpton and for a restrctve class of algorthms. Whether the Ω(log(1/ǫ)) bound n [25] can be easly extended to multple queues stll remans a queston. B. Multple Keywords, Multple Clents and Multple Webpage Slots We now extend ths lower bound to the orgnal general model, whch can have multple keywords, multple clents and multple webpage slots. It s easy to see that the per-clent revenue regon C n Defnton 1 s a polytope, whch can then be rewrtten as { } C = λ 0 : h (n) λ d (n), 1 n L, (21) where h (n) 0 and d (n) > 0 for all and n.the outer boundary of the polytope C conssts of the L hyperplanes,.e., h(n) λ = d (n) for all n [1,L]. Under condton (16), L s at least equal to the number of clents (.e., number of budget constrants), so (21) gves a more precse descrpton of the stablty condton for ths multqueue system, compared to the orgnal defnton of C. Thus, correspondng to the normal vector of each hyperplane, we convert the orgnal mult-queue system nto a new one wth L queues: For each n [1,L], we frst scale the th queue descrbed n (15) by h (n), so that t has a queue length equal to h (n) Q (k), wth h (n) A (k) arrvals and h (n) b (k) potental departures n tme slot k, for all. Next, we treat h(n) Q (k) as the n th queue, and snce any λ C satsfes h(n) λ d (n), ts maxmum achevable average departure rate equals d (n), where d (n) h(n) b, because the potental departure rate of each ndvdual scaled queue may not be fully acheved when all of them are coupled together.

16 16 We then come up wth the formal defnton of the class of algorthms whch acheves a nearoptmal average revenue. Defnton 2 ( ǫ-neghbourhood of the maxmum): Let λ be one optmal pont n C such that λ = R. The ǫ-neghbourhood of λ s defned as N ǫ {λ C \ C : 0 < N ( R R ) ǫ}, (22) where C represents the outer boundary of C, and t should be noted that the average revenue s evaluated per tme slot whle λ s evaluated per N tme slots. Note that n the above defnton, snce λ N ǫ s not on any boundary, R s strctly larger than R, whch s easy to see from some basc prncples of lnear programmng. The followng theorem shows the unversal lower bound Ω(log(1/ǫ)) for the general case. Theorem 3: For any algorthm s.t. λ N ǫ, we have M =1 Q log(1/ǫ) C 2 C 1 1, where ϕ Pr(no query arrval n a budgetng cycle) = (1 ν) N > 0, P + Pr( b (k) > 0, ) > 0, and C 1 2(1 log(ϕp + )) max,n h(n) (0, ), C 2 max{log(max,n h(n) ),0} [0, ). (23) Proof: We gnore the superscrpt for brevty. Accordng to some basc prncples of lnear programmng, an optmal pont λ s at a corner of C. If there are several optmal ponts, any convex combnaton of them s also optmal. Denote ths optmal pont sets as Λ and λ Λ, n [1,L], s.t. ) h(n λ = d(n ). Gven a λ N ǫ, θ s.t. θ = λ and θ λ for all (but at least one nequalty s strct). Besdes, for ths θ, ñ [1,L], s.t. h(ñ) θ d (ñ) (otherwse, θ C \ C wll hold and hence θ < λ, whch leads to a contradcton). Therefore, d (ñ) h (ñ) λ = h (ñ) max h (ñ) (θ λ ) (a) h (ñ) max (θ λ ) (λ λ ) h (ñ) maxǫ, (24) where h (ñ) max max h (ñ) > 0 and nequalty (a) holds because θ λ for all. Lettng P + Pr( h(ñ) b (k) > 0), t s easy to see that P + Pr( b (k) > 0, ) = P + > 0. Together wth Theorem 2, we can conclude that h (ñ) Q log(1/ǫ) log(h(ñ) max) 2(1 log(ϕp +)) 1 log(1/ǫ) log(h(ñ) max) 1. 2(1 log(ϕp + ))

17 17 Fg. 2: An llustraton of the dea n the proof of Theorem 3 Snce h(ñ) Q h max (ñ) Q, t s further concluded that Q log(1/ǫ) log(h(ñ) max) 2h max(1 log(ϕp (ñ) + )) 1 log(1/ǫ) C 2 1, C 1 where the unversal constants are defned n (23), and t s guaranteed that C 1 (0, ) and C 2 [0, ). Ths completes the proof. Remark 3: We brefly explan the dea behnd choosngθ n the above proof: For thoseλ N ǫ such that λ λ for all (at least one s strct), θ can be drectly chosen as λ to make nequalty (a) n (24) hold. But for the other λ N ǫ whch do not satsfy the above condton, t s necessary to ntroduce a θ other than λ, whch both les on the maxmum revenue lne (.e., θ = λ ) and domnates λ component-wse, n order to derve nequalty (24). Note that θ s not unque and furthermore, θ les ether on C or n the exteror of C and t can be chosen as a boundary pont only f the optmal revenue pont s not unque. Fgure 2 llustrates ths dea usng an example wth one keyword, two clents and one webpage slot, specfcally for showng where such a θ s located. The basc dea n our proof s to use Theorem 2 to frst get a lower bound for those new sngle queues wrtten as a weghted sum of the orgnal queues (descrbed above). Ths dea s smlar to one part n the proof for the lower bound on the expected queue length of a departurecontrolled mult-queue system n [26], but some technque n ther proof cannot drectly apply to arrval-controlled queues lke ours. C. Tghtness of the Lower Bound We want to show that the Ω(log(1/ǫ)) unversal lower bound s tght,.e., achevable by some algorthms. Consder the followng smple queueng model: the arrval process a(k) s..d. across tme, a(k) = 2 w.p. ν and a(k) = 0 otherwse. The servce rate s constant and equal

18 18 to 1. Assume that ν (1/2,1). Wth the controlled arrval process â(k), we want to acheve a throughput E[â(k)] 1 ǫ for a gven small ǫ > 0. A threshold polcy based on a threshold T s proposed below: When Q(k) > T, reject all arrvals; When Q(k) = T, accept one arrval w.p. p 1, accept two arrvals w.p. p 2, and reject all of them otherwse. When Q(k) < T, accept all arrvals. Defnng π as the steady-state probablty that Q(k) = (0 T + 1) for the resultng Markov chan, the local balance equatons are gven below: π ν = π +1 (1 ν), 0 T 2; π T 1 p 1 ν = π T (1 (p 1 +p 2 )ν); π T p 2 ν = π T+1 ; T+1 π = 1. (25) =0 Combnng these equatons wth the throughput requrement, we get [ ] T 1 ν 2 π +π T (2p 2 +p 1 ) = 1 ǫ, (26) and one can fnally show that (gnorng detaled calculatons) where =0 T = log(1/ǫ)+logc(ǫ) log ( ), ν 1 ν C(ǫ) (2ν 1+ǫ)(1 ν(p 1 +p 2 )). ν(2 2(1 ν)p 2 p 1 ) The above result further mples that Q Θ(log(1/ǫ)). we can also see that as ν 1, T 0, whch s consstent wth the fact the lower bound gven n Theorem 2 goes to 0 as the zero arrval probablty ϕ 0. Another example showng the tghtness of an Ω(log(1/ǫ)) bound s the dynamc packet droppng algorthm n [25] (note that ths unversal lower bound s proved based on a strct convexty assumpton as mentoned before n Subsecton IV-A). V. CLICK-THROUGH RATE MAXIMIZATION PROBLEM In ths secton, we consder another onlne ads model, n whch the objectve s to maxmze the long-term average total clck-through rate of all queres. Instead of average budget, clent specfes n the contract an average mpresson requrement m, whch s the mnmum number of tmes an ad of ths clent should be posted by the servce provder per requrement cycle (equal to N tme slots) on average. The other parameters are the same as n the model proposed n Secton I for the revenue maxmzaton problem. The correspondng optmzaton formulaton now becomes max p F J(p) = q ν q M s c qs (27),s

19 19 where the feasble set F s characterzed by N ν q M s m, q s ; (28) 0 1, q, M M q ; (29) 1, q. (30) Dfferent from the revenue maxmzaton problem, here the feasble set can become empty f some m s too hgh. Bascally, wthout constrant (28), F s relaxed to F 0 {p : 0 1, q,m M q ; 1, q}. (31) We can then defne the followng capacty regon whch characterzes how large the average number of mpressons can be acheved for each clent per requrement cycle: C µ : µ =N ν q M s,, s.t. p F 0. q s Clearly, m C must hold to ensure the exstence of a soluton for the above optmzaton problem. Through a smlar approach as n Subsecton III-A, we can wrte down a smlar onlne algorthm based on the same stochastc model as defned n Subsecton III-B. We defne q(k) { q(t), for kn t kn + N 1}. Smlar to b (k), m(k) = m w.p. m m and m(k) = m otherwse. Onlne Algorthm: (n each requrement cycle k 0) In each tme slot t [kn,kn +N 1], f q(t) > 0, choose the assgnment matrx ( ) M c q(t)s (t, q(t),q(k)) arg max M s +Q (k). (32) M M q(t) ǫ At the end of requrement cycle k, for each clent, update where Q (k +1) = [Q (k)+ m(k) S (k,q(k),q(k))] +, S (k,q(k),q(k)) kn+n 1 t=kn,s [ M (t, q(t),q(k))] s. (33) s In real onlne advertsng busness, some clents may only have short-term contracts,.e., clents may not be nterested n the average number of mpressons per tme slot but may be nterested n a mnmum number of mpressons n a gven duraton (such as a day). Further, query arrvals may not form a statonary process. In fact, they are more lkely to vary dependng on the tme of day. These extensons are consdered n Appendx E. Such extensons also make sense for the revenue maxmzaton model consdered n the prevous sectons, but the approach s smlar to Appendx E and so wll not be consdered here.

20 20 A. Performance Evaluaton S (k,q(k),q(k)) defned n (33) represents the actual number of mpressons for clent s ads durng requrement cycle k. The queue length ncreases when the average mpresson requrements n a partcular requrement cycle cannot be fulflled. Hence, a postve queue represents accumulated credts, whch enhances the chance of beng assgned wth a webpage slot n the future, much lke a negatve queue n the revenue maxmzaton problem. We thus call ths queue a credt queue. Unlke the revenue maxmzaton problem n whch an O(1/ǫ) upper bound on the transent queue length s automatcally mposed by the onlne algorthm, here we need to prove the stablty of the queues and show an upper bound on the mean queue length. Snce {Q(k)} defnes an rreducble and aperodc Markov chan, n order to prove ts stablty (postve recurrence), we wll frst bound the expected drft of Q(k) for a sutable Lyapunov functon. Lemma 2: Consder the Lyapunov functon V(Q) = 1 2 requrement cycle k, Here, D Q2. For any ǫ > 0 and each E[V(Q(k +1)) Q(k) = Q] V(Q) D 3 ǫ +D 1 D 2 Q. (34) ( N(N 1)L 2 +NL+ where L s the number of webpage slots; D 2 mn {N q for some ˆp F such that D 2 > 0; and where F 0 s defned n (31). ) m 2 (m m )+ m 2 (1 m + m ), (35) ν q ˆ M s m }, (36) s D 3 N max p F 0 J(p) (37) The proof s smlar to the proof of Lemma 1 wth some modfcatons n the fnal steps, whch wll be gven brefly n Appendx D. Wth ths lemma, we can conclude that Q(k) s postve recurrent because the expected Lyapunov drft s negatve except for a fnte set of values of Q(k), accordng to Foster-Lyapunov theorem ([2], [23]). Remark 4: Note that compared to the defnton of B 2 n (11) of Lemma 1 where B 2 0, D 2 needs to be strctly postve n order to prove the stablty of queues. Such a ˆp n the defnton of D 2 can always be found unless F s a degenerate set wth at most one element. The stablty of the queues drectly mples the followng corollary: Corollary 2 (Overservces n the long term): [ ] lm E 1 K S (k,q(k),q(k)) m,. K K k=1

21 21 In addton to provng stablty, Lemma 2 wll be used to evaluate the upper bound on the expected total queue length n the steady state, as shown n the followng theorem: Theorem 4: Under the onlne algorthm, [ ] E Q ( ) 1 D2 ( D 1 + D ) 3, (38) ǫ where D 1 and D 3 are respectvely defned n (35) and (37); D 2 s defned as where D 2 s defned n (36) (regarded as a functon of p). D 2 max p F 0 D 2 (p). (39) Proof: Averagng both sdes of nequalty (34) over 0 k K 1, takng K and dong some smple algebra, one obtans [ ] K 1 1 ( lm sup E Q (k) 1D2 D 1 + D ) 3. K K ǫ k=0 The LHS equals to E[ Q ( )] accordng to Theorem n [23]. The RHS s mnmzed through maxmzng D 2 over all p F 0 (whch wll certanly satsfy p F and D 2 > 0). Ths completes our proof. The followng theorem shows that the onlne algorthm proposed above acheves a long-term average clck-through rate wthn O(ǫ) of the offlne optmum. The proof s smlar to the one for Theorem 1 and hence wll be omtted. Theorem 5: For any ǫ > 0, 0 lm K E [ J(p ) 1 K 1 J(k) KN k=0 ] D 1ǫ N, for some constant D 1 > 0 (defned n (35) n Lemma 2). Here, J(k) s defned as the total number of clck-through events wthn requrement cycle k. B. Customzng Impresson Requrements {m } Based on Query Arrval Rates {ν q } Snce a postve queue measures how much the servce provder owes a clent, reducng the coeffcent of the 1/ǫ term n the upper bound on the mean queue length becomes mportant. Besdes, we also need to guarantee m C. In order to handle these two ssues, we ntroduce an approach to customzng {m } based on known (or estmated) query arrval rates {ν q }, Replacng D2 n Theorem 4 by a commond 2 defned n equaton (36), f we want the expected total queue length to be upper bounded by Q max, t suffces to let D 2 ξ 1 Q max ( D 1 + D 3 ǫ ), (40)