6.896 Topics in Algorithmic Game Theory February 16, Lecture 4

Transcription

1 6.896 opcs n Algorthmc Game heory February 6, 200 Lecture 4 Lecturer: Constantnos Daskalaks Scrbe: Jason Bddle, Alan Deckelbaum NOE: he content of these notes has not been formally revewed bhe lecturer. It s recommended that they are read crtcally. Introducton In ths lecture we cover the followng topcs: Hedgng Usng Learnng n Zero-Sum Games hs s our last lecture on zero-sum games. he reason we have spent several lectures on zero-sum games s that zero-sum games are one of the few cases n the socal scences where we re farly confdent about our mathematcal predctons of how agents behave recall Aumann s quote from the frst lecture. In ths lecture, we begn by fnshng our dscusson on hedgng and expert algorthms, and then relate ths topc to convergence of dstrbuted algorthms to equlbra n zero-sum games. 2 Expert Algorthms Recall our setup from last lecture. here are n experts, and at each tme-step t =, 2, 3... there s an assocated loss vector l t 0, n whch assgns a loss value to each expert. In each tme-step, we pck a probablty dstrbuton P t over the n experts. Our loss obtaned up to s gven by L := l t P t. As a benchmark for our algorthm s performance, we compare L aganst the performance of the sngle best expert over these rounds. hat s, our benchmark s mn l t. For notatonal purposes, we wll refer to the quantty lt as L. As mentoned n the last lecture, the follow-the-leader algorthm (at each tme τ, choose an expert whch has mnmal L τ value can be worse than our benchmark by a factor of n. In ths lecture, we wll show that the multplcatve weghts update algorthm or hedgng algorthm has a sgnfcantly better performance guarantee. 2. Hedgng Algorthm (Multplcatve Weghts Update Algorthm We now recall the hedgng algorthm ntroduced last lecture. At each tme-step, we wll mantan a weght vector w t assgnng a weght to each expert. Our probablty dstrbuton P t wll smply assgn probabltes to experts proportonal to ther weghts: 4-

2 P t = wt w t. o update the weghts each round, we smply set w t w t u b (l t where u b s any functon whch satsfes the followng condtons: b (0, For all x 0,, we have b x u b (x ( bx We now have the followng performance guarantee of the multplcatve weghts update algorthm: heorem. For all l, l 2,..., l t,... and all t, we have Proof: L t (mn L t ln (/b + ln n b b. We wll defne a potental functon at tme t to be ln ( n = wt. We compute n = w t+ = n (w t u b (l t = n (w( t ( bl t. We now note that w t = pt wt, and hence the rght-hand sde above s just ( = (p t ( ( bl t. We now take the natural log of both sdes to obtan ( n ( n ln ln + ln ( ( bp t l t. = w t+ Snce ln( x x, we have: ( n ln = w t+ w t = ln w t ( n = w t = ( bp t l t. Summng both sdes of the nequalty from t = to and cancellng the terms whch appear on both sdes yelds: ( n ( n ln ln ( bl. = w + We can ntally dstrbute the weghts unformly, so that (for example w = n for each. We notce that n ths case, ln ( w = 0. herefore, we have L ln ( n + = w. b By monotoncty of the negatve log functon, we see that the for any partcular, we have = w L + ln(w. b 4-2

3 We now observe that our update rule w t+ w tu b(l t combned wth the nequalty u b(x b x mples that w + w bl b l 2 b l and hence w + w bl = n bl. herefore, we see that for all, we have L ln( n blt b Snce was arbtrary, the proof s complete. = ln(n b Lt ln(b b. If we set b = ɛ for some ɛ (0, /2, our above bound becomes ɛ L (mn L ln( ɛ + ln(n. ɛ Usng the standard nequalty ln( z z + z 2 for all z (0, /2, we obtan L mn L ( + ɛ + ln(n. ɛ Suppose that we know the tme horzon n advance. hen we can set ( ln(n ɛ = mn n, 2 to obtan the bound herefore, we can bound the average loss by L mn L + 2 ln(n. L mn L 4 ln(n. Even f we do not know the fnal tme horzon n advance, we can use a doublng trck to obtan a smlar bound. he dea behnd ths trck s to start by choosng ɛ for a of 2. If the tme horzon exceeds 2, we now select a new ɛ correspondng to = 4. If ths tme horzon s surpassed, we select a new ɛ correspondng to = 8, and so on. Instead of the above doublng trck, we could also change ɛ wth each step. By settng ɛ to have a form such as ɛ t = mn ( ln(n t, 2, we can do slghtly better than we dd wth the doublng trck. 2.2 ghtness of the Multplcatve Weghts Bound We now look at how close the performance of the multplcatve weghts update method s to the optmal learnng algorthm. We wll argue that our bound s asymptotcally close to optmal, by gvng two examples Example Suppose we have n experts. In each round, an expert wll ether receve a loss of 0 or a loss of (that s, l t {0, } n. he losses are assgned accordng to the followng random process: At t =, select a random subset S n of sze n/2. Assgn loss 0 to the experts n S, and assgn loss to the experts n S. At t = 2, select a random subset S 2 S of sze n/4. Assgn loss 0 to the experts n S 2, and assgn loss to the remanng experts. 4-3

4 At t = 3, select a random subset S 3 S 2 of sze n/8. Assgn loss 0 to the experts n S 3, and assgn loss to the remanng experts. Contnue the above process up untl t = log 2 n. At each step t a total of n/(2 t experts wll have loss 0. It s clear that, after t = log 2 n, the best expert wll have loss 0. Furthermore, t s clear that any learnng algorthm A wll have expected performance El t A 2. (All of the experts from S t wll have loss, and every expert from S t wll have expected loss /2 at tme t. herefore, for any learnng algorthm A, we see that EL A /2, where log 2 n. Snce log 2 n, our above bound for the multplcatve weghts learnng algorthm of 2 ln n s wthn a constant factor of the best possble performance of log 2 n Example 2 he above example had a bounded tme horzon of = log 2 n. However, we can also provde an example wth an unbounded tme horzon. In ths example, we have 2 experts. At everme t, we choose l t to be ether (0, or (, 0 unformly at random. (hat s, we unformly at random select one expert to receve pont of loss, and the other expert receves no ponts of loss. It s obvous that every learnng algorthm A wll have EL A /2 (snce both experts have an expected loss of /2 at each round. Our benchmark (the loss of the best expert at tme wll be, wth constant probablty, 2 Ω(. (hs bound comes from the thought experment of flppng a far con tmes and estmatng the mnmum of the number of heads and the number of tals. We know that, wth sgnfcant probablty, the average number of heads wll be wthn a few standard devatons of the average number of tals. hus, the number of heads and the number of tals should each be, wth hgh probablty, wthn ± from /2. hus, we see that the term n our performance guarantee s necessary. 3 Back to Zero-sum Games Recall the defnton of a two-player zero-sum game defned by a par of m n payoff matrces (R, C where R + C = 0. For the remander of ths secton, we assume w.l.o.g. that m = n. Now suppose row player and column player both use a multplcatve weghts update (MWU experts algorthm to generate ther respectve mxed strateges, x t and for the zero-sum game at tme step t. Each player could use a dfferent MWU algorthm, however we assume both algorthms are of low regret (.e. the algorthm acheves the MWU bound dscussed n the prevous secton. Snce the game s zero-sum, row player s loss at tme step t s determned by column player s strategy, l t row = C. Smlarly, column player s loss at tme step t s determned by row player s strategy, (l t col = (x t R l t col = R x t. Recall our assumpton of bounded losses, l t 0, n n the experts algorthm settng. Here, however, the losses of row and column player are not restrcted to that range snce the payoffs n R and C can take on any values as long as R + C = 0. In order to use the proved MWU bounds, we must frst normalze R and C so that l t row, l t col 0, n. 4-4

5 Before proceedng, let us ntroduce the followng scalar term: M = max,j R,j = max C,j snce R + C = 0.,j Now we apply an affne transformaton to R and C to produce normalzed payoff matrces R and C R = C = R + M C + M, where denotes the matrx of ones. Next we derve the cumulatve loss for row and column player s experts algorthms for tme horzon usng the normalzed payoff matrces. o avod confuson between the tme horzon and the tranpose operator, we henceforth use the conjugate transpose A to denote the transpose of A. Entres n the strategy vectors and payoff matrces are real, so A concdes wth the transpose of A. L row = = (x t l t row = (x t C = (x t C + M (x t (x t ( C + M = (x t C + M, ( snce t, (x t =. Usng the same constructon for column player, we fnd the followng: L col = (l t col = (x t R = (x t R + M. (2 Now we bound the cumulatve loss for row player s experts algorthm gven a known tme horzon, L row mn L + 2 ln(n L + 2 ln(n, ( L = e l t row = e C = e C + M = e C + M e L row e C + M whch we combne wth equaton and smplfy, (x t C + M ( (x t C = e C + M + 2 ln(n,, e C + M + 2 ln(n, e C + 4M ln(n,. (3 4-5

6 Smlarly, we bound the cumulatve loss for column player s experts algorthm, L col mn L j + 2 ln(n L j + 2 ln(n, j j ( L j = (l t col e j = (x t R e j = (x t R + M e j L col = (x t Re j + M (x t Re j + M whch we combne wth equaton 2 and smplfy, (x t R + M ( (x t R (x t e j = (x t Re j + M + 2 ln(n, j, (x t Re j + M + 2 ln(n, j (x t Re j + 4M ln(n, j. (4 Equatons 3 and 4 tell us that the cumulatve losses of the row and column player s experts algorthms are bounded bhe no-regret loss tmes a multplcatve term, whch s lnear n the maxmum absolute value M n the payoff tables. In the followng theorem, we see what these bounds tell us about the average payoff of the game. heorem 2. If (x, x 2,..., x and (y, y 2,..., y are the sequences of strateges generated for the row player and column player respectvely bhe MWU algorthm, then ( x t, s a ( 8M ln(n approxmate Nash Equlbrum. Proof: For an ɛ-approxmate N.E., we must show the followng: ( ( ( x t R ( x t C ( e R ( ɛ, {,..., m} x t Ce j ɛ, j {,..., n}. 4-6

7 We begn by replacng C wth R (snce R + C = 0 n equaton 3, (x t ( R e ( R + 4M ln(n, (x t R e R + 4M ln(n, (x t R e R 4M ln(n, (x t R e R 4M ln(n, Next we rearrange the terms n equaton 4 and applhe above bound, mn j mn j (x t Re j ( x t Re j ( x t Re j. (x t R 4M ln(n, j (x t R 4M ln(n e R 8M ln(n, Fnally we ntroduce the average strategy t yt whch s a probablty dstrbuton over the determnstc strateges e j. We use the fact that any lnear functon appled to t yt must have value at least as large as the mnmum value t takes over the determnstc strateges e j ( ( ( ( ( x t R ( x t R ( x t R. ( mn x t Re j j e R 8M ln(n, ( e R ln(n 8M,. Followng the same constructon for column player, that s by replacng R by C n equaton 4 and rearrangng the terms n equaton 3, we fnd the followng: ( ( ( ln(n x t C x t Ce j 8M, j Remark. heorem 2 demonstrates two mportant facts about the two-player zero-sum game as the tme horzon ncreases (.e. : he average payoffs obtaned bhe row and column player of the game converge to ther values n the game. 2 (Informal he hstogram of play, or average strateges of the players, converges to an equlbrum of the game. 4-7

8 Remark 2. In hereom 2, the ɛ error term decreases as the nverse square root of tme. It s an open queston whether there exsts a smple no-regret algorthm for zero-sum games convergng faster (e.g. exponentally fast. 4-8