Incremental calculation of weighted mean and variance

Transcription

1 Icremetal calculatio of weighted mea ad variace Toy Fich Uiversity of Cambridge Computig Service February 009 Abstract I these otes I eplai how to derive formulae for umerically stable calculatio of the mea ad stadard deviatio, which are also suitable for icremetal o-lie calculatio. I the geeralize these formulae to weighted meas ad stadard deviatios. I upick the difficulties that arise whe geeralizig further to ormalized weights. Fially I show that the epoetially weighted movig average is a special case of the icremetal ormalized weighted mea formula, ad derive a formula for the epoetially weighted movig stadard deviatio. Simple mea Straightforward traslatio of equatio ito code ca suffer from loss of precisio because of the differece i magitude betwee a sample ad the sum of all samples. Equatio 4 calculates the mea i a way that is more umerically stable because it avoids accumulatig large sums. µ i ) + i ) ) This formula also provides us with some useful idetities. Simple variace + )µ ) 3) µ + µ ) 4) µ µ µ ) 5) µ µ µ ) µ + µ )µ µ ) 6) The defiitio of the stadard deviatio i equatio 7 below requires us to already kow the mea, which implies two passes over the data. This is t feasible for olie algorithms that eed to produce icremetal results after each sample becomes available. Equatio solves this problem sice it allows us to calculate the stadard deviatio from two ruig sums. σ i µ) 7) i i µ + µ ) 8)

2 3 Icremetal variace i µ i + µ 9) i µµ + µ 0) i µ ) ) ) i i ) Kuth otes [] that equatio is proe to loss of precisio because it takes the differece betwee two large sums of similar size, ad suggests equatio 4 as a alterative that avoids this problem. However he does ot say how it is derived. I the followig, equatio 0 is derived from the previous step usig equatio 5. Let S σ 3) i µ ) 4) i µ 5) S S i µ i + )µ 6) µ + )µ 7) µ + µ µ ) 8) µ + µ µ )µ + µ ) 9) µ + µ )µ + µ ) 0) µ + µ µ µ + µ µ ) µ µ + µ µ ) µ ) µ ) 3) S S + µ ) µ ) 4) σ S / 5) Mathworld [] has a alterative derivatio of a similar formula, which i our otatio is as follows. S i µ ) 6) i µ ) µ µ )) 7) i µ ) + µ µ ) i µ )µ µ ) 8) Simplify the first summatio: i µ ) µ ) + i µ ) 9) µ ) + S 30) S + µ µ ) 3)

3 Simplify the secod summatio: Simplify the third summatio: µ µ ) µ µ ) 3) i µ )µ µ ) µ µ ) i µ ) 33) ) µ µ ) µ + i µ ) ) µ µ ) µ )µ + i µ µ ) µ )µ + )µ ) 36) µ µ ) µ ) 37) µ µ ) 38) Back to the complete formula: S S + µ µ ) + µ µ ) µ µ ) 39) S + µ µ ) µ µ ) 40) S + )µ µ ) 4) We ca use equatios 6 ad 5 to show this is equivalet to equatio 4. S S + )µ µ ) 4) S + µ µ ) µ ) 43) S + µ ) µ ) 44) 4 Weighted mea The weighted mea is defied as follows. 34) 35) µ w i i w i 45) It is equivalet to the simple mea whe all the weights are equal, sice µ w i w w i i 46) w If the samples are all differet, the weights ca be thought of as sample frequecies, or they ca be used to calculate probabilities where p i w i / w i. The followig derivatio of the icremetal formula equatio 53) follows the same patter as the derivatio of equatio 4. For brevity we also defie as the sum of the weights. µ w i 47) w i i 48) ) w + w i i 3 49)

4 Useful idetities derived from this formula are: 5 Weighted variace w + W µ ) 50) w + w )µ ) 5) µ + w w µ ) 5) µ + w µ ) 53) µ µ ) w µ ) 54) µ µ ) w µ 55) µ w µ µ ) µ + µ w w µ µ ) 56) w µ ) 57) Similarly, we derive a umerically stable formula for calculatig the weighted variace equatio 68) usig the same patter as the derivatio of the uweighed wersio equatio 4). σ w i i µ) w i i µ 58) Let S σ 59) w i i µ 60) S S 6 Variable weights w i i µ w i i + W µ 6) w µ + W µ 6) w µ + w )µ 63) w µ ) + W µ µ ) 64) w µ ) + W µ µ )µ + µ ) 65) w µ + µ )µ + µ ) ) 66) w µ ) µ ) 67) S S + w µ ) µ ) 68) σ S / 69) I the previous three sectios, I have assumed that weights are costat oce assiged. However a commo requiremet is to ormalize the weights, such that W w i 70) If we are repeatedly addig ew data to our workig set, the we ca t have both costat weights ad ormalized weights. To allow us to keep weights ormalized, we eed to allow the weight of each 4

5 sample to chage as the set of samples chages. To idicate this we will give weights two idices, the first idetifyig the set of samples usig the sample cout as we have bee doig for µ etc.) ad the secod beig the ide of the sample i that set. We will ot make ay assumptios about the sum of the weights, that is we will ot require them to be ormalized. For eample, w,i, µ w,i i 7) Havig doe this we eed to re-eamie some of the logical steps i the previous sectios to esure they are still valid. I equatios 49 5, we used the fact that i the fied-weight settig, w i i W µ w )µ 7) I the ew settig, this equality is fairly obviously o loger true. For eample, if we are keepig weights ormalized the W.) Fortuately there is a differet middle step which justifies equatio 7 whe weights vary, so the results of sectio 4 remai valid. w,i i w, )µ 73) w,i i w,i i w,i w,i w i,i w,j w,i w w,i i,i w,i i w,i w,i 74) 75) w,i w i 76),i w,j w,i where j 77) This says that for the weighted mea formulae to remai valid the ew ad old weights should be cosistet. Equatio 75 says that we get the same result whe we calculate the mea of the previous workig set whether we use the old weights or the ew weights. Equatio 77 says that whe we ormalize the weights across the previous set up to ) we get the same set of weights whether we start from the old weights or the ew oes. This requiremet is t eough by itself to make the weighted variace formulae work, so we will eamie it agai below. 7 The epectatio fuctio At this poit it is worth defiig some better otatio to reduce the umber of summatios we eed to write. The epectatio fuctio is a geeralized versio of the mea, whose argumet is some arbitrary fuctio of each sample. E f)) w,i f i ) 78) E k) k 79) E af)) ae f)) 80) E f) + g)) E f)) + E g)) 8) µ E ) 8) σ E µ ) ) 83) E + µ µ ) 84) E ) + µ µ E ) 85) 5

6 E ) µ 86) E ) E ) 87) The icremetal formula is derived i the usual way. Equatio 9 is particularly useful. E f)) w,i f i ) 88) 8 Variable-weight variace w, f ) + w,i f i ) 89) w, f ) + w, ) w,if i ) w,i 90) w, f ) + w, ),if i ) w,i 9) w, f ) + w, )E f)) 9) E f)) E f)) + w, f ) E f))) 93) I equatios 6 63 we made the followig assumptios which are ot true whe weights ca vary. w,i i µ w,i i + W µ w, µ + W µ w, µ + w, )µ If we try to re-do the short derivatio of the icremetal stadard deviatio formula startig from S S the we soo get stuck. Fortuately the loger derivatio shows how to made it work. S σ 94) E µ ) ) 95) Simplify the first term: E [ µ ] [µ µ ]) ) 96) E [ µ ] + [µ µ ] [ µ ][µ µ ] ) 97) E [ µ ] ) + E [µ µ ] ) E [ µ ][µ µ ]) 98) E [ µ ] ) w, [ µ ] + w, )E [ µ ] ) 99) Simplify the secod term: Simplify the third term: w, [ µ ] + w, ) S W 00) w, S + [µ µ ] 0) W w, E [µ µ ] ) [µ µ ] 0) E [ µ ][µ µ ]) [µ µ ] E [ µ ] 03) 6

7 Back to the complete formula: [µ µ ] w, [ µ ] + w, )E [ µ ]) 04) [µ µ ] w, [ µ ] + w, )[E ) E µ )]) 05) [µ µ ] w, [ µ ] + w, )[µ µ ]) 06) [µ µ ]w, [ µ ] 07) [µ µ ] 08) S w, S + [µ µ ] + [µ µ ] [µ µ ] 09) W w, w, S + [µ µ ] w, [µ µ ] 0) W w, w, w, S + w, ) [µ µ ] ) W w, w, W S + w, )[µ µ ] µ ) ) S w, W S + w, µ ) µ ) 3) This is the same as equatio 68, ecept for the multiplier W w, W which captures the chage i weights betwee the old ad ew sets. w,,i W w w,j,i w,j where j 4) Now that we kow the rescalig trick which makes it work, we ca write dow the short versio. S w, S W E ) µ ) W w, ) E ) µ ) 5) E ) µ ) W E ) + w, + w, )µ 6) w, µ + w, )µ 7) w, µ ) + W µ µ ) 8) w, µ ) + W µ µ )µ + µ ) 9) w, µ + µ )µ + µ ) ) 0) w, µ ) µ ) ) 9 Epoetially-weighted mea ad variace Startig from equatio 53, let s set w, / to a costat 0 < α < ad let a α. This produces the stadard formula for the epoetially weighted movig average. µ µ + α µ ) ) α)µ + α 3) aµ + a) 4) I the followig it s more coveiet to use a lower boud of 0 istead of, i.e. 0 i. We are goig to show that the weights are reormalized each time a datum is added. First, we epad out the iductive defiitio of the mea. µ aµ + a) 5) a µ + a a) + a) 6) 7

8 a 3 µ 3 + a a) + a a) + a) 7) µ a 0 + a i a) i 8) This allows us to write dow the weights directly. Note that w, is idepedet of. w,0 a 9) w,i a i a), for i 30) w, a α 3) Sice w, α w, / we ca see that., that is, the weights are always ormalized. We ca get the same result by summig the geometric series. a i a j a a w,i j0 3) a i a) a 33) w,0 + These weights satisfy the cosistecy requiremet because w,j aw,j w,j w,i aw,i w,i w,i a + a ) 34) We ca use the epectatio fuctio to write dow the aïve formula for the variace. 35) E f)) E f)) + w, f ) E f))) 36) E f)) + αf ) E f))) 37) E ) E ) + α E )) 38) σ E ) µ 39) So usig the formula from the previous sectio we ca write the icremetal versio: S w, W S + w, µ ) µ ) 40) S α S + α µ ) µ ) 4) σ S S as + a) µ ) µ ) 4) This latter form is slightly more coveiet for code: diff : - mea icr : alpha * diff mea : mea + icr variace : - alpha) * variace + diff * icr) Refereces α)s + α µ ) ) 43) [] Doald E. Kuth. Semiumerical Algorithms, volume of The Art of Computer Programmig, chapter 4.., page 3. Addiso-Wesley, Bosto, third editio, 998. [] Eric W. Weisstei. Sample variace computatio. From Mathworld, a Wolfram web resource, 8