Quaderni di Dipartimento. Rate of Convergence of Predictive Distributions for Dependent Data. Patrizia Berti (Università di Modena e Reggio Emilia)

Transcription

1 Quaderi di Dipartimeto Rate of Covergece of Predictive Distributios for Depedet Data Patrizia Berti Uiversità di Modea e Reggio Emilia Iree Crimaldi Uiversità di Bologa Luca Pratelli Accademia Navale di Livoro Pietro Rigo Uiversità di Pavia # Dipartimeto di ecoomia politica e metodi quatitativi Uiversità degli studi di Pavia Via Sa Felice, 5 I Pavia Geaio 2009

2 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS FOR DEPENDENT DATA PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO Abstract. This paper deals with empirical processes of the type C B = { µ B PX +1 B X 1,..., X }, where X is a sequece of radom variables ad µ = 1/ i=1 δ X i the empirical measure. Coditios for sup B C B to coverge stably i particular, i distributio are give, where B rages over a suitable class of measurable sets. These coditios apply whe X is exchageable, or, more geerally, coditioally idetically distributed i the sese of [6]. By such coditios, i some relevat situatios, oe obtais that sup B C B P 0, or eve that sup B C B coverges a.s.. Results of this type are useful i Bayesia statistics. 1. Itroductio ad motivatios A umber of real problems reduce to evaluate the predictive distributio a = P X +1 X 1,...,X for a sequece X 1, X 2,... of radom variables. Here, we focus o those situatios where a ca ot be calculated i closed form, ad oe decides to estimate it basig o the available data X 1,...,X. Related refereces are [1], [2], [3], [5], [6], [8], [10], [15], [18], [20]. For otatioal reasos, it is coveiet to work i the coordiate probability space. Accordigly, we fix a measurable space S, B, a probability P o S, B, ad we let X be the -th caoical projectio o S, B, P, 1. We also let G = σx 1,...,X ad X = X 1, X 2,... Sice we are cocered with predictive distributios, it is reasoable to make some qualitative assumptios o them. I [6], X is said to be coditioally idetically distributed c.i.d. i case E I B X k G = E IB X +1 G, a.s., for all B B ad k > 0, where G 0 is the trivial σ-field. Thus, at each time 0, the future observatios X k : k > are idetically distributed give the past G. I a sese, this is a weak form of exchageability. I fact, X is exchageable if ad oly if it is statioary ad c.i.d., ad various examples of o exchageable c.i.d. sequeces are available. I the sequel, X = X 1, X 2,... is a c.i.d. sequece of radom variables Mathematics Subject Classificatio. 60G09, 60B10, 60A10, 62F15. Key words ad phrases. Bayesia predictive iferece Cetral limit theorem Coditioal idetity i distributio Empirical distributio Exchageability Predictive distributio Stable covergece. 1

3 2 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO I that case, a soud estimate of a is the empirical distributio µ = 1 δ Xi. The choice of µ ca be defeded as follows. Let D B ad let deote the suporm o D. Suppose also that D is coutably determied, as defied i Sectio 2. The latter is a mild coditio, oly eeded to hadle measurability issues. The, i=1 µ a = sup µ B a B a.s. 0 1 B D provided X is c.i.d. ad µ coverges uiformly o D with probability 1; see [5]. For istace, µ a a.s. 0 wheever X is exchageable ad D a Gliveko- Catelli class. Or else, µ a a.s. 0 if S = R, D = {, t] : t R}, ad X 1 has a discrete distributio or if ǫ>0 lim if P X +1 X < ǫ = 0; see [4]. To sum up, uder mild assumptios, µ is a cosistet estimate of a with respect to uiform distace for c.i.d. data. This is i lie with de Fietti [10] i the particular case of exchageable idicators. Takig 1 as a startig poit, the ext step is to ivestigate the covergece rate. That is, to ivestigate whether α µ a coverges i distributio, possibly to a ull limit, for suitable costats α > 0. This is precisely the purpose of this paper. A first piece of iformatio o the covergece rate of µ a ca be gaied as follows. For B B, defie µb = limsupµ B, W B = {µ B µb}. By the SLLN for c.i.d. sequeces, µ B a.s. µb; see [6]. Hece, for fixed 0 ad B B, oe obtais E µb G = limk E µ k B G = limk 1 k k i=+1 = E I B X +1 G = a B a.s.. E I B X i G I tur, this implies { µ B a B } = EW B G a.s., so that µ a 1 sup E 1 W B G E W G a.s.. B D If sup E W k < for some k 1, it follows that E { α µ a k} α k E W k 0 wheever α 0. Eve if obvious, this fact is potetially useful, as sup E W k < for all k 1, if X is exchageable, 2 for various choices of D; see Remark 3. I particular, 2 holds if D is fiite.

4 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 3 The itriguig case, however, is α =. For each B B ad probability Q o S, B, write C Q B = E Q W B G ad C B = C P B = { µ B a B }. I Theorem 3.3 of [6], the asymptotic behaviour of C B is ivestigated for fixed B. Here, istead, we are iterested i C = sup C B = µ a. B D Our mai result Theorem 1 is the followig. Fix a radom probability measure N o R ad a probability Q o S, B such that The, C Q N stably uder Q ad W is uiformly itegrable uder both P ad Q. C N stably wheever P Q. 3 A remarkable particular case is N = δ 0. Suppose i fact that, for some Q, oe has C Q Q 0 ad W uiformly itegrable uder P ad Q. The, C P 0 wheever P Q. Stable covergece i the sese of Reyi is a stroger form of covergece i distributio. The defiitio is recalled i Sectio 2. I geeral, oe caot dispese with the uiform itegrability coditio. However, the latter is ofte true. For istace, W is uiformly itegrable uder P ad Q provided D meets 2 ad X is exchageable uder P ad Q. To make 3 cocrete, a large list of referece probabilities Q is eeded. Various examples are available i the Bayesia oparametrics framework; see e.g. [16] ad refereces therei. The most popular is perhaps the Ferguso-Dirichlet law, deoted by Q 0. If P = Q 0, the X is exchageable ad a B = α PX 1 B + µ B α + a.s. for some costat α > 0. Sice µ a α/ whe P = Q 0, somethig more tha C P 0 ca be expected i case P Q 0. Ideed, we prove that µ a = C coverges a.s. wheever P Q 0 with a desity satisfyig a certai coditio; see Theorem 2 ad Corollary 5. Oe more example should be metioed. Let X = Y, Z, where Z > 0 ad P α PY 1 B + Y +1 B G = i=1 Z i I B Y i α + i=1 Z a.s. i for some costat α > 0. Uder some coditios, X is c.i.d. but ot ecessarily exchageable, W is uiformly itegrable ad C coverges stably. See Sectio 4. The above material takes a icer form whe the coditio P Q ca be give a simple characterizatio. This happes, for istace, if S = {x 1,...,x k, x k+1 } is

5 4 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO fiite, X exchageable ad PX 1 = x > 0 for all x S. The, P Q 0 for some choice of Q 0 if ad oly if µ{x1 },...,µ{x k } has a absolutely cotiuous distributio with respect to Lebesgue measure. I this particular case, however, a part of our results ca also be obtaied through Berstei - vo Mises theorem; see Sectio 3. Fially, we make two remarks. i If X is exchageable, our results apply to Bayesia predictive iferece. Suppose i fact S is Polish ad B the Borel σ-field, so that de Fietti s theorem applies. The, P is a uique mixture of product probabilities o B ad the mixig measure is called prior distributio i a Bayesia framework. Now, give Q, P Q is just a assumptio o the prior distributio. This is plai i the last example where S = {x 1,...,x k, x k+1 }. I Bayesia terms, such a example ca be summarized as follows. For a multiomial statistical model, C P 0 if the prior is absolutely cotiuous with respect to Lebesgue measure, ad C coverges a.s. if the prior desity satisfies a certai coditio. ii To our kowledge, there is o geeral represetatio for the predictive distributios of a exchageable sequece. Such a represetatio would be very useful. Eve if partially, results like 3 cotribute to fill the gap. As a example, for fixed B B, oe obtais a B = µ B + o P 1 as far as X is exchageable ad P Q for some Q such that C Q B Q 0 ad W B is uiformly itegrable. 2. Mai results A few defiitios eed to be recalled. Let T be a metric space, B T the Borel σ-field o T ad Ω, A, P a probability space. A radom probability measure o T is a mappig N o Ω B T such that: i Nω, is a probability o B T for each ω Ω; ii N, B is A-measurable for each B B T. Let Z be a sequece of T-valued radom variables ad N a radom probability measure o T. Both Z ad N are defied o Ω, A, P. Say that Z coverges stably to N i case P Z H E N H weakly for all H A such that PH > 0. Clearly, if Z N stably, the Z coverges i distributio to the probability law E N just let H = Ω. Stable covergece has bee itroduced by Reyi i [17] ad subsequetly ivestigated by various authors. See [9] for more iformatio. Next, say that D B is coutably determied i case, for some fixed coutable subclass D 0 D, oe obtais sup B D0 ν 1 B ν 2 B = sup B D ν 1 B ν 2 B for every couple ν 1, ν 2 of probabilities o B. A sufficiet coditio is that, for some coutable D 0 D, ad for every ǫ > 0, B D ad probability ν o B, there is B 0 D 0 satisfyig ν B B 0 < ǫ. Most classes D ivolved i applicatios are coutably determied. For istace, D = {, t] : t R k } ad D = {closed balls} are coutably determied if S = R k ad B the Borel σ-field. Or else, D = B is coutably determied if B is coutably geerated.

6 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 5 We are ow i a positio to state our mai result. Let N be a radom probability measure o R, defied o the measurable space S, B, ad let Q be a probability o S, B. Theorem 1. Let D be coutably determied. Suppose C Q N stably uder Q, ad W : 1 is uiformly itegrable uder P ad Q. The, C = µ a N stably wheever P Q. Proof. Sice D is coutably determied, there are o measurability problems i takig sup B D. I particular, W ad C are radom variables ad C is G -measurable. Let f be a versio of dp dq ad U = f E Q f G. The, C B = E W B G = E Q f W B G Lettig M = E Q E Q f G = C Q B + E Q U W B G, P-a.s., for each B B. E Q f G U W G E Q f G ad takig sup B D, it follows that C Q M C C Q + M, P-a.s.. We first assume f bouded. Sice C Q N stably uder Q, give a bouded radom variable Z o S, B, oe obtais φ C Q Z dq Nφ Z dq, for each bouded cotiuous φ : R R, where Nφ = φx N, dx. Lettig Z = f I H /PH, with H B ad PH > 0, it follows that C Q N stably uder P. Therefore, it suffices to prove EM 0. Give ǫ > 0, sice W is uiformly itegrable uder Q, there is c > 0 such that { } ǫ E Q W I { W >c} < for all. supf Sice M is G -measurable, EM = E Q f M = EQ EQ f G M = EQ U W c E Q U + supf E Q W I { W >c} < ce Q U + ǫ for all. Therefore, the martigale covergece theorem implies lim sup EM c lim supe Q U + ǫ = ǫ. This cocludes the proof whe f is bouded. Next, let f be ay desity. Fix k > 0 such that Pf k > 0 ad defie K = {f k} ad P K = P K. The, P K has the bouded desity f I K /PK with respect to Q. By what already proved, C P k N stably uder P K, where C P E { } I k K W B G B = E PK W B G =, P K -a.s.. EI K G

7 6 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO Lettig R = I K EI K G, it follows that E { I K C C P k } = E { I K sup E{ } R W B G } B D EI K G E { E { } R W G } { I K = E R W } EI K G c E R + E { W I { W >c}} for all c > 0. Sice E R 0 ad W is uiformly itegrable uder P, arguig as above implies E PK C C P k E{ I K C C P k } 0. PK Therefore, C N stably uder P K. Fially, fix H B, PH > 0, ad a bouded cotiuous fuctio φ : R R. The PH K = P H {f k} > 0, for k large eough, ad 2 sup φ Pf > k + PH E φ C H E Nφ H E φ C H K E Nφ H K. Sice E φ C H K E Nφ H K as, ad Pf > k 0 as k, this cocludes the proof. We ext deal with the particular case Q = Q 0, where Q 0 is a Ferguso-Dirichlet law o S, B. If P Q 0 with a desity satisfyig a certai coditio, the covergece rate of µ a ca be remarkably improved. Theorem 2. Suppose D is coutably determied ad sup E Q0 W 2 <. The, C = µ a coverges a.s. provided P Q 0 ad E Q0 f 2 E Q0 { EQ0 f G 2} = O 1, for some versio f of dp dq 0. Proof. Let D B = C B. The, D is G -measurable as D is coutably determied ad E +1 D +1 G = E sup I B X i + 1EµB G +1 G B D i=1 sup E +1 I B X i G + 1E µb G B D i=1 = sup I B X i EµB G = D B D i=1 Sice D is a G -submartigale, it suffices to prove that sup E D <. Let U = f E 0 f G, where E 0 stads for E Q0. By assumptio, there are c 1, c 2 > 0 such that a.s.. E 0 W 2 c 1, E 0 U 2 = { E 0 f 2 E 0 E 0 f G 2 } c 2 for all.

8 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 7 As oted i Sectio 1, sice Q 0 is a Ferguso-Dirichlet law, there is α > 0 such that C Q 0 = sup E 0 W B G α for all. B D Defie M = E0 U W G ad recall that C C Q0 + M, P-a.s.; see the E 0 f G proof of Theorem 1. The, for all, oe obtais E D = E C E C Q0 + EM α + E0 f M = α + E 0 U W α + E 0 U 2 E 0 W 2 α + c 1 E 0 U 2 α + c 1 c 2. Fially, we specify a poit raised i Sectio 1. Remark 3. There is a log list of coutably determied choices of D such that sup E W k ck, for all k 1, if X is i.i.d., where ck is some uiversal costat; see e.g. Subsectios ad of [21]. Fix oe such D, k 1, ad suppose S is Polish ad B the Borel σ-field. If X is exchageable, de Fietti s theorem yields E W k T ck a.s. for all, where T is the tail σ-field of X. Hece, E W k = E { E W k T } ck for all. This proves iequality Exchageable data with fiite state space Whe X is exchageable ad S fiite, there is some overlappig betwee Theorem 1 ad a result of Berstei ad vo Mises Coectios with Berstei - vo Mises theorem. For each θ i a ope set Θ R k, let P θ be a product probability o S, B that is, X is i.i.d. uder P θ. Suppose the map θ P θ B is Borel measurable for fixed B B. Give a prior probability π o the Borel subsets of Θ, defie PB = P θ B πdθ, B B. Roughly speakig, Berstei - vo Mises BVM theorem ca be stated as follows. Suppose π is absolutely cotiuous with respect to Lebesgue measure ad the statistical model P θ : θ Θ is suitably smooth we refer to [13] for a detailed expositio of what smooth meas. For each, suppose θ admits a cosistet maximum likelihood estimator θ. Further, suppose the prior π possesses the first momet ad deote θ the posterior mea of θ. The, θ θ P θ0 0 for each θ 0 Θ such that the desity of π is strictly positive ad cotiuous at θ 0. Actually, BVM-theorem yields much more tha asserted, what reported above beig just the corollary coected to this paper. We refer to [13] ad [14] for more iformatio ad historical otes. See also [18].

9 8 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO Assumig a smooth, fiite-dimesioal statistical model is fudametal; see e.g. [11]. Ideed, BVM-theorem does ot apply whe the oly iformatio is X exchageable or eve c.i.d. ad P Q for some referece probability Q. Oe exceptio, however, is S fiite. Let us suppose S = {x 1,...,x k, x k+1 }, X exchageable, PX 1 = x > 0 for all x S, ad D = B = power set of S. Also, let λ deote Lebesgue measure o R k ad π the probability distributio of θ = µ{x 1 },...,µ{x k }. As oted i Sectio 1, π λ if ad oly if P Q 0 for some choice of Q 0. Sice D is fiite ad X exchageable uder P ad Q 0, the W is uiformly itegrable uder P ad Q 0. Thus, Theorem 1 yields C P 0 wheever π λ. O the other had, π is the prior distributio for this problem. The uderlyig statistical model is smooth ad fiite-dimesioal it is just a multiomial model. Further, for each, the maximum likelihood estimator ad the posterior mea of θ are, respectively, θ = µ {x 1 },...,µ {x k }, θ = a {x 1 },...,a {x k }. Thus, BVM-theorem implies C P 0 as far as π λ ad the desity of π is cotiuous o the complemet of a π-ull set. To sum up, i this particular case, the same coclusios as Theorem 1 ca be draw from BVM-theorem. Ulike the latter, however, Theorem 1 does ot require ay coditio o the desity of π Some cosequeces of Theorems 1 ad 2. I this subsectio, we focus o S = {0,1}. Thus, D = B = power set of S ad λ is Lebesgue measure o R. Let N0, a deote the oe-dimesioal Gaussia law with mea 0 ad variace a 0 where N0,0 = δ 0. Our first result allows π to have a discrete part. Corollary 4. With S = {0,1}, let π be the probability distributio of µ{1} ad = {θ [0,1] : π{θ} > 0}, Defie the radom probability measure N o R as A = {ω S : µω, {1} }. N = 1 I A δ 0 + I A N 0, µ{1}1 µ{1}. If X is exchageable ad π does ot have a sigular cotiuous part, the C {1} N stably ad C N h 1 stably where hx = x, x R, is the modulus fuctio. Proof. By stadard argumets, the Corollary holds whe π 0, 1 provided it holds whe π = 0 ad π = 1. Let π = 0. The π λ, as π does ot have a sigular cotiuous part, ad the Corollary follows from Theorem 1. Thus, it ca be assumed π = 1. Sice C {0} = C {1}, C = C {1} ad the modulus fuctio is cotiuous, it suffices to prove that C {1} N stably. Next, exchageability of X implies W {1} N 0, µ{1}1 µ{1} stably; see e.g. Theorem 3.1 of [6]. Sice π = 1, the N = N 0, µ{1}1 µ{1} a.s.. Hece, it is eough to show that E C {1} W {1} 0.

10 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 9 Fix ǫ > 0 ad let M = W {1}. Sice X is exchageable, M is uiformly itegrable. Therefore, there is c > 0 such that supe ǫ M I { M >c} < 4. Defie φx = x if x c, φx = c if x > c, ad φx = c if x < c. Sice C {1} = EM G a.s., it follows that E C {1} W {1} E EM G EφM G + +E EφM G φm + E φm M E EφM G φm + 4E M I { M >c} < E EφM G φm + ǫ for all. Write = {a 1, a 2,...} ad M,j = µ {1} a j. Sice σm,j G ad Pµ{1} = π = 1, oe also obtais E EφM G φm = E E φm,j I {µ{1}=aj} G φm,j I {µ{1}=aj} j = E φm,j { } Pµ{1} = a j G I {µ{1}=aj} j m c E Pµ{1} = a j G I {µ{1}=aj} + 2c j>mπ{a j } for all m,. j=1 By the martigale covergece theorem, E Pµ{1} = a j G I {µ{1}=aj} 0, as, for each j. Thus, lim supe C {1} W {1} ǫ + 2c j } for all m. j>mπ{a Takig the limit as m cocludes the proof. If π is sigular cotiuous, we cojecture that C {1} coverges stably to a o ull limit. But we have ot a proof. I the ext result, a real fuctio g o 0,1 is said to be almost Lipschitz i case x gxx a 1 x b is Lipschitz o 0,1 for some reals a, b < 1. Corollary 5. Suppose S = {0,1}, X is exchageable ad π is the probability distributio of µ{1}. If π admits a almost Lipschitz desity with respect to λ, the C coverges a.s. to a real radom variable. Proof. Let V = µ{1}. By assumptio, there are a, b < 1 ad a versio g of dπ dλ such that φθ = gθθ a 1 θ b is Lipschitz o 0,1. For each u 1, u 2 > 0, we ca take Q 0 such that V has a beta-distributio with parameters u 1, u 2 uder Q 0. Let Q 0 be such that V has a beta-distributio with parameters u 1 = 1 a ad u 2 = 1 b uder Q 0. The, for ay 1 ad x 1,...,x {0,1}, oe obtais P 1 X 1 = x 1,...,X = x = θ r 1 θ r πdθ = = c V r 1 V r φv dq 0 where r = θ r a 1 θ r b φθ dθ x i ad c > 0 is a costat. i=1

11 10 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO Let h = c φ. The, h is Lipschitz ad f = hv is a versio of dp dq 0. Let V = E 0 V G, where E 0 stads for E Q0. Sice h is Lipschitz, f E 0 f G hv hv + E 0 hv hv G d V V + d E 0 V V G where d is the Lipschitz costat of h. Sice E 0 C Q0 2 E 0 W 2 ad V V = C Q0 {1} W {1} C Q0 + W, it follows that E 0 f 2 E 0 E0 f G 2 = E 0 { f E0 f G 2} 4 d 2 E 0 { V V 2} 4 d2 E { 0 C Q 0 + W 2 } 16 d2 E 0 W 2. Sice sup E 0 W 2 <, the E 0 f 2 E 0 E0 f G 2 = O1/. A applicatio of Theorem 2 cocludes the proof. Corollaries 4 ad 5 deal with S = {0,1} but similar results ca be proved for ay fiite S. See also [12] ad [19]. 4. Geeralized Polya urs I this sectio, basig o Examples 1.3 ad 3.5 of [6], the asymptotic behaviour of C is ivestigated for a certai c.i.d. sequece. Let Y, B Y be a measurable space, B + the Borel σ-field o 0, ad S = Y 0,, B = B Y B +, X = Y, Z, where Y ω = y, Z ω = z for all ω = y 1, z 1, y 2, z 2,... S. Give a law P o B, it is assumed that P Y +1 B G = α PY 1 B + i=1 Z i I B Y i α + i=1 Z i a.s., 1, 4 P Z +1 C X 1,...,X, Y +1 = PZ1 C a.s., 0, 5 for some costat α > 0 ad all B B Y ad C B +. Note that Z is i.i.d. ad Z +1 is idepedet of Y 1, Z 1,...,Y, Z, Y +1 for all 0. I real problems, the Z should be viewed as weights while the Y describe the pheomeo of iterest. As a example, cosider a ur cotaiig white ad black balls. At each time 1, a ball is draw ad the replaced together with Z more balls of the same colour. Let Y be the idicator of the evet {white ball at time } ad suppose Z is chose accordig to a fixed distributio o the itegers, idepedetly of Y 1, Z 1,...,Y 1, Z 1, Y. The, the predictive distributios of X are give by 4-5. Note also that the probability law of Y is Ferguso- Dirichlet i case Z = 1 for all. It is ot hard to prove that X is c.i.d.. We state this fact as a lemma. Lemma 6. The sequece X assessed accordig to 4-5 is c.i.d.. Proof. Fix k > 0 ad A B Y B +. By a mootoe class argumet, it ca be assumed A = B C where B B Y ad C B +. Further, it ca be assumed

12 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 11 k = +2. Let = 0 ad G 0 the trivial σ-field. Sice X 2 X 1 as it is easily see, E I B Y 2 I C Z 2 G 0 = E IB Y 1 I C Z 1 G 0 a.s.. If 1, defie G = σx 1,...,X, Z +1. O otig that E I B Y +1 G = E IB Y +1 G a.s., oe obtais E I B Y +2 G { = E EIB Y +2 G +1 G } = α PY 1 B + i=1 Z i I B Y i + Z +1 EI B Y +1 G α + +1 i=1 Z i = α + i=1 Z i EI B Y +1 G + Z +1 EI B Y +1 G α + +1 i=1 Z. i = EI B Y +1 G = EI B Y +1 G a.s.. Fially, sice G G, the previous equality implies E I B Y +2 I C Z +2 G = PZ1 C E { EI B Y +2 G G } = PZ 1 C E { EI B Y +1 G G } = E IB Y +1 I C Z +1 G Therefore, X is c.i.d.. Usually, oe is iterested i predictig Y more tha Z. Thus, i the sequel, we focus o PY +1 B G. For each B B Y, we write C B = C B 0,, a B = a B 0, = PY+1 B G, ad so o. I Example 3.5 of [6], assumig EZ 2 1 <, it is show that C B N0, σb 2 stably, where σb 2 = varz 1 µb 1 µb. EZ 1 2 Here, we prove that C coverges stably whe regarded as a map C : S l D, where l D is the space of real bouded fuctios o D equipped with uiform distace; see Sectio 1.5 of [21]. I particular, stable covergece of C as a radom elemet of l D implies stable covergece of C = sup B D C B. Ituitively, the stable limit of C whe it exists is coected to Browia bridge. Let B 1, B 2,... be pairwise disjoit elemets of B Y ad k D = {B k 0, : k 1}, T 0 = 0, T k = µb i. Also, let G be a stadard Browia bridge process o some probability space Ω 0, A 0, P 0. For fixed ω S, varz1 { Lω,B k = GTk ω GT k 1 ω } EZ 1 is a real radom variable o Ω 0, A 0, P 0. Sice the B k are pairwise disjoit ad G has cotiuous paths, Lω, B k 0 as k. So, it makes sese to defie Mω, as the probability distributio of Lω = Lω,B 1, Lω,B 2,..., that is, Mω,A = P 0 Lω A i=1 for each Borel set A l D. Similarly, let Nω, be the probability distributio of sup k 1 Lω, B k, i.e., Nω, A = P 0 sup Lω,B k A for each Borel set A R. k 1 a.s..

13 12 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO Theorem 7. Suppose B 1, B 2,... B Y are pairwise disjoit ad D, M, N are defied as above. Let X be assessed accordig to 4-5 with a Z 1 b a.s. for some costats 0 < a < b. The supe W 2 c P Y 1 k B k, 6 for some costat c idepedet of the B k, ad C M stably i the metric space l D. I particular, C N stably. Let Q 1 deote the probability law of a sequece X satisfyig 4-5 ad a Z 1 b a.s.. I view of Theorem 7, Q 1 ca play the role of Q i Theorem 1. That is, for a arbitrary c.i.d. sequece X with distributio P, oe has C N stably provided P Q 1 ad W is uiformly itegrable uder P. The coditio of pairwise disjoit B k is actually rather strog. However, it holds i at least two relevat situatios: whe a sigle set B is ivolved ad whe S = {x 1, x 2,...} is coutable ad B k = {x k } for all k. Proof of Theorem 7. This proof ivolves some simple but log calculatios. Accordigly, we just give a sketch of the proof ad we refer to [7] for details. Sice X is c.i.d., for fixed B B Y oe has a B = E µb G a.s.. Hece, a B : 1 is a G -martigale with a B a.s. µb, ad this implies E { a +1 B µb 2} = E { a j B a j+1 B 2} = E { a j B a j+1 B 2}. j> Replacig a j B by 4 ad usig that a Z i b a.s. for all i, a log but straightforward calculatio yields j> E{ a j B a j+1 B 2} c 1 PY 1 B where c 1 is a costat idepedet of B. It follows that E a +1 µ 2 = E { sup k k = E { a j B k a j+1 B k 2} c 1 PY 1 B k k j> k = c 1 P Y 1 k B k as the B k are pairwise disjoit. Precisely as above, after some algebra, oe obtais E µ a +1 2 c 2 P Y 1 k B k for some costat c 2 idepedet of B 1, B 2,... Therefore, E W 2 = E µ µ 2 2 E µ a E a +1 µ 2 c P Y 1 k B k j> a+1 B k µb k 2} E { a +1 B k µb k 2} where c = 2c 1 + c 2. This proves iequality 6. It remais to prove that C M stably i the metric space l D. For each m 1, let Σ m be the m m matrix with elemets σ k,j = varz 1 EZ 1 2 µbk B j µb k µb j, k, j = 1,...,m. By Theorems ad of [21], for C M stably, it is eough that i-fiite dimesioal covergece: C B 1,...,C B m N m 0, Σm stably for each m 1,

14 RATE OF CONVERGENCE OF PREDICTIVE DISTRIBUTIONS 13 where N m 0, Σm is the m-dimesioal Gaussia law with mea 0 ad covariace matrix Σ m ; ii-asymptotic tightess: For each ǫ, δ > 0, there is m 1 such that lim supp sup C B r C B s > ǫ < δ. r,s>m Fix m 1, b 1,...,b m R, ad defie R = m k=1 b ki Bk Y. Sice R : 1 is c.i.d., arguig exactly as i Example 3.5 of [6], oe obtais m b k C B k = i=1{ Ri ER +1 G } N 0, b k b j σ k,j stably. k=1 k,j Sice b 1,...,b m are arbitrary, i holds. To check ii, give ǫ, δ > 0, take m such that P ǫ 2 δ 2 Y 1 r>m B r < 4 c where c is the costat ivolved i 6. By what already proved, P sup C B r C B s > ǫ P 2 sup C B r > ǫ r,s>m r>m P 2 E 4 sup W B r G > ǫ r>m ǫ 2 E{ sup W B r 2} r>m 4 c ǫ 2 P Y 1 r>m B r < δ. Thus, ii holds, ad this cocludes the proof. Ackowledgmets: This paper beefited from the helpful suggestios of two aoymous referees. Refereces [1] Algoet P.H Uiversal schemes for predictio, gamblig ad portfolio selectio, A. Probab., 20, [2] Algoet P.H Uiversal predictio schemes Correctio, A. Probab., 23, [3] Berti P. ad Rigo P A uiform limit theorem for predictive distributios, Statist. Probab. Letters, 56, [4] Berti P., Pratelli L. ad Rigo P Almost sure uiform covergece of empirical distributio fuctios, Iterat. Math. J., 2, [5] Berti P., Mattei A. ad Rigo P Uiform covergece of empirical ad predictive measures, Atti Sem. Mat. Fis. Uiv. Modea, L, [6] Berti P., Pratelli L. ad Rigo P Limit theorems for a class of idetically distributed radom variables, A. Probab., 32, [7] Berti P., Crimaldi I., Pratelli L. ad Rigo P Rate of covergece of predictive distributios for depedet data, Techical Report, available at: [8] Blackwell D. ad Dubis L.E.1962 Mergig of opiios with icreasig iformatio, A. Math. Statist., 33, [9] Crimaldi I., Letta G. ad Pratelli L A strog form of stable covergece, Semiaire de Probabilites XL, Lect. Notes i Math., 1899,

15 14 PATRIZIA BERTI, IRENE CRIMALDI, LUCA PRATELLI, AND PIETRO RIGO [10] de Fietti B La previsio: ses lois logiques, ses sources subjectives, Aales Istit. Poicare, 7, [11] Freedma D O the Berstei - vo Mises theorem with ifiitedimesioal parameters, A. Statist., 27, [12] Ghosh J.K., Siha B.K. ad Joshi S.N Expasios for posterior probability ad itegrated Bayes risk, Statistical Decisio Theory ad Related Topics III, Vol. 1, Academic Press, [13] Ghosh J.K. ad Ramamoorthi R.V Bayesia oparametrics, Spriger. [14] Le Cam L. ad Yag G.L Asymptotics i statistics: some basic cocepts, Spriger. [15] Morvai G. ad Weiss B Forward estimatio for ergodic time series, A. Ist. H. Poicare Probab. Statist., 41, [16] Pitma J Some developmets of the Blackwell-MacQuee ur scheme, I: Statistics, Probability ad Game Theory Ferguso, Shapley ad Mac Quee Eds., IMS Lecture Notes Moogr. Ser., 30, [17] Reyi A O stable sequeces of evets, Sakhya A, 25, [18] Romaovsky V Sulle probabilita a posteriori, Giorale dell Istituto Italiao degli Attuari,. 4, [19] Strasser H Improved bouds for equivalece of Bayes ad maximum likelihood estimatio, Theor. Probab. Appl., 22, [20] Stute W O almost sure covergece of coditioal empirical distributio fuctios, A. Probab., 14, [21] va der Vaart A. ad Weller J.A Weak covergece ad empirical processes, Spriger. Patrizia Berti, Dipartimeto di Matematica Pura ed Applicata G. Vitali, Uiversita di Modea e Reggio-Emilia, via Campi 213/B, Modea, Italy address: patrizia.berti@uimore.it Iree Crimaldi, Dipartimeto di Matematica, Uiversita di Bologa, Piazza di Porta Sa Doato 5, Bologa, Italy address: crimaldi@dm.uibo.it Luca Pratelli, Accademia Navale, viale Italia 72, Livoro, Italy address: pratel@mail.dm.uipi.it Pietro Rigo correspodig author, Dipartimeto di Ecoomia Politica e Metodi Quatitativi, Uiversita di Pavia, via S. Felice 5, Pavia, Italy address: prigo@eco.uipv.it