ON LOCAL LIKELIHOOD DENSITY ESTIMATION WHEN THE BANDWIDTH IS LARGE

Transcription

1 ON LOCAL LIKELIHOOD DENSITY ESTIMATION WHEN THE BANDWIDTH IS LARGE Byeong U. Park 1 and Young Kyung Lee 2 Department of Statistics, Seoul National University, Seoul, Korea Tae Yoon Kim 3 and Ceolyong Park 3 Department of Statistics, Keimyung University, Taegu, Korea Sinto Eguci 3 Institute of Statistical Matematics, Tokyo, Japan August 31, 2004: Final version for JSPI Abstract In tis paper, we provide a large bandwidt analysis for a class of local likeliood metods. Tis work complements te small bandwidt analysis of Park, Kim and Jones (2002. Our treatment is more general tan te large bandwidt analysis of Eguci and Copas (1998. We provide a iger order asymptotic analysis for te risk of te local likeliood density estimator, from wic a direct comparison between various versions of local likeliood can be made. Te present work, being combined wit te small bandwidt results of Park et al. (2002, gives an optimal size of te bandwidt wic depends on te degree of departure of te underlying density from te proposed parametric model. AMS 2000 subject classifications. Primary 62G07; secondary 62G20. Key words and prases. Density estimation, local likeliood, kernel function, bandwidt. 1 Researc of Byeong U. Park was supported by KOSEF troug Statistical Researc Center for Complex Systems at Seoul National University. 2 Researc of Young Kyung Lee was supported by te Brain Korea 21 project in Researc of Sinto Eguci, Tae Yoon Kim and Ceolyong Park was supported by KOSEF(F and JSPS troug Korea-Japan Joint Researc Program. 1

2 1. Introduction Local likeliood metods for density estimation are very promising, not least on account of teir extraordinary flexibility and adaptivity. Tey afford efficient estimation if te proposed parametric family includes a good model for te data, as well as te usual good beaviour of nonparametric density estimation if it does not. Eac formulation of local likeliood estimation is based on a locally weigted log-likeliood were te weigts are determined by a kernel function K and a bandwidt. Wen is large, te resulting estimator is close to te fully parametric maximum likeliood estimator. On te oter and, wen is small, performance of te resulting estimator would not depend muc on te proposed parametric model. For consistent estimation, te metods require a correction term added to te naive locally weigted log-likeliood. Precise details of te correction term allow scope for variation. See Copas (1995, Loader (1996, Hjort and Jones (1996, and Kim, Park and Kim (2001 for statistical analysis on different formulations of local likeliood wic employ a specific coice of te correction term. Recently, Eguci and Copas (1998 made an important contribution to local likeliood density estimation. Tey provided a unified formulation of various local likeliood approaces by introducing an arbitrary function, denoted by ξ, for te additional correction term. Tis unified formulation includes as special cases te C-version due to Copas (1995, te U-version due to Loader (1996 and Hjort and Jones (1996, and te T-version due to Eguci and Copas (1998. Te definition of te general local likeliood function and tose of te U-, C-, and T-versions are given in te next section. Later, Park et al. (2002 presented a small asymptotics of te class of local likeliood estimators. It is based on a general condition on ξ tailored for te small analysis. Eguci and Copas (1998 found some interesting large properties of te class. However, teir results are based on a rater stringent condition on ξ. Te condition excludes some important local likeliood metods suc as Copas (1995. See, for more details, te paragrap including (2.4 in te current paper, or te discussion in Eguci and Copas (1998 immediately below te proof of teir Teorem 1. Furtermore, teir results are restricted to te case were te underlying density is undetectably close to te parametric model. In tis paper, we give a more relevant large analysis wit a quite general condition on te function ξ. Te condition is different from te one used in Park et al. (2002 for a small analysis. Te U-, C- and T-versions of local likeliood are seen to satisfy our new 2

3 condition. We provide a useful approximation of te risk wen te bandwidt tends to infinity as sample size n grows. We sow tat te mean integrated squared error takes te form 1 C B,ML + C V,ML n C 1 B,SM C 1 2 V1,SM n + C 1 2 V2,SM n. (1.1 4 Te first term C B,ML represents te model misspecification error of te proposed parametric family. Tis and C V1,SM are zero if te true density actually belongs to te parametric model. In fact, C B,ML and n 1 C V,ML are te integrated squared bias and te integrated variance, respectively, of te parametric maximum likeliood estimator. Te next tree terms are originated from local smooting. Te constants C B,SM and C V1,SM depend on te distance between te true density and te parametric model, wile te last C V2,SM does not. Depending on ow close te true density is to te parametric model, te last term n 1 4 C V2,SM may dominate n 1 2 C V1,SM. We give explicit expressions for tese constants. Our formula is more useful tan te one given by Eguci and Copas (1998 in te sense tat its dependence on te coice of ξ is more transparent, tus a direct comparison between various local likeliood metods can be made. Let {f(, θ : θ Θ} be a parametric model proposed for te data were θ is a p-dimensional parameter vector. Let θ denote te solution of te population version of te parametric likeliood equation. In fact, f(, θ minimizes te Kullback-Leibler divergence from te true density among all members in te parametric model. It is te best parametric approximant to te true density in tat sense. A precise definition of θ is given at (2.8 in Section 2. We evaluate te risk in a srinking neigborood of te parametric model. In particular, we assume tat te true density g g n satisfies {g(x f(x, θ } 2 dx cn (1+α, (1.2 for some α > 1 and c > 0. Note tat α = corresponds to te case were te true density belongs to te parametric model. We derive an approximation of te risk of te form given at (1.1 under tis condition. It is sown tat te constants C B,ML and C B,SM in (1.1 are bot O(n 1 α. Te constant C V1,SM is seen to be O(n (1+α/2. Our large asymptotics is valid for all α > 1. Note tat, wen α = 1, a risk analysis in te usual nonparametric context of te bandwidt tending to zero as n grows is more appropriate. Eguci and Copas (1998 gave some interesting large properties of teir class of local likeliood density estimators, but only in te case were α > 0. 3

4 Tus, te present work is also an extension of Eguci and Copas (1998 in tis regard. Since features of te underlying density (suc as location and scale may be estimated at best to an accuracy of O(n 1/2, treatment of te case 1 < α 0, tat is not dealt in Eguci and Copas (1998, is useful to describe te large properties wen te underlying density lies in a region were one can distinguis it from te parametric model. Our large asymptotics and te small results of Park et al. (2002, te latter being valid for all α 1, may be combined to give an optimal size of te bandwidt going to zero or infinity. It is argued tat te value of α at wic a transition from a large to a small bandwidt is desirable does exist in te range 1 < α 0. We observe in an example tat te optimal bandwidt indeed ranges from a very small to a very large value depending on ow close te true density is to te parametric model. Our treatment of large bandwidt asymptotics is more rigorous tan tat of Eguci and Copas (1998. We obtain iger order expansions for te bias and te variance of te local likeliood estimator. We sow tat by coosing te bandwidt in an optimal way te local likeliood estimator may ave smaller risk tan te parametric maximum likeliood estimator except te case were α =. Te risk considered in tis paper is te mean integrated weigted squared error. An expansion of te relative entropy risk (expected Kullback-Leibler divergence considered in Eguci and Copas (1998 is readily obtained from our results by coosing proper weigt functions. Tis paper is organized as follows. Section 2 introduces te unified formulation of local likeliood density estimation and provides te new condition on ξ for large analysis. It also contains some preliminary results for te risk analysis. In Section 3, we give a detailed account of large asymptotics for all α > 1. In Section 4, te risks of te U-, C- and T-versions of local likeliood estimation are compared wit an example. Also discussed is te issue of optimal bandwidt size. Some tecnical proofs are given in te appendix. 2. Preliminaries Let X 1,..., X n be independent d-variate random vectors from a common density g( supported on X. Let f(, θ wit θ being a p-dimensional parameter vector be a parametric model proposed for te data. Define u(t, θ = ( / θ log{f(t, θ}. Let K be a nonnegative symmetric kernel function on IR d. For simplicity of presentation we take a 4

5 scalar bandwidt. For an arbitrary function ξ(,, te general form of local likeliood estimating equation considered by Eguci and Copas (1998 and Park et al.(2002 is given by ˆΨ (x, θ = 0 were ˆΨ (x, θ = 1 n ( x Xi K u(x i, θ 1 n ξ n i=1 n i=1 ( x X1 E θ {K { K u(x 1, θ }. ( x Xi, E θ K ( } x X1 Here and below E θ means expectation wit respect to te parametric density f(, θ wile E wit no subscript means expectation wit respect to te true density g. If we take =, tis reduces to te parametric maximum likeliood estimating equation: n u(x i, θ = 0. i=1 Let ˆθ (x denote te solution of te estimating equation ˆΨ (x, θ = 0. likeliood density estimator may be defined by A local ˆf (x = f(x, ˆθ (x. (2.1 Note tat tis estimator may not integrate to one since ˆθ (x depends on x. A bona fide density estimator is given by Te function ξ is assumed to satisfy E θ ξ { K ĝ (x = ˆf (x/ ˆf (x dx. (2.2 ( x X1, E θ K ( } x X1 = 1. (2.3 Tis condition is required for te estimating equation to be unbiased wen g( = f(, θ for some θ. In fact, it guarantees te first-order Bartlett s identity: E θ ˆΨ (x, θ = 0 for all θ. Examples of ξ tat satisfy te condition (2.3 include te U-version ξ(u, v 1 of Hjort and Jones (1996, te C-version ξ(u, v = (1 u/(1 v of Copas (1995, and te T-version ξ(u, v = u/v considered in Eguci and Copas (1998. Eguci and Copas (1998 assumed α > 0 and discussed te properties of f(x, ˆθ (x wen tends to infinity as n grows. Teir results are based on te following additional condition on ξ: E ξ { K ( x X1, E θ K ( } x X1 5 ( 1 = 1 + O 2 (2.4

6 as tends to infinity. But, tis condition is too restrictive. Te C-version ξ(u, v = (1 u/(1 v, for example, does not satisfy tis condition. To see tis, assume as in Eguci and Copas (1998 tat K(t = 1 κ 2 t 2 + O( t 4 as t 0. Ten, E ξ { K ( ( } x X1 x X1 1, E θ K 1+ x y 2 {g(y f(y, θ E θ x X 1 2 } dy. Te second term in te above approximation is not O( 2, but equals O{n (1+α/2 } wen te true density g g n satisfies te condition (1.2. We consider ere a more relevant condition on ξ wic replaces (2.4. We note tat, under te condition K(t = 1 κ 2 t 2 + O( t 4 as t 0, te large properties of te local likeliood estimator depend on ξ troug te beaviour of ξ(1 y, 1 z wen bot y and z approac to zero from above. Tis can be seen from te fact tat bot arguments of ξ in te definition of ˆΨ converge to one as tends to infinity. Te condition on ξ sould be different from te one in small setting, were performance of te estimator relies on te properties of ξ(y, z near y = z = 0 since bot arguments of ξ tend to zero as converges to zero. See Park et al. (2002 for a suitable condition on ξ in small setting. Te condition on ξ for our large analysis is given in (A1 below. Te condition on te kernel K is stated in (A2 were we specify te coefficient of O( t 4 term for more detailed analysis. Assumptions. (A1 In addition to te consistency condition (2.3, ξ satisfies lim z 0 sup 0 y c 1 z {ξ(1 yz, 1 z ξ 0(y ξ 1 (yz} = 0 (2.5 for some functions ξ 0, ξ 1 and a constant c > 0. Te function ξ 0 is continuously differentiable and ξ 1 is continuous. Also, ξ(y, z is twice continuously differentiable wit respect to z on (0, 1 for eac y [0, 1]. (A2 Te kernel function K(t is continuous at t = 0 and satisfies K(t = 1 κ 2 t 2 + κ 4 t 4 + o( t 4 as t 0. Te tree versions of local likeliood mentioned above satisfy te condition (A1: for te U-version, ξ 0 (y 1 and ξ 1 (y 0; for te C-version, ξ 0 (y = y and ξ 1 (y 0; 6

7 and for te T-version, ξ 0 (y 1 and ξ 1 (y = 1 y. Under te condition (A1, we may differentiate te left and side of (2.3 wit respect to θ. Tis yields ( ( ] ( x E θ ξ [K (1 X1 x X1 x X1, E θ K E θ K [ ( ( x X1 x X1 = E θ ξ K, E θ K for all θ, were ξ (1 (y, z = ( / zξ(y, z. u(x 1, θ ] u(x 1, θ (2.6 Below, we give some preliminary results for te discussion in Section 3. Let θ (x denote te solution of te equation E ˆΨ (x, θ = 0. (2.7 Also, define a population version of ˆf (x by f (x = f(x, θ (x. Tese two quantities are tose to wic ˆθ (x and ˆf (x, respectively, get closer as te sample size n grows. Next, define θ to be te solution of Eu(X, θ = 0. (2.8 Later, it will be seen tat θ is te limit of θ (x as tends to infinity. If te true density g belongs to te parametric model, i.e., if it equals f(, θ for some θ, ten θ equals tat value of te parameter and g = f(, θ. Trougout te paper, we assume tat te true density lies in a n (1+α/2 neigborood of te parametric model in te following sense. (A3 Te true density g g n satisfies for some α > 1 and c > 0 {g(x f(x, θ } 2 dx cn (1+α. A relevant expansion of ξ[k( 1 (x X 1, E θ K( 1 (x X 1 ] plays an important role for te asymptotic analysis in Section 3. Define Y (x = x X 1 2 E θ x X 1 2. Tis is te limit of Y (x = {1 K( 1 (x X 1 }/{1 E θ K( 1 (x X 1 } as tends to infinity. Also, let W (x = κ 4 κ 2 ξ 0(Y (xy (x [ E θ {Y (x x X 1 2 } x X 1 2] +κ 2 ξ 1 (Y (xe θ { x X 1 2 } (2.9 7

8 In te following lemma we give a useful expansion for ξ. Lemma 1. In addition to te conditions (A1 and (A2, assume tat g as compact support. Ten, for any constant c > 0 tere exists ɛ going down to zero as tends to infinity suc tat [ ( ( } P sup x {K ξ X1 x X1, E θ K x <c ξ 0 (Y (x 1 W (x ɛ ] = Proof. ξ [ K Let z (x = 1 E θ K( 1 (x X 1. Ten, we can write ( x X1, E θ K ( ] x X1 = ξ [1 Y (xz (x, 1 z (x]. (2.10 From te condition (A2 and compactness of te support of g, it follows tat for any constant c > 0 tere exists ɛ going down to zero as tends to infinity suc tat wit probability one { sup Y (x Y (x κ ( 4 x <c 2 Eθ [Y (x x X 1 2 ] x X 1 2} ɛ κ 2. ( Applying (2.5 to (2.10 wit te fact tat sup x <c z (x c / 2 for some c > 0, and using (2.11 yields te lemma. Remark 1. Te condition tat g as compact support in Lemma 1 may be relaxed to a weaker one. In fact, it may be proved wit more deliberate arguments tat te lemma still olds wen g as exponentially decaying tails. However, in tis case one needs some stronger conditions on ξ, instead. For example, in place of (2.5 one needs lim z 0 sup 0 y z 1+β 1 z {ξ(1 yz, 1 z ξ 0(y ξ 1 (yz} = 0 for an arbitrarily small β > 0. In addition, to control its property at tails one needs sup 0<z<ε sup z γ ξ(1 yz, 1 z ξ 0 (y ξ 1 (yz < c z 1+β <y z 1 for some ε, c > 0 and γ 0. Te tree versions of te local likeliood estimation still satisfy tese conditions on ξ: for te U-version, c = 1 and γ = 0; for te C-version, c = 1 and γ = 1; and for te T-version, c = (1 ε 1 and γ = 0. 8

9 Te following two lemmas demonstrate te beaviour of f (x, defined immediately below (2.7, and tat of ˆf (x wen tends to infinity. Tese are useful to quantify te asymptotic risk of te estimator in te next section. To state te lemmas, let U (x, θ = ˆΨ (x, θ E ˆΨ (x, θ. It as mean zero and variance of order O(n 1. Define { ( } { } x X1 f I (x, θ = E K u(x 1, θu(x 1, θ T E f (X 1, θ { [ ( ( ] } x X1 x X1 E ξ K, E θ K u(x 1, θ ( } x X1 E θ {K u(x 1, θ T. (2.12 It will be seen tat tis is an approximation of ( / θe ˆΨ (x, θ. Above and in te subsequent arguments, we let ṗ(x, θ and p(x, θ for a function p denote, respectively, te first and te second derivatives of p(x, θ wit respect to θ. Write f (x = f(x, θ, and define a p p matrix I (θ = E { u(x 1, θu(x 1, θ T f } f (X 1, θ. Te two lemmas rely on some tecnical assumptions in addition to (A1 (A3. Tey are stated in te appendix. Proofs of te lemmas are also deferred to te appendix. Lemma 2. Assume (A1 (A3 and te conditions listed in te appendix. Ten, for all α 1 it follows tat uniformly for x in any compact subset of X f (x = f (x + f(x, θ T I (θ 1 E ˆΨ (x, θ + O(ρ (2.13 as n and, were ρ ρ(n, = n (1+α/2 4. Lemma 3. Assume te conditions of Lemma 2. Ten, for all α 1 it follows tat uniformly for x in any compact subset of X ˆf (x = f (x + f(x, θ T I (x, θ 1 U (x, θ + O p (δ as n and, were δ δ(n, = n 1 (α/2 2 (log n 1/2 + n 1 log n. Remark 2. Since U (x, θ at Lemma 3 is a sum of independent random vectors, asymptotic normality of ˆf (x follows immediately from te lemma. 3. Risk analysis 9

10 error: For te risk of an estimator ĝ of g, we consider te mean integrated weigted squared E {ĝ(x g(x} 2 w(x dx, were w( is a weigt function wose support is compact and contained in te support of g. In tis section, we provide te asymptotic risks of te estimator ˆf (x and its scaled version ĝ (x. First, we consider te unscaled estimator ˆf (x. Te asymptotic risk of te estimator ˆf (x is given by te risk of its approximation f (x wic is defined by f (x = f (x + f(x, θ T I (x, θ 1 U (x, θ. (3.1 We decompose te risk of f into two parts: b U (n, = {f (x g(x} 2 w(x dx, v U (n, = E { f (x f (x} 2 w(x dx. Te first term b U (n, represents te bias of te unscaled estimator f due to model misspecification, wile v U (n, measures its sampling variability. In te following teorem, we give approximations for tese components. To state te teorem, let I 0, = Eu(X 1, θ u(x 1, θ T and D = E{ f(x 1, θ /f(x 1, θ }. Note tat D is not zero since we take te expectation wit respect to te true density instead of f(, θ. It follows tat Define ν 0 = τ 0 = I 0, = I (θ D. (3.2 {f (x g(x} 2 w(x dx, f(x, θ T I (θ 1 I 0, I (θ 1 f(x, θ w(x dx, U(x = f(x, θ T I 1 0, D I 1 0, u(x 1, θ, Z U (x = f(x, { θ T I0, 1 x X1 2 u(x 1, θ ξ 0 (Y (xe θ x X 1 2 u(x 1, θ }, ν 1,U = 2 {E g f Z U (x} {f (x g(x} w(x dx, τ 1,U = 2 {EZ U (xu(x} w(x dx, τ 2,U = [EZU (x 2 {EZ U (xu(x 1, θ T }I 1 0, {Eu(X 1, θ Z U (x} ] w(x dx. 10

11 Here E g f denotes E E θ. Note tat ν 0 = O(n 1 α and ν 1,U = O(n 1 α by te condition (A3. Also, τ 1,U = O(n (1+α/2 since D = O(n (1+α/2 by te fact E θ { f(x 1, θ /f(x 1, θ } = 0. Now, τ 2,U is a constant wic does not depend on n. It is strictly positive. Tis follows from te Caucy-Scwartz inequality: for any IR p -valued ψ and real-valued φ ( ( Eφψ T Eψψ T 1 (Eψφ Eφ 2 wit = olding if and only if φ = a T ψ for some constant vector a. Taking ψ = U(X 1, θ and φ = Z U (x sows tat τ 2,U > 0. Teorem 1. Under te conditions of Lemma 2, we get as n and ( ν 1,U b U (n, = ν 0 κ 2 + o 1, 2 n α+1 2 v U (n, = τ ( 0 n κ τ 1,U 2 n + τ 2,U 2 κ2 2 n + o 1 4 n 3/2+α/ n 4. Write Next, we consider te scaled estimator ĝ (x. Define g (x = f (x/ f (x dx. v (x = f(x, θ T I (θ 1 E ˆΨ (x, θ, V (x = f(x, θ T I (x, 1 U (x, θ. We may obtain expansions for g (x and ĝ (x, analogous to tose given at Lemmas 2 and 3, as follows: g (x = f (x + v (x f (x v (x dx + O(ρ, (3.3 ĝ (x = g (x + V (x f (x V (x dx + O p (δ. (3.4 In te next teorem, we give te risk of g (x, an approximation of ĝ (x defined by g (x = g (x + V (x f (x V (x dx. Similarly to te case of te unscaled f (x, te risk of g (x is decomposed into two parts: b S (n, = {g (x g(x} 2 w(x dx, (3.5 v S (n, = E { g (x g (x} 2 w(x dx. (3.6 11

12 To state te teorem, write Z S (x = Z U (x f (x Z U (x dx. Define te following scaled versions of ν 1,U, τ 1,U and τ 2,U : ν 1,S = 2 {E g f Z S (x} {f (x g(x} w(x dx, τ 1,S = 2 {EZ S (xu(x} w(x dx, τ 2,S = [EZS (x 2 {EZ S (xu(x 1, θ T }I 1 0, {Eu(X 1, θ Z S (x} ] w(x dx. Teorem 2. Under te conditions of Lemma 2, we get as n and ( ν 1,S b S (n, = ν 0 κ 2 + o 1, 2 n α+1 2 v S (n, = τ ( 0 n κ τ 1,S 2 n + τ 2,S 2 κ2 2 n + o 1 4 n 3/2+α/ n 4. Te first term ν 0 in te expansions of b U (n, and b S (n, in Teorems 1 and 2 represents te model misspecification error of te proposed parametric family. It is te integrated squared bias of te parametric maximum likeliood estimator f(, ˆθ MLE, were ˆθ n MLE is defined as te solution of te equation u(x i, θ = 0. It is zero if te true density actually belongs to te parametric model. Next, te first term τ 0 /n in te expansions of v U (n, and v S (n, is te integrated variance of te parametric maximum likeliood estimator. Since ν 0 and τ 0 do not depend on ξ, te first order properties of all te members in te class of te local likeliood estimation are te same. Te oter terms in te expansions depend on te bandwidt. Tese terms also depend on ξ, but only troug ξ 0. Tus, te U- and T-versions ave te same second order properties, too, as tey bot ave ξ 0 1. We note tat Eguci and Copas (1998 neglected te term κ 2 τ 1 /n 2 in teir analysis of te asymptotic variance because teir main concern was te case were 0 < α < 1. Wen 0 < α < 1, te optimal is of order n α/2 (see Section 4 and wit tis coice te term κ 2 τ 1 /n 2 is negligible. We may find an optimal size of te bandwidt by minimizing te sum of b(n, and v(n,. Tis will be discussed in Section 4. Te formulas given in Teorems 1 and 2 are more useful tan te one given by Eguci and Copas (1998 since dependence of te risks on te function ξ are more transparent. A direct risk comparison between various local 12 i=1

13 likeliood metods can be made from te formulas, wic will be dealt too in te next section. Below, we give a proof of Teorem 1. Proof of Teorem 2 is omitted as it may be proved in a similar fasion using (3.3 and (3.4 instead of Lemmas 2 and 3. Proof of Teorem 1. From te consistency condition (2.3 on ξ, E ˆΨ ( [ ( ( ] x X1 x X1 x X1 (x, θ = E g f {K u(x 1, θ ξ K, E θ K ( } x X1 E θ K u(x 1, θ. Since E θ u(x 1, θ = 0, we obtain from te condition (A2 on te kernel tat uniformly for x in any compact subset of X ( x X1 E θ K u(x 1, θ = κ 2 2 E θ x X 1 2 u(x 1, θ + O ( 1 4. Tus, from Lemma 1 and te fact E g f u(x 1, θ = 0, it follows tat uniformly for x in any compact subset of X f(x, θ T I (θ 1 E ˆΨ (x, θ = κ 2 2 E g f Z U (x + o ( 1. (3.7 n (1+α/2 2 Te first part of te teorem ten follows immediately from (2.13 at Lemma 2 and (3.7. To find te formula for v(n,, we need to approximate I (x, θ, defined at (2.12, and var {U (x, θ }. Define I k, (x = E { x X 1 k u(x 1, θ u(x 1, θ T }, Z(x = x X 1 2 u(x 1, θ ξ 0 (Y (xe θ x X 1 2 u(x 1, θ. By similar arguments as in deriving (3.7, we get I (x, θ = I (θ κ 2 E Z(xu(X 1, θ 2 T + 1 [ κ 4 4 I 4, (x uniformly for x in any compact subset of X. +κ 2 E W (xu(x 1, θ { E θ x X 1 2 u(x 1, θ }] T (3.8 ( 1 +o 4 We compute var {U (x, θ }. From Lemma 1 and te condition (A2 on te kernel K, we may verify tat uniformly for x in any compact subset of X var {U (x, θ } = 1 [ n var u(x 1, θ κ 2 Z(x + 1 { κ4 x X u(x 1, θ + κ 2 W (xe θ x X 1 2 u(x 1, θ }] ( 1 + o. (3.9 n 4 13

14 Next, write H 2 (x = κ 2 E Z(xu(X 1, θ T and H 4 (x = κ 4 I 4, (x + κ 2 E W (xu(x 1, θ { E θ x X 1 2 u(x 1, θ T }. Note tat wit tese notations te equation (3.8 can be written as Also, te equation (3.9 reduces to I (x, θ = I (θ 1 2 H 2(x H 4(x. (3.10 n var {U (x, θ } = I 0, 1 2 { H2 (x + H 2 (x T } { H4 (x + H 4 (x T } ( + κ2 2 1 E 4 Z(xZ(xT + o. ( From (3.10 it follows tat uniformly for x in any compact subset of X I (x, θ 1 = I (θ I (θ 1 H 2 (xi (θ 1 We plug (3.11 and (3.12 into + 1 I (θ 4 [ 1 H 2 (xi (θ 1 H 2 (x H 4 (x ] I (θ 1 (3.12 ( 1 +o. 4 n f(x, θ T I (x, θ 1 var{u (x, θ }{I (x, θ 1 } T f(x, θ, and collect terms involving 2 and 4. We find tat te 2 terms are 1 f(x, θ 2 T I (θ { 1 H 2 (xi (θ 1 I 0, + I 0, I (θ 1 H 2 (x T H 2 (x H 2 (x } T I (θ 1 = 2 2 f(x, θ T I (θ 1 { H 2 (xi (θ 1 E f f (X 1, θ f(x, θ (3.13 } I (θ 1 f(x, θ. Te equation (3.13 follows from (3.2. We can replace I (θ 1 in (3.13 by I 1 0, wit an error O(n 1 α 2 since E{ f(x 1, θ /f(x 1, θ } = O(n (1+α/2. Tis gives tat (3.13 equals 2 κ 2 2 E{Z U (xu(x} + O(n 1 α 2. Similarly, we find tat te 4 terms reduce to 1 f(x, θ 4 T I (θ [ 1 κ2e 2 Z(xZ(x T H 2 (xi (θ 1 H 2 (x ] T I (θ 1 f(x, θ + o ( 1 4 = κ2 [ 2 EZU (x 2 {EZ 4 U (xu(x 1, θ T }I0, {Eu(X 1 1, θ Z U (x} ] ( 1 + o 4 14

15 uniformly for x in any compact subset of X. Te second part of te teorem now follows. Remark 3 (Kullback-Leibler risk. An expansion of te Kullback-Leibler risk E log {g(x/ĝ (x} g(x dx may be derived from Teorem 2 wit specific coices of te weigt function w. In fact, under te condition tat f is bounded away from zero on te support of g, we may approximate te Kullback-Leibler risk by { } g (x f (x KL(g, f g(x dx + 1 { } 2 f (x 2 E g (x g (x g(x dx (3.14 g (x were KL(g, f = log {g(x/f(x} g(x dx denotes te Kullback-Leibler divergence of f from g. Te first term ν KL 0 KL(g, f at (3.14 is te minimal Kullback-Leibler divergence from g among all members in {f(, θ : θ Θ}. Te second term equals {g (x f (x} {g(x f (x} f (x 1 dx (3.15 since {g (x f (x} dx = 1 1 = 0. From te definition of b S (n, at (3.5 and its expansion given at Teorem 2, we may get an approximation of (3.15. By applying te first part of Teorem 2 wit w(x = 1/{2f (x}, we obtain (3.15 equals κ 2 ν KL o(n (1+α 2 were ν1 KL = {E g f Z S (x} {f (x g(x} f (x 1 dx. Wen ξ 0 1 (tus for U- and T-version, it may be proved tat ν KL 1 = 2 {E g f X 1 u(x 1, θ } T I 1 0, {E g f X 1 u(x 1, θ }. (3.16 Tis matces te bias results in Teorem 1 and Corollary 1 of Eguci and Copas (1998. Tis means tat te results of Eguci and Copas (1998 are valid only for te case were ξ 0 1. Note tat te unscaled version f does not integrate to one. Tus, te second term at (3.14 corresponding to te unscaled estimator ˆf would be of order n (1+α/2 2 wic is slower tan n (1+α 2 of te scaled estimator. Terefore, normalizing te local likeliood estimator by its integral is important for te Kullback-Leibler risk. Te tird term at (3.14 is te variance part. We may get an expansion of tis from te second part of Teorem 2, now wit w(x = g(x/{2f (x 2 }. It equals n κ τ1 KL 2 n + τ KL ( 2 κ2 2 2 n + o 1 4 n 3/2+α/ n 4 τ KL 0 15

16 were τ KL i w(x. For instance, for i = 0, 1, 2 are defined as τ 0, τ 1,S and τ 2,S wit g(x/{2f (x 2 } replacing τ0 KL = 1 2 u(x, θ T I (θ 1 I 0, I (θ 1 u(x, θ g(x dx. 4. Comparison and optimal bandwidt In Teorems 1 and 2, ν 0 + (τ 0 /n is te mean integrated squared error of te parametric maximum likeliood estimator. Tus, te risk improvement acieved by te local likeliood estimators upon te parametric maximum likeliood estimator is given by r d ( κ 2 (ν 1 + τ 1 n 1 2 κ 2 2τ 2 n 1 4. (4.1 Here and below in tis section, we simply write ν 1 and τ i (i = 1, 2 for ν 1,k and τ i,k, respectively, were k = U or S. As a function of t = 2, r d ( is a concave parabola on t 0. It as te maximum value at t 0 = (ν 1 n + τ 1 /(2κ 2 τ 2 if ν 1 n + τ 1 > 0. Tus, in tis case te optimal bandwidt is given by ( 2κ2 τ 1/2 2 opt =, (4.2 ν 1 n + τ 1 and te maximum risk improvement equals (ν 1 n + τ 1 2 /(4nτ 2. Wen ν 1 n + τ 1 0, te risk improvement r d is a strictly decreasing function of t on t 0, tus it is maximized at t = 0, i.e. at = wit te maximum value being zero. Note tat = corresponds to te fully parametric maximum likeliood estimator. It is not clear to us weter ν 1 n + τ 1 > 0 in general. However, we found in an example below ν 1 and τ 1 are positive (see Figure 2. For te Kullback-Leibler risk, it can be seen from (3.16 tat te U- and T-versions ave ν KL 1 n + τ KL 1 > 0 for sufficiently large n. In te subsequent discussion we assume ν 1 n+τ 1 > 0. Now, recall tat ν 1 n (1+α and τ 1 n (1+α/2. Tus, te formula (4.2 is valid only wen α > 0 since it is derived in te large setting were tends to infinity as n grows. If 0 < α < 1, ten (ν 1 n dominates τ 1. Tus, in tis case opt n α/2 and te maximum risk improvement is of order n (1+2α. Next, wen α 1, te optimal bandwidt is asymptotic to n (1+α/4 wit te maximum risk improvement being of order n (2+α. In te remaining case were 1 < α 0, we see from (4.1 tat letting tend to infinity at a slower rate makes r d larger. Tus, a bandwidt tending to infinity at an ultimately slow rate would be preferable in tis case. 16

17 We combine te results of Park et al. (2002 into our large analysis. Recall d is te dimension of X i and p is te dimension of te parameter. Wen d = 1, it was sown tat te optimal bandwidt in te small setting is asymptotic to n 1/{1+4[(p+1/2]} wit te minimal risk being of order n q 1, were q 1 = 4[(p + 1/2]/{1 + 4[(p + 1/2]} and [(p + 1/2] denotes te greatest integer wic is less tan or equal to (p + 1/2. Tis can be generalized to an arbitrary d. Let q d = 4[(p + 1/2]/{d + 4[(p + 1/2]}. It can be seen tat in te d-variate case te minimum risk n q d is acieved by te optimal bandwidt of order n 1/{d+4[(p+1/2]}. Note tat te first order in te risk expansion for large is n (1+α +n 1. Comparing tis wit te small optimal risk n q d and taking into account te discussion in te previous paragrap, we arrive at te following conclusion. We find tat te value of α at wic a transition from a small to a large bandwidt is desirable is α = q d 1. (i 1 α < q d 1: opt n 1/{d+4[(p+1/2]} ; (ii q d 1 < α 0: tending to infinity at an ultimately slow rate is preferable; (iii 0 < α < 1: opt n α/2 ; (iv 1 α: opt n (1+α/4. Te large asymptotic formula (4.2 and te small results provided in Park et al. (2002 may be used to produce useful bandwidt selectors. For example, plug-in metods are immediate from te formula (4.2, were θ is replaced by te solution of te likeliood equation n i=1 u(x i, θ = 0 and oter unknown quantities by teir obvious empirical versions. Least squares cross-validation is an alternative way of coosing a datadriven bandwidt selector, and is readily applicable to local likeliood density estimation. Te latter is not so tied to asymptotics and does not depend on te knowledge of α. Tus, it may be used for a goodness-of-fit test of a parametric model, were te parametric model is rejected for small values of cross-validatory bandwidt selector. Determination of te cut-off values in tis case requires te sampling distribution of te bandwidt selector. Tis would be a callenging problem for future researc. 5. A skewed normal example We compare te large properties of te U- and C-versions of te local likeliood estimation. Note tat te T-version as te same first and second order properties wit 17

18 te U-version as we pointed out in te paragrap immediately after te statement of Teorem 2. We consider N(θ, 1 as te parametric model. We take w(x 1 in te definition of te risk. Te true density is taken to be g(x g β (x = 2φ(xΦ(βx, (4.1 were φ and Φ are te standard normal density and its distribution function. Tis is te so-called skewed normal distribution of Azzalini (1985, and was also considered by Eguci and Copas (1998. Here, β acts as a discrepancy parameter. Wen β = 0, te density g is identical to φ. As β increases, it becomes increasingly skewed. In tis setting, we find θ = EX = 2 β π. ( β 2 Te integrated squared distance between te true density g and its best parametric approximant φ( θ, wic is ν 0 function of β. Figure 1 depicts ν 0 as a function of β. = {φ(x θ 2φ(xΦ(βx} 2 dx, is a symmetric (Insert Figure 1 about ere We calculate some ingredients to evaluate te risks given in Teorems 1 and 2. We find I (θ = 1, I 0, = 1 θ 2 and D = θ 2. For computing ν i and τ i, we use te formula for te odd moments of te skewed normal distribution given in Corollary 4 of Henze (1986. In particular, we find in addition to (4.2 E X 3 = ( 2 3 ( π β β β β 2 E X 5 = ( 2 5 ( 3 π 3 β β β β 2 ( β 1 + β 2. For te even moments we obtain E X 2k = (2k!/(2 k k!, and tus EX 2 = 1, EX 4 = 3, EX 6 = 15. Tese formula may be also obtained by applying Corollaries 3.2 and 5.3 of Aldersof et al. (1995. It may be seen tat all ν i and τ i are symmetric as functions of β. Furtermore, E Z U (y dy U(x = 2 θ2 C (1 θ 2 3 (x θ φ(x θ E(X 1 θ 3, were C = 1 for te U-version and C = {y 2 /(1 + y 2 }φ(y dy for te C-version. Tus, since τ 1,S = τ 1,U 2 f (x{e Z U (y dy U(x} dx and f (x = φ(x θ, we ave 18

19 τ 1,S = τ 1,U for bot te U- and C-versions. Similarly, we may find τ 2,S = τ 2,U, but in tis case only for te U-version. Figure 2 sows ν 1, τ 1 and τ 2. We find tat ν 1 and τ 1 are positive and converge to zero as β tends to zero. Also, from Figure 2(a we find tat te U-versions ave less bias tan te C-version, and tat te scaling improves te bias. If one plugs te values of ν 1, τ 1 and τ 2 into te formula (4.2, one may see ow opt canges as β increases. We found tat for a sample of size 100 te optimal bandwidt for te U-version of ˆf decreases from infinity to.31 as β increases from zero to 10, and for te C-version it takes values from infinity to.30. (Insert Figure 2 about ere Now, we evaluate te maximum risk improvements r d ( opt = (ν 1 n + τ 1 2 /(4nτ 2 acieved by te U- and C-versions upon te parametric maximum likeliood estimator. Note tat te maximum risk improvement does not depend on te coice of kernel. It is symmetric about zero as a function of β. Figure 3 depicts r d ( opt wen n = 100 and 400. Comparing te scaled estimator ĝ wit te unscaled ˆf, we find bot U(T- and C- versions of ĝ outperform te corresponding versions of ˆf for all β. Also, it is interesting to find tat te U(T-version is better tan te C-version for all β in te case of ĝ, but tat te risks of te unscaled estimators ˆf are indistinguisable altoug te C-version now is sligtly better. We found tat tis is true for oter sample sizes, too. (Insert Figure 3 about ere We conducted a small simulation to ceck te validity of our discussion on te optimal bandwidt size. For tis, we took te standard normal density as te kernel function. We calculated an optimal bandwidt wic minimizes te sum of squared n deviations { ˆf (X i g β (X i } 2. Our simulation consists of te tree steps; (i for eac i=1 β = 0, 0.5, 1, 2, 5, and 10, generate a random sample of size n = 100 from te density g β at (4.1 by te rejection metod; (ii compute ˆf for te U- and C-versions at eac data point X 1, X 2,..., X n ; (iii find te optimal bandwidt over te interval (0, 30 wic n minimizes te sum of squared deviations { ˆf (X i g β (X i } 2. Tese steps were repeated i=1 100 times. Table 1 sows te average of te 100 calculated optimal bandwidts for eac value of β. We see tat it clearly justifies our teoretical observation tat te optimal bandwidt traverses from a large to a small value as te degree of discrepancy from te 19

20 parametric model (in tis example, β increases. We note tat te teoretical values.31 and.30 at β = 10 discussed two paragraps above do not matc well wit.13 in te table because te teoretical values are obtained from te formula tat is valid in te near parametric case. (Insert Table 1 about ere Table 1: Average of te optimal bandwidt U-version β C-version β

21 Appendix A.1. Additional assumptions. following assumptions for te lemmas and te teorem. (A4 te solution θ (x defined at (2.7 is unique; In addition to te assumptions (A1 (A3, we need te (A5 te underlying density g and its best parametric approximation f ave compact supports; (A6 f(x, θ is tree times partially differentiable wit respect to θ and all te partial derivatives are continuous in x and θ; (A7 tere exists a function G wic is continuous and satisfies for all x 2 sup u(x, θ θ Θ θ2 G(x; A.2. Proof of Lemma 2. Here, all O expressions are uniform for x in any compact subset S of X, i.e. for a sequence of functions Q n, we say simply Q n, (x = O{r(n, } instead of sup x S Q n, (x = O{r(n, }. It follows from (3.7 tat Tis implies θ (x = θ E ˆΨ ( (x, θ = O 1 n (1+α/2 2 [ ] 1 θ E ˆΨ (x, θ E ˆΨ ( (x, θ + O θ=θ. (A.1 1 n (1+α 4. (A.2 Using te conditions (A2 and (A3, te identity (2.6, and te fact f(x, θ dx = 0, we may verify [ ] θ E ˆΨ ( (x, θ = I (x, θ + O θ=θ = I (θ + O 1 n (1+α/2 2 ( 1 2. Plug te second approximation at (A.3 into (A.2 and use (A.1 to get (A.3 θ (x = θ + I (θ 1 E ˆΨ (x, θ + O(ρ. (A.4 Te lemma follows immediately from (A.4. 21

22 A.3. Proof of Lemma 3. compact subset S of X. First, we observe ˆθ (x = θ (x In tis proof, all O p expressions are also uniform for x in any [ ] 1 θ E ˆΨ (x, θ θ=θ (x ( log n ˆΨ (x, θ (x + O p. (A.5 n Te proof of (A.5 is similar to tat of (4.1 in Park et al. (2002. Te only difference is tat we let tend to infinity instead of zero and tus only ave O p (n 1 log n instead of O p (n 1 1 log n for te remainder. Now, we can replace ˆΨ (x, θ (x by U (x, θ (x in (A.5 since E ˆΨ (x, θ (x = 0 by definition of θ (x. Also, we can replace [ ( / θe ˆΨ (x, θ ] θ=θ (x by I (x, θ wit an error O p {n 1 (α/2 2 (log n 1/2 }. Tis is due to te facts tat U (x, θ (x = O p {n 1/2 (log n 1/2 } and tat [ ( / θe ˆΨ (x, θ ] [ θ=θ (x ( / θe ˆΨ (x, θ ] as magnitude of order O(n (1+α/2 2 by (A.1 and (A.2. Also, [ θ=θ ( / θe ˆΨ (x, θ ] +I (x, θ = O(n (1+α/2 2 by te first approximation at (A.3. θ=θ Tis yields ˆθ (x = θ (x + I (x, θ 1 U (x, θ (x + O p (δ. Te lemma ten follows immediately from te facts U (x, θ = O p {n 1/2 (log n 1/2 } and θ (x θ = O(n (1+α/

23 Acknowledgments. of an associate editor and two reviewers. We are grateful for te elpful and constructive comments 23

24 References Aldersof, B., Marron, J. S., Park, B. U. and Wand, M. P. (1995. Facts about te Gaussian probability density function. Applicable Analysis 59, Azzalini, A. (1985. A class of distributions wic includes te normal ones. Scand. J. Statist. 12, Copas, J. B. (1995. Local likeliood based on kernel censoring. J. R. Statist. Soc. 57, B Eguci, S. and Copas, J. B. (1998. A class of local likeliood metods and nearparametric asymptotics. J. R. Statist. Soc. B 60, Henze, N. (1986. A probabilistic representation of te skewed-normal distribution. Scan. J. Statist. 13, Hjort, N. L. and Jones, M. C. (1996. Locally parametric nonparametric density estimation. Ann. Statist. 24, Kim, W. C., Park, B. U., and Kim, Y. G. (2001. On Copas local likeliood density estimator. J. Kor. Statist. Soc. 30, Loader, C. R. (1996. Local likeliood density estimation. Ann. Statist. 24, Park, B. U., Kim, W. C. and Jones, M. C. (2002. On local likeliood density estimation. Ann. Statist. 30,