Nonparametric Estimation: Smoothing and Data Visualization

Transcription

1 Nnparametric Estimatin: Smthing and Data Visualizatin Rnald Dias Universidade Estadual de Campinas 1 Clóqui da Regiã Sudeste Abril de 2011

2

3 Preface In recent years mre and mre data have been cllected in rder t etract infrmatin r t learn valuable characteristics abut eperiments, phenmena, bservatinal facts, etc.. This is what it s been called learning frm data. Due t their cmpleity, several datasets have been analyzed by nnparametric appraches. This field f Statistics impse minimum assumptins t get useful infrmatin frm data. In fact, nnparametric prcedures, usually, let the data speak fr themselves. This wrk is a brief intrductin t a few f the mst useful prcedures in the nnparametric estimatin tward smthing and data visualizatin. In particular, it describes the thery and the applicatins f nnparametric curve estimatin (density and regressin) prblems with emphasis in kernel, nearest neighbr, rthgnal series, smthing splines methds. The tet is designed fr undergraduate students in mathematical sciences, engineering and ecnmics. It requires at least ne semester in calculus, prbability and mathematical statistics. iii

4

5 Cntents Preface List f Figures 1 Intrductin 1 2 Kernel estimatin The Histgram Kernel Density Estimatin The Nearest Neighbr Methd Sme Statistical Results fr Kernel Density Estimatin Bandwidth Selectin Reference t a Standard Distributin Maimum likelihd Crss-Validatin Least-Squares Crss-Validatin Orthgnal series estimatrs Kernel nnparametric Regressin Methd k-nearest Neighbr (k-nn) Lcal Plynmial Regressin: LOWESS Penalized Maimum Likelihd Estimatin Cmputing Penalized Lg-Likelihd Density Estimates Spline Functins Acquiring the Taste Lgspline Density Estimatin Splines Density Estimatin: A Dimensinless Apprach The thin-plate spline n R d Additive Mdels Generalized Crss-Validatin Methd fr Splines nnparametric Regressin Regressin splines, P-splines and H-splines Sequentially Adaptive H-splines P-splines Adaptive Regressin via H-Splines Methd A Bayesian Apprach t H-splines Final Cmments 55 Bibligraphy 57 iii vii v

6

7 List f Figures 2.1 Naive estimate cnstructed frm Old faithful geyser data with h = Kernel density estimate cnstructed frm Old faithful geyser data with Gaussian kernel and h = Bandwidth effect n kernel density estimates. The data set incme was rescaled t have mean Effect f the smthing parameter k n the estimates Cmparisn f tw bandwidths, ˆσ (the sample standard deviatin) and ˆR (the sample interquartile) fr the miture 0.7 N( 2, 1) N(1, 1) Effect f the smthing parameter K n the rthgnal series methd fr density estimatin Effect f bandwiths n Nadaraya-Watsn kernel Effect f the smthing parameter k n the k-nn regressin estimates Effect f the smthing parameter using LOWESS methd Basis Functins with 6 knts placed at Basis Functins with 6 knts placed at lgspline density estimatin fr.5n(0,1)+.5n(5,1) Histgram, SSDE, Kernel and Lgspline density estimates True, tensr prduct, gam nn-adaptive and gam adaptive surfaces Smthing spline fitting with smthing parameter btained by GCV methd Spline least square fittings fr different values f K Five thusand replicates f y() = ep( ) sin(π/2) cs(π) + ɛ Five thusand replicates f the affinity and the partial affinity fr adaptive nnparametric regressin using H-splines with the true curve Density estimates f the affinity based n five thusand replicates f the curve y i = i 3 + ɛ i with ɛ i N(0,.5). Slid line is a density estimate using beta mdel and dtted line is a nnparametric density estimate A cmparisn between smthing splines (S-splines) and hybrid splines (H-splines) methds smth spline and P-spline H-spline fitting fr airmiles data vii

8 6.8 Estimatin results: a) Bayesian estimate with a = 17 and ψ(k) = K 3 (dtted line); b) (SS) smthing splines estimate (dashed line). The true regressin functin is als pltted (slid line). The SS estimate was cmputed using the R functin smth.spline frm which 4 degrees f freedm were btained and λ was cmputed by GCV One hundred estimates f the curve 6.8 and a Bayesian cnfidence interval fr the regressin curve g(t) = ep( t 2 /2) cs(4πt) with t [0, π].. 53

9 Chapter 1 Intrductin Prbably, the mst used prcedure t describe a pssible relatinship amng variables is the statistical technique knwn as regressin analysis. It is always useful t begin the study f regressin analysis by making use f simple mdels. Fr this, assume that we have cllected bservatins frm a cntinuus variable Y at n values f a predict variable T. Let (t j, y j ) be such that: y j = g(t j ) + ε j, j = 1,..., n, (1.1) where the randm variables ε j are uncrrelated with mean zer and variance σ 2. Mrever, g(t j ) are the values btained frm sme unknwn functin g cmputed at the pints t 1,..., t n. In general, the functin g is called regressin functin r regressin curve. A parametric regressin mdel assumes that the frm f g is knwn up t a finite number f parameters. That is, we can write a parametric regressin mdel by, y j = g(t j, β) + ε j, j = 1,..., n, (1.2) where β = (β 1,..., β p ) T R p. Thus, t determine frm the data a curve g is equivalent t determine the vectr f parameters β. One may ntice that, if g has a linear frm, i.e., g(t, β) = p j=1 β j j (t), where { j (t)} p j=1 are the eplanatry variables, e.g., as in plynmial regressin j (t) = t j 1, then we are dealing with a linear parametric regressin mdel. Certainly, there are ther methds f fitting curves t data. A cllectin f techniques knwn as nnparametric regressin, fr eample, allws great fleibility in the pssible frm f the regressin curve. In particular, assume n parametric frm fr g. In fact, a nnparametric regressin mdel makes the assumptin that the regressin curve belngs t sme infinite cllectin f curves. Fr eample, g can be in the class f functins that are differentiable with square integrable secnd derivatives, etc. Cnsequently, in rder t prpse a nnparametric mdel ne may just need t chse an apprpriate space f functins where he/she believes that the regressin curve lies. This chice, usually, is mtivated by the degree f the smthness f g. Then, ne uses the data t determine an element f this functin space that can represent the unknwn regressin curve. Cnsequently, nnparametric techniques rely mre heavily n the data fr infrmatin abut g than their parametric cunterparts. Unfrtunately, nnparametric estimatrs have sme disadvantages. In general, they are less efficient than the parametric estimatrs when the parametric mdel is crrectly specified. Fr 1

10 2 Chapter 1: Intrductin mst parametric estimatrs the risk will decay t zer at a rate f n 1 while nnparametric estimatrs decay at a rate f n α, where the parameter α (0, 1) depends n the smthness f g. Fr eample, when g is twice differentiable the rate is usually, n 4/5. Hwever, in the case where the parametric mdel is incrrectly specified, ad hc, the rate n 1 cannt be achieved. In fact, the parametric estimatr des nt even cnverge t the true regressin curve.

11 Chapter 2 Kernel estimatin Suppse we have n independent measurements {(t i, y i )} i=1 n, the regressin equatin is, in general, described as in (1.1). Nte that the regressin curve g is the cnditinal epectatin f the independent variable Y given the predict variable T, that is, g(t) = E[Y T = t]. When we try t apprimate the mean respnse functin g, we cncentrate n the average dependence f Y n T = t. This means that we try t estimate the cnditinal mean curve g(t) = E[Y T = t] = y f TY(t, y) dy, (2.1) f T (t) where f TY (t, y) dentes the jint density f (T, Y) and f T (t) the marginal density f T. In rder t prvide an estimate ĝ(t) f g we need t btain estimates f f TY (t, y) and f T (t). Cnsequently, density estimatin methdlgies will be described. 2.1 The Histgram The histgram is ne f the first, and ne f the mst cmmn, methds f density estimatin. It is imprtant t bear in mind that the histgram is a smthing technique used t estimate the unknwn density and hence it deserves sme cnsideratin. Let us try t cmbine the data by cunting hw many data pints fall int a small interval f length h. This kind f interval is called a bin. Observe that the well knwn dt plt f B, Hunter and Hunter (1978) is a particular type f histgram where h = 0. Withut lss f generality, we cnsider a bin centered at 0, namely the interval [ h/2, h/2) and let F X be the distributin functin f X such that F X is abslutely cntinuus with respect t a Lesbegue measure n R. Cnsequently the prbability that an bservatin f X will fall int the interval [ h/2, h/2) is given by: P(X [ h/2, h/2)) = h/2 h/2 f X ()d, where f X is the density f X. A natural estimate f this prbability is the relative frequency f the bservatins in this interval, that is, we cunt the number f bservatins falling int the interval and 3

12 4 Chapter 2: Kernel estimatin divide it by the ttal number f bservatins. In ther wrds, given the data X 1,..., X n, we have: P(X [ h/2, h/2)) 1 n #{X i [ h/2, h/2)}. Nw applying the mean value therem fr cntinuus bunded functin we btain, P(X [ h/2, h/2)) = h/2 h/2 f ()d = f (ξ)h, with ξ [ h/2, h/2). Thus, we arrive at the fllwing density estimate: ˆf h () = 1 nh #{X i [ h/2, h/2)}, fr all [ h/2, h/2). Frmally, suppse we bserve randm variables X 1,..., X n whse unknwn cmmn density is f. Let k be the number f bins, and define C j = [ 0 + (j 1)h, 0 + jh), j = 1,..., k. Nw, take n j = i=1 n I(X i C j ), where the functin I( A) is defined t be : { 1 if A I( A) = 0 therwise, and, k j=1 n j = n. Then, ˆf h () = 1 k nh n j I( C j ), j=1 fr all. Here, nte that the density estimate ˆf h depends upn the histgram bandwidth h. By varying h we can have different shapes f ˆf h. Fr eample, if ne increases h, ne is averaging ver mre data and the histgram appears t be smther. When h 0, the histgram becmes a very nisy representatin f the data (needle-plt, Härdle (1990)). The ppsite, situatin when h, the histgram, nw, becmes verly smth (b-shaped). Thus, h is the smthing parameter f this type f density estimate, and the questin f hw t chse the histgram bandwidth h turns ut t be an imprtant questin in representing the data via the histgram. Fr details n hw t estimate h see Härdle (1990). 2.2 Kernel Density Estimatin The mtivatin behind the histgram can be epanded quite naturally. Fr this cnsider a weight functin, and define the estimatr, K() = ˆf () = 1 nh { 12, if < 1 0, therwise n i=1 K( X i ). (2.2) h

13 2.2: Kernel Density Estimatin 5 We can see that ˆf etends the idea f the histgram. Ntice that this estimate just places a b f side (width) 2h and height (2nh) 1 n each bservatin and then sums t btain ˆf. See Silverman (1986) fr a discussin f this kind f estimatr. It is nt difficult t verify that ˆf is nt a cntinuus functin and has zer derivatives everywhere ecept n the jump pints X i ± h. Besides having the undesirable character f nnsmthness (Silverman (1986)), it culd give a misleading impressin t a untrained bserver since its smewhat ragged character might suggest several different bumps. Figure 2.1 shws the nnsmth character f the naive estimate. The data seem t have tw majr mdes. Hwever, the naive estimatr suggests several different small bumps Eruptins length density estimate Figure 2.1: Naive estimate cnstructed frm Old faithful geyser data with h = 0.1 T vercme sme f these difficulties, assumptins have been intrduced n the functin K. That is, K must be a nnnegative kernel functin that satisfies the fllwing prperty: K()d = 1. In ther wrds K() is a prbability density functin, as fr instance, the Gaussian density, it will fllw frm definitin that ˆf will itself be a prbability density. In additin, ˆf will inherit all the cntinuity and differentiability prperties f the kernel K.

14 6 Chapter 2: Kernel estimatin Fr eample, if K is a Gaussian density then ˆf will be a smth curve with derivatives f all rders. Figure 2.2 ehibits the smth prperties f ˆf when a Gaussian kernel is used Eruptins length density estimate Figure 2.2: Kernel density estimate cnstructed frm Old faithful geyser data with Gaussian kernel and h = 0.25 Nte that an estimate based n the kernel functin places bumps n the bservatins and the shape f thse bumps is determined by the kernel functin K. The bandwidth h sets the width arund each bservatin and this bandwidth cntrls the degree f smthness f a density estimate. It is pssible t verify that as h 0, the estimate becmes a sum f Dirac delta functins at the bservatins while as h, it eliminates all the lcal rughness and pssibly imprtant details are missed. The data fr the Figure 2.3 which is labelled incme were prvided by Charles Kperberg. This data set cnsists f 7125 randm samples f yearly net incme in the United Kingdm (Family Ependiture Survey, ). The incme data is cnsiderably large and s it is mre f a challenge t cmputing resurces and there are severe utliers. The peak at 0.24 is due t the UK ld age pensin, which caused many peple t have nearly identical incmes. The width f the peak is abut 0.02, cmpared t the range 11.5 f the data. The rise f the density t the left f the peak is very steep. There is a vast (Silverman (1986)) literature n kernel density estimatin studying its mathematical prperties and prpsing several algrithms t btain estimates based n it. This methd f density estimatin became, apart frm the histgram, the mst cmmnly used estimatr. Hwever it has drawbacks when the underlying

15 2.2: Kernel Density Estimatin 7 Histgram f incme data Relative Frequency h=r default h=.12 h=.25 h= transfrmed data Figure 2.3: Bandwidth effect n kernel density estimates. The data set incme was rescaled t have mean 1. density has lng tails Silverman (1986). What causes this prblem is the fact that the bandwidth is fied fr all bservatins, nt cnsidering any lcal characteristic f the data. In rder t slve this prblem several ther Kernel Density Estimatin Methds were prpsed such as the nearest neighbr and the variable kernel. A detailed discussin and illustratin f these methds can be fund in Silverman (1986) The Nearest Neighbr Methd The idea behind the nearest neighbr methd is t adapt the amunt f smthing t lcal characteristics f the data. The degree f smthing is then cntrlled by an integer k. Essentially, the nearest neighbr density estimatr uses distances frm in f () t the data pint. Fr eample, let d( 1, ) be the distance f data pint 1 frm the pint, and fr each dente d k () as the distance frm its kth nearest neighbr amng the data pints 1,..., n. The kth nearest neighbr density estimate is defined as, ˆf () = k 2nd k (), where n is the sample size and, typically, k is chsen t be prprtinal t n 1/2. In rder t understand this definitin, suppse that the density at is f (). Then, ne wuld epect abut 2rn f () bservatins t fall in the interval [ r, + r] fr

16 8 Chapter 2: Kernel estimatin each r > 0. Since, by definitin, eactly k bservatins fall in the interval [ d k (), + d k ()], an estimate f the density at may be btained by putting k = 2d k ()n ˆf (). Nte that while estimatrs like histgram are based n the number f bservatins falling in a b f fied width centered at the pint f interest, the nearest neighbr estimate is inversely prprtinal t the size f the b needed t cntain a given number f bservatins. In the tail f the distributin, the distance d k () will be larger than in the main part f the distributin, and s the prblem f under-smthing in the tails shuld be reduced. Like the histgram the nearest neighbr estimate is nt a smth curve. Mrever, the nearest neighbr estimate des nt integrate t ne and the tails f ˆf () die away at rate 1, in ther wrds etremely slwly. Hence, this estimate is nt apprpriate if ne is required t estimate the entire density. Hwever, it is pssible t generalize the nearest neighbr estimatr in a manner related t the kernel estimate. The generalized kth nearest neighbr estimate is defined by, ˆf () = 1 n nd k () K( X i d i=1 k () ). Observe that the verall amunt f smthing is gverned by the chice f k, but the bandwidth used at any particular pint depends n the density f bservatins near that pint. Again, we face the prblems f discntinuity f at all the pints where the functin d k () has discntinuus derivative. The precise integrability and tail prperties will depend n the eact frm f the kernel. Figure 2.4 shws the effect f the smthing parameter k n the density estimate. Observe that as k increases rugher the density estimate becmes. This effect is equivalent when h is appraching t zer in the kernel density estimatr. 2.3 Sme Statistical Results fr Kernel Density Estimatin As starting pint ne might want t cmpute the epected value f ˆf. Fr this, suppse we have X i,..., X n i.i.d. randm variables with cmmn density f and let K( ) be a prbability density functin defined n the real line. Then we have, fr a nnstchastic h E[ ˆf ()] = 1 nh n i=1 E[K( X i )] h = 1 h E[K( X i )] h = 1 K( u ) f (u)du h h = K(y) f ( + yh)dy. (2.3) Nw, let h 0. We see that E[ ˆf ()] f () K(y)dy = f (). Thus, ˆf is an asympttic unbiased estimatr f f.

17 2.3: Sme Statistical Results fr Kernel Density Estimatin bs. frm N(0.5,0.1) density True k=40 k=30 k= Figure 2.4: Effect f the smthing parameter k n the estimates data T cmpute the bias f this estimatr we have t make the assumptin that the underlying density is twice differentiable and satisfies the fllwing cnditins Prakasa- Ra (1983): Cnditin 1. sup K() M < ; K() 0 as. Cnditin 2. K() = K( ), (, ) with 2 K()d <. Then by using a Taylr epansin f f ( + yh), the bias f ˆf in estimating f is b f [ ˆf ()] = h2 2 f () y 2 K(y)dy + (h 2 ). We bserve that since we have assumed the kernel K is symmetric arund zer, we have that yk(y)h f ()dy = 0, and the bias is quadratic in h. Parzen (1962) Using a similar apprach we btain : Var f [ ˆf ()] = nh 1 K f () + ( nh ), where K 2 2 = K() 2 d MSE f [ ˆf ()] = 1 nh f () K h4 4 ( f () y 2 K(y)dy) 2 + ( 1 nh ) + (h4 ), where MSE f [ ˆf ] stands fr mean squared errr f the estimatr ˆf f f. Hence, when the cnditins h 0 and nh are assumed, the MSE f [ ˆf ] 0, which means that the kernel density estimate is a cnsistent estimatr f the underlying density f. Mrever, MSE balances variance and squared bias f the estimatr in such way that the variance term cntrls the under-smthing and the bias term cntrls ver-smthing. In ther wrds, an attempt t reduce the bias increases the variance, making the estimate t nisy (under-smth). On the cntrary, minimizing the variance leads t a very smth estimate (ver-smth) with high bias.

18 10 Chapter 2: Kernel estimatin 2.4 Bandwidth Selectin It is natural t think f finding an ptimal bandwidth by minimizing MSE f [ ˆf ] in h > 0. Härdle(1990) shws that the asympttic apprimatin, say, h fr the ptimal bandwidth is ( f () K 2 2 h = ( f ()) 2 ( ) 1/5 n 1/5 y 2 K(y)dy) 2. (2.4) n The prblem with this apprach is that h depends n tw unknwn functins f ( ) and f ( ). An apprach t vercme this prblem uses a glbal measure that can be defined as: IMSE[ ˆf ] = MSE f [ ˆf ()]d = 1 nh K h4 4 ( y 2 K(y)dy) 2 f ( 1 nh ) + (h4 ). (2.5) IMSE is the well knwn integrated mean squared errr f a density estimate. The ptimal value f h cnsidering the IMSE is define as it can be shwn that, ( h pt = c 2/5 2 h pt = arg min h>0 IMSE[ ˆf ]. ) 1/5 ( 1/5n K 2 ()d f 2) 2 1/5, (2.6) where c 2 = y 2 K(y)dy. Unfrtunately, (2.6) still depends n the secnd derivative f f, which measures the speed f fluctuatins in the density f f Reference t a Standard Distributin A very natural way t get arund the prblem f nt knwing f is t use a standard family f distributins t assign a value f the term f 2 2 in epressin (2.6). Fr eample, assume that a density f belngs t the Gaussian family with mean µ and variance σ 2, then ( f ()) 2 d = σ 5 (ϕ ()) 2 d = 3 8 π 1 2 σ σ 5, (2.7) where ϕ() is the standard nrmal density. If ne uses a Gaussian kernel, then h pt = (4π) 1/10 ( 3 8 π 1/2 ) 1/5 σ n 1/5 = ( 4 3 ) 1/5 σ n 1/5 = 1.06 σ n 1/5 (2.8) Hence, in practice a pssible chice fr h pt is 1.06 ˆσ n 1/5, where ˆσ is the sample standard deviatin.

19 2.4: Bandwidth Selectin 11 If we want t make this estimate mre insensitive t utliers, we have t use a mre rbust estimate fr the scale parameter f the distributin. Let ˆR be the sample interquartile, then ne pssible chice fr h is ˆR ĥ pt = 1.06 min(ˆσ, (Φ(3/4) Φ(1/4)) ) n 1/5 = 1.06 min( ˆσ, ˆR ) n 1/5, (2.9) where Φ is the standard nrmal distributin functin. Figure 2.5 ehibits hw a rbust estimate f the scale can help in chsing the bandwidth. Nte that by using ˆR we have strng evidence that the underlying density has tw mdes. Histgram f a miture f tw nrmal densities Relative Frequency True sigmahat interquartile Figure 2.5: Cmparisn f tw bandwidths, ˆσ (the sample standard deviatin) and ˆR (the sample interquartile) fr the miture 0.7 N( 2, 1) N(1, 1). data Maimum likelihd Crss-Validatin Cnsider kernel density estimates ˆf and suppse we want t test fr a specific h the hypthesis ˆf () = f () vs. ˆf () = f (), fr a fied The likelihd rati test wuld be based n the test statistic f ()/ ˆf (). Fr a gd bandwidth this statistic shuld thus be clse t 1. Alternatively, we wuld

20 12 Chapter 2: Kernel estimatin epect E[lg( f (X) )] t be clse t 0. Thus, a gd bandwidth, which is minimizing this ˆf (X) measure f accuracy, is in effect ptimizing the Kullback-Leibler distance: d KL ( f, ˆf ) = ( f () ) lg f ()d. (2.10) ˆf () Of curse, we are nt able t cmpute d KL ( f, ˆf ) frm the data, since we d nt knw f. But frm a theretical pint f view, we can investigate this distance fr the chice f an apprpriate bandwidth h. When d KL ( f, ˆf ) is clse t 0 this wuld give the best agreement with the hypthesis ˆf = f. Hence, we are lking fr a bandwidth h, which minimizes d KL ( f, ˆf ). Suppse we are given a set f additinal bservatins X i, independent f the thers. The likelihd fr these bservatins is i f (X i ). Substituting ˆf in the likelihd equatin we have i ˆf (Xi ) and the value f this statistic fr different h wuld indicate which value f h is preferable, since the lgarithm f this statistic is clse t d KL ( f, ˆf ). Usually, we d nt have additinal bservatins. A way ut f this dilemma is t base the estimate ˆf n the subset {X j } j =i, and t calculate the likelihd fr X i. Denting the leave-ne-ut estimate Hence, n i=1 ˆf (X i ) = (n 1) 1 h 1 j =i ˆf (X i ) = (n 1) n h n n i=1 K( X i X j ). h j =i K( X i X j ). (2.11) h Hwever it is cnvenient t cnsider the lgarithm f this statistic nrmalized with the factr n 1 t get the fllwing prcedure: CV KL (h) = 1 n = 1 n n i=1 lg[ f h,i (X i )] n lg i=1 Naturally, we chse h KL such that: [ j =i K( X i X ] j ) lg[(n 1)h] (2.12) h h KL = arg ma CV KL (h). (2.13) h Since we assumed that X i are i.i.d., the scres lg ˆf i (X i ) are identically distributed and s, E[CV KL (h)] = E[lg ˆf i (X i )]. Disregarding the leave-ne-ut effect, we can write [ E[CV KL (h)] E lg ˆf ] () f ()d lg[ f ()] f ()d. (2.14) E[d KL ( f, ˆf )] +

21 2.4: Bandwidth Selectin 13 The secnd term f the right-hand side des nt depend n h. Then, we can epect that we apprimate the ptimal bandwidth that minimizes d KL ( f, ˆf ). The Maimum likelihd crss validatin has tw shrtcmings: When we have identical bservatins in ne pint, we may btain an infinite value if CV KL (h) and hence we cannt define an ptimal bandwidth. Suppse we use a kernel functin with finite supprt, e.g., the interval [ 1, 1]. If an bservatin X i is mre separated frm the ther bservatins than the bandwidth h, the likelihd ˆf i (X i ) becmes 0. Hence the scre functin reaches the value. Maimizing CV KL (h) frces us t use a large bandwidth t prevent this degenerated case. This might lead t slight ver-smthing fr the ther bservatins Least-Squares Crss-Validatin Cnsider an alternative distance between f h and f. The integrated squared errr (ISE) d ISE (h) = ( f h f ) 2 ()d = fh 2()d 2 ( f h f )()d + f 2 ()d d ISE (h) f 2 ()d = fh 2()d 2 ( f h f )()d (2.15) Fr the last term, bserve that ( f h f )()d = E[ f h (X i )] where the epectatin is understd t be cmputed with respect t an additinal and independent bservatin X. Fr estimatin f this term define the leave-ne-ut estimate This leads t the Least-squares crss-validatin: E X [ fˆ h (X)] = 1 n n f h,i (X i ) (2.16) i=1 CV LS (h) = The bandwidth minimizing this functin is, fh 2 n ()d 2 f h,i (X i ) (2.17) i=1 h LS = arg min CV LS (h). h This crss-validatin functin is called an unbiased crss-validatin criterin, since, E[CV LS (h)] = E[d ISE (h) + 2(E X [ f h (X)] E[ 1 n n f h,i (X i )]) f 2 2 i=1 = IMSE[ f h ] f 2 2. (2.18) An interesting questin is, hw gd is the apprimatin f d ISE by CV LS. T investigate this define a sequence f bandwidths h n = h(x 1,..., X n ) t be asympttically ptimal, if d ISE (h n ) 1, a.s. when n. inf h>0 d ISE (h)

22 14 Chapter 2: Kernel estimatin It can be shwn that if the density f is bunded then h LS is asympttically ptimal. Similarly t maimum likelihd crss-validatin ne can fund in Härdle (1990) an algrithm t cmpute the least-squares crss-validatin. 2.5 Orthgnal series estimatrs Orthgnal series estimatrs apprach the density estimatin prblem frm a quite different pint f view. While kernel estimatrs is clse related t statistical thinking rthgnal series relies n the ideas f apprimatin thery. Withut lss f generality let us assume that we are trying t estimate a density f n the interval [0, 1]. The idea is t use the thery f rthgnal series methd and then t reduce the estimatin prcedure by estimating the cefficients f its Furier epansin. Define the sequence φ v () by φ 0 () = 1 φ 2r 1 () = 2 cs 2πr r = 1, 2,... φ 2r () = 2 sin 2πr r = 1, 2,... It is well knwn that f can be represented as Furier series i=0 a iφ i, where, fr each i 0, a i = f ()φ i ()d. (2.19) Nw, suppse that X is a randm variable with density f. Then written a i = Eφ i (X) and s an unbiased estimatr f f based n X 1,..., X n is (2.19) can be â i = 1 n n φ i (X j ). j=1 Nte that the i=1 âiφ i cnverges t a sum f delta functins at the bservatins, since ω() = 1 n where δ is the Dirac delta functin. Then fr each i, â i = 1 0 n δ( X i ) (2.20) i=1 ω()φ i ()d and hence the â i are eactly the Furier cefficients f the functin ω. The easiest t way t smth ω is t truncate the epansin â i φ i at sme pint. That is, chse K and define a density estimate ˆf by ˆf () = K â i φ i (). (2.21) i=1 Nte that the amunt f smthing is determined by K. Small value f K implies in ver-smthing, large value f K under-smthing.

23 2.5: Orthgnal series estimatrs bs frm N(.5,.1) density True K=3 K=10 K= data Figure 2.6: Effect f the smthing parameter K n the rthgnal series methd fr density estimatin A mre general apprach wuld be, chse a sequence f weights λ i, such that, λ i 0 as i. Then ˆf () = λ i â i φ i (). i=0 The rate at which the weights λ i cnverge t zer will determine the amunt f smthing. Fr nn finite interval we can have weight functins a() = e 2 /2 and rthgnal functins φ() prprtinal t Hermite plynmials. The data in figure 2.6 were prvided t me by Francisc Cribari-Net and cnsists f the variatin rate f ICMS (impst sbre circulaçã de mercadrias e serviçs) ta fr the city f Brasilia, D.F., frm August 1994 t July 1999.

24 16 Chapter 2: Kernel estimatin

25 Chapter 3 Kernel nnparametric Regressin Methd Suppse we have i.i.d. bservatins {(X i, Y i )} i=1 n and the nnparametric regressin mdel given in equatin (1.1). By equatin (2.1) we knw hw t estimate the denminatr by using the kernel density estimatin methd. Fr the numeratr ne can estimate the jint density using the multiplicative kernel f h1,h 2 (, y) = 1 n n K h1 ( X i )K h2 (y Y i ). i=1 where, K h1 ( X i ) = h1 1 K(( X i)/h 1 ), K h2 ( Y i ) = h2 1 K(( Y i)/h 2 ). It is nt difficult t shw that y f h1,h 2 (, y)dy = 1 n n K h1 ( X i )Y i. i=1 Based n the methdlgy f kernel density estimatin Nadaraya (1964) and Watsn (1964) suggested the fllwing estimatr g h fr g. g h () = n i=1 K h( X i )Y i n j=1 K h( X j ) (3.1) In general, the kernel functin K h () = K(( j )/h) is taken as prbability density functin symmetric arund zer and parameter h is called smthing parameter r bandwidth. Nw, cnsider the mdel (1.1) and let X 1,..., X n be i.i.d. randm variables with density f X such that X i is independent f ε i fr all i = 1,..., n. Assume the cnditins given in Sectin 2.3 and suppse that f and g are twice cntinuusly differentiable in neighbrhd f the pint. Then, if h 0 and nh as n, we have ĝ h g in prbability. Mrever, suppse E[ ε i 2+δ ] and K( 2+δ d <, fr sme δ > 0, then nh(ĝ h E[ĝ h ]) N(0, ( f X ()) 1 σ 2 (K()) 2 d) in distributin, where N(, ) stands fr a Gaussian distributin, (see details in Pagan and Ullah (1999)). As an eample, figure 3.1 shws the effect f chsing h n the Nadaraya-Watsn prcedure. The data cnsist f the speed f cars and the distances taken t stp. It is imprtant t ntice that the data were recrded in the 1920s. (These datasets can be 17

26 18 Chapter 3: Kernel nnparametric Regressin Methd fund in the sftware R) The Nadaraya-Watsn kernel methd can be etended t the multivariate regressin prblem by cnsidering the multidimensinal kernel density estimatin methd (see details in Sctt (1992)). dist h=2 h= speed Figure 3.1: Effect f bandwiths n Nadaraya-Watsn kernel 3.1 k-nearest Neighbr (k-nn) One may ntice that regressin by kernels is based n lcal averaging f bservatins Y i in a fied neighbrhd f. Instead f this fied neighbrhd, k-nn emplys varying neighbrhds in the X variable supprt. That is, where, g k () = 1 n W ki () = n W ki ()Y i, (3.2) i=1 { n/k if i J 0 therwise, (3.3) with J = {i : X i is ne f the k nearest bservatins t } It can be shwn that the bias and variance f the k-nn estimatr g k with weights (3.3) are given by, fr a fied and E[g k ()] g() 1 24( f ()) 3 [g () f () + 2g () f ()](k/n) 2 (3.4) Var[g k ()] σ2 k. (3.5)

27 3.2: Lcal Plynmial Regressin: LOWESS 19 We bserve that the bias increasing and the variance is decreasing in the smthing parameter k. T balance this trade-ff ne shuld chse k n 4/5. Fr details, see Härdle (1990). Figure 3.2 shws the effect f the parameter k n the regressin curve estimates. Nte that the curve estimate with k = 2 is less smther than the curve estimate with k = 1. The data set cnsist f the revenue passenger miles flwn by cmmercial airlines in the United States fr each year frm 1937 t 1960 and is available thrugh R package. airmiles data airmiles Data K=1 K= Passenger miles flwn by U.S. cmmercial airlines Figure 3.2: Effect f the smthing parameter k n the k-nn regressin estimates. 3.2 Lcal Plynmial Regressin: LOWESS Cleveland (1979) prpsed the algrithm LOWESS, lcally weighted scatter plt smthing, as an utlier resistant methd based n lcal plynmial fits. The basic idea is t start with a lcal plynmial (a k-nn type fitting) least squares fit and then t use rbust methds t btain the final fit. Specifically, ne can first fit a plynmial regressin in a neighbrhd f, that is, find β R p+1 which minimize n 1 n p W ki (y i β j j) 2, (3.6) i=1 j=0 where W ki dente k-nn weights. Cmpute the residuals ˆɛ i and the scale parameter ˆσ = median( ˆɛ i ). Define rbustness weights δ i = K( ˆɛ i /6ˆσ), where K(u) = (15/16)(1 u) 2, if u 1 and K(u) = 0, if therwise. Then, fit a plynmial regressin as in (3.6) but with weights (δ i W ki ()). Cleveland suggests that p = 1 prvides gd balance

28 20 Chapter 3: Kernel nnparametric Regressin Methd between cmputatinal ease and the need fr fleibility t reprduce patterns in the data. In additin, the smthing parameter can be determined by crss-validatin as in (2.13). Nte that when using the R functin lwess r less, f acts as the smthing parameter. Its relatin t the k-nn nearest neighbr is given by where n is the sample size. k = n f, f (0, 1), lwess(cars) dist f = 2/3 f = speed Figure 3.3: Effect f the smthing parameter using LOWESS methd. 3.3 Penalized Maimum Likelihd Estimatin The methd f penalized maimum likelihd in the cntet f density estimatin cnsist f estimating a density f by minimizing a penalized likelihd scre L ( f ) + λj( f ), where L ( f ) is a gdness-f-fit measure, and J( f ) is a rughness penalty. This sectin is develped cnsidering histrical results, beginning with Gd and Gaskins (1971), and ending with the mst recent result given by Gu (1993). The maimum likelihd (M.L.) methd has been used as statistical standard prcedure in the case where the underlying density f is knwn ecept by a finite number f parameters. It is well knwn the M.L. has ptimal prperties (asympttically unbiased and asympttically nrmal distributed) t estimate the unknwn parameters. Thus, it wuld be interesting if such standard technique culd be applied n a mre general scheme where there is n assumptin n the frm f the underlying density by assuming f t belng t a pre-specified family f density functins.

29 3.3: Penalized Maimum Likelihd Estimatin 21 Let X 1,..., X n be i.i.d. randm variables with unknwn density f. The likelihd functin is given by: n L( f X 1,..., X n ) = f (X i ). i=1 The prblem with this apprach can be described by the fllwing eample. Recall ˆf h () a kernel estimate, that is, ˆf h () = 1 n nh K( X i h i=1 ), with h = h/c, where c is cnstant greater than 0, i.e., fr the mment the bandwidth is h/c. Let h be small enugh such that X i X i h/c > M > 0, and assume K has been chsen s that K(u) = 0, if u > M. Then, ˆf h (X i ) = c nh K(0). If c > 1 K(0) then ˆf h (X i ) > 1 nh. Fr fied n, we can d this fr all X i simultaneusly. Thus, L ( 1 nh )n. Letting h 0, we have L. That is, L( f X 1,..., X n ) des nt have a finite maimum ver the class f all densities. Hence, the likelihd functin can be as large as ne wants it just by taking densities with the smthing parameter appraching zer. Densities having this characteristic, e.g., bandwidth h 0, apprimate t delta functins and the likelihd functin ends up t be a sum f spikes delta functins. Therefre, withut putting cnstraints n the class f all densities, the maimum likelihd prcedure cannt be used prperly. One pssible way t vercme the prblem described abve is t cnsider a penalized lg-likelihd functin. The idea is t intrduce a penalty term n the lglikelihd functin such that this penalty term quantifies the smthness f g = lg f. Let us take, fr instance, the functinal J(g) = (g ) 2 as a penalty term. Then define the penalized lg-likelihd functin by L λ (g) = 1 n n g(x i ) λj(g), (3.7) i=1 where λ is the smthing parameter which cntrls tw cnflicting gals, the fidelity t the data given by n i=1 g(x i) and the smthness, given by the penalty term J(g). The pineer wrk n penalized lg-likelihd methd is due t Gd and Gaskins (1971), wh suggested a Bayesian scheme with penalized lg-likelihd (using their ntatin) becmes: ω = ω( f ) = L( f ) Φ( f ), where L = i=1 n g(x i) and Φ is the smthness penalty. In rder t simplify the ntatin, let h have the same meaning as h()d. Nw, cnsider the number f bumps in the density as the measure f rughness r

30 22 Chapter 3: Kernel nnparametric Regressin Methd smthness. The first apprach was t take the penalty term prprtinal t Fisher s infrmatin, that is, Φ( f ) = ( f ) 2 / f. Nw by setting f = γ 2, Φ( f ) becmes (γ ) 2, and then replace f by γ in the penalized likelihd equatin. Ding that the cnstraint f 0 is eliminated and the ther cnstraint, f = 1, turns ut t be equivalent t γ 2 = 1, with γ L 2 (, ). Gd and Gaskins(1971) verified that when the penalty 4α (γ ) 2 yielded density curves having prtins that lked t straight. This fact can be eplained nting that the curvature depends als n the secnd derivatives. Thus (γ ) 2 shuld be included n the penalty term. The final rughness functinal prpsed was: Φ( f ) = 4α (γ ) 2 + β (γ ) 2, with α, β satisfying, 2ασ β = σ4, (3.8) where σ 2 is either an initially guessed value f the variance r it can be estimated the sample variance based n the data. Accrding t Gd and Gaskins (1971), the basis fr this cnstraint is the feeling that the class f nrmal distributins frm the smthest class f distributins, the imprper unifrm distributin being limiting frm. Mrever, they pinted ut that sme justificatin fr this feeling is that a nrmal distributin is the distributin f maimum entrpy fr a given mean and variance. The integral (γ ) 2 is als minimized fr a given variance when f is nrmal (Gd and Gaskins, 1971). They thught was reasnable t give the nrmal distributin special cnsideratin and decided t chse α, β such that ω(α, β; f ) is maimized by taking the mean equal t and variance as i=1 N ( i ) 2 /N 1. That is, if f () N (µ, σ 2 ) then (γ ) 2 = 1, (γ ) 2 = 3 and hence we have, 4σ 2 16σ 4 ω(α, β; f ) = N 2 lg(2πσ2 ) 1 2σ 2 N i=1 ( i µ) 2 α σ 2 3β 16σ 4. The scre functin ω(α, β; f ) is maimized when µ = and σ is such that, N + N i=1 ( i ) 2 σ 2 + 2α σ 2 + 3β = 0. (3.9) 4σ4 If we put σ 2 = N i=1 ( i ) 2 /N 1, the equatin (3.9) becmes, σ 4 (N 1) + 2ασ 2 + 3β 4 = σ4 N, and s we have the cnstraint (3.8). Pursuing the idea f Gd and Gaskins, Silverman (1982) prpsed a similar methd where the lg density is estimated instead f the density itself. An advantage f Silverman s apprach is that using the lgarithm f the density and the augmented Penalized likelihd functinal, any density estimates btained will autmatically be psitive and integrate t ne. Specifically,

31 3.3: Penalized Maimum Likelihd Estimatin 23 Let (m 1,..., m k ) be a sequence f natural numbers s that 1 i=1 k m i m, where m > 0 is such that g (m 1) eists and is cntinuus. Define a linear differential peratr D as: D(g) = c(m 1,..., m k )( ) m 1... ( ) m k(g). 1 k Nw assume that at least ne f the cefficients c(m 1,..., m k ) = 0 fr m i = m. Using this linear differential peratr define a bilinear functinal, by g 1, g 2 = D(g 1 )D(g 2 ). where the integral is taken ver a pen set Ω with respect t Lebesgue measure. Let S be the set f real functins g n Ω fr which: the (m 1)th derivatives f g eist everywhere and are piecewise differentiable, g, g <, e g <. Given the data X 1,..., X n i.i.d. with cmmn density f, such that g = lg f, ĝ is the slutin, if it eists, f the ptimizatin prblem ma{ 1 n n g(x i ) λ g, g }, 2 i=1 subject t e g = 1. And the density estimate ˆf = eĝ, where the the null space f the penalty term is the set {g S : g, g = 0}. Nte that the null space f g, g is an epnential family with at mst (m 1) parameters, fr eample, if g, g = (g (3) ) 2 then g = lg f is in an epnential family with 2 parameters. See Silverman (1982). Silverman presented an imprtant result which makes the cmputatin f the cnstrained ptimizatin prblem a relatively easy cmputatinal scheme f finding the minimum f an uncnstrained variatinal prblem. Precisely, fr any g in a class f smth functins (see details in Silverman (1982)) and fr any fied psitive λ, let and ω 0 (g) = 1 n ω(g) = 1 n n g(x i ) + λ 2 i=1 n g(x i ) + i=1 e g + λ 2 (g ) 2 (g ) 2. Silverman prved that uncnstrained minimum f ω(g) is identical with the cnstrained minimum f ω 0, if such a minimizer eists.

32 24 Chapter 3: Kernel nnparametric Regressin Methd Cmputing Penalized Lg-Likelihd Density Estimates Based n Silverman s apprach, O Sullivan(1988) develped an algrithm which is a fully autmatic, data driven versin f Silverman s estimatr. Furthermre, the estimatrs btained by O Sullivan s algrithm are apprimated by linear cmbinatin f basis functins. Similarly t the estimatrs given by Gd and Gaskins(1971), O Sullivan prpsed that cubic B-splines with knts at data pints shuld be used as the basis functins. A summary f definitins and prperties f B-splines were given in the sectin 4. The basic idea f cmputing a density estimate prvided by penalized likelihd methd is t cnstruct apprimatins t it. Given 1,..., n, the realizatins f randm variables X 1,..., X n, with cmmn lg density g. We are t slve a finite versin f (3.7) which are reasnable apprimatins t the infinite dimensinal prblem (Thmpsn and Tapia, 1990, ). Gd and Gaskins (1971) based their cmputatinal scheme n the fact that since γ L 2 (, ) then fr a given rthnrmal system f functins {φ n }, n=0 a n φ n m.s. g L 2, with n=0 a n < and {a n } R. That is, γ in L 2 can be arbitrarily apprimated by a linear cmbinatin f basis functins. In their paper, Hermite plynmials were used as basis functins. Specifically: where, f φ n () = e 2 /2 H n ()2 n/2 π 1/4 (n!) 1/2, H n () = ( 1) n e 2 ( dn d n e 2 ). The lg density estimatr prpsed by O Sullivan (1988) is defined as the minimizer 1 n b b n g( i ) + e g(s) ds + λ (g (m) ) 2 ds, (3.10) i=1 a a fr fied λ > 0, and data pints 1,..., n. The minimizatin is ver a class f abslutely cntinuus functins n [a, b] whse mth derivative is square integrable. Cmputatinal advantages f this lg density estimatrs using apprimatins by cubic B-splines are: It is a fully autmatic prcedure fr selecting an apprpriate value f the smthing parameter λ, based n the AIC type criteria. The banded structures induced by B-splines leads t an algrithm where the cmputatinal cst is linear in the number f bservatins (data pints). It prvides apprimate pintwise Bayesian cnfidence intervals fr the estimatr. A disadvantage f O Sullivan s wrk is that it des nt prvide any cmparisn f perfrmance with ther available techniques. We see that the previus cmputatinal framewrk is unidimensinal, althugh Silverman s apprach can be etended t higher dimensins.

33 Chapter 4 Spline Functins 4.1 Acquiring the Taste Due t their simple structure and gd apprimatin prperties, plynmials are widely used in practice fr apprimating functins. Fr this prpse, ne usually divides the interval [a, b] in the functin supprt int sufficiently small subintervals f the frm [ 0, 1 ],..., [ k, k+1 ] and then uses a lw degree plynmial p i fr apprimatin ver each interval [ i, i+1 ], i = 0,..., k. This prcedure prduces a piecewise plynmial apprimating functin s( ); s() = p i () n [ i, i+1 ], i = 0,..., k. In the general case, the plynmial pieces p i () are cnstructed independently f each ther and therefre d nt cnstitute a cntinuus functin s() n [a, b]. This is nt desirable if the interest is n apprimating a smth functin. Naturally, it is necessary t require the plynmial pieces p i () t jin smthly at knts 1,..., k, and t have all derivatives up t a certain rder, cincide at knts. As a result, we get a smth piecewise plynmial functin, called a spline functin. Definitin 4.1 The functin s() is called a spline functin (r simply spline ) f degree r with knts at { i } k i=1 if =: 0 < 1 <... < k < k+1 :=, where =: 0 and k+1 := are set by definitin, fr each i = 0,..., k, s() cincides n [ i, i+1 ] with a plynmial f degree nt greater than r; s(), s (),..., s r 1 () are cntinuus functins n (, ). The set S r ( 1,..., k ) f spline functins is called spline space. Mrever, the spline space is a linear space with dimensin r + k + 1 (Schumaker (1981)). Definitin 4.2 Fr a given pint (a, b) the functin (t ) r + = { (t ) r if t > 0 if t is called the truncated pwer functin f degree r with knt. 25

34 26 Chapter 4: Spline Functins Hence, we can epress any spline functin as a linear cmbinatin f r + k + 1 basis functins. Fr this, cnsider a set f interir knts { 1,..., k } and the basis functins {1, t, t 2,..., t r, (t 1 ) r +,..., (t k) r +}. Thus, a spline functin is given by, s(t) = r θ i t i k + θ j (t j r ) r + i=0 j=r+1 It wuld be interesting if we culd have basis functins that make it easy t cmpute the spline functins. It can be shwn that B-splines frm a basis f spline spaces Schumaker (1981). Als, B-splines have an imprtant cmputatinal prperty, they are splines which have smallest pssible supprt. In ther wrds, B-splines are zer n a large set. Furthermre, a stable evaluatin f B-splines with the aid f a recurrence relatin is pssible. Definitin 4.3 Let Ω = { j } {j Z} be a nndecreasing sequence f knts. The i-th B-spline f rder k fr the knt sequence Ω is defined by B k j (t) = ( k+j j )[ j,..., k+j ](t j ) k 1 + fr all t R, where, [ j,..., k+j ](t j ) k 1 + is (k 1)th divided difference f the functin ( j) k + evaluated at pints j,..., k+j. Frm the Definitin 4.3 we ntice that B k j (t) = 0 fr all t [ j, j+k ]. It fllws that nly k B-splines have any particular interval [ j, j+1 ] in their supprt. That is, f all the B- splines f rder k fr the knt sequence Ω, nly the k B-splines B k j k+1, Bk j k+2,..., Bk j might be nnzer n the interval [ j, j+1 ]. (See de Br (1978) fr details). Mrever, B k j (t) > 0 fr all ( j, j+k ) and j Z B k j (t) = 1, that is, the B-spline sequence Bk j cnsists f nnnegative functins which sum up t 1 and prvides a partitin f unity. Thus, a spline functin can be written as linear cmbinatin f B-splines, s(t) = β j B k j (t). j Z The value f the functin s at pint t is simply the value f the functin j Z β j B k j (t) which makes gd sense since the latter sum has at mst k nnzer terms. Figure 4.1 shws an eample f B-splines basis and their cmpact supprt prperty. This prperty makes the cmputatin f B-splines easier and numerically stable. Of special interest is the set f natural splines f rder 2m, m N, with k knts at j. A spline functin is a natural spline f rder 2m with knts at 1,..., k, if, in additin t the prperties implied by definitin (4.1), it satisfies an etra cnditin: s is plynmial f rder m utside f [ 1, k ]. Cnsider the interval [a, b] R and the knt sequence a := 0 < 1 <... < k < k+1 := b. Then, N S 2m = {s S(P 2m ) : s 0 = s [a,1 ) and s k = s [k,b) P m }, is the natural plynmial spline space f rder 2m with knts at 1,..., k. The name natural spline stems frm the fact that, as a result f this etra cnditin, s satisfies the s called natural bundary cnditins s j (a) = s j (b) = 0, j = m,..., 2m 1. Nw, since the dimensin f S(P 2m ) is 2m + k and we have enfrced 2m etra cnditins t define N S 2m, it is natural t epect the dimensin f N S 2m t be k.

35 4.2: Lgspline Density Estimatin 27 B splines Figure 4.1: Basis Functins with 6 knts placed at Actually, it is well knwn that N S 2m is linear space f dimensin k. See details in Schumaker (1981). In sme applicatins it may be pssible t deal with natural splines by using a basis fr S(P 2m ) and enfrcing the end cnditins. Fr ther applicatins it is desirable t have a basis fr N S 2m itself. T cnstruct such a basis cnsisting f splines with small supprts we just need functins based n the usual B-splines. Particularly, when m = 2, we will be cnstructing basis functins fr the Natural Cubic Spline Space, N S 4. Figure 4.2 shw an eample f the natural splines basis. 4.2 Lgspline Density Estimatin Kperberg and Stne (1991) intrduced anther type f algrithm t estimate an univariate density. This algrithm was based n the wrk f Stne (1990) and Stne and K (1985) where the thery f the lgspline family f functins was develped. Cnsider an increasing sequence f knts {t j } K j=1, K 4, in R. Dente by S 0 the set f real functins such that s is a cubic plynmial in each interval f the frm (, t 1 ], [t 1, t 2 ],..., [t K, ). Elements in S 0 are the well-knwn cubic splines with knts at {t j } K j=1. Ntice that S 0 is a (K + 4)-dimensinal linear space. Nw, let S S 0 such that the dimensin f S is K with functins s S linear n (, t 1 ] and n [t K, ). Thus, S has a basis f the frm 1, B 1..., B K 1, such that B 1 is linear functin with negative slpe n (, t 1 ] and B 2,..., B K 1 are cnstant functins n the same interval. Similarly, B K 1 is linear functin with psitive slpe n [t K, ) and B 1,..., B K 2 are cnstant n the interval [t K, ). Let Θ be the parametric space f dimensin p = K 1, such that θ 1 < 0 and θ p > 0

36 28 Chapter 4: Spline Functins Natural Splines Figure 4.2: Basis Functins with 6 knts placed at fr θ = (θ 1,..., θ p ) R p. Then, define and c(θ) = lg( R K 1 f (; θ) = ep{ K 1 ep( j=1 j=1 θ j B j ()d)) θ j B j () c(θ)}. The p-parametric epnential family f (, θ), θ Θ R p f psitive twice differentiable density functin n R is called lgspline family and the crrespnding lglikelihd functin is given by L(θ) = lg f (; θ); θ Θ. The lg-likelihd functin L(θ) is strictly cncave and hence the maimum likelihd estimatr ˆθ f θ is unique, if it eists. We refer t ˆf = f (, ˆθ) as the lgspline density estimate. Nte that the estimatin f ˆθ makes lgspline prcedure nt essentially nnparametric. Thus, estimatin f θ by Newtn-Raphsn, tgether with small numbers f basis functin necessary t estimate a density, make the lgspline algrithm etremely fast when it is cmpared with Gu (1993) algrithm fr smthing spline density estimatin. In the Lgspline apprach the number f knts is the smthing parameter. That is, t many knts lead t a nisy estimate while t few knts give a very smth curve. Based n their eperience f fitting lgspline mdels, Kperberg and Stne prvide a table with the number f knts based n the number f bservatins. N indicatin was fund that the number f knts takes in cnsideratin the structure f the data (number f mdes, bumps, asymmetry, etc.). Hwever, an bjective criterin