A short note on quantile and expectile estimation in unequal probability samples

Transcription

1 Cataogue o X ISS Survey Methodoogy A short ote o quatie ad expectie estimatio i uequa probabiity sampes by Lida Schuze Watrup ad Göra Kauerma eease date: Jue 22, 206

2 How to obtai more iformatio For iformatio about this product or the wide rage of services ad data avaiabe from Statistics Caada, visit our website, You ca aso cotact us by emai at teephoe, from Moday to Friday, 8:30 a.m. to 4:30 p.m., at the foowig to-free umbers: Statistica Iformatio Service atioa teecommuicatios device for the hearig impaired Fax ie Depository Services Program Iquiries ie Fax ie Stadards of service to the pubic Statistics Caada is committed to servig its ciets i a prompt, reiabe ad courteous maer. To this ed, Statistics Caada has deveoped stadards of service that its empoyees observe. To obtai a copy of these service stadards, pease cotact Statistics Caada to-free at The service stadards are aso pubished o uder Cotact us > Stadards of service to the pubic. ote of appreciatio Caada owes the success of its statistica system to a og stadig partership betwee Statistics Caada, the citizes of Caada, its busiesses, govermets ad other istitutios. Accurate ad timey statistica iformatio coud ot be produced without their cotiued co operatio ad goodwi. Stadard tabe symbos The foowig symbos are used i Statistics Caada pubicatios:. ot avaiabe for ay referece period.. ot avaiabe for a specific referece period... ot appicabe 0 true zero or a vaue rouded to zero 0 s vaue rouded to 0 (zero) where there is a meaigfu distictio betwee true zero ad the vaue that was rouded p preimiary r revised x suppressed to meet the cofidetiaity requiremets of the Statistics Act E use with cautio F too ureiabe to be pubished * sigificaty differet from referece category (p < 0.05) Pubished by authority of the Miister resposibe for Statistics Caada Miister of Idustry, 206 A rights reserved. Use of this pubicatio is govered by the Statistics Caada Ope Licece Agreemet. A HTML versio is aso avaiabe. Cette pubicatio est aussi dispoibe e fraçais.

3 Survey Methodoogy, Jue Vo. 42, o., pp Statistics Caada, Cataogue o X A short ote o quatie ad expectie estimatio i uequa probabiity sampes Lida Schuze Watrup ad Göra Kauerma Abstract The estimatio of quaties is a importat topic ot oy i the regressio framework, but aso i sampig theory. A atura aterative or additio to quaties are expecties. Expecties as a geeraizatio of the mea have become popuar durig the ast years as they ot oy give a more detaied picture of the data tha the ordiary mea, but aso ca serve as a basis to cacuate quaties by usig their cose reatioship. We show, how to estimate expecties uder sampig with uequa probabiities ad how expecties ca be used to estimate the distributio fuctio. The resutig fitted distributio fuctio estimator ca be iverted eadig to quatie estimates. We ru a simuatio study to ivestigate ad compare the efficiecy of the expectie based estimator. Key Words: Quaties; Expecties; Probabiity proportioa to size; Desig-based; Auxiiary variabe; Distributio fuctio. Itroductio Quatie estimatio ad quatie regressio have see a umber of ew deveopmets i recet years with Koeker (2005) as a cetra referece. The pricipe idea is thereby to estimate a iverted cumuative distributio fuctio, geeray caed the quatie fuctio Q = F for 0,, where the 0.5 quatie Q 0.5, the media, pays a cetra roe. For survey data tracig from a uequa probabiity sampe with kow probabiities of icusio Kuk (988) shows how to estimate quaties takig the icusio probabiities ito accout. The cetra idea is to estimate a distributio fuctio of the variabe of iterest ad ivert this i a secod step to obtai the quatie fuctio. Chambers ad Dusta (986) propose a mode-based estimator of the distributio fuctio. ao, Kovar ad Mate (990) propose a desig-based estimator of the cumuative distributio fuctio usig auxiiary iformatio. Bayesia approaches i this directio have recety bee proposed i Che, Eiott, ad Litte (200) ad Che, Eiott, ad Litte (202). Quatie estimatio resuts from miimizig a L oss fuctio as demostrated i Koeker (2005). If the L oss is repaced by the L 2 oss fuctio oe obtais so caed expecties as itroduced i Aiger, Amemiya ad Poirier (976) or ewey ad Powe (987). For 0,, this eads to the expectie fuctio M which, ike the quatie fuctio Q, uiquey defies the cumuative distributio fuctio F ( y ). Expecties are reativey easy to estimate ad they have recety gaied some iterest, see e.g., Schabe ad Eiers (2009), Pratesi, aai, ad Savati (2009), Sobotka ad Keib (202) ad Guo ad Härde (203). However sice expecties ack a simpe iterpretatio their acceptace ad usage i statistics is ess deveoped tha quaties, see Keib (203). Quaties ad expecties are coected i that a uique ad ivertibe trasformatio fuctio hy : 0, 0, exists so that M h = Q, see Yao ad Tog (996) ad De ossi ad Harvey (2009). This coectio ca be used to estimate quaties. Lida Schuze Watrup, Busiess Admiistratio ad Socia Scieces, Ludwig Maximiia Uiversity of Muich, Ludwigstraße 33, Muich, Germay. E-mai: [email protected]; Göra Kauerma, Busiess Admiistratio ad Socia Scieces, Ludwig Maximiia Uiversity of Muich, Ludwigstraße 33, Muich, Germay. E-mai: [email protected].

4 80 Schuze Watrup ad Kauerma: A short ote o quatie ad expectie estimatio i uequa probabiity sampes from a set of fitted expecties. The idea has bee used i Schuze Watrup, Sobotka, Keib ad Kauerma (204) ad the authors show empiricay that the resutig quaties ca be more efficiet tha empirica quaties, eve if a smoothig step is appied to the atter (see Joes 992). A ituitive expaatio for this is that expecties accout for a the data whie quaties based o the empirica distributio fuctio oy take the eft (or the right) had side of the data ito accout. That is, the media is defied by the 50% eft (or 50% right) part of the data whie the mea (as 50% expectie) is a fuctio of a data poits. I this ote we exted these fidigs ad demostrate how expecties ca be estimated for uequa probabiity sampes ad how to obtai a fitted distributio fuctio from fitted expecties. The paper is orgaized as foows. I Sectio 2 we give the ecessary otatio ad discuss quatie regressio i uequa probabiity sampig. This is exteded i Sectio 3 towards expectie estimatio. Sectio 4 utiizes the coectio betwee expecties ad quaties ad demostrates how to derive quaties from fitted expecties. Sectio 5 demostrates i simuatios the efficiecy gai i quaties derived from expecties ad a discussio cocudes the paper i Sectio 6. 2 Quatie estimatio We cosider a fiite popuatio with eemets ad a cotiuous survey variabe Y. We are iterested i quaties of the cumuative distributio fuctio F y = Yi y ad defie as Q =ifargmi w Yi q Yi q (2.) q i = the Quatie fuctio of Y (see Koeker 2005), where for > 0 w = for 0. The if argumet i (2.) is required i fiite popuatios sice the arg mi is ot uique. We draw a sampe from the popuatio with kow icusio probabiities i, i =,,. Deotig by y,, y the resutig sampe, we estimate the quatie fuctio by repacig (2.) through its weighted sampe versio with w w y q i = Qˆ =ifargmi w, y q q = (2.2), = as defied above. It is easy to see that the sum i (2.2) is a desig-ubiased estimate for the sum i Q give i (2.). oetheess, because we take the arg mi it foows that Qˆ is ot ubiased for Q. We therefore ook at cosistecy statemets for Qˆ as foows. Let i q = w yi q yi q ad q := i q. i Statistics Caada, Cataogue o X

5 Survey Methodoogy, Jue We draw a sampe from q, i =,, ad assume we appy a cosistet sampig scheme i that i r q := r q is desig-cosistet for q, where r q deotes the sampe of i q. ote that r q ad hece r q, i q ad q aso deped o which is suppressed i the otatio for readabiity. Let q 0 be the miimum of q which is ot ecessariy uique due to the fiite structure of the popuatio. We ca take the if argumet, i.e., q0 =ifargmi q, but for simpicity we assume a superpopuatio mode (see Isaki ad Fuer 982) by cosiderig the fiite popuatio to be a sampe from a ifiite superpopuatio. I the atter we assume that survey variabe Y has a cotiuous cumuative distributio fuctio so q 0 resuts i a uique quatie. We get for >0 Pr q0 < r q0 P r q0 r q0 < 0. = ote that the argumet i the probabiity statemet is a desig-cosistet estimate for q0 q0, which is ess tha zero sice q 0 is the miimum of. Hece, the probabiity teds to oe i the sese of desig cosistecy defied i Isaki ad Fuer (982). The same hods of course for <0. With this statemet we may cocude that the estimated miimum ˆ =argmi q0 = r q is a desig-cosistet estimate for q 0 so that Qˆ i (2.2) is i tur desig-cosistet for Q. It is easiy show that Qˆ is the iverse of the ormed weighted cumuative distributio fuctio = Fˆ y := = y = y usig the same otatio as i Kuk (988). ote that Fˆ y is the Haek (97) estimate of the cumuative distributio fuctio (see aso ao ad Wu 2009) ad as such ot a Horvitz-Thompso estimate. As a cosequece Qˆ is ot desig-ubiased. oetheess, Fˆ y is a vaid distributio fuctio, ad hece it ca be cosidered as ormaized versio of the Lahiri or Horvitz-Thompso estimator of the distributio fuctio (see Lahiri 95) which is deoted by Fˆ y := y y. L = Kuk (988) proposes to repace FˆL with aterative estimates of the distributio fuctio: Istead of estimatig the distributio fuctio itsef he suggests to estimate the compemetary proportio Sˆ y which the eads to the estimate y defied through Fˆ Fˆ y = S y = y > y. ˆ = Statistics Caada, Cataogue o X

6 82 Schuze Watrup ad Kauerma: A short ote o quatie ad expectie estimatio i uequa probabiity sampes esutig directy from these defiitios we ca express F i terms of F through ˆ ˆ F = F ad F = F. (2.3) = ˆ ˆ ˆ ˆ L L = Kuk (988) shows that, uder sampig with uequa probabiities, estimatio of the media derived from F ˆ outperforms media estimates derived from F ˆ ad F ˆL i terms of mea squared estimatio error. ote that the estimators F ˆ, F ˆL ad F ˆ coicide i the case of simpe radom sampig without repacemet where = =. 3 Expectie estimatio A aterative to quaties are expecties. The expectie fuctio the L oss i (2.) by the L 2 oss eadig to M is thereby defied by repacig =argmi 2 M w Yi m Yi m. (3.) m i = ote that M is cotiuous i eve for fiite popuatios. Moreover M 0.5 equas the mea vaue Y = Y. i = i Usig the sampe y,, y with icusio probabiities,, we ca estimate M by repacig the sum i (2.2) by its sampe versio, i.e., Mˆ w y m m = =argmi 2, with w, as defied above. It is easy to see that the sum i ˆM is a desig-ubiased estimate for the sum i M. The estimate itsef is however ot desig-ubiased ike for the quatie fuctio above. However the same argumets as for i (2.2) may be used to estabish desig-cosistecy. 4 From expecties to the distributio fuctio Q Both, the quatie fuctio Q ad the expectie fuctio M uiquey defie a distributio fuctio F.. Whie Q is ust the iversio of F. the reatio betwee M ad F. is more compicated. Foowig Schabe ad Eiers (2009) ad Yao ad Tog (996), we have the reatio M GM M 0.5 GM =, F M F M (4.) where Gm is the momet fuctio defied through Gm = Y. i = i Yi m Expressio (4.) gives the uique reatio of fuctio M to the distributio fuctio F.. The idea is ow to sove (4.) for F., that is to express the distributio F. i terms of the expectie fuctio M.. Apparety, Statistics Caada, Cataogue o X

7 Survey Methodoogy, Jue this is ot possibe i aaytic form but it may be cacuated umericay. To do so, we evauate the fitted fuctio ˆM at a dese set of vaues 0< < 2 < L < ad deote the fitted vaues as mˆ ˆ = M. We aso defie eft ad right bouds through mˆo = mˆ c0 ad mˆ L = mˆl cl, where c 0 ad c L are some costats to be defied by the user. For istace, oe may set c 0 = mˆ 2 mˆ ad cl = mˆ L mˆl. By doig so we derive fitted vaues for the cumuative distributio fuctio F. at m ˆ which we write as Fˆ := ˆ ˆ = ˆ F m = for o-egative steps ˆ L 0, =,, L with ˆ. = L We defie ˆ ˆ L = = to make F ˆ. a distributio fuctio. Assumig a uiform distributio betwee the dese supportig poits m ˆ we may express the momet fuctio G. by simpe stepwise itegratio as m = Gˆ := Gˆ mˆ = xdfˆ x = dˆˆ, where dˆ = mˆ mˆ 2 y With the steps ˆ, =,,. = = with the costrait that Gˆ ˆ L = M 0.5 ad M ˆ 0.5 = L we ca ow re-express (4.) as ˆˆ ˆ ˆˆ d M 0.5 d = = mˆ =, =,, L, ˆ ˆ = = which is the be soved for ˆ ˆ,, L. This is a umerica exercise which is coceptuay reativey straightforward. Detais ca be foud i Schuze Watrup et a. (204). Oce we have cacuated ˆ ˆ,, L we have a estimate for the cumuative distributio fuctio which is deoted as F ˆ M y = ˆ. m : ˆ < y We may aso ivert ˆ M F. which eads to a fitted quatie fuctio which we deote with ˆ M Q. As Kuk (988) shows, both theoreticay ad empiricay, F ˆ. is more efficiet tha F ˆ.. We make use of this reatioship ad appy it to ˆ M F., which yieds the estimator Fˆ :=. M = ˆ M F = I the ext sectio we compare the quaties cacuated from the expectie based estimator F ˆ M with quaties cacuated from F ˆ. ote that either F ˆ M or F ˆ are proper distributio fuctios sice they are ot ormed to take vaues betwee 0 ad. 5 Simuatios We ru a sma simuatio study to show the performace of the expectie based estimates. I the foowig, we make use of the Mizuo sampig method (see Midzuo 952) ad defie the icusio Statistics Caada, Cataogue o X

8 84 Schuze Watrup ad Kauerma: A short ote o quatie ad expectie estimatio i uequa probabiity sampes probabiities proportioa to a measure of size x, see package sampig by Tié ad Matei (205). We examie two data sets aso used i Kuk (988). The first data set (Dweigs) cotais two variabes, the umber of dweig uits X, ad the umber of reted uits Y, which are highy correated (with a correatio of 0.97); see aso Kish (965). The secod data set (Viages) icudes iformatio o the popuatio X ad o the umber of workers i househod idustry Y for 28 viages i Idia; see Murthy (967). I the secod data set the correatio betwee Y ad X is I order to compare our simuatio resuts with the resuts of Kuk (988) we choose the same sampe size of = 30 (from a tota popuatio of = 270 for the Dweigs data ad = 28 for the Viages data). We compare quaties defied by iversio of F ˆ with quaties defied by iversio of ˆ M F. I Tabe 5. we give the root mea squared error (MSE) ad the reative efficiecy for specified quaties. We ote that the media for the viage data ad for the Dweig data aso upper quaties derived from expecties yied icreased efficiecy. Aso the efficiecy gai does ot hod uiformy as we observe a oss of efficiecy for ower quaties. Tabe 5. Compariso of mea squared error o a basis of 500 repicatios quaties MSE Qˆ quaties from expecties ˆ M MSE Q reative efficiecy ˆ M MSE Q MSE Qˆ Dweigs Viages To obtai more isight we ru a simuatio sceario which ivoves a arger sampe size of =00 seected from popuatios of sizes =,000ad = 0,000. We draw Y ad X from a bivariate og stadard orma distributio with = 0 ad =. The variabes Y ad X are draw such that the correatio betwee the variabes is equa to 0.9. We agai cacuate the root mea squared error for a rage of vaues ad show the reative efficiecy of the expectie based approach i Figure 5.. For better visua presetatio we show a smoothed versio of the reative efficiecy. We otice a reductio i the root mea squared error for both cases =,000ad = 0,000. We may cocude that the expecties ca easiy be fitted i uequa probabiity sampig ad the reatio betwee expecties ad the distributio fuctio ca be used umericay to cacuate quaties with icreased efficiecy. This efficiecy gai hods for upper quaties oy, that is for bouded away from zero. ote however that the sampig scheme is such that arge vaues of Y are samped with higher probabiity, refectig that the sampig scheme aims to get more reiabe estimates for the right had side of the distributio fuctio, i.e., for arge quaties. If we are Statistics Caada, Cataogue o X

9 Survey Methodoogy, Jue iterested i sma quaties we shoud use a differet samig scheme by givig idividuas with sma vaues of Y a icreased icusio probabiity. I this case the behavior show i Figure 5. woud be mirrored with respect to. Smooth fit for =,000 Smooth fit for = 0,000 MSE Quaties from / MSE Quaties from MSE Quaties from / MSE Quaties from α Figure 5. eative oot Mea Squared Error (MSE) of quaties ad quaties from expecties for the Probabiity Proportioa to Size (PPS) desig cacuated from 500 repetitios (eft: =,000, right: =0,000. α 6 Discussio I Sectio 4 we exteded the toobox of expecties to the estimatio of distributio fuctios i the framework of uequa probabiity sampig. We defied expecties for uequa probabiity sampes. Whe comparig quaties based o F ˆ with quaties based o the expectie based estimator ˆ M F, we observed that the proposed estimator performs we i compariso to existig methods. The cacuatio of empirica expecties is impemeted i the ope source software (see Core Team 204) ad ca be foud i the -package expectreg by Sobotka, Schabe, ad Schuze Watrup (203). The cacuatio of the expectie based distributio fuctio estimator F ˆ M is aso part of the -package expectreg. The cacuatio of F ˆ M is, however, more demadig as the cacuatio of F ˆ because it ivoves three steps: First, we cacuate the weighted expecties as described i Sectio 3; secod, we estimate ˆ F, ad i a third step, we derive F from F ˆ (see Sectio 4). I the Log-orma-Simuatio it takes about 2-3 secods for =,000to ˆ M cacuate F ˆ M whereas the computatioa effort of F ˆ is barey oticeabe. Ackowedgemets Both authors ackowedge fiacia support provided by the Deutsche Forschugsgemeischaft DFG (KA 88/7-). Statistics Caada, Cataogue o X

10 86 Schuze Watrup ad Kauerma: A short ote o quatie ad expectie estimatio i uequa probabiity sampes efereces Aiger, D.J., Amemiya, T. ad Poirier, D.J. (976). O the estimatio of productio frotiers: Maximum ikeihood estimatio of the parameters of a discotiuous desity fuctio. Iteratioa Ecoomic eview, 7(2), Chambers,.L., ad Dusta,. (986). Estimatig distributio fuctios from survey data. Biometrika, 73(3), Che, Q., Eiott, M.. ad Litte,.J.A. (200). Bayesia peaized spie mode-based iferece for fiite popuatio proportio i uequa probabiity sampig. Survey Methodoogy, 36,, Che, Q., Eiott, M.. ad Litte,.J.A. (202). Bayesia iferece for fiite popuatio quaties from uequa probabiity sampes. Survey Methodoogy, 38, 2, De ossi, G., ad Harvey, A. (2009). Quaties, expecties ad spies. oparametric ad robust methods i ecoometrics. Joura of Ecoometrics, 52(2), Guo, M., ad Härde, W. (203). Simutaeous cofidece bads for expectie fuctios. AStA - Advaces i Statistica Aaysis, 96(4), Haek, J. (97). Commet o A essay o the ogica foudatios of survey sampig, part oe. The Foudatios of Survey Sampig, 236. Isaki, C.T., ad Fuer, W.A. (982). Survey desig uder the regressio superpopuatio mode. Joura of the America Statistica Associatio, 77, Joes, M. (992). Estimatig desities, quaties, quatie desities ad desity quaties. Aas of the Istitute of Statistica Mathematics, 44(4), Kish, L. (965). Survey Sampig. ew York: Joh Wiey & Sos, Ic. Keib, T. (203). Beyod mea regressio (with discussio ad reoider). Statistica Modeig, 3(4), Koeker,. (2005). Quatie egressio, Ecoometric Society Moographs. Cambridge: Cambridge Uiversity Press. Kuk, A.Y.C. (988). Estimatio of distributio fuctios ad medias uder sampig with uequa probabiities. Biometrika, 75(), Lahiri, D.B. (95). A method of sampe seectio providig ubiased ratio estimates. Bueti of the Iteratioa Statistica Istitute, (33), Midzuo, H. (952). O the sampig system with probabiity proportioa to sum of size. Aas of the Istitute of Statistica Mathematics, 3, Murthy, M.. (967). Sampig Theory ad Methods. Cacutta: Statistica Pubishig Society. ewey, W.K., ad Powe, J.L. (987). Asymmetric east squares estimatio ad testig. Ecoometrica, 55(4), Statistics Caada, Cataogue o X

11 Survey Methodoogy, Jue Pratesi, M., aai, M. ad Savati,. (2009). oparametric M-quatie regressio usig peaised spies. Joura of oparametric Statistics, 2(3), Core Team (204). : A Laguage ad Eviromet for Statistica Computig. Viea, Austria: Foudatio for Statistica Computig. ao, J., ad Wu, C. (2009). Empirica ikeihood methods. Hadbook of Statistics, 29B, ao, J..K., Kovar, J.G. ad Mate, H.J. (990). O estimatig distributio fuctios ad quaties from survey data usig auxiiary iformatio. Biometrika, 77(2), Schabe, S.K., ad Eiers, P.H. (2009). Optima expectie smoothig. Computatioa Statistics & Data Aaysis, 53(2), Schuze Watrup, L., Sobotka, F., Keib, T. ad Kauerma, G. (204). Expectie ad quatie regressio - David ad Goiath? Statistica Modeig, 5, Sobotka, F., ad Keib, T. (202). Geoadditive expectie regressio. Computatioa Statistics & Data Aaysis, 56(4), Sobotka, F., Schabe, S. ad Schuze Watrup, L. (203). Expectreg: Expectie ad Quatie egressio. With cotributios from P. Eiers, T. Keib ad G. Kauerma, package versio Tié, Y., ad Matei, A. (205). Sampig: Survey Sampig. package, versio Yao, Q., ad Tog, H. (996). Asymmetric east squares regressio estimatio: A oparametric approach. Joura of oparametric Statistics, 6(2-3), Statistics Caada, Cataogue o X