Richard W. Andrews and William C. Birdsall, University of Michigan Richard W. Andrews, Michigan Business School, Ann Arbor, MI

Transcription

1 SIMULTANEOUS CONFIDENCE INTERVALS: A COMPARISON UNDER COMPLEX SAMPLING Rchard W. Andrews and Wllam C. Brdsall, Unversty of Mchgan Rchard W. Andrews, Mchgan Busness School, Ann Arbor, MI EY WORDS: Bayesan, smulaton, earnngs 1. Introducton Procedures have been developed to valdate the output of mcroeconomc smulaton models by comparng such output to survey data. The propertes of the valdaton procedures are not well understood and are not easy to determne analytcally, partcularly when the survey data has been sampled wth a complex desgn. Ths paper reports on a smulaton experment whch compares the propertes of three smultaneous confdence nterval procedures. These procedures are compared wthn the context of smulaton valdaton. Consder two fnte populatons, desgnated I and II. Defned on each populaton there s a sngle categorcal varable whch has one of values. Let Pk be the proporton of populaton I unts n category k, (k : 1,..., ). Lkewse, defne qk on populaton II. The purpose of ths paper s to report on a smulaton study of three methods for developng smultaneous confdence ntervals for (pk -qk), (k = 1... ). The samplng desgn for populaton I s smple random samplng (SRS), and for populaton II a stratfed two stage ($2S) desgn s used. The choce of these two desgns s consstent wth the motvatng applcaton whch s dscussed n secton 3. Three smultaneous confdence nterval (SCI) procedures are descrbed n the next secton and referred to as: (1) ordnary-x 2, (2) full desgn, and (3) Bayesan. The smulaton experment as descrbed n secton 3, uses a survey sample of ndvdual earnngs to generate the fnte populatons. The purpose of the applcaton s to compare the generated output from a mcroeconomc smulaton model wth survey sample data. The smulaton results ndcate that, when pk -- qk, (k - 1,..., ), the full desgn method yelds acceptable ntervals. However, when pk ~ qk, the full desgn SCIs frequently cause one to falsely conclude that ~he two populatons have the same values for p and q. The ordnary-x 2 and Bayesan SCIs do not falsely conelude equvalence of the two populatons as frequently. In addton, the nterval lengths of the ordnary-x 2 and Bayesan methods show better propertes than the full desgn. There s lttle evdence of dfference between Bayesan and ordnary-x 2 methods; however, where dfferences do occur the Bayesan procedure ponts toward the correct concluson more often and ts length s smaller wth less varaton. The full desgn methodology has been recommended [1] as the procedure to use when full desgn nformaton s avalable. Because of the hgh frequency wth whch the full desgn method falsely ndcates that two populatons have no dfference, t s not recommended. 2. Confdence Interval Proeedures The three CIPs consdered are (1) ordnary-x 2, (2) full de- sgn, and (3) a Bayesan posteror nterval. The ordnary-x 2 s derved under the assumpton of smple random samplng. The full desgn method takes nto account that populaton II s sampled usng a $2S desgn. The Bayesan method gnores the desgn and states the posteror dstrbuton of {Pk -qk}~=l, gven the data. The presentaton of these ntervals follows. Ordnary X 2 From populaton I, a SRS of sze n s selected. Defne/~k -- pro- porton of populaton I observatons from category k; ~k=1 pk = I. From populaton If, a $2S sample of total sze n s selected. In lke manner, defne qk - proporton of populaton II observatons from category k; ~-]ksl qk -- I. Usng Scheffe type smultaneous conf- dence ntervals, the 100(I - a)% smultaneous confdence ntervals for (Pk - qk); k = 1,..., are: where, (Pk -qk) ::k Sko\/~x~( - 1); S~o= /~j:(1--/~k)+4k(1--qk); x~( - 1) = the 100(1 -a) dstrbuton wth - 1 degrees of freedom. Full Desgn Populaton II s stratfed nto H strata. (k = 1,...,g). (k = l,...,g). percentle of the ch-squared Stratum h has Mh (h = 1,..., H) clusters; and the jth cluster from the h th stra- tum has Lhj ndvduals. where, The full desgn Scheffe type SCIs for (pk -qk) are: (Pk--qk)±SklX/x~(--1); (k=l,...,).,_,q~l = /~k(1 --/~k) + l)q(kk); (k - 1,...,). T/, f~(kk) = ~ 1-s~4~ ~ + A(1 - f2 )~,~,,~ m ml h=l m 1 The between cluster varance s gven by: 82kh I = (~- 1) 240

2 The wthn cluster varance s gven by: $2h2 --- m l - )2 s(~ I~1 = r(z _a ~+v ~- 1 (~, + ~,111] (r(~, ~" + ~,11" =1 =1 The ndcator varable, Ikuj s one f the jth observaton from the th cluster of the h th stratum belongs to the k th category; and zero otherwse. The estmates of the proportons are as follows: Defne, Z~ q = (pl - ql,p2 - q2,...,p-1 -- q-1), the dfference n the frst - 1 components of 6 and ~. As an approxmaton to the posteror dstrbuton of/~ gven ~ and ~, we let ~[f, f be multvarate (- 1) normal. The mean vector s j=l m l =1 j=l = E[~ I~] - El4 I~]; where the expectaton s taken on the frst - 1 components of/3 and ~. Assumng the same pror for 15 and ~, Bayesan The Bayesan approach uses a multnomal dstrbuton wth a conjugate Drchlet pror. For the sample of sze n from populaton I defne f = (ul,u2,...,ug); such that uk = the number of observatons n the k th category (k = 1,2,...,). Lkewse, for the sample of sze n from populaton II, defne ~ = (vl,v~,..., V). Let 16 = (Pl,P2,...,P) and ~ = (q~,q2... q/c). The condtonal dstrbuton of gven 15 s assumed multnomal wth parameters (15, n); that s, f [ MN (15, n). So, a+n ~--I--~_1 The varance--covarance of/~ s v - v(~ l e) + v(q I ~). g l~ f(~ ] 16) = n! H (~) =l "" E u = n, =l E =1 P = 1. for whch, The dagonal terms are v~ = (~ + =~ )(~ + = - ~ - ~) + (~ + "~ )(~ + ~ - ~ - ~ ). (~ + ~)~(~ + ~ + 1) for k = 1,2,..., - 1. Also assume, f]q ~ MN(q,n). The conjugate pror dstrbuton on 15 s Drchlet wth param- eters c~; 15 ~ I)(5). So, The off dagonal terms are y~, = (~k + ~)(~, + ~) + (~k + ~)(~, ~) (~ +,~)~(~ +,~ + 1) f (/5) = l"(a) H ( ~ =1 E P = 1, e~.-1 P ) for whch, for k = 1,2,..., - 1;/=/- k. From Berger[2; p.143] the 100(1 -a)% credble set for A wll be { " (/~ -/2)V-I(Z~ -/2) < x~( - 1)}. For each component of A, Ak = Pk --qk, the SCI s gven by the projecton whch s the nterval Throughout ths analyss we assume D() and ndependently, ~ D(1); so, al = a2... a = 1. Ths pror can be thought of as a unform dstrbuton over the smplex of the - 1 dmenson space spanned by the vectors/5 and q; under the constrant that E:I P-- 1 and E=I q-- 1. By conjugate pror dstrbuton theory we have that the posteror dstrbuton of 5 gven f s Drchlet wth parameter & + ~; where & + f = ( rl + Ul,a2 + u2,...,ag + Ug). 8% a,q-u "-- 1) ~, / V x~( Akk ; where Akk s the k ta dagonal element of the nverse of V. 3. Smulaton Experment The purpose of the experment was to nvestgate the statstcal propertes of smultaneous confdence nterval procedures appled to fnte populatons. The motvatng example for ths experment s the output valdaton of a mcroeconomc smulaton model 241

3 (MSM). Specfcaly, we want to compare the earnngs dstrbuton from the MASSII [3] MSM wth the earnngs dstrbuton from the Panel Study of Income Dynamcs (PSID) [4]. We consder the output from the MSM as a smple random sample. The desgn Of the PSID s stratfed two stage. To make the conclusons from the smulaton experment as applcable as possble we used the 1980 results of the PSID to generate the fnte expermental populaton from whch the samples were drawn. The purpose of the experment s to nvestgate the statstcal behavor of SCI procedures on the.dfference of proportons, when one populaton s sampled usng a SRS and the other populaton s sampled usng a $2S desgn. Usng the 1980 PSID as a bass, a fnte populaton wth a stratfed-cluster structure s constructed. The PSID s a stratfed cluster sample wth 32 strata wth 2 clusters per stratum. The PSID s a combnaton of two samples. One s an equal probablty sample (EPSEM) and the other s a sample whch over sampled ndvduals at the lower end of the earnngs scale. As our bass we used only the EPSEM sample. Furthermore, we used only the subset of ths EPSEM sample whch were prme age (35-50) whte males, wth reported earnngs n the nterval between zero (we excluded those wth no earnngs) and $99,999. Any ndvdual wth earnngs above $99,999. was recorded as havng earnngs of $99,999; therefore, ndvduals wth that response were not ncluded n our bass sample. Ths resulted n 2089 ndvduals. On each ndvdual we had the recorded earnngs, and the strata and cluster desgnaton. Let, Yhj =the earnngs of the jth ndvdual from the th cluster of the h th stratum. To generate the ndvdual earnngs, we used the components of varance model, whch s now descrbed. Snce the varable of nterest s earnngs, whch has postve skewness we used the transformaton Xno - ln(yhj ). The components of varance model on the transformed varable s Xhj -- ]~h 4- O~h Q- ehj h= 1,2,...,32 =1,2 j = 1, 2,..., Jh. The assumptons on the components are: for any h, ahl,ah2 are ndependent ndentcally dstrbuted (ld) normal wth a mean of zero and a varance desgnated by cr2(h); and ehl,eh2,...,ehj~.~ are ld normal wth varance desgnated era2. The as and es are mutually ndependent. We used the standard classcal methods from Graybll [5, Sec. 16.5] ~o estmate the parameters. Only 8 of the 32 strata resulted n postve estmates of the varance of the duster effect. Therefore, n generatng the fnte populaton we used 8 strata. The parameter values for the generaton are the estmates n the 8 strata. Ths gves the fnte populaton on whch the smulaton was bult a realstc bass. For each stratum, we generated 50 clusters, and wthn each cluster we generated 100 ndvdual earnngs. The smulaton experment was executed usng a FORTRAN program. The steps of the experment are as follows: 1. Usng the components of varance model wth the estmated parameters, Populaton I earnngs were generated. Therewere 8 strata, 50 clusters per strata, and 100 ndvduals per duster, resultng n a total populaton sze of 40, For each ndvdual earnngs n Populaton I, a correspondng earnngs was generated n Populaton II by multplyng the Populaton I earnngs by 6. The values of 6 nvestgated were 6 = 1.00, 1.05, 1.10, 1.20, 1.50, and These values are seen across the top of Table 1, whch gve the smulaton results. 3. The ndvduals were classfed nto one of three categores k = 1,2,3 =. Category 1 = Earnngs less than $10,000. Category 2 = Earnngs between $10,000. and $20,000. Category 3 = Earnngs greater than $20, From Populaton I a SRS of sze 400 was selected and the earnngs categores were noted. 5. From Populaton II a $2S sample was selected. In each stratum, 5 clusters were sampled; and wthn each cluster, 10 ndvduals were selected. Agan the earnngs categores were noted. 6. Usng the three methods descrbed n secton 2, SCIs were calculated for (p~ -qk); k = 1, 2, 3; as defned n secton For each value of 6, steps 4, 5, and 6 were replcated 500 tmes. 8. Measures of effectveness (MOE) of the SCIs were calculated and are reported n Table Smulaton Results The smulaton results are reported n Table 1 by gvng 5 measures of effectveness. Also, for the 3 categores of earnngs, the fnte populaton proportons are at the bottom of the table (pl,p2,p3 for populaton I; ql,q2, q3 for populaton II). The MOEs follow the gudelnes gven n Schrber and Andrews[6]. The MOE gven n the frst row measures the coverage functon ntroduced and dscussed by Schruben[7]. Defne the random varable r/~ as follows: r 1" = nf{rl: Vk, (Pk --qk) E C(r/)}; where, C(r/) s an 100r/% confdence nterval employng the procedure under nvestgaton. Hence, ~ s the confdence level that just succeeds n smultaneously coverng all components of 5 -~. 242

4 It s shown n Schruben[7] that f C(~?) s an approprate confdence nterval procedure then the random varable ~* wll be unformly dstrbuted on the [0, 1] nterval. Therefore, the value of,7* was determned for each of the SCI procedures on every replcaton. A ch-squared statstc of goodness-of-ft, usng 10 equal szed ntervals, was calculated to see f the ~*s followed a unform dstrbuton. These ch--squared values are reported n the frst row of Table 1. The small ch-squared values ndcate a good ft. The 95 th percentle of the ch-squared wth the approprate 9 degrees of freedom s Therefore, at ths testng level, the ordnary-x 2 and Bayesan procedures behave approprately for all values of ~. However, for all values of ~ the full desgn methodology fals to meet ths crtera. The next four rows of Table 3 pertan to the smultaneous ntervals calculated wth 95% confdence. The second row gves the percent coverage of the actual fnte populaton dfference. Ths should be approxmately.95 for the 500 replcatons. For all three procedures, at all ~ values, the coverage s close to.95. However, the coverage usng the full desgn methodology s consstently larger than ether of the other two methodologes. Ths ndcates that the full desgn ntervals are larger than requred. The thrd row gves the coverage of the pont pk -qk = 0; /c = 1, 2, 3. Coverage of ths pont could lead one to the concluson that the proportons n the two populatons are the same. Ths s only true when ~ = 1.0. For all three methods ths coverage remans hgh untl 6 = Ths coverage s essentally zero for all three methods when 6 = 1.50 or However, for ~s smaller than 1.50 the full desgn method consstently had a false coverage of zero at a larger percent than the other two methodologes. The last two MOEs deal wth the wdth of the ntervals. We defne I/V" to be the wdth of the wdest nterval among the smultaneous ntervals beng calculated for the three components. E[W] s the average over the replcatons of these wdest ntervals; and SD[W]/E[W] s the correspondng coeffcent of varaton. Ideally a confdence nterval procedure needs to have the correct coverage wth a small average wdth and furthermore, the wdth should have a small varaton over the replcatons. Of the three confdence ntervals nvestgated, the Bayesan procedure had the best propertes accordng to these deal crtera. Its value of E[W] was always the smallest, even though t was only slghtly smaller than the ordnary-x ~ method. The full desgn average wdths were always the largest whch explans some of the coverage problems. In addton the coeffcent of varaton of the full desgn wdths were larger than the other two by a factor of 10. Ths ndcates that the sze of the ntervals reported by the full desgn wll vary consderably and n some nstances gve a false sense of precson by reportng very small ntervals. Ths smulaton study casts some serous doubts on the recommended full desgn methodology. It s especally troublesome on two accounts. Frst, there s a large probablty that the method wll lead to falsely concludng that two populatons have the same proportons. Secondly, the wdths of the full desgn ntervals have excessve varance. 5. Acknowledgment Ths research was supported by the Department of Health and Human Servces, Socal Securty Admnstraton. under grant No. 10-P REFERENCES [1 Holt, D., Scott, A. J. and Ewngs, P. D., (1980). Ch-squared Tests wth Survey Data. Journal of the Royal Statstcal Socety, A 143, Part 3, [2] Berger, J. O., (1985). Statstcal Decson Theory and Bayesan A~alyss. Sprnger-Verlag, New York. [3 Unted States Department of Health and Human Servces (1985). Report on Earnngs Sharng Informaton Study, SSA Publ. No ICN443150, Washngton, D.C. [4] Insttute of Socal Research (1984). User Gude to the Panel Study of Income Dynamcs. Survey Research Center, Unversty of Mchgan, Ann Arbor. [5] Graybll, F. A. (1961) An Introducton to Lnear Statstcal Models, Vol., McGraw-Hll, New York. [6] Schrber, T. J. and Andrews, R. W. (1981) A Conceptual Framework for Research n the Analyss of Smulaton Output. Communcatons of the ACM, Vol. 24, No.4, pp [7] Schruben, L. W. (1980) Coverage functon of nterval estmators of smulaton response. Management Scence, 26, 1, pp

5 TABLE I Smulaton Results (SRS vs. Two Stage) (500 replcatons, Sample Sze = 400) (Seeds = 4543, 8771, 811) Pror = (I,I,I) =I.0 t~ =I.0 =.20 =~.5o =2.0 Ord FD Bayes Ord! FD Bayes Ord ~ FD!Bayes Ord FD Bayes Ord ~ FD Bayes Ord FD ~Bayes Ch-squared Test of Coverage of Dff. % Coverage of Populaton Dfference % Coverage of Zero ~. [w*] S~D [W* ] E[W*] ~o.~4aa.4a ~.~ "! "!, ", o.95ao.~aoo.~4~ !. o.a~a " ! o69o oo55 ~.~a a~.ao ~.~o " o.3oo!o.42oo.:~sa! I.! oo49 o664 o049! t ~16.52 J ~ o.95oo.972o.94o : o.oooo.oooo.ooo J ! : 3.92 ~19.16! 1.92 o.9s21o.96o!o.932,. o.oooo.oooo.ooo.1608!.1759".1602 " :!.0134 o137.o645. P1 P2P ,0.339, ,0.339,0.404 Q1 Q2 Q , , ,0.318, ,0.297, ,0.235, ,0.170,