Chapter XX More advanced approaches to the analysis of survey data. Gad Nathan Hebrew University Jerusalem, Israel. Abstract

Transcription

1 Household Sample Surveys n Developng and Transton Countres Chapter More advanced approaches to the analyss of survey data Gad Nathan Hebrew Unversty Jerusalem, Israel Abstract In the present chapter, we consder the effects of complex sample desgn used n practce n most sample surveys on the analyss of the survey data The cases n whch the desgn may or may not nfluence analyss are specfed and the basc concepts nvolved are defned Once a model for analyss has been set up, we consder the possble relatonshps between the model and the sample desgn When the desgn may have an effect on the analyss and addtonal explanatory varable related to the desgn cannot be added to the analytcal model, two basc methodologes may be used: classcal analyss, whch could be modfed to take the desgn nto account; or a new analytcal tool, whch could be developed for each desgn Dfferent approaches are llustrated wth real-data applcatons to lnear regresson, lnear models, and categorcal data analyss Key terms: complex sample desgn, analyss of survey data, lnear regresson, lnear models, categorcal data analyss, model-based analyss 49

2 Household Sample Surveys n Developng and Transton Countres A Introducton Sample desgn and data analyss The prmary purpose of the vast majorty of sample surveys, both n developed and n developng countres, s a descrptve one, namely, to provde pont and nterval estmates of descrptve measures of a fnte populaton, such as means, medans, frequency dstrbutons and cross-tabulatons of qualtatve varables Nevertheless, as demonstrated n chapters V-I and as wll be demonstrated n chapter I, there s ncreasng nterest n makng nferences about the relatonshps among the varables nvestgated, as opposed to smply descrbng phenomena In the present chapter, we shall try to assess the effects of commonly used complex sample desgns on the analyss of survey data We shall attempt to dentfy cases where the desgn can nfluence the analyss Usually, the sample desgn has no effect on the analyss when the varables on whch the sample desgn s based are ncluded n the analytcal model Frequently, however, some desgn varables are not ncluded n the model, ether owng to msspecfcaton or to a lack of nterest n those desgn varables as explanatory factors Ths can result n serous bases There are two basc methodologes we shall dscuss for handlng data from a complex sample when addtonal desgn-related varables are not added to the analyss The frst modfes a classcal analytcal tool developed for handlng data from a smple random sample The second develops a new analytcal tool for the specfc complex desgn 4 In what follows, we present some examples of the possble effects of sample desgn on analyss, defne a few basc concepts, and dscuss the role of desgn effects n the analyss of complex sample data Secton B descrbes the two basc approaches to the analyss of complex sample data In sectons C and D, we dscuss examples relatng to the analyss of contnuous and categorcal data, respectvely The fnal secton contans a summary and some conclusons Formal defntons and techncal results are gven n the annex Examples of effects (and of non-effect) of sample desgn on analyss 5 In order to demonstrate the potental effects of sample desgn on the analyss, we consder the followng smple, but llumnatng, example (for detals, see Nathan and Smth, 989) Let Y be the varable of nterest and be an auxlary varable Assume that the lnear regresson model Y = α + β + ε, wth ε ~ N( 0, σ ), holds for the populaton The model nd holds as well for any smple random sample selected from the populaton Sometmes the assumpton of ndependence across the ε s better suted for the smple random sample than the populaton from whch t was drawn In a human populaton, for example, ε values may be correlated for dfferent members of the same household, whle n a smple random sample of ndvduals, wth a very small probablty that more than a sngle ndvdual s selected per household, the correlaton would be neglgble 40

3 Household Sample Surveys n Developng and Transton Countres 6 Under smple random samplng the standard estmate of the regresson coeffcent s unbased The dsplay n fgure plots Y aganst for the total populaton; the dsplay would look the same for a smple random sample In the fve dsplays of fgures -6, samples are selected from the populaton usng methods very dfferent from smple random samplng Consder sample selecton based entrely on the value of, for nstance, by truncatng data ponts wth values beyond (or wthn) fxed lmts as n fgures and It s clear from these fgures that the selecton has no effect on the estmaton of the ntercept (α ) and slope ( β ) parameters of the regresson (although t may effect the varance of the estmators) 7 Now consder sample selecton based on the values of the target varable, Y, for nstance, by truncatng those data ponts wth Y values beyond (or wthn) fxed lmts as n fgures 4-6 Fgure No selecton Fgure Selecton on : L<<U Y Y Fgure Selecton on : <L; >U Fgure 4 Selecton on Y: YL<Y<YU 4 Y Y Fgure 5 Selecton on Y: Y<YL;Y>YU 4 Y Fgure 6 Selecton on Y: Y>YU 4 Y

4 Household Sample Surveys n Developng and Transton Countres In these cases, t s clear that the estmates of the slopes of the regresson become based In the last case (fgure 6), truncaton s not symmetrc, and the estmate of the ntercept s also based These examples are extreme, snce selecton based on the truncaton of the dependent varable s rare n sample surveys It s qute common, however, n expermental or observatonal studes, such as case control studes n epdemology or choce-based studes n economcs [see, for example, Scott and Wld (986) and Mansk and Lerman (977)] Nevertheless, n many cases, samples are selected on the bass of desgn varables that may be closely related to the dependent varable Thus, a common samplng procedure, wdely used n surveys of establshments and of farms, s to select unts wth probablty proportonal to sze The sze measure, say, the prevous year s producton, wll obvously be related to the varable of nterest when that varable s the current year s producton Standard estmates of model parameters, such as regresson coeffcents, can be based when the sample desgn s gnored 8 The examples above llustrate the dangers of carryng out an analyss based on complex sample data as f they came from smple random samplng The examples reveal a need for dentfyng when t s lkely that the desgn affects the analyss and for takng the desgn nto account when t does Basc concepts 9 Most sample surveys are desgned prmarly for descrptve (or enumeratve) purposes They am at estmatng the values of fnte populaton parameters, such as the medan household ncome or the proporton of all adults wth AIDS These are statstcs that, n prncple, can be measured exactly f the whole populaton were ncluded n the survey, that s to say, f a census of the populaton was enumerated rather than a random sample of the populaton The standard theory of survey samplng ensures that data from a random sample can be used to provde unbased estmates of the fnte populaton parameters and of ther samplng errors no matter how complex the sample desgn Ths assumes that the sample desgn s a probablty sample desgn, that s to say, each unt n the populaton has a known postve probablty of beng sampled These classcal methods of estmaton of fnte populaton parameters are known as desgn-based (or randomzaton-based) methods, snce all nference s based on the propertes of the sample desgn, va the sample probablty dstrbuton It should be noted, however, that the effcency of dfferent estmaton strateges (a sample desgn coupled wth an estmaton formula) can usually be evaluated only when extensve nformaton about the populaton s avalable Ths s usually not the case n practce Thus, even classcal samplng texts (for example, Cochran, 977) often rely on models to justfy specfc methods of samplng or estmaton If the populaton values follow a smple regresson model, for example, then the rato estmator wll be more effcent than the smple expanson estmator, under certan assumptons Desgn-based methods are thus often model-asssted, but they are not model-based (or model-dependent): wth model-asssted methods of sample desgn and estmaton of descrptve statstcs, model assumptons are not requred for the nearly unbased estmaton of fnte populaton parameters 0 The model-based (or predcton theory) approach to sample desgn and estmaton assumes that the fnte populaton values are, n fact, realzatons of a super-populaton dstrbuton based on a hypothetcal model wth super-populaton (model) parameters For further detal and dscusson, see Brewer and Mellor (97), Hansen, Madow and Teppng (98), 4

5 Household Sample Surveys n Developng and Transton Countres Särndal, Swensson and Wretman (99) and Vallant, Dorfman and Royall (000) In contrast wth model-asssted estmaton methods, the ncreased effcency attaned by model-based estmaton methods reles on the valdty of the assumed model Thus, f there s any doubt about the valdty of the model assumptons, the apparent reducton n mean square may not justfy the use of purely model-based analyss A good example of ths s demonstrated n Hansen, Madow and Teppng (98), where assumng no ntercept n a regresson model, even when the ntercept s n fact very close to zero, results n nvald model-based nference Increasngly, surveys are beng used for analytcal purposes as well as descrptve ones Often surveys are desgned wth ther analytcal uses n mnd Ths s because decson makers and researchers are nterested n the processes underlyng the raw data, n modellng the relatonshps among the varables nvestgated Such analyses obvously requre assumptons about models The am of an analyss s to confrm the valdty of an assumed model and to estmate the model s parameters rather than the parameters of the fnte populaton Thus analyss s nherently model-based Inference about model parameters must, practcally by defnton, be based on the models of nterest It should be ponted out, however, that when the populaton s very large and the hypotheszed model ndeed holds, there s n practce very lttle dfference between the model parameters and ther fnte populaton counterparts For nstance, f the standard lnear regresson model Y = α + β + ε, wth ε ~ N( 0, σ ) holds, and the populaton sze s very nd large, then the value of the standard populaton regresson coeffcent, B (see annex), wll be very close to the value of the model parameter, β, because of to the Central Lmt Theorem Thus, although we shall focus on the estmaton of model parameters n what follows, these parameters wll sometmes be replaced by ther fnte-populaton counterparts For the sake of smplcty n presentaton, most of the examples n what follows wll be formulated n terms of unvarate dstrbutons (that s to say, a sngle dependent varable and a sngle explanatory varable) The extenson of the results to the multvarate case s usually straghtforward To summarze, hypothetcal models are an ntegral part of statstcal analyss In order to analyse data from sample surveys, the choce of a good model to ft the data s a crtcal part of the analyss Researchers and analysts must have a good understandng of the models underlyng the processes they wsh to study before applyng analytcal methods As we shall see n the followng sectons, the applcaton to data from complex sample desgns requres both an understandng of the underlyng model and of the way n whch the analyss can be affected by the complex desgn 4 Desgn effects and ther role n the analyss of complex sample data 4 The topcs of desgn effects and ther estmaton have been treated extensvely n chapters VI and VII, prmarly n relaton to ther role n the desgn and estmaton for enumeratve surveys We shall see n ths chapter that they also play an mportant role n the analyss of data from complex sample surveys The underlyng dea s based on the fact that, assumng that the model assumptons hold, unbased estmates of the model parameters and estmates of the varances of these estmates are readly avalable under smple random samplng These estmates and varance estmates form the bass for testng hypotheses relatng to the model 4

6 Household Sample Surveys n Developng and Transton Countres parameters For example, under smple random samplng and assumng the smple regresson model Y = α + β + ε, wth ε ~ N( 0, σ ), the ordnary least squares sample estmator, b, nd s an unbased estmator of β, and an unbased estmator of ts varance, v(b), s avalable (see annex) The standard test of the null hypothess that β=0 s then based on the test statstc b / v( b), by nvokng the Central Lmt Theorem When the sample desgn s a complex one, for example, a stratfed cluster sample, the estmator b remans model-unbased, f the regresson model holds and the sample desgn does not depend on the values of Y (for example, n constrast to the stuaton n fgures 4-6) By ths we mean that under the specfed regresson model, the expected value of b s β, where the expectaton s wth respect to the super-populaton dstrbuton of the values of Y As we shall see n secton C, ths may no longer be true f the model does not hold However, even f the model assumptons hold, v(b) s no longer a vald estmator of the model varance of b and must be modfed Often, a drect estmator of the correct model varance can be computed, for example, by usng one of the software packages descrbed n chapter I, and can be used to replace v(b) If a drect estmator s not avalable, often a good estmator of the desgn effect, denoted by d (b), can be obtaned Ths can be used for the modfcaton of the test statstc to replace v(b) by d (b) x v(b) We shall present further specfc uses of the desgn effects to modfy standard test statstcs for other applcatons below B Basc approaches to the analyss of complex sample data Model specfcaton as the bass of analyss 5 Correct specfcaton of the underlyng model s a fundamental step n any analyss The consequences of model ms-specfcaton - both of excluson of relevant explanatory varables (or the ncluson of superfluous ones) and of usng a wrong functonal form (for example, lnear nstead of quadratc) - are well known and documented n standard texts They can take the form of bases n the estmaton of the model parameters (prmarly the excluson of relevant varables), losses of effcency (mostly connected wth erroneous ncluson of explanatory varables) and altered szes and power n tests of hypotheses These effects may be exacerbated when the ms-specfcaton relates drectly to sample desgn varables or to varables correlated wth desgn varables Nevertheless, t s mportant to realze that the survey desgn varables may not be relevant to the research objectves Moreover, there may be no subject-matter justfcaton for ther ncluson n the analytcal model 6 There are two basc approaches to ncorporatng survey desgn varables n the model The aggregated approach consders the model of nterest to be at the populaton level and conceptually ndependent of the sample desgn employed to obtan the data Under ths approach, desgn varables would be ncluded n the model only when they are relevant to the subjectmatter analyss For nstance, say we wsh to explan the bnary varable employed/unemployed by the explanatory varable years of educaton, rrespectve of geographcal locaton The sample s stratfed, say, by geographcal regons, for whch dfferent models may be relevant Ths would be the case even had smple random samplng been used As a result, the stratfcaton can be ncluded n the model (see the dsaggregated approach dscussed below) to reflect regonal varatons n the relatonshps among the model varables If, by contrast, the stratfcaton and 44

7 Household Sample Surveys n Developng and Transton Countres the sample allocaton to strata were carred out smply for operatonal reasons (convenence or cost), the sample weghts would lkely not be relevant to the populaton model The ncorporaton of samplng weghts nto an analyss otherwse free of stratum effects wll lead to some loss of effcency Nevertheless, t s susceptble to the easy nterpretaton of a model free from stratum effects, whle beng robust to model falure f some of the gnored stratum effects really exst 7 The dsaggregated approach extends the analyst s model to nclude not only the survey varables of nterest but also varables used n the survey desgn and those relatng to the structure of the populaton reflected n the desgn Desgn varables relatng to the stratfcaton and clusterng are ncluded n the model to reflect the complex structure of the populaton For nstance, n the prevous example, the model would contan a dfferent set of coeffcents (both an ntercept and a slope) for each geographcal stratum Inference under the dsaggregated approach takes the sample desgn fully nto account, assumng that all desgn varables are correctly ncluded n the model The large number of parameters that need to be estmated under ths approach may cause dffcultes and lead to less accurate estmates when compared wth those of more parsmonous aggregated models The dsaggregated approach s approprate only when the analyst beleves that the hypotheszed model s relevant for hs purposes 8 The approprate approach to be used - the aggregated or dsaggregated approach - wll depend on the analyst s ams The aggregated approach s more sutable for studyng factors affectng the populaton as a whole and, as such, may be more useful for evaluatng natonalpolcy actons The dsaggregated approach s more sutable for studyng mcro-effects and the effects of local and sector-specfc decson-makng For further examples and dscusson, see Sknner, Holt and Smth (989) and Chambers and Sknner (00) Possble relatonshps between the model and sample desgn: nformatve and unnformatve desgns 9 It s mportant to draw a dstncton between nformatve and non-nformatve sample desgns when analysng complex survey data Once a model has been hypotheszed, the analyst must consder whether, after condtonng on the model covarates, the sample selecton probabltes are related to the values of the response varable A samplng process s nformatve f the jont condtonal model dstrbuton of the observatons for the sample, gven the values of the covarates n the model, dffers from ther condtonal dstrbuton n the populaton Only when these dstrbutons are dentcal s the sample desgn non-nformatve (or gnorable), n whch case standard analytcal methods can be employed as f the observatons came from a smple random sample When the sample desgn s nformatve, the model holdng for the sample data s dfferent from the populaton model Ignorng the samplng process n such a case may yeld based pont estmators and dstort the analyss just as when varables are excluded from the model n a conventonal analyss Note that the correct ncluson of desgn-related varables n the model wll ensure that the desgn s non-nformatve 0 There are two major problems wth ncludng all desgn-related varables n a model Frst, exactly whch varables were used n the desgn may not be known or, f known, ther values may not be avalable Even when the desgn varables are dentfed and measured, the 45

8 Household Sample Surveys n Developng and Transton Countres analyst may not know the exact form of the relatonshp (for example, lnear or exponental) between them and the varable of nterest For nstance, f the desgn s a stratfed one, then the possblty that a regresson relatonshp has dfferent slopes and ntercepts for the dfferent strata wll need to be checked Second, when desgn varables are correctly ncluded n the model, resultng estmates may be of lttle value to the analyst, snce the varables added are not of ntrnsc subject-matter nterest (recall the dscusson on aggregated and dsaggregated analyss) Ths mples that the effect of a complex sample desgn on analyss cannot always be dealt wth solely by modfyng the underlyng model In what follows, we shall consder both how to modfy standard analytcal methods to take a complex desgn nto account and how to construct specal desgn-specfc methods of estmaton and analyss Problems n the use of standard software analyss packages for analyss of complex samples The nearly unversal use of standard software for statstcal analyss has led to wdespread abuse of sound statstcal practce Ths abuse s frequently exacerbated when analysng complex sample survey data The advantages of statstcal software n facltatng analyss unfortunately come wth the possbltes for performng analyss wthout any basc understandng of the underlyng prncples nvolved Ths has become a serous problem n quanttatve work, especally n the socal scences Ths problem s compounded by the fact that most commonly avalable software treats data as f they resulted from smple random samplng As ponted out prevously, ths can lead to serously based nference when the desgn s nformatve Nevertheless, wth due care, standardzed software can often be adapted to approxmately capture or account for the effect of a complex desgn In partcular the SURVEYREG procedure n the latest versons of SAS (versons 8 and 9) features regresson analyss that takes the sample desgn nto account n ways smlar to those descrbed below [see An and Watts (00)] 4 For nstance, consder the lnear (heteroscedastc) regresson model defned by: Y = α + β + ε, wth ε ~ N( 0, σ ) Standard computer programmes ordnarly compute nd b, the ordnary least squares (OLS) estmator of β, or the generalzed least squares (GLS) estmator, b G, where sums and products are weghted by the recprocals of σ, whose values (or relatve values) are assumed to be known (see annex) Both of these are unbased estmators for the parameter β, f the model holds, although b G s a more effcent estmator n the heteroscedastc case The standard programmes also provde estmators of the varances of the OLS estmator, v(b), and of the GLS estmator, v(b G ), whch are each model-unbased under the approprate model (the homoscedastc model n the case of v(b)) 5 In many cases, there may be doubt about the valdty of the model, so that nstead of estmatng β, t may be more approprate to estmate the fnte populaton counterpart of β, whch we denote as B (see annex) Although b (the OLS estmator) s a model-unbased estmator for β B, t s not n general desgn-unbased The sample-weghted (Horvtz-Thompson) estmator, 46

9 Household Sample Surveys n Developng and Transton Countres b W, wth cross-products and squares weghted by the recprocals of the ncluson probabltes, s both desgn-consstent and model-unbased, under approprate condtons Furthermore b W can be obtaned from the weghted regresson optons of many standard programmes by usng the values of w as the weghts Alternatvely b W can be obtaned by unweghted regresson of the transformed varables Y π, π wth the ntercept replaced by π It must be emphaszed, however, that under both these alternatves, the estmates of the varance-covarance matrx reported by most standard programmes are ncorrect -- both as estmators for desgn mean squared error and as estmators for model varance -- except n unusual crcumstances 6 In summary, the use of standard software programmes that do not take nto account complex survey desgn should be avoded unless t can be determned that the complex desgn does not have a serous effect on estmaton Ths can often be acheved through the sutable use of standard software See the example n secton C The use of software packages specfcally dealng wth complex sample desgns s recommended (see chap I) C Regresson analyss and lnear models Effect of desgn varables not n the model and weghted regresson estmators 7 Regresson analyss and lnear modellng are very common applcatons where standard models developed for smple random samples are routnely appled to data from complex sample surveys As already ponted out, ths can often lead to erroneous analyses and conclusons A key source of protecton aganst error s the dentfcaton of varables determnng or nfluencng the sample desgn, so that they can be ncluded n the model As we have seen, however, even when these varables are dentfed, ther ncluson n the model may not be warranted from the subject-matter pont of vew In the present subsecton, we wll study the effects on tradtonal estmators of not ncludng desgn varables n the model and nvestgate the possbltes for modfyng these estmators so as to take the complex desgn nto account For ease of exposton, we consder the case of a sngle dependent varable, Y (denoted n ths secton for techncal reasons by ), where the model of nterest has a sngle explanatory varable ( ), and there s a sngle desgn varable ( ) The model of nterest s therefore E ( ) = µ + β ( µ ), where β s the parameter of nterest, rather than the full model, whch ncludes the desgn varable, See the annex for the formulae used n ths subsecton 8 Under farly general condtons specfed n Nathan and Holt (980), the standard OLS estmator for β, b = s s, can be both (model-) based condtonal on and on the sample, S, and based uncondtonally Expressons for the condtonal model expectaton and ts uncondtonal (jont model and desgn) expectaton show that, n general, b s asymptotcally based, unless ρ, the correlaton between and, s zero or f the smple sample varance of s an unbased estmator for ts true varance It can be shown that ths second condton does hold asymptotcally for a large number of equal probablty (epsem) sample desgns, but rarely for unequal probablty desgns (for example, non-proportonal stratfed sample desgns) 47

10 Household Sample Surveys n Developng and Transton Countres 9 A corrected, asymptotcally unbased estmator based on the maxmum lkelhood estmator under normalty, ˆβ, can be used nstead of the OLS estmator Expressons for the varances of b and of β ˆ are gven n Nathan and Holt (980) It should be noted that the usual estmator for the varance of b, v( b ), may not be nearly unbased, even when b s a consstent estmator for β Ths can happen when the ε s are not ndependent and dentcally dstrbuted among the observatons n the sample 0 Nether the estmator b nor βˆ depends on the sample desgn, though ther propertes do Informaton on the sample desgn may be useful for mprovng these estmators, ether when nformaton on values of the desgn varable s not avalable for the whole populaton (so that S cannot be used for estmaton) or when the analysts wshes to ensure robustness to departures from the model Ths can be done by usng sample-weghted estmators based on Horvtz- Thompson estmaton for each of the components of the unweghted estmators One can replace the unweghted sample moments by ther weghted versons n the expressons for b and for β ˆ to obtan the weghed estmators, b and * ˆβ * * Note that b can be used when the populaton varance of, S, s unknown, but that * ˆβ cannot be used n that stuaton It s easly seen that, under farly general condtons, both of these estmators are desgn-consstent estmators of the fnte populaton parameter, B Emprcal comparsons of the performance of these four estmators were made by Nathan and Holt (980) for a populaton of N =,850 farms, about whch data on crop land ( ), total acreage ( ), and total value of produce n prevous year ( ), were avalable Farms were stratfed on the bass of the values resultng n sx strata of szes 56, 584, 854, 998, 696 and 55 The followng sx sample desgns were used to select samples of sze n = 400 (see table ): (A) (B) (C) (D) Smple random samplng; Proportonal stratfed smple random samplng; Fxed sze stratfed smple random samplng; Stratfed smple random samplng wth hgher-than- proportonal allocaton to strata wth hgh values (5, 0, 60, 80, 0, 75); (E) Stratfed smple random samplng wth U-shaped allocaton (00, 80, 0, 0, 80, 00) 48

11 Household Sample Surveys n Developng and Transton Countres Table Bas and Mean square of ordnary least squares estmator and varances of unbased estmators for populaton of,850 farms usng varous survey desgns Survey * * E(b desgn )- β MSE(b ) V( ˆβ ) V( b ) V( ˆβ ) A B C D E Source: Nathan and Holt (980); table The results demonstrate the bas of b for the non-epsem desgns (C,D,E), whereas the other estmators are ether desgn-consstent or model-consstent (or both) They also demonstrate the advantage of βˆ over the weghted estmators for all the desgns consdered Ths holds even though the full model assumptons appear not to hold for the populaton When S s unknown, however, the less effcent, but stll consstent, b *, s a reasonable estmator 4 To summarze, when data are based on unequal probablty desgns, t s worthwhle to consder both weghted and unweghted maxmum lkelhood estmators, rather than the smple OLS estmators The unweghted estmator seems to be more effcent In many applcatons, however, the analyst wll not have the nformaton needed to compute maxmum lkelhood estmators; and less effcent, but consstent, sample-weghted estmators are approprate and, ndeed, are routnely used [see Korn and Graubard (999)] Testng for the effect of the desgn on regresson analyss 5 Many analysts prefer to use smple weghted or unweghted estmators of regresson coeffcents, whch can be obtaned from standard packages, rather than the modfed estmators proposed n secton C We have seen that the smple OLS estmator s consstent when the desgn s non-nformatve or the effect of the desgn s neglgble, but that ts weghted counterpart s preferable when that s not the case DuMouchel and Duncan (98) proposed a smple test, based on standard software packages, for decdng whether weghts should be used when based on data from a non-clustered sample Consder the unvarate case wth a sngle explanatory varable The extenson to the multvarate case s straghtforward Lettng ^ ˆ = b W b, the goal s to test the hypothess: = E = 0 DuMouchel and Duncan showed that the test for = 0 s the same as the test for γ = 0, under the model Y = α + β + γz + ε, where Z = w and ε ~ N( 0, σ ) The authors gave a numercal example for the multvarate nd case nvolvng a subset of data from the Unversty of Mchgan Survey Research Center s Panel Study of Income Dynamcs The sample of 658 ndvduals was selected wth varyng probabltes, resultng n weghts rangng from to 8 The fnal model used to explan educatonal attanment ncluded a constant and 7 explanatory varables, such as parents 49

12 Household Sample Surveys n Developng and Transton Countres educaton, ncome, age, race employment and nteractons The followng analyss of varance (ANOVA) table s obtaned: Table ANOVA table comparng weghted and unweghted regressons Source df Sum of squares Mean square F Sgnfcance Regresson <000 Weghts Error Total Taken together, the 8 varables correspondng to Z (the 7 explanatory varable and the constant, each multpled by w ) have an F value of 97 and a sgnfcance level of only 494 Thus an unweghted regresson s justfed, even though t may ental some loss of power 7 In general, analysts may be equally concerned about acceptng the null hypothess that = E( ˆ ) = 0, when t s false, as about rejectng t when t s true As a result, they may decde to conduct a weghted analyss (wth the approprate software) or develop a less parsmonous model when the sgnfcance level s consderable larger that the standard 05 In the example above, the sgnfcance level s very close to 05, whch suggests that the weghts can be gnored In an earler verson of the model, however, the sgnfcance level for the Z was 056, at whch pont, DuMouchel and Duncan added some nteracton terms The fnal results are those dsplayed n the table 8 The DuMouchel-Duncan test descrbed above assumes that the ε s are ndependent and dentcally dstrbuted Often survey data come from multstage sample desgns When the ε values of observatons from the same sample cluster are correlated or when the observatons have an unknown heteroscedastcty regardless of the desgn, ths test n napproprate Nevertheless, an analyst may feel that usng sample weghts adds unneeded varance to the resultng estmates A Wald test along the lnes proposed by Fuller (984) can be employed In practce ths nvolves usng software lke SAS/SURVEYREG and enterng each data pont twce, once wth the sample weght set to and once wth the sample weght set to the actual weght 9 Pfeffermann and Sverchkov (999) proposed an alternatve set of weghts for use when the lnear model s correct and the errors are ndependent and dentcally dstrbuted but the sample desgn s nformatve The test descrbed above can be used to assess ther weghts relatve to the sample weghts For further dscusson of the role of samplng weghts when modellng survey data, see Pfeffermann (99) and Korn and Graubard (999) Multlevel models under nformatve sample desgn 40 Recently, there has been ncreased use of multlevel models for the analyss of data from populatons wth complex herarchc structures For nstance, n most household surveys, ndvduals nested wthn households are the unts of nvestgaton, and there s nterest n both 40

13 Household Sample Surveys n Developng and Transton Countres relatonshps among the ndvduals and among the households Smlar herarchc structures exst for surveys of pupls wthn schools and employees wthn establshments 4 The usual sngle-level lnear models can easly be extended to take a herarchy nto account by usng mxed (random and fxed effects) models wth an error structure that reflects the herarchc confguraton For example, what s known as the random ntercept model can be formulated (for a sngle explanatory varable) as follows: j = β o + βxj + ε j ; ε j xj ~ N 0, ε ( σ ); ( =, K, N; j =, K M ) y, where y j s the outcome varable for frst-level unt j (say, ndvdual) wthn second-level unt (say, household), x j s a known explanatory varable and β an unknown parameter The ntercept, β, s here a random varable, whch s further modelled as o ( ); ( =, K N ) β o = α + γz + u ; u z ~ N 0, σ u, where z s a known second-level unt explanatory varable and α and γ are unknown parameters 4 Under smple random samplng, models of ths type can be analysed usng straghtforward extensons of sngle-level lnear model theory Unfortunately, closed forms for the estmates of the model parameters (α, β, γ, σ ε, σ u, for the model above) are not avalable Instead, an teratve procedure, Iterated Generalzed Least Squares (IGLS), s used It produces estmates that converge to maxmum-lkelhood solutons Thus, the closed-form methods of adaptng weghted least squares to take sample desgn nto account cannot be employed n ths case A sample-weghted verson of IGLS (PWIGLS), whch weghts the frst- and second-level estmatng equatons by approprate weghts based on the selecton probabltes, has been developed to obtan consstent estmators of the parameters [see Pfeffermann and others (998) for the detals] 4 More recently, Pfeffermann, Moura and Slva (00) have proposed a model-dependent (purely model-based) approach for multlevel analyss that accounts for nformatve samplng The dea behnd the proposed approach s to extract the herarchc model holdng for the sample data as a functon of the populaton model and the frst-order sample ncluson probabltes, and then ft the sample model usng classcal estmaton technques The selecton probabltes become addtonal outcome varables to be modelled and to thereby strengthen the performance of the estmators Further detal s beyond the scope of ths chapter but can be found n Pfeffermann, Moura and Slva (00) A smulaton experment that follows closely the desgn of the Ro de Janero Basc Educaton Evaluaton study of 996 ndcates that the results of applyng the proposed method are promsng 4

14 Household Sample Surveys n Developng and Transton Countres D Categorcal data analyss Modfcatons to ch-square tests for tests of goodness of ft and of ndependence 44 Intal attempts to assess the effects of complex sample desgn on the analyss of categorcal data (data such that each pont falls nto one of a fnte number of categores or cells) concerned modfcatons to the ch-squared tests that are commonly employed ether to assess the goodness of ft between the dstrbuton of a sngle categorcal varable and a hypotheszed dstrbuton or to test for ndependence between two categorcal varables Although several modfed ch-squared tests have been proposed n the lterature for data from proportonate stratfed smple random samplng, the effect of that desgn s usually very small n practce Thus, n a study of modfed ch-square statstcs n eght data sets from proportonate stratfed samples n Israel, presented n table [from Ksh and Frankel (974)], none of the fnal teraton statstcs dffered by more than 4 per cent from those that would have been obtaned under smple random samplng (SRS) assumptons, and most dffered by less than per cent Data set Table Ratos of three terated ch-squared tests to SRS tests a/ No of strata Row x columns Sample sze Frst teraton Nathan s three tests Last teraton χ G χ 4 x x x x x x x x Source: Adapted from data n Nathan (97) a/ Eght contngency tables based on proportonate stratfed samples from Israel: Nos -4 of savngs, No 5 of atttudes, No 6 of hosptal data, No 7 of poultry medcament and No 8 of percepton experments G 45 Although the mpact of the desgn on categorcal data analyss under proportonate stratfed smple random samplng s usually small, ths s often not the case under clustered samplng, as was demonstrated n a semnal paper by Rao and Scott (98) When testng for goodness of ft, they showed that, under the null hypothess, the usual ch-square statstc,, s dstrbuted asymptotcally as a weghted sum of k- ndependent χ (that s to say, squared normal) random varables The weghts are the egenvalues of a matrx D (see annex) The matrx can be vewed as a natural multvarate extenson of the desgn effect for unvarate statstcs (see chaps VI and VII) Its egenvalues, λ 0, are termed generalzed desgn effects and can be shown to be the desgn effects for certan lnear combnatons of the desgn effects, d, of pˆ (= the estmated proporton of the populaton n category ) A modfed ch-square statstc, 4

15 Household Sample Surveys n Developng and Transton Countres C, can be obtaned by dvdng the standard statstcs by the average of estmates of these generalzed desgn effects, denoted by ˆλ Ths modfcaton requres knowledge only about the desgn effects of the cell estmates Although C does not have an asymptotc χ k dstrbuton under the null hypothess, t has the same asymptotc expected value as χ k (that s to say, k-), but wth a larger varance It turns out that C can be used emprcally to test goodness-of-ft by comparng the value of ths statstc to the crtcal value of χ k Ths can be seen n table 4 [from Rao and Scott (98)], whch dsplays the true szes of tests based on and on C, respectvely, for sx tems of data from the 97 General Household Survey of the Unted Kngdom of Great Brtan and Northern Ireland The survey had a stratfed three-stage desgn Table 4 Estmated asymptotc szes of tests based on and on C for selected tems from the 97 General Household Survey of the Unted Kngdom of Great Brtan and Northern Ireland; nomnal sze s 05 Varable k m ˆ λ Sze Sze ( ) ( C ) G: Age of buldng G: Ownershp type G: Type of accommodaton G4: Number of rooms G5: Household gross weekly ncome G6: Age of head of household The results show that the use of the standard ch-square statstc,, can be very msleadng, whereas the modfed statstc,, performs very well C 46 Smlar results hold when testng for ndependence n a two-way contngency table For a contngency table wth r columns and c rows, the null hypothess of nterest s H 0 : hj = pj p+ p+ j = 0 ( =, K, r; j =, K, c), where p j s the populaton proporton n the (,j)th cell and p +, p+ j are the margnal totals The usual ch-square statstc for data from a smple random sample, I, s asymptotcally dstrbuted as ch-square wth b=(r-)(c-) degrees of freedom under the null hypothess Ths need not be true when the sample desgn s complex In fact, the asymptotc dstrbuton of I s a weghted sum of b ndependent χ random varables, smlar to the case when testng for goodness-of-ft 47 A generalzed Wald statstc can be constructed based on estmatng the complete varance-covarance matrx of the estmates, h ˆ j = pˆ j pˆ + pˆ + j [see detals n Rao and Scott (98)] Fortunately, a frst-order correcton, whch requres only estmates of the varances of vˆ, seems to be an adequate approxmaton The modfed statstc s defned as j ĥ j = ˆ I δ ( C ) I ĥ, ( ), where ˆ δ s a weghted average of the estmated desgn effects of ĥ j When 4

16 Household Sample Surveys n Developng and Transton Countres estmates of these desgn effects are unavalable, as often happens wth secondary analyss of publshed data, an alternatve modfcaton can be obtaned by replacng δ by ˆλ, a weghted average of the estmated desgn effects of the cell proportons, d ˆj The adequacy of these approxmatons depends to a large extent on the relatve varance of the desgn effects A second-order correcton s avalable for use when the relatve varance s large 48 Emprcal results for 5 two-way contngency tables based on data from the General Household Survey of the Unted Kngdom of Great Brtan and Northern Ireland are gven n table 5 [from Rao and Scott (98)] They ndcate agan that: (a) the uncorrected ch-square statstc, I, performs very poorly n many cases; (b) the corrected statstc,, based on ˆ δ I (C ) attans the nomnal sze almost exactly; and () the corrected statstc based on ˆλ errs on the conservatve sde Table 5 Estmated asymptotc szes of tests based on I, ˆ I δ, and on ˆλ I for cross-classfcaton of selected varables from the 97 General Household Survey of the Unted Kngdom of Great Brtan and Northern Ireland; nomnal sze s 05 Cross Classfcaton r + c ˆ δ ˆλ Sze ( ) I ˆ Sze ˆ δ ) ( I Sze ˆλ ) ( I G G G G G G G G G G G G G G G G G G G G G G G G G4 G G4 G G5 G Generalzatons for log-lnear models 49 The results above for two-way tables have been generalzed by Rao and Scott (984) to the log-lnear model used n analysng mult-way tables Denote by π the T-vector of populaton cell proportons, π, n the mult-way table wth T π = t (for example, T = 4 for a x table) Denote the saturated log-lnear model (that whch ncludes all possble nteractons) 44

17 Household Sample Surveys n Developng and Transton Countres as M We consder testng of the hypothess that a reduced nested sub-model, M, s suffcent Let πˆ be the pseudo maxmum lkelhood estmator of π under M Ths s defned as the soluton of the sample estmate of the census lkelhood equatons (those that would have been obtaned on the bass of the populaton data) and s based on a desgn-consstent estmator of π under the survey desgn (see annex) Smlarly, let πˆ be the pseudo maxmum lkelhood estmator of π under M The standard Pearson ch-square statstc for testng H 0, based on πˆ, and on πˆ, does not usually have an asymptotc ch-square dstrbuton under the null hypothess Ths case s smlar to that of the two-way table, nasmuch as the standard Pearson ch-square statstc s asymptotc dstrbuton s a weghted sum of u ndependent χ random varables wth weghts δ, whch are the egenvalues of a generalzed desgn effect matrx (see annex for detals) 50 In order to take the complex desgn nto account, modfed ch-square statstcs, δ, ˆλ and ˆd are proposed Here ˆ δ s the average of the estmated egenvalues, λ s the average of the estmated desgn effects of pˆ ˆ, and d, s the average of the estmated cell ˆ desgn effects (see annex for detals) It should be noted that λ and d do not depend on the null hypothess, H 0, whereas ˆ ˆ δ does Furthermore, d requres knowledge only of the cell desgn effects as does λ when M s the saturated model 5 In the mportant case of models admttng explct solutons for πˆ and for πˆ, Rao and Scott (984) show that ˆ δ can be computed knowng only the cell desgn effects and those of ther margnals For nstance, for the hypothess of complete ndependence n a three-way I J K table, H 0 :π jk = π ++ π + j+ π + + k, where π + +, π + j+, π + + k are the three-way margnals, the value of ˆ δ can be calculated explctly as a functon of the estmates of the desgn effects of the threeway margnals and of the estmates of the cell desgn effects 5 The relatve performances of these modfed statstcs and of the unmodfed one are gven n table 6 [from Rao and Scott (984)] based on a 5 4 table from the Canada Health Survey The varables are gender (I=), drug use (J=5) and age group (K=4) The hypotheses tested were: (a) complete ndependence (denoted by ); (b) partal ndependence (for example, π π ; jk = jk ˆ π ) and, smlarly, ( ) and ( ) (c) condtonal ndependence (for example, π π π π ) and, smlarly, ( ) and ( ) jk = + k + jk + + k The desgn was complex, nvolvng stratfcaton and multstage samplng Moreover, post-stratfcaton was used to mprove the estmates 45

18 Household Sample Surveys n Developng and Transton Countres Table 6 Estmated asymptotc sgnfcance levels (SL) of and the corrected statstcs ˆ δ, ˆλ, ˆd : x 5 x 4 table and nomnal sgnfcance level α = 005 Hypothess (a) (b) (c) SL ( ) SL ( ˆ δ ) SL ( ˆλ ) SL ( ˆd ) δˆ CV ( δˆ ) The comparsons relate the actual sgnfcance levels (SL) to the desred nomnal level, α = 005 The results agan show unacceptably hgh values of SL for the uncorrected statstc The modfed statstcs ˆλ and ˆd, whch do not depend on the hypothess, perform very smlarly wth values of SL rangng from 006 to 09, whch are too hgh The modfcaton based on margnal and cell desgn effects, ˆ δ, has a more stable performance, wth SL values rangng from 0095 to 06, all above the nomnal level, probably owng to the large coeffcent of varaton (CV) of the ˆ δ ' s 54 To summarze, correcton methods are avalable for standard ch-squared test statstcs n categorcal data analyss These correctons are often necessary for vald analyss, gven a clustered sample, and can be appled wth relatve ease usng estmated margnal and cell desgn effects Detals on avalable software to deal wth the effects of complex sample desgn on chsquared tests and logstc regresson can be found n chapter I E Summary and conclusons 55 In ths chapter, we have llustrated methods for assessng the effects of commonly used complex sample desgns on the analyss of survey data The materal s ntended prmarly as an ntroductory exposton of the ssues rather than a prescrptve one The assessment and treatment of the effects of sample desgn on analyss can be dffcult and are not amenable to the formulaton of easly applcable how-to-do rules As we have shown, dfferent problems can have dfferent (or several dfferent) possble methods of resoluton These are hghly dependent on the hypotheszed model and the valdty of ts underlyng assumptons, on varous aspects of the sample desgn (for example, unequal selecton probabltes, clusterng, etc) and on the type of analyss contemplated Knowledge about the relatonshp between the model and the sample desgn varables s mperatve Unfortunately, ths nformaton s not always readly avalable, whch means that assumptons and approxmatons may have to be used nstead 46

19 Household Sample Surveys n Developng and Transton Countres 56 A frst and fundamental step n any analyss s the correct specfcaton of the underlyng model Ths s the responsblty of the subject-matter analyst, although the fnal dentfcaton of the model can and should be based on approprate statstcal technques The ntal exploratory analyss needed to dentfy the approprate model can be conducted usng standard graphcal and descrptve methods wthout takng nto account the effects of the sample desgn 57 Once an ntal workng model has been hypotheszed, t s necessary to determne whether the desgn has a confoundng nfluence on the analyss Ths can be done, for example, wth a test comparng the weghted and the unweghted estmates of lnear regresson coeffcents (see sect C) If the complex desgn needs to be ncorporated nto the analyss, then one must choose the approprate method for dong that The dsaggregated approach smply adds varables to the model related to the sample desgn 58 In many stuatons, however, the model cannot be modfed to fully reflect the effects of sample desgn n a meanngful way When ths s the case and the aggregated approach s to be used, two basc methodologes have been proposed to deal wth the potental mpact of the sample desgn One entals the modfcaton of classcal analytcal tools to take the desgn nto account Ths s the method best suted for dealng wth categorcal data analyss, where standard ch-square statstcs can be modfed on the bass of generalzed desgn effects The second approach s the development of approprately defned analytcal tools especally for the desgn Sample-weghted estmators and a large-sample Wald statstc have been proposed A relable estmator of the covarance matrx s needed before usng the Wald statstc Ths s not always avalable n practce 59 Consderable research nto the problems of handlng the effects of complex sample desgn on analyss has produced practcal methods, some of whch have been descrbed n ths chapter Further research s under way, and many of the exstng methods have already been ncorporated n new and exstng software Unfortunately, owng to the complexty of the problem, t s unlkely that any overall unform method wll be developed n the future The avalable methods and software must be appled wth extreme cauton Ther applcaton requres both basc knowledge of the underlyng theory and thorough understandng and experence n practcal model constructon 47

20 Household Sample Surveys n Developng and Transton Countres Annex Formal defntons and techncal results Regresson models (sects B and B) Standard lnear regresson model: Y α + β + ε =, wth ε ~ N( 0, σ ) nd Standard populaton regresson coeffcent: Ordnary least squares (OLS) estmator of β: B = b = N Y = N = n y = n = ( ) ( ) ( x x) ( x x) Unbased estmator of varance of b: ( b) s v =, n ( x x) = where s s the unbased estmator of σ, based on the varance of the estmated regresson resduals General lnear (heteroscedastc) regresson model: Y = α + β + ε, ε ~ N 0, σ wth ( ) nd Weghted populaton regresson coeffcent: B N * = = N ( ) ( σ ) = Y σ σ σ, where N = σ = N = σ σ 48