Model Quality Report in Business Statistics

Transcription

1 Model Quality Report in Business Statistics Mats Bergdal, Ole Blac, Russell Bowater, Ray Cambers, Pam Davies, David Draper, Eva Elvers, Susan Full, David Holmes, Pär Lundqvist, Sixten Lundström, Lennart Nordberg, Jon Perry, Mar Pont, Mie Prestwood, Ian Ricardson, Cris Sinner, Paul Smit, Ceri Underwood, Mar Williams General Editors: Pam Davies, Paul Smit Volume II Comparison of Variance Estimation Software and Metods

2 Preface Te Model Quality Report in Business Statistics project was set up to develop a detailed description of te metods for assessing te quality of surveys, wit particular application in te context of business surveys, and ten to apply tese metods in some example surveys to evaluate teir quality. Te wor was specified and initiated by Eurostat following on from te Woring Group on Quality of Business Statsitics. It was funded by Eurostat under SUP-COM 1997, lot 6, and as been undertaen by a consortium of te UK Office for National Statistics, Statistics Sweden, te University of Soutampton and te University of Bat, wit te Office for National Statistics managing te contract. Te report is divided into four volumes, of wic tis is te second. Tis volume deals wit te software available for variance estimation in sample surveys, comparing a range of pacages and metods, and evaluating some of teir properties troug a simulation study using a nown population Oter volumes of te report contain: a review and development of te teory and metods for assessing quality in business surveys (volume I); example assessments of quality for an annual and a montly business survey from Sweden and te UK (volume III); guidelines for and experiences of implementing te metods (volume IV). An outline of te capters in te report is given on te following pages. Acnowledgements Apart from te autors, several oter people ave made large contributions witout wic tis report would not ave reaced its current form. In particular we would lie to mention Tim Jones, Anita Ullberg, Jeff Evans, Trevor Fenton, Jonatan Goug, Dan Hedlin, Sue Hibbitt and Steve James, and we would also lie to tan all te oter people wo ave been so elpful and understanding wile our attention as been focussed on tis project!

3 Outline of Model Quality Report Volumes Volume I 1. Metodology overview and introduction Part 1: Sampling errors. Probability sampling: basic metods 3. Probability sampling: extensions 4. Sampling errors under non-probability sampling Part : Non-sampling errors 5. Frame errors 6. Measurement errors 7. Processing errors 8. Non-response errors 9. Model assumption errors Part 3: Oter aspects of quality 10. Comparability and coerence Part 4: Conclusions and References 11. Concluding remars 1. References Volume II 1. Introduction. Evaluation of variance estimation software 3. Simulation study of alternative variance estimation metods 4. Variances in STATA/SUDAAN compared wit analytic variances 5. References Volume III 1. Introduction Part 1: Annual statistics. Quality assessment of te 1995 Swedis Annual Production Volume Index 3. Quality assessment of te 1996 UK Annual Production and Construction Inquiries Part : Sort-term statistics 4. Quality assessment of te Swedis Sort-term Production Volume Index 5. Quality assessment of te UK Index of Production 6. Quality assessment of te UK Montly Production Inquiry Part 3: Te UK s Sampling Frame 7. Sampling frame for te UK

4 Volume IV 1. Introduction. Guidelines on implementation 3. Implementation report for Sweden 4. Implementation report for te UK 5. Visit to Statistisces Bundesamt, Wiesbaden, Germany, 3-4 Marc Visit to CSO, Cor, Ireland, 3 April Visit to INE, Madrid, SPain, 6 July 1998

5 Contents 1 Introduction... Evaluation of variance estimation software Requirements on software for business statistics Introduction Parameters Point estimators Variance estimation metods Te Taylor linearisation metod Te Jacnife metod Te Bootstrap metod Te Balanced Repeated Replication (BRR) metod Summary of requirements Critical comparison of software pacages Sample designs Nonresponse models and outlier treatment Parameters Estimators Variance estimators Interfaces, documentation and elp Initial reactions of new users to te software Correctness and speed Ease of integration wit processing systems Costs....3 Recommendations for variance estimation software for use in EU member states... 3 Simulation study of alternative variance estimation metods Te simulated population A model for data generation Domains and estimators Data features Processing Results Comparison of estimators Comparison of variance estimators Naïve variance estimators Comparison of software pacage outputs General conclusions Variances in STATA/SUDAAN compared wit analytical variances Expansion estimator Ratio estimator Wat does SUDAAN do? References...36 i

6 1 Introduction Paul Smit, Office for National Statistics One of te ey indicators of quality in sample surveys is te sampling variance arising from te random sampling mecanism troug te randomisation distribution. Tis indicates te variability introduced by coosing a sample instead of enumerating te wole population, assuming tat te information collected in te survey is oterwise exactly correct. For a discussion of te teory underlying tese calculations, see capters M 1 and M3 of te metodology report (volume I). For any given survey, an estimator of tis sampling variance can be evaluated and used to indicate te accuracy of te estimates. Te forms of tese estimators are often complex, especially wen te design contains strata or clusters, and wen te estimation model uses auxiliary information to improve te accuracy. In order to mae tese calculations feasible, appropriate software is required, and altoug it is possible to construct a program witin most survey processing systems to do tis for a specific survey, tere as been a recent trend towards te production of generalised software wic will calculate te appropriate variances in a wide range of commonly met survey situations. Tese must ten be incorporated into te survey process. Sampling variances are often not time-critical information, and any difficulties wit data transfer to or setup of tis software are offset by te generalised nature of te programs. In tis paper we evaluate five generalised pacages wic are publicly available: CLAN, GES, SUDAAN, STATA and WesVar PC. Tere are four main variance estimation metods, Taylor, jacnife, bootstrap and balanced repeated replication (tese are explained in section.1.4), and between tem tese pacages cover all te available metods except te bootstrap (Table 1.1). Tese are te pacages wic were available at te time of putting togeter te tender for tis study, wit te exception of PC-CARP wic was available but as not been studied. Oter pacages are being developed; tose nown to te Model Quality Report team are BASCULA and POULPE but neiter of tese seems to be fully functional in its current version. Metod Direct + Taylor series metods Jacnife Bootstrap Balanced repeated replication Software CLAN GES STATA SUDAAN GES SUDAAN WesVarPC None SUDAAN WesVarPC Table 1.1: Variance estimation metods available in te evaluated software pacages. 1 Reference is made trougout tis document to te Metodology report by prefixing section references wit an M.

7 Te requirements for a variance estimation pacage are discussed in section.1, and tere is a comparative description of te pacages in section.. Section.3 draws conclusions about te suitability of te pacages for general use in business surveys in EU member states, and maes recommendations for wic sould be adopted. A separate simulation study as been undertaen to loo at te properties of te available variance estimators, and tis is presented in capter 3 of tis report. A more detailed description of te differences in underlying metods between STATA/SUDAAN and te oter pacages for te Taylor linearisation approac to ratio estimation is given in capter 4. 3

8 Evaluation of variance estimation software Paul Smit, Office for National Statistics Sixten Lundström, Statistics Sweden Ceri Underwood, Office for National Statistics.1 Requirements on software for business statistics.1.1 Introduction Te units in business surveys can be of various types, suc as enterprises and ind-of-activity units. Mostly a Business Register (BR) is used as te frame for te survey. Tere is a set of units on te BR, suc as enterprises, legal units, local units, and possibly ind-of-activity units. Tere is a set of variables for eac type of unit, some common to oter types of unit, some unique. Ordinarily, te BR contains information on wic industry eac unit belongs to and a measure of te size of te unit. Te size variable is often te number of employees, or peraps a measure of turnover (depending on unit level). Tese variables and teir reference dates affect te use of auxiliary information in te sampling design and in te estimation process. In business surveys two typical inds of probability sampling design can be identified, namely (i) one-step element and (ii) one-step cluster. Typical examples are (i) surveys wit te enterprise as bot te sampling unit and observation unit, and (ii) surveys wit te enterprise as te sampling unit and all its ind-of-activity units or all its local units as te observation units. Te population is often stratified by industry and size, and from eac stratum a simple random sample is drawn. Te stratification variable industry is used wit regard to te domains of estimation tat are mostly defined by industry. Size is usually an effective variable for reducing te sampling variability (see capter M). Business surveys are ordinarily carried out continuously, eiter annually, quarterly or montly. Te samples may be co-ordinated over time, using a panel system or possibly a tecnique based on permanent random numbers (Olsson 1995). Units in business statistics typically cange fairly rapidly; tey can die, tey can merge wit anoter unit and tey can split into several units. Te industrial classification may cange, and te size of te unit can vary..1. Parameters Let us loo at te various types of finite population parameters tat are typical for a business survey. Consider te finite population of N units U { u u u } interested in te population total t y = 1,...,,...,. Sometimes we are = y (.1) U N 4

9 were y is te value of te study variable, y, for te t element. Moreover, totals for domains typically industries are also common. Let us denote te domain set by U d, d = 1,..., D, and set y ( d) = y if unit U 0 oterwise 5 d. Ten te total for domain d is ty = U y( d) = U y (.) d Ratios of different types are common in business statistics. To define tese types let z be anoter study variable and let te population total for z be denoted t z and te domain total t zd. One type of ratio is d y d z d d R = t t (.3) A typical example ere is production per ead wit industry as domain. Anoter type of ratio is R = t y t y (.4) sowing for example te production of an industry, relative to te wole population. Anoter parameter of interest is d d I = t t (.5) y d z d were prime ( ) indicates relative to anoter population. A typical application of (.5) is te relative cange in production (say) by industry from one period to anoter, tat is, te totals in te numerator and te denominator ave different reference times, but oterwise relate to te same variable and domain. Te sample units (involved in te numerator and denominator) are partly te same, partly different, and units tat contribute to te total on bot occasions may ave canged domain (industry) in between. Indices of production (say) are examples of complex sets of parameters, typically built up from components lie (.5), and usually also deflated by price indices. Yet (.5) is already a callenge for te available software. Te complexity also depends on te way samples are coordinated over time..1.3 Point estimators To estimate te parameters defined in section.1., a sample s of size n is drawn from U (or actually from te frame). Stratification is commonly used in business surveys, tat is, a simple random sample s of size n is drawn from te stratum U, = 1,..., H, were U = for H U =1 U. Let te stratum sizes be N, = 1,..., H, and te design weigts are d = N n s. However, nonresponse occurs in te survey process, and te response set r of size m is obtained, were r s. Tere are two main ways of treating tis problem, namely weigting and imputation. In weigting, te nonresponse compensation adjustment weigt v is

10 constructed primarily wit te aim of reducing te nonresponse bias, but is also used to reduce te additional component of sampling error caused by nonresponse (see capter M8). Wen using te weigting approac te estimator consists of te sum of te weigted values for elements in r, were te weigt consists of te product of d and v, were v is te tool for maing te inference from r to s and d from s to U. Wen imputation is used, values for all n elements are used in te estimation, but now n-m of tese values are estimates (approximations) of te real values. None of tese metods is expected to completely eliminate te bias. Wen a substantial nonresponse bias is still present te variance estimate and te confidence interval will be an unrelevant and incomplete measure of te quality of te point estimate. As indicated above, nonresponse will also cause an additional component of sampling error. Tis is obvious in weigting, since te number of observations is reduced from n to m. In te following, we describe estimators used in business surveys. Here we describe te estimator using a nonresponse compensation adjustment weigt, wic as a more complex form tan te estimator based on imputation. Te nonresponse compensation adjustment weigt is an approximation of te inverse of te response probability. Tat is, one sees a relevant model of te response probabilities. Commonly, tis model consists of a grouping of te sample s. Särndal, Swensson & Wretman (199) denote tem Response Homogeneity Groups (RHGs). In te following we will coose among tree different types of RHGs, namely (i) (ii) (iii) strata and RHGs coincide RHGs are subgroups of strata RHGs cut across te strata (.6) Te simplest estimator is te Horvitz-Tompson estimator, combined wit nonresponse model (i). Tat means tat we find it plausible tat eac sampled element in te stratum responds wit te same probability. In tis case te nonresponse compensation weigt is n v = and since d = N n te resulting weigt is N m and te estimator as te m form H $t = N y y r =1 (.7) were y r 1 = m r y. A somewat more complex estimator is obtained wen using nonresponse model (ii), namely $t y H N L = n = 1 n q= 1 q y rp (.8) 6

11 were n q is te size of te part of s tat falls into RHG q; m q is te size of r q, te response set in RHG q, and y r p 1 = m p r p y Wen using nonresponse model (iii) an even more complex estimator is obtained. Let us ere express it by te general version. $t = d v y (.9) y r Frames used in te Member States regularly contain more information tan industry and number of employees, for example, te turnover from a previous time of reference. Moreover, geograpical information for te local units is commonly available. Tus, tere may be register information, wic is correlated wit te study variables and/or te response probabilities, but not used in te estimator of te form (.9). A simple version of suc information is a partition of te population. To demonstrate estimators based on suc a partition we let U,..., U,..., U be groups tat form a mutually exclusive and exaustive 1 p P partition of te population. Assume tat we now te sizes of tese groups, N1,..., Np,..., NP. Ten tey can be used as poststrata. Suc an estimator, using te nonresponse model (i) mentioned above, as te form t$ yr P N p H N = r N$ y p m p = 1 = 1 p (.10) wit N$ = N$ p H =1 p, were $ N to te union of U and U p. p N = m m p ; m p is te size of r p, te response set tat belongs Estimator (.10) is a special case of te following general estimator were $t = d v g y (.11) yr r g = T T 1 ( x ) ( / ) U d r v x d r v x x σ x σ 1+ (.1) By coosing te positive factors σ te approac can be made very flexible. Tis will become apparent in subsequent sections. Te vector x is called te auxiliary vector in wat follows. Estimator (.11) is based on a general approac to regression for two-pase sampling following Särndal & Swensson (1987). It is ere used in te nonresponse situation, but since we do not now te response probabilities te second-pase inclusion probabilities ave to be estimated in some way (see also M.3.1.5). Te inverse of tis estimate is denoted by v. In wat follows te estimator (.11) is called te GREG estimator. 7

12 In te case of poststratification te auxiliary vector is defined by x = ( γ γ γ ) T 1,..., p,..., P 1 if unit U p were, for p = 1,..., P, γ p = and σ = 1 for all. Tis poststratification 0 oterwise approac gives us one simple metod of dealing wit outlying observations in a survey, since tey can be moved into an appropriate poststratum for estimation. Most of te classical estimators can be derived as special cases from te GREG estimator. For example, if x = x for all and σ x, were x is a continuous variable, and wen nonresponse model (i) is used, ten te following estimator is obtained: $t yr = H N y = 1 H N x = 1 r r U x (.13) Estimator (.13) is sometimes called te combined ratio estimator. Sometimes te group totals U p x are nown and, in tis general case, te p-groups are called model groups. Let us present a simple example. As before assume tat x is a continuous variable, but ere we now te quantities ; p = 1,..., P. Let ( γ x γ x γ x ) T x = 1,..., p,..., P, σ x for eac p-group, and te RHGs coincide wit strata (nonresponse model (i)) ten te GREG estimator taes te form U p x t$ yr = H $ P N = 1 H p= 1 N$ = 1 p p y x rp rp U p x (.14) If strata and model groups coincide ten estimator (.14) can be written $t yr H yr = =1 x r U x (.15) Estimator (.15) is sometimes called te separate ratio estimator. Wen x = ( 1, x ) for all and σ = constant, ten te classical regression estimator is obtained. Many business surveys are subject to occasional unusual observations, or outliers, wic can ave a large effect on te estimates. In tese cases, robust versions of point estimators are often used, wit te simplest being te poststratification estimator wit te outliers in teir own (completely enumerated) poststratum. Tis follows from te metod above (.13). Oter metods involve adjusting te weigts or te responding values, and winsorisation is becoming widely used witin te UK for treating outliers. Tis leads to a different estimator, wic does not necessarily fit completely into te GREG framewor. 8

13 Te parameters (.1)-(.4) are totals or functions of totals from te same period of reference. Estimators for tese parameters can be obtained by replacing tese totals by teir estimators. Parameter (.5) is muc more complex since it contains totals from two periods of reference. In most surveys two consecutive samples are drawn in suc a way tat tey overlap eac oter. Tat maes it possible to construct combined estimators tat are more effective tan just replacing te totals by teir estimators. However, variance estimation becomes complicated. We do not go deeper into tis problem but just refer to Nordberg (1998), wo as found a solution to te special sampling procedure used at Statistics Sweden. So far we ave only discussed one-step element sampling designs, but it is easy to see ow te one-step cluster alternative affects te formulas. Auxiliary information can be nown at te cluster level or at te unit level. In te latter case we can coose to use te auxiliary information eiter at cluster level or at unit level. Wen te auxiliary information is nown only at te cluster level te model groups are, of course, defined for tat level..1.4 Variance estimation metods Tere are four principal ways of calculating variances (Wolter 1985), eac unbiassed or asymptotically unbiassed in most widely-used design-estimation strategies if full response is assumed, but eac (in general) producing a different value for te unbiassed estimate: direct calculation and Taylor linearisation; jacnife; bootstrap; balanced repeated replication metod. Before we discuss tese metods just a few words about variance estimation wen imputation is used, following te discussion in section.1.3. Te literature describes many imputation metods suc as nearest neigbour donor, current ratio, current mean, auxiliary trend, etc. However, te teoretical development of variance estimators wen data contain imputations is still in its initial pase. Two examples of articles on tis problem are Särndal (199) and Deville & Särndal (1994). In surveys were te complete data set is treated as if it were te full-response set, owever, tis will commonly underestimate te variance (see, for example, Rubin 1986) Te Taylor linearisation metod Direct calculation involves application of (normally) te Sen-Yates-Grundy estimator (Sen 1953, Yates & Grundy 1953) to form te variances of simple survey estimates. More complex survey estimates are first linearised by taing te first-order terms in an appropriate Taylor-series expansion, and ten te SYG estimates are inserted into te linearised formula. Tis is basically a set of appropriate linear expressions for te variances of estimators, wic as to be coded into te software. Every different design-estimand combination requires a different formula wic must be (essentially) ard-coded; separate formulae are not required for different estimation models if te GREG estimator (see equation (.11)) is present, as all te commonly used models are eiter GREG or special cases of it. 9

14 .1.4. Te Jacnife metod Te jacnife involves dropping an observation and recalculating te estimates from te remaining observations, repeating successively until all observations ave been dropped, and ten finding te variance of te resulting series of estimates (wit a suitable multiplier to give approximate unbiassedness). Te drop-one jacnife is usually used, as it can be sown to give te variance estimate wit te smallest sampling variability, altoug it is possible to drop pairs of observations (or even more) too; tis strategy is usually adopted to speed up processing since drop-one is te most processor-intensive metod. We consider only dropone metods ere. More information on te jacnife estimator is in M.4..-M It sould be noted tat te jacnife is only strictly applicable in wit-replacement designs. It can be used in witout-replacement designs were te sampling fractions are sufficiently small (Wolter 1985, p168), but in many business survey designs, te sampling fractions are relatively large. Te dangers of tis approac are illustrated in te simulation in capter 3 below Te Bootstrap metod Te bootstrap involves resampling a number of times wit replacement from te sampled observations, and calculating an estimate for eac of te bootstrap samples. Te variance of tese bootstrap estimates is ten calculated, again wit a suitable multiplier to ensure unbiassedness. Te metod is described in more detail in M Te Balanced Repeated Replication (BRR) metod Tis is derived from te balanced alf samples (BHS) metod wic as a very specific application in cluster designs were eac cluster as exactly two final stage units. By successively deleting one of tese units and canging te weigt of te oter to compensate, a range of estimates can be produced wose variance can be calculated and suitably adjusted to give an appropriate variance estimator (Wolter 1985). Various adaptations of tis can be applied in designs were te clusters ave variable numbers of units, based on dividing tese into two groups. Recent researc (Rao & Sao 1996) sows tat only by using repeated divisions ( repeatedly grouped balanced alf samples (RGBHS)) can an asymptotically correct estimator be obtained. Tis metod, ten, can only be used for te usual stratified designs in business surveys if we are prepared to treat a stratum as if it were a cluster, and to run te pacage a number of times wit different divisions of te elements into two groups; were tere is an odd number of elements in te stratum te results are biassed, and ways of reducing tis bias (but not eliminating it) are described in Slootbee (1998). Tere are ways in wic tis can be done, but te results are typically unsatisfactory and te manipulation of bot data and software becomes very involved..1.5 Summary of requirements Tere is a number of requirements for point and variance estimation in business surveys wic any software sould satisfy. We ave pointed out several suc requirements in te ting to be estimated 10

15 previous sections. However, in order to simplify te evaluation we will ere present a structured summary of tese requirements. Te demands on te software will certainly vary between Member States (MS). Consequently, pacages wic only meet some of te requirements mentioned aead may be sufficient for a particular MS, provided tat tey meet te requirements of tis MS. Te pacages will be evaluated wit respect to teir ability to cope wit te following situations. Sampling designs: One-step stratified sampling of units or clusters. In eac stratum a simple random sample is drawn. In some strata te finite population correction (fpc) as a large effect; in tae-all strata it reduces te sampling variance to zero. Panels or random number tecniques are used in te sampling procedure. Nonresponse models and outlier adjustment: Weigting witin RHGs (i)-(iii), as described in.1.3 and equation (.6) or imputation as described in sections.1.3 and.1.4, and outlier treatment using poststratification or winsorisation as described in.1.3. Parameters: Parameters for measuring levels as in (.1)-(.4) and parameters for measuring cange as in (.5). More complex parameters suc as indices are also of great interest. Estimators: Estimators for totals as defined in (.7) to (.15). Ratios and oter functions of tese estimators are also of interest. Point estimates and te corresponding variance estimates for parameters suc as (.5), for example measures of cange between two consecutive periods (a demanding tas for te pacages) are of interest. Variance estimators: availability of different variance estimation metods (Taylor, Jacnife, BRR, Bootstrap). Te pacages will also be evaluated wit respect to: interface, documentation and elp functions; weter computations are correctly done; execution time; simplicity to integrate into production systems; cost for purcase or licenses.. Critical comparison of software pacages Te software pacages evaluated ere fall into two distinct groups based on te way tey are designed and te type of situations in wic tey can be used. It maes sense to structure te discussion around tese two groups, as te metods employed witin te pacages are very similar witin groups, and quite different between tem. Group I: CLAN and GES are designed for stratified designs wit estimation models up to te complexity of te generalised regression (GREG) estimator. Tey are caracterised by aving two parts to teir processing, one in wic te appropriate weigts are calculated for te survey observations, and ten a second pase were te estimates and teir associated variances are produced. Te variances specifically tae account of tese weigts, and are based on te variances of te residuals from te GREG model (or a specific (simpler) case). 11

16 Group II: STATA, SUDAAN and WesVar are designed principally for cluster designs wit versions of te Horvitz-Tompson (HT) estimator (in most cases optionally involving poststratification); te ey ere is tat GREG-type estimators (including most of te simpler cases suc as ratio and regression estimation) are not supported. STATA and SUDAAN bot wor in a straigtforward way wit stratified designs, but WesVar needs clusters at te penultimate sampling stage in order to wor effectively (mainly because of te BRR variance estimation metod employed). Tis group is caracterised by not aving a weigt calculation pase and requiring te (HT) weigt to be input. In some cases te software can be made to produce valid or approximately valid results for estimators oter tan HT, but tis is typically not easy and may require te pacage to be run more tan once for eac survey...1 Sample designs CLAN and GES ave te following designs built-in: 1. simple random sampling;. stratified designs; 3. probability proportional to size (wit replacement) designs; 4. one stage cluster designs (optionally wit te clusters in strata). Tese cover te main probability designs used for business surveys in Member States, but do not extend to te more complex designs used in some social surveys. It is possible to force more complex designs troug CLAN and GES by accepting some assumptions about variances at lower stages; one option is to set appropriate jacnife adjustment weigts witin GES for two-stage designs. All of tese metods, owever, are vanisingly rare in business surveys, and require considerable expertise and input from te user, so tey are not considered furter ere. Statistics Canada ave just begun to develop two-stage cluster sampling for inclusion in te next version of GES (version 5.0). STATA and SUDAAN ave te following designs built in: 1. simple random sampling;. stratified designs; 3. one stage cluster designs; 4. two- and multi-stage cluster designs. Tese cover a wider range of designs, but te complex cluster designs are not typically used for business surveys, and we now of no examples of teir current use in business surveys in member states. However, tis does give some added flexibility in te use of te pacage for various surveys. WesVar as te following two designs available: 1. simple random sampling;. two-stage cluster designs wit exactly two primary sampling units in eac cluster. Tese designs are very restrictive in te context of business surveys were clusters are rarely used, and were treating a stratum as if it were a cluster typically gives more ten two primary sampling units in eac cluster. For tis reason we will not concentrate muc discussion on WesVar. 1

17 Te finite population correction (fpc) can ave a large effect on te variance estimates; witin GES and CLAN it is included automatically (except for te jacnife estimator in GES). In STATA a specific command option must be used to get te fpc, and in SUDAAN it depends on te design weter te fpc is included or not. GES and SUDAAN alie include te fpc automatically in witout-replacement designs, and exclude it in wit-replacement designs. However, it can in some circumstances be reasonable to use wit-replacement variance estimators as approximate variance estimators in witout-replacement designs, wen inclusion of te fpc can become important; inclusion of te fpc is unliely, owever, to solve all te difficulties of tis approac... Nonresponse models and outlier treatment CLAN is te only software pacage to include te specification of non-response models. Tis is done by defining response omogeneity groups, wic can be defined differently from te stratification and model groups, and provide a flexible way of defining te weigting adjustment for non-response in line wit equations (.7)-(.10). Tis additional option witin CLAN is similar to te sort of metodology wic would arise in a two-stage stratified design, wit first stage selection being sampling from te frame and te second pase being sampling respondents from te selected sample. Tis means tat te extra functionality can be used to mae CLAN give appropriate answers in some complex designs if tere is (or can be assumed to be) no non-response. For te oter software pacages considered ere, only two alternatives are available, eiter to assume tat non-responding units were not sampled, wic is equivalent to imputing teir value wit te mean under te estimation model for te stratum in wic tey were selected, or to fill in te missing values using some imputation procedure and ten use te completed dataset. In bot tese cases (but particularly te second), it is very liely tat te calculated variance underestimates te true variability. Te only reasonable metod of calculating variances wit pacages oter tan CLAN would be to use a stocastic imputation procedure to create multiple datasets (multiple imputation, Rubin 1987) and use te pacages to produce a series of estimates wic can ten be suitably combined. Tis approac involves a lot of additional processing not available witin te pacages, and as not been attempted ere. Outlier treatment by moving outliers into a poststratum can be appropriately set up in most of te software described ere (in GES and CLAN by setting up appropriate model groups, and in SUDAAN by using te poststratification options). Exact variance calculations for oter metods, specifically winsorisation (Koic & Smit 1999a, b), are not available in any pacage, but a good (first-order) approximation can be obtained by using te winsorised values as if tey were te survey values. 13

18 ..3 Parameters Te parameters wic can be estimated in GES are: (a) count (an estimate of domain size); (b) total (equations (.1) and (.)); (c) mean; (d) ratio(equations (.3) and (.4)). Witin CLAN, te user needs to construct several macros to specify te estimation to be undertaen, and at tis stage it is possible to include arbitrary rational functions of totals, so tat purpose-built estimands can be constructed and teir sampling variances calculated explicitly witin te pacage. GES allows only te four estimands described above, but in a similar way te variances of linear combinations can be found afterwards outside te pacage. In general owever, tis will require more expertise and effort tan setting up te appropriate macros in CLAN. Te PC-CARP documentation suggests tat it estimates quantiles (wit te appropriate variances) too, a facility not available in eiter GES or CLAN. STATA and SUDAAN ave: (a) count; (b) mean; (c) total; (d) ratio; (e) regression parameters; (f) Wald statistics; (g) logistic regression parameters; () quantiles; and for STATA only (i) arbitrary linear combinations of parameters. Some of tese are not currently widely used in business surveys, but tere seems to be some development in te field of estimating distributions, wic will mae te estimation of quantiles more important, and te facility to produce estimates and variance estimates for arbitrary linear combinations of parameters can be used to assist in te estimation of variances of complex population parameters suc as canges, index numbers and so on (see capter M3). WesVar produces a similar range of statistics to STATA and SUDAAN, including arbitrary linear and non-linear combinations of statistics. Te sampling variances of te non-linear statistics can be found because WesVar relies on replication metods. Of particular interest in repeating business surveys are estimates of movement or cange. Were te units are exactly common between two periods (almost never true even if te design is set up in tis way because of differential non-response), ten any of te pacages ere can be used to estimate te movement by including te responses for different periods as two survey variables. Wen te units are not te same, ten it becomes very callenging to produce an appropriate estimate of cange and its variance. Witin CLAN tis can be acieved by including te union of te two samples as te sample, and specifying te 14

19 response omogeneity groups in suc a way tat weigting adjustments are made for te units wic were not sampled because of te sample rotation as well as tose units wic did not respond. Because te variance estimation reflects te additional uncertainty due to imputation, it gives an approximately correct variance for te estimate of cange taing account of te substitution of units (if te non-response weigting completely adjusts for bias). A similar imputation can be done to fill in te missing data for rotated (and non-responding) units before entry into te oter pacages, but because te pacages do not appropriately account for imputation wen estimating te sampling variance, it will typically be underestimated. More complex statistics are also of interest, for example deflated index numbers. None of te software is currently able to tacle suc combinations of information, and te only reasonable approaces are (i) linearisation of te target statistic and calculation of te appropriate components of te linear combination in CLAN or STATA or from results produced by any of te software pacages, or (ii) a sensitivity-type analysis sowing te effect of sampling errors on te overall statistic (see M3.4 and Koic (1998))...4 Estimators A range of estimators is available for use in business surveys, depending on te range of auxiliary information available from te business register. Te simplest estimation metod is Horvitz-Tompson (HT) estimation (also called simple raising, expansion estimation and number raised estimation), wic involves weigting eac unit by te inverse of its selection probability. Tis estimator is available in CLAN, GES, STATA and SUDAAN, but is not given in WesVar wic is designed purely for variance estimation and does not provide point estimates. Tis estimator is unusual in ONS business surveys, altoug tere are some examples of its use in recent years; in oter member states, for example at Statistics Sweden, it is widely used. Te only information wic is normally required is te number of units (altoug HT for πps sampling as already used additional information in setting up te selection probabilities). Were additional auxiliary information is available from te business register, more complex estimators are often used. In te ONS te ratio estimator (separate or combined, equation (.13) and te simplification of it wit a single stratum) is almost ubiquitous. Te true ratio estimator is available only in CLAN and GES, were it is andled appropriately wit te correct model used to calculate residuals to feed into te sampling variance calculation. In SUDAAN and STATA only te HT estimator is available. However, it is possible to obtain approximately correct variances for (one-variable) ratio estimation by (i) calculating te ratio of te survey variable to te auxiliary value, witin strata (for separate ratio estimation), taing account of te selection probabilities; (ii) constructing an additional variable as te residual between te observed value and te ratio applied to te auxiliary value, and (iii) calculating te variance of tis residual witin strata again taing account of te selection weigts. Tis involves two passes troug te software wit some additional manipulation and produces only te variance directly tere is no point estimate, and if tis is required it 15

20 needs some additional processing after te ratios ave been calculated to produce it. Crossstratum ratio estimation can naturally be done in te same way by defining appropriate groups witin wic to calculate te ratio. Te additional feature of coosing te variance function is not available; for estimation of ratios in SUDAAN only te ratio of averages ( rˆ = wy wx, wit appropriate weigts w) metod is supplied (tat is, oter ratios suc as te average ratio ~ 1 y r = w are not available). It is naturally also possible to w x supply different weigts to te expansion estimator, suc as tose taen from ratio, regression and GREG estimators, but naïve application of tese weigts in te standard HT estimator does not give te correct variances (more detail is given in Capter 4). Neverteless te effects of using tis sceme are investigated in te simulation in capter 3. Furter complexity in te estimator can be introduced by using more variables witin a regression estimation framewor, altoug tere are very few current examples of tis sort of estimation in business surveys in te UK and Sweden (only te Annual Employment Survey uses tis metod in te UK). However, it seems liely tat tese metods will become more important in te future. As before, CLAN and GES cover tese metods directly, wereas SUDAAN and STATA do not include te direct estimator, but can be used to estimate te regression parameters and ence calculate residuals to use in calculating te sampling variance. We ave not attempted to verify tat tis wors using classical regression estimation (tat is, wit te variance approximately constant wit size). Getting an appropriate (non-constant) variance function in regression may be extremely involved (especially were tere is more tan one explanatory variable), but tis is properly dealt wit under full calibration in te next paragrap. Te most general estimator, te GREG estimator, wic allows calibration to many auxiliary totals and provides a facility to add constraints to bound te weigts, is available in only CLAN and GES, and cannot be incorporated into STATA or SUDAAN. We now of no business surveys in member states wic rely on tis tecnology at te moment. One side effect of te inclusion of te GREG estimator is tat te variance function for te ratio and regression estimators can be defined by te user, by supplying suitable values to te software (normally σ x α were x is one of te auxiliary variables and α = 1, see (.1), or sometimes for some oter value of α). By maing te variance proportional to an extremely large number for any particular observation, its effect can be removed from estimation (tat is, its g-weigt will be 1), giving a rudimentary outlier treatment/robust estimation metodology...5 Variance estimators Te use of BRR wit business surveys is typically difficult, as described in section 0. WesVar relies almost entirely on te metod of BRR, and so is not a serious contender for recommendation for business surveys. SUDAAN also as tis metod available as one option among several, but tere seems to be little to commend it over te oter metods in te current context. 16

21 Te four main pacages investigated (CLAN, GES, STATA, SUDAAN) all include te direct ( Taylor ) metod of variance estimation (te SYG estimator). Te implementation is basically a set of appropriate expressions for te variance of estimators, wic as to be coded into te software. Tis is te way in wic most business survey variances are calculated, and as suc eac of te four software pacages fulfils our requirement for a basic design-based variance estimator. For te simpler cases of expansion and ratio estimation wit model groups corresponding wit strata were te full complexity is not needed, tere can be little to be gained from te software; in tese cases, purpose-written programmes may be perfectly adequate. Most pacages include te finite population correction automatically witin teir variance calculation for witout-replacement designs, but STATA requires it to be specified explicitly as a command option if it is required. Jacnife variance estimators are available in GES, SUDAAN and WesVar. It sould be noted tat te jacnife is only strictly applicable in wit-replacement designs, and te documentation for te pacages points tis out. It can be used in witout-replacement designs were te sampling fraction is sufficiently small, but in many business survey designs, te sampling fractions are ig. A furter adjustment can be made by including te fpc, but none of te pacages do tis automatically. In GES it is not obvious from te documentation tat tis is missing. Te validity of te outputs is discussed as part of te results of te simulation exercise (capter 3). In GES te jacnife option requires te user to set up jacnife groups explicitly. Te dropone jacnife is te most efficient variance estimator, and te easiest and quicest set-up is to use tis metod, by maing every element a jacnife group, and giving eac group an equal jacnife adjustment weigt. Altoug tis is fairly intuitive, it is a same tat te software does not contain a default to allow it to appen automatically. If speed of processing is vital it is possible to set up jacnife groups containing several elements (faster, less efficient and less intuitive), in wic case tere are also several ways to form appropriate jacnife adjustment weigts usually te weigt is equal to te number of elements in a group, but for multi-stage designs te weigts can be set to te number of secondary sampling units to give a variance estimate under te complex design. Tis flexibility is useful in concept but unliely to be applied in practice in business surveys. SUDAAN provides a default jacnife metod by simply coosing te eyword for jacnife variances; tis is in fact te drop-one metod. Tere is no facility for user-defined jacnife groups. In WesVar two forms of te jacnife estimator are provided one is dependent on te specific design wit two elements in eac final stage cluster, and te oter is te drop-one jacnife, wic is available only for simple random sampling. By processing strata separately and using te drop-one jacnife it is possible to force te software to deal wit some business surveys, but it is not in general suited to tem. None of te software pacages considered implements a bootstrap variance estimator. 17

22 ..6 Interfaces, documentation and elp In many NSIs it seems tat SAS is becoming te main tool for survey analysis, and tis is reflected in te software seen ere. CLAN and GES are bot written as a series of SAS macros, so tat te SAS pacage is required to use tem. CLAN uses only CORE and BASE SAS, wereas GES uses CORE, BASE, AF, FSP and IML. SUDAAN is available in two versions, one free-standing and one wic can be called directly from SAS. WesVarPC is designed to loo somewat lie SAS but oterwise as no connection wit it. Following an agreement between te autors, SPSS versions will now include te WesVar software. STATA is stand-alone (but provides a complete statistical pacage), and only available for Windows 95, Windows NT or later operating systems. Tere are two basic approaces to setting up te data and commands for te software, and tese are not related to te groupings described at te ead of section.. Te first is to provide appropriate commands and leave te user to construct a programme or script wic is ten submitted to te software, wic returns wit te completed calculations, and tis is te basis for CLAN and SUDAAN. CLAN in fact goes a stage furter and requires te user to construct several macros as well as putting togeter te code to produce te final outputs. CLAN is basically a series of macros, wic accept data and oter macros as input. Once te user-defined parts are written, te user calls te macros in te appropriate order and combination in order to get te results. Because te program is written in SAS, te entire interface is supplied by SAS. Tis metod maes it relatively easy for te software to be flexible and to cope wit cases were unusual estimates are required; it also, by dint of requiring te user to now a fair amount about te way in wic te pacage is constructed in order to use it, prevents te mindless application of default metods in situations were tey are not appropriate. By te same toen, owever, a reasonable amount of expertise in estimation teory and in SAS programming are required to use te pacage. Fortunately te recently produced CLAN manual (Andersson & Nordberg 1998) is very clearly written and sows in a very straigtforward way ow to set up te appropriate macros and data. Tere is no on-line elp system available wit CLAN. Output is sent only to a SAS dataset, wic can ten be printed, exported or furter manipulated using SAS. Tere is no formal support system for CLAN, but informal support from Statistics Sweden is available on a case by case basis. SUDAAN can be viewed in a similar way, except tat te macros are called procedures, and in te SAS-callable version tey beave lie SAS procedures. In stand alone SUDAAN tere are Program Editor and Output windows (te output ere doubles as bot Log window and Output window according to SAS s view of te world). All tat is required is for te user to learn te appropriate syntax and to type in tese commands. Te submit button is ten cliced, and te pacage processes te data as required, sending results to te output window (and/or an appropriate file). Most of te syntax is easily learnt, but tere are a few oddities: two procedures ave different names in te SAS version to avoid reserved eywords; te formatting statements in SUDAAN are notoriously long-winded and do not ave sort forms. 18

23 Tere is a two-volume manual wic describes te syntax and te basic usage of SUDAAN, wic is a very useful guide for te beginning user. However, it does not contain any explanation of te teory used in te software, and in several places tere are bald statements from wic it is almost impossible to wor out exactly wat te software is doing (for example, a poststratified estimator is mentioned for several procedures). Tere is in fact a metodology guide (Sa et al. 1995), but tis wasn t sent out as part of te documentation to accompany te license. Wit tis guide to and te pacage is well-documented. Te on-line elp system covers only te main user-guide part of te manual. SUDAAN as te advantage of reading and writing files in several formats, including SAS files (in te standalone and SAS-callable versions), text, and SPSS (in te stand-alone version only). Te SAScallable version is particularly useful wen combining SUDAAN processing wit oter operations, for example in producing a ratio estimator or doing experiments or simulations were te procedure can be embedded in a macro or loop. Tere is support for SUDAAN, and an support address, during office ours on te West Coast of te USA (approximately 1600 to 400 GMT). Te second approac is one wic provides an interface to lead te user troug te stages of setting up te appropriate files, meanwile writing te commands eiter in te foreground or beind te scenes. GES as an interface wic leads te user troug te stages. From v4.0 (te latest version) most of te information for a single run of te pacage is contained on one screenful; te catc is tat a 17 (43cm) screen is required to be able to view all te appropriate buttons, and tis does not seem to be mentioned in te documentation(!). At any stage te input files must first be defined to GES, so tat tey must be selected even if tey already exist as SAS datasets, and oterwise imported to SAS; te import facility is built in to GES so tat tere is no need to exit and return. At te same time as a file is defined, te variables corresponding to certain ey definitions (strata, etc) are cosen. All te identifiers are intended to be text variables, and altoug numerics can be used in teir place in some (but not all) parts of te software, tey can t be cosen from lists of available variables unless tey are text. Tis is frequently frustrating were, for example, te stratum is identified by a number in a numeric field, wic must be converted to a string containing te number. Once GES is running it is also not possible to run any code from te program editor witout exiting GES (te only way to amend a dataset witout exiting is to use ASSIST). GES does contain facilities to generate input files in most of te cases wic one would use in practice, normally using a SAS By statement. GES maintains its previous settings and data files from run to run, wic can be convenient wen several similar surveys are to be analysed, or several alternative models are compared for te same dataset. It also as a good system of survey organisation; eac survey is individually labelled, and witin eac survey multiple periods can be eld, wit te files for eac period stored in an individually named directory (te same directory can be reused for several periods as long as file names are not duplicated). Tis maes it very easy to produce results for repeating surveys wen tey are using te same definitions and procedures. It also means tat by selecting a new survey te previous information for tat survey is available on te definition screen. Te SAS versions of output files are constructed to contain (meta-)information on te input files wic ave 19