The application of alternative modes of data collection in UK Government social surveys. A report for the Government Statistical Service

Transcription

1 The application of alternative modes of data collection in UK Government social surveys A report for the Government Statistical Service Peter Betts and Charles Lound Office for National Statistics May 2010

2 [blank page]

3 Contents 1 Purpose of the report Summary: Key findings Introduction...3 The Quality Improvement Fund Project...3 Context: GSS survey traditions Drivers of change Where we are now the current state of research into mode-related issues Emerging design options for UK social surveys Considerations of alternative/mixed mode designs References...14 Figure 1. Illustration of broad trade-offs between survey costs and quality for different design options...3 1

4 1 Purpose of the report 1 This report is to inform the Government Statistical Service (GSS) about a project which considered the potential for the application of alternative modes of data collection in UK Government social surveys. The report describes the background to the project, how it was conducted and the findings, and considers their implications. It is intended to provide information and guidance to assist the GSS in addressing some of the challenges faced in social survey data collection. The primary audience includes members of the GSS with responsibility for survey taking strategies within departments (particularly Heads of Profession), and survey managers and developers. The wider social survey research community, in the UK and internationally, may also find the report of use. 2 Summary: Key findings 2 The project has led to the conclusion that cost savings may be achievable in government social surveys, by maximising the use of cheaper modes of data collection (mail, internet and telephone) in mixed-mode data collection survey designs. However, it is important to be aware of the likely impact on quality. There are several important considerations, as follow. a Any fundamental change to survey design may lead to discontinuity in time series. To retain continuity requires complex statistical adjustment which can be difficult to implement and interpret and will need resources for experiments to measure the discontinuity. b The need to retain full population coverage and for response rates not to further decline argues against the introduction of mail, internet or telephone in single mode surveys. c Mixing data collection modes to maximise the use of cheaper modes is unlikely to increase response rates. Options designed to increase response rates would be more expensive. d Adequate control over respondent mode of response is required to minimise self-selection and nonresponse biases, so mixes of modes should either be sequential (from the cheaper to the more expensive) or employ subsampling, matrix sampling or dual frame sampling, and not offer sample survey respondents choice of mode at the initial recruitment stage. e f Surveys are likely to need to retain face-to-face interviewing for a proportion of the sample, in order to reach sample members without access to, or who do not respond by, other modes. Shortening or modularisation of questionnaires is required, meaning data requirements may have to be simplified. g Organisational commitment, capability and resources to develop, test and maintain new survey designs or strategies are required. h Capable field management and processing systems will need to be developed. i j Survey redesign strategies must consider future social and technological change, and initiatives such as ONS s Beyond 2011 project into the potential integration of survey, census and administrative data sources. Several areas require research, such as: the optimal mixes of modes with regard to costs and data quality (minimising various forms of bias and error); 2

5 mode effects (where the properties of a particular mode have an influence on the responses obtained), other causes of measurement error, and their separation from coverage, selection and nonresponse biases; the design of questions for multiple modes; and statistical adjustment methods. Experiments and field trials will require substantial resources. k The GSS will need to consider how to proceed from here, recognising that choices between cost and quality will need to be made, and that drivers of change and strategies for proceeding will vary across GSS departments and individual surveys. 3 Figure 1 below attempts to show schematically the complex variety of options, with the relative trade-offs between the cost of data collection and the dimensions quality relating to accuracy (precision and bias). It is not possible to quantify these parameters without experiments and trials. The illustration necessarily conceals much complexity and variability, for example relating to the population of interest and question type (e.g. sensitive questions may obtain better quality data by selfcompletion). Figure 1. Illustration of broad trade-offs between survey costs and quality for different design options High Cost Low Single mode: web Concurrent, respondent choice: selfcompletion only Concurrent, respondent choice: inc. interviewermode(s) Concurrent, control over mode: selfcompletion only Concurrent, control over mode: inc. interviewermode(s) Sequential from cheaper mode to more expensive Face-to-face single mode Sequential: face-to-face then followups Low Accuracy High 3 Introduction The Quality Improvement Fund Project 4 The project was funded by the UK Statistics Authority s Quality Improvement Fund (QIF), which helps the Government Statistical Service (GSS) address matters relating to statistical quality. The project was led jointly by ONS and the Home Office. In response to various drivers of change (see section 4) the project 3

6 considered whether it is feasible to implement alternative social survey data collection designs to those currently employed. 5 The project objectives were to review the current use of and research into alternative or mixed modes of data collection internationally, and to consider their application on GSS social surveys. The work conducted in late 2009 and early 2010 consisted of: a review of literature; consultation with National Statistical Institutes (NSIs) and other research organisations; and a two day workshop held at ONS, London in December 2009 involving academic/international expert researchers, and survey managers, researchers and methodologists from across the GSS, including the devolved governments in Wales, Scotland and Northern Ireland. 6 There are three outputs from the project: the detailed findings from the literature review and consultation exercise (Betts and Lound, 2010), which also includes further project background and context; an account of the workshop (see Betts, 2010); and this report, which consolidates and reflects on the findings from the review/consultation and the workshop. Context: GSS survey traditions 7 It is important to understand the context in which the project was conducted by summarising the GSS survey taking traditions, against which any future changes to data collection strategies or designs need to be assessed. 8 Social surveys are commissioned by many GSS departments and conducted by ONS s Social Survey Division or other survey organisations. Surveys are crosssectional (conducted on a continuous or ad hoc basis) or, increasingly, longitudinal (with periodic wave designs or follow-up surveys). 9 Note that this project did not consider the Population Census or business surveys, which have their own considerations. Modes of data collection 10 The primary data collection mode is face-to-face computer-assisted personal interviewing (CAPI). Some computer-assisted telephone interviewing (CATI) is conducted. Paper and pencil interviewing (PAPI) and paper self-completion surveys are little employed. Self-administration is used mainly in the context of otherwise interviewer-administered surveys such as self-completion modules for sensitive topics, using computer-assisted self-interviewing (CASI), and record keeping paper diaries of expenditure, nutrition and travel. Internet data collection has not been used much, with the exception of small scale tests or pilots. Coverage and sampling 11 To produce nationally representative statistics, surveys i) (mostly) cover the household population and ii) employ probability samples. The sampling frame 4

7 used for national household surveys is the Royal Mail s Postcode Address File (PAF), from which addresses are selected. Alternative sources are not as comprehensive. 12 Most GSS surveys interview at the household level and attempt to interview all adults (aged 16 and over) preferably in person, or by proxy. There are exceptions when one or more members of the household are sampled to respond. 4 Drivers of change 13 The basic survey model as set out above has worked well over several decades. It has developed to include computer-assisted methods, harmonisation and integration, and efficient targeting of samples. However, there are strong reasons for considering alternative survey designs, as follow. The rising cost of traditional data collection methods. Increasing numbers of visits by interviewers to sampled addresses are required to make contact and gain cooperation, and interviews have become longer (a typical interview might take an hour, though some are shorter or longer) and more complex. Reduced departmental budgets and efficiency savings required across government. Declining response rates on many of the major surveys. Societal changes have made people harder to contact or less willing to participate when contacted. GSS departments and survey contractors have made efforts to slow or reverse the decline with varying success. Technological advancements mean a greater variety of survey methods are available and continue to develop. Newer methods include computer-assisted web interviewing (CAWI), CASI, audio-casi, interactive voice recognition (IVR), telephone data entry (TDE). Portable computing and telecommunications devices with internet access (e.g. tablet computers, smart phones) are becoming widespread. 14 Meanwhile there is continuing need to meet the requirements of survey sponsors, other data users and policy makers in terms of quality in its different dimensions (accuracy, timeliness, relevance and so on). In the project we sought to investigate whether alternative modes may make it easier to make contact, gain cooperation and enable busy or reluctant people to respond by a means more convenient to them, and provide greater value for money. We also explored associated challenges with using alternative modes relating to nonobservation, representation, measurement differences and security. 5 Where we are now the current state of research into mode-related issues 15 From the research conducted on the project we have gained an understanding of the broad range of issues relating to the application of alternative modes. 5

8 Coverage, sampling, estimation 16 A key challenge for internet and telephone data collection relates to nonobservation and representation: coverage bias, lack of adequate sampling frames, selection effects and nonresponse bias. There is a digital divide between the characteristics of households with internet access (70% of UK households in 2009) and those without. Seven per cent of UK households are without a landline telephone; mobile-only households are biased towards younger people; and people are increasingly unlikely to include their number in directories. Probability sampling must be used in official statistics to avoid self-selection bias, but there is no suitable sampling frame for internet users or telephone numbers (landline or mobile). 17 However there is scope for using internet and telephone modes within a probability sampling framework and mixed-mode survey system (for example, drawing addresses from the PAF then posting a letter containing information about the survey and how to log-in to a website to respond or to request a paper questionnaire or a telephone interview). There may be potential in Random Digit Dialling (RDD) for telephone sampling. Options for mixing modes 18 There are two main options for mixing modes on a survey. The first option is to mix the mode of communication with respondents (e.g. advance and reminder letters) with a single data collection mode, resulting in better coverage and/or response rates and with no mode effects. The second option is to (additionally) mix data collection modes with potentially greater coverage, higher response rates and lower costs; but with potential drawbacks of mode effects and confounds between the effects of mode, selection bias, nonresponse biases and time. The latter option has implications for identifying measurement differences and methods to make statistical adjustments. Mixed mode data collection can be concurrent (e.g. offering respondents a choice of mode), or sequential (e.g. progressing from the cheapest to the most expensive). It should be noted that a single mode of data collection may contain bias. Response rates and nonresponse bias by different modes and systems 19 Key findings regarding response rates and nonresponse bias in single modes and in mixed-mode survey systems included the following. Response rates in single mode surveys are substantially lower by internet than mail or telephone interviewing and lower by mail and telephone than face-toface interviewing. Some differences in respondent characteristics have been observed between face-to-face, telephone, internet and mail responders. Mixed-mode design may not increase the overall response rate. The literature covers a complex variety of concurrent and sequential designs. In concurrent designs offering choice of mode respondents tend to choose other modes in preference to the internet. Using a concurrent design may even decrease response rates. Sequential mixed mode designs may not increase the overall response rate, but can obtain a large proportion of response by cheaper modes. Each additional mode can chip away at nonresponse at extra cost. 6

9 Web follow-up surveys to interviewer surveys result in significant accumulated sample loss (where agreement to follow-up has not been sought at the initial interview). Interview length is asserted to affect response rates (including agreement to be reinterviewed in longitudinal surveys). The typical length of face-to-face surveys may be a barrier to moving them to other modes, but more needs to be known about sensitivity to length in different modes. Capturing data from all individuals in a household may be difficult when using mail or internet modes. One person may respond by proxy on behalf of other members, with potential for item nonresponse, inaccurate responses, or there is the possibility of nonresponse from some individuals. Furthermore, there is less control over who within a household is providing responses. Mode effects and other causes of measurement differences 20 Mode effects on measurement error are when properties of the mode have an effect on the responses collected. Observed differences between (sub)samples may result from mode of collection rather than being real differences. Such effects are a challenge for mixed mode data collection. The interactions between features of modes, question types and designs, and a respondent s characteristics, cognitive ability and motivation are complex, involving several factors co-varying with mode. 21 Mode effects include satisficing (response processing is not thorough or stages are skipped) which is greater in self-administration than in interviewer administered mode; greater in telephone interviewing than in face-to-face interviewing; and greater in internet mode than postal mode. However, evidence for satisficing is not overwhelming and sometimes inconsistent. The second main type of mode effect is social desirability bias, where the true answer is not reported by the respondent in order to portray him/herself favourably or to conform to social norms (for example, in respect of sensitive behaviour, such as smoking or drinking, or opinions). It is higher in interviewer-administered than self-administered modes; though evidence is mixed when comparing telephone and face-to-face. However there is debate relating to: whether social desirability is a characteristic of question or a personality trait; what the desired values/norms are; and whether it is subject to conscious control or relates to response process shortcutting, recall error or self-deception. 22 Measurement differences between modes cannot always be attributed to mode effects. Mode effects may be confounded with sampling, selection into mode and nonresponse biases; or with questionnaire construction and administration, which can change the stimulus. The latter includes the tendency to construct questions differently in different modes; question context and order effects; recall and memory effects; interviewer practices differing between face-to-face and telephone interviewers; and the visual design of self-completion questionnaires. 23 There exists a dichotomy between interviewer modes and self-completion, but no mode is superior all round. The implications of the findings are that optimum design has to weigh up the advantages/disadvantages of mixing modes: lower costs, better coverage and higher response rates (than those from telephone, internet or mail modes individually); against 7

10 measurement error compounded with differential coverage or nonresponse error. Approaches to mixed mode questionnaire design 24 The above implications raise the question of whether instruments which are insensitive to mode can be developed. There are various approaches to designing question(naire)s for mixed mode data collection, in recognition of there being no single solution and that there are different ways of combining modes and reasons for doing so. They include: Unified mode construction (unimode): questions are written and presented the same (or nearly) with the intention of giving common mental stimulus. Mode-specific construction: structure, wording, presentation of questions are modified to suit particular mode capabilities, to achieve the same stimulus. Mode-enhancement construction: when one mode is primary, others auxiliary (and equivalence is less important), mode-specific features are used to improve quality in the primary mode. 25 Variants on these approaches include universal presentation and generalised mode design. There remain uncertainties: further research is needed into what constitutes equivalent stimulus in different modes; it may be an illusory goal. A pragmatic approach may be needed, designing on a question by question basis, adjusted for the survey context and needs. 26 As there is little experience of internet data collection in GSS surveys, specific design aspects which were identified in the literature review should be taken into consideration if it is adopted as part of the future data collection strategy. Weighting and statistical adjustment 27 Weighting and statistical adjustment for biases undercoverage, nonresponse, systematic mode effects will be required. Possible approaches are design-based (traditional random sampling) and model-based (opt-in/volunteer samples where probabilities of inclusion are modelled). 28 Assessing the magnitude, direction and impact of mode effects were identified in the literature as challenges. Some techniques to deal with them were proposed: controlling for respondent characteristics, creation of adjustment factors and regression analysis. The magnitude of differences needs to be reported. Significant differences between estimates may not matter; what is important is the effect on substantive interpretation, for example, of the relationship between variables. Assessing which mode obtains better quality data can be done by internal and external validation and with prior knowledge of the expected direction (for example, a higher reported rate of alcohol consumption might be expected). Survey management 29 Other matters relating to the use of alternative or mixed modes which were identified by the project included survey management, infrastructure and logistics. The implications for the GSS of aspects such as costs, field management systems, data processing, delivery timetables and technical issues are covered in section 7. 8

11 Further research agenda and experimental design 30 The literature is clear that much is still unknown about the effect of applying alternative or mixed mode designs. Several areas for further research are identified in the literature review (Betts and Lound, 2010). Good practice in experimental design should be followed, to avoid confounds of the kind mentioned previously. Alternative/mixed mode data collection research and development in other research organisations 31 The experiences of other organisations provide useful guidance and lessons from which the GSS can learn. Several National Statistical Institutes and other research organisations have been through, or are currently in, the process of developing new strategies, systems and guidelines for mixed mode data collection, and/or are conducting experimental research. Some which are of particular interest follow, although findings may vary as to how generalisable they are to the UK context. Statistics Netherlands ongoing programme of modernisation of household surveys in which mixed mode designs are being implemented. This includes a phased redesign of their Labour Force Survey (LFS) to ultimately collect as much Wave 1 data as possible by internet, followed in sequence by telephone and face to face. Statistics Canada have several initiatives underway, relating to development of: best practice in mixed mode surveys and a framework for data collection research; an e-collection Service for all of its social and business surveys and the Census; and a Labour Force Survey Internet Response Option. The US Census Bureau has developed Guidelines for Designing Questionnaires for Administration in Different Modes and has various crossorganisational groups looking at mode-related issues. An Economic and Social Research Council (ESRC) Survey Design and Measurement Initiative (SDMI) project is being conducted by the National Centre for Social Research (NatCen) and the Institute for Social and Economic Research (ISER) at the University of Essex. The objectives are: to provide researchers with practical advice as to which modes are likely to produce comparable responses and which question types are susceptible to mode effects; and to develop design principles to improve mode portability. The cross-national, comparative European Social Survey has been conducting a programme to investigate the feasibility of changing to a mixed mode data collection strategy. The Understanding Society survey (the UK Household Longitudinal Study), conducted by ISER for the Economic and Social Research Council (ESRC), includes an Innovation Panel designed to enable methodological research. The Wave 2 Innovation Panel was conducted using a mixed mode experimental design. 6 Emerging design options for UK social surveys 32 A number of key high-level themes emerged from the workshop, literature review and consultation. Potential options for using alternative or mixed-mode designs on GSS social surveys included the following. 9

12 Maximise the use of cheaper modes (mail, internet and telephone), in mixedmode data collection designs, to reduce the proportion of face-to-face interviews so lowering costs. Use sequential mixed mode data collection designs in cross-sectional surveys, or over the waves of panel surveys, or for follow-up surveys. Single-mode mail, internet and telephone surveys are not recommended as an alternative to face-to-face surveys due to inadequate response rates and nonresponse bias, and, for internet and telephone modes, coverage and selection biases. Use dual frame sampling, subsampling and matrix sampling in a concurrent mixed mode design to obtain a proportion of response by cheaper modes and/or cut down the number of face-to-face interviews conducted. Some control is maintained over mode of response, rather than allowing respondents to selfselect which may introduce bias. Redesign questionnaires: o use short form (containing core variables) and long form versions (core and additional variables); o modularise the design: including one-off modules, rotating modules; breaking a survey into modules administered over several periods rather than in a single interview; o revisit the concept of a modular Integrated Household Survey. Also maximise the use of cheaper modes in respondent communications (prenotification, reminders, follow-ups of nonresponders, keeping in touch between waves), combined with data collection in mixed mode survey systems. Survey paradata and other information may be used to enable targeting of particular modes, for example, in areas with lower rates of access to broadband internet, mail might be used. 7 Considerations of alternative/mixed mode designs 33 There are a number of considerations to be weighed up when making strategic decisions about future data collection designs for GSS surveys. They include the benefits and drawbacks relating to the potential approaches outlined above, and some overarching considerations. Survey costs 34 The primary benefit of alternative approaches to data collection appears to lie in reduced costs due to obtaining a larger proportion of response via cheaper modes. There are, however, other costs, relating to: systems design and set-up; printing and postage; systems maintenance and technical support; and data processing and editing. Little is known about cost models and optimal strategies for mixed mode designs, and how much money may be saved. Cost-benefit analysis based on experimental evidence will be needed. Savings may only be realised in the medium to long term. Coverage, sampling and estimation 35 The issues of coverage, sampling and selection bias mean the internet mode can only be used as part of a mixed mode design. Internet cannot be used without making initial contact via another mode (e.g. by post as per current practice on many surveys). Similarly more use may be made of telephone mode, for example 10

13 by inviting respondents to request a telephone interview, by following up a faceto-face interview using telephone, or by using RDD. 36 Response rates may not be increased by mixing modes to maximise the use of cheaper modes. They may even be reduced if concurrent design offering respondents a choice of response mode is used. 37 The sensitivity of interview length in different modes and its effect on response rates and other aspects of data quality require further research. 38 The requirement on many GSS surveys to obtain response from all members of a household means that using mail or internet modes has potential implications: more data being collected by proxy, or more partial data (item nonresponse or data missing for individuals within the household if partial household response is allowed) or higher unit nonresponse (if partial household response is not allowed). 39 If modular panel designs are used there is risk of attrition before all required data has been collected from a respondent or household. Measurement error and data comparability 40 Mode effects and other forms of measurement error may have an effect on the overall estimates of a survey with mixed modes of data collection, according to the proportions of cases collected in different modes. 41 Mode effects may also affect comparisons over time from data collected by different modes: observed changes may be real or may be due to mode effects. Such comparisons are made, for example, in longitudinal analysis on the LFS where Wave 1 is primarily face-to-face and subsequent waves are largely collected by telephone. 42 Mode effects may also affect the comparability of data from different survey sources. Comparability between sources is an issue in so far as it is intended that GSS departments use harmonised questions in surveys, for variables such as socio-demographic characteristics, economic activity, income and receipt of benefits and tax credits. 43 There will always be some form of measurement error in any mode. Mode effects may or may not be large or affect substantive conclusions. Identification and separation of coverage/selection bias, nonresponse bias, mode effects and other causes of measurement differences, in order to make statistical adjustments, is difficult and requires costly experiments. 44 These issues may need to be considered separately where the focus is measuring change over time, as opposed to measuring levels cross-sectionally, as consistent biases may cancel each other out. Shorter questionnaires and modularisation 45 The use of shorter questionnaires or shorter versions of forms, for example for administration by mail or internet, may be limited unless data requirements can be cut or simplified. 11

14 Continuity 46 The potential effect of survey redesign on the continuity of time series is a risk. Careful management of the transition and clear communications between data users and survey conductors is required, with regard to needs and expectations with regard to maintaining quality and deciding where to compromise. 47 There is a need to understand the potential impact of survey design changes by conducting well-designed research and testing. A break in a time series can be estimated by embedded experiment, parallel run or time series methods to evaluate change. The estimated break can be used to backcast or forecast results. Survey management 48 To conduct mixed mode surveys there is a need for effective fieldwork management and processing systems, with capacity to deal with multiple response modes, which do not currently exist. Bespoke systems may need to be developed. 49 Maintenance of multiple versions of a questionnaire for use in different modes may become complex, particularly as questionnaires change on continuous or repeated surveys. Flexibility and agility will be required. 50 When designing internet data collection consideration must be given to respondents internet connection speeds, browsers, computer operating systems and screen settings, to minimise the risk of incompatibility, break off before completion, and to the likelihood of respondent keying error when logging on to websites. 51 Using face-to-face interviewers mainly to deal with non-responders to cheaper modes, who may be more difficult to contact and persuade to respond, may have implications for survey management: a requirement for additional interviewer training, or higher interviewer turnover. 52 Furthermore, the cases from which it is difficult to gain response may be more scattered or unpredictably located, although it is possible they may cluster in, for example, urban areas. This would be likely to increase the cost per face-to-face interview. 53 Sequential mixed mode fieldwork may take longer than using a single mode, once the process of prenotification, delivery of the instrument, reminders, reallocation to a different mode and follow-ups of nonresponders, perhaps more than once, has been conducted. 54 Questionnaires must be less complex in self-completion modes. In paper selfcompletion there can be no automated routing of questions or checking of invalid or inconsistent responses. Randomisation of questions, modules or response category order is possible but complex to administrate. In internet mode, routing and randomisation are possible, but less sophisticated online checking is possible than in interviewer-administered modes. Processing in-house, including data entry, validation and coding will be required, creating another potential source of error and extending the survey timetable. 12

15 55 Survey designs which extend fieldwork and/or processing time may impact on delivery timetables, which are often required by users of GSS survey data to be shortly after the field reference period (relating to the timeliness dimension of quality). Reference periods for questions may need to be changed to allow subsamples from different modes to be combined to form a representative sample. Alternatively, provisional estimates may be produced from a subset of cases with final estimates later once all cases have been processed. Future proofing of design changes 56 Major design changes need to be future proof, that is sufficiently robust to be durable and not launched without consideration of potential social and technological changes, both positive and negative. Such changes relate to issues of coverage, sampling and representation (for example, wider but still not universal - internet access; changing patterns of mobile and landline telephone ownership; and reduced public availability of telephone numbers) and to measurement (for example, developing computer and telephone technology; mobile internet access; portability; interactivity; visual design). There is a balance to strike between early adoption of new developments and waiting for evidence of their impact and longevity. 57 Additionally, the potential development of a central population database or greater use of administrative data to replace or be linked with survey and census data, as being addressed by ONS s Beyond 2011 project, should be considered. The cost of data collection may need to take account of the overall cost of surveys and census taking. 58 Pressure to offer alternative survey modes may come from expectations that interaction between the public and government should be possible via the internet and minimise the burden on people. It is important to recognise that in the survey context it is the respondent who is providing the service to government, by responding, rather than the government providing a service to the public. Surveys with additional elements 59 Surveys with elements in addition to an interview such as record keeping diaries, health examinations, physical measurements and physical surveys (and which may involve a visit by a professional such as a surveyor, dentist or nurses as well as an interviewer) - have additional considerations with regard to use of cheaper modes. Research could be conducted into the feasibility of online diaries and technological developments which might aid collection of such data (some work has already been done, for example into use of GPS devices for the National Travel Survey, or is planned, such as use of digital pens on the International Passenger Survey). Ethical and legal implications 60 There are some ethical and legal issues relating to the use of alternative modes of data collection. Differential respondent access to modes, due to, for example, the language spoken/read or disability should be taken into consideration. Those unable to respond in one mode should be given an opportunity to respond in a more accessible mode. If using mail or internet modes in longitudinal surveys, 13

16 dependent interviewing may not be possible since the lack of control over who in a household is responding may compromise within-household confidentiality. Use of administrative data has ethical considerations (for example seeking respondent permission to link survey data with administrative records) and legislative considerations (for example, conforming with the Statistics and Registration Act, and to data protection legislation). Consideration might be given to introducing legislation making participation in some surveys mandatory, if the burden was reasonable and justifiable, for example, by reducing the number of questions or facilitating respondent response mode preferences. Implications of redesign for individual departments and across the GSS 61 Organisational commitment, capability and resources to develop, implement and maintain new survey designs or strategies are important. Investment of financial resources and technical expertise will be required in the development stages (for example to conduct research into the areas identified previously, and development of systems) and possibly subsequently (such as embedding small scale experiments to identify various types of error and to develop methods for their reduction and statistical adjustment). 62 It should be recognised that often there are no easy or pain free solutions. The business case for change will need to be carefully considered, balancing investment cost, financial savings and other benefits as well as data quality and scope. 63 Drivers of change may vary across GSS departments, with their differing financial situations and levels of concern over response rates or other aspects of data quality. Departments may need to weigh up the pros and cons of piecemeal versus wholesale change. Different approaches may be appropriate for ONS, which conducts numerous surveys in house (and its need to take account of the requirements of the Census and the potential synergies with Beyond 2011), and for other departments, which individually sponsor fewer surveys and who need to consider the capabilities of survey contractors. Within ONS considerations and solutions may vary from survey to survey. At the same time, however, there is a need to consider cross-gss harmonisation of questions and outputs and comparative research. 8 References Betts P (2010) The application of alternative modes of data collection in UK Government social surveys: Report on a workshop held at the Office for National Statistics, 8-9 December 2009, Office for National Statistics. Available at: Betts P and Lound C (2010) The application of alternative modes of data collection in UK Government social surveys: Review of literature and consultation with National Statistical Institutes, Office for National Statistics. Available at: 14