Beyond 2011: Population Coverage Survey Field Test 2013 December 2013

Transcription

1 Beyond 2011 Beyond 2011: Population Coverage Survey Field Test 2013 December 2013 Background The Beyond 2011 Programme in the Office for National Statistics (ONS) is currently reviewing options for taking the census in future. Improvements in technology and in government data sources offer opportunities to either modernise the existing census process, or to develop an alternative census method that reuses existing data already held within government. Our research has resulted in two potential approaches for census taking in future Once a decade, like that conducted in 2011, but primarily online Using existing government data and compulsory annual surveys A consultation on the two approaches is being run from 23rd September to 13th December 2013 The consultation document describes these approaches, their strengths and weaknesses and the different types of information they could provide. We will publish our findings in About this paper This paper provides a summary of a Population Coverage Survey (PCS) field test run by the Programme in February 2013 to understand how address history information could help improve the quality of matching of coverage survey data to administrative sources. This field test was a voluntary postal exercise using a self-completion paper questionnaire, designed to collect basic demographic information similar to that collected by the 2011 Census and CCS. The field test surveyed 30,000 households across Lambeth, Coventry and Reading, chosen due to differences between population estimates and NHS Patient Register data. This document is one of a series of papers to be published over coming months. These will report our progress on researching and assessing the options, discuss our policies and methods and summarise what we find out about individual data sources. For more information Paper references (Paper M1, O1, R2 etc) used throughout refer to other papers published by Beyond 2011 all of which are available on the Beyond 2011 pages of the ONS website. Search Beyond or contact : beyond2011@ons.gov.uk Crown Copyright 2013 Beyond 2011: Population Coverage Survey Field Test 2013

2 Table of Contents 1 Executive Summary Introduction Purpose of a PCS Challenges in using administrative data to estimate the population Previous address history questions PCS February 2013 Field Test Collection of data Processing of data Initial research findings Response Address history and matching Conclusions and next steps Appendix A: PCS Questionnaire and Covering Letter References Beyond 2011: Population Coverage Survey Field Test

3 1 Executive Summary The census and existing administrative sources have one thing in common - they do not perfectly capture the number of people who are resident in England and Wales. Consequently we need a way to assess the quality and reliability of each of these sources to enable us to make adjustments when estimating the size of the population. This is the purpose of a coverage survey. A coverage survey will be required for either an online census once a decade, or a census using administrative data and surveys. We have undertaken research to develop the design of a coverage survey to support a census using administrative data to produce population estimates. We call this a Population Coverage Survey (PCS) to distinguish it from a Census Coverage Survey (CCS). Accurate, up-to-date address information on administrative data is, in most cases, dependent on the individual informing the service provider 1 of a change of address or circumstance. We know that not everyone does this immediately upon moving house; some people take a while to update their details and some may never change. Therefore, the PCS design may be different to that of a CCS supporting an online census once a decade. The 2011 Census and CCS included a question on usual address one year ago to provide information on migrant status in the previous year and produce migration estimates. In an administrative data based approach this question could help understand lags in administrative sources. However, we cannot be certain that just asking about address one year ago is sufficient to identify and adjust for these time lags. We ran a field test in February 2013 to understand how address history information could help improve the quality of matching of coverage survey data to administrative sources. This field test was a voluntary postal exercise using a self-completion paper questionnaire, designed to collect basic demographic information similar to that collected by the 2011 Census and CCS. It included questions asking respondents to provide up to four previous addresses over the past four years. This field test surveyed 30,000 households across Lambeth, Coventry and Reading, chosen due to differences between population estimates and NHS Patient Register (PR) data. The test achieved an overall response rate of 19.9% in line with similar voluntary tests undertaken by ONS in the past. PCS data was matched to the 2012 PR using the methods proposed in Paper M9. Indications are that a PCS asking for current address, any alternative current address and the most recent previous address could improve the quality of matching of a coverage survey to administrative sources. However, the bias in response to this voluntary test could mean that more historical address history information could be more useful for those people who have not responded. This field test has provided some evidence that address history information could be used to validate the matching methods proposed by Beyond 2011, where records have been matched using partial or no postcode information. This suggests that address history information could be used to provide on-going quality assurance for these methods. The benefits of address history information could also be greater for more mobile population subgroups including students and young people, who may be less likely to update their information on 1 Examples of service providers include NHS General Practitioners (GPs), Department for Work and Pensions (DWP) and Schools. Beyond 2011: Population Coverage Survey Field Test

4 administrative sources. Analysis of address history information could also help to further understand delays in updating information on administrative sources in future research. These findings will feed into further development and field testing of any CCS or PCS, and into further development of matching methods and their quality assurance. 2 Introduction 2.1 Purpose of a PCS The census and existing administrative sources have one thing in common - they do not perfectly capture the number of people who are resident in England and Wales. Consequently we need a way to assess the quality and reliability of each of these sources to enable us to make adjustments when estimating the size of the population. This is the purpose of a coverage survey. A coverage survey will be required for either an online census once a decade, or a census using administrative data and surveys. The CCS was an important element in the coverage assessment in the 2011 Census. ONS used the CCS to estimate the population missed by the census, and to adjust the census results for those estimated to have been missed or counted in the wrong location (ONS 2012). Research has been undertaken to develop the design of a coverage survey to support a census using administrative sources for producing population estimates. We call this a PCS to distinguish it from a CCS. This paper outlines the design and findings from the February 2013 field test to understand how address history information could help improve the quality of matching of coverage survey data to administrative sources. 2.2 Challenges in using administrative data to estimate the population There are two important statistical challenges in using administrative data to estimate the population; definitional differences and time lags. These challenges are discussed further in Paper M2. We define a population estimate as the number of men and women of different ages who are usually resident in an area, such as a local authority (LA), at a given time. We currently use the United Nations definition of usually resident, the place at which a person has lived continuously for at least the last 12 months, not including temporary absences for holidays or work assignments, or intends to live for at least 12 months (United Nations, 2008). Unlike a census once a decade, administrative data sources are collected primarily for operational rather than statistical purposes. There may be definitional differences between those used in the normal operations involving administrative sources and the statistical usually resident definition. This can result in: individuals who have moved abroad who are no longer usually resident in England and Wales, remaining on administrative sources short term migrants, who are outside of the scope of the usually resident definition, being recorded on administrative sources Beyond 2011: Population Coverage Survey Field Test

5 Delays in updating individual information, or systems updating can also result in: people who have died remaining on administrative sources (unrecorded deaths) people moving house can result in out-of-date address information, or registration in two places (unrecorded internal migration) Accurate, up-to-date address information on administrative data is, in most cases, dependent on the individual to inform the service provider 2 of a change of address or circumstance. We know that not everyone does this immediately upon moving house; some people take a while to update their details and some may never change. In order to produce population estimates using administrative data a PCS is required to adjust for those people who have, for example, moved house. Paper M9 sets out how a PCS would work with a Statistical Population Dataset 3 to produce population estimates. 2.3 Previous address history questions The 2011 Census and CCS included a question on usual address one year ago to provide information on migrant status in the previous year. This question was used to produce census estimates of internal migration within England and Wales, and from Northern Ireland and Scotland. It was also used to help count international migrants whose previous address was outside the United Kingdom. In a PCS supporting a census using administrative data, this type of question could help understand the time lags in individuals and systems updating address information. It could also help improve the quality of anonymously matching and linking administrative sources to a coverage survey, as there may be different addresses for the same individual in different sources. This can result in some records being matched (or not matched) incorrectly, or records assigned to an out of date geographic location. High quality matching is essential in the production of robust population estimates, as any matching error will impact on the level of error in the final population estimates. However, we cannot be certain that just asking about address one year ago is enough to ascertain and adjust for these time lags in administrative sources. This question has also historically been asked in the once a decade census to help in the production of internal migration estimates rather than understanding lags. In February 2013 we ran a PCS field test to understand how previous address information could help understand administrative sources and be used in our methods, including matching survey and administrative data and adjustment processes. This paper outlines the design and findings from this field test. 3 PCS February 2013 Field Test The PCS field test in February 2013 was a voluntary postal exercise using a self-completion paper questionnaire, similar to previous tests undertaken in preparation for the 2011 Census. Information 2 Examples of service providers include GPs, DWP and Schools. 3 Once individual records have been matched across data sources using the approach outlined in Paper M9, the information is pulled together into a single, coherent dataset that forms the basis for estimating the population. We call this a Statistical Population Dataset (SPD). Beyond 2011: Population Coverage Survey Field Test

6 about the PCS and the field test was provided to respondents on the questionnaire form and in a separate covering letter to the household (Appendix A). The PCS questionnaire was designed to collect basic demographic information similar to that collected by the 2011 Census and CCS. It included questions asking respondents to provide up to four previous addresses over the past four years. As these questions were asked of all persons living in the household, the PCS required a different questionnaire design to previous householdbased CCS questionnaires where only one person completed questions relating to everyone in the household. In the PCS, person-based and household-based questions were split into two sections. A reference date of 31 January 2013 was used in this field test, to represent a short time lapse between a reference date and the timing of survey field operations. The ONS Ethics Advisory Panel provided feedback on the proposed field test and this exercise was conducted in line with the ONS Respondent Charter commitments to respondents 4. The PCS questionnaire and covering letter was sent to 30,000 households across three local authorities: Coventry, Lambeth and Reading. These areas were selected on the basis that they experience high levels of migration which mean that there are differences between mid-year population estimates and PR data 5. The differences between PR and population estimates could arise from definitional differences or time lags. The aim was to maximise the proportion of records where people were likely to have incorrect geographic information. A likely response rate of 20% was assumed for this voluntary test, consistent with postal tests conducted in preparation for the 2011 Census. Assuming a 20% response rate, a sample size of 30,000 addresses was deemed large enough to detect whether the additional address information would improve the match rates. A simple random sample of 10,000 addresses in each of the three local authorities was selected and drawn from the Royal Mail Postcode Address File. 3.1 Collection of data The field period began in early February 2013 and respondents were provided with a pre-paid envelope to return their paper questionnaires. If respondents had any queries they were asked to contact the ONS Survey Enquiry Line or use the address provided. Small scale tests in preparation for the 2011 Census in 2006 and 2009 focused on maximising response amongst target population groups and testing delivery and follow up procedures. There was no follow up procedure included in this exercise as the focus was on the potential for address information to help improve matching, rather than maximising response. This aspect will be tested in any future field tests of a PCS. 3.2 Processing of data Both administrative data and PCS data will be subject to errors in data collection. The quality of PCS data used for matching to a SPD, and error arising from the matching process, can be affected by missing data through: 4 Further information about the Respondent Charter for Surveys of Households and Individuals is available on the ONS website annual mid-year population estimates were the most recent published when sample areas were selected, and are available on the ONS website Beyond 2011: Population Coverage Survey Field Test

7 respondents returning a blank form respondents not responding to an individual question processing error, such as incorrect data capture through scanning or keyed data entry, for example, errors in scanning, or typos or data entered into the incorrect field The questionnaires were scanned and the information captured from digital images using optical character recognition. Name and address information were keyed by experienced data entry staff exactly as written on the form. These data were then anonymised and processed for analysis as described in Beyond 2011: Matching Anonymised Data (Paper M9). This exercise matched PCS data to 2012 PR data (using different concatenations of name, sex, date of birth and geographic information) to determine whether address history information could improve the quality of matching to administrative sources. Using the matching approach proposed by Beyond 2011, PCS data was matched to 2012 PR data using current, alternative current and all previous addresses separately. If a match was made on current address then this would take precedence over matches on other previous or alternative addresses. 4 Initial research findings 4.1 Response The target response rate was 20% for this voluntary field test and a response rate of 19.9% was achieved. Response was around five percentage points lower in Lambeth (16.4%) than in Coventry or Reading (Table 1). This is consistent with lower survey responses in London compared with other areas in the 2011 CCS (ONS 2012), and in other social surveys, for example the Crime Survey for England and Wales (TNS 2012). Non-response bias occurs when the characteristics of people who respond to a survey are different to those who did not respond. Figure 1 compares the age and sex of PCS respondents with the 2011 mid-year population estimates. This shows that older age groups are over-represented (or were more likely to respond) in this sample, with people under 45 years of age under-represented (who were less likely to respond). This means that there is non-response bias in this sample and this must be considered when interpreting research findings. Internal migration statistics show that younger adults are more likely to move between local authorities than older age groups, with peaks at student ages (19 and 22 years; ONS 2013). This suggests address information is more likely to be up to date for older people, who are overrepresented in this field test. It may be more likely for matches to be made between the SPD and PCS for these older individuals, compared to other groups whose information on administrative sources may be out of date. The findings from this voluntary field test and matching exercise must therefore be interpreted with caution, as they are specific to this group of respondents and cannot be generalised to all population groups. Beyond 2011: Population Coverage Survey Field Test

8 Quinary Age Table 1 Response to the PCS February 2013 Field Test LA Number of responses % Coventry 2, Lambeth 1, Reading 2, Overall Response Rate 5, Source: ONS Population Coverage Survey Test 2013 Figure 1 Comparison of age-sex distribution of all PCS respondents with combined 2011 mid-year population estimates for Lambeth, Coventry and Reading Percent 0-4 PCS Females PCS Males MYE Estimate Sources: ONS Population Coverage Survey Test 2013; Annual Mid-year Population Estimates for England and Wales Mid 2011 NB this graph compares the shape of the proportional distributions of age groups of PCS respondents and mid-year population estimates, not the counts themselves. Beyond 2011: Population Coverage Survey Field Test

9 4.2 Address history and matching Over a quarter of PCS respondents provided some form of additional address information other than their current address (27.2%), with 13.6% of all PCS respondents providing only one previous address (Table 2). All PCS respondents have a current address, as this was the address which the questionnaire was sent to. When PCS responses were processed, geo-referencing software was used to attach geographic information to each of the addresses. Analysis of the type of information this software used to assign geographic information showed that the quality of address information reduced as addresses became older. This could be due to partial or incorrect address information, or addresses from outside the UK that are not recognised by the software. This suggests that people struggle to provide high quality address information the further back in time they try to remember. The quality of address information reduced as the number of previous addresses provided increased. Overall, 92.1% of PCS respondents were successfully matched to the 2012 PR (Table 3). Most matches were found on current address (91.5%) with most of additional matches in the sequential process found on most recent previous address, contributing 0.5 percentage points to the overall match rate. This suggests that for the type of people who responded to this voluntary field test, the most recent previous address was most useful in terms of improving the overall match rate. Address history information could also help validate matches made using partial postcodes (e.g. PO15 rather than PO15 5RR) or no postcode information. Matches were made on current address using partial postcode information (e.g. PO15) for 3.3% of PCS respondents that were matched to the 2012 PR (367 records). In 72% of the records matched using partial postcode information, an exact match on name, sex, date of birth and full postcode was made on a previous or alternative address (267 records). This suggests that in these cases the respondent is registered with their GP at a previous address and have not updated their information when they have moved. In these cases, address histories could provide information on the lags in administrative sources and be valuable in quality assurance of the matching methods in future. In the PCS records that could not be successfully matched to the PR, a higher proportion of respondents were: babies under 1 year old (who would not be present on the 2012 PR) respondents completing the intention to stay question (Person Question 5) indicating they are likely to be a migrant Unmatched PCS records also included those not registered with a GP. Further, if data is missing, it is more likely that a match will not be made as the information is incomplete. Further findings from this matching exercise are available in an internal technical report on request, subject to ONS Statistical Disclosure Control. Beyond 2011: Population Coverage Survey Field Test

10 Table 2: Address types provided by PCS respondents Type of address information PCS respondents providing address information Count % Current address only 8, Most recent previous address 1, Other previous addresses Alternative address Previous and alternative address Total PCS respondents (persons) 12, Source: ONS Population Coverage Survey Test 2013 Table 3: PCS match rates to 2012 PR by address type Type of address information PCS respondents matched to 2012 PR Count % Current address 11, Most recent previous address Other previous or alternative address Total PCS respondents matched (persons) 11, Unmatched PCS respondents (persons) Total PCS respondents (persons) 12, Source: ONS Population Coverage Survey Test 2013; 2012 NHS Patient Register Beyond 2011: Population Coverage Survey Field Test

11 5 Conclusions and next steps The findings from this initial analysis for the type of people responding to this voluntary field test indicate that: the most recent previous address was most useful in terms of improving the overall match rate the quality of address information reduces as it becomes more historic address history information could be used to validate matches made that use partial or no postcode information, and could provide on-going quality assurance for matching methods proposed by Beyond 2011 However, the bias in response to this voluntary test means that more historic addresses than the most recent address could be more useful for those who have not responded, or for more mobile population groups, such as students, who may be less likely to update their information on administrative sources. Analysis of address history information could also help to further understand delays in updating information on administrative sources. These findings will feed into further development and field testing of either a CCS or PCS, and into further development of matching methods and their quality assurance. Beyond 2011: Population Coverage Survey Field Test

12 Appendix A: PCS Questionnaire and Covering Letter Beyond 2011: Population Coverage Survey Field Test

13 Beyond 2011: Population Coverage Survey Field Test

14 Beyond 2011: Population Coverage Survey Field Test

15 Beyond 2011: Population Coverage Survey Field Test

16 Beyond 2011: Population Coverage Survey Field Test

17 Beyond 2011: Population Coverage Survey Field Test

18 Beyond 2011: Population Coverage Survey Field Test

19 Beyond 2011: Population Coverage Survey Field Test

20 Glossary Abbreviation CCS DWP GP LA ONS NHS PCS PR Meaning Census Coverage Survey Department for Work and Pensions NHS General Practitioner Local Authority Office for National Statistics National Health Service Population Coverage Survey 2012 NHS Patient Register Beyond 2011: Population Coverage Survey Field Test

21 References Beyond 2011 Exploring the Challenges of Using Administrative Data Methods & Policies Report (M2), ONS (2012). Available at: projects/beyond-2011/news/reports-and-publications/methods-and-policies/beyond exploring-the-challenges-of-using-administrative-data.pdf Beyond 2011 Producing Population Estimates Using Administrative Data Methods & Policies Report (M6), ONS (2013). Available at Beyond 2011 Producing Population Estimates Using Administrative Data: In Practice Methods & Policies Report (M7), ONS (2013). Available at: Beyond 2011 Producing Population Estimates Using Administrative Data: In Theory Methods & Policies Report (M8), ONS (2013). Available at: Beyond 2011 Matching anonymous data Methods & Policies Report (M9), ONS (2013). Published in July Available at: projects/beyond-2011/news/reports-and-publications/methods-and-policies/index.html Beyond 2011: Administrative Data Sources Report: NHS Patient Register ONS (S1) (2012). Published in November Available at do/programmes---projects/beyond-2011/news/reports-and-publications/sources/beyond administrative-data-sources-report--nhs-patient-register.pdf ONS (2013) Statistical bulletin: Internal migration by local authorities in England and Wales, Year Ending June 2012, published 26 June Available at wales/year-ending-june-2012/stb-internal-migration-for-england-and-wales--year-ending-june html#tab-Key-figures ONS (2012) 2011 Census Coverage Survey Summary, October Available at: TNS (2012) The 2011/12 Crime Survey for England and Wales Technical Report Volume One. Available at: United Nations (2008). Principles and Recommendations for Population and Housing Censuses. Statistical Papers, series M, No.67/Rev.2. New York. Available at Beyond 2011: Population Coverage Survey Field Test