Data Integration Manual

Transcription

1

2 Acknowledgement This report was prepared by Statistics New Zealand s Statistical Methods team and produced by the Product Development and Publishing unit. Further information For further information on the statistics in this report, or on other reports or products, contact Statistics New Zealand s Information Centre. Visit our website: or us at: info@stats.govt.nz or phone toll free: Auckland Wellington Christchurch Private Bag PO Box 2922 Private Bag 4741 Phone Phone Phone Fax Fax Fax Information Centre Your gateway to Statistics New Zealand Statistics New Zealand collects more than 60 million pieces of information each year. New Zealanders tell us how and where they live and about their work, spending and recreation. We also collect a complete picture of business in New Zealand. This valuable resource is yours to use. But with all the sophisticated options available, finding exactly what you need can sometimes be a problem. Giving you the answers Our customer services staff can provide the answers. They are the people who know what information is available and how it can be used to your best advantage. Think of them as your guides to Statistics New Zealand. They operate a free enquiry service where answers can be quickly provided from published material. More extensive answers and customised solutions will incur costs, but we always give you a free, no-obligation quote before going ahead. Liability statement Statistics New Zealand gives no warranty that the information or data supplied in this report is error free. All care and diligence has been used, however, in processing, analysing and extracting information. Statistics New Zealand will not be liable for any loss or damage suffered by customers consequent upon the use, directly or indirectly, of information in this report. Reproduction of material Any table or other material published in this report may be reproduced and published without further licence, provided that it does not purport to be published under government authority and that acknowledgement is made of this source. Published in August 2006 by Statistics New Zealand PO Box 2922 Wellington Crown Copyright ISBN

3 Contents Preface... vi Abbreviations... vii 1 Introduction to Data Integration Introduction What is data integration? Levels of data integration The role of data integration Why integrate? Legal and policy considerations Some key data integration concepts Integration for statistical and administrative purposes Exact linkage and probabilistic linkage Quality assessment Data integration scenarios Data integration at Statistics NZ and elsewhere The emergence of data integration Data integration at Statistics NZ Key steps in a data integration project Legal and Policy Considerations Introduction The Statistics Act The Privacy Act General information Use of unique identifiers The Statistics NZ Data Integration Policy Codes of practice Data integration business case Privacy impact assessment Consultation with other agencies The Statistics NZ Confidentiality Protocol The Statistics NZ Microdata Access Protocols Operational Aspects of a Statistics NZ Data Integration Project Early stages of a data integration project Other relationships...17 iii

4 3.3 Obtainment and safe keeping of external data Data extract Data transfer, storage, security and internal access controls Documentation and quality assurance Documenting record linkage methodology Reviewing record linkage methodology Supporting documentation IT considerations Preparing for Record Linkage Introduction Gathering information about source data Preliminary investigation of source data Target population and units Identification of population Identification of units Understanding the source data metadata Implications from the metadata Procedure for obtaining data Request for supply of data Data transfer Data verification Feedback to provider Preparing data for record linkage Typical errors in linking variables Standardisation: editing, parsing, formatting, concordance Editing Parsing and standardisation of linking variables Concordances Deduplication Anonymisation of unique identifiers Statistical Theory of Record Linkage Introduction Exact matching Terminology Matching files The human approach The mathematical approach...38 iv

5 5.6.1 The m probability The u probability The field weight The composite weight Example Changing the m and u probabilities Weights Distribution Cut-off thresholds Clerical review Blocking Passes Record Linkage in Practice Types of matching Pre-matching process Deduplication A data integration process flow Standardised datasets Matching method Choice of blocking variables Choice of linking variables Commonly used comparison functions for linking variables The m and u probabilities Quality assessment of linked data Setting the cut-off threshold False positives, false negatives and match rates Measurement error in integration Adding data over time...60 Appendix: Statistics New Zealand s Uses of Data Integration...61 Glossary...62 Bibliography...66 v

6 Preface The Data Integration Manual provides a guide to data integration as carried out at Statistics New Zealand. The manual was written by Statistics NZ staff, following involvement in several large interagency data integration projects. The aim of the manual is to provide a guide to best practice and to share the insights gained from Statistics NZ s experience. We hope the manual will assist agencies collaborating with Statistics NZ, and others interested in data integration, to understand the basic concepts, theory and processes involved in data integration, as well as providing practical advice. The manual begins with an introduction to data integration that describes what data integration is and why data integration is carried out, and outlines the key steps involved. Chapter 2 introduces the legal environment and Statistics NZ policy on data integration. Chapter 3 describes operational aspects of Statistics NZ data integration projects. The remaining chapters focus on technical aspects of the linkage itself: the preparation of data needed before record linkage can be undertaken, the statistical theory of record linkage and the practical implementation of record linkage techniques. vi

7 Abbreviations ACC Accident Compensation Corporation CD compact disc CURF confidentialised unit record file DSW Department of Social Welfare DVD digital video disc FAQs frequently asked questions IRD Inland Revenue Department IT information technology IUID internally assigned unique identifier LEED Linked Employer-Employee Data MOU Memorandum of Understanding NHI National Health Index NMDS National Minimum Dataset NYSIIS New York State Identification and Intelligence Algorithm NZHIS New Zealand Health Information Services OPC Office of the Privacy Commissioner PGP Pretty Good Privacy PIA privacy impact assessment SLA Service Level Agreement UID unique identifier vii

8

9 1 Introduction to Data Integration Summary This chapter provides an introduction to data integration, describes why data integration is carried out, and presents a brief history of data integration at Statistics New Zealand. 1.1 Introduction What is data integration? Data integration is defined broadly as the combination of data from different sources about the same or a similar individual or unit. This definition includes linkages between survey and administrative data, as well as between data from two or more administrative sources. An alternative application of data integration theory is in identifying records on a single file that belong to the same individual or unit. Other terms used to describe the process of data integration include record linkage and data matching Levels of data integration When integration occurs at the micro level, information on one individual (unit) can be linked to: (i) a different set of information on the same person (unit) (ii) information on an individual (unit) with the same characteristics. At the macro level, collective statistics on a group of people or a region can be compared and used together. The main focus of this manual is micro-level data integration of type (i) that is, linkage of records that are likely to belong to the same individual or unit The role of data integration The role of data integration in helping to produce an effective official statistical system is becoming increasingly apparent. The process of bringing together information from different sources paves the way for a broader range of questions to be answered. Through integration it becomes possible to examine underlying relationships between various aspects of society, thus improving our knowledge and understanding about a particular subject Why integrate? Linking administrative data from different sectors creates a valuable source of information for statistical and research purposes because relationships that previously could not have been considered can be examined. There may sometimes be other possible methods of investigating relationships of particular interest, for example conducting a survey. However, data integration can offer a less time consuming and less costly alternative, although it does still require a significant level of time and resource. Data integration also has the advantage of reducing respondent burden by making more effective use of existing data sources. 1

10 1.1.5 Legal and policy considerations Data integration raises a range of legal and policy issues, some of which can be complex to resolve. These are discussed in more detail in Chapter Some key data integration concepts Integration for statistical and administrative purposes When data is linked for statistical purposes, individuals (or units) are identified only to enable the link to be made. When the linkage is complete, the identity of the individual (or unit) is no longer of any statistical interest. The linked dataset is used to report statistical findings about the population or sub-populations. In contrast, when data is linked for administrative purposes, individuals are identified not only to enable the link to be made, but also for administrative use subsequent to the linkage. This may sometimes result in adverse action, such as prosecution, being taken against individuals. Statistics NZ undertakes data integration only for statistical purposes Exact linkage and probabilistic linkage There are two key methods for matching records. Exact linkage involves using a unique identifier (for example a tax number, passport number or driver s license number) that is present on both files to link records. It is the easiest and most efficient way to link datasets, and standard statistical software such as SAS can be used. Where a unique identifier is not available, or is not of sufficient quality or coverage to be relied on alone, probabilistic linkage 1 is employed. This involves the use of other variables common to both files (for example names, addresses, date of birth and sex). Probabilistic linking is more complex and sophisticated data integration software is required in order to achieve high-quality results Quality assessment Either linking method can result in two types of errors: false positive matches and false negative matches. A false positive match is where two records are linked together, when in reality they are not the same person or unit. A false negative match is where two records are not linked together, when they do in fact belong to the same person or unit. Generally there is a trade-off between the two types of errors since, for example, reducing the rate of false positives may increase the rate of false negatives. Thus it is important to consider the consequences of each type of error and to determine whether one is more critical than the other. 1 There are a range of terms used to describe types of linkage, including probabilistic, statistical, stochastic, and demographic. In the literature, different authors use these to communicate different concepts. Sometimes they are used interchangeably within a paper. The term probabilistic as defined above is used consistently throughout this manual. 2

11 An assessment of the size of each of these sources of linkage error should be undertaken as part of the integration and results made available. Analysis of an integrated dataset should take into account possible impacts of the linkage error. Further details on quality assessment can be found in section 6.4, below Data integration scenarios There are different ways in which two datasets being integrated relate to each other. Situation 1 Situation 2 Situation 3 A B A A B B Situation 1 This is where every individual on dataset A is also on dataset B and vice versa. For example, dataset A might consist of addresses while dataset B contains rates information for each address. Situation 2 This is where every individual on dataset B is on dataset A but there are individuals who appear on dataset A who are not on dataset B. For example, dataset A could be student enrolments and dataset B could be information for those students who have student loans. Situation 3 This is where some individuals appear on both dataset A and dataset B. However, other individuals will appear on only one dataset or the other. For example, dataset A might be Accident Compensation Corporation (ACC) clients, while dataset B could be people who are admitted to hospital. It should be noted that these are theoretical relationships between pairs of files. Real life is rarely that perfect. For example, in situation 1 there could be duplication and omissions within the files, and timing differences between the two files, which mean that they do not have 100 percent overlap. There are also different desired results from a pair of integrated datasets: the union or the intersection. A B Intersection A B Union 3

12 For example, if dataset A was ACC claims and dataset B was hospitalisations for injury, the intersection would be of interest if statistics were wanted on the number of ACC claimants admitted to hospital as a result of their injury. The union would be of interest if statistics were wanted on the total number of injuries, without double counting injuries represented in both datasets. Sometimes a different combination of records may be required for example, all records on B, with information added from A if it is available. A B All records on B Continuing the previous example, this combination may be of interest if hospitalisation costs were to be combined with costs to ACC, for the population of ACC claims. 1.3 Data integration at Statistics NZ and elsewhere The emergence of data integration The most significant early contributions to record linkage came in the 1950s, in the field of medical research. 2 Two of the most influential early papers were by Newcombe et al 3 and Fellegi and Sunter Data integration at Statistics NZ Data integration has been used in a variety of ways at Statistics NZ, beginning in the 1990s, and gaining momentum in recent years. The earliest uses were mostly Statistics NZ driven, but more recent momentum has come from external interest in integrating datasets collected from different agencies for unrelated purposes. In 1997, the Government directed that where datasets are integrated across agencies from information collected for unrelated purposes, Statistics NZ should be custodian of these datasets in order to ensure public confidence in the protection of individual records. 5 In the same Cabinet meeting, it was agreed that Statistics NZ should carry out a feasibility study 2 Gill L (2001). Methods for Automatic Record Matching and Linkage and their use in National Statistics, National Statistics Methodological Series No 25, National Statistics, United Kingdom. 3 4 Newcombe H, Kennedy J, Axford S, and James A (1959). Automatic Linkage of Vital Records, Science 130, Fellegi I and Sunter A (1969). A theory of record linkage, Journal of the American Statistical Association 64, Cabinet meeting minutes CAB (1997) M 31/14 [electronic copy unavailable]. 4

13 into the costs and benefits of integrating cross-sectoral administrative data to produce new social statistics. This feasibility study was subsequently carried out, including a trial integration of Inland Revenue Department (IRD) income information with beneficiary data from the Department of Social Welfare (DSW). The feasibility, costs, barriers and benefits of that integration were assessed. The resulting final report, completed in 1998, 6 was instrumental in laying the foundation for following data integration projects. The Statistics New Zealand Statement of Intent for the year ended 30 June 2004 stated: 7 Data from administrative and transaction databases will increasingly be used for producing statistics. Users will expect this data to be made available, and where applicable, be integrated with data from other sources, resulting in greater information richness. Data integration has been used at Statistics NZ to create survey frames, supplement survey data, and produce new datasets. The main uses of probabilistic linkage have been as follows: New Zealand Census and Mortality Study Student Loan Data Integration Project Linked Employer-Employee Data (LEED) Project Injury Statistics Project. More detail on Statistics NZ s uses of data integration is given in the Appendix. 1.4 Key steps in a data integration project Through experience gained over the various data integration projects, Statistics NZ have identified a number of key steps that must be undertaken for a successful outcome. The list below is a high-level summary of what must be addressed. Each of these is fundamental, and none are trivial. The importance of clear and well-defined objectives cannot be overemphasised. The objectives will inform decisions at every other step of the project, from gaining approval to undertake the project (under Statistics NZ policy requirements), to assessing whether the integrated data is able to support outputs that are fit for purpose. Most data integration projects go through a feasibility or development stage, where the steps will be investigated and carried out to a greater or lesser extent. The results of the feasibility study determine whether full production systems will be developed. Many of the steps have parallels in the production of statistics from survey data. Differences occur where the nature of the data collection is different, and because there is an additional step of matching two or more data sources. 6 7 Statistics New Zealand (1998). Final report on the feasibility study into the costs and benefits of integrating cross-sectoral administrative data to produce new social statistics. (Internal report available on request.) Statistics New Zealand (2003c). Statistics New Zealand Statement of Intent: Year ending 30 June 2004, Statistics New Zealand, Wellington. 5

14 Key steps in a data integration project: develop clearly defined objectives address legal, policy, privacy, and security issues define governance structures and establish relationships with data providers and data users gain a thorough understanding of data sources decide how you will do the matching define and build information technology (IT) data storage and processing requirements obtain the source data carry out the matching validate the matching and provide quality measures consider provision of access to microdata and confidentiality of published outputs carry out the analysis and disseminate results. These aspects are discussed in more detail in the remainder of the manual. 6

15 2 Legal and Policy Considerations Summary All data integration projects are subject to a whole raft of legislation, codes of practice, protocols and policies. This section gives an overview of the relevant guidelines and a practical guide to their application within Statistics NZ. 2.1 Introduction Staff working on a data integration project should be aware of the various policies and legislative provisions that affect their project. Some of these, such as the Statistics Act 1975, 8 are applicable to much of the business carried out by Statistics NZ. Others, such as the Statistics NZ Data Integration Policy, are much more specific to data integration projects. Sometimes it can be difficult to interpret legislation and apply it in a practical way to a particular situation. Different parties have differing views on what is or is not acceptable, and the first project teams to work on data integration projects have had to debate and work through issues, which often establish precedents. Questions can often be resolved by discussion with experienced colleagues, the project manager and stakeholders. Sometimes advice must be sought from external parties such as a reference group or the Privacy Commissioner (see section 2.6.2, below). The following sections give an overview of some relevant documents and processes. 2.2 The Statistics Act 1975 Statistics NZ operates under the authority of the Statistics Act The Act provides the framework for the production of official statistics in New Zealand. It covers statistics collected in surveys of households and businesses, as well as statistics derived from administrative records of central and local government agencies. It covers official statistics produced by Statistics NZ as well as by other government agencies. As stated in the Statistics New Zealand Statement of Intent 2006, 9 Statistics NZ s main roles are to: lead New Zealand s Official Statistics System be the key contributor to the collection, analysis and dissemination of official statistics relating to New Zealand s economy, environment and society build and maintain trust in official statistics ensure official statistics are of high integrity and quality, and are equally available to all guarantee that statistical information provided to Statistics NZ remains confidential, and that it will be used for statistical purposes only Internal document available on request. The latest published version can be found at: 7

16 The Statistics Act 1975 does not specifically refer to data integration, so it is necessary to interpret its provisions in the data integration context. It includes the following points relevant to data integration: Official statistics shall be collected to provide information required by the Executive Government of New Zealand, Government Departments, local authorities, and businesses for the purpose of making policy decisions, and to facilitate the appreciation of economic, social, demographic, and other matters of interest to the said Government, Government Departments, local authorities, businesses, and to the general public [section 3(1)]. Official statistics means statistics derived by Government Departments from: (a) (b) statistical surveys as defined in this section; and administrative and registration records and other forms and papers the statistical analysis of which are published regularly, or are planned to be published regularly, or could reasonably be published regularly [section 2]. Information may be required of any person in a position to provide it to enable the production of official statistics of any or all of the kinds specified [section 4]. Independence of the Government Statistician in respect of deciding: (a) the procedures and methods employed in the provision of statistics produced by the Statistician; and (b) the extent, form, and timing of publication of those statistics [section 15(1)]. Furnishing of information required [section 32]. Security of information provided [section 37]. Information furnished under the Act to be used only for statistical purposes [section 37(1)]. Only employees of the department may view individual schedules [section 37(2)]. No information from an individual schedule is to be separately published or disclosed [section 37(3)], except as authorised by the Statistics Act 1975 (the Act permits others to see information from an individual schedule, but only when it is in a form that prevents identification of the respondent concerned, and then only under strict security conditions). All statistical information published is to be arranged in such a manner as to prevent any particulars published from being identifiable by any person as particulars relating to any particular person or undertaking [section 37(4)]. Data obtained by Statistics NZ for integration, and all integrated datasets, are considered to be furnished under the Statistics Act 1975, and therefore subject to the provisions of the Act. 2.3 The Privacy Act General information The Privacy Act 1993 aims to promote and protect individual privacy. It relates to personal information (not information about businesses). In section 6, twelve principles relating to the collection, storage, security, access, retention, use and disclosure of personal information are outlined. A Statistics NZ corporate document exists that outlines how the Privacy Act 1993 relates to statistics. 10 Several of the principles provide for exemption on the grounds that the information is used for statistical or research purposes and will not be published in a form that could reasonably be expected to identify the individual concerned. Regardless of these exemptions, it is important to consider the ideals expressed by the principles. Each situation must be 10 Statistics New Zealand (1999b). Statistics and the Privacy Act (Internal report available on request.) 8

17 evaluated in the light of the other privacy principles, and likely public perception of the proposed use. There are no guidelines in the Act regarding linkage for statistical purposes. In lieu of this, Statistics NZ has produced a set of data integration principles and guidelines (see section 2.4). The Privacy Act 1993 contains a chapter governing information matching. This does not relate to data integration as carried out by Statistics NZ; it relates to the comparison of two files for the purpose of producing or verifying information that may be used for the purpose of taking adverse action against an identifiable individual. The Statistics NZ Data Integration Policy states that Statistics New Zealand must not provide information to data providers about individual records in integrated data that could assist the data provider in carrying out any administrative purpose [principle 7(d)]. There is a designated channel for communication with the Office of the Privacy Commissioner on privacy issues in data integration projects (see section 2.6.2, below) Use of unique identifiers Principle 12 of section 6 the Privacy Act regarding unique identifiers has particular relevance for data integration projects and has been discussed at length with the Office of the Privacy Commissioner. Principle 12 states (under the heading Unique identifiers ): (1) An agency shall not assign a Unique identifier to an individual unless the assignment of that identifier is necessary to enable the agency to carry out any one or more of its functions efficiently. (2) An agency shall not assign to an individual a unique identifier that, to that agency s knowledge, has been assigned to that individual by another agency, unless those 2 agencies are associated persons within the meaning of section OD 7 of the Income Tax Act (3) An agency that assigns unique identifiers to individuals shall take all reasonable steps to ensure that unique identifiers are assigned only to individuals whose identity is clearly established. (4) An agency shall not require an individual to disclose any unique identifier assigned to that individual unless the disclosure is for one of the purposes in connection with which that unique identifier was assigned or for a purpose that is directly related to one of those purposes

18 A unique identifier is defined in section 2 of the Privacy Act 1993 as follows: Unique identifier means an identifier (a) That is assigned to an individual by an agency for the purposes of the operations of the agency; and (b) That uniquely identifies that individual in relation to that agency but, for the avoidance of doubt, does not include an individual s name used to identify that individual. Not all of the identifiers used by other departments are unique identifiers in terms of the above. However, in practical terms, this privacy principle impacts on Statistics NZ s ability to use unique identifiers in data integration projects, particularly in retention of the unique identifiers over time. The Statistics NZ Data Integration Policy states that unique identifiers assigned by an external agency must not be retained in an integrated dataset. It goes on to say: (a) (b) Data can be received that includes unique identifiers assigned by an external agency. These identifiers can be used to verify the integrity of the data, or to clean the data. They can also be used for integration of data, but they must be removed immediately after integration. When linking needs to occur on an ongoing basis, then the externally assigned identifier must be replaced by a new identifier. This new identifier can be used for integration. It must not be possible to derive the externally assigned identifier from the new identifier [principle 11]. 2.4 The Statistics NZ Data Integration Policy The Statistics NZ Data Integration Policy 12 states: This Data Integration Policy states Statistics NZ policy on integrating personal data. Data integration involves linking together information from different sources. This can be used to produce new statistics, enhance the value of existing statistics, and also enable a greater level of research. This can benefit New Zealand by increasing knowledge on the country s people, economy and environment. Individuals have legitimate privacy expectations. Integration of personal data can be privacy intrusive when it uses information for purposes other than those for which the information was originally provided. New Zealand legislation recognises individual expectations of privacy and the wider benefits of statistical information. The Statistics Act 1975 allows Statistics NZ to require responses to its surveys while also requiring that these responses are kept confidential. The Privacy Act 1993 requires adherence to a set of information privacy principles while also recognising exceptions when information is used for statistical or research purposes. This policy describes how Statistics NZ ensures that any integration of personal data is justified. It details the care taken by Statistics NZ when integrating personal data to

19 ensure any impact on privacy is minimised. This policy provides strict conditions, often beyond statutory obligations, on how Statistics NZ undertakes data integration. The Government Statistician s approval is required for all data integration projects. The Government Statistician will only give the go-ahead to a data integration project if satisfied that the principles set out in this policy will be fully observed. The policy covers: applicability data integration principles applying the data integration principles. There are 12 data integration principles. The following principles govern when integration of personal data for statistical or related research purposes can occur: 1. Statistics NZ must only undertake data integration if integration will produce or improve official statistics. 2. Data integration should be considered when it can reduce costs, increase quality or minimise compliance load. 3. Data integration benefits must clearly outweigh any privacy concerns about the use of data and risks to the integrity of the official statistics system. 4. Data integration must not occur when it will materially threaten the integrity of the source data collections. 5. Data must not be integrated where any undertaking has been given to respondents that would preclude this. 6. Data integration must be approved at an appropriate level by all the agencies involved. The following principles govern how integration of personal data for statistical or related research purposes will be done: 7. Integrated data must only be used for approved statistical or related research purposes. 8. The size and data variables of the linked dataset must be no larger than necessary to support the approved purposes. 9. Integrated data will be stored apart from other data. 10. Names and addresses can only be kept in an integrated dataset while necessary for linking. 11. Unique identifiers assigned by an external agency must not be retained in an integrated dataset. 12. Data integration must be conducted openly. Statistics NZ s Data Integration Policy Guidelines 13 also describes practical steps that can be taken to comply with the policy. These guidelines explain what must be done at various 13 Statistics New Zealand (2005a). Data Integration Policy Guidelines. (Internal report available on request.) 11

20 stages of a data integration project: before data integration starts, during data integration and when bringing a data integration project to a close. The Statistics NZ Data Integration Policy is essential reading for anyone embarking on a data integration project. 2.5 Codes of practice The Privacy Commissioner 14 may issue codes of practice which modify the Information Privacy Principles set out in the Privacy Act to take into account the special characteristics of specific industries, agencies or types of personal information. The provisions in a code may be more stringent or less stringent than the principles. Although there is no specific code of practice for data integration, a number of other codes of practice have been issued by the Privacy Commissioner. Where the code applies it substitutes for the principles [in the Privacy Act 1993]. For example, an action that would otherwise be a breach of one of the principles is deemed not to breach that principle if done in accordance with the code. Also a failure to comply with the code, where it applies, is for the purposes of the complaints procedures under the Privacy Act, deemed to be a breach of the principles. 15 Examples of such codes are: The Health Information Privacy Code 1994 The Post-Compulsory Education Unique Identifier Code Staff working on data integration projects should be aware of the existence of codes of practice that relate to their project and, in particular, how these affect the agency supplying the data. 2.6 Data integration business case It is important that before data is actually acquired and linked, appropriate discussions are held and approvals given. A specific requirement of the Statistics NZ Data Integration Policy is that a business case documenting how the proposed project will comply with the policy be submitted to the Government Statistician for approval. An approved data integration business case sets the boundaries for the data integration project. To produce the data integration business case (Statistics NZ, 2005a), the following tasks need to be undertaken: (1) Define purpose(s) It is essential to identify the statistical and related research purposes that the integrated data can be used for. Once this business case is approved, these purposes cannot be changed. The data integration business case must explain how these purposes will produce or improve official statistics. Improving official statistics 14 Office of the Privacy Commissioner: 15 Health Information Privacy Code 1994: 12

21 could mean improving: accuracy, reliability, timeliness, consistency, coverage, concepts, definitions or methodologies. (2) Stakeholder consultation As part of producing a data integration business case, it is necessary to consult with stakeholders in the project. This is likely to include consultation with at least the following groups: data suppliers, respondent representatives, the Privacy Commissioner, the Chief Archivist. (3) Prepare a privacy impact assessment A privacy impact assessment (PIA) needs to be undertaken and included in the data integration business case. (4) Prepare the data integration business case The data integration business case needs to document how the proposed data integration project will comply with the Statistics NZ Data Integration Policy. A list of requirements for a data integration business case is available in Attachment A to that policy. (5) Obtain approval First, approval for a data integration business case must be obtained from the chief executives of any agencies that would supply data for integration (other than Statistics NZ). Following this, the business case can be submitted to the Government Statistician for approval. Approval must be received from the Government Statistician before any data integration occurs. Statistics NZ can undertake a pilot study to determine whether using data integration to produce or improve official statistics is feasible in a particular case prior to full project approval [principle 1(b)]. A pilot study is likely to be needed for any significantly new or largescale data integration work. Any pilot study still needs to conducted in accordance with the Data Integration Policy and consequently needs its own business case. Once approved, the Minister of Statistics needs to be formally notified that the data integration project has been approved and the project must also be included in the list of data integration projects maintained on Statistics NZ s website Privacy impact assessment A key component of a data integration business case is the PIA. PIAs are not unique to Statistics NZ they are used in many situations where risks to privacy arise for example, from a new technology, the convergence of existing technologies, the use of a known privacy intrusive technology in new circumstances, a major endeavor, or a change in practice. Privacy impact assessments for Statistics NZ s data integration projects include a description of procedures for collection, use, disclosure and retention of personal information. They analyse the risks to privacy, and state how these risks will be managed, avoided or reduced. Statistics NZ has compiled a summary of privacy issues that must be considered in an integration project. 16 The following is an extract: 16 Statistics New Zealand (2002c). Pro forma Privacy Impact Assessment Report Data Integration Projects (draft). (Internal report available on request.) 13

22 The risks associated with failing to address the privacy implications of a given proposal can take many forms, and may include: failing to comply with either the letter or the spirit of the Privacy Act, or fair information practices generally, resulting in criticism from the public or Privacy Commissioner or complaints under the Act stimulating public outcry as a result of a perceived loss of privacy or a failure to meet expectations regarding the protection of personal information loss of credibility or public confidence when the public feels that a proposed project has not adequately considered or addressed privacy concerns underestimating privacy requirements with the result that systems need to be redesigned or retro-fitted at considerable expense. An important consideration is the expectations of the general public, customers, clients or employees. Proposals may be subject to public criticism even where the requirements of the various Acts have been met. If people perceive their privacy is seriously at risk, they are unlikely to be satisfied by justification that the project has not technically breached the law. Risks to actual and perceived privacy can arise in many circumstances. Collecting excessive information, using intrusive means of collection, or obtaining sensitive details in unexpected circumstances all represent risks to the individual, and might already be there to some extent with the various source datasets. Unexpected or unwelcome use or disclosure of that information, as in a [data integration] project could put perceived privacy at risk. One task of the PIA is to sort out which risks are serious and which are trivial. The privacy impact report should identify the avoidable risks and suggest cost-effective measures to reduce them to an appropriate level. PIAs are usually compiled by the project manager, with contribution from team members and policy staff. However, it is important for project staff to be aware of privacy issues relating to their project and how they are to be managed. PIAs for established projects are a useful way of getting to grips with privacy issues, for example the Linked Employer-Employee Data PIA. 17 While not a formal part of a PIA, any issues regarding appropriate protection for business information should also be considered Consultation with other agencies (i) Office of the Privacy Commissioner It is sometimes necessary to consult with the Office of the Privacy Commissioner (OPC) regarding proposed data integration projects. A role of the Privacy Commissioner is to:... investigate on receipt of a complaint or on his own initiative and form an opinion about whether there had been an interference with the privacy of an individual. He 17 CE070FFC4807/0/LEEDPrivacyImpactAssessment.pdf 14

23 does not give a decision. Nor does his opinion bind anyone. Only the Complaints Review Tribunal can give a decision which could be accurately described as a ruling. Therefore, Statistics NZ cannot and does not seek approval from the Privacy Commissioner to proceed with a data integration project, but does seek advice from the Privacy Commissioner as to whether the proposed approach raises any concerns, and works closely with him or her to determine the most appropriate solution to any such concerns. For a Statistics NZ project manager, faced with making a choice between continuing with an approach that precedent and best current understanding can accept, or arguing for a change in what the OPC is comfortable with to get a better outcome, it is much easier and more certain to take the first option. The OPC is not resourced for quick responses to such approaches, because it is not their core business. It is up to Statistics NZ to be compliant with the Privacy Act 1993 and, if necessary, to seek professional advice on issues. (ii) Other agencies Integration projects must recognise the direct interests of stakeholders and take any concerns into account in a decision to integrate. Providers of source datasets, as well as groups that represent the interests of those whose information is being integrated, must be consulted. 2.7 The Statistics NZ Confidentiality Protocol The Statistics NZ Confidentiality Protocol 18 is another document that has an impact on data integration projects. It includes sections on restricting use of information to statistical purposes, protecting confidential information, and rules for avoiding disclosure of confidential information in outputs and microdata. Although this protocol is not specific to integrated data, its provisions do apply to integrated data, and it is therefore important to be aware of its content. There is greater risk of disclosure from integrated datasets and therefore extra care is required to protect the data. 2.8 The Statistics NZ Microdata Access Protocols Integrated datasets potentially pose more risk in terms of disclosure so any access to microdata needs careful consideration. The Statistics NZ Data Integration Policy notes that the Government Statistician s decision on provision of microdata access will be made after consultation with data providers and will take into account: the legislation under which the data was collected Statistics NZ s Microdata Access Protocols 19 the data integration business case (Statistics NZ, 2005a) (eg allowed uses of integrated data) any agreements entered into with data suppliers Statistics New Zealand (1999a). Confidentiality Protocol. (Internal report available on request.) 15

24 In some cases, the data providers might advise Statistics NZ of legal requirements or other conditions that preclude some of the forms of access that would otherwise be possible. Tax data is a particular instance where this has occurred. Statistics NZ s Microdata Access Protocols provide guidance on: the type of research that will be considered eligible for use of Statistics NZ microdata the methods of access that are available, including on-site Data Laboratory, off-site Data Laboratory, confidentialised unit record files (CURFs) or remote access the conditions placed on the researchers and their obligations in using microdata and producing output the data that is potentially eligible for use. 16

25 3 Operational Aspects of a Statistics NZ Data Integration Project Summary Individual data integration projects can vary greatly in terms of the data sources and methods used. This chapter outlines some of the operational aspects that are common to most data integration projects carried out by Statistics NZ. 3.1 Early stages of a data integration project A data integration project begins with the approval processes outlined in Chapter 2. It is recommended that a pilot study be undertaken to assess the feasibility of a data integration project. Once approved, a successful data integration proposal moves into the usual phases of a Statistics NZ project, with the development of project initiation documents, and the bringing together of a project team. It is also important to establish inter-agency relationships and support from interest groups as early as possible in the initiation phase. 3.2 Other relationships Statistics NZ s data integration projects usually involve data from external agencies, and produce outputs that are of wide interest outside the organisation. In addition to a project team and internal Statistics NZ governance, each data integration project is likely to have a number of critical relationships with external groups. The nature of these relationships differs, depending on how the project is structured. These are broadly described in the following table, with examples from each of the current data integration projects. Role Student loans Linked Employer- Employee Data (LEED) Provide independent purchase advice to the relevant Minister(s) Oversee development Provide expert review and advice Represent user views and needs, provide advice Provide data, and advice on data IRD-led steering committee Inter-departmental working group Sponsors group Expert advisory group Users group Data providers Injury Ministerial advisory panel External reference group A critical factor in the success and efficiency of any data integration project is the quality of the relationship with the data providers and users. 17

26 3.3 Obtainment and safe keeping of external data Data extract The data integration business case submitted for Statistics NZ s approval needs to detail the data it proposes integrating. This should include a list of the variables from each data source. If it is not possible to determine these details, then a business case for a pilot study should be produced instead. A main objective of the pilot study will be to determine the variables that are needed for integration and whether the integrated dataset is suitable for achieving the statistical objectives of the project. The specifications for the data extract should use the table and variable names of the source. Specific inclusions and exclusions must be clear and unambiguous. The data received should be examined to ensure it complies with expectations (see section 4.3, below) Data transfer, storage, security and internal access controls In keeping with Statistics NZ s core value of Security, it is important that data integration projects provide adequate protection to the data that is involved. The following checklist, developed by Paul Maxwell based on international practice and now used in Statistics NZ privacy impact assessments, contains a range of issues that need to be addressed within data integration projects. The project team should actively contribute to the maintenance of high standards of security. Security checklist: Have security procedures for the collection, transmission, storage and disposal of personal information, and access to them, been developed and documented? Are privacy controls in place for the project? Have technological tools and system design techniques been considered that may enhance both privacy and security? Has there been an expert review of all the security risks and the reasonableness of countermeasures to secure the system against unauthorised or improper collection, access, modification, use, disclosure and disposal? Have staff been trained in requirements for protecting personal information and are they aware of policies regarding breaches of security or confidentiality? Are there authorisations controls defining which staff may add, change or delete information from records? Is the system designed so that access and changes to data can be audited by date and user identification? Does the system footprint inspection of records and provide an audit trail? Are user accounts, access rights and security authorisations controlled and recorded by an accountable systems or records management process? Are access rights only provided to users who actually require access for the stated purposes of collection or consistent purposes? Is user access to personal information limited to that required to discharge the assigned functions? Are the security measures commensurate with the sensitivity of the information recorded? Are there contingency plans and mechanisms in place to identify security breaches or disclosures of personal information in error? Are there mechanisms 18