Migrating digital records

Transcription

1 Migrating digital records A guideline for Queensland public authorities June 2012 Version 1.0 Queensland State Archives Department of Science, Information Technology, Innovation and the Arts

2 Document details Security Classification PUBLIC Date of review of security classification June 2012 Authority Queensland State Archives Author Queensland State Archives Document Status Final Version Version 1.0 Contact for enquiries All enquiries regarding this document should be directed in the first instance to: Digital Archives Unit Queensland State Archives Copyright Migrating digital records: a guideline for Queensland public authorities Copyright The State of Queensland (Department of Science, Information Technology, Innovation and the Arts) 2012 Licence Migrating digital records: a guideline for Queensland public authorities by Queensland State Archives is licensed under a Creative Commons Attribution 3.0 Australia Licence. To view a copy of this licence, please visit Information security This document has been security classified using the Queensland Government Information Security Classification Framework (QGISCF) as PUBLIC and will be managed according to the requirements of the QGISCF. Migrating digital records, version 1.0 page 2

3 Executive Summary This guideline provides advice to public authorities on the key recordkeeping issues to be considered when planning for and carrying out data migration projects where the business system contains public records. Moving public records from one hardware/software configuration to another, or from one generation of computer technology to a subsequent generation, requires explicit planning to ensure that the public records maintain their authenticity and integrity in accordance with the Public Records Act Digital source records that have been migrated cannot be disposed of unless they have met certain criteria as outlined under the General Retention and Disposal Schedule for Digital Source Records (QDAN 678), and this guideline provides guidance to Queensland public authorities about the migration of digital records and the lawful disposal of the source records. The guideline assumes: familiarity with the requirements of the Public Records Act 2002 and a knowledge of the various supporting standards and guidelines, and that records management expertise is incorporated into the overall migration project planning and implementation. The guideline is organised into three parts: 1. Introduction outlines the context of migrating digital records as it is applied in the guideline. 2. Fundamental issues when migrating public records examines the challenge of evolving technology, drivers for migration, risks, obligations and the treatment of source records after migration. 3. Recordkeeping considerations for migration projects outlines key recordkeeping considerations to be incorporated into the overall migration project, including various checklist tools included as appendices. This document forms part of a wider framework of digital continuity advice under development by Queensland State Archives to support public authorities in meeting their obligations in managing their digital public records. Migrating digital records, version 1.0 page 3

4 Contents 1. Introduction Purpose Audience Scope Authority Relationship to other Queensland State Archives publications Definitions Acknowledgements Important considerations for migrating public records Drivers for migration Risks Obligations Treatment of source records after migration Recordkeeping considerations for migration projects Determining what is to be migrated Determining where and how to migrate Ensuring quality Performing migration Post migration Appendices...35 Appendix A IS40 Principle 7 and the migration of digital records Appendix B Records manager s project stages flowchart Appendix C Records manager s migration project checklist Appendix D Recordkeeping controls in the target system Appendix E Testing and validation checklist...54 Appendix F Lessons learnt hints and tips Migrating digital records, version 1.0 page 4

5 1. Introduction 1.1 Purpose The purpose of this guideline is to assist public authorities to meet their recordkeeping obligations under the Public Records Act 2002 when they are planning and implementing data migration projects involving business systems that contain digital public records. The document also forms part of a wider framework of digital continuity advice developed by Queensland State Archives, to support public authorities in digital recordkeeping obligations. Specifically the guideline provides advice on: key recordkeeping considerations where the migrated version of the record will act as the official public record, and the correct disposal of the digital source records in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). 1.2 Audience The intended target audience for this guideline is information and records management staff responsible for advising on a public authority s recordkeeping compliance and who will be working with: project team members, business analysts, business system owners and ICT colleagues, and other key stakeholders responsible for data and business system migration projects or activities. 1.3 Scope This guideline applies to all Queensland public authorities, as defined in Schedule 2 of the Public Records Act The guideline provides support for the migration of digital public records, where the migration of the digital public records is from one hardware/software configuration to another or from one generation of information technology to a subsequent generation, for ongoing business and legislative purposes. This guideline: does not seek to provide all the technical advice on carrying out a data migration project does not replace the existing project management methodology adopted by the public authority, and does not cover the digitisation of records from a physical hardcopy format to a digital format such as when paper records are digitised Authority The State Archivist has issued this guideline in accordance with section 25(1)(f) of the Public Records Act 2002, which enables the State Archivist to make policy, standards, and guidelines about the making, keeping, preserving, managing and disposing of public records. 1 For guidance about digitisation or microfilming of records, please see the Queensland State Archives Digitisation Disposal Policy and associated toolkit, or the Microfilming Disposal Policy and guideline. Migrating digital records, version 1.0 page 5

6 1.5 Relationship to other Queensland State Archives publications This guideline is specifically supported by: the QSA publication Metadata for digital continuity: a companion guideline to the Queensland Recordkeeping Metadata Standard, and the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). Figure 1 Migration, metadata and retention and disposal connections It is also recommended that recordkeeping and information management staff have an understanding of the following documentation: Information Standard 40: Recordkeeping (IS40) Information Standard 31: Retention and Disposal of Public Records (IS31) Related public records briefs: Use of the Guidelines and Functional Requirements for Records in Business Systems by Queensland public authorities Identifying a Public Record in the Electronic Environment Advice on the Destruction of Public Records Decommissioning Business Systems (currently under review) Keeping digital records useable Ten steps for ensuring the continued accessibility of digital records. Online access to these documents is available on the Queensland State Archives website. It is assumed that the recordkeeping considerations detailed in this guideline will be incorporated into the public authority s own project management methodology, for example the Queensland Government Project Management Methodology 2. 2 Accessible at Queensland Government Project Management Methodologies Migrating digital records, version 1.0 page 6

7 1.6 Definitions Records and information management specific terms are defined in Queensland State Archives Glossary of Archival and Recordkeeping Terms available on Queensland State Archives website. Some specific terms as they are used in the context of migration processes in this guideline are described below. Migration There are many different definitions and understandings of the term migration across disciplines. Within the context of this guideline, the term migration is used to encompass the process of preserving the authenticity of records when relocating digital records from one hardware/software configuration to another, or from one generation of computer technology to a subsequent generation. 3 Source records and migrated records Source record and migrated record are terms used in this document. For clarity, source record refers to the record being migrated, and the migrated record is the record resulting from the migration, as illustrated in Figure 2 below. Old home relocation to New home Source system Target system Source records Migration Migrated records Figure 2 Migration of source records to a new target location 3 Referred to by R. Harvey (2005), Preserving Digital Materials, K. G. Saur, Munich, page 148. Originally used in Preserving Digital Information: Report of the Task Force on Archiving of Digital Information, Available at: Migrating digital records, version 1.0 page 7

8 Authorised delegate As stated in Information Standard 31: Retention and Disposal of Public Records, an authorised delegate refers to the Chief Executive Officer or their appointed delegate who has responsibility for authorising the disposal of a public authority s records in accordance with an approved Retention and Disposal Schedule. Data cleansing Data cleansing involves the removal of out-of-date, duplicated, inaccurate or redundant information and is a recommended practice prior to undertaking migration activities in order to minimise the relocation of unnecessary or inaccurate information. In the context of this document, data cleansing processes are applied to the review of the data/records/metadata in the source system in order to make decisions about what data/records/metadata will be migrated into the target system. It should be noted that data cleansing activities are likely to trigger the need for an authorised disposal of time-expired records. This means that any public records identified for deletion during data cleansing in the source system prior to the migration activity will need to be disposed of in accordance with an approved Retention and Disposal Schedule. Data mapping Data mapping refers to the process of establishing connections or relationships between the data elements of the source and target systems that will enable the migration of the records and their associated metadata, and contribute to the maintenance of the authenticity and integrity of the records. Deletion vs destruction vs disposal In this guideline, these three terms have distinct meanings and are described below: Deletion in this guideline is a generalised term referring to the erasing of digital information contained in a business system. Destruction refers to the process of eliminating or deleting records that do not have continuing value, beyond the point of any possible reconstruction. Disposal of records as described in the Public Records Act (2002) includes the actions of destroying, damaging, abandoning, transferring, selling, donating or giving away a record, or part of a record. In the context of this guideline, the term disposal has been used to refer to: o the deletion of the source record, following migration, or o the deletion of eligible time-expired records prior to migration, undertaken in accordance with the terms stated in an approved Retention and Disposal Schedule. Digital continuity Digital continuity used in the context of this guideline is the ability to ensure that despite organisational, business and technological change, the access, readability and the authenticity of digital records is maintained for as long as they are required to be kept for business, legal and/or archival requirements. Digital continuity involves more than preserving digital records, and requires a range of robust records and information management policies and practices. Queensland State Archives is responsible for developing a digital continuity strategy and tools for state government and this guideline is one of the tools developed to support the continuity of digital public records. Migrating digital records, version 1.0 page 8

9 Encryption and decryption Encryption refers to a process where security coding is applied to data that converts it into a secured format for protected transmission across networks. To read the data the receiver must have the decryption key coding to restore the data to its original form. 1.7 Acknowledgements A range of advice was referenced in the development of this guideline. Queensland State Archives in particular acknowledges State Records New South Wales migration advice in their Managing Digital Records Guideline. 4 Queensland State Archives would like to thank the many public authorities that contributed to the development of the guideline and their representatives who also commented on the exposure draft. 4 State Records New South Wales (2009) Managing Digital Records Guideline: Effectively Manage the Migration of your Digital Records. Accessible from Migrating digital records, version 1.0 page 9

10 2. Important considerations for migrating public records Two important recordkeeping values need to be upheld when undertaking migration projects. They are: maintaining the authenticity, useability, integrity and reliability of public records for as long as they are required to be kept for business, legal and / or archival purposes, and ensuring the disposal of records is authorised and in accordance with an approved Retention and Disposal Schedule. These values have broad reaching implications for records migration projects, and the recordkeeping and information professional not only needs to have a grounding knowledge of these values but also needs to be able to translate them and have input into applying these values to the overall migration project. Before dealing with specific recordkeeping considerations in Section 3, this section will specify the: drivers for migration potential risks to records recordkeeping obligations, and the treatment of source records after migration and will assist in setting the scene for a recordkeeping and information management professional s input into a records migration project. 2.1 Drivers for migration Digital records are dependent on technology for their access and use. Over a period of time, the migration of the records from one hardware/software configuration to another may be required for the continuing access, useability and ongoing management of the public records in accordance with business, legal and archival obligations. The need to migrate public records may be triggered by a range of drivers, for example: obsolescence and technological changes that require the upgrading or decommissioning of business systems changing business needs that lead to the adoption of a new business system system and storage rationalisation, e.g. as a result of the implementation of the Application Rationalisation Methodology (ARM) 5, and / or machinery of government changes that require the transfer of functions or activities recorded in business systems from one public authority to another entity. Figure 3 Examples of drivers for migration 5 Migrating digital records, version 1.0 page 10

11 2.2 Risks The processes for the relocation of public records from one hardware/software configuration to new technological environment/s may threaten the legal authenticity and integrity of digital public records. To mitigate risks to the public records, migrations must be controlled by authorised and planned policies and procedures that incorporate recordkeeping considerations within the public authority s overall project management methodology. Specifically the threats and risks to records during migration include: failure to maintain the authenticity and integrity of the records (refer to Information Standard 40 for details) loss of readability and accessibility, particularly for records that are required to be kept for the longterm or in perpetuity, and non-compliance with legal obligations for the maintenance of the public records, for example through inadvertent loss or unauthorised deletion through failure to identify and manage the existence of digital public records when decommissioning a system. It is important that risk mitigation assessments and strategies relevant to recordkeeping be incorporated into planning and that ownership of risk is delegated, managed and documented. Risk mitigation strategies may include: identifying and appraising the nature of the digital records to be migrated in accordance with approved Retention and Disposal Schedules, to assist in determining the potential risk value determining and documenting the capacity of the target system to capture the required recordkeeping metadata to sustain the authenticity and integrity of the migrated records adopting robust quality assurance planning and processes. Section 3 of this guideline will highlight the key recordkeeping considerations that need to be integrated into risk mitigation planning. 2.3 Obligations The Queensland Government Recordkeeping Policy Framework sets out the recordkeeping obligations of public authorities. 6 Public authorities must meet all legislative and regulatory requirements relating to the management of records during migration projects and activities. This includes compliance with: Public Records Act 2002 this Act is concerned with ensuring the public records of Queensland are made, managed, kept and, if appropriate, preserved in a useable form for the benefit of present and future generations. It mandates a number of obligations for Queensland public authorities which include making and keeping full and accurate records of its activities, ensuring the safe custody and preservation of records in its possession, and who must authorise disposal. Information Standard 40: Recordkeeping this standard outlines seven mandatory principles for compliance by public authorities. These principles are summarised in Table 1. Appendix A provides further guidance on the application of Principle 7 in the digital records migration context. 6 Available from the Queensland State Archives website Migrating digital records, version 1.0 page 11

12 Principle Principle 1 Principle 2 Principle 3 Principle 4 Principle 5 Principle 6 Principle 7 Requirement Public authority recordkeeping must be compliant and accountable Recordkeeping must be monitored and audited for compliance Recordkeeping activity must be assigned and implemented Recordkeeping must be managed Recordkeeping systems must be reliable and secure Recordkeeping must be systematic and comprehensive Full and accurate records must be made and kept for as long as they are required for business, legislative, accountability and cultural purposes. Table 1 Mandatory principles of Information Standard 40: Recordkeeping Information Standard 31: Retention and Disposal of Public Records - the Public Records Act 2002 prohibits the disposal of public records without the permission of the State Archivist. The primary purpose of this Information Standard is to assist public authorities to lawfully retain and dispose of public records. This standard, among other obligations, requires that public authorities must: - dispose of public records in accordance with a Retention and Disposal Schedule approved by the State Archivist that is current at the time of disposal - ensure all disposal is endorsed by the Chief Executive or an authorised delegate - ensure the method of destruction of public records is appropriate to the sensitivity of the records and conforms with local environment regulations - document the disposal of public records. 2.4 Treatment of source records after migration Decisions about the treatment of the source records after their successful migration need to be incorporated into the migration project plan. Deletion of source records needs to be in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678), which addresses the treatment of the remaining source records after they have been migrated and enables lawful disposal. Section and of this guideline provide further advice about the application of retention and disposal obligations within the records migration project stages. Key points: The processes of moving records from one system to a new technological environment can threaten the authenticity, integrity, reliability and useability of digital records. To mitigate risks, migrations must be controlled by authorised and planned policies, processes and procedures that are designed to incorporate key recordkeeping standards. Source records are not automatically eligible for deletion. Certain obligations outlined in this guideline and the General Retention and Disposal Schedule for Digital Source Records (QDAN 678) need to have been met before deletion can be authorised to be undertaken. Migrating digital records, version 1.0 page 12

13 3. Recordkeeping considerations for migration projects This section outlines the recordkeeping considerations that need to be included in planning migration projects. The structure of this section follows a generic migration project methodology, however, it is acknowledged that public authorities will not necessarily all use the same methodology for migration projects. A public authority s approach can be based on this document but will need to be adapted to the specific project management methodology adopted by the public authority. Table 2 below summarises the considerations that are detailed further in this section. A more readily useable version of the information in this table has been prepared as a flowchart, which is available at Appendix B. Additionally, the flowchart stages have been expanded into a checklist for records managers which provides, in essence, a summary of the material in the rest of this section. The checklist is available at Appendix C. Records migration stages 1 Migration project initiation 2 Determining what is to be migrated 3 Determining where & how to migrate 4 Ensuring quality 5 Performing migration 6 Post migration Description The initial stage of high-level scheduling and resource planning for the project. The stage of the project when the scope of the migration is being established. The stage when decisions are being explored and made about the approach, method and target system(s) for the migration. The stage when quality assurance and testing and validation processes are identified, planned and undertaken. The stage concerned with the actual undertaking of the migration to extract/export, transfer and load/import the records into the target system(s). The final recordkeeping considerations prior to project closure. Recordkeeping considerations Project planning Roles and responsibilities Stakeholder communications Identifying the records Establishing retention obligations Determining recordkeeping metadata Understanding characteristics of the records Verifying target system/s recordkeeping capabilities Making file / record format decisions Establishing migration approach Preparing records for migration Testing and validation Testing roll back strategy Recordkeeping during cut over period Migration of records Quality assuring the results of the migration Verifying conditions for disposal of source records Disposal of digital source records Capture mandatory recordkeeping documentation Stakeholder relations and continuous improvement Table 2 Project stages for recordkeeping considerations Migrating digital records, version 1.0 page 13

14 3.1 Migration project initiation Project planning A migration project that is systematically planned, initiated and carried out to well-defined procedures will help protect records and is required to ensure their evidential integrity. This guideline assumes that appropriate processes and documentation will be established to initiate a migration project by the broader organisational project team. This would include project planning documentation which clearly describes the drivers, approaches, legislative and policy requirements and expected outcomes for the project. It also assumes that appropriate change management activities are planned. It therefore does not address these planning issues in detail and assumes that the records manager will engage with the broader project. During the planning process it is important to ensure that: there is recorded acknowledgement within plans of the need to meet a public authority s recordkeeping obligations throughout the project risk mitigation planning is incorporated that is commensurate with the level of value associated with the records, and there is appropriate scheduling of time and resources for the recordkeeping activities associated with the migration of digital public records. Key point: Ensure recordkeeping obligations, standards and activities are well understood by the project team and are sufficiently incorporated into the planning and scheduling for the project Roles and responsibilities As outlined in section 2.2 migrations can be a high risk activity for records. Those responsible for the management of records must be involved in all records migration projects. This includes projects to migrate records stored in any business system, not just for projects that involve dedicated recordkeeping systems such as electronic document and records management systems (edrms). 7 When migration projects involving records are established within a public authority, and roles and responsibilities are planned, records and information managers are well placed to contribute expertise to the project through: assisting with the identification and appraisal of the records within business systems, how long the records must be legally retained and which records are eligible for lawful disposal determining how the records are being managed within the existing system and their current and ongoing recordkeeping compliance needs 7 In this guideline business systems refer to automated systems that have a primary purpose of facilitating business transactions (e.g. finance, licensing, client-relationship systems, etc.) and that do not necessarily contain records management functionality. This is distinct from electronic document and records management systems (edrms) that are designed with records management functionality to manage the records it captures throughout their lifecycle. Migrating digital records, version 1.0 page 14

15 facilitating the process for the authorised disposal of records supporting quality assurance of migrated records in collaboration with business area experts and others assisting in the data mapping process between the source and target systems to ensure records, including their metadata, remain full and accurate records of business and that their authenticity can be verified advising on the recordkeeping controls needed in the design and configuration of the target system developing and delivering training and awareness materials and sessions related to records management, and advising on and promoting continuous improvement in recordkeeping functionality within new systems Stakeholder communication planning As mentioned above, this guideline assumes that appropriate change management activities are planned as part of the overall project management plan. However it is important that those responsible for records and information management engage with the overall communications strategy and the proposed timings for the migration of the records. This is needed in order to understand and plan for the potential impact on recordkeeping requirements, and on the business and its stakeholders, during the time of the records changeover period between the source system and the target system. (Please note: Appendix F provides some real-life insights into recordkeeping considerations during this time. Refer to F.3.) There will be a need to ensure that appropriate training of staff is undertaken. This may require targeted promotion and the development of new procedures relating to the management of the records in the newly migrated environment. Key points: Those with responsibility for records management should be engaged and consulted in migration projects to ensure that recordkeeping considerations are integrated into the overall project management planning. The key objective of records/information managers being involved is to support the achievement of maintaining the authenticity and evidential integrity of the public records throughout the migration process and to assist with retention and disposal compliance. Migrating digital records, version 1.0 page 15

16 3.2 Determining what is to be migrated A key decision point in a migration project is determining what information will be migrated. To determine this, there are some fundamental details about the records themselves that must first be established. Understanding these details will help to inform decisions about what data to migrate and what data does not need to be migrated. The processes for determining this are described below Identifying the records Records are the consequence or product of a business activity. Digital records are not simply data but comprise a complex set of related information which includes the: content of the record contextual metadata, for example, information about who created the record, how it has been managed, relationships to other records, etc, and the structure that enables the record to be accessible and meaningful, e.g. database table headings. Knowing what the records are within a system is critical to protecting and maintaining their completeness, integrity and authenticity. It is important to clearly scope and document them prior to undertaking a migration. Depending on the nature of the migration project, the records to be migrated may be easily identified and can be clearly described. However, identifying records in business systems can often be challenging as they are generally complex and are not always packaged as neatly discrete objects. The process of determining what records are contained in business systems involves undertaking an analysis of the business processes associated with the system s use, in order to identify the information that records the evidence of business activity and transactions. For further advice on this process see Part 2.3 of the Guidelines and Functional Requirements for Records in Business Systems. 8 This process of analysis will also highlight ephemera within the system, including any duplicated copies of information. Ephemera are items of short-term value that are not required to be kept as records and therefore may be able to be disposed of, if not required for other business or legal purposes, prior to the migration. Key point: Know what records there are and describe them to ensure the records can be successfully maintained throughout and following the migration process. 8 The use of this guideline has been endorsed by the State Archivist and is accessible from Queensland State Archives website at Also refer to the public records brief at Please note that in 2008 the International Council on Archives (ICA) published three modules of its Principles and Functional Requirements for Records in Electronic Environment (ICA-Req). These modules have since been adopted by the International Standards Organisation and published as ISO Migrating digital records, version 1.0 page 16

17 3.2.2 Establishing retention obligations The retention periods for records are documented in authorised Retention and Disposal Schedules. 9 Once the records have been identified, an authorised schedule can be consulted and applied to determine how long they must be kept. Some records may have passed their retention period and can be deleted, which will reduce the volume of records requiring migration. IMPORTANT: The deletion of records must be in accordance with the authorised disposal requirements. It is also important to consider other aspects that may apply to the records prior to deletion, such as whether a disposal freeze 10 is in place or another business or legislative reason overrides the Retention and Disposal Schedule/s. For example, are the records in question potentially needed in evidence in an existing judicial proceeding, including any reasonably possible judicial proceeding? When deletion is to be undertaken in the source system prior to migration, all disposal actions must be documented in accordance with requirements under Principle 2 of Information Standard 31: Retention and Disposal of Public Records. For further information about disposing of records, see section 4 of the Guideline for the Implementation of Retention and Disposal Schedules. 11 The disposal of public records must be authorised by the State Archivist under section 26 of the Public Records Act Such authorisation is usually given in a Retention and Disposal Schedule. In the event the identified records are not covered by an authorised Retention and Disposal Schedule, records cannot be deleted prior to migration. Key points: Identify the lawful retention period for the records through consultation with authorised Retention and Disposal Schedule/s and consider any other legal obligations e.g. records subject to litigation, discovery process. After establishing the lawful retention period, if there are no longer any legal, business or legislative requirements to retain the records, undertake an authorised process to dispose of any records that have met their minimum retention period. If there is no authorised Retention and Disposal Schedule covering the migrated records, disposing of them is unlawful. 9 See for further information. 10 A disposal freeze is an authority issued by the State Archivist that requires the temporary cessation of the destruction of public records, relating to a specific topic or event, that are covered by an approved Retention and Disposal Schedule. A disposal freeze, issued by the State Archivist for a particular group of public records, overrides an existing Retention and Disposal Schedule that applies to the records of a public authority for a specified period of time. Refer to 11 Accessible from %20Retention%20and%20Disposal%20Schedules.pdf Migrating digital records, version 1.0 page 17

18 3.2.3 Determining recordkeeping metadata Recordkeeping metadata is a fundamental tool for creating and maintaining full and accurate records and is a key means by which the integrity and authenticity of a record can be proven. A key feature that differentiates recordkeeping metadata from other types of metadata is that it is not a static profile of a document or other resource. Recordkeeping metadata initially defines a record at the point of capture, but is also dynamic and accrues through time, to provide information on how a record has been used or managed. This characteristic of recordkeeping metadata is essential for preserving the authenticity of records. Safeguarding metadata relationships Effective migration of records requires that the associated metadata including the connections between a record and its metadata are persistently maintained. This requires an understanding of the records existing metadata. Documenting this metadata will help to highlight all the necessary data that must be migrated, particularly metadata which records the information about relationships and connections between data. Relationships requiring safeguarding may be: structural that is within the document, such as a document containing a linked spreadsheet or images between records e.g. between records documenting related aspects of business between records and/or containers e.g. documents aggregated to files/folders between records and other entities e.g. between records and creating agents between records and control tools such as Business Classification Schemes, Retention and Disposal Schedules, access and security controls and mandates. Some of this information may be outside the system in question, for example, the Business Classification Scheme, however processes will need to be established to ensure the relationships are successfully maintained with the migrated records. Once the metadata and records are understood and documented, the process of mapping the metadata from source to destination system is essential to ensure that all the necessary metadata is carried over to the target system and remains useable and linked to the records it describes. This will contribute significantly to the maintenance of the records context and authenticity. Metadata and mapping analysis is an opportunity to address any insufficiencies and/or apply improved metadata capture that ensures that the minimum recordkeeping metadata requirements are met. In some migrations the date of the migration can become the default date for all recordkeeping actions, including date of creation of records. This severely impacts recordkeeping processes such as disposal and undermines the authenticity, useability and integrity of the records. Checks should therefore be in place to ensure dates that trigger recordkeeping actions in the source system remain accurate and unchanged post migration. Migrating digital records, version 1.0 page 18

19 The Queensland Recordkeeping Metadata Standard and Guideline (QRKMS) can be used as a tool to help identify the metadata that requires migration, and the links that must be maintained. For example The table below contains columns to identify the QRKMS required metadata, the relevant field in the current system and the target field in the new system. The comments column can be used to note any system limitations or issues that require resolution before migration. QRKMS element Current system Target system Comments Identifier Transaction reference Transaction ID Check if ID format is supported by new system, or whether new format is required Date created Transaction date Date Must ensure dates carry across as is from old system and are not updated A metadata implementation matrix is available that can be adapted for this purpose. 12 Capture metadata about migration When migration of digital records occurs, it is a key event in the lifecycle of the records. Metadata must be captured to document the process of the migration of the record. The mandatory Record Event History element within the Queensland Recordkeeping Metadata Standard requires documenting the preservation, retrieval, disposal, control, access or use related activities performed on a record. This metadata provides a visible and auditable trail of records management actions and decisions, and can be used to record the migration activities on the record. Other metadata considerations In addition to specific recordkeeping metadata mentioned above, the system may hold other metadata that is crucial to the meaning of the records. For example, a database containing demographic data will need information that explains that a 1 in the gender column means the individual is a man, while a 2 indicates a woman. It is important that all metadata dependencies are identified when preparing for migration. It is acknowledged that the export of metadata from systems can be technically challenging. It is important that the time and effort required to undertake this step is not underestimated. Where it is not possible to export particular metadata, the effects on the functionality and authenticity of the records must be carefully analysed and documented. For further advice about recording metadata to manage digital records over time, see the Metadata and digital continuity: a companion guideline to the Queensland Recordkeeping Metadata Standard. Key points: Analyse and document the recordkeeping metadata that exists about the records. Plan for the mandatory capture of metadata about the migration activities to be undertaken on the record in the line with Queensland Recordkeeping Metadata Standard. Take the opportunity to improve the capture of metadata about the records, if appropriate, by influencing the design of the target system. 12 Accessible from Migrating digital records, version 1.0 page 19

20 3.2.4 Understanding characteristics of the records Continued access to a record requires preservation of its content and other information and characteristics if the record is to remain authentic, meaningful and useable. All migrations carry risk related to the loss of these characteristics, information and content. One of the aims in any migration project is to minimise the amount of loss. An ideal approach may be to preserve all the records characteristics where possible, in order to maximise future access and use opportunities. However it may not be technically or economically feasible to achieve this, nor may it be necessary. For example, a new format may not technically be able to support all features of the preceding format but retains sufficient characteristics for business purposes. Defining the characteristics of the records needs to be driven by business requirements in consideration of obligatory recordkeeping requirements. This process includes consultation with the business stakeholders and considering the access and use requirements that current and potential future users of the records may have. Characteristics that enable these requirements can then be identified and migration decisions made to ensure their maintenance. Examples If a report contains a map where colours are used to signify different agricultural areas, these colours have meaning and the report could not be interpreted accurately if these colours were lost through migration to another format. For this report, the colours are a critical characteristic and any migration performed on this report must ensure that they are maintained. An experiment in the United States sought to determine whether complex three-dimensional, geometric CAD (Computer Aided Design) records of high tolerance machined piece parts could be converted from their native CAD environment to an open source archival format to facilitate their long-term preservation. The experiment showed that, at the time of the study, it was not possible to successfully convert all aspects of the records to the non-proprietary format. The open format could not adequately represent the fine accuracy and measurement levels (down to a millionth of an inch) that were necessary to sustain the accuracy of the engineering drawing. Because the exacting engineering requirements that were a core characteristic of the records could not be reproduced, a different conversion option was required. 13 Characteristics may encompass: Content information content within the record, e.g. text, still and moving images, audio, and other intellectual productions. Context - information that describes the environment in which the content was created and managed, e.g. creator name, date of creation, file format. Structure - information that describes the extrinsic or intrinsic relationship between content, as required to reconstruct the performance of the record, e.g. s and their attachments. Appearance/rendering - any information that contributes to the re-creation of the performance of the record, e.g. font type, colour and size, bit depth. Behaviour - properties that indicate the method in which content interacts with other stimuli, e.g. hyperlinks This case study is discussed in L Duranti and R Preston, International research on permanent authentic records in electronic systems (InterPARES) 2: Experiential, interactive and dynamic records, 2008, pp , Accessible from 14 InSPECT Project (Dec 2009) Investigating the significant properties of electronic content over time: Final report. Accessible from Migrating digital records, version 1.0 page 20

21 For example, records in business systems may possess the following characteristics: 15 Content the actual content of the database tables queries used so the required content can be retrieved and represented Context metadata identifying: the name of the creator and creating organisation the business process(es) that produced the record date of creation of the record specified relationships to other records original and current file formats name and version of the query languages that are used history of the recordkeeping actions performed on the record Structure the physical structure of the database. This includes: - the tables - relationships between the tables, including the constraints - the views - field attributes the structural composition of the data as presented onscreen the logical structure of the database can be preserved in an entity relationship diagram or an XML schema Appearance/ rendering the onscreen representation Behaviour the behaviour of the user application. This may be preserved in the form of descriptions of system-supported functions, as well as screenshots of the displays used for entering and amending data, generating reports, etc. It may also be preserved in system documentation and user manuals. Table 3 Characteristics of records in business systems Where it is not possible to maintain all of the characteristics, a risk analysis of the characteristics which will and will not be preserved should be undertaken and justified, and the decisions made, approved and documented. At a minimum, characteristics should be prioritised for preservation by their capacity to ensure the authenticity, accessibility, usability and meaningfulness of the record as evidence of business activity. Key points: Assess, document and prioritise the characteristics of records. Consider the characteristics from different business perspectives and usage requirements. Ensure decisions about acceptable loss and/or change of record characteristics between the source and target systems are documented, and the consequences for useability, reliability, and authenticity of the migrated records are understood and accepted. 15 This example has been taken from the Digital Preservation Testbed Project led by the National Archives of the Netherlands. It should be noted that these characteristics are generic and are based on a specific file format type. There will be other specific characteristics unique to an agency s business that will need to be identified and managed for databases. Migrating digital records, version 1.0 page 21

22 3.3 Determining where and how to migrate Following the determination of what requires migration to the target system, the following may require consideration: the target system s recordkeeping options and capabilities the format into which the records will be migrated, and the overall approach, method and tools required to achieve the migration Verifying target system/s recordkeeping capabilities Options for migration of inactive records A key consideration in determining where the records will go and how they will be treated is whether the records are required for ongoing business purposes (active records) or can be less immediately accessible (inactive records). This guideline assumes that all active records are migrated to an online target system and that different migration options can be applied to the inactive records. The table below contains some examples with the related issues and outcomes: Source system Migration options Issues or outcomes Migrate to the new Maintain business continuity Active records online target system Preservation monitoring is less of an overhead as records are in the current online environment May be most efficient and sustainable way of managing inactive records that must be retained for longer period 1. Convert and move to a data Requires a conversion process on the records into an accessible format that is not dependent on the originating software store, repository Business system functionality will not remain or removable media Requires active monitoring of the hardware, media and file formats for potential obsolescence Inactive records Requires active monitoring and management in compliance with recordkeeping requirements under IS40 and IS31 Higher risk of loss of accessibility due to technology obsolescence 2. Retain in the existing and now off-line system Inability to decommission the existing system which carries associated ongoing costs Requires ongoing maintenance and monitoring of the obsolete business system from the hardware, software and file format points of view Requires active monitoring and management in compliance with recordkeeping requirements under IS40 and IS31 Table 4 Inactive records migration options Migrating digital records, version 1.0 page 22