Migrating digital records A guideline for Queensland public authorities June 2012 Version 1.0 Queensland State Archives Department of Science, Information Technology, Innovation and the Arts
Document details Security Classification PUBLIC Date of review of security classification June 2012 Authority Queensland State Archives Author Queensland State Archives Document Status Final Version Version 1.0 Contact for enquiries All enquiries regarding this document should be directed in the first instance to: Digital Archives Unit Queensland State Archives 07 3131 7777 info@archives.qld.gov.au Copyright Migrating digital records: a guideline for Queensland public authorities Copyright The State of Queensland (Department of Science, Information Technology, Innovation and the Arts) 2012 Licence Migrating digital records: a guideline for Queensland public authorities by Queensland State Archives is licensed under a Creative Commons Attribution 3.0 Australia Licence. To view a copy of this licence, please visit http://creativecommons.org/licenses/by/3.0/au/. Information security This document has been security classified using the Queensland Government Information Security Classification Framework (QGISCF) as PUBLIC and will be managed according to the requirements of the QGISCF. Migrating digital records, version 1.0 page 2
Executive Summary This guideline provides advice to public authorities on the key recordkeeping issues to be considered when planning for and carrying out data migration projects where the business system contains public records. Moving public records from one hardware/software configuration to another, or from one generation of computer technology to a subsequent generation, requires explicit planning to ensure that the public records maintain their authenticity and integrity in accordance with the Public Records Act 2002. Digital source records that have been migrated cannot be disposed of unless they have met certain criteria as outlined under the General Retention and Disposal Schedule for Digital Source Records (QDAN 678), and this guideline provides guidance to Queensland public authorities about the migration of digital records and the lawful disposal of the source records. The guideline assumes: familiarity with the requirements of the Public Records Act 2002 and a knowledge of the various supporting standards and guidelines, and that records management expertise is incorporated into the overall migration project planning and implementation. The guideline is organised into three parts: 1. Introduction outlines the context of migrating digital records as it is applied in the guideline. 2. Fundamental issues when migrating public records examines the challenge of evolving technology, drivers for migration, risks, obligations and the treatment of source records after migration. 3. Recordkeeping considerations for migration projects outlines key recordkeeping considerations to be incorporated into the overall migration project, including various checklist tools included as appendices. This document forms part of a wider framework of digital continuity advice under development by Queensland State Archives to support public authorities in meeting their obligations in managing their digital public records. Migrating digital records, version 1.0 page 3
Contents 1. Introduction...5 1.1 Purpose... 5 1.2 Audience... 5 1.3 Scope... 5 1.4 Authority... 5 1.5 Relationship to other Queensland State Archives publications... 6 1.6 Definitions... 7 1.7 Acknowledgements... 9 2. Important considerations for migrating public records...10 2.1 Drivers for migration... 10 2.2 Risks... 11 2.3 Obligations... 11 2.4 Treatment of source records after migration... 12 3. Recordkeeping considerations for migration projects...13 3.2 Determining what is to be migrated... 16 3.3 Determining where and how to migrate... 22 3.4 Ensuring quality... 26 3.5 Performing migration... 29 3.6 Post migration... 31 Appendices...35 Appendix A IS40 Principle 7 and the migration of digital records... 36 Appendix B Records manager s project stages flowchart... 38 Appendix C Records manager s migration project checklist... 40 Appendix D Recordkeeping controls in the target system... 53 Appendix E Testing and validation checklist...54 Appendix F Lessons learnt hints and tips... 55 Migrating digital records, version 1.0 page 4
1. Introduction 1.1 Purpose The purpose of this guideline is to assist public authorities to meet their recordkeeping obligations under the Public Records Act 2002 when they are planning and implementing data migration projects involving business systems that contain digital public records. The document also forms part of a wider framework of digital continuity advice developed by Queensland State Archives, to support public authorities in digital recordkeeping obligations. Specifically the guideline provides advice on: key recordkeeping considerations where the migrated version of the record will act as the official public record, and the correct disposal of the digital source records in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). 1.2 Audience The intended target audience for this guideline is information and records management staff responsible for advising on a public authority s recordkeeping compliance and who will be working with: project team members, business analysts, business system owners and ICT colleagues, and other key stakeholders responsible for data and business system migration projects or activities. 1.3 Scope This guideline applies to all Queensland public authorities, as defined in Schedule 2 of the Public Records Act 2002. The guideline provides support for the migration of digital public records, where the migration of the digital public records is from one hardware/software configuration to another or from one generation of information technology to a subsequent generation, for ongoing business and legislative purposes. This guideline: does not seek to provide all the technical advice on carrying out a data migration project does not replace the existing project management methodology adopted by the public authority, and does not cover the digitisation of records from a physical hardcopy format to a digital format such as when paper records are digitised. 1 1.4 Authority The State Archivist has issued this guideline in accordance with section 25(1)(f) of the Public Records Act 2002, which enables the State Archivist to make policy, standards, and guidelines about the making, keeping, preserving, managing and disposing of public records. 1 For guidance about digitisation or microfilming of records, please see the Queensland State Archives Digitisation Disposal Policy and associated toolkit, or the Microfilming Disposal Policy and guideline. Migrating digital records, version 1.0 page 5
1.5 Relationship to other Queensland State Archives publications This guideline is specifically supported by: the QSA publication Metadata for digital continuity: a companion guideline to the Queensland Recordkeeping Metadata Standard, and the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). Figure 1 Migration, metadata and retention and disposal connections It is also recommended that recordkeeping and information management staff have an understanding of the following documentation: Information Standard 40: Recordkeeping (IS40) Information Standard 31: Retention and Disposal of Public Records (IS31) Related public records briefs: Use of the Guidelines and Functional Requirements for Records in Business Systems by Queensland public authorities Identifying a Public Record in the Electronic Environment Advice on the Destruction of Public Records Decommissioning Business Systems (currently under review) Keeping digital records useable Ten steps for ensuring the continued accessibility of digital records. Online access to these documents is available on the Queensland State Archives website. It is assumed that the recordkeeping considerations detailed in this guideline will be incorporated into the public authority s own project management methodology, for example the Queensland Government Project Management Methodology 2. 2 Accessible at Queensland Government Project Management Methodologies Migrating digital records, version 1.0 page 6
1.6 Definitions Records and information management specific terms are defined in Queensland State Archives Glossary of Archival and Recordkeeping Terms available on Queensland State Archives website. Some specific terms as they are used in the context of migration processes in this guideline are described below. Migration There are many different definitions and understandings of the term migration across disciplines. Within the context of this guideline, the term migration is used to encompass the process of preserving the authenticity of records when relocating digital records from one hardware/software configuration to another, or from one generation of computer technology to a subsequent generation. 3 Source records and migrated records Source record and migrated record are terms used in this document. For clarity, source record refers to the record being migrated, and the migrated record is the record resulting from the migration, as illustrated in Figure 2 below. Old home relocation to New home Source system Target system Source records Migration Migrated records Figure 2 Migration of source records to a new target location 3 Referred to by R. Harvey (2005), Preserving Digital Materials, K. G. Saur, Munich, page 148. Originally used in Preserving Digital Information: Report of the Task Force on Archiving of Digital Information, 1996. Available at: http://www.clir.org/pubs/reports/pub63watersgarrett.pdf Migrating digital records, version 1.0 page 7
Authorised delegate As stated in Information Standard 31: Retention and Disposal of Public Records, an authorised delegate refers to the Chief Executive Officer or their appointed delegate who has responsibility for authorising the disposal of a public authority s records in accordance with an approved Retention and Disposal Schedule. Data cleansing Data cleansing involves the removal of out-of-date, duplicated, inaccurate or redundant information and is a recommended practice prior to undertaking migration activities in order to minimise the relocation of unnecessary or inaccurate information. In the context of this document, data cleansing processes are applied to the review of the data/records/metadata in the source system in order to make decisions about what data/records/metadata will be migrated into the target system. It should be noted that data cleansing activities are likely to trigger the need for an authorised disposal of time-expired records. This means that any public records identified for deletion during data cleansing in the source system prior to the migration activity will need to be disposed of in accordance with an approved Retention and Disposal Schedule. Data mapping Data mapping refers to the process of establishing connections or relationships between the data elements of the source and target systems that will enable the migration of the records and their associated metadata, and contribute to the maintenance of the authenticity and integrity of the records. Deletion vs destruction vs disposal In this guideline, these three terms have distinct meanings and are described below: Deletion in this guideline is a generalised term referring to the erasing of digital information contained in a business system. Destruction refers to the process of eliminating or deleting records that do not have continuing value, beyond the point of any possible reconstruction. Disposal of records as described in the Public Records Act (2002) includes the actions of destroying, damaging, abandoning, transferring, selling, donating or giving away a record, or part of a record. In the context of this guideline, the term disposal has been used to refer to: o the deletion of the source record, following migration, or o the deletion of eligible time-expired records prior to migration, undertaken in accordance with the terms stated in an approved Retention and Disposal Schedule. Digital continuity Digital continuity used in the context of this guideline is the ability to ensure that despite organisational, business and technological change, the access, readability and the authenticity of digital records is maintained for as long as they are required to be kept for business, legal and/or archival requirements. Digital continuity involves more than preserving digital records, and requires a range of robust records and information management policies and practices. Queensland State Archives is responsible for developing a digital continuity strategy and tools for state government and this guideline is one of the tools developed to support the continuity of digital public records. Migrating digital records, version 1.0 page 8
Encryption and decryption Encryption refers to a process where security coding is applied to data that converts it into a secured format for protected transmission across networks. To read the data the receiver must have the decryption key coding to restore the data to its original form. 1.7 Acknowledgements A range of advice was referenced in the development of this guideline. Queensland State Archives in particular acknowledges State Records New South Wales migration advice in their Managing Digital Records Guideline. 4 Queensland State Archives would like to thank the many public authorities that contributed to the development of the guideline and their representatives who also commented on the exposure draft. 4 State Records New South Wales (2009) Managing Digital Records Guideline: Effectively Manage the Migration of your Digital Records. Accessible from http://www.records.nsw.gov.au/recordkeeping/government-recordkeepingmanual/guidance/documents/recordkeepingguidelines/4%20effectively%20manage%20the%20migration%20of%20your%20digital%20records.pdf Migrating digital records, version 1.0 page 9
2. Important considerations for migrating public records Two important recordkeeping values need to be upheld when undertaking migration projects. They are: maintaining the authenticity, useability, integrity and reliability of public records for as long as they are required to be kept for business, legal and / or archival purposes, and ensuring the disposal of records is authorised and in accordance with an approved Retention and Disposal Schedule. These values have broad reaching implications for records migration projects, and the recordkeeping and information professional not only needs to have a grounding knowledge of these values but also needs to be able to translate them and have input into applying these values to the overall migration project. Before dealing with specific recordkeeping considerations in Section 3, this section will specify the: drivers for migration potential risks to records recordkeeping obligations, and the treatment of source records after migration and will assist in setting the scene for a recordkeeping and information management professional s input into a records migration project. 2.1 Drivers for migration Digital records are dependent on technology for their access and use. Over a period of time, the migration of the records from one hardware/software configuration to another may be required for the continuing access, useability and ongoing management of the public records in accordance with business, legal and archival obligations. The need to migrate public records may be triggered by a range of drivers, for example: obsolescence and technological changes that require the upgrading or decommissioning of business systems changing business needs that lead to the adoption of a new business system system and storage rationalisation, e.g. as a result of the implementation of the Application Rationalisation Methodology (ARM) 5, and / or machinery of government changes that require the transfer of functions or activities recorded in business systems from one public authority to another entity. Figure 3 Examples of drivers for migration 5 http://qgcio.govnet.qld.gov.au/govnet/architectureandstandards/methodol/pages/arm.aspx Migrating digital records, version 1.0 page 10
2.2 Risks The processes for the relocation of public records from one hardware/software configuration to new technological environment/s may threaten the legal authenticity and integrity of digital public records. To mitigate risks to the public records, migrations must be controlled by authorised and planned policies and procedures that incorporate recordkeeping considerations within the public authority s overall project management methodology. Specifically the threats and risks to records during migration include: failure to maintain the authenticity and integrity of the records (refer to Information Standard 40 for details) loss of readability and accessibility, particularly for records that are required to be kept for the longterm or in perpetuity, and non-compliance with legal obligations for the maintenance of the public records, for example through inadvertent loss or unauthorised deletion through failure to identify and manage the existence of digital public records when decommissioning a system. It is important that risk mitigation assessments and strategies relevant to recordkeeping be incorporated into planning and that ownership of risk is delegated, managed and documented. Risk mitigation strategies may include: identifying and appraising the nature of the digital records to be migrated in accordance with approved Retention and Disposal Schedules, to assist in determining the potential risk value determining and documenting the capacity of the target system to capture the required recordkeeping metadata to sustain the authenticity and integrity of the migrated records adopting robust quality assurance planning and processes. Section 3 of this guideline will highlight the key recordkeeping considerations that need to be integrated into risk mitigation planning. 2.3 Obligations The Queensland Government Recordkeeping Policy Framework sets out the recordkeeping obligations of public authorities. 6 Public authorities must meet all legislative and regulatory requirements relating to the management of records during migration projects and activities. This includes compliance with: Public Records Act 2002 this Act is concerned with ensuring the public records of Queensland are made, managed, kept and, if appropriate, preserved in a useable form for the benefit of present and future generations. It mandates a number of obligations for Queensland public authorities which include making and keeping full and accurate records of its activities, ensuring the safe custody and preservation of records in its possession, and who must authorise disposal. Information Standard 40: Recordkeeping this standard outlines seven mandatory principles for compliance by public authorities. These principles are summarised in Table 1. Appendix A provides further guidance on the application of Principle 7 in the digital records migration context. 6 Available from the Queensland State Archives website http://www.archives.qld.gov.au/ Migrating digital records, version 1.0 page 11
Principle Principle 1 Principle 2 Principle 3 Principle 4 Principle 5 Principle 6 Principle 7 Requirement Public authority recordkeeping must be compliant and accountable Recordkeeping must be monitored and audited for compliance Recordkeeping activity must be assigned and implemented Recordkeeping must be managed Recordkeeping systems must be reliable and secure Recordkeeping must be systematic and comprehensive Full and accurate records must be made and kept for as long as they are required for business, legislative, accountability and cultural purposes. Table 1 Mandatory principles of Information Standard 40: Recordkeeping Information Standard 31: Retention and Disposal of Public Records - the Public Records Act 2002 prohibits the disposal of public records without the permission of the State Archivist. The primary purpose of this Information Standard is to assist public authorities to lawfully retain and dispose of public records. This standard, among other obligations, requires that public authorities must: - dispose of public records in accordance with a Retention and Disposal Schedule approved by the State Archivist that is current at the time of disposal - ensure all disposal is endorsed by the Chief Executive or an authorised delegate - ensure the method of destruction of public records is appropriate to the sensitivity of the records and conforms with local environment regulations - document the disposal of public records. 2.4 Treatment of source records after migration Decisions about the treatment of the source records after their successful migration need to be incorporated into the migration project plan. Deletion of source records needs to be in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678), which addresses the treatment of the remaining source records after they have been migrated and enables lawful disposal. Section 3.6.1 and 3.6.2 of this guideline provide further advice about the application of retention and disposal obligations within the records migration project stages. Key points: The processes of moving records from one system to a new technological environment can threaten the authenticity, integrity, reliability and useability of digital records. To mitigate risks, migrations must be controlled by authorised and planned policies, processes and procedures that are designed to incorporate key recordkeeping standards. Source records are not automatically eligible for deletion. Certain obligations outlined in this guideline and the General Retention and Disposal Schedule for Digital Source Records (QDAN 678) need to have been met before deletion can be authorised to be undertaken. Migrating digital records, version 1.0 page 12
3. Recordkeeping considerations for migration projects This section outlines the recordkeeping considerations that need to be included in planning migration projects. The structure of this section follows a generic migration project methodology, however, it is acknowledged that public authorities will not necessarily all use the same methodology for migration projects. A public authority s approach can be based on this document but will need to be adapted to the specific project management methodology adopted by the public authority. Table 2 below summarises the considerations that are detailed further in this section. A more readily useable version of the information in this table has been prepared as a flowchart, which is available at Appendix B. Additionally, the flowchart stages have been expanded into a checklist for records managers which provides, in essence, a summary of the material in the rest of this section. The checklist is available at Appendix C. Records migration stages 1 Migration project initiation 2 Determining what is to be migrated 3 Determining where & how to migrate 4 Ensuring quality 5 Performing migration 6 Post migration Description The initial stage of high-level scheduling and resource planning for the project. The stage of the project when the scope of the migration is being established. The stage when decisions are being explored and made about the approach, method and target system(s) for the migration. The stage when quality assurance and testing and validation processes are identified, planned and undertaken. The stage concerned with the actual undertaking of the migration to extract/export, transfer and load/import the records into the target system(s). The final recordkeeping considerations prior to project closure. Recordkeeping considerations Project planning Roles and responsibilities Stakeholder communications Identifying the records Establishing retention obligations Determining recordkeeping metadata Understanding characteristics of the records Verifying target system/s recordkeeping capabilities Making file / record format decisions Establishing migration approach Preparing records for migration Testing and validation Testing roll back strategy Recordkeeping during cut over period Migration of records Quality assuring the results of the migration Verifying conditions for disposal of source records Disposal of digital source records Capture mandatory recordkeeping documentation Stakeholder relations and continuous improvement Table 2 Project stages for recordkeeping considerations Migrating digital records, version 1.0 page 13
3.1 Migration project initiation 3.1.1 Project planning A migration project that is systematically planned, initiated and carried out to well-defined procedures will help protect records and is required to ensure their evidential integrity. This guideline assumes that appropriate processes and documentation will be established to initiate a migration project by the broader organisational project team. This would include project planning documentation which clearly describes the drivers, approaches, legislative and policy requirements and expected outcomes for the project. It also assumes that appropriate change management activities are planned. It therefore does not address these planning issues in detail and assumes that the records manager will engage with the broader project. During the planning process it is important to ensure that: there is recorded acknowledgement within plans of the need to meet a public authority s recordkeeping obligations throughout the project risk mitigation planning is incorporated that is commensurate with the level of value associated with the records, and there is appropriate scheduling of time and resources for the recordkeeping activities associated with the migration of digital public records. Key point: Ensure recordkeeping obligations, standards and activities are well understood by the project team and are sufficiently incorporated into the planning and scheduling for the project. 3.1.2 Roles and responsibilities As outlined in section 2.2 migrations can be a high risk activity for records. Those responsible for the management of records must be involved in all records migration projects. This includes projects to migrate records stored in any business system, not just for projects that involve dedicated recordkeeping systems such as electronic document and records management systems (edrms). 7 When migration projects involving records are established within a public authority, and roles and responsibilities are planned, records and information managers are well placed to contribute expertise to the project through: assisting with the identification and appraisal of the records within business systems, how long the records must be legally retained and which records are eligible for lawful disposal determining how the records are being managed within the existing system and their current and ongoing recordkeeping compliance needs 7 In this guideline business systems refer to automated systems that have a primary purpose of facilitating business transactions (e.g. finance, licensing, client-relationship systems, etc.) and that do not necessarily contain records management functionality. This is distinct from electronic document and records management systems (edrms) that are designed with records management functionality to manage the records it captures throughout their lifecycle. Migrating digital records, version 1.0 page 14
facilitating the process for the authorised disposal of records supporting quality assurance of migrated records in collaboration with business area experts and others assisting in the data mapping process between the source and target systems to ensure records, including their metadata, remain full and accurate records of business and that their authenticity can be verified advising on the recordkeeping controls needed in the design and configuration of the target system developing and delivering training and awareness materials and sessions related to records management, and advising on and promoting continuous improvement in recordkeeping functionality within new systems. 3.1.3 Stakeholder communication planning As mentioned above, this guideline assumes that appropriate change management activities are planned as part of the overall project management plan. However it is important that those responsible for records and information management engage with the overall communications strategy and the proposed timings for the migration of the records. This is needed in order to understand and plan for the potential impact on recordkeeping requirements, and on the business and its stakeholders, during the time of the records changeover period between the source system and the target system. (Please note: Appendix F provides some real-life insights into recordkeeping considerations during this time. Refer to F.3.) There will be a need to ensure that appropriate training of staff is undertaken. This may require targeted promotion and the development of new procedures relating to the management of the records in the newly migrated environment. Key points: Those with responsibility for records management should be engaged and consulted in migration projects to ensure that recordkeeping considerations are integrated into the overall project management planning. The key objective of records/information managers being involved is to support the achievement of maintaining the authenticity and evidential integrity of the public records throughout the migration process and to assist with retention and disposal compliance. Migrating digital records, version 1.0 page 15
3.2 Determining what is to be migrated A key decision point in a migration project is determining what information will be migrated. To determine this, there are some fundamental details about the records themselves that must first be established. Understanding these details will help to inform decisions about what data to migrate and what data does not need to be migrated. The processes for determining this are described below. 3.2.1 Identifying the records Records are the consequence or product of a business activity. Digital records are not simply data but comprise a complex set of related information which includes the: content of the record contextual metadata, for example, information about who created the record, how it has been managed, relationships to other records, etc, and the structure that enables the record to be accessible and meaningful, e.g. database table headings. Knowing what the records are within a system is critical to protecting and maintaining their completeness, integrity and authenticity. It is important to clearly scope and document them prior to undertaking a migration. Depending on the nature of the migration project, the records to be migrated may be easily identified and can be clearly described. However, identifying records in business systems can often be challenging as they are generally complex and are not always packaged as neatly discrete objects. The process of determining what records are contained in business systems involves undertaking an analysis of the business processes associated with the system s use, in order to identify the information that records the evidence of business activity and transactions. For further advice on this process see Part 2.3 of the Guidelines and Functional Requirements for Records in Business Systems. 8 This process of analysis will also highlight ephemera within the system, including any duplicated copies of information. Ephemera are items of short-term value that are not required to be kept as records and therefore may be able to be disposed of, if not required for other business or legal purposes, prior to the migration. Key point: Know what records there are and describe them to ensure the records can be successfully maintained throughout and following the migration process. 8 The use of this guideline has been endorsed by the State Archivist and is accessible from Queensland State Archives website at http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/guidelines_functional_requirements.pdf. Also refer to the public records brief at http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/prb_ica_project.pdf. Please note that in 2008 the International Council on Archives (ICA) published three modules of its Principles and Functional Requirements for Records in Electronic Environment (ICA-Req). These modules have since been adopted by the International Standards Organisation and published as ISO 16175. Migrating digital records, version 1.0 page 16
3.2.2 Establishing retention obligations The retention periods for records are documented in authorised Retention and Disposal Schedules. 9 Once the records have been identified, an authorised schedule can be consulted and applied to determine how long they must be kept. Some records may have passed their retention period and can be deleted, which will reduce the volume of records requiring migration. IMPORTANT: The deletion of records must be in accordance with the authorised disposal requirements. It is also important to consider other aspects that may apply to the records prior to deletion, such as whether a disposal freeze 10 is in place or another business or legislative reason overrides the Retention and Disposal Schedule/s. For example, are the records in question potentially needed in evidence in an existing judicial proceeding, including any reasonably possible judicial proceeding? When deletion is to be undertaken in the source system prior to migration, all disposal actions must be documented in accordance with requirements under Principle 2 of Information Standard 31: Retention and Disposal of Public Records. For further information about disposing of records, see section 4 of the Guideline for the Implementation of Retention and Disposal Schedules. 11 The disposal of public records must be authorised by the State Archivist under section 26 of the Public Records Act 2002. Such authorisation is usually given in a Retention and Disposal Schedule. In the event the identified records are not covered by an authorised Retention and Disposal Schedule, records cannot be deleted prior to migration. Key points: Identify the lawful retention period for the records through consultation with authorised Retention and Disposal Schedule/s and consider any other legal obligations e.g. records subject to litigation, discovery process. After establishing the lawful retention period, if there are no longer any legal, business or legislative requirements to retain the records, undertake an authorised process to dispose of any records that have met their minimum retention period. If there is no authorised Retention and Disposal Schedule covering the migrated records, disposing of them is unlawful. 9 See http://www.archives.qld.gov.au/recordkeeping/retentiondisposal/pages/default.aspx for further information. 10 A disposal freeze is an authority issued by the State Archivist that requires the temporary cessation of the destruction of public records, relating to a specific topic or event, that are covered by an approved Retention and Disposal Schedule. A disposal freeze, issued by the State Archivist for a particular group of public records, overrides an existing Retention and Disposal Schedule that applies to the records of a public authority for a specified period of time. Refer to http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/disposal_freeze_policy.pdf. 11 Accessible from http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/guideline%20for%20the%20implementation%20of %20Retention%20and%20Disposal%20Schedules.pdf Migrating digital records, version 1.0 page 17
3.2.3 Determining recordkeeping metadata Recordkeeping metadata is a fundamental tool for creating and maintaining full and accurate records and is a key means by which the integrity and authenticity of a record can be proven. A key feature that differentiates recordkeeping metadata from other types of metadata is that it is not a static profile of a document or other resource. Recordkeeping metadata initially defines a record at the point of capture, but is also dynamic and accrues through time, to provide information on how a record has been used or managed. This characteristic of recordkeeping metadata is essential for preserving the authenticity of records. Safeguarding metadata relationships Effective migration of records requires that the associated metadata including the connections between a record and its metadata are persistently maintained. This requires an understanding of the records existing metadata. Documenting this metadata will help to highlight all the necessary data that must be migrated, particularly metadata which records the information about relationships and connections between data. Relationships requiring safeguarding may be: structural that is within the document, such as a document containing a linked spreadsheet or images between records e.g. between records documenting related aspects of business between records and/or containers e.g. documents aggregated to files/folders between records and other entities e.g. between records and creating agents between records and control tools such as Business Classification Schemes, Retention and Disposal Schedules, access and security controls and mandates. Some of this information may be outside the system in question, for example, the Business Classification Scheme, however processes will need to be established to ensure the relationships are successfully maintained with the migrated records. Once the metadata and records are understood and documented, the process of mapping the metadata from source to destination system is essential to ensure that all the necessary metadata is carried over to the target system and remains useable and linked to the records it describes. This will contribute significantly to the maintenance of the records context and authenticity. Metadata and mapping analysis is an opportunity to address any insufficiencies and/or apply improved metadata capture that ensures that the minimum recordkeeping metadata requirements are met. In some migrations the date of the migration can become the default date for all recordkeeping actions, including date of creation of records. This severely impacts recordkeeping processes such as disposal and undermines the authenticity, useability and integrity of the records. Checks should therefore be in place to ensure dates that trigger recordkeeping actions in the source system remain accurate and unchanged post migration. Migrating digital records, version 1.0 page 18
The Queensland Recordkeeping Metadata Standard and Guideline (QRKMS) can be used as a tool to help identify the metadata that requires migration, and the links that must be maintained. For example The table below contains columns to identify the QRKMS required metadata, the relevant field in the current system and the target field in the new system. The comments column can be used to note any system limitations or issues that require resolution before migration. QRKMS element Current system Target system Comments Identifier Transaction reference Transaction ID Check if ID format is supported by new system, or whether new format is required Date created Transaction date Date Must ensure dates carry across as is from old system and are not updated A metadata implementation matrix is available that can be adapted for this purpose. 12 Capture metadata about migration When migration of digital records occurs, it is a key event in the lifecycle of the records. Metadata must be captured to document the process of the migration of the record. The mandatory Record Event History element within the Queensland Recordkeeping Metadata Standard requires documenting the preservation, retrieval, disposal, control, access or use related activities performed on a record. This metadata provides a visible and auditable trail of records management actions and decisions, and can be used to record the migration activities on the record. Other metadata considerations In addition to specific recordkeeping metadata mentioned above, the system may hold other metadata that is crucial to the meaning of the records. For example, a database containing demographic data will need information that explains that a 1 in the gender column means the individual is a man, while a 2 indicates a woman. It is important that all metadata dependencies are identified when preparing for migration. It is acknowledged that the export of metadata from systems can be technically challenging. It is important that the time and effort required to undertake this step is not underestimated. Where it is not possible to export particular metadata, the effects on the functionality and authenticity of the records must be carefully analysed and documented. For further advice about recording metadata to manage digital records over time, see the Metadata and digital continuity: a companion guideline to the Queensland Recordkeeping Metadata Standard. Key points: Analyse and document the recordkeeping metadata that exists about the records. Plan for the mandatory capture of metadata about the migration activities to be undertaken on the record in the line with Queensland Recordkeeping Metadata Standard. Take the opportunity to improve the capture of metadata about the records, if appropriate, by influencing the design of the target system. 12 Accessible from http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/qrkmsmatrix.xls. Migrating digital records, version 1.0 page 19
3.2.4 Understanding characteristics of the records Continued access to a record requires preservation of its content and other information and characteristics if the record is to remain authentic, meaningful and useable. All migrations carry risk related to the loss of these characteristics, information and content. One of the aims in any migration project is to minimise the amount of loss. An ideal approach may be to preserve all the records characteristics where possible, in order to maximise future access and use opportunities. However it may not be technically or economically feasible to achieve this, nor may it be necessary. For example, a new format may not technically be able to support all features of the preceding format but retains sufficient characteristics for business purposes. Defining the characteristics of the records needs to be driven by business requirements in consideration of obligatory recordkeeping requirements. This process includes consultation with the business stakeholders and considering the access and use requirements that current and potential future users of the records may have. Characteristics that enable these requirements can then be identified and migration decisions made to ensure their maintenance. Examples If a report contains a map where colours are used to signify different agricultural areas, these colours have meaning and the report could not be interpreted accurately if these colours were lost through migration to another format. For this report, the colours are a critical characteristic and any migration performed on this report must ensure that they are maintained. An experiment in the United States sought to determine whether complex three-dimensional, geometric CAD (Computer Aided Design) records of high tolerance machined piece parts could be converted from their native CAD environment to an open source archival format to facilitate their long-term preservation. The experiment showed that, at the time of the study, it was not possible to successfully convert all aspects of the records to the non-proprietary format. The open format could not adequately represent the fine accuracy and measurement levels (down to a millionth of an inch) that were necessary to sustain the accuracy of the engineering drawing. Because the exacting engineering requirements that were a core characteristic of the records could not be reproduced, a different conversion option was required. 13 Characteristics may encompass: Content information content within the record, e.g. text, still and moving images, audio, and other intellectual productions. Context - information that describes the environment in which the content was created and managed, e.g. creator name, date of creation, file format. Structure - information that describes the extrinsic or intrinsic relationship between content, as required to reconstruct the performance of the record, e.g. e-mails and their attachments. Appearance/rendering - any information that contributes to the re-creation of the performance of the record, e.g. font type, colour and size, bit depth. Behaviour - properties that indicate the method in which content interacts with other stimuli, e.g. hyperlinks. 14 13 This case study is discussed in L Duranti and R Preston, International research on permanent authentic records in electronic systems (InterPARES) 2: Experiential, interactive and dynamic records, 2008, pp. 34-35, Accessible from http://www.interpares.org/display_file.cfm?doc=ip2_book_complete.pdf 14 InSPECT Project (Dec 2009) Investigating the significant properties of electronic content over time: Final report. Accessible from http://www.significantproperties.org.uk/inspect-finalreport.pdf Migrating digital records, version 1.0 page 20
For example, records in business systems may possess the following characteristics: 15 Content the actual content of the database tables queries used so the required content can be retrieved and represented Context metadata identifying: the name of the creator and creating organisation the business process(es) that produced the record date of creation of the record specified relationships to other records original and current file formats name and version of the query languages that are used history of the recordkeeping actions performed on the record Structure the physical structure of the database. This includes: - the tables - relationships between the tables, including the constraints - the views - field attributes the structural composition of the data as presented onscreen the logical structure of the database can be preserved in an entity relationship diagram or an XML schema Appearance/ rendering the onscreen representation Behaviour the behaviour of the user application. This may be preserved in the form of descriptions of system-supported functions, as well as screenshots of the displays used for entering and amending data, generating reports, etc. It may also be preserved in system documentation and user manuals. Table 3 Characteristics of records in business systems Where it is not possible to maintain all of the characteristics, a risk analysis of the characteristics which will and will not be preserved should be undertaken and justified, and the decisions made, approved and documented. At a minimum, characteristics should be prioritised for preservation by their capacity to ensure the authenticity, accessibility, usability and meaningfulness of the record as evidence of business activity. Key points: Assess, document and prioritise the characteristics of records. Consider the characteristics from different business perspectives and usage requirements. Ensure decisions about acceptable loss and/or change of record characteristics between the source and target systems are documented, and the consequences for useability, reliability, and authenticity of the migrated records are understood and accepted. 15 This example has been taken from the Digital Preservation Testbed Project led by the National Archives of the Netherlands. It should be noted that these characteristics are generic and are based on a specific file format type. There will be other specific characteristics unique to an agency s business that will need to be identified and managed for databases. Migrating digital records, version 1.0 page 21
3.3 Determining where and how to migrate Following the determination of what requires migration to the target system, the following may require consideration: the target system s recordkeeping options and capabilities the format into which the records will be migrated, and the overall approach, method and tools required to achieve the migration. 3.3.1 Verifying target system/s recordkeeping capabilities Options for migration of inactive records A key consideration in determining where the records will go and how they will be treated is whether the records are required for ongoing business purposes (active records) or can be less immediately accessible (inactive records). This guideline assumes that all active records are migrated to an online target system and that different migration options can be applied to the inactive records. The table below contains some examples with the related issues and outcomes: Source system Migration options Issues or outcomes Migrate to the new Maintain business continuity Active records online target system Preservation monitoring is less of an overhead as records are in the current online environment May be most efficient and sustainable way of managing inactive records that must be retained for longer period 1. Convert and move to a data Requires a conversion process on the records into an accessible format that is not dependent on the originating software store, repository Business system functionality will not remain or removable media Requires active monitoring of the hardware, media and file formats for potential obsolescence Inactive records Requires active monitoring and management in compliance with recordkeeping requirements under IS40 and IS31 Higher risk of loss of accessibility due to technology obsolescence 2. Retain in the existing and now off-line system Inability to decommission the existing system which carries associated ongoing costs Requires ongoing maintenance and monitoring of the obsolete business system from the hardware, software and file format points of view Requires active monitoring and management in compliance with recordkeeping requirements under IS40 and IS31 Table 4 Inactive records migration options Migrating digital records, version 1.0 page 22
The chosen approach will be influenced by a range of business, recordkeeping and technical issues. An important initial step to undertake is to understand how the records are controlled in the current environment, for example whether recordkeeping controls are built into the system itself, or whether the records are exported and managed through a separate recordkeeping system. Ability of target system to meet recordkeeping obligations Whichever target system is selected, it is important to ensure there are adequate recordkeeping controls to manage the records in line with Information Standards 31 and 40. Refer to Appendix D for a checklist on the key obligations. The Guidelines and functional requirements for records in business systems 16 provides additional guidance. Retention periods influencing target system decisions The retention and disposal obligations attached to the records may influence decisions about the migration approach. For inactive records that only need to be kept for relatively short periods of time beyond the planned migration, transfer into a target online business system and subsequent disposal of them once their retention period is reached may be the most appropriate choice, as migrating or storing them elsewhere may not be as cost-effective. Retention in the old system may also be appropriate, provided there is a commitment to sentence and dispose of these records when they are lawfully able to be disposed and to keeping the legacy system alive so the records remain findable and actionable until they can be disposed of. However leaving records in legacy systems for extended periods without active management to ensure they remain reliable, useable and authentic will not meet a public authority s obligations under the Public Records Act 2002, nor necessarily comply with other legislation such as the Right to Information Act 2009. 3.3.2 Making file / record format decisions Repeated migrations can compromise the integrity, reliability, useability and authenticity of records, particularly those that are required to be retained for long periods of time. To minimise the number of migrations, conversion of those records that are no longer needed for active business purposes and have lengthy retention periods into an open file format which is not dependent on the originating software is recommended. 17 Open formats are stable, longer-term formats that are widely used. They are usually non-proprietary, platform-independent formats with freely available specifications. 18 The QSA Public Records Brief Advice on choosing file formats for the management of digital records contains more detailed information about choosing file formats. 16 The use of this guideline has been endorsed by the State Archivist. Refer to the public records brief at http://www.archives.qld.gov.au/recordkeeping/grkdownloads/documents/prb_ica_project.pdf. Please note that in 2008 the International Council on Archives (ICA) published three modules of its Principles and Functional Requirements for Records in Electronic Environment (ICA-Req). These have now been endorsed by the International Standards Organisation and published as ISO 16175 parts 1 to 3. 17 It is acknowledged conversion is not the only longer-term preservation approach. There are other techniques such as emulation that can be applied. 18 State Records New South Wales (2009) Managing Digital Records Guideline: Effectively Manage the Migration of your Digital Records, accessible from http://www.records.nsw.gov.au/recordkeeping/government-recordkeepingmanual/guidance/documents/recordkeepingguidelines/4%20effectively%20manage%20the%20migration%20of%20your%20digital%20records.pdf Migrating digital records, version 1.0 page 23
Some examples of open standard formats include: PDF/A ODF HTML XHTML JPEG TIFF PNG FLAC Portable document format/ archive OpenDocument Format Hypertext Markup Language Extensible Hypertext Markup Language Joint Photographic Experts Group Tagged Image File Format Portable Network Graphics Free Lossless Audio Codec Table 5 Examples of open standard formats This approach still requires an active and ongoing commitment to monitoring and managing these migrated records to ensure their ongoing access. Appendix F provides an example of a record format conversion decision made by a public authority when it was required to decommission a business system. Key points: Ensure that the target system/s have recordkeeping capabilities and/or will meet the responsibilities for retention management. Plan to minimise the need for record migrations by determining a target system/s and record format(s) that suit the records retention periods and record types. Understanding the retention requirements, conversion formats and their associated stability also involves an active and ongoing commitment to their monitoring and maintenance. 3.3.3 Establishing a migration approach There may be a number of approaches required to successfully migrate records. 19 After determining what is to be migrated as described in section 3.2 and considering the points raised above, a documented and approved plan needs to be devised which comprehensively outlines the approaches to be taken for all of the information in the business system. The plan should describe: both the source and target business system formats for the information/records the tools that will be used to achieve the migration details of any manual intervention activities security requirements for the information business rules and quality assurance procedures, and roll back strategy for mitigating risks to the records if affected by an unforseen migration event. 19 Refer to Appendix F for examples of approaches taken by some public authorities. Migrating digital records, version 1.0 page 24
The migration approaches should be appropriately selected based on defined requirements, informed by the analysis of the records and recordkeeping obligations. Technological tools - Migrations may be facilitated by technological tools for extracting the data from the existing system, transforming the data and loading the data into the target system. For example the conversion of records to stable open standard formats can be achieved through a range of freely available tools such as the National Archives of Australia s Xena. 20 Manual intervention - Migration processes are likely to involve some manual intervention supported by appropriate business rules and quality assurance procedures. For example, if the mapping of information from the existing system to the target system proved complex, manual data entry of some of the information into the new system may be an appropriate approach when it is supported by appropriate checking processes to ensure the accuracy and completeness of the entered data. Key points: Document and secure approval for the plans that describe what will happen to all of the information within the system including the corresponding security classifications. Select tools which can best facilitate the maintenance of full and accurate records throughout the migration process. Ensure migration approaches, including any transforming of the data, loading and any reliance on manual extraction, is supported by clear business rules and quality checking processes. Ensure the approach includes a tested roll back strategy, as part of risk mitigation planning. 20 Xena is free and open source software developed by the National Archives of Australia to converts records to an XML-based archival data format. Xena is an acronym meaning Xml Electronic Normalising for Archives. See http://xena.sourceforge.net/ for further information. Migrating digital records, version 1.0 page 25
3.4 Ensuring quality For a record to retain its evidentiary value, it must be possible to demonstrate that migration has resulted in a full and accurate reproduction of the source record which has not been altered in an accidental or unauthorised way since it was made. The General Records Disposal Schedule for Digital Source Records (QDAN 678) requires that digital source records are retained until quality assurance has been undertaken and the results accepted. Trusted systems and quality assurance processes are therefore a critical component of migration activities. 3.4.1 Preparing records for migration Prior to migration, it is important to take the opportunity to ensure that all records and metadata are as accurate and up-to-date as possible. This will improve the efficiency and quality of the information migrated. A migration project provides the opportunity to cleanse and maximise the access to data, for example through: Reviewing the location metadata about records when migrating between recordkeeping systems, for example to ensure people who have left the organisation are not recorded as holding records. Ensuring all records to be migrated are in the source system. If there is a high likelihood that there are records which should be migrated stored on shared directories, email, or personal and any other systems outside the identified system to be migrated, awareness activities can be scheduled to encourage transfer of the records into the source system prior to the migration. Ensuring any records stored in encrypted forms are decrypted as appropriate, as maintaining unnecessary encryption poses significant risks to the continuing useability and readability of public records over time. As migration can be a resource-intensive activity, it may also be appropriate to minimise its impact where relevant, by deleting any records that have met their minimum lawful retention period prior to the performance of the migration activity, as outlined in section 3.2.2. Any data cleansing activity undertaken should be documented and decisions to dispose of records prior to migration endorsed and documented as required by IS31, to help ensure the authenticity of the records and to provide justification for decisions to dispose of existing records prior to migration. Key points: Strive to minimise the overheads associated with migration processes by ensuring records and metadata are accurate and up-to-date and undertaking sentencing projects prior to migration to dispose of records that have met their minimum retention periods. Document any disposal decisions arising from data cleansing activities prior to migration, and ensure the documentation is retained as required by the General Retention and Disposal Schedule for Administrative Records (QDAN 249 ver. 6). Migrating digital records, version 1.0 page 26
3.4.2 Testing and validation To have assurance that migrated records retain their full and accurate attributes, testing and validation that includes recordkeeping considerations must be undertaken, both pre-migration in a test environment and post-migration in the live system. This will help to minimise and mitigate any unanticipated and unacceptable results at the go live stage of the new system. A thorough and documented testing and validation process that is undertaken successfully also provides valuable assurances to a third party as to the authenticity, reliability, integrity and useability of the records. It is also required before source records can be disposed of lawfully in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). The recordkeeping and information professional s role and responsibilities through the testing and validation stages will need to be defined and incorporated into the overall project management planning. Appropriate tests to be undertaken to ensure the quality of the records will need to be devised in collaboration with the wider project team. A knowledge and understanding of the source and target states of the records, as set out in sections 3.2 and 3.3, will assist in this process. In addition, a guide to the key recordkeeping related testing and validation checks that should be considered is provided in Appendix E. The process will involve working with the project team, to establish and document baseline information about the records themselves and establish recordkeeping benchmark standards and allowances. Testing procedures and processes should include steps to follow when validation or testing falls short of the standards set. For example, where more than an agreed percentage of the total records and the required recordkeeping metadata examined are found to be defective, the project migration phase pauses and a review of technical migration elements are undertaken and issues resolved before resuming the migration project. The level of risk to the business if the authenticity of a particular class or group were not effectively maintained influences the testing nature and scope. For example, testing approaches could include: scripted and manual testing of migrated information within a business process context; sampling a broad range, percentage or block of migrated records for validation; analysis and confirmation that migration mapping outcomes have been achieved; log and error-message checking, etc. Under the General Retention and Disposal Schedule for Administrative Records there are a number of record classes that require long term retention with conditional criteria that applies. This would influence the nature and scope of testing and validation required. The level of validation required should be considered in the context that includes the possibility of any future activities that could bring the authenticity of the migrated records into question. This is in addition to the ability to be able to recall and access them in a useable format. The following are two examples of types of records that would need to have a risk analysis applied to determine the scope and level of testing and validation required: a workers compensation incident report concerning an individual that did not involve an immediate claim that would need to be held on the individual s personnel file and retained for 70 years from date of birth of the individual or seven years from date of separation, or resignation, whichever is later; records relating to the removal and disposal of hazardous materials (including asbestos and radioactive material) from buildings or structures which require retention for 70 years after removal of the hazardous materials. In both of the above cases, if there was a thorough and documented testing and validation process this would assist in providing evidence to justify the authenticity, reliability and integrity of the records to a third party if this were ever required into the future. Migrating digital records, version 1.0 page 27
The extent of testing and validation should therefore be commensurate with the risks associated with the records retention requirements and the level of on-going importance required to justify their authenticity for the life of the records. IMPORTANT: The time and resources needed to undertake testing and validation is often underestimated. It is important that the records manager factors some flexibility into planning for this time-consuming and resource intensive activity to ensure that it can be undertaken adequately. Pre-migration testing is strongly recommended to be undertaken within a test environment and allows for a revised strategy to be devised in the event that adverse effects are observed. For example, the test may result in an undesirable outcome in its connection with other integrated software systems and these unanticipated results may be unacceptable to the business and put a halt to the migration until a solution is devised as part of the overall migration planning. If this were to occur once the new system had gone live, it could have serious adverse affects on the stakeholders confidence and business operations. Best practice recommends, and project management methodology calls for, pre and post migration testing and validation results to be documented and signed off by the appropriate project delegate at both the pre and post migration phases. As mentioned previously, this documentation will provide some assurance that the right practices were undertaken and results anticipated were validated and/or remediation was implemented. Key points: A thorough and documented testing and validation process must be undertaken and signed-off so that assurances can be made to a third party that the authenticity, reliability, integrity and useability of the records have been maintained. In collaboration with the broader project team members, contribute to the inclusion of recordkeeping baseline standards and the target results/outcomes. Ensure the extent, range and types of tests are commensurate with the risks and value associated with the records. Be detailed in your checking for any unanticipated results or consequences from the testing or migration including any unacceptable affects on integrated systems. Be aware that time and resources are often underestimated for testing and validation processes. Ensure that planning includes flexibility of time and resources for this important element. Migrating digital records, version 1.0 page 28
3.5 Performing migration 3.5.1 Testing roll back strategy Prior to migration, it is important that a roll back strategy has been established and thoroughly tested. Sometimes, despite comprehensive pre-migration efforts and particularly with complex migrations, unexpected outcomes can occur in migration exercises. To mitigate the risks to the records, due to unexpected and unacceptable outcomes, having a tested roll back option means that records remain protected and business processes can be resumed with the old system until another migration attempt is made. 3.5.2 Recordkeeping during cut over period Business arrangements will need to be made to maintain recordkeeping continuity and compliance during the period when the migration activities are performed up to when the target system becomes live and available for use. For example certain business transactions may need to be suspended or recorded manually until the new system is available, with any updates made or repeated in the new system once it is live. 3.5.3 Migrating records The process of migration will involve some element of extraction/export, transformation and import/loading of data into the target system. As outlined in section 3.2.3, metadata should be captured to record the actions carried out on the data as it progresses through this extraction, transformation and loading process. The process of migration in collaboration with members of the broader project team should be undertaken with the minimum of intervention and time delay as possible. This helps to demonstrate an unbroken chain of custody and can help to prove the authenticity and reliability of the migrated records, particularly in the event of legal proceedings. The process should be designed to prevent irreversible activity so in the event that problems arise the migration can be rolled back as necessary. To further strengthen the evidential integrity which may be required to be demonstrated in the event of legal proceedings, documentation should include the actions taken to validate that the particular equipment and systems used in the migration are in good operating order. Once records have been successfully migrated and all testing/validation processes accepted it is important that stakeholder expectations are clearly communicated to the business. For example, it should be clear that the source record is no longer the official record and business rules and processes should be established to ensure that the source records cannot be altered inadvertently or continue to be used for documenting business activity. Further guidance about the management of source records postmigration is detailed in sections 3.6.1 and 3.6.2. Migrating digital records, version 1.0 page 29
Key points: Ensure that a roll back strategy is included in planning and has been tested prior to undertaking migration activities. Record information about the process of migration that includes recordkeeping continuity arrangements during the cut over period. In collaboration with the broader project team, keep time delays and interventions to a minimum when moving records from the existing system to the target system to strengthen evidential integrity of the records and reduce data updates or repeated actions needed in the new system. Ensure business rules and processes are put in place to protect the authenticity of the records. 3.5.4 Quality assuring the results of the migration It is necessary to validate the results of the complete migration of the records into the live environment. This is similar in its process to preliminary testing and validation undertaken in the test environment, prior to go-live, as described in section 3.4.2. Once again the results should be assessed, documented and a report compiled and signed off by the assigned project team delegate prior to the completion of the project. This report provides essential context to records that have undergone migration and is a key event in the lifecycle of digital records. This documentation should indicate, at a minimum, that the migrated records were inspected, that metadata and characteristics were compared to the original records and were intact in their entirety, that the records were properly migrated, and that all necessary control procedures were applied during the process. Only after this report attesting to the successful migration of the records has been signed-off, should the disposal authorisation for the digital source records be activated. Public authorities are encouraged to maintain this documentation for the life of the record, in order to help demonstrate the authenticity of the migrated records. Key points: Testing and validation of recordkeeping functionality post-migration is similar to the process undertaken pre-migration and requires the same thoroughness. Only after the post-migration testing and validation report has been signed-off, should the disposal authorisation for the digital source records be activated. Migrating digital records, version 1.0 page 30
3.6 Post migration Migration project initiation Determining what is to be migrated Determining where & how to migrate Ensuring quality Performing migration Post migration 3.6.1 Verifying conditions for disposal of source records Migration should not immediately cause the destruction of the source records that are the subject of, or input to, those processes. The source records generally continue to exist in tandem with the output of the migration process. An informed business decision therefore needs to be made about when it is appropriate to dispose of the source records. The General Retention and Disposal Schedule for Digital Source Records (QDAN 678) sets out the minimum conditions for the retention of source records. However the period of time for which the source records might require further retention beyond the minimum, should be established by individual public authorities through a documented risk assessment as necessary. When conducting the risk assessment, public authorities must consider any other legal or business continuity issues that may influence the duration for the further retention of the source records. These issues could include, but are not necessarily restricted to, the following: Verifying conditions for disposal - risk assessment considerations business purpose, litigation potential and value of the records and the risks associated with this, including the potential business, financial and legal implications of any loss of trustworthiness or access level of assurance that full and accurate records have been achieved through the migration, including the maintenance of the completeness and authenticity of records level of assurance that the migrated record is being well managed in a robust recordkeeping environment thoroughness of the migration processes, including quality assurance processes and the size and complexity of the migration extent of usage and level of integration into the operational environment of the records. (Please note: Records that are likely to be immediately used after migration may require a shorter retention period as use will quickly highlight any problems. Those records that are unlikely to be used in the immediate period following migration, may need to be considered for a longer retention that fits into a quality evaluation and validation plan before disposal.) Table 6 Risk assessment considerations prior to disposal During the established period of retention, source records must be stored and protected to ensure that they remain accountable, well-managed records of business that can be used again as appropriate source records, should there be a need to roll back and re-initiate migration if issues arise. Also during this retention period, isolation of the original source records from further access by general stakeholders is highly recommended to eliminate the possibility of their inadvertent use. Migrating digital records, version 1.0 page 31
It is recommended that the authorised deletion or disposal of source records be performed in a timely way as part of good business practice in the context of the closure of the migration project. Key points: Source records cannot be deleted until the conditions outlined in the General Retention and Disposal Schedule for Digital Source Records (QDAN 678) are met. Undertake a risk assessment to validate the need to retain source records for a longer period than specified in the General Retention and Disposal Schedule for Digital Source Records (QDAN 678) Make sure processes are in place to ensure retained source records cannot be confused with or used for working business transactions. Deletion of source records at the end of the authorised retention period should be scheduled as part of the project plan closure. 3.6.2 Mandatory requirements before disposing of digital source records The disposal of public records including digital source records requires the authorisation of the State Archivist under section 26 of the Public Records Act 2002. Authorisation for the disposal of digital source records following a successful records migration activity is given through the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). Before digital source records are deleted under the General Retention and Disposal Schedule for Digital Source Records (QDAN 678), public authorities must ensure that: agreed quality assurance procedures as described in sections 3.4 and 3.5 have been conducted and any remediation need has been satisfied, the verification of the conditions for disposal readiness as outlined in 3.6.1 have been met, and their eligibility for disposal signed off by the Chief Executive Officer or their authorised delegate in line with IS31. Once the above conditions have been satisfied, the source records can be deleted under the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). The disposal of source records must be documented. 21 For disposal of the source records in a migration project, document at system level information about the disposal of the digital source records. Approval/endorsement and sign-off of the disposal of the source records can, for example be retained as an administrative record in the public authority s EDRMS. Note that the General Retention and Disposal Schedule for Administrative Records (QDAN 249) requires that master disposal documentation be retained permanently (reference 9.4.2). Public authorities should ensure that the source records are securely deleted once the approved retention period has expired so that they cannot be retrieved or recreated. While some business systems have recordkeeping functionality built-in that includes workflow for an authorised destruction process, there are many business systems that do not have this specific utility and secure deletion involves a separate process and documentation. To cover a public authority s specific circumstances, it is recommended that the secure and irretrievable deletion process be incorporated into the overall project planning and be devised in consultation with the broader team for the project. 21 See: http://qgcio.qld.gov.au/qgcio/architectureandstandards/informationstandards/current/pages/is31- Implementationadvice.aspx Migrating digital records, version 1.0 page 32
As detailed in section 3.6.1, performing a migration results in the existence of two versions of the same record the source record and the new migrated record (Refer to Figure 3). Maintaining two records is not cost effective and may cause confusion as to which record is the official record of the business. Digital source records should therefore be deleted once quality assurance and risk assessment procedures have been completed in line with the General Retention and Disposal Schedule for Digital Source Records QDAN 678. At a minimum they should be isolated and made inaccessible for general use. Key points: Source records that have been subject to migration to a new target system can only be destroyed in accordance with the General Retention and Disposal Schedule for Digital Source Records (QDAN 678). Quality assurance procedures and validation must be completed and signed off by the authorised delegate prior to deletion. Disposal must be endorsed by the Chief Executive Officer or their authorised delegate and information about the source records disposal retained as required by IS31 and the General Retention and Disposal Schedule for Administrative Records (QDAN 249 ver. 6) Source records must be securely destroyed so that they cannot be retrieved or recreated. 3.6.3 Capturing mandatory recordkeeping documentation An essential part of all project management methodologies is the documentation and sign-offs on the finalisation of the process to confirm that the project was completed successfully. In the case of the migration of public records the sign-offs need to include sign-off on the recordkeeping considerations. This is important because if migrated records were ever called into question during legal proceedings, it may be necessary to certify that the migration processes were reliable and produced accurate and authentic records. Generally, the greater the risk associated with the need to rely on the records in court or in business transactions, the greater the need to make and keep adequate documentation that attests to the accuracy and authenticity of the records. A range of documentation will have been produced throughout the project in accordance with the public authority s project management methodology. As a guide for records and information managers, documentation of the aspects of the migration project outlined in the Table below is important for recordkeeping purposes to provide assurance about the integrity of the records treatment throughout the migration process and the results achieved. Guide to recordkeeping documentation: descriptions of the identified records and their characteristics, and the metadata being migrated along with reports associated with decisions made any data cleansing performed, including records of decisions and the processes undertaken the formal migration plans and processes the dates/times of the migration activities and the people involved, including their roles and positions system configurations, including metadata definitions and mappings risk assessments and mitigation planning and activities testing and quality assurance plans and results, pre and post migration Migrating digital records, version 1.0 page 33
Guide to recordkeeping documentation: any reports that compare original system functionality to target system functionality all sign-offs/approvals and all decisions, including any decisions not to migrate certain data any variations to plans around the recordkeeping considerations any necessary variation in records structure, metadata, format or content that will or has resulted from the migration the disposal of the source records including approved assurances that the minimum conditions were met and the appropriate further period of retention had elapsed. Table 7 Guide to documentation requirements Key points: Projects should not be closed until there are appropriate assurances that the records have been successfully migrated and the original source records have been attended to in accordance with the authorised plan. Documenting the recordkeeping considerations and results applicable to the records migration process is important as this helps to demonstrate the integrity and authenticity of the records in the future. 3.6.4 Stakeholder relations and continuous improvement While the migration project may have been completed and signed off by the business, the work of those responsible for records and information management continues. The following pointers highlight some of those areas for consideration. Staff training Although this is not dealt with in detail in this guideline, an important aspect arising from migration projects for a records and information manager to consider is the need to ensure that appropriate training of staff is undertaken. This may require targeted promotion and the development of new procedures relating to the management of the records in the new migrated environment. Ongoing monitoring To ensure that the migrated records continue to be maintained as expected, and as a mechanism for identifying opportunities for recordkeeping improvements, ongoing monitoring of system usage and the system s treatment of the records is required. Lessons learnt and continuous improvement opportunities There will be many lessons learned from the migration project that can be applied to future projects to improve the process and outcomes of records migration projects. By documenting the learnings for potential future reference and having records management expertise involved in business system procurement and implementation projects, ongoing recordkeeping considerations can be included and ongoing business benefits can be achieved. Key points: Migration projects are an opportunity to improve recordkeeping practices. Recordkeeping training material may need updating. Monitor ongoing usage of the system to ensure records continue to be managed in accordance with requirements. Migrating digital records, version 1.0 page 34
Appendices Attached supporting materials for the records and information manager To further assist recordkeeping and information professionals in their engagement with recordkeeping considerations during a migration project, various supporting documents are appended here as quick reference materials. These are referred to in the text where appropriate and include: Appendix A IS40 Principle 7 and the migration of digital records a guide Appendix B Appendix C Appendix D Appendix E Appendix F Records manager s project stages flow chart Records manager s migration project check list Recordkeeping controls in the target system a summary Testing and validation check list Lessons learnt migration hints and tips for recordkeeping professionals Migrating digital records, version 1.0 page 35
Appendix A IS40 Principle 7 and the migration of digital records The following is a guide to the application of IS40 Principle 7 to the migration of digital records. To be a full and accurate record a public record must, amongst other requirements, be: Attribute Standard meaning Digital migration context Complete Meaningful Accurate Authentic Inviolate Complete records comprise contextual and structural data as well as content. Contextual data is information about the creation and use of the record, including the business processes that give rise to it. Structural data includes formal internal structures of the information object and the structural relations between records. Content is the information contained within the information object. Meaningful records may be understood in the context of the processes and business for which they are created and in which they are used. An accurate record reflects what was communicated, decided or done (or not done). That is, the record s content, context and structure can be trusted as a true and accurate representation of the transactions, activities or facts that they document and can be depended upon in the course of subsequent use. An authentic record is a record that can be proven and trusted to be what it purports to be and to have been created, used, transmitted or held by the person to whom these actions have been attributed. The maintenance of authentic records is facilitated by well-planned, comprehensive and successful migration processes and metadata capture. Inviolate records are time-bound (that is they are fixed at a point in time) and complete. To be inviolate, a record must be securely maintained to prevent alteration and unauthorised removal. Where alteration or changes must occur, the inviolate nature of the records can be protected through the capture of metadata about the alterations. A complete migrated record is an accurate replica of the source record, including all the record s content, context and structure i.e. metadata. For digital records to be meaningful, metadata that records the history of the recorded information s use is required to ensure it is understood in a factual context. An accurate migrated record is one that remains a reliable representation of the business activity after the migration. An authentic migrated record is one where it can be demonstrated, particularly through the capture of metadata about the migration process, that it maintains its trustworthiness as being what it purports to be. The requirement for authenticity means that any migration must be managed and documented with care so that the reasons for the process are explicit, the method used to validate the quality of the record following the process is clear, and the metadata detailing who did what and when is recorded and maintained. An inviolate migrated record is one where the content data acknowledges its timebound elements (fixed at the point in time and evidence of business activity it is recording) and where it can be demonstrated that no unauthorised changes have been made to it. If changes are authorised, these changes and the authorisation are recorded in metadata. Accessible Accessible records are records that can be An accessible migrated record is one that is captured in a format, media and system Migrating digital records, version 1.0 page 36
Attribute Standard meaning Digital migration context Useable identified, located and viewed as required. Useable records are those that may be viewed and are functional and re-useable. in which the current technologies will enable timely retrieval and readable access of that migrated record A useable migrated record is one that can be used with the appropriate level of functionality required to provide a factual and reliable interpretation of the source record. For example, if the hyperlinks do not successfully work after migration, and these links have been determined as being an essential characteristic of the record, then the record is not useable. Migrating digital records, version 1.0 page 37
Appendix B Records manager s project stages flowchart Project planning 1. Migration project initiation Roles and responsibilities Stakeholder communications Identifying the records 2. Determining what is to be migrated Establishing retention obligations Determining recordkeeping metadata Understanding the characteristics of the records 3. Determining where and how to migrate Verifying target system/s recordkeeping capabilities Making file/record format decisions Establishing migration approach Migrating digital records, version 1.0 page 38
Preparing records for migration 4. Ensuring quality Testing and validation Testing roll back strategy 5. Performing migration Recordkeeping during cut over period Migration of records Quality assuring the results of the migration 6. Post migration Verifying conditions for disposal of source records Disposal of digital source records Capture of mandatory recordkeeping documentation Stakeholder relations and continuous improvement Migrating digital records, version 1.0 page 39
Appendix C Records manager s migration project checklist Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website 1. Migration project initiation The initial stage of highlevel scheduling and resource planning for the project. Project planning ensure that recordkeeping obligations and activities are well understood by the project team and are incorporated into the project brief and other project initiation documentation e.g. project stage plans. recordkeeping matters for consideration include: 3.1.1 o o risk mitigation planning is commensurate with the level of value associated with the records e.g. vital records, permanent value records, etc appropriate scheduling of the associated records related activities is incorporated into the project stage plan (as listed in the rest of this check list). Roles and responsibilities Records management role may include: 3.1.2 the appraisal / identification of records, how long the records must be retained and which records are eligible for lawful disposal determining whether the system is currently managing the records or whether this is through another recordkeeping system or combination of systems contribute to the mapping process between the existing and target system to ensure recordkeeping elements are included, including the relevant metadata to maintain the authenticity of the records and that they will be Migrating digital records, version 1.0 page 40
Appendix C Records manager s migration project checklist Stages for records manager involvement Key recordkeeping considerations full and accurate in their characteristics Guideline section reference Supporting guidance available on QSA Website contribute to the design or configuration of the target system to ensure that adequate recordkeeping controls are established facilitate the process for the authorised disposal of eligible records participate in testing and validation processes support the process of the quality assurance of migrated records provide training and awareness in records management procedures advocate for continuous improvement of recordkeeping functionality within business system/s Stakeholder communications engage with the overall project s communications strategy 3.1.3 ensure that recordkeeping considerations are integrated into planning e.g. in communications and system training Migrating digital records, version 1.0 page 41
Appendix C Records manager s migration project checklist Stages for records manager involvement 2. Determining what is to be migrated The stage of the project when the scope of the migration is being established. Key recordkeeping considerations Identifying the records undertake an analysis of the business processes identify information that records evidence of business transactions (the records) Guideline section reference Supporting guidance available on QSA Website 3.2.1 Public Records Brief Identifying a Public Record in the Electronic Environment go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/identifying_pub_record_e_e nviron_200503.pdf Part 2.3 Guidelines and Functional Requirements for Records in Business Systems go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/guidelines_functional_requir ements.pdf Establishing retention obligations disposal of public records must be authorised and given under a Retention and Disposal Schedule that has been approved by the State Archivist for use by the public authority. determine and document the lawful retention obligations that apply to the records in accordance with the authorised Retention and Disposal Schedule/s applicable to the public authority. determine which records have passed their retention period and that are eligible for authorised disposal. after establishing the lawful retention period, and if time and resources allow and there are 3.2.2 Guideline for the Implementation of Retention and Disposal Schedules go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/guideline%20for%20the%2 0Implementation%20of%20Retenti on%20and%20disposal%20sched ules.pdf Information Standard 31: Retention and Disposal of Public Records go to http://qgcio.qld.gov.au/qgcio/archit ectureandstandards/informationsta ndards/current/pages/retention%2 0and%20Disposal%20of%20Public Migrating digital records, version 1.0 page 42
Appendix C Records manager s migration project checklist Stages for records manager involvement Key recordkeeping considerations no other legal, business or legislative requirements to retain the records, undertake an authorised process to dispose of any records that have met their minimum retention period. if disposal of eligible records has occurred prior to migration, capture metadata about the records that have been disposed in accordance with IS31. The documentation about the disposed public records must be retained permanently. Guideline section reference Supporting guidance available on QSA Website %20Records.aspx General Retention and Disposal Schedule for Administrative Records go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/generaldisposalschedule.p df Guideline for the Implementation of Retention and Disposal Schedules go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/guideline%20for%20the%2 0Implementation%20of%20Retenti on%20and%20disposal%20sched ules.pdf Determining recordkeeping metadata analyse and document the recordkeeping metadata that exists about the records plan for the capture of metadata about the migration activities to be undertaken on the records 3.2.3 Queensland Recordkeeping Metadata Standard and Guideline (QRKMS) go to http://www.archives.qld.gov.au/rec ordkeeping/grkdownloads/docu ments/qrkms.pdf if possible and appropriate, improve the capture of metadata about the records by contributing to the design and configuration decisions about the target system. Metadata and digital continuity: a companion guideline to the Queensland Recordkeeping Metadata Standard. Understanding the characteristics of the records analyse and document the characteristics of the records including: content, context, structure, appearance/rendering, behavioural 3.2.4 Migrating digital records, version 1.0 page 43
Appendix C Records manager s migration project checklist Stages for records manager involvement Key recordkeeping considerations properties (e.g. interactions with other software/information, hyperlinks, etc). consider the characteristics from different business perspectives and usage requirements. Guideline section reference Appendix A Appendix F.2 Supporting guidance available on QSA Website advise on the priority of the characteristics that need to be carried over with the migration. at a minimum, those characteristics that contribute to the authenticity, access, usability and meaningfulness of the records should be preserved. undertake a risk analysis of the characteristics which will or will not be carried over, and justify and document decisions about acceptable change and/or loss. Migrating digital records, version 1.0 page 44
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website 3. Determining where and how to migrate The stage when decisions are being explored and made about the approach, method for the migration to the target system(s). Verifying target system/s recordkeeping capabilities in collaboration with the relevant project team members, explore the migration options for active and inactive records and ascertain how recordkeeping compliance can be maintained for each. gain an understanding of the ability of target system/s to meet recordkeeping obligations. note that retention periods that apply to the records may have an influence on target system decisions. 3.3.1 Information Standard 31: Retention and Disposal of Public Records go to http://qgcio.qld.gov.au/qgcio/architectur eandstandards/informationstandards/c urrent/pages/retention%20and%20dis posal%20of%20public%20records.as px Information Standard 40: Recordkeeping go to http://qgcio.qld.gov.au/qgcio/architectur eandstandards/informationstandards/c urrent/pages/recordkeeping.aspx Making file / record format decisions minimise the need for record migrations by determining record format(s) that take into account the records retention periods and the record types 3.3.2 Appendix F.1 Public record brief is under development Establishing migration approach Contribute to the: documentation and approval of the plans that describe what will happen to all the information within the system including the corresponding security classifications 3.3.3 Appendix F.3 selection of tools which can best facilitate the maintenance of full and accurate records throughout the migration process development of the business rules and quality checking processes to support the chosen migration approach/es, including any transforming of the data, loading and any reliance on manual extraction. Migrating digital records, version 1.0 page 45
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website 4. Ensuring quality The stage when quality assurance, including testing and validation processes, are identified, planned and undertaken. Preparing records for migration Contribute to the minimisation of the overheads associated with migration processes by: ensuring records and metadata are accurate and up-to-date undertaking sentencing projects prior to migration to dispose of records that have met their retention periods 3.4.1 documenting any data cleansing and disposal activities in accordance with the requirements of IS31. Testing and validation Contribute to the: 3.4.2 development of testing that incorporates the maintenance of the authenticity of the full and accurate records extent, range and types of testing that is commensurate with the risks and value associated with the records documenting of the testing and validation plans, including baseline information and target results / outcomes engagement of appropriate recordkeeping skilled people within the migration stage sign-off process by ensuring that where appropriate recordkeeping elements are signed-off by an authorised delegate in addition to any other authorised sign-offs Appendix E Migrating digital records, version 1.0 page 46
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website provision of advice on adequate time and resources to be allocated to the testing and validation of recordkeeping elements including the recommendation of conducting testing on copies of records within a test environment prior to migration documentation of the recordkeeping elements of the testing and validation processes and outcomes. Migrating digital records, version 1.0 page 47
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website 5. Performing migration The stage concerned with the actual undertaking of the migration to extract / export, transfer and load / import the records into the target system(s). Testing roll back strategy check that roll back strategy has been tested prior to migration exercise Recordkeeping during cut over period prior to performing the migration, ensure that procedures are in place to protect recordkeeping elements during the period of cut over to the new system 3.5.1 3.5.2 Appendix F as some system transactions may have been suspended or recorded manually during this period, factor in time and resources to perform the system updates Migration of records Contribute to: 3.5.3 documentation of the recordkeeping elements of the migration process engagement of appropriate recordkeeping skilled people within the migration stage development of recordkeeping business rules and the establishment of subsequent processes to protect the authenticity and integrity of the records supporting the minimisation of time delays and interventions. Quality assuring the results of the migration Similar to the testing and validation stage above, contribute to the: 3.5.4 development of testing that incorporates the maintenance of the authenticity of the full and Migrating digital records, version 1.0 page 48
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website accurate records extent, range and types of testing that is commensurate with the risks and value associated with the records documenting of the testing and validation plans, including baseline information and target results / outcomes engagement of appropriate recordkeeping skilled people within the migration stage sign-off process by ensuring that where appropriate recordkeeping elements are signed-off by an authorised delegate in addition to any other authorised sign-offs provision of advice on adequate time and resources to be allocated to the testing and validation of recordkeeping elements including the recommendation of conducting testing on copies of records within a test environment prior to migration documentation of the recordkeeping elements of the testing and validation processes and outcomes. Migrating digital records, version 1.0 page 49
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website 6. Post migration The final recordkeeping considerations prior to project closure Verifying conditions for disposal of source records Contribute to ensuring that recordkeeping considerations are incorporated into: processes that eliminate any business user confusion, access or potential inadvertent use of the now redundant source records 3.6.1 General Retention and Disposal Schedule for Digital Source Records review and update the risk level attached to the records that have been subject to migration, based on the success of the validation process, and confirm the lawful period of retention for the original source records left behind in the old home location, and once conditions for retention have been met, source records can be deleted in accordance with the General Retention and Disposal Schedule for Digital Source Records Disposal of digital source records source records can only be destroyed in accordance with the General Retention and Disposal Schedule for Digital Source Records 3.6.2 General Retention and Disposal Schedule for Digital Source Records quality assurance procedures must be completed and signed off by the appropriate delegate disposal of the source records must first be endorsed by the Chief Executive Office or their authorised delegate disposal must be documented and documentation must capture the minimum metadata requirements as outlined under IS31 Migrating digital records, version 1.0 page 50
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website once these conditions have been satisfied, source records must be securely destroyed so that they cannot be retrieved or recreated Capture mandatory recordkeeping documentation the greater the risk associated with the need to rely on the records in court or in business transactions, the greater the need to make and keep adequate documentation that attests to the accuracy and authenticity of the records the disposal of source records post migration must be documented in line with IS31 projects should not be closed until there are appropriate assurances that the recordkeeping considerations have been satisfied 3.6.3 Information Standard 31: Retention and Disposal of Public Records go to http://qgcio.qld.gov.au/qgcio/architectur eandstandards/informationstandards/c urrent/pages/retention%20and%20dis posal%20of%20public%20records.as px documenting the migration projects recordkeeping considerations and the associated results will help to demonstrate the integrity and authenticity of the records Stakeholder relations and follow-up with your stakeholders to confirm continuous improvement system results and gauge user adaptation 3.6.4 ensure that training and awareness materials are up-to-date ensure that communications and awareness sessions are completed document the recordkeeping related lessons learned from the migration project Migrating digital records, version 1.0 page 51
Stages for records manager involvement Key recordkeeping considerations Guideline section reference Supporting guidance available on QSA Website monitor ongoing usage and management of the system and the treatment of the records by the new system and its business users contribute to continuing education of staff in recordkeeping matters continue to promote opportunities for business process improvement in recordkeeping matters maintain relationship/s with system owner and business users and ensure that recordkeeping responsibilities continue to be assigned Migrating digital records, version 1.0 page 52
Appendix D Recordkeeping controls in the target system Key recordkeeping considerations for the target system can protect the integrity, authenticity and context of the records This includes the maintenance of recordkeeping metadata connections. Migrating records without effective recordkeeping controls, for example migrating records onto a shared drive, may not capture adequately the contextual metadata (e.g. author, version history, true date of creation, etc) required to maintain the authenticity and integrity of the records. is reliable and secure Under Information Standard 40, all systems used to create or maintain records must work reliably and be secure to ensure that records are credible and authoritative. Public authorities are therefore required to implement systems which are secure from unauthorised access, damage and misuse. This requires supporting processes and tools that prevent the corruption or loss of the digital records, e.g. backup/restore procedures, disaster recovery procedures, sampling of media and digital records to detect deterioration and corruption, and the refreshing of media. An ability to ensure the reliability and security of the records will also influence choices made about the storage medium of records. For example, storage of records on RAID (Redundant Array of Independent Disks) servers may be considered more secure and reliable than storage of records on removable media such as CDs, or DVDs. can enable access to the records can preserve the characteristics of the records Under section 14 of the Public Records Act 2002, public authorities controlling records that can only be produced or made available with the use of particular equipment or information technology must take all reasonable action to ensure the information remains able to be produced or made available. At a minimum, because of technology obsolescence, the selected system should have the ability to easily export records so that at a future point in time when the system s lifespan has been reached and decommissioning occurs, the records are able to be transferred again to a successor for continued access if required. Some approaches will enable easier and wider access to records than others. For example, records in offline systems that become legacy systems are arguably less accessible than other online records or records stored in accessible file formats. To be able to access a record requires that first the records are structured and stored in a way that they can be searched for and are discoverable. The ability of the system to facilitate discoverability would factor in the selection of a target system, as a system that does not enable discoverability prevents access to the records. For example, converting records and storing them in a data store that does not provide adequate searching facilities will not enable access to the records. The lifespan of the system and associated technologies, software licensing arrangements or any other constraints which may limit access, should also be considered when selecting an appropriate target system. The characteristics of the records must be maintained throughout a migration process, as mentioned in section 3.2.4. The ability to maximise the extent of the characteristics that can be preserved may influence the choice of file format or system. Should a particular file format not be able to maintain, for example, the appearance of the content of the record, and the appearance is critical to the meaning of the record, it may not be an appropriate format to choose. Migrating digital records, version 1.0 page 53
Appendix E Testing and validation checklist 22 Details Guideline reference if applicable Check accuracy of the content and metadata of the records. For example: - expected fields are populated with data, e.g. mandatory value 3.2.3 - field values are in keeping with expected lengths and valid date ranges - values are populated with expected data types (e.g. numeric, alphanumeric, date, etc.) - metadata values remain accurate, particularly those such as the date of creation and dates which trigger disposal countdowns Appendix F.2 - accurate metadata about the migration process itself is captured 3.2.4 Check for completeness that all records identified for migration, have been migrated and that the completeness of the records themselves is verified. For example: - numbers of records remains the same Appendix A - linkages/relationships between metadata and records are maintained Appendix F.2 - results of standard reports and searches are repeatable, consistent, and accurate Check for maintenance of the integrity and authenticity of the records. For example: - all agreed characteristics of the records as previously identified for migration during the planning process are maintained - any errors or corruption during transmission are flagged, e.g. as identified through checksums an IT computational process that can highlight errors that might have occurred during the migration process. 3.2.4 Check for meaningfulness and the access and usability of the records. For example: - real time checks of a sample of the records to ensure they remain readable and understandable Appendix A Check the reliability of the system/s including functionality, processes, and integrated system/s and peripherals. For example: - after roll-out the system is stable and operating within acceptable error log margins that have been established in collaboration with IT colleagues 22 A guide only. The individual example items in this checklist should be adapted and expanded to suit the public authority s specific migration project and developed in collaboration with IT colleagues. Migrating digital records, version 1.0 page 54
Appendix F Lessons learnt hints and tips The following hints and tips have been provided by records and information managers from public authorities that have been involved with records migration projects. F.1 File format decisions When migrating documents from an old system that was marked for decommissioning, single page tiff files were identified that were bound together by the proprietary software. Once migrated, these documents would have separated into single pages causing many access issues and creating thousands more documents than were necessary. The public authority was able to copy them, optical character recognition (OCR) them, convert them to single (multi-page) PDF documents prior to placing them into the new edrms, along with the treatment history of these records to ensure their authenticity could be verified. Overall there were significant benefits to the business in undertaking this exercise. These included: considerable improvements to the searchability and accessibility of these records, and maintenance of the digital continuity of the records. F.2 Complexities of determining what is to be migrated and what is to be tested and validated Identifying the full characteristics of the records within the business system and their dependencies and relationships with data elements can be complex. Records managers need to work with their IT colleagues to ensure that the records to be migrated will retain their authenticity and remain full and accurate. Decisions made need to be justified and documented to ensure that the integrity of the records can be substantiated. For example: Data elements in a database record may be: mandatory or not constrained to specific data types (e.g. date, text or number) and character lengths limited to a set of allowed values, and/or default values. There may also be complex editing rules. For example, if element A has a value of X then element B may be restricted to a particular set of values, etc. These attributes should be documented and included in validation testing. Sets of allowed values are often rationalised or updated in migration processes. These changes and the justifications for these changes should be documented in the migration process. Relationships between records can be constrained. They may be one to one, one to many, many to many or reflexive. For example, an employee may be the manager or supervisor of another employee. These constraints may be important in the design of the migration processes and the validation testing of the new system. Migrating digital records, version 1.0 page 55
F.3 Approaches to migrating digital records and the recordkeeping implications Within the planning phases for a migration project, a roll back strategy is highly recommended in order to mitigate any risks that may occur if unacceptable deficiencies, despite pre-migration testing, come to light after the live/production migration. The roll back strategy should itself be tested and validated before final migration takes place, and should include recordkeeping functionality testing. In addition, the following migration approaches require careful consideration and action planning by the records manager in collaboration with their IT colleagues: A cut over strategy could be planned. Ideally use of the source system should cease and the source system be frozen before migration commences. Business transactions may be suspended or recorded manually until the data is migrated and the new system is up and running. If the system is critical to the functioning of the business, a snapshot may need to be taken. Data additions, changes or deletions or functions performed must be logged so the changes can be repeated in the new system. Certain non critical functions in the source system might be suspended during the cut over period to simplify this process. For example, only new records may be allowed to be entered. Changes to existing records may be manually recorded and performed in the new system once the initial migration is validated and signed off. New records could be migrated to the new system in a second step to synchronise the two systems once the new system is signed off as functional and the source system frozen. Migrating digital records, version 1.0 Migrating digital records, version 1.0 page 56