DIGITAL ARCHIVING DISCUSSION PAPER: INFORMING AN APPROACH TO THE LONG-TERM MANAGEMENT AND PRESERVATION OF GOVERNMENT DIGITAL RECORDS

Transcription

1 DIGITAL ARCHIVING DISCUSSION PAPER: INFORMING AN APPROACH TO THE LONG-TERM MANAGEMENT AND PRESERVATION OF GOVERNMENT DIGITAL RECORDS May 2010 Queensland State Archives Department of Science, Information Technology, Innovation and the Arts

2 Document details Security Classification PUBLIC Date of review of security classification May 2010 Authority Author Document Status Queensland State Archives Queensland State Archives Final Version Version Version 1.0 Contact for enquiries All enquiries regarding this document should be directed in the first instance to: Queensland State Archives Copyright Digital Archiving Discussion Paper: Informing an approach to the long-term management and preservation of Government digital records Copyright (c) The State of Queensland (Department of Science, Information Technology, Innovation and the Arts) 2010 Licence Digital Archiving Discussion Paper: Informing an approach to the long-term management and preservation of Government digital records by Queensland State Archives is licensed under a Creative Commons Attribution 2.5 Australia Licence. To view a copy of this licence, please visit Information security This document has been security classified using the Queensland Government Information Security Classification Framework (QGISCF) as PUBLIC and will be managed according to the requirements of the QGISCF. 2

3 CONTENTS 1 Introduction Background Purpose and audience Introducing digital archiving 5 2 The context for digital preservation Changing nature of Government business Public records and their legislative context 7 3 Understanding digital archiving and digital archives Scope of the challenge Key issues Impact of not addressing the problem 11 4 Developing a solution Standards for a digital archive Examples from international and national practice Challenges for agencies 14 5 The Way Forward Program elements Program phases Guiding principles 18 Appendix A: List of questions 19 Appendix B: Examples of digital preservation tools 20 Appendix C: Open Archival Information System Reference Model 22 Appendix D: Selected International and National Examples 24 Appendix E: Glossary 26 3

4 1 Introduction 1.1 Background About Queensland State Archives Queensland State Archives is the lead agency for recordkeeping across Queensland s public authorities and is the custodian of the State s largest and most significant collection of (nondigital) government records. As mandated under the Public Records Act 2002, the powers of the State Archivist include establishing and managing repositories and other facilities to store, preserve, exhibit and make available for use public records and other materials, and to undertake research and issue policy advice on contemporary recordkeeping and information management issues. In September 2009, the Toward Q2 through ICT 1 Strategy was released, which outlines the Queensland Government s information and communication technology priorities and targets to help create more accessible, efficient, and effective services for the benefit of all Queenslanders. Under the associated implementation plan, Queensland State Archives (QSA), assisted by the Queensland Government Chief Technology Office (QGCTO) and Queensland Health are responsible, by the end of 2010, for reviewing the current digital archiving methods across the government, and developing an approach to this issue, including the potential for a Queensland digital archive. This Discussion Paper is an initial step in this project. It is anticipated the paper will be followed by a survey of public authorities later in 2010, to assist in ascertaining the specific digital archiving methods in use across Government, and inform the development of recommendations on a suitable way forward for the long-term management and preservation of digital records across the Queensland Government. 1.2 Purpose and audience The purpose of this Discussion Paper is to bring attention to the importance of the complex problem of digital preservation and to seek feedback on how Queensland public authorities are responding, or preparing to respond, to the challenge of ensuring access to digital records over time. It seeks views from public authorities on appropriate plans, principles and elements to inform a coordinated whole-of- Government research and development program. The program will seek to define and establish how Queensland Government electronic records can be preserved over time, including the development of a future Queensland Government Digital Archive. Digital preservation is not simply a technology issue but has wider business, policy and economic impacts for an organisation. Therefore, Chief Executive Officers and Senior Management Teams, in consultation with Information Steering Committees and Chief Information Officers, are requested to consider and provide coordinated feedback on the issues raised in this Discussion Paper. Questions are highlighted throughout the Discussion Paper and summarised in Appendix A. A feedback form can be downloaded from Queensland State Archives website at DigitalArchivingDiscussion.asp. Comments are required by Tuesday, 15 June Key terms used in this paper are defined in Appendix E. Other recordkeeping terms are defined in Queensland State Archives Glossary of Archival and Recordkeeping Terms, available on Queensland State Archives website at: GlossaryOfArchivalRKTerms.pdf 1 Available from TowardQ2throughICT.pdf 4

5 1.3 Introducing digital archiving One of the key issues facing executives in the digital age is the long-term preservation and management of digital information to support organisational sustainability, accountability and decision-making effectiveness. With information being the lifeblood of any organisation, it is essential to develop and implement strategies to ensure that information remains accessible for as long as required. A 2006 survey of government bodies, scientific research organisations, pharmaceutical, environmental, nuclear, engineering and financial institutions in the United Kingdom revealed that the loss of electronic data is commonplace, with more than 70% of respondents saying data had been lost in their organisation. 2 Digital information requires early intervention by trained professionals to ensure its survival in context. Government archival authorities are increasingly taking the lead in this work, recognising their responsibility to set standards for government recordkeeping and ensure the preservation of government records over time. Queensland State Archives works collaboratively with other archival authorities through the Australasian Digital Recordkeeping Initiative (ADRI) and with the International Council on Archives (ICA). Such collaboration gives QSA access to major developments in these areas. Around the world many governments are investing in the infrastructure required for long-term preservation, including research initiatives such as the PLANETS (Preservation and Long-Term Access through Networked Services) project funded by the European Union; research and advocacy organisations, such as Digital Preservation Europe; and the development of relevant standards, in addition to the development of digital archiving facilities. However, in Australia there has been a relative under-investment in digital preservation to date. 2 Digital Preservation Coalition (2006) Mind the Gap: Assessing digital preservation needs in the UK. Available at 5

6 2 The context for digital preservation 2.1 Changing nature of Government business Government business is increasingly being conducted electronically. For example, the Council of Australian Governments (COAG) agenda includes a number of national information-sharing initiatives, including a proposed e-conveyancing solution and ehealth initiatives. The roll-out of the National Broadband network will also see an increased push for electronic content. All of these initiatives in addition to actions taken by individual agencies to improve services through the use of new technology mean that the volume and complexity of information is increasing exponentially. Research indicates there was 281 exabytes of information created, captured or replicated in digital form around the world in 2007 and that this volume of information is increasing. Indeed, by 2011, the digital universe is expected to be 10 times the size it was in The community is also driving a shift to increased electronic service delivery. As shown in Figure 1 below, in 2008, over two in five Australians nominated online as their preferred service delivery channel. 4 Figure 1: Preferred service delivery channel 3 IDC White Paper (2007) The Expanding Digital Universe: A Forecast of Worldwide Information Growth through Available at Universe_IDC_WhitePaper_ pdf. An exabyte is one million terabytes. It has been estimated that 10 terabytes could hold the printed collection of the Library of Congress. 4 Australian Government Information Management Office (2008) Interacting with Government: Australians use and satisfaction with e-government services. Available at interacting-with-government-report.pdf 6

7 Common misconceptions Electronic public records Electronic public records are not just those that have been created through the process of digitisation or scanning. They include records that have been born-digital, for example those created in business systems and online, e.g. licensing systems, financial management systems, human resource management systems, websites, blogs and other social media technologies. Public and publicly -available records Public records are not only records that are available to citizens. The term public should be understood in the same broad sense as public sector ; not as in open to the public. Under the Public Records Act 2002, a public record is any record received or kept by a public authority in exercise of its statutory, administrative or other public responsibilities, or for a related purpose. Back-up Back-up involves copying systems in case of system failure. The back-ups are only kept for a defined period to support business continuity or disaster recovery purposes, in order that system can be restored to the last known good copy. Back-ups are not a substitute for records management systems and strategies for managing the electronic records over their authorised retention period. Scope of digital archiving The ICT community often refers to digital archiving as the process of moving data from expensive storage to cheaper storage. Use of the term in this way does not reflect the same focus as understood by the archival and information and records management sectors, and as used in this paper. 2.2 Public records and their legislative context Electronic records are not new, but for a long time organisations considered that the official record was the paper version. This is no longer the case. According to the Public Records Act 2002, a record is: recorded information created or received by an entity in the transaction of business or the conduct of affairs that provides evidence of the business or affairs. This definition covers records in all forms, including digital. As business is conducted electronically, the records of this business will be in digital form. It is true that some simple electronic records, such as word-processed documents, can be printed and managed as paper. But as the volume and complexity of digital information increases, a print to paper mindset is not feasible or sustainable. Common systems in use across Government include financial management systems, case management and licensing. There are also more specific and specialised forms, for example: Queensland Fire & Rescue Service manage and retain electronic building plans Queensland Police utilise digital technology to record fingerprints Queensland Mines and Energy has sponsored hyperspectral surveys, generating GIS image maps, to help explorers evaluate their mineral deposit models. The day-to-day management of digital records does present challenges. Queensland State Archives, through its Recordkeeping Policy Framework, issues guidelines and advice on many aspects of managing digital records, such as the Queensland Recordkeeping Metadata Standard and Guideline 5 and Guidelines and Functional Requirements for Records in Business Systems. 6 However, there is also a need for wider strategic solutions to ensure that records can be retained appropriately in accessible formats over time. The retention and disposal of public records is governed by the Public Records Act The periods for which records need to be retained are set out in Retention and Disposal Schedules authorised by the State Archivist. There is no default retention period. The retention period for each group of records created and kept by an agency is based on an analysis of the business and legislative requirements for that type of record. Retention periods can vary from as little as when reference ceases or one year after last action to 50, 75 or 100 years. Some types of records are to be retained permanently, due to their ongoing legal, historical or community importance. These permanent records ultimately form part of the archival record of the State, coming into the collection of Queensland State Archives. 5 Available from 6 Available from 7

8 3 Understanding digital archiving and digital archives Digital archiving aims to preserve the accessibility of digital records through time. In the analogue world, the rate of degradation or depreciation of an asset is usually not rapid. Consequently, decisions about the long-term preservation of these materials can often be postponed for a considerable period, especially if they are kept in appropriate conditions. The digital world does not afford a similar luxury. Degradation and obsolescence of hardware, software and media can be rapid as new technology replaces older versions, driven by faster, more productive and higher capacity devices, with improved functionality. This obsolescence presents a challenge for access to and preservation of digital information. For example: File formats While most software is upgraded on a regular basis, and generally the new application will read the files created by the previous versions, the ability to read older versions is often not supported over time. Over time, files that have not been migrated to newer formats, because they are no longer active for business purposes, may no longer be accessible by the latest version of the software, and the older version software may no longer be available, or may not run on a current computer, or under a current version of the operating system, thereby rendering the files inaccessible. 7 Storage media Digital information stored on media where refreshment of the information to new storage media is not actively undertaken, may become inaccessible over time because devices needed to read the storage medium are no longer manufactured. For example, the evolution of personal computers means that current generations no longer support floppy disks. 8 New risks continue to emerge: for example the loss of control of storage networks inherent in the use of some Web 2.0 services and cloud computing. Many electronic records will need to be retained for far longer than the life of a system or storage medium. For example, some records of working with children checks blue cards need to be kept for 100 years from the date of birth of the applicant, far longer than the life of the current Employment Screening System database. 7 Digital Preservation Management Tutorial: Implementing Short-term Strategies for Long-term Problems, Cornell University Library 8 See the Digital Preservation Management Tutorial s Chamber of Horrors, available from 8

9 A key difference between data centres and digital archives: A data centre is not a digital archive. A data centre provides a controlled storage facility for electronic information. While a digital archive must include storage (and may use a data centre service for this purpose), the management and control of information required in a digital archive is not included in current storage services or storage technologies. 3.1 Scope of the challenge Strategies and solutions need to be developed to ensure that digital records can remain accessible, authentic and useable for their full retention period. Two aspects of digital archiving need to be considered: Preservation of temporary records for ongoing business purposes, and Permanent preservation for archival purposes. While the preservation of temporary records for business purposes presents a more immediate problem for many organisations, public authorities are also asked to turn their attention to the longer-term issues which need urgent action now if the archival records of the state are to be safeguarded into the future. Queensland State Archives has a dual legislative role: as custodian of the State s archival collection and a leadership role with regards to recordkeeping. Recognising this dual role, it is intended that the future program of work will incorporate both the development of a Queensland Government Digital Archive and provision of advice on the preservation of temporary records. While these challenges are broadly similar, there are some differences. Preservation for archival purposes involves the records becoming part of the Queensland State Archives collection and cultural heritage of the State of Queensland and means they will be available for public access in accordance with the Public Records Act As a future Queensland Digital Archive will collect public records from a wide range of public authorities (the Public Records Act 2002 covers more than 600 organisations), the records will be in a wide range of formats. For this reason, it will be necessary to develop strategies to ensure that the records can remain accessible over time, despite changes to format. A key obligation is to ensure that the information remains authentic and trustworthy, despite preservation treatment. Authentic information also needs to be understood in terms of the context of its creation by what organisation, at what date, for what purpose. Efficient and effective processes for transferring digital records and their metadata from public authorities systems to a future digital archive will be an essential requirement. For many public authorities, the bulk of records required for longer-term retention for business purposes may be in a more limited range of known formats, although storing the large volumes involved may present its own challenges. Digital storage, or what is commonly referred to by the IT industry as archiving, does not encompass preservation actions on file formats. Digital storage repositories, whilst often responsible for storing information for extended periods, will not typically encompass the same management and operations as a digital archive.? Q1: What sorts of digital records does your public authority need to manage for longer periods (more than 20 years)? 9

10 3.2 Key issues The preservation of digital information requires consideration of a number of key issues that continue to challenge the digital preservation community worldwide, such as: What must be preserved? The nature of digital objects is very different from their physical analogues. By virtue of the tangibility of a physical item, it is generally clear what must be preserved. But in the digital environment where the information is mediated by technology, digital records are not stable objects but can be considered a series of performances across time. Each performance is a combination of characteristics, some of which are incidental and some of which are essential to the meaning of the performance. 9 How should it be preserved? A key issue for digital preservation is the challenge of ensuring records remain accessible despite the obsolescence of software formats. A number of strategies have emerged in recent years to address this challenge and common approaches are outlined below: Migration Emulation Normalisation Encapsulation Migration is the transferral of digital records from older or obsolete hardware and software configurations or generations, to current configurations or generations in order to maintain accessibility. Emulation uses software to recreate the information s original operating environment to enable the original performance of the software to be recreated on current computer systems. The result is that the original data format is preserved and may be accessed in an environment that allows for the recreation of the original look and feel of the record. 10 Normalisation involves transforming the original form to create another that is not specific to a particular technology or application and is deemed to be more preservable, i.e. an archival data format. This process reduces the need for repeated migrations. Generally, archival data formats are open source and/or non-proprietary formats that provide greater potential longevity and fewer preservation constraints, however a key issue is the determination of which file formats to recommend as archival data formats. Encapsulation requires metadata to be bundled with the digital object. The metadata allows the record to be intellectually understood and technologically accessible in the future. A viewer is then required to display the records. Encapsulation ensures the integrity and authenticity of records over time, but cannot preserve digital records on its own. 9 The National Archives of Australia refers to essential characteristics as the essence of a record. There are a number of research programs to explore the nature of these characteristics, such as the UK INSPECT (INvestigating the Significant Properties of Electronic Content over Time) project. 10 Emulation consists of keeping the original executable program that was used to create and/ or manipulate digital information. A specification is also kept that records the details of the original environment that are required to play the program on another computer in the future. 10

11 When is it best to conduct preservation treatment? Organisations around the world are taking different approaches to the timing of undertaking preservation treatments. Some, for example, require conversion into an archival format at the point of decommissioning systems or transferring information to a digital archive; others retain the information in its native format in a digital archive until they are alerted to the impending obsolescence of the file format Impact of not addressing the problem Globally, it has been recognised that strategic digital archiving approaches are required to ensure the sustainability of electronic government information. Many leading international authorities, such as UNESCO, have recognised that digital evolution has been too rapid and costly for governments and institutions to develop timely and informed frameworks encompassing information management and preservation strategies. UNESCO has asserted the lack of an overarching framework threatens the economic, social, intellectual and cultural potential of an organisation. 12 There are often systemic challenges which create barriers for sustainable digital access and preservation within organisations. These are commonly recognised to include: 13 Misalignment of incentives between those who benefit from it and those who are in a position to undertake digital preservation Confusion and/or lack of alignment between stakeholders, roles and responsibilities with respect to digital access and preservation, with a prevailing assumption that it is someone else s problem Inadequacy of funding models to address long-term access and preservation needs which is underpinned by challenges in placing a monetary value on digital preservation, and translating this value into a willingness to pay Complacency that current practices are good enough to ensure ongoing access Fear that digital access and preservation is too immense an endeavour to take on. Without a strategic digital archiving approach, poor information management practices across the lifecycle of digital information means that records are at risk of losing their trustworthiness, integrity and authenticity, or simply becoming lost or inaccessible. Preservation of valuable information assets is a form of investment for Government. It is a way to ensure that the value-creating capacity of these assets remains available in the future. 14 The Queensland Government recognises the value of information when it makes major investments in change initiatives enabled by Information and Communications Technology (ICT). However, these initial investments in new ICT systems may remain under-utilised when there is no program in place to ensure the ongoing management and accessibility of the public records generated within these systems over the information s required retention period. 11 Automated obsolescence notifications occur through integration with file format registries, such as PRONOM 12 United Nations Educational, Scientific and Cultural Organization (2003) Charter on the Preservation of the Digital Heritage. Available at: files/13367/ charter_en.pdf/charter_en.pdf 13 Blue Ribbon Taskforce on Sustainable Digital Preservation and Access (December 2008) Sustaining the Digital Investment: Issues and Challenges of Economically Sustainable Digital Preservation Interim Report. Available from 11

12 Faced with the need to address digital archiving challenges today in the absence of a whole-of-government framework, a number of Queensland public authorities have expressed their intention to develop agency-specific solutions. This siloed approach raises the risk of an inconsistent whole-of-government strategy for addressing the problem, increased costs associated with the duplication of efforts and a risk that any work undertaken in the absence of a whole-of-government approach may need to be reviewed or reversed at a future point in time. The lack of a defined approach to digital preservation plays a part in the inaccessibility and loss of public records. In research undertaken by Queensland State Archives in 2006, 34% of respondents to a survey advised they are not confident of the future accessibility of their electronic records or have already lost access to some electronic records. This loss carries significant risks such as legal liability; inability to provide consistent client service; lack of accountability; loss of business intelligence and continuity; inefficiency (and the associated economic loss); and damage to the reputation of, and therefore public trust in, public authorities and by extension, the Government as a whole. The impact of this risk cannot be underestimated. For example, in 2007, due to poor electronic recordkeeping practices by Japan s Social Insurance Agency it was revealed that millions of pension scheme records were unidentifiable, could not be matched to the correct beneficiaries, or had not been recorded in the system. 15 The rapid take up of new technologies, the ever-increasing volume and size of information and the lack of a coordinated, comprehensive and strategic approach to digital archiving within the Queensland Government means that a critical point has been reached. Analysis of other archival authorities approaches to the issue of longterm preservation of information indicates the development and establishment of a framework and digital archive can take a number of years. Without action to progress this work, Queensland Government only prolongs its exposure to these risks. The action in the Toward Q2 though ICT Implementation Plan is an initial step towards ensuring the Queensland Government has acknowledged the need to develop the necessary strategies and infrastructure to preserve valuable information over time.?q2: Is your public authority confident that all of its digital records will remain accessible for as long as required? What evidence supports this level of confidence? Q3: Has your public authority implemented any strategies to ensure records remain accessible over time? If so, what are these strategies? Do they cover all of your organisation s information, or only selected areas? Q4: Are there specific challenges your public authority is facing that create barriers to ensuring sustainable digital preservation of public records? 14 Blue Ribbon Task Force on Sustainable Digital Preservation and Access (2010) Sustaining the Digital Investment: Issues and Challenges of Economically Sustainable Digital Preservation. Available at 15 Japan s politics The Economist, 14 Jun Available at 12

13 4 Developing a solution A successful digital archive requires an integrated framework of policy and standards to underpin its operation and support whatever technology options are adopted....there is no silver bullet for solving the set of issues. New knowledge must be developed, pilot approaches must be tested and the efforts that are currently in progress for developing technology solutions need to be coordinated and communicated to the appropriate user community. 16 As the quote above highlights, while a number of tools and approaches have emerged for digital preservation see Appendix B for examples there is no single out of the box solution or policy approach that can be selected and deployed. Ensuring electronic records are managed, preserved and accessible into the future requires more than an ICT storage system. A broad range of functions need to be addressed, including identification, appraisal, description, storage, preservation, management, retrieval and dissemination, as well as the development of the associated policies, guidelines and systems to ensure the integrity of public records is maintained over time. 4.1 Standards for a digital archive A range of standards have emerged in recent years that provide guidance on the essential elements and functionality of a digital archive. Two key standards are ISO 14721:2003 Open archival information system Reference model and Trustworthy Repositories Audit & Certification: Criteria and Checklist. 17 The Open Archival Information System (OAIS) reference model is a conceptual model that outlines how digital objects can be prepared, submitted to an archive, stored, managed and maintained for long periods, and retrieved as needed, although it does not prescribe any technical approach to preservation. More detail on the OAIS standard and its implications for a digital archive is provided in Appendix C. The Trusted Repositories Audit & Certification (TRAC) framework provides tools for the audit, assessment, and potential certification of digital archives, establishes the documentation required for audit, delineates a process for certification, and establishes appropriate methodologies for determining the soundness and sustainability of digital archives. It is widely recognised as a defacto standard within the digital preservation community and work is currently underway to transform this framework into a formal international standard. 16 National Association of State Chief Information Officers (2007) Electronic Records Management and Digital Preservation: Protecting the Knowledge Assets of the State Government Enterprise Part II: Economic, Legal and Organizational Issues. 17 See for further information. 13

14 4.2 Examples from international and national practice Many government archival authorities around the world have invested in developing strategies for long-term preservation. Appendix D includes a brief overview of some selected initiatives. Key lessons from this work include: Prioritisation of key areas: different types of records document-based, databases, websites present different preservation challenges. Experience has shown that it is not feasible to tackle all at once. Use of standard components where possible: a number of open source tools are available to facilitate different aspects of a digital archive. However, the appropriate selection and integration of these tools requires careful consideration. The importance of partnerships: the development of a successful digital archive requires both technological and archival expertise, and must be done in consultation with the eventual contributors to the archive. For these reasons a range of partnerships are essential. Some organisations focus on one main preservation strategy; others use a selection of techniques based on the particular requirements of different categories of records. For example, the National Archives of Australia normalises text-based records transferred to the digital archive into XML-based open formats, whereas the UK National Archives uses a mix of strategies, including emulating older Microsoft operating systems and formats under agreement with Microsoft Challenges for agencies As Queensland s lead agency for recordkeeping, Queensland State Archives has released extensive advice on many aspects of managing electronic records, including advice on managing as public records and the use of digital rights management technologies. Through partnerships, such as the Australasian Digital Recordkeeping Initiative, Queensland State Archives has also contributed to key resources such as Guidelines and Functional Requirements for Records in Business Systems. 19 However, extensive challenges remain. Anecdotal feedback suggests that a number of Queensland public authorities are currently maintaining legacy systems without knowledge of the nature of the information stored within them and therefore whether they are lawfully able to dispose of and decommission them. It is acknowledged that any approach to a digital archive and long-term preservation must also include support for agencies to enable them to properly appraise the information they hold and identify its lawful retention period. With the application of proper appraisal and implementation of retention and disposal decisions, ICT systems may in fact be properly decommissioned, allowing 18 Warning of a data ticking time bomb, BBC News, 3 July Available at 19 Available from 14

15 for potential savings in storage, supporting technical infrastructure and energy consumption. Without this, agencies may continue to expend unnecessary resources on maintenance of these legacy systems. Agencies that delay consideration of the need for preservation over time only compound the financial impact, when, as technology becomes obsolete, the efforts and costs associated with ensuring accessibility increase. A digital archive will only be successful if public authorities are able to identify records of permanent value in accordance with an authorised Retention and Disposal Schedule and extract them with necessary metadata from ICT systems for transfer into the archive; a task made much simpler if recordkeeping considerations are included in the design of all systems.? Q5: What challenges does your agency face in effectively managing records in the electronic environment? For example, has your agency identified all of its electronic public records and their authorised retention periods? Has your agency calculated the storage volume of its electronic records? Is your agency able to demonstrate the authenticity of its electronic records when legal proceedings arise? Q6: In addition to the guidance already released by Queensland State Archives, what further support and guidance is required to ensure records are well-managed in the electronic environment? 15

16 5 The Way Forward While much can be learnt from other jurisdictions approaches, it is clear from international research that developing a digital archiving approach is not simply a case of adopting another jurisdiction s solution and implementing this unconditionally within Queensland. This Discussion Paper has raised a broad range of issues that requires a co-ordinated plan for resolution in order to develop a program of work to establish a Queensland Government Digital Archive over time. 5.1 Program elements It is anticipated that a program management office will be established within Queensland State Archives in the future to strategically plan the program for preserving government digital public records over time. This office will plan and oversee specific projects that research, develop and implement solutions for digital preservation, including those issues outlined in Figure 2 below, supported by appropriate partnerships and consultation. 20 Queensland State Archives will actively seek input from public authorities as the program of work develops. Resources Framework Program resources Costs of digital preservation and archiving Building capability within QSA & across Government Physical infrastructure Procurement planning Partnership opportunities Queensland Government s Digital Public Records Organisational & Policy Framework Business requirements Digital archiving policy positions e.g. recommended archival file formats & digital preservation treatments Legislative framework Technological Framework Functional requirements Applications and Technology platform Security Technology forecasting Program Vision & Governance Stakeholder Engagement Resources Framework Technological Framework Organisational & Policy Framework Figure 2: Examples of digital archiving program elements 20 These program elements are based on the three legs for digital preservation developed by Cornell University. 16

17 5.2 Program phases This program will involve a number of phases and activities as proposed in Figure 3 below. It is intended that the key activities for each stage will be refined with input from public authorities and evolve as expertise in digital preservation increases across government. Preliminary Planning Key activities: Discussion Paper, survey & forums Business case for research & development Establishment of governance & Program Management Office Research on digital archiving trends Concept Concept Definition Development Key activities: Definition of program scope White Paper/Vision Requirements and Enterprise Architecture & Design AS-IS and TO-BE Options analysis Policies, principles & standards framework Legislative framework Skills framework Assessment of agencies support needs Partnership strategy Developing a transition plan for Queensland State Archives Key activities: Procurement Physical & technological infrastructure Policies & standards Proof of Concept/Pilots Determination of costs associated with digital archiving Building capability Implementation business case and business model Re-engineering Queensland State Archives Operationalise Key activities: Go-live Service Level Agreements with agencies Capability development Ongoing preservation planning & technology forecasting, & associated policy development Transition governance arrangements for normal operations Program Governance Stakeholder Engagement Figure 3: Draft digital archiving program 17

18 5.3 Guiding principles It is also proposed that this program will be directed by a set of high-level principles to help guide projects. These principles, to be refined with further expert input, have been drafted to date as follows: 1. The program will be guided by best practice and established standards for digital archiving. The program is committed to developing a digital archive that is compliant with the Open Archival Information System (OAIS) standard. The Open Archival Information System Reference Model (ISO 14721:2003) is an internationally accepted standard for developing a digital archive and compliance with this standard assists to demonstrate its trustworthiness as a digital archive. The program is also committed to developing a Trusted Digital Repository, in line with the international efforts to standardise the characteristics of a trustworthy digital preservation repository, and will also be guided by other best practice digital archiving approaches and standards. 2. The program will initially focus its efforts on permanent digital public records. Permanent records are those with ongoing archival value which cannot be destroyed and must be retained indefinitely by either the agency or Queensland State Archives. Permanent records are stipulated in authorised Retention and Disposal Schedules. 21 Because of the value of these records, the program will primarily concentrate on developing solutions for permanent digital public records. 3. Approaches for preservation will be designed to be sustainable. Public records must be preserved with due regard to their significant characteristics, recognising that all digital preservation treatments involve some form of prioritisation of the elements of the record that need to be maintained over time. Within the constraints of ensuring appropriate preservation, specific preservation approaches and strategies will be designed to be sustainable, both through minimising the number of preservation treatments required and consideration of the resourcing impacts on both public authorities and Queensland State Archives.?Queensland State Archives welcomes your feedback on these proposals. Q7: Does your public authority agree with the proposed program elements, plan and principles? If not, why not? Q8: Are there any significant aspects that are not covered? If so, what are these and why does your public authority consider they are required? Q9: Does your public authority have any other comments to make in regard to the issues raised in the Discussion Paper? 21 Schedules that have been approved by the State Archivist can be found at: 18

19 Appendix A: List of questions Q1: What sorts of digital records does your public authority need to manage for longer periods (more than 20 years)? Q2: Is your public authority confident that all of its digital records will remain accessible for as long as required? What evidence supports this level of confidence? Q3: Has your public authority implemented any strategies to ensure records remain accessible over time? If so, what are these strategies? Do they cover all of your organisation s information, or only selected areas? Q4: Are there specific challenges your public authority is facing that create barriers to ensuring sustainable digital preservation of public records? Q5: What challenges does your agency face in effectively managing records in the electronic environment? For example, has your agency identified all of its electronic public records and their authorised retention periods? Has your agency calculated the storage volume of its electronic records? Is your agency able to demonstrate the authenticity of its electronic records when legal proceedings arise? Q6: In addition to the guidance already released by Queensland State Archives, what further support and guidance is required to ensure records are well-managed in the electronic environment? Q7: Does your public authority agree with the proposed program elements, plan and principles? If not, why not? Q8: Are there any significant aspects that are not covered? If so, what are these and why does your public authority consider they are required? Q9: Does your public authority have any other comments to make in regard to the issues raised in the Discussion Paper? 19

20 Appendix B: Examples of digital preservation tools DROID JHOVE Plato DROID (Digital Record Object Identification) is a software tool developed by The National Archives, United Kingdom to perform automated batch identification of file formats. DROID is a platform-independent Java tool, which is available to download under an open source license from JHOVE (pronounced jove ) is an extensible framework for format validation developed by JSTOR and the Harvard University Library. JHOVE provides functions to perform: Format specific identification the process of determining the format to which a digital object conforms; it answers the question: I have a digital object; what format is it? Format validation the process of determining the level of compliance of a digital object to the specification for its purported format, e.g.: I have an object purportedly of format F; is it? Format characterization the process of determining the format specific significant properties of an object of a given format, e.g.: I have an object of format F; what are its salient properties? See for further information. Plato is a decision support tool for implementing a preservation planning process which integrates services for content characterisation, preservation action and automatic object comparison in a service-oriented architecture to provide maximum support for preservation planning endeavours. It helps an organisation to determine the appropriate preservation treatment for a digital object. Plato was developed by the PLANETS (Preservation and Long-term Access through NETworked Services) project a research and development initiative in the field of digital preservation, funded by the European Commission. PREMIS See for further information. The PREMIS (PREservation Metadata Implementation Strategies) Data Dictionary defines a core set of semantic units that digital archives must know in order to perform their preservation functions. The Data Dictionary is maintained through the Library of Congress, USA. See for further information. 20

21 SIARD UDFR The Software-Independent Archiving of Relational Databases format and software, developed by the Swiss Federal Archives, are tools for preserving relational databases. The software converts proprietary databases (MS Access, MS SQL, Oracle, etc.) into a file archive in a non-proprietary SIARD format. See for further information. The Unified Digital Formats Registry (UDFR) is an initiative begun in April 2009 to build a single shared formats registry for the global community. UDFR builds on the PRONOM and Global Digital Formats Registries to create an online registry of technical information that provides impartial and definitive information about the file formats, software products and other technical components required to support long-term access to electronic records and other digital objects of cultural, historical or business value. Xena See for further information. Free and open source software developed by the National Archives of Australia to aid in the long-term preservation of digital records through: detecting the file formats of digital objects converting digital objects into open formats for preservation. Xena is an acronym meaning Xml Electronic Normalising for Archives. See for further information. 21

22 Appendix C: Open Archival Information System Reference Model This internationally accepted model has been widely adopted as the standard for developing a digital archive and applied by a range of communities that are concerned with the long-term preservation of digital information, including libraries, universities, archives, research organisations and international aeronautical and space agency bodies. 22 Some of the key principles of the OAIS model include: the importance of the non-technical aspects of a digital archive such as organisational infrastructure and staffing, sustainable resourcing and procedural accountability the need for an ongoing program of preservation planning, including routine monitoring of the technological change occurring in the archive s environment, and the importance of metadata to record all preservation, management and use processes that take place. The OAIS model, shown in Figure 1, outlines the key functions that must be performed by a digital archive. Figure 1 - OAIS Functional Entities Ingest occurs when the records, in the form of a Submission Information Package (SIP) are taken into archival custody. It is at this stage any required preservation treatments will occur (e.g. converting to selected archival formats), and the generation of additional technical metadata and archival description. Following Ingest, the Archival Information Package (AIP) is moved into the archival repository (Archival Storage) and metadata into other appropriate systems (Data Management). 22 The Standard is based on the original work undertaken by a Consultative Committee for Space Data Systems. 22

23 The model also highlights the access function of the archive. The consumer entity refers to users who will wish to access records from the archive, including public authorities and public clients and also includes consideration of access rules and requirements. Depending on the needs of the consumer, the dissemination information packages (DIP the access version of the record) may or may not be in the same format as the AIP. The model also identifies the vital roles of: Management, which sets overall policy and direction of the archive Administration, or the day to day operation of the archive; and Preservation planning, highlighting that preserving a digital record is not a one-off step and that archival staff will need to constantly monitor the stability of formats in the archive, and develop strategies to manage new formats as they emerge. Not shown in the model is a recursive process between archival storage and ingest. As preservation strategies change, records may need to be exported from the repository and re-treated, for example moving to a new archival format. This process will also generate more technical metadata, for the data management function. OAIS responsibilities The OAIS standard also defines the responsibilities which an archive must fulfill to conform to the standard. A conforming archive must: Negotiate for and accept appropriate information from information producers. Obtain sufficient control of the information provided to the level needed to ensure long-term preservation (for example, ensure there are no copyright barriers to copying or otherwise preserving the information). Determine, either by itself or in conjunction with other parties, which communities should become the Designated Community, and therefore should be able to understand the information provided (this relates to knowing your target client audience). Ensure that the information to be preserved is independently understandable to the designated community. In other words, the community should be able to understand the information without needing the assistance of the experts who produced the information. Follow documented policies and procedures which ensure that the information is preserved against all reasonable contingencies, and which enable the information to be disseminated as authenticated copies of the original, or as traceable to the original. Make the preserved information available to the designated community. 23