Suitable file formats for transfer of digital records to The National Archives

Transcription

1 Suitable file formats for transfer of digital records to The National Archives The National Archives September 2011 Crown copyright 2011 You may re-use this information (excluding logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit or Any enquiries regarding the content of this publication can be sent to Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned. This publication is available for download at

2 1. Introduction 1.1 What is the purpose of this guidance? This guidance sets out the range of digital file formats that The National Archives can currently sustain over the long term. We are able to accept these formats from public record bodies, preserve the information they contain and provide access to future policy-makers and researchers. Following this guidance will help you comply with The National Archives requirements for transfer of digital records. It will also help you ensure that the information which supports your own business requirements remains complete and available to you for as long as you need it. The National Archives does not stipulate which file formats you should use for the creation of public records. This decision must be made on the basis of your own business requirements; however, we strongly recommend that requirements for sustaining access to your records are considered as an integral part of evaluating file formats for use in your organisation. If you have selected digital records for transfer to The National Archives that are not in a format we can currently accept, please talk to us before you take any other action (see section 4 below for a discussion of the available options). This guidance applies to the transfer of digital records and information to The National Archives. It excludes the transfer of government websites and datasets as part of the UK government web archive. This activity is covered by separate guidance Who is this guidance for? This guidance is primarily aimed at information managers and IT managers who need to assess the use of file formats in different business situations across their organisation. 2. What is a file format? A file format is a method of storing digital information in a computer file, allowing its later use by computer systems or people. There are thousands of different file formats for different kinds of digital content and there may be several different versions of the same file format. A file format is often confused with the software most commonly used to create or use it. For example, we talk about Microsoft Word files, or Acrobat PDF files. Despite these naming conventions, in principle a file format is not bound to any particular software even if this is sometimes the case in practice

3 A file format is like a language which is only spoken by certain pieces of software. In general, the greater the number of languages, and the fewer speakers there are for each, the harder it is to maintain the ability to read and understand them over long periods of time. 3. Choosing file formats Before you select file formats for use within your organisation, you will need to understand your requirements: Business requirements, for example, your need to record, read, work with, share or publish information now and in the future Technical requirements, for example, compatibility with your technical environment; your ability to provide supporting storage and infrastructure services Legislative and policy requirements, for example: the Public Records Act, S46 Code of Practice 2, Government IT strategy 3, the Data Protection Act, Public sector transparency agenda Having understood your requirements, you should evaluate your file formats in this context. We suggest that you use the following criteria for your assessment: Capability how well the format meets your requirements Resilience how resilient the format is to time Bear in mind that you may require access to your digital records over very long periods of time. If your records are selected for permanent preservation, they will also need to meet The National Archives technical criteria for transfer. Flexibility how well the format can adapt to changing requirements Consider the ability of your formats to adapt to your changing business requirements, and to changes in the technical and legislative environment in which you operate. For example, the Freedom of Information Act and the Transparency agenda have both introduced new requirements for maintaining and providing access to public sector information. The National Archives guidance on Evaluating File Formats provides a full discussion of evaluating file formats against your business and technical requirements 4. This guidance will focus on choosing file formats that support long-term preservation; however, the same recommendations will help you to maintain the continuity of your digital information for business use in the short and medium term. 2 Lord Chancellor s Code of Practice on the management of records issued under section 46 of the Freedom of Information Act The Government IT strategy, published by Cabinet Office in March 2011 states The Government believes that citizens should be able to read government documents with the standardised document format reader of their choice. The first wave of compulsory open standards will determine, through open consultation, the relevant open standard for all government documents

4 4. File formats for long-term preservation The digital environment is continuously evolving: new formats appear, older formats undergo revisions or version enhancements, supporting devices and infrastructure can change beyond recognition in a matter of a few years. While some formats have stood the test of time, others have become increasingly difficult and costly to maintain. In general, the formats that have proved difficult to sustain are generally those that were not widely adopted, were poorly supported, evolved rapidly or were heavily platform restricted. The sheer number of different formats in existence makes it economically unsustainable for The National Archives to commit to managing and maintaining access to all formats in perpetuity. While the individual bytes of a digital file can be preserved relatively easily, the ability to access and understand the information contained in those bytes will be lost if the required software and technical environment are no longer available. The challenge increases where the file format specification is not available either because it is proprietary or was never formally defined. The level of resource that would be needed to provide long-term access to an everexpanding list of file formats prohibits this being a viable option. The only reasonable alternative is for The National Archives to limit the range of formats we can accept and maintain. This strategy enables us to continue to deliver our core task of providing access to the digital public record, whilst operating within our budgetary constraints. The National Archives has identified a list of file formats that are currently within our technical and budgetary capacity to receive, maintain and make available. When using this list to inform your choice of file formats for use in your organisation, you should note the following: The list will grow as new formats emerge, our understanding of existing formats develops, and the available tools and technologies improve. Although unlikely, formats may be removed from the list as new risks are discovered or formats are superseded. If you have records in formats that are not on the list, you should not convert the files for the purpose of transfer to The National Archives: your format may be in the process of being added to the list; the new format you select may be at risk of being removed from the list; the process of conversion may place the authenticity of the records in question unless appropriate procedures are followed. In this situation, please contact us to discuss the options before you take any action. If you have formats that you believe should be added to the list, please contact us. The list of file-formats that The National Archives is currently able to accept is given in Section 6 below. This list is reviewed regularly. Please refer to The National Archives website 5 to make sure you have the latest version of this document before making any decisions based on this list

5 5. Transferring digital records to The National Archives When appraising digital records prior to transfer, you should establish the range of file formats contained within the selected records. You will need to send us this information at an early stage of the digital transfer process. File format identification tools such as DROID 6 can assist you in this. In addition to identifying the base file format, DROID will distinguish between different versions of the format, for example it can distinguish fifteen different format versions which all use the extension.doc If records which have been selected for preservation include files in formats which do not appear on our list of suitable formats for transfer you should discuss this with us at the earliest opportunity. Formats that are not listed will need to be assessed on an individual basis to determine whether they can be transferred, or whether you (or your sponsoring department) will need to preserve them until acceptable arrangements can be made for transfer, accessioning and access at The National Archives. Factors such as the format, version, content of the records and number of files will influence this discussion. Regardless of format, The National Archives will not accept records if there is anything that would prevent full and safe access to the data, for example: computer viruses, worms or associated malware full or partial encryption of the data, including DRM full or partial password protection of the data full or partial corruption All digital records selected for transfer should be checked for these issues alongside the sensitivity review process and we will require confirmation from the transferring department that this has been done. 6. Acceptable File Formats for Transfer: version 1.0 September 2011 The list that follows does not detail different versions of individual file formats; however, in certain cases, the particular version of a format may affect our ability to accept the files. We will require a comprehensive list of file formats, including version information, from you at an early stage of the digital transfer process. 6.1 Document and Text Formats Open Document Text Open Office Write Microsoft Rich Text Format Plain Text.odt.sxw.rtf.txt 6 DROID is a free and open source file format identification tool, developed by The National Archives ( 5

6 Portable Document Formats HTML extensible Mark-up language Microsoft Word, Works and Wordpad Word Perfect 6.2 Spreadsheet Formats Comma Separated Values OpenDocument Spreadsheet OpenOffice Calc Lotus Worksheet Lotus Symphony Spreadsheet Microsoft Excel Microsoft Works Spreadsheet 6.3 Presentation Formats OpenDocument Presentation OpenOffice Impress Microsoft PowerPoint Harvard Graphics Presentation 6.4 Graphic Formats JPEG 2000 JPEG Tagged image file format OpenOffice Draw OpenDocument Graphics Portable Network Graphics Graphics Interchange Format Windows Bitmap Encapsulated Postscript Corel Draw Adobe digital negative Microsoft Visio Drawing 6.5 Audio Formats Waveform Audio Format Broadcast Wave Format Ogg Vorbis.pdf.htm.html.xml.docx.doc.wps.wpd.wpd.csv.ods.sxc.wks.wrk.wr1.xlsx.xls.wks.odp.sxi.pptx.ppt.shw.jp2.jpg.tif.sxd.odp.png.gif.bmp.eps.cdr.dng.vsd.wav.wav.ogg.oga 6

7 Windows Media Audio MIDI Audio MPEG 1/2 Audio Layer 3 MPEG Video Formats Motion JPEG 2000 Ogg Theora Windows Media Video Macromedia Director Macromedia Flash Quicktime MPEG Formats Internet Message Format Microsoft Outlook Personal Folders Microsoft Outlook Message 6.8 Other Formats Microsoft Project.wma.mid.mp3.mp4.mj2.ogg.ogv.wmv.dir.swf.mov.mp4.eml.pst.msg.mpp 7. Further reading Digital Continuity service For information and guidance on maintaining the usability of digital records over time. Digital Transfer Guidance for departments preparing and transferring digital records to The National Archives. DROID Free, open source, file profiling and characterisation tool developed by the National Archives. Sustainable digital preservation Report of the Blue Ribbon Task Force on Sustainable Digital Preservation and Access. 7