Two Approaches to Digital Long-Term Storage and Preservation at the National Archives of Sweden (NAS) and the Challenge of File Format Standards

Transcription

1 Two Approaches to Digital Long-Term Storage and Preservation at the National Archives of Sweden (NAS) and the Challenge of File Format Standards Karl-Magnus Drake

2 Topics of this presentation Background First approach: current digital storage and preservation system at NAS - Digitization activities; digital chains; digital repository content and quantities - Adopted archival storage and preservation standards and metadata schemes - The Radar platform and the archival storage system ESSArch at NAS Second approach: a holistic approch to microfilm-based digital storage and preservation - The Archivator, Milos and Astor projects - Applying the Archivator workflow on digital data objects and metadata from NAS The challenge of file format standards Conclusions

3 Contributors to this presentation have mainly been the following colleagues and project partner At NAS: Martin Bjersby Karin Bredberg Mats Berggren Magnus Geber In the Archivator; Milos and Astor projects: Piql A/S in Norway (former Cinevation A/S)

4 Background NAS have recieved digital data objects and metadata since the 1970s During the last decades more resources have been allocated to develop an OAIS-based digital long-term storage and preservation system at NAS The system consists of number of modules, which have gradually been implemented and now form the so called RADAR platform at NAS During the last two years NAS have been involved more or less in the three European research projects Archivator; Astor and Milos, based on microfilm for long-term storage and preservation Digital data objects and metadata (AIPs) from NAS have been used in a test bench within the now beta-testing Archivator system

5 First approach: current digital storage and preservation system at NAS Main scanning and long-term storage/preservation activities at NAS (April, 2012) The FOSAM project: scanning of documents of church records (younger than 1860) at MKC (finalized) The DAF-Access project: scanning of highly requested documents at NAS and further processing at MKC or SVAR (two production sites) Ongoing microfilm scanning of various (old) microfilm collections of NAS at SVAR The GSU I project starting 2007: microfilm scanning of Swedish church book older than 1860 by FamilySearch in Salt Lake City, USA and then delivery to SVAR (finalized) The GSU II project starting 2011: microfilm scanning of Swedish judicial records by FamilySearch in Salt Lake City, USA and then delivery to SVAR

6 The digital chain for church books scanning at MKC

7 Current digital repository content and quantities at NAS (April, 2012) : Born-digital files from agencies: 3 TB Audio-video files and multimedia: approximately 100 TB Digitized volumes (one AIP per volume): Digitized images in the TIFF format (mainly from church books): 1400 TB Images total (mainly from church books): 121 million Images published on the web/internet: 63 million DJVU-files (web-presentation files extracted from TIFF files of church books): 18 TB Total storage: >2300 TB on offline LTO-tapes; 140 TB in an old LTO tape robot system

8 General metadata levels Metadata Encoding And Transmission Encoded Archival Context Encoded Archival Descript. XML-based Metadata for Markup Language

9 Adopted archival standards and metadata schemes at NAS METS METS is used by NAS to describe a SIP or an AIP - In addition the ADDML and PREMIS/MIX metadata schemes are used - The current METS-profile was developed in May 2010

10 Adopted archival standards and metadata schemes at NAS PREMIS PREMIS metadata is created by KRAM (NAS application for control and ingest of information received from as well as accessed by agencies) - PREMIS metadata is created for scanned images - PREMIS metadata is embedded in METS by the NAS storage system ESSArch - Metadata derived from PREMIS is also stored in ARKIS (NAS Archival Information System)

11 Adopted archival standards and metadata schemes at NAS MIX NAS use MIX as metadata scheme for scanned images in TIFF-format - MIX-metadata is embedded in METS and PREMIS by NAS ESSArch system when AIP:s containing scanned images are ingested

12 Adopted archival standards and metadata schemes at NAS ADDML-2001/ ADDML-metadata is embedded in METS by NAS ESSArch system (via the KRAM and RALF applications) - Metadata derived from ADDML is also imported and stored in ARKIS - SIP:s with METS and embedded ADDML will be delivered to NAS by e.g. Statistics Sweden

13 The NAS platform for digital long time preservation RADAR ( ) Tjänsteman Riksarkivet Allmänhet Sökning via NAD och SVAR:s webbplats Tjänsteman Myndighet Tjänsteman Riksarkivet ARKIS RALF Application for control and preparation at the agencies KRAM Application for Ingest and control Archival Information System KRAM Access and dissemination of databases Digital Chain ESSArch Ingest from scanning Archival Storage System Tjänsteman Riksarkivet

14 The archival storage system ESSArch at NAS

15 The main characteristics of the ESSArch archival storage system Store and retrieve AIP:s. Stores AIP:s in several bitwise identical copies in the TAR-format Reads and writes checksums for packages and files Manage event log and access control Use the PREMIS 2.0. data model in local MySQL-databases Generate automatic updates to the Archival Information system ARKIS Is used by the National Archives in Sweden and Norway

16 Second approach: a holistic approch to microfilm-based digital storage and preservation By using all results from the three European research projects - Archivator (finalized) - Astor (running) - Milos (running)

17 The Archivator project The Archivator project was an EU funded R&D project with a total budget of 7.5 M over three years The goal was to develop a closed loop solution for secure, migration free long term archiving based on microfilm media The project is now finished The system is now in Beta testing (2014)

18 The Astor project funded by The Research Council of Norway: New Packaging and Logistics The development of new polymer materials for cartridge of existing medium and automated logistics

19 The Eurostars supported Milos project: New Storage Medium The development of new polymer materials for a microfilm-based Digital Optical Tape (DOT) medium and reel cartridge as well as processing technology & equipment A total budget of over 35 MSEK over a 30 months project period

20 An Archivator test bench Digital data objects and metadata (AIPs) from NAS was ingested Church books Old maps of Sweden Illustrated medieval parchments Video Sound Digital born material

21 Preserve digital data objects and metadata from NAS by using the Archivematica digital preservation system (see Archivematica Client data Create DIP Create AIP Archivator Archivist ICA AtoM Search/ Preview Restore

22 AIP Encoding Recording Film Processing Scanning Decoding AIP An AIP is uploaded to Archivator The AIP keeps unique ID from Archivematica

23 AIP Encoding Recording Film Processing Scanning Decoding AIP First you prepare the AIP for writing to microfilm: From the AIP you then identify file formats, generate a file list and create previews (optional). Then you encode the digital data object to square dots in image frames. The encoding also includes Forward Error Correction (FEC) and generates checksums. Finlly you create the reel structure with table of content, human readable section with decoding instructions and file format descriptions.

24 AIP Encoding Recording Film Processing Scanning Decoding AIP You transfer the created reel to film using Writer When recorded, the reel is sent to the film processor Writer

25 AIP Encoding Recording Film Processing Scanning Decoding AIP You develop the microfilm in the film processor. You put the microfilm in the labeled package. The label includes unique identifiers in both human and machine readable form. Then you verify the microfilm by reading back the frames, decoding the digital data objects and comparing with the original AIP. Finally you store the verified microfilm reel in the automated storage system.

26 Restore digital data objects and metadata (AIPs) from NAS Archivator ICA AtoM Search/ Preview Restore Client data Client

27 AIP Encoding Recording Film Processing Scanning Decoding AIP You process the request from the retrieving system (ICA- Atom) to link AIP with reel in the storage. Then you fetch the reel from the automated storage system. Finally you read the frames from the microfilm. Reader

28 AIP Encoding Recording Film Processing Scanning Decoding AIP Each scanned frame is first decoded Then you combine all frames to file and compare checksums An AIP is finally restored

29 AIP Encoding Recording Film Processing Scanning Decoding AIP An AIP is now available for download You notify the data owner Finally the reel is brought back to storage

30 The storage medium is self-contained The reel structure Control section Metadata section Content Area Control section Metadata section

31 The control section A zero reference frame for reader to autodetect start of medium

32 The control section A control frame, digital

33 The control section A control frame in human readable form

34 The control section The table of content, digital

35 The control section Table of content in human readable form

36 The metadata section System metadata in human readable form

37 Metadata section File format description

38 The human readable content section Metadata of church book in human readable form

39 The human readable content section Church book in human readable form

40 The digital content section First frame of digital section

41 The digital content section Second frame of digital section

42 The challenge of file format standards when long-term storage and preservation Besides large-scale disasters; media faults and media/hardware obsolescence memory institutions worldwide also have to face software obsolescence such as: - proprietary, closed format specification (i.e. not open or ISO-standarized) - limited number of rendering applications (e.g. editors; viewers; web browsers) - few application developers/vendors - the format is withdrawn due to copyright dispute; commercial or other reasons - no/limited backward compatibility - limited range of functionality - very complex functionality (e.g. only editors created the file could read the file) - limited technical documentation

43 The challenge of file format standards when long-term storage and preservation The file format implementation dilemma when generating SIPs/AIPs: - File format compliance cannot be fully guaranteed as the application implementing the file format is not in full control of neither the information owners or the service providers - Even if the file format is validated to check compliance the validating tools are not fully reliable as different tools could end up in different results - Also it is always an uncertainty in what extent a specific standard is implemented in a specific application - To minimize the file format dilemma when long-term storage and preservation you always have to apply validation and conformance tools as well as consider well-established sustainability factors

44 Conclusions A comparison of the main strength and weakness of the Archivator and NAS long-term storage and preservation system: Archivator is a migration-free (or at least a less-migration) off-line solution and hence inexpensive in a long-term perspective. NAS system is based on repetitive migrations, with negative implications on long-term cost. Archivator is based on a microfilm-based storage media with a proved longevity of up to 500 years. The estimated longevity of LTO tapes of NAS is only at the most 30 years (hence the need of repetitive migration) Archivator is based on non-magnetic (optical) storage media (i.e. microfilm). LTO tapes of NAS could be exposed to external threats such as electricity shortage; electromagnetic pulses (EMP); etc. - Archivator stores and preserves digital data object on a media (i.e. microfilm), which cannot be edited, changed or altered. Once the data is written they are like carved in stone (true WORM) in contrary to the rewriteable LTO tape media of NAS to be continued!

45 Conclusions A comparison of the main strength and weakness of the Archivator storage and preservation system and the NAS "electronic archive" (NAS-EA): Archivator is the only solution that allows you to preserve a combination of digital data, human-readable text/metadata and visual information/pictures on the same medium. Such a hybrid solution reduces the need for using multiple technologies and processes, and thus increasing your efficiency as well as the reliability of your data storage and preservation. Archivator as well as the system of NAS addresses the file format standards implementation dilemma Archivator (currently) stores archival content at a higher initial cost per GB than the LTO tape solution of NAS Archivator is (currently) an unknown brand and technology as well as an unknown developer outside the motion picture industry which is not the case of LTO tapes Thank you!