By Piql AS Rev. 140610 Alternative Storage Technologies Whitepaper
The complex needs of preservation The world is overflowing with digital data; in fact the digital universe is doubling every two years according to IDC 1. A share of this digital universe has value in the long-term. This is for instance data that is considered part of our cultural heritage, strategic business information or data that must be preserved for legal reasons. In order to ensure long-term survival, these data need to be treated differently than short-lived information. What does «digital» really mean? Do the existing storage technologies meet the complex When a file is stored digitally, the data is requirements involved in long-term preservation of data? stored as binary codes, i.e. sequences of What are the unique characteristics of the available numbers (0 and 1,) which are written on alternatives? These are some of the issues we will look into. to a storage medium. In our evaluation of the alternatives we will in particular By decoding these numbers, a perfect focus on the following aspects: copy of the original file is brought back. How well is data security and integrity maintained? How easy will it be to access the data in the future? How cost-efficient is the technology in a long term perspective? Digital storage, including the Cloud, normally means digital application of analogue storage medium. Magnetic tape What it is: Magnetic tape is an analogue medium for magnetic recording, made of a thin magnetizable coating on a long, narrow strip of plastic film. The plastic film is coated with magnetic material, i.e. ferric oxide powder which is mixed with a binder to attach it to the plastic. It includes a dry lubricant to avoid damaging the tape drive. The magnetic coating makes it possible to retain electronically encrypted data in digital format. The tape is normally packaged in a plastic cartridge for protection, and tape drives are used to read and write the data. Cheap mass storage: Magnetic tape allows large amounts of digital data to be stored at a relatively low cost, making it well suited for a mass storage purpose. Magnetic tape Handle with care: However, when using tape for archiving purposes one has to bear in mind that tape is a sensitive material that requires proper handling and environmental conditions. For instance, magnetic fields are obviously a threat to any magnetic storage, and smoke and small particles can cause damage and loss of data. It is therefore imperative to have clean operating conditions, with the right temperature and humidity levels. With minimal usage, handled properly and stored in optimal conditions, magnetic tapes have a lifetime expectancy ranging between 10 to 30 years. 2 However, due to the delicate material and its high density of 1 1 IDC, The Digital Universe in 2020, December 2012 ; www.emc.com/leadership/digital-universe PAGE 2
Linear Tape Open (LTO) is a magnetic tape storage open technology that was developed in the late 1990s by HP, IBM and Certance (now Quantum). LTO is widely used for mass storage, and the latest generations (LTO 6) can store up to 2.5 terabyte. For long-term preservation purposes, it is important to be aware that data capacity increases for each generation, but this has a negative impact on the expected lifetime. As for other magnetic tapes, LTO requires a migration-based archiving strategy in order to ensure data safety and accessibility. storage, the risk of damage and failure increases each time the tape is used. Consequently, the usage should be kept to a minimum, and it is considered best practice to migrate the data at least every 5-10 years to newer tape formats. Migrations and vendor lock-in: A drawback with magnetic tape is that data retrieval is dependent on specific reading devices, i.e. tape drives, which need to be maintained. New tape drives are normally not able to bring back data from older generations of tapes. This means that when using magnetic tape for long-term preservation, regular and endless investments in migration is needed. Optical discs What it is: Optical discs are flat circular disks that encode digital data on one of its flat surfaces. They are usually made of aluminium, but for preservation purposes it is preferred that the reflective layer of the optical disk is gold 3. Optical disc drives (ODD) with laser beams are used for engraving the data onto the disc, and the laser light is also used for reading back the data whilst the disk spins at high speeds. CDs and DVDs: The two main types of optical discs used by archivists are CDs and DVDs. These can be further split into: CD-R and DVD-R: Write once, Optical discs read many CD-RW and DVD-RW: Rewritable DVD-RAM: Rewritable discs formatted for random access (like a computer hard disc) Ease of use: Optical discs such as CDs and DVDs are easy to use, and both the media and the hardware are relatively inexpensive to purchase. In addition, the laser-based device reader is not in contact with the disc which makes mechanical failure less likely, and most of the readers are backward compatible. These factors have made optical discs widely used by archivists. High failure rates: Originally developed primarily as a mass consumer product, elaborate measures are required when using optical discs for long-term preservation. Optical discs have a relatively high risk of 2 Dr. John W.C. Van Bogart; Magnetic Tape Storage and Handling - A Guide for Libraries and Archives, June 1995; www.clir.org/pubs/reports/pub54/estimation_of_les.html 3 DPC; Handbook in Digital Preservation ; www.dpconline.org/advice/preservationhandbook/media-and-formats/media PAGE 3
failure 4, and it is strongly recommended to always do thorough testing to ensure required standards are met. Scratching and environmental factors such as dust, heat or UV light can cause severe damage to the disks. Optical discs also offer low data capacity compared to alternative technologies. Frequent migrations: It is challenging to reliably estimate the real lifetime of optical discs. They are fragile, and the life span depends on factors such as manufactured quality, how well it is recorded and its physical handling and storage. To lower the risk of failure, best practice is to frequently migrate onto newer formats. This is in sharp contrast to manufacturers who tend to claim an expected lifetime of up to several hundred years. Blu-Ray is an optical disc designed to supersede the DVD format with a higher capacity. However, it is more popular among households than professional archivists. M-Disk is an optical disc which was introduced in 2013, claiming a lifetime of 1000 years. However, a Blu-Ray reader is needed for data retrieval, and the future accessibility of these is a challenge the manufacturer does not give an answer to. Hard Disk Drive (HDD) What it is: A hard disk is a rapidly rotating disk used for storing digital information. The data is recorded by magnetizing a thin film of ferromagnetic material. A read/write head on an arm accesses the data when the disk is spinning. Instant access: Hard disks provide instant access to the data and the reading and writing device is an integrated part of the storage medium. They enable cost-efficient storage of large amounts of data, with a storage capacity of up to 4TB per disk (2013). Short durability: High failure rates make hard disks Hard Disk Drive inappropriate for long-term preservation. A recent study projects an average lifespan of six years. 5 It is therefore Solid-State Drives (SSD) use flash memory common to combine a number of disks for redundancy, for digital storage, unlike HDD s magnetic also known as a RAID. technology. They are more mechanically robust From a mechanical point-of-view, a frequent issue is that than HDDs and require less power, but have less the disk will not spin, and the failure rates increase with data capacity and are more expensive in use. age. Another common reason for failure is that the read/write head scrapes the rotating platter and thereby SSD is well suited for portable devices and causes damage and data loss. Furthermore, a hard drive applications requiring fast data access, but are is a magnetic storage technology and its magnetic not considered to have the attributes required for strength will naturally deteriorate over time when placed long-term digital preservation. offline for archiving upon a shelf. 4 Risks Associated with the Use of Recordable CDs and DVDs as a Reliable Storage Medium in Archival Collections Strategies and Alternatives, Memory of the World Programme, UNESCO, 2006; http://unesdoc.unesco.org/images/0014/001477/147782e.pdf 5 http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives-last, retrieved November 21 st, 2013 PAGE 4
The Cloud What it is: The Cloud is a term used for storage services where the data is stored in virtualized pools operated by third parties. These hosting companies operate large data centres, filled with servers, which could be located anywhere in the world. They are normally subject to strict security measures such as restricted access, environmental control and emergency backup power supply. Store and ignore: For the end user, the Cloud offers clear advantages such as instant access, ease of use with no need for maintenance, and it is inexpensive to buy or lease storage space. This makes it well suited for basic storage needs. However, for the valuable data that requires secure, long-term preservation the Cloud involves some major concerns related to security, privacy and ownership. Ownership and privacy concerns: The Cloud involves entrusting Data centre with servers a third party to store the data. This means the service provider has access to the data and may accidentally or deliberately disclose or alter the information. The service provider may also go out of business, posing questions as to who owns the servers and the data on them. The data owner will in this situation have limited control, and the data is inaccessible from the moment the network connection or power supply is switched off. Amazon Glacier is an online file storage service developed for long-term storage of data that does not require instant access. Like other Cloud services, Glacier is based on hard drives, but the data retrieval time is 3 to 5 hours according to Amazon. Storage is cheap, but users are charged for retrieving the data hence the risk of vendor lock-in should be considered. The privacy, security and ownership concerns related to cloud storage also apply to Amazon Glacier. Security issues: Data accessible through the web is vulnerable to hacking. Although the service providers strive to increase the security level, there is no guarantee that the data is out of reach of individual hackers, companies or national security agencies that might have an interest in accessing the information. Some data centres offer offline back-up, normally in the form of magnetic tape, in addition to cloud storage. This makes the data more secure, but a migration-based archiving strategy is then needed when the time perspective is long. Microfilm What it is: Microfilm is 16mm or 35mm film stored on open reels or in cassettes. It contains small images, usually in black and white. Although some microfilm formats also support digital data. The two main types of microfilms are: Silver halide film: The recommended alternative for long-term preservation. The image is transferred to the film by using silver emulsion on a polyester strip. Vesicular film: Creates the image on the polyester strip by using microscopic bubbles instead of silver emulsion, making it a less expensive but also a less durable solution. PAGE 5
Special cameras capable of photographing at reduced size are used in order to transfer an image to microfilm. The image is then printed on the film and chemically processed in a laboratory. The film processing makes the recording process more complex compared with some of the alternative technologies. In spite of this, microfilm is widely used for long-term preservation of data, for a number of reasons. Long durability and no need for decoding: Microfilm has a lifetime expectancy of up to several hundred years when stored properly at the recommended temperature and humidity levels. Additionally, as the data is Microfilm reader normally analogue, data stored on microfilm can be read back by simply using a magnifying glass. Hence data retrieval is independent of specific reading devices, and the images on the film require no software decoding. Desktop readers with large screens and zooming lenses are normally used for data retrieval. The downside is that it is a manual and cumbersome process to access and reproduce data stored on traditional microfilm. A migration-free WORM: The durability and secure read-back of analogue data make microfilm a migration-free storage alternative, potentially creating large cost savings and eliminating the risk of migration-related data loss. By being an analogue medium stored offline, microfilm is also a secure solution. As a true WORM (Write Once, Read Many) it is literally impossible to edit or delete the information that is written to the film. Hence microfilm meet many of the criteria for long-term preservation of valuable data. A microfiche has many of the same attributes as microfilm, but is formatted as a card rather than as a film reel. They are normally stored in open top envelopes in drawers or boxes. An ultrafiche is a compact version of a microfiche, storing images at much higher densities and often made directly from computers. Manual handling: The major drawback with microfilm is that the workflow for reproducing data is time-consuming and largely manual. Microfilm was designed for storing analogue images, and in a world overflowing with digital data, traditional microfilm is simply not an efficient alternative. Piql Preservation Services What it is: Piql Preservation Services is a solution that has been designed with the needs of long-term digital preservation in mind. The result is a system that allows digital data to be preserved safely and efficiently, and easily retrievable independent of future access to specific technologies or vendors. With Piql Preservation Services, data is written on to photosensitive film, which has a proven longevity of several hundred years. The solution includes all equipment and processes needed to write and read-back data. A data scanner is developed for high-quality data retrieval, but the data can also be recovered with simple means such as a light source and some sort of digital camera and computer. Future-proof WORM: Piql Preservation Services offers many of piqlbox: film final package PAGE 6
the same advantages as microfilm when used for long-term preservation. As a true WORM stored offline it is impossible to manipulate the data once it is written and it has a long projected lifespan. It is also migration-free, both in terms of storage medium, reading device and file format (provided that archival formats are used). The latter is due to the fact that the file format specification can be written in readable text on the storage medium. For digital and analogue data: Piql Preservation Services differs from traditional microfilm in enabling digital data to be stored alongside visual images, making it suitable for the digital era we have entered. Digital data written as 0 and 1 obviously need decoding to get back the original file. To make data retrieval future-proof, explanations of how to decode the information is written as human readable text on the storage medium. The flexibility of combining digital and analogue information opens up for new possibilities and additional security. Search and access files: Piql Preservation Services allows for metadata Digital and analogue data searches and accessibility on a different level than traditional microfilm. As a seamless element within a standard IT infrastructure, users can search for the requested file and get it back in original format within minutes (the real retrieval time will vary according to each user s set-up and the size of the film). Migration-free long-term preservation: Piql Preservation Services is not recommended for data that needs to be instantly accessible to ensure business continuity. It is rather a safe, efficient and future-proof option for valuable data that needs to be preserved for future use. Different technologies for different needs: We have seen that each of the storage technologies have different characteristics, making them suitable for different needs. Data owners need to thoroughly consider their requirements in terms of e.g. acceptable data retrieval time, security level and budget limitations during the archival period. The following table provides an overview of some of the attributes of each of the storage technologies. It is important to keep in mind that this is a general overview of technologies constantly in development. Within each category, there are differences between vendors and products. PAGE 7
Different Storage Technologies Magnetic Tape Optical Disks Hard Disk Drive Cloud Microfilm Piql Preservation Services Projected lifetime 6 30 100 6 N/A 500 500 Analogue data 7 Digital data True WORM 8 9 Offline format 10 Instant retrieval Vendor-neutral reading device Migration-free storage medium True archival format 11 6 General figures, there are variations from vendor to vendor 7 Human readable text or pictures 8 Write Once, Read Many, meaning the data is unalterable after being recorded on the storage medium 9 CD-R and DVD-R only 10 Does not require any maintenance of the medium to ensure data integrity 11 Self-contained format: descriptions of how the data is stored, file formats, information on how to decode the data, etc. Piql AS provides solutions within digital preservation through a worldwide network of partners. With its cutting-edge technology Piql has reinvented the use of photosensitive film by converting it to a digital preservation medium. Piql is also behind Cinevator, the world s leading digital film printer. Founded in 2002, the company is based in Drammen, Norway. PAGE 8 Piql AS, Grønland 56, 3045 Drammen, Norway 1+47 90 53 34 32 piql.com