The Digital Domain and Media Storage: How the Moving Image Archive Survives in Two Different Worlds Joshua S. Harris Archives Manager National Geographic Society Film and Audiovisual Archives Digital Media Group Introduction: One of the largest issues for all archives globally is STORAGE. This is even more important and complex in the Digital Domain as archivists become responsible for both physical and digital storage. In this regard, the modern audiovisual archives truly survives in two different worlds 1
Background: Three years ago The National Geographic Society embarked on a large scale moving image digitization project. A multitude of theories and assumptions were made at the beginning of this project. The archives has learned a great deal about physical and digital storage. Many theories have been proved WRONG! Intro to The National Geographic Film Archive Main collections repository for the film and audiovisual legacy and continuing work of NGS. All collections materials from 1903 to present including the archives for all NGS documentary production. Field rolls, outtake rolls, interviews, show masters, research tapes, misc. collections ~25,000 film elements, ~70,000 analog tapes, over ~40,000 digital tapes, ~3,000 HD tapes 2
35mm film 16mm film Super 16mm film 8mm film Super 8mm film 70mm film 2 quad 1 quad 3/4 Umatic 1/2 VHS Betacam BetacamSP BetacamSX Digital betacam Mini dv Some Film/Video Formats DvcamDvcpro Hi-8 Da-88 D-1 D-2 D-3 D-5 HD HDcam HDcamSR DVCPro-HD DV-HD DVD DLT (Digital linear tape) HDD (Hard-disc drive) On site Collections Vaults: Master Tape Vault: 42,300 items 3
P1 Cold Vault: ~45,000 Film and Tape Elements Research Tape storage: ~4,000 items 4
The Active Archive Inside the NGS Film Archive supports institution-wide projects Support documentary film production and post-production units Active library for film, video and audio materials: circulation and loan unit. Ever-increasing need to view, listen and utilize footage in multimedia projects throughout NGS Duplication facility The Two Different Worlds Analog World: All tape, film, and physical collections. The traditional archive situation. Storage: Traditional Vaults with Shelving. Retrieval: By hand. Database searching and physical retrieval. Pre-2005: Archive responsible for accessioning, retrieval, access and collections maintenance of tape-based and film materials. 5
The Two Different Worlds DIGITAL WORLD: Collections of digital materials. Either digitally generated collections: analog and tapebased assets migrated to digital or digitally-born assets (Example: Web-films, or digital capture). NGS storage: -Off-line > DLT tapes, -Near-line > Servers and accessible hard-drives -On-line > Web flavors and viewing copies on web-servers 2005-0n: Archive responsible for retrieval and access to analog AND DIGITAL COLLECTIONS. Storage Theory: Digitization of an asset would render access to the original element less important. After successful digitization, assets would be sent to deep archival storage offsite. This would free up very valuable physical space at the Society headquarters The original asset would be preserved and would not be needed for archival needs. 6
A very simple scenario: Original Archival Tape Tape Encoded Digital Elements sent to Digital Storage For archival needs. Tape Sent to Offsite Storage for permanent preservation Space Cleared in NGSvaults Our Theory in Practice Digital encoding of analog and storage of digital assets rises at a very high pace. -Expected. (with some unexpected occurrences) Analog and physical tape collections continue to grow at a very high pace. -Unexpected! 7
Digital Storage Digital Materials stored on variety of off-line, near-line and on-line devices. -Petasite, DLT Tapes -RAID Storage -Isolon Drives -NGS infrastructure servers. UNEXPECTED: -How fast these fill up. -SLOW Migration off of encoders to storage. -Difficult retrieval at times Our Theory in Practice Reasons for the unexpected rise in tape/film collections: -Newly discovered collections of film - Clean-up of entire NGS -Changes in infrastructure reveal unknown collections (~9000 tapes) -Increase in television and media production: from ~50 hours in 2000 to over 150 hours in 2007! 8
Our Theory in Practice Found collections: 9
Increase in TV and documentary production NGT Tape Vault: ~15,000 Items Cleanup of NGS and infrastructure changes Our Theory in Practice Result: Boxes everywhere! 10
Our Theory in Practice The Traditional archival practices, needs and storage practices DO NOT END because of the move to a digital environment. We needed to INCREASE onsite storage rather then decrease as we thought. Boxes Everywhere! Thousands of unprocessed collections elements Our Theory in Practice: The Digital World Digital Storage has proved not as easy as expected: -Storage failures -Slow migrations. -Inability to locate files / bad metadata: often due to rushing through work due to encode priorities. -Restoring of files often slow and unreliable. - Playout software often difficult to work with. -NGS IS infrastructure problems: Web research not possible. Much negative feedback from NGS users 11
Access to Footage Tape encode problems often make access to original tapes necessary. Problems with searching and retrieving useful files for researchers has made access to tape even more necessary. Time to restore uncompressed files may make it easier to Grab the Tape in a time bind. As physical storage of collections grows, digital storage of collections grow Collection Growth 2005-08 35000 30000 25000 20000 15000 10000 5000 0 2005 2006 2007 2008 Video/Film/ Audio Digitized Elements 12
As we well know: As physical collections continue to grow Physical Storage decreases Add Digital Growth + + = Digital collection growth: Including time, expense, resources Physical collection Growth Physical Archival Storage: With decrease in time and resources 13
= Collection Neglect! Over the past three years: <1% film collections preserved, recanned or examined <1% of untransferred films transferred or retransferred <1% of older formats migrated to more stable carrier (ie. over 6000 1 tapes in collection) AND <1% Digitized COLLECTIONS MOVED OFFSITE Neglect of Off-site Storage Vaults: Over 75,000 Items inside over 8,000 boxes! Materials sent here to move them out of the way. Not due to informed collection decisions. 14
= Collection Neglect! Fortunately, this is climate controlled Although HVAC system is 25 yrs. OLD! = Facilities Neglect! 15
Conclusions Digitization take carefully planned storage of both digital files AND traditional archival storage. The proper resources MUST be set up far in advance in order to best survive in the TWO DIFFERENT WORLDS DO not expect that as the migration to a Digital World occurs that traditional analog collections will decrease: They may increase MORE! Too much digital, too fast can have a negative impact on storage of digital and physical collections! 16
Our Theory? DISPROVED!! Collection digitization has made access to the originals equally if not more important. Our storage situation and preservation of original collections has sufferred rather than improved! 17