NORTH CAROLINA STATE ARCHIVES. Preservation Storage in Real Life

Transcription

1 NORTH CAROLINA STATE ARCHIVES Preservation Storage in Real Life

2 North Carolina Department of Cultural Resources Works closely w/ State Library Electronic Records Branch (2 FTE) State Archives of North Carolina

3 WHERE ARE WE NOW?

4 INTRASTATE BACKUPS Offsite storage: Asheville, NC Local storage Images from

5 INTRASTATE BACKUPS THE GOOD Duplicate copies of AIPs Other side of the state Our storage / control Easily expandable THE NOT-SO-GOOD Slow Other side of the state Not automated (yet) Offsite storage: Asheville, NC Local storage Images from

6 CONTEXT: OAIS

7 CONTEXT: OAIS

8

9 CLOUD STORAGE Offsite storage: Asheville, NC Local storage Cloud storage Images from and

10 CLOUD STORAGE 1X uploads DuraCloud Offsite storage: Asheville, NC Local storage Images from and

11 WISH LIST Transfer AIPs to backup storage Validate transfer Integrity checks of AIPs in backup storage Replace backup AIPs that fail integrity checks Integrity checks of local AIPs Replace local AIPs that fail integrity checks

12 DURACLOUDFUNCTIONS Transfer AIPs to backup storage? Validate transfer Integrity checks of AIPs in backup storage Replace backup AIPs that fail integrity checks? Integrity checks of local AIPs Replace local AIPs that fail integrity checks

13 DURACLOUDFUNCTIONS Transfer AIPs to backup storage Validate transfer Careful use of tools to prevent Integrity checks of AIPs in backup storage local errors being propagated to the cloud Replace backup AIPs that fail integrity checks Integrity checks of local AIPs Replace local AIPs that fail integrity checks

14 DURACLOUDFUNCTIONS Transfer AIPs to backup storage Validate transfer We do nothing. DuraCloud does everything. Integrity checks of AIPs in backup storage Replace backup AIPs that fail integrity checks Integrity checks of local AIPs Replace local AIPs that fail integrity checks

15 DURACLOUDFUNCTIONS Transfer AIPs to backup storage Validate transfer Integrity checks of AIPs in backup storage DuraCloud provides us the Replace backup AIPs that fail integrity checks means to do this: fixity values of uploaded files (MD5 hashes). Integrity checks of local AIPs Replace local AIPs that fail integrity checks

16 DURACLOUDFUNCTIONS Transfer AIPs to backup storage Validate transfer Integrity checks of AIPs in backup storage Tricky! Replace backup AIPs that fail integrity checks Integrating tools is tough. Integrity checks of local AIPs Replace local AIPs that fail integrity checks

17 DURACLOUDFUNCTIONS Transfer AIPs to backup storage We need to know when to put DuraCloud to use! Validate transfer Integrity checks of AIPs in backup storage Tricky! Replace backup AIPs that fail integrity checks Integrating tools is tough. Integrity checks of local AIPs Replace local AIPs that fail integrity checks

18 CLOUD STORAGE: DURACLOUD We need to know when to put DuraCloud to use! DuraCloud Offsite storage: Asheville, NC Local storage

19 ADDING CONTENT TO DURACLOUD Part 2

20 ~1,400,000 files ~25 TB digitized & born-digital government records & special collections CONTENT structured by agency/collection usually Images from

21 CONTENT STRUCTURE -Hierarchical directories with many layers BagIt bag -AIPs often packaged in BagIt bag structure

22 DURACLOUD SYNC TOOL Sync tool DuraCloud Local storage

23 DURACLOUD SYNC TOOL Sync tool can do all this Run continuously in the background Upload new files to cloud as soon as they are added to local storage Delete existing files as soon as they are removed from local storage Replace existing files as soon as they are altered on local storage Sync tool But we only use it for this One-off uploads Resume uploads that were interrupted

24 DURACLOUD - AIP-blind -Flat structure - Original file path is unique identifier

25 DuraCloudID: Original file path DURACLOUD Modified timestamp of record about file - AIP-blind -Flat structure Spaces (single level of hierarchy) MD5 hash - Original file path is unique identifier Timestamps of local file (timestamp tool)

26 TESTING: WHAT WE LEARNED Easy to overwrite files on upload Integration of unique identifiers is important Local Storage unique ID Cloud Storage unique ID d7a8fbb307d ca9a bcb0082e4f8d5651e46d3c db762d02d0bf37c9e592 Creating backups is easy. Knowing when to use them is tough. Preservation/Digitized_Maps /County_Based/Dare/1947_ Aerial_Photos/BodieIsland S13-1.tif

27 Part 3

28 ALL FILES CREATED EQUAL? Digitized Born-digital +

29 TRUSTED DIGITAL REPOSITORY? The repository shall have adequate hardware and software support for backup functionality sufficient for preserving the repository content and tracking repository functions The repository shall have effective mechanisms to detect bit corruption or loss The repository shall have mechanisms in place to ensure any/multiple copies of digital objects are synchronized The repository shall have suitable written disaster preparedness and recovery plan(s), including at least one off-site backup of all preserved information together with an offsite copy of the recovery plan(s).

30 OAIS?

31 Third party storage is a vital part of our strategy DuraCloud has been receptive to feedback Definite room for improvement (see comment above) in terms of what we want it to do For some, this solution is exactly what they need

32 CONTACT INFO Kelly Eubank Rachel Trent