Storage Solutions. Jeffrey van der Hoeven (National Library of the Netherlands) IDCC Workshop on Sustainability, January 2013.

Transcription

1 Storage Solutions Jeffrey van der Hoeven (National Library of the Netherlands) IDCC Workshop on Sustainability, January 2013 Co-ordinated by

2 Outline Introduction Two cases: a library and a research institute Cloud storage: hot or not Costs Survey & statements

3 Partners involved Libraries: British Library, Deutsche Nationalbibliothek, National Library of the Netherlands (KB) Data centres: CINES, CSC IT Center for Science Research organisations: European Space Agency, Fondazione Rinascimento Digitale, CINI ICT companies: IBM

4 Data diversity

5 Two storage cases Case 1: National Library of the Netherlands (KB) Deposit library for all published content in and about the Netherlands. Large-scale digitization of books, newspapers, magazines. Safe place for national & international scientific publications. 458 TB of data stored, 300+ million files.

6 Two storage cases Case 2: CERN European Organization for Nuclear Research, is one of the world s largest and most respected centres for scientific research. Known for its Large Hadron Collider (LHC) recently discovering the Higgs-particle. Distributed storage solution for LHC data 78 PB (80113 TB) of data stored, 340 million files

7 Case 1: storage solution at KB In-house data centre for primary storage and computing Off-site location for back-up Hosting 230 servers (80 physical / 150 virtual) Managing 550 TB of storage space Managing 300 million files

8 Storage prospect at KB 1800m 1 PB 828m Burj Khalifa Dubai 0,5 PB 443m 324m Empire State Building 1.5 million CD-ROM s Eiffel tour

9 Hierarchical Storage Management Books Newspapers Storage tiers Gold Very fast, very expensive Used: indexing, databases SAN with HiPerf SAS disks Magazines Webarchive Journals Parliament Multimedia Silver Steel Bronze Fast, expensive Used: web hosting, processing SAN with HiCap SAS disks Slow (45 sec), sustainable Used: long-term archiving Disk-based NAS with WORM Very slow (> 45 sec) Used: back-up & restore LTO tape

10 Storage strategy Pre-ingest Digital processing Access Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Shared file system File system Storage management Off-site Storage on-site Bronze Bronze Steel Silver Gold Platinum Back-up

11 Case 2: storage solution at CERN Distributed data centres for primary storage and computing CERN Advanced STORage manager (CASTOR) is a hierarchical storage management system developed at CERN for physics data files. Files can be: stored, listed, retrieved, accessed remotely Combination of disk and tape Targeted to large files (> 1 GB) Used for raw data and simulation files Retrieval time: around 4 hours

12 Important factors in long-term storage Volume (size and number of files) Type of data (structured / unstructured) Growth rate Retention (long term or temporary) Availability Costs per TB

13 Cloud storage: hot or not? Storage in the cloud

14 Cloud basics: IaaS: Infrastructure as a Service storage/compu:ng op afstand PaaS: Pla<orm as a Service computer pla<orm op afstand SaaS: So=ware as a Service so=ware tools in de cloud

15 Benefits of cloud storage Scalable Availability Pay per TB per month No need for own ICT infrastructure Less maintenance

16 However Is it sustainable? Who is responsible for the data? Which jurisdic:on is applied? What if I want to migrate to other cloud? No money? No data! Conclusion: be aware of this if you consider cloud for long- term storage. Read on: hkp://

17

18 Storage costs

19 Total Cost of Ownership TCO

20 Many ways of computing TCO for storage A personal favorite (KB): Four Principles for Reducing Total Cost of Ownership (2011 Hitachi) * Describes 34 storage cost components Agnostic for type of storage system * Available at:

21 Cloud storage costs vs in-house KB

22 Can we still afford it? Recent insights *: Disk storage is becoming more popular in archiving. Kryder s law seems to fail, as disk storage density seems not to keep up the pace of a yearly 30-40% increase of storage density. Physical limits of hard disk drive seems reached. Monopoly of hard disk producers Seagate and Western Digital is risky as prices might go up, especially in case of shortage. Conclusion: storage costs are becoming a bottleneck for long-term preservation. * David Rosenthal blog post, available at:

23 We value your input! Short APARSEN survey on storage solutions Topics addressed such as: Mission, preservation policy, legal status Type of data, volume, number of items, ownership Access to data Storage solution, in-house/outsourced Storage storage management (tiering) Back-up or replication (or both) Costs Risk analysis

24 Survey: how to participate? Survey will be out in February / March 2013 Go to And browse to WP23 Storage Solutions Start the online survey Or contact one of the APARSEN project partners.

25 Statements for discussion As storage will become the largest cost component in the long run, we need to be picky about what to preserve. Cloud storage is a nice solution for mid-size organisations.

26 Network of Excellence