How To Build A Cloud Storage System

Transcription

1 Reference Architectures for Digital Libraries Keith Rajecki Education Solutions Architect Sun Microsystems, Inc. 1

2 Agenda Challenges Digital Library Solution Architectures > Open Storage/Open Archive > Cloud Computing Solution Benefits Summary Next Steps 2

3 Challenges 3

4 Value of Reference Architectures Minimizes cost, complexity and deployment time >Lowers administrative costs via automated data management and migration across storage tiers >Cost-effectively matches the value of data with the appropriately priced media >Economical and power-friendly cost of operation Flexibility to build performance, economy or mixed archive repository >Infinitely scalable archive management 4

5 Sun Reference Architectures Develop Collaborative, Replicable Reference Architectures Fedora Fedora/Drupal DSpace EPrints Duraspace Ex Libris Rosetta VTLS VITAL Internet Archive in a Sun Modular Datacenter 5

6 Digital Library Sun PASIG Community Government Librarians Students Staff Faculty Researchers Business Museums Integrated Library Digital Asset Mgt. System (ILS) Acquisition & ILL Content Repositories: VITAL Rosetta,Fedora, EPrints, DSpace ejournals, Serials Cataloging Digital Objects Long-term Preservation Search and Discovery Systems Data Curation & Collab. Digital Library and Repository Consulting, Storage, and Data Management Networking Identity Web Services Scalability, Security, Sustainability, Sharability 6

7 Sun Open Storage Product Positioning StorageTek 7410 SAM/QFS StorageTek7210 Identity Management and SOA Digital repositories for data and metadata storage Fedora, EPrints, and D-Space communities Ex Libris Rosetta and VTLS VITAL applications Large scale preservation projects needing policies Digital asset management eresearch databases Federated repositories 7

8 Virtualized Repository Appliance Digital Objects Single Virtualized Server Virtual Machine 1: Repository Entity Preservation Index Creation Metadata Management Security Search Engine Archive Repository App. Oracle, MySQL Solaris + ZFS Virtualized Server Physical Storage Virtual Machine 2: Archive DB Policies Metadata Virtual Machine 3: Open Storage Mgmt Storage Preservation Physical Storage 8

9 Open Repository Tiered Components User Digital Objects Application Server Entity Preservation Metadata Management Relationship connections Security Search Engine Policy Driven Open Storage Storage Preservation Abstraction OpenSolaris, ZFS, SAM Physical Storage Components Media Migration DB Server Digital Asset Policies Metadata Tape Libraries Full Sun portfolio of supported tape libraries 9

10 Sun s Infinite Archive System Approach Factory integrated, solution tested, simple, scalable, and economical Library DB Surveillance Web Unstructured IP Communications Protocol Infinite Archive System (IAS) Models: Value and Midrange X4200 T5220 Intelligent, Policy-Based Automated Archive IAS GUI, Storage Archive Manager SAM-QFS, Sun Cluster 3.2, Solaris Infinite Archive System (IAS) Tier 1 + Tier 2 Disk IAS Options 2500 series SAS 2500 series SATA Scalable, Eco-Efficient Tape Tier Options SL8500 SL3000 LibrariesEncryption SL500 Access & Capacity Drives 10

11 11

12 Cloud Computing 12

13 Emerging Cloud Deployment Patterns Test and Development Functional Offload (Batch Processes TimesMachine) Functional Offload (Storage SmugMug) Augmentation (Temporary Load Animoto) Web Service 13

14 Storage Service What It is > On-demand, API-based access to storage on the network Features > Ability to store and retrieve data as objects or files > REST API with open, AWS S3-like semantics for > > > > object storage WebDAV for file storage Fast and inexpensive cloning of objects and files High availability Detailed metering of storage used, I/O requests, bandwidth, etc. Customer Benefit > Scalable, highly available storage without big hardware investments 14

15 TACC: World s Top Supercomputer Sun Fire X Systems 1.7 petabytes 64.8 GB/sec total bandwidth Sun Fire X systems 800 cores SunBlade ,936 blades 15,744K CPUs 62,976 cores 125 TB/RAM Sun Data Center Switch 3456 Dual redundant 110 TB/sec bisectional bandwidth The world s largest largest computing system in the world for open science research Sun Constellation Linux Cluster and Sun StorageTek Mass Storage Facility Tflops peak performance 15

16 Internet Archive Highest Density Integrated Storage Architecture Key Requirements Build a server infrastructure to support massive amounts of data 2 PB of storage, growing by 1 PB per year Provide an efficient, reliable, and scalable datacenter Keep space, energy, management and maintenance costs low Web snapshot 100 TB of data - approximately 4 billion Web pages. Sun Solution Client Results Sun Modular Datacenter S20 Sun Fire X4500 Server Solaris 10 with ZFS Sun Remote Operations Management Gained a reliable and flexible datacenter that supports multiple PB of storage Increased storage capacity of its servers Reduced space and energy needs for lower costs Superior data integrity to guard against data loss Rapid time to deployment Sun MD unit delivered in less than 45 days Support up to 500 user queries per second. 16

17 Next Steps Sun Matching Grant Up to 50% Sun Edu Essentials On-going Discounts Try & Buy 60 days up to 40% off Sun Products Open Archive Architecture Assessment Participate in the Development of Open Cloud APIs Sign up for Early Access to Sun Cloud Services Contact Your local Sun Partner or Account Team 17

18 For More Information Storage Archive Manager Join the Sun Preservation and Archiving Community Join the OpenSolaris Storage community Open Storage Open Storage Servers 18

19 Thank You Keith Rajecki Education Solutions Architect Global Education & Research Sun Microsystems, Inc. 19