Data Services for Campus Researchers

Transcription

1 Data Services for Campus Researchers Research Data Management Implementations Workshop March 13, 2013 Richard Moore SDSC Deputy Director & UCSD RCI Project Manager

2 SDSC Cloud: A Storage Paradigm Shift for Access, Sharing and Collaboration Launched September 2011 Largest, highest-performance known academic cloud 5.5 Petabytes (raw), ~2PB usable, 8 GB/sec Automatic dual-copy and verification Capacity and performance scale linearly to 100 s of petabytes Open source platform based on NASA and RackSpace software for more information 2

3 Key Features of SDSC Cloud Easy access to shared data for any users with permission under range of mechanisms (http, APIs, portals, gateways ) Simple data owner user interfaces to data, its management, its access and setting permissions for sharing data Always-there disk-based availability of data High reliability Disk RAID; automatic dual-copy; continuous background checksum verification/ restoration; offsite replication soon Encryption readily incorporated Transaction history logged track usage, assess utility, support provenance Scalable system in both capacity and bandwidth Interfaces to commercial and open-source products

4 Applications of SDSC Cloud Shared/published/curated data collections HPC simulation data storage and sharing Web/portal applications and site hosting Application integration using supported APIs Serving images/videos Backup services

5 SDSC Cloud Interfaces Data Owners Traditional Clients GUI Applications Command Line SDSC Web I/F Load Balanced Proxy Servers External Users Web Services API Amazon S3 Rackspace CloudFiles / Openstack API Swift Object Storage Cluster Commercial Products Amanda Backup Tools Commvault Crashplan Panzura User- Developed Web Portals/ Gateways

6 SDSC Cloud Explorer

7 Developing interface between Globus Online and SDSC Cloud Developing a GridFTP Data Storage Interface to OpenStack s Swift Object Storage (Swift-DSI) modules are 80% complete Developed authentication system between GridFTP and Swift Directory-level operations under development Going beyond Swift s one-level directory hierarchy (folders in Swift)

8 First Year s Experience w/ SDSC Cloud Adoption primarily by large-scale users w/ commercial interfaces Backup services (Commvault), national lab (Panzura) More typical feedback from individual researchers Interfaces are not easy for integration with normal tools POSIX access is more familiar than object-based storage Service is too expensive Data sharing only slowly emerging as a bottoms-up priority amongst individual researchers Recovering total cost of ownership, without campus or extramural investments, is a challenge And depreciation costs just get tougher to recapture later in equipment life We expect this to be an increasingly valuable resource but adoption takes time and there remain barriers to entry for individual PIs Want to improve interfaces and access/authentication (e.g. InCommon)

9 UCSD s Research Cyberinfrastructure Program : Research Cyberinfrastructure Design Team (RCIDT) Chaired by Mike Norman and Phil Papadopoulos; broad campus participation Campus-wide survey of research cyberinfrastructure needs (2008) April 2009: RCIDT issued Blueprint for the Digital University ( : CyberInfrastructure Planning & Operations Committee (CIPOC) developed a business plan with recommendations A principle of shared costs between PIs and campus RCI investments April 2010: CIPOC issued their report 2011-Present: RCI Oversight Committee charged to implement RCI January CIPOC business plan accepted, oversight committee charged Chaired by Mike Norman and Mike Gilson; broad campus representation

10 Data Center Colocation Networking Elements of UCSD s Integrated Research CyberInfrastructure Centralized Storage Data Curation Research Computing Technical Expertise

11 A Data-Centric view of UCSD s RCI: From the 2009 Blueprint report

12 RCI is evolving SDSC s Triton system to the Triton Shared Computing Cluster (TSCC) Condo model: Researchers purchase compute nodes which are operated as part of shared cluster for 3-4 years PI buys hardware & modest ops fee Lower operations cost than local PI cluster; larger-scale resource available (core count and capacity); professionally-managed Hotel: Purchase time by the corehour; shared queue Research Computing (in production now)

13 Completing a two-year pilot phase Data Curation in pilot (production FY13-14) How do lab personnel work with librarians to curate their data? How much work is required to curate data and what are options? What is a sustainable business model for curation within RCI project? Five representative programs across UCSD selected as pilots The Brain Observatory (Annese) Open Topography (Baru) Levantine Archaeology Laboratory (Levy) SIO Geological Collections (Norris) Laboratory for Computational Astrophysics (Wagner) Using existing tools whenever possible Storage at SDSC, campus high-speed networking, Digital Asset Management System (DAMS) at UCSD Libraries, Chronopolis digital preservation network Also, develop Data Management Plan tools and provide training Anticipate production curation services in FY13-14

14 Centralized Storage (phased production thru 2013) Completing interviews of a broad sample of ~50 representative PIs to understand technical and cost requirements Identify common needs, and define sustainable RCI business model with strong adoption Anticipate production centralized storage services in CY13 RCI Network-Attached Storage (RCI/NAS) available Spring CY13 Further services to be rolled out throughout the year, based on requirements analysis

15 PI Interview Responses: How do You Handle Data Storage/Backup? Numbers reflect percentages of PIs surveyed that utilize each solution ; Individual PIs use multiple solutions, so % s add up to >100%. Storage Devices Network accessible storage (NAS), USB and server local drives dominate Use of Dropbox for sharing Others use Google Drive, Hadoop, XSEDE, SDSC colocation Backup modes Replicated copies in two NAS A copy in the NAS, A copy in local hard drive (laptop/workstation), And a copy in a USB drive Maybe a copy in /dropbox Problems: Out of sync Lost track of its location Lost version control High cost of recovery

16 PI Interview Responses: Where is Your Data Coming From? Numbers reflect percentages of PIs surveyed that utilize each solution ; Individual PIs use multiple solutions, so % s add up to >100%. Indicates use cases for connectivity requirements Data sources: ~50% campus instruments ~30% simulations (XSEDE, campus, lab systems) ~20% field instruments ~15% other external sources % s reflect PIs, not data volume

17 PI Interview Responses: How much storage do you need: now, future, permanently? For PIs interviewed, current needs TB Increasing in future Perceptions of permanent storage interesting none for some, intermediate for many, large for a few

18 Interview responses: Metadata and Do you need metadata annotation capability? longevity requirements How long do you need to retain your data?

19 Interview responses: What are you willing to pay for storage? Intentionally posed a relative question rather than absolute $ s PIs comfortable with hardware costs but Often ignore the cost of lab staff to operate equipment Hard to compete with slave labor (Jim Pepin)

20 Data Life Cycle Services for Campus Researchers Initial Services (currently available) High-performance Lustre PFS for HPC scratch and medium-term parking Centralized network-attached storage (NFS, CIFS mounts) available from campus HPC cluster and remotely across campus (10 Gbps infrastructure) Support to Data Management Plans Next-phase services Build on experience with pilots to develop basic data curation services Improve interfaces for cloud storage to facilitate sharing/access Low-cost backup services for research data HIPAA/PII protected data services Data management tools and improved support to DMPs Looking ahead Tools/technology to support data sharing (via SDSC Cloud, portals, gateways) Develop tools for search, discovery and sharing of data Integrate data management practices into routine research practices