EMC IRODS RESOURCE DRIVERS

Transcription

1 EMC IRODS RESOURCE DRIVERS PATRICK COMBES: PRINCIPAL SOLUTION ARCHITECT, LIFE SCIENCES 1

2 QUICK AGENDA Intro to Isilon (~2 hours) Isilon resource driver Intro to ECS (~1.5 hours) ECS Resource driver Possibilities (~1 hour) Requests (rest of the evening) 2

3 EMC ISILON SCALE OUT NAS 3

4 ISILON SCALE-OUT NAS Simplicity and ease of use Single file system, single volume, global namespace Massive scalability Scales to 40 PB in a single file system World record performance Over 100 GB/s throughput, 1.6M SPECsfs ops Unmatched efficiency More than 85% storage utilization, and automated storage tiering Enterprise data protection Efficient backup and recovery, reliable disaster recovery, and N+1 thru N+4 redundancy Robust security options Roles-based administration; Authentication Zones; SEC 17a-4 compliant WORM data security Operational flexibility Integrated support for industry-standard protocols including NFS, SMB, HTTP, FTP, and HDFS 4

5 THE MOST RELIABLE STORAGE SYSTEM Built-in high availability clustered architecture With N+2, N+3, and N +4 With protection, N+1 protection, data is is 100% available if even multiple if a single drives drive or nodes or node fail fails FAILED 100% 100% 100% 100% Fastest rebuild time, and with Isilon, the more nodes in the cluster, the faster drive rebuild time FAILED 100% 100% 100% 100% 5

6 Mix and Match Any NODE at Any Time Tune Performance to Achieve the Needed Performance and Capacity Cache Acceleration High IOP Performance High Concurrent Throughput High Capacity Nearline Archive A-Series NODES S-Series NODES X-Series NODES NL- Series NODES OneFS 6

7 HDFS IMPLEMENTED LIKE A NAS PROTOCOL 1) Request( /file ) DFSClient Hadoop Node OneFS runs a daemon that speaks NameNode and DataNode natively NameNode DataNode OneFS Node 2) Response (block locations) NameNode DataNode OneFS Node 3) GetBlock(block) NameNode DataNode OneFS Node OneFS Clustered FileSystem NameNode DataNode OneFS Node 7 7

8 IRODS RESOURCE DRIVER FOR ISILON Uses HDFS to communicate with the cluster Parallel reads, massive throughput Setup as primary or archive resource (compound or hierarchical) 8

9 EMC ELASTIC CLOUD STORAGE (ECS) 9

10 COMPLETE CLOUD STORAGE PLATFORM Lower cost than public cloud Unmatched combination of storage efficiency and data access Anywhere read/write access with strong consistency simplifies application development No single points of failure increases availability and performance Universal accessibility eliminates storage silos and inefficient ETL processes Comprehensive types satisfy the broadest range of application needs 10

11 ECS SOFTWARE ARCHITECTURE CLOUD-SCALE STORAGE SERVICES ON COMMODITY ECS Software OBJECT STORAGE HDFS STORAGE Geo-Replicated Data Protection Active-Active read/write support with strong consistency No single point of failure Performance and efficiency for small and large objects SITE 1 SITE 2 SITE 3 11

12 GEO-REPLICATED DATA PROTECTION PROETECTION AGAINST LOCAL AND GLOBAL FAILURES Handles local hardware and full data center failures Disk, Node, Rack, Data Center are failure domains Site 1 Site 3 Site 2 Local hardware failure recovery requires no WAN traffic New Hybrid encoding approach enables low storage overhead 12

13 GLOBAL CONTENT REPOSITORY ON-PREMISE UNSTRUCTURED STORAGE PLATFORM PROBLEM Applications Tiering, Archiving, Backup Can t cost-effectively manage or scale storage to support explosive growth in unstructured content. Traditional storage not suited for new Web, mobile and cloud applications. Difficult and costly to manage data lifecycle and retention policies across archive silos and sites SOLUTION EMC ECS Appliance (Object and HDFS) VALUE Reduce complexity and cost one globally accessible, geo-efficient archive that serves multiple applications and content types at lower cost than public cloud. Anywhere data access All data globally accessible by Web, mobile and cloud apps. Enterprise-grade data protection Efficient geo-protection and policy-based retention for basic compliance and governance. Memphis L.A. U.K. 13

14 IRODS RESOURCE DRIVER FOR ECS Uses the Atmos protocol Parallel reads and writes Auto-balancing across the ECS nodes 14

15 WITH EACH DRIVER ViPR-Sync HDFS Atmos 15

16 REQUESTS Connection based/pooled streams for I/O plugins (in 4.1?) Parallel/batch job execution in irods Bulk operations (copy, move, etc.) of objects & metadata 16

17