ECMWF HPC Workshop: Accelerating Data Management

Transcription

1 October 2012 ECMWF HPC Workshop: Accelerating Data Management Massively-Scalable Platforms and Solutions Engineered for the Big Data and Cloud Era Glenn Wright Systems Architect, DDN

2 Data-Driven Paradigm Shift Massive Data Growth Cloud & Web Ubiquity Big Data Processing 40 35x Growth In 10 Years Zettabytes Source: IDC Storing over 100PB of content 1 33x Sensor Data Growth Over Just Two Years 2 Information, Today s Global Currency, is Shared & Distributed Globally via Cloud Information managers must fundamentally rethink their approach to data. 1 Facebook: 2012 Form S-1 Registration Statement 2 Defense Industry Daily: Too Much Information 2 6/8/12

3 DDN Big Data Leadership Validated If any company is well poised to take on the challenges of EXAScale computing and big data, it's DDN, since this is its heritage. February 2012 Strategic Big Data HW Supplier DDN Ranks #2 in 2012 CIO Survey #1 = IBM #2 = DDN #3 = EMC #4+ = NTAP, FIO, Dell 24% of Respondents Identify DDN as Their Top Choice for Big Data June /8/12

4 DDN Designed for Big Data & Cloud Massively Scalable Storage Technology Cloud Storage & Computing Infrastructure Big Data Processing for Actionable Insight HyperScale, High Performance Platform Peer to Peer Cloud Infrastructure Big Data Processing System DDN s Massively Scalable SFA :S/W Engine on Enhanced H/W Platforms Over 1TB/s In Only 25 Systems, Millions of IOPS DDN s WOS Cloud-Based Data Delivery 55 Billion Objects Per Day, 100+ Locations DDN s SFA In-Storage Processing Nanosecond Latency, 16 Virtual Machines 4 6/8/12

5 DDN A Decade+ Investment In Scale DDN FOUNDED 1 st Customer NASA SFA Project Inception WOS Project Inception Largest private storage co. (IDC) 500+ EMPLOYEES S2A6000 S2A3000 S2A8000 S2A9550 S2A9900 AWARDS K 12K 5 6/8/12

6 The Broadest Big Data Portfolio File Storage Object Storage SAN File Storage Enterprise NAS xstreamscaler 100s SAN/LAN Clients HSM Capable Streaming Optimized NAS Scaler 1-16 NAS Servers Fully-Featured High Performance Parallel File Storage EXAScaler 10Ks of Clients 1TB/s+ GRIDScaler 1Ks of Clients 1TB/s+ Scale-Out NAS Storage Engines Silicon Storage Architecture 9900 Storage Fusion Architecture K 12K WOS 256 Billion Objects GeoReplicated 6GB/s in Real-Time 1,200 Drives: 2 Racks 2GB/s, 350K IOPS 120 Drives in 8U 15GB/s, 1.2M IOPS 1,200 Drives: 2 Racks Embedded Computing Cloud Foundation Mobile Cloud Access 40GB/s/1.7M IOPS 1,680 Drives: 2 Racks Embedded Computing Platforms Support Any Drive Technology to Optimize Data Intense Workflows SAS 6 6/8/12 SATA SSD

7 DDN Storage Fusion Architecture (SFA) Accelerating Big Data and Cloud, Optimizing TCO Over 1 Million Lines of S/W Code First Customer Shipped 2008 Designed Specifically for Big Data and Cloud Workloads DirectMon : Infrastructure Management Scaler File System Family In-Storage Processing Engine & DMA Driver Storage Fusion Architecture [Core Storage S/W Engine] Low-Latency Connect: FC, IB, Memory Interrupt-Free Storage Processing ReACT Adaptive Cache Technology DirectProtect Data Integrity Management Quality of Service Engine Storage Fusion Fabric Parallel State-Machine Design Maximum Performance, Lowest Latency Virtualized Processing Optimized Environment for Big Data Application Hosting Robust Data Protection Quality of Service and Performance Without Compromise Flexible & Massively Scalable Best-In-Class Scalability and Density 7 6/8/12

8 Data Sensors: The Challenge Increasing amount of data captured per hour by the next generation sensors. Increasing number of multi-sensor, ISR platforms being developed and deployed. Increasing linger capability has a multiplying effect on the amount of data

9 Conventional Storage 9 6/8/12

10 Object Storage Explained Object storage stores data into containers, called objects Each object has both data and user defined and system defined metadata (a set of attributes describing the object) File Systems Objects Flat Namespace Traversals Extent Lists File Allocations Metadata Metadata Metadata Data Data Data File systems were not designed to store billions of files on a global scale Objects are stored in an infinitely large flat address space that can contain billions of files without file system complexity

11 DDN WOS Enabling Real-Time Global Collaboration Web-Scale, High-Performance Cloud Storage Appliances 99% Efficiency, Petabyte-Class Peer:Peer Technology Connectors WOS Cluster Management WOS API HTTP, C++, Java Replication Engine Android & ios Client WOS NAS Access API: S3 Client Desktop Client WOS Cloud Multi-Tenancy WOS Core [Peer:Peer Object Storage] Latency-Aware Access Manager Self-Healing Object Storage Clustering WOS Policy Engine ObjectAssure Erasure Coding De-clustered Data Management Limitless Scalability Eliminates the limitations of traditional file systems Store Objects Intelligently User-Defined Metadata allows customers to understand their data Global, Peer:Peer Distribute data across 100s of sites in one namespace Self-Healing Intelligent Data Management system recovers from failures rapidly and autonomously 11 6/8/12

12 DDN irods 12 6/8/12

13 DDN The Big In Big Data 98% Fortune 500 Customers of Online File Service PBs of Cloud Storage 768 Thousand # of Supercomputer CPUs 35PB Storage Processing System 23 Million Broadcast Viewers 9PB of Real-Time Play-To-Air Systems If any company is well poised to take on the challenges of exascale computing and big data, it's DDN, since this is its heritage. 451 Group 13 6/8/12

14 October 2012 Accelerated Data! Massively-Scalable Platforms and Solutions Engineered for the Big Data and Cloud Era Thank You