Storage Challenges for Petascale Systems

Transcription

1 Storage Challenges for Petascale Systems Dilip D. Kandlur Director, Storage Systems Research IBM Research Division 2004 IBM Corporation

2 Outline Storage Technology Trends Implications for high performance computing Achieving petascale storage performance Manageability of petascale systems Organizing and finding Information IBM Corporation

3 Extreme Scaling There have been recent inflection points in CAGR of processing and storage in the wrong direction! Programs like HPCS are aimed at maintaining throughput at or above the CAGR of Moore s Law in spite of these technology trends Frequency (GHz) Pentium 4 (130 nm) Pentium 4 (180 nm) 2002 Roadmap ~35% yr/yr % yr/yr Prescott (90 nm) Initial Ship Date Maximum Internal Bandwidth Maximum internal Bandwidth MB/s Areal Density Gb/sq.in Disk Areal Density Trend % CAGR 25-35% CAGR Year of Production IBM Corporation

4 Peta-scale systems: DARPA HPCS, NSF Track 1 HPCS goal: Double value every 18 months in the face of flattening technology curves NSF track 1 goal: at least a sustained petaflop for actual science applications New technologies like multi-core will keep processing power on the rise but will make storage relatively more expensive Maintaining balanced system scaling constants for storage will be expensive Storage bandwidth:.001byte/second/flop capacty: 20 bytes/flop Cost per drive will be same order of magnitude, so proportionally the same amount of storage will be a higher fraction of total system cost How to make reliable a system with 10x today s number of moving parts? System Year TF GB/s Nodes Cores Storage Disks Blue P TB 5040 White TB 8064 Purple/C TB NSF Track 1 (possible) TB IBM Corporation

5 HPCS Storage ,000 drives 4 TF 3.6 GB/s 165,000 drives 11,000 drives 6 PF 6 TB/s 100 TF 120 GB/s Fast 5 TB/sec sequential bandwidth 30,000 file creates/sec on one node Capable of running fsck on 1 trillion files CPU Performance File System Capacity CPU Performance Number of Disk Drives File System Throughput Number of Disk Drives File System Throughput 300,000 processors 150,000 disk drives Managable Robust Fix 3 or more concurrent errors Detect undetected errors Only minor slowing during disk rebuild Detect and manage slow disks Unified manager for files, storage End-end discovery, metrics, events Managing system changes, problem fixes GUI scaled to large clusters IBM Corporation

6 GPFS Parallel File System GPFS file system nodes Cluster: thousands of nodes, fast reliable communication, common admin domain. Shared disk: all data and metadata on disk accessible from any node, coordinated by distributed lock service. Parallel: data and metadata flow to/from all nodes from/to all disks in parallel; files striped across all disks. Control IP network Disk FC network GPFS file system nodes Data / control IP network GPFS disk server nodes: VSD on AIX, NSD on Linux RPC interface to raw disks IBM Corporation

7 Scaling GPFS HPCS file system performance and scaling targets Balanced system DOE metrics (.001B/s/F, 20 B/F) This means 2-6 TB/s throughput, PB storage!! Other performance goals 30 GB/s single node to single file for data ingest 30K file opens per second on a single node 1 trillion files in a single file system Scaling to 32K nodes (OS images) IBM Corporation

8 Extreme Scaling: Metadata Metadata: the on-disk data structures that represent hierarchical directories, storage allocation maps, Why is it a problem? Structural integrity requires proper synchronization. Performance is sensitive to the latency of these (small) I/O s. Techniques for scaling metadata Scaling synchronization (distributing the lock manager) Segregating metadata from data to reduce queuing delays Separate disks Separate fabric ports Different RAID levels for metadata to reduce latency, or solid-state memory Adaptive metadata management (centralized vs. distributed) GPFS provides for all these to some degree; work always ongoing Sensible application design can make a big difference! IBM Corporation

9 Data loss in Petascale Systems Petaflop systems require tens to hundreds of petabytes of storage Evidence exists that manufacturer MTBF specs may be optimistic (Schroeder & Gibson) Evidence exists that failure statistics may not be as favorable as simple exponential distribution MTTDL in years for 20PB system Hard error rate of 1 in means one rebuild in 30 will get an error Rebuild of 8+P array of 500GB drives reads 4TB, or bits RAID-5 is dead at petascale; even RAID-6 may not be sufficient to prevent data loss Simulations of file system size, drive MTBF, failure probability distribution show 4%-28% chance of data loss over five-year lifetime for 8+2P code Stronger RAID (8+3P) increase MTTDL by 3-4 orders of magnitude for extra 10% overhead. Stronger RAID is sufficiently reliable even for unreliable (commodity) disk drives MTTDL in years P 600K hrs Exponential 8+3P 300K hrs Exponential 4% 8+2P 600K hrs Exponential Configuration 16% 8+2P 300K hrs Exponential 8+3P 600K hrs Weibull 28% 8+2P 600K hrs Weibull IBM Corporation

10 GPFS Software RAID Implement software RAID in the GPFS NSD server Motivations Better fault-tolerance Reduce the performance impact of rebuilds and slow disks Eliminate costly external RAID controllers and storage fabric Use the processing cycles now being wasted in the storage node Improve performance by file-system-aware caching Approach Storage node (NSD server) manages disks as JBOD Use stronger RAID codes as appropriate (e.g. triple parity for data and multi-way mirroring for metadata) Always check parity on read Increases reliability and prevents performance degradation from slow drives Checksum everything! Declustered RAID for better load balancing and non-disruptive rebuild IBM Corporation

11 Declustered RAID Partitioned RAID Declustered RAID 16 logical tracks 20 physical disks 20 physical disks IBM Corporation

12 Rebuild Work Distribution failed disk Relative read and write throughput for rebuild IBM Corporation

13 Rebuild (2) Upon the first failure, begin rebuilding the tracks that are affected by the failure (large arrows). Many disks involved in performing rebuild, so work is balanced, avoiding hot spots IBM Corporation

14 Declustered vs. Partitioned RAID Data losses per year per 100PB 1E+5 1E+4 1E+3 1E+2 1E+1 1E+0 1E-1 1E-2 partitioned distributed Simulation results 1E Failure tolerance IBM Corporation

15 Autonomic Storage Management Making Complex Tasks Simple IBM TotalStorage Productivity Center Standard Edition A Single Application with modular components Disk Data Fabric Business Resiliency Integrated Repication Manager Metro Disaster Recovery Global Disaster Recovery Cascaded Disaster Recovery Application Disaster Recovery Console Enhancements End-to-End DataPath Explorer Integrated Storage Planner Configuration Change Rover Configuration Checker Personalization TSM Integration Ease of Use Streamlined Installation and Packaging Single User Interface Single Database Single Set of services for consistent administration and operations Policy Based Storage Management SAN Best Practices SAN Configuration Validation Storage Subsystem Planning Fabric Security Planning Host Planning (Multi-path) IBM Corporation

16 Integrated Management Seamlessly integrate systems management across servers, storage and network & provide end-to-end problem determination and analytics capabilities Integrated Web 2.0 GUI Best Practices Deployment Systems Knowledge DB Orchestration Analytics Discovery Monitoring Reporting Configuration Applications Middleware OS File System Server Network Storage IBM Corporation Applications Middleware Operating systems Virtualization software Hardware

17 PERCS Management A unified and standards based management for GPFS and PERCS Storage A GUI that is designed for large-scale clusters Supporting PERCS scale GPFS The PERCS UI will support: Information collection: asset tracking, end-end discovery, metrics, events Management: system changes, problem fixes, configuration changes Rich visualizations to help them maintain situational awareness of system status Essential for large systems Also enable GPFS to satisfy commercial customers requiring easeof-use GPFS File System PERCS Storage Retrieves Data CIM Provider CIM Model Uses PERCS GUI CIM Client Management CIMOM Server Uses CIM Repository Simulator Systems DB IBM Corporation

18 Analytics Problem Determination and Impact Analysis Root cause analysis: discover the finest-grain events that indicate the root cause of the problem Symptom suppression: correlate alarms/symptoms caused by a common cause across the integrated infrastructure Bottleneck Analysis Post-mortem, live and predictive analysis Workload and Virtualization Management Automatically monitor multi-tiered, distributed, heterogeneous or homogeneous workloads Migrate virtual machines to satisfy performance goals Integrated Server, Storage and Network Allocation and Migration Integrated allocation accounting for connectivity, affinity, flows, ports based on performance workloads Disaster Management Provides integrated server/storage disaster recovery support IBM Corporation

19 Visualization Integrated Management is centered around Topology Viewer capabilities based on Web 2.0 technologies Data Path Viewer for Applications, Servers, Networks and Storage Progressive Information Disclosure Semantic Zooming Information Overlays Mixed Graphical and Tabular Views Integrated Historical and Real Time Reporting IBM Corporation

20 The Changing Nature of Archive Current Archive: Data landfill Store and forget Not easily accessible, typically offline and offsite with access time measured in days Not organized for usage, retained just in case needed Emerging Archive: Leverage information for business advantage Readily accessible, access time measured in seconds Indexed for effective discovery Mined for business value IBM Corporation

21 Building Storage Systems Targeted at Archive Scalability Scale to huge capacity Exploit tiered storage with disk and tape Leverage commodity disk storage Handle extremely large number of objects Support high ingest rates Effect data management actions in a scalable fashion Emerging Archive: Leverage info for business advantage Current Archive: Data landfill Functionality Consistently handle multiple kinds of objects Manage and retrieve based on data semantics E.g. Logical groupings of objects Support effective search and discovery Provide for compliance with regulations Reliability Ensure data integrity and protection Provide media management and rejuvenation Support long-term retention IBM Corporation

22 GPFS Information Lifecycle Management (ILM) GPFS ILM abstractions Storage pool group of LUNs Fileset subtree of a file system namespace Policy rule for file placement, retention, or movement among pools GPFS Clients Application Application Application GPFS Placement Application GPFS Policy Posix Placement GPFS Policy Placement GPFS Policy Placement Policy ILM Scenarios Tiered storage fast storage for frequently used files, slower for infrequently used files Project storage separate pools for each project, each with separate policies, quotas, etc. Differentiated storage e.g. place media files on media-friendly storage (QoS) GPFS RPC Protocol Gold Pool Storage Network Silver Pool Pewter Pool GPFS Manager Node Cluster manager Lock manager Quota manager Allocation manager Policy manager System Pool Data Pools GPFS File System (Volume Group) IBM Corporation

23 GPFS 3.1 ILM Policies Placement policies, evaluated at file creation, example Migration policies, evaluated periodically Deletion policies, evaluated periodically GPFS RPC Protocol GPFS Clients Application Application Application GPFS Placeme Application Posix GPFS nt Policy Placeme GPFS nt Policy Placeme GPFS nt Policy Placeme nt Policy GPFS Manager Nod Cluster manager Lock manager Quota manager Allocation manager Policy manager Storage Network Gold Pool Silver Pool Pewter Pool System Pool Data Pools GPFS File System (Volume Group) IBM Corporation

24 GPFS Policy Engine Migrate and delete rules scan the file system to identify candidate files Conventional backup and HSM systems also do this Usually implemented with readdir() and stat() This is slow random small record reads, distributed locking Can take hours or days for a large file system GPFS Policy Engine uses efficient sort-merge rather than slow readdir()/stat() Directory walk builds list of path names (readdir() but no stat()!) List sorted by inode number, merged with inode file, then evaluated Both list building and policy evaluation done in parallel on all nodes > 10 5 files/sec per node! IBM Corporation

25 Storage Hierarchies the old way Normally implemented one of two ways: Explicit control archive command (IBM TSM, Unitree) copy into special archive file system (IBM HPSS) copy to archive server (HPSS, Unitree) all of which are troublesome and error-prone for the user Implicit control through an interface like DMAPI File system sends events to HSM system (create/delete, low space) Archive system moves data and punches holes in files to manage space Access miss generates event; HSM system transparently brings file back IBM Corporation GPFS Session Node GPFS Disk Arrays GPFS I/O Nodes HPSS SAN Disk HPSS Data Store HPSS HSM processes HPSS Interface Client Domain HPSS 6.2 API Architecture DB2 Tape Libraries 2. Client transfers file to HPSS Disk or Tape Over TCP/IP LAN or WAN using an HPSS Mover HPSS FC SAN Moverless SAN Data transfers Client Cluster Computers Metadata Disks HSM Control Information IP LAN Data transfers HPSS API IP Network 1. Client issues HPSS Write or Put to HPSS Core Server HPSS Cluster Computers Core Server and Movers HPSS Disk Arrays DB2 HPSS Movers GPFS Cluster HPSS Cluster GPFS 3.1 and HPSS 6.2 DMAPI Architecture HPSS Core Server HPSS Tape Libraries Tape disk transfers

26 DMAPI Problems Namespace events (create, delete, rename) Synchronous and recoverable Each is multiple database transactions Slow down the file system Directory scans DMAPI low-space events trigger directory scans to determine what to archive can take hours or days on large FS Scans have little information upon which to make archiving decisions (what you get from ls l ) As a result, data movement policies are usually hard-coded and primitive Read/write managed region Blocks the user program while data brought back from HSM system Parallel data movement isn t in the spec, but everyone implements it anyway Data movement is actually the one thing about DMAPI worth saving GPFS Session Node GPFS Disk Arrays HPSS HSM processes GPFS I/O Nodes HPSS Interface DB2 Moverless SAN Data transfers HSM Control Information IP LAN Data transfers GPFS Cluster GPFS 3.1 and HPSS 6.2 DMAPI Architecture HPSS Disk Arrays HPSS Core DB2 Server HPSS Movers HPSS Cluster Tape disk transfers HPSS Tape Libraries IBM Corporation

27 GPFS Approach: External Pools External pools are really interfaces to external storage managers, e.g. HPSS or TSM External pool rule defines script to call to migrate/recall/etc. files RULE EXTERNAL POOL PoolName EXEC InterfaceScript [ OPTS options ] GPFS policy engine builds candidate lists and passes them to external pool scripts External storage manager actually moves the data Using DMAPI managed regions (read/write invisible, punch hole) Or using conventional Posix API s IBM Corporation

28 GPFS ILM Demonstration NERSC Oakland, CA HPSS Archive on tapes with disk buffering, connected via 10Gb link High bandwidth, parallel data movement across all devices and networks SC 06 Tampa, FL GPFS 1M active files FC, SATA disks IBM Corporation

29 Nearline Information conceptual view NFS/CIFS Client TSM Archive Client/API Admin / Search NFS/CIFS Server Scale-out Archiving Engine (GPFS Cluster) DMAPI TSM Archive API TSM Archive Client Global Index and Search Capability Migration via TSM Archive Client Provides capability to handle extended metadata Meta-data may be derived from data content Extended attributes (integrity code, retention period, retention hold status, and any application meta-data) Global index on content and EA meta-data Allow for application-specific parsers (e.g., DICOM) TSM Deep Storage IBM Corporation

30 Summary Storage environments moving from petabytes to exabytes Traditional HPC New archive environments Significant challenges for reliability, resiliency, and manageability Meta-data becomes key for information organization and discovery IBM Corporation