Isilon: Scalable solutions using clustered storage TERENA Storage WG Conference September, 2008 Rob Anderson Systems Engineering Manager, UK & Ireland rob@isilon.com
Isilon at HEAnet HEAnet were looking for a large, scalable storage system for Video (such as parliamentary recordings) Multimedia delivery Data backup Hosting Virtualisation Wanted to be able to allot storage out to their client organisations They selected clustered storage as the solution Why clustered storage?
Why clustered storage? Organisatons face rapid data growth Organisations are adopting clustered computing Traditional SAN/NAS solutions are often struggling
Organisations face rapid data growth (PB) By 2011, 75% of all storage capacity sold will be for file-based data File Based: 79.3% CAGR Block Based: 31% CAGR * Source: IDC, 2007
Organisations are (finally) adopting clustering Evolution of Servers Benefits of Clustering Vastly superior performance Vastly superior scalability modular and incremental Substantially lower cost to own and operate Accelerates business workflow and productivity
Traditional solutions are often struggling Information Availability Network Attached Storage (NAS) Limited file system size Inherent performance limits and bottlenecks Single points of failure File-based Many file systems to manage Clustered Storage 100x larger file systems 20x higher total performance Unmatched reliability and self healing File/object-based Easy to manage 1000+ TBs of storage Storage Area Networks (SAN) Limited file system size Designed for I/Os not throughput High cost and complexity Block-level storage (not file) Difficult to manage and maintain Performance
Isilon Elements Clustered of clustered Storage storage System Unlimited Scalability High Performance Maximum Data Protection Ease of Use/Management High Availability Enterprise Services (Windows, Unix, Security, authentication, etc.)
Isilon clustered storage system
Scaling clustered storage OneFS intelligent software Enterpriseclass hardware Isilon IQ Complete storage system A 3-node Isilon IQ Cluster Expandable to more than 96 nodes and 2.3 PB in a single file system 9
Product suite Performance Isilon IQ 1920, 3000, 6000, 9000 or 12000 + IQ Accelerator Independent Performance Scalability Isilon IQ 1920, 3000, 6000, 9000 and 12000 Platform Nodes Isilon IQ 200 Platform Node Linear Performance and Capacity Scaling Isilon IQ 6000/9000/12000 + EX 6000/9000/12000 Independent Capacity Scalability Capacity 10
Cluster file striping Cornerstone for a truly symmetric clustered architecture High performance Linear scalability Unmatched reliability Ease of Use 1 1
Isilon IQ network architecture Servers NFS, CIFS, FTP, HTTP Servers (optional 2nd switch) (optional 2nd switch) Servers Client/Application Layer Standard GbE or 10 GbE Ethernet Layer Isilon IQ Storage Layer Infiniband Intracluster Communication 1 2
Writing a file with Isilon IQ Servers NFS, CIFS, FTP, HTTP Servers (optional 2nd switch) (optional 2nd nd (optional 2switch) switch) Servers 1 3
Writing a file with Isilon IQ (optional 2nd switch) 1 4
Reading with Isilon Writing aa filefile with Isilon IQIQ Servers NFS, CIFS, FTP, HTTP Servers (optional 2nd switch) (optional 2nd nd (optional 2switch) switch) Servers 1 5
Reading a file with Isilon IQ Servers NFS, CIFS, FTP, HTTP Servers (optional 2nd switch) (optional 2nd switch) Servers 1 6
Fortunately scaling is easy OneFS can scale from 4 TB to 2.3 PB in a single file system Traditional SAN/NAS limited to 2-16TB max file system Under 60 seconds to scale, with no user disruption 17
Data in a liquid pool AutoBalance: Automated data balancing across nodes Reduces costs, complexity and risks for scaling storage BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY AutoBalance migrates content to new storage nodes while system is online and in production Requires no manual intervention, no reconfiguration, no server or client mount point or application changes 18
If it s big, it must be resilient Built-in high availability clustered architecture EMC and NetApp require costly, redundant heads and software With N+2, N+3, and N+4 protection, data is 100% available if multiple With N+1 protection, drives or nodes fail data is 100% available even if a single drive or node And fails Isilon IQ offers the industry s fastest 100% AVAILABILITY 100% AVAILABILITY FAILED 100% AVAILABILITY 100% AVAILABILITY drive 100% AVAILABILITY rebuild times in less than an hour 100% AVAILABILITY (optional 2nd switch) 100% AVAILABILITY FAILED 100% AVAILABILITY 19
Investment protection Unlike with SANs, you can keep your existing investment Virtualise different ages of hardware in same cluster (With some rules) different sizes of node in same cluster too 20
Suite of software applications Data Protection Simple, scalable and flexible data protection with SnapshotIQ Data Management Policy-based load balancing with failover with SmartConnect Quota management and thin provisioning with SmartQuotas Data Migration Automated content migration across storage tiers with MigrationIQ Data Replication Fast and flexible file-base asynchronous replication with SyncIQ High-Performance Content Delivery High performance wide area file and content delivery with Aspera for Isilon IQ 21
Local Data Protection A snapshot is a locally simple, retained read-only scalable point-in-time & flexible image of data SnapshotIQ is a software application that leverages OneFS and Isilon s clustered architecture Facilitates frequent, near instantaneous, lowimpact, user-recoverable backup of data First line of defense against local data corruption, accidental deletion (i.e. fat finger deletion ), or modifications Snapshots vastly improve the frequency and granularity of backups/restore 22
Load Balancing with Failover SmartConnect Advanced software application provides seamless enterprise storage access to an Isilon IQ cluster Client Connection: Load balancing based on intelligent policies Redundant: Dynamic failover of NFS client connections between nodes Performance: Zoning for priority user, group & application optimization Clients SmartConnect 23
Quota Management and Thin Provisioning Simple, scalable & flexible management for clustered storage simple, scalable & flexible Data management and provisioning feature of Isilon s OneFS operating system software Partition and control your cluster (single pool) usage exactly the way you want, on the fly Industry-leading flexibility Cluster, directory, subdirectory, user and group Hard, soft and advisory thresholds Robust storage provisioning Thin provisioning and oversubscription Manage user exposure to capacity 24
Asynchronous File-based Replication SyncIQ is a software application that delivers file-based, asynchronous replication for DR, D2D backup and distributed workflow Enterprise-wide: One-to-many replication locations throughout enterprise Flexible: Choose what/when content to replicate based on policy engine Intelligent: Cluster-aware design delivers unmatched performance LAN/ WAN connection Additional cluster Additional cluster Primary source cluster Additional cluster 25
High-Performance Wide Area File and Content Delivery Fasp Clients IP Network Aspera Enterprise Server for Isilon IQ Speeds file transfers across any wide area network slow or fast Scales beyond the limits of a single-head storage system Supports interoperability between Windows, Mac, Linux, Isilon Ensures predictable speeds and transfer times Reduced infrastructure costs as software lives on Isilon clustered storage
OneFS 5.0 features Symmetric multi-processor (SMP) support Allows OneFS to take advantage of multi-core and multiprocessor Isilon IQ nodes. Support for 64-bit memory addressing Allowing OneFS to scale far beyond 4 GB of RAM in Isilon IQ nodes. Support for the Accelerator-x platform New Accelerator node with up to 10 Gigabit Ethernet, 32 GB of memory, and 8 processing cores. Native Windows opportunistic locks Allows CIFS clients to take advantage of client-side caching.
OneFS 5.0 features FlexNet version 2.0 Includes support for multiple subnets, VLANs, tighter integration with SmartConnect Advanced, and a completely redesigned WebAdmin and command-line interface. NIC Bonding for performance Improves concurrent performance Flexible Allocation A new OneFS per-file attribute which allows file layout to be optimized for streaming or random access.
OneFS 5.0 features Drive Timeout & Threshold Improvements Unique capability which allows OneFS to proactively quarantine drives which are failing to respond and analyze whether they should be SmartFailed or returned to service. Isilon Firmware Tool Separate software upgrade from firmware upgrade. EUC-KR and CP949 Korean encodings. Thousands of Quality Improvements
X-Series storage node performance Performance Metric Read (MB/s) Write (MB/s) Linux NFS Windows NFS Windows CIFS Mac NFS Concurrent 1 NIC 112 100 96 96 Sequential Bonded 206 195 228 231 (per node) NICs Single File 1 thread 91 82 54 84 72 79 File Stream 1 thread 88 60 / 66* 48 63 4 threads 96 94 / 114* 72 86 Concurrent 1 NIC 112 98 117 Sequential Bonded 201 150 178 (per node) NICs Single File 1 thread 120 97 107 77 117 File Stream 1 thread 91 92 / 86* 64 72 4 threads 112 112 / 196* 113 110 FTP HTTP * Run with bonded NICs
Accelerator X-series performance Performance Metric Read (MB/s) Write (MB/s) Linux NFS Windows NFS Windows CIFS Mac NFS Concurrent Sequential (per node) 518 564 854 849 Single File 1 thread 341 228 112 269 230 233 File Stream 1 thread 180 / 220* 106 / 152* 104 / 131* 132 4 threads 380 297 161 191 8 threads 420 378 170 Concurrent Sequential (per node) 730 581 442 Single File 1 thread 400 332 246 303 251 File Stream 1 thread 273 248 129 213 4 threads 504 419 286 244 8 threads 540 512 FTP HTTP * Run with filename prefetch turned on
OneFS 5.0 Performance (by Cluster Size) OneFS 4.7 Performance Metric (MBps) Per Node 3 Node Cluster 6 Node Cluster 10 Node Cluster 42 Node Cluster 96 Node Cluster Sequential Concurrent Read NFS 110 330 660 1,100 4,620 10,560 Sequential Concurrent Read CIFS 110 330 660 1,100 4,620 10,560 Sequential Concurrent Write NFS 88 264 528 880 3,696 8,448 Sequential Concurrent Write CIFS 87 261 522 870 3,654 8,352 OneFS 5.0* Performance Metric (MBps) Per Node 3 Node Cluster 6 Node Cluster 10 Node Cluster 42 Node Cluster 96 Node Cluster Sequential Concurrent Read NFS 206 618 1,236 2,060 8,652 19,776 Sequential Concurrent Read CIFS 195 585 1,170 1,950 8,190 18,720 Sequential Concurrent Write NFS 201 603 1,206 2,010 8,442 19,296 Sequential Concurrent Write CIFS 150 450 900 1,500 6,300 14,400 * Run with bonded NICs
OneFS 5.0 WebAdmin
Example scenario: share some storage Click on FileSystem -> Windows File Sharing Click Create a New Share Fill in details, click Next Click Finish Done! Your users can now start using the share. 3
Example scenario: (thin) provision space click Modules -> SmartQuotas -> Create Quota select Apply quota to Directory, and choose directory Specify enforcement criteria click on Save Done! You ve provisioned space for a user, group or set of users. 3
Enterprise clustered storage customers Central repository for software development Scene7 delivers cloud-based software as a service (SaaS) 400TB+ Isilon IQ Clustered Storage deployed Reduced management complexity vs. SAN/NAS storage Orange is the key brand of France Telecom, one of the world's leading telecommunications operators Pioneering multimedia solutions offering voice, internet, mobile & video convergence 70TB s of Isilon Clustered Storage is primary storage repository for Orange s video-on-demand global architecture
Enterprise clustered storage customers One media repository for entire editorial staff Real-time playback & editing from Isilon Clustered Storage Rapid, cost-effective scalable storage w/no configuration With Isilon, we ve done about five years worth of work in two. Ken Brady, Director of Technical Operations and Digital Media, Turner Broadcasting
Questions? Rob Anderson Systems Engineering Manager, Isilon UK & Ireland rob@isilon.com