Panasas at the RCF. Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory. Robert Petkus Panasas at the RCF

Transcription

1 Panasas at the RCF HEPiX at SLAC Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory

2 Centralized File Service Single, facility-wide namespace for files. Uniform, facility-wide POSIX-like access to data. Transparent to end-user.

3 Current Storage Implementation Traditional SAN ~220 TB fibre-channel disks in RAID 5 arrays 13 TB IDE storage 24 Brocade fibre-channel switches 37 Sun Solaris 9 servers (E450, V480, V240) using Veritas 4.0 (VxVM, VxFS) NFS transfer rate: 70MB/sec/server I/O throughput to disks: MB/sec writes, 75 MB/sec reads

4 Current Storage Implementation Panasas 100 TB on 20 shelves. 10 Realms. 2 trunked 1000BaseT network ports (out of 4) enabled. Data transfer rate from server ~250 MB/sec max per shelf. DirectFLOW client deployed on ~2000 node RHIC analysis/reconstruction farm.

5 Current view of the facility

6 NFS Load balancing Scaling Issues NFS Concerns Horizontal (management) Vertical (performance) Security concerns

7 VXFS and VXVM Pros Veritas Issues Ability to shrink file systems. Easy to import/deport volumes among servers. Dynamic multipathing Cons Quotas broken on file systems > 1 TB Expensive Poor customer support

8 Panasas Features Fast, scalable, reliable and fault tolerant Direct and parallel data access Load balancing (Efficient resource utilization) Incremental growth Integrated hardware / software solution Economic benefits of IDE disk Security and centralized management Global namespace Distributed metadata Active development Excellent customer support

9 Spectrum of Solutions GPFS: data block striping across disks. Parallel reads/writes. Fail-over, replication, distributed metadata. Xrootd: Load balancing, single namespace, proxy servers. Client access through TXNetFile (ROOT framework) BlueArc: ASICs dedicated to NFS, network, and filesystem. Feature-rich, fast and robust NFS solution. Ibrix: Meta servers assigned to segments in a disk pool. Traditional NFS or Fusion. Linux LVS model. Dcache: Management of heterogeneous storage repositories. Hot spot detection. Lustre: Object-based storage solution. Software only. Active development. Older versions are free.

10 Various Models POSIX-like vs Non POSIX-like Hardware vs Software Existing Protocol vs New Protocol Ethernet vs Fibre Channel Unlimited Scalability vs Just Fast Enough Dedicated vs Non-Dedicated Resources Proprietary vs Open

11 Panasas Architecture Components of each shelf at the RCF 1 Director Blade 10 Storage Blades 1 Gigabit internal ethernet ports per blade (11 total) 4 Gigabit links to external network 5 or 8 TB raw disk/shelf

12 Panasas Architecture Director Blade Director Blades (The brains) File Namespace server(s) Manages Metadata object map Coordinates between clients and Storage Blades Determines RAID characteristics of a file. Determines distribution of file objects over OSD's Specifications 2.4 GHz Intel Xeon 4 GB RAM BaseT ports FreeBSD

13 Panasas Architecture Storage Blade Storage Blades (Object Storage Devices or OSD's) Store and retrieve data objects Handles I/O to client Specifications Each Storage Blade (10) 500GB or 800GB storage (2 SATA 250GB or 400GB HDD) 1.2 GHz Intel Celeron 512 MB RAM 100/1000BaseT FreeBSD

14 Panasas Architecture ActiveScale Operating System Runs on Director Blade Divides files into data objects, which are arbitrary in size, and stripes them across storage blades Dynamically distributes workload across storage blades Each storage blades is only filled to 90% capacity. The remaining 10% is reserved for rebuilding parity. Direct Flow Software Installed on the Linux compute node Direct data path from client to storage blades Optimizes data layout, caching and prefetching File is reconstructed at the compute node

15 Panasas performance snapshots

16 Panasas performance snapshots

17 Issues No LDAP support No user or group quotas volume quotas only A new command pan_du is in development to list the amount of storage used per user but no enforcement. No on-line director blade failover Can't downgrade ActiveScale DirectFLOW memory leaks fixed? DHCP server stubborn to release addresses need an excess pool of IP addresses in case a blade crashes.

18 Issues Maxed-out volumes are unstable. Hard volume quotas need to be set at 95% of total usable capacity for protection. Atime records written to disk on writes not reads (JLAB) Hard limit on files per directory (350k) Mass-deletes are not handled gracefully.

19 Conclusions Panasas scalability and performance much better than NFS. Easy to administrate and reliable. Responsive, knowledgeable support staff. Intermittent client-side DirectFLOW crashes are frustrating on a very large cluster. Possibility of separating software from hardware? Emergence of strong competitors. NFSv4.1? What role will centralized storage play in a grid environment?