Large Scale Storage. Orlando Richards, Information Services LCFG Users Day, University of Edinburgh 18 th January 2013

Transcription

1 Large Scale Storage Orlando Richards, Information Services LCFG Users Day, University of Edinburgh 18 th January 2013

2 Overview My history of storage services What is (and is not) a SAN? Parallel File Systems Our systems Any questions just stick your hand up any time

3 2001 Linux or Solaris Workstation with mixture of external SCSI drives Ext2, shared by NFS

4 2003 Linux workstation with internal drives, old ones still clinging on Ext3 filesystem, shared by NFS

5 2005 Moving to rack mounted servers with internal disks Ad-hoc in shared racks with network kit Ext3 filesystem, shared by NFS

6 2007 ~1000 drives Dual raid controllers Multipathing Parallel file systems Fibre Channel Storage Area Network Fibre Channel backplanes Cache mirroring Controller failover iscsi Shared by NFS/GPFS/CIFS

7 Storage Area Network (SAN)

8 What is a SAN? SAN = Storage Area Network Moves data packets between system A and system B A SAN is not a file system A SAN is not disks What is a SAN not? A SAN is not SCSI, or SAS, or SATA

9 Transportation TCP is a transport for application data packets IP is a transport for TCP packets Ethernet is a transport for IP packets SCSI is a transport for data packets Fibre Channel Protocol (FCP) is a transport for SCSI packets Fibre Channel (FC) is a transport for FCP packets

10 A local file system Application does a write to the file system via kernel interface, into memory Separate process orders memory, and at some point sends write commands to the disk Reads come from memory, populated with reads from disk if needed

11 A SAN file system Application does a write to the file system via kernel interface, into memory Separate process orders memory, and at some point sends write commands to the HBA Reads come from memory, populated with reads from the (SAN connected) disk if needed Added complexity is hidden from the Application and the File System

12 Cache A memory cache is typically used where ever possible to accelerate reads and optimise writes Memory access time is ~100ns Disk seek time is ~10,000,000 ns (10ms) Writes from an application can come in all shapes and sizes buffering them in memory allows transactions to be optimised before sending to disk (e.g. bundle a number of small writes into a single disk operation)

13 Shared disk, shared file system? With a SAN, you can connect the same disk to multiple hosts So can you build a file system that they can both use? Nope! * * Actually, yes but it s not that simple

14 So what good is having a SAN then? If you can t use it to share data, what good is it? Performance: a modern SAS connection can move 6Gbit/sec, typically to ~1 -> ~10 drives a modern FC SAN connection can move 16Gbit/sec, to 100 s-1000 s of drives Scalability: Easy to add more storage just plug it into the network Manageability: Able to assign storage to multiple hosts in exactly the desired quantity, and move it when required. Smarty-pants: once you get all the storage lumped together, you can pool it all (think RAID0) (= performance), thin provision it (= efficient use), replicate it, snapshot it, tier it (= fancy stuff)

15 Parallel File Systems

16 Shared disk, shared file system? With a SAN, you can connect the same disk to multiple hosts If the hosts know how, they can share information in memory, as well as on disk. Enter. parallel file systems

17 Parallel File Systems Typical SAN shared storage environment All nodes can see all disks Locking information shared across nodes Tokens for control of cache coherency Manager nodes keep everyone talking

18 Distributed File Systems Data index is held on metadata server Data on the storage servers Clients transfer data directly to the storage servers at the instruction of the metadata server

19 Why? Parallel/distributed file systems offer: Performance typically scale out performance Need more performance? Then add more things! Resilience active/active/active/active Single name space all your eggs, in one basket But, they have downsides: Performance all that locking/caching info needs to be passed between systems Cost & complexity particularly noticeable on a small scale All your eggs, in one basket

20 Summary A SAN is a network It lets you use lots of disks easily A parallel file system lets you use lots of disks well

21 Eddie 163TB usable High IOPS tier for HPC SSD for metadata 8x storage servers at 10GE 10GE network backbone

22 ECDF NAS 389TB usable 3x IBM DS3512 SAN arrays 1x IBM DS3524 SAN array (metadata) 2 frontends, 3 backends 2x1GE connectivity Off-site backups

23 SMB,CIFS,WTF,NFS,CTDB,GPFS,SFTP,SSHFS Presentation to end users typically from Samba server, but also NFS, SSHFS Plays nicely with GPFS ACLs, snapshots, locking Clustering with CTDB Shared locking info Shared user/group mapping info Very fast failover ( < 0.5s mostly)

24 The Big One Planning for Q PB free-at-the-pointof-use service Ecdfnas++ Higher performance, higher capacity, lower cost Enhanced DR

25 ECDF Marketing Slide ECDF / IS Services (available to everyone in the University, and beyond) include: High Performance Computing cluster Eddie Pre-paid for CSCE Pay-as-you-go for CMVM/CHSS (pre-paid by arrangement) Data storage service ecdfnas Cloud storage at 205/TB/year (this year) available via standard NAS interfaces (CIFS/NFS/SFTP/SSHFS) Version control service ECDF Subversion No charge for most use cases