High Performance Computing Specialists. ZFS Storage as a Solution for Big Data and Flexibility

Transcription

1 High Performance Computing Specialists ZFS Storage as a Solution for Big Data and Flexibility

2 Introducing VA Technologies UK Based System Integrator Specialising in High Performance ZFS Storage Partner of E4 Computing Engineering Delivering ZFS Storage and HPC Solutions

3 VA HPC Powered by E4 New HPC Solutions ARKA ARM and Quadro + Tegra 3 Blades Joint New Solutions Lustre on ZFS & Hadoop on ZFS

4 But First. ZFS 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

5 Architecture 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

6 Architecture ZFS Pool Layout Clones Pool Configuration zvol Information zvol File System zvol File System Snapshots

7 Architecture ZFS Layer View raw swapdumpiscsi?? ZFS NFS CIFS?? ZFS Volume Emulator (Zvol) ZFS POSIX Layer (ZPL) e.g. pnfs?? Transactional Object Layer Pooled Storage Layer Block Device Driver HDD SSD iscsi FC??

8 Data Integrity 5 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

9 Data Integrity Designers Quote The job of any file system boils down to this: when asked to read a block, it should return the same data that was previously written to that block. If it can't do that -- because the disk is offline or the data has been damaged or tampered with -- it should detect this and return an error. Jeff Bonwick Father of ZFS Dec 2008

10 Data Integrity Merkle Trees & Checksums

11 Data Integrity Validating Checksum

12 Data Integrity Do you have a Write Hole?

13 Redundancy 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

14 Redundancy Mirrored Disks in 2 vdevs root vdev Logical vdevs top-level vdev children[0] mirror top-level vdev children[1] mirror vdev type = disk children[0] Vdev vdev type = disk type = disk children[0] children[0] Physical or leaf vdevs vdev type = disk children[0]

15 Redundancy RAIDz2 in 3vdevs vdev-0 RAID-Z2 HDD HDD HDD HDD HDD HDD zpool vdev-1 RAID-Z2 HDD HDD HDD HDD HDD HDD vdev-x RAID-Z2 HDD HDD HDD HDD HDD HDD Physical or leaf vdevs

16 Redundancy Dynamic Striping RAID-0 Column size = 128 kbytes, stripe width = 384 kbytes 384 kbytes ZFS Dynamic Stripe recordsize = 128 kbytes vdevs Total write size = 2816 kbytes

17 Transactional Copy on Write (COW) 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

18 Transactional Copy on Write (COW) Journaling FSCK

19 1. Initial block tree 2. COW some data 3. COW metadata 4. Update Uberblocks & free

20 Snapshots 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

21 Snapshots

22 Hybrid Storage Pools Mixing SSD with HDD 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

23 Hybrid Storage Pools Mixing SSD with HDD Old Old New ZFS

24 Hybrid Storage Pools Mixing SSD with HDD Adaptive Replacement Cache (ARC) Separate ZFS Intent Log (ZIL) Main Pool Level 2 ARC (L2ARC) Write optimized device (SSD) HDD HDD HDD Read optimized device (SSD)

25 Hybrid Storage Pools ARC Evict the oldest single-use entry Miss Hit LRU Recent Cache MRU MFU Frequent Cache LFU RAM Evict the oldest multiple accessed entry

26 Hybrid Storage Pools L2ARC Data soon to be evicted from the ARC sent to the Layer 2 ARC (L2ARC) which is usually a SSD vdev Works well when cache vdev is optimized for fast reads lower latency than pool disks inexpensive way to read performance SSD vdev can be striped for better performance Non Persistent Requires a rebuild after power off (soon to change!) ARC data soon to be evicted cache

27 Hybrid Storage Pools ZIL/SLOG Known as the Write Cache Non Volatile Stores small (<32k Configurable ) sync writes in high speed persistent storage or a separate log device (SLOG) Flushes to disk backend periodically sequential write stream as part of the TXG group Assigned on a per pool basis Perfect use case for RAM Based SAS Devices

28 ZFS Limitations Parallel Access? Distributed File System? pnfs

29 Relevancy to the HPC Community

30 Hadoop on ZFS Rebuild Times Utilize SSD s with HDD Better Administration Single Drive Replacement Integrated Management Warning and Reporting

31 Linux and Lustre on ZFS Native Kernel Module now available.

32 Linux and Lustre on ZFS 55PB Now Active to Sequoia Users Lustre + ZFS Fully Configured 768 OSS & OST s

33

34 Lustre on ZFS Write Performance Single shared file IOR (10G block, 1M transfers) Sequoia Workload 768 OSS Nodes 2048 Tasks per OSS 1,572,864 Compute Cores MB/s LDISKFS Increased tasks per OSS degrades performance ZFS - Constant performance Increase I/O size for RAIDZ2 LDISKFS+RAI D6 ZFS+RAID6 ZFS+RAIDZ2

35 Lustre on ZFS Read Performance LDISKFS mballocallowslargeri/o ZFS 128Kmaximumblocksize IOPs limited for ZFS+RAID6 Perfect Opportunity for Read Caching LDISKFS+RAI D6 ZFS+RAID6 ZFS+RAIDZ2

36 Lustre on ZFS Coming Soon New Hardware Optimised for Lustre on ZFS Low Power Consumption OSS & OST Customisable for your Lustre deployment Full Lustre and HW Support

37 Thanks Very Much! Ryan Tyler VA Technologies