A client side persistent block cache for the data center. Vault Boston Luis Pabón - Red Hat

Transcription

1 PBLCACHE A client side persistent block cache for the data center Vault Boston Luis Pabón - Red Hat

2 ABOUT ME LUIS PABÓN Principal Software Engineer, Red Hat Storage IRC, GitHub: lpabon

3 QUESTIONS: What are the benefits of client side persistent caching? How to effectively use the SSD? Compute Node Storage SSD

4 MERCURY* Use in memory data structures to handle cache misses as quickly as possible Write sequentially to the SSD Increase storage backend availability by reducing read requests Cache must be persistent since warming could be time consuming * S. Byan, et al., Mercury: Host-side flash caching for the data center

5 MERCURY QEMU INTEGRATION

6 PBLCACHE

7 PBLCACHE Persistent BLock Cache Persistent, block based, look aside cache for QEMU User space library/application Based on ideas described in the Mercury paper Requires exclusive access to mutable objects

8 GOAL: QEMU SHARED CACHE

9 PBLCACHE ARCHITECTURE PBL Application Cache Map Log SSD

10 PBL APPLICATION Sets up the cache map and log Decides how to use the cache (writethrough, read-miss) Inserts, retrieves, or invalidates blocks from the cache Pbl App Msg Queue Cache map Log

11 CACHE MAP Composed of two data structures Maintains all block metadata Address Map Block Descriptor Array

12 ADDRESS MAP Implemented using as a hash table Translates object blocks to Block Descriptor Array (BDA) indeces Cache misses are determined extremely fast Address Map Block Descriptor Array

13 BLOCK DESCRIPTOR ARRAY Contains metadata for blocks stored in the log Length is equal to the maximum number of blocks stored in the log Handles CLOCK evictions Invalidations are extremely fast Address Map Block Descriptor Array Insertions always append

14 CACHE MAP I/O FLOW Block Descriptor Array

15 CACHE MAP I/O FLOW G et In address map N o Y es M iss H it S et CLOCK bit in BDA Read from log

16 CACHE MAP I/O FLOW Invalidate Free BDA index Delete from map

17 LOG Block location determined by BDA CLOCK optimized with segment read-ahead Segment pool with buffered writes Contiguous block support Segments SSD

18 LOG SEGMENT STATE MACHINE

19 LOG READ I/O FLOW Read In a segment? Yes No Read from segment Read from SSD

20 PERSISTENT METADATA Save address map to a file on application shutdown Cache warm on application restart Not designed to be durable System crash will cause metadata file not to be created

21 PBLIO BENCHMARK PBL APPLICATION

22 PBLIO Benchmark tool Uses an enterprise workload workload generator from NetApp* Cache setup as write through Can be used with or without pblcache Documentation * S. Daniel et al., A portable, open-source implementation of the SPC-1 workload *

23 ENTERPRISE WORKLOAD Synthetic OLTP enterprise workload generator Tests for maximum number of IOPS before exceeding 30ms latency Divides storage system into three logical storage units: ASU1 - Data Store - 45% of total storage - RW ASU2 - User Store - 45% of total storage - RW ASU3 - Log - 10% of total storage - Write Only BSU - Business Scaling Units 1 BSU = 50 IOPS

24 SIMPLE EXAMPLE $ fallocate -l 45MiB file1 $ fallocate -l 45MiB file2 $ fallocate -l 10MiB file3 $ $./pblio -asu1=file1 \ -asu2=file2 \ -asu3=file3 \ -runlen=30 -bsu= pblio Cache : None ASU1 : 0.04 GB ASU2 : 0.04 GB ASU3 : 0.01 GB BSUs : 2 Contexts: 1 Run time: 30 s Avg IOPS:98.63 Avg Latency: ms

25 RAW DEVICES EXAMPLE $./pblio -asu1=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde \ -asu2=/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi \ -asu3=/dev/sdj,/dev/sdk,/dev/sdl,/dev/sdm \ -runlen=30 -bsu=2

26 CACHE EXAMPLE $ fallocate -l 10MiB mycache $./pblio -asu1=file1 -asu2=file2 -asu3=file3 \ -runlen=30 -bsu=2 -cache=mycache pblio Cache : mycache (New) C Size : 0.01 GB ASU1 : 0.04 GB ASU2 : 0.04 GB ASU3 : 0.01 GB BSUs : 2 Contexts: 1 Run time: 30 s Avg IOPS:98.63 Avg Latency: ms Read Hit Rate: Invalidate Hit Rate: Read hits: 1120 Invalidate hits: 347 Reads: 2513 Insertions: 1906 Evictions: 0 Invalidations: 513 == Log Information == Ram Hit Rate: Ram Hits: 1120 Buffer Hit Rate: Buffer Hits: 0 Storage Hits: 0 Wraps: 1 Segments Skipped: 0 Mean Read Latency: 0.00 usec Mean Segment Read Latency: usec Mean Write Latency: usec

27 ----- pblio Cache C Size ASU1 ASU2 ASU3 BSUs : 32 Contexts: 1 Run time: 600 s LATENCY OVER 30MS : /dev/sdg (Loaded) : GB : GB : GB : GB Avg IOPS: Avg Latency: ms Read Hit Rate: Invalidate Hit Rate: Read hits: Invalidate hits: Reads: Insertions: Evictions: Invalidations: == Log Information == Ram Hit Rate: Ram Hits: 75 Buffer Hit Rate: Buffer Hits: 0 Storage Hits: Wraps: 0 Segments Skipped: 0 Mean Read Latency: usec Mean Segment Read Latency: usec Mean Write Latency: usec

28 EVALUATION

29 TEST SETUP Client using 180GB SAS SSD (about 10% of workload size) GlusterFS 6x2 Cluster 100 files for each ASU pblio v0.1 compiled with go1.4.1 Each system has: Fedora 20 6 Intel Xeon 2GHz 64 GB RAM 5 300GB SAS Drives 10Gbit Network

30 CACHE WARMUP IS TIME COMSUMING 16 hours

31 INCREASED RESPONSE TIME 73% Increase

32 STORAGE BACKEND IOPS REDUCTION BSU = 31 or 1550 IOPS ~75% IOPS Reduction

33 CURRENT STATUS

34 MILESTONES 1. Create Cache Map - COMPLETED 2. Create Log - COMPLETED 3. Create Benchmark application - COMPLETED 4. Design pblcached architecture - IN PROGRESS

35 NEXT: QEMU SHARED CACHE Work with the community to bring this technology to QEMU Possible architecture: Some conditions to think about: VM migration Volume deletion VM crash

36 FUTURE Hyperconvergence Peer-cache Writeback Shared cache QoS using mclock* Possible integrations with Ceph and GlusterFS backends * A. Gulati et al., mclock: Handling Throughput Variability for Hypervisor IO Scheduling

37 JOIN! Github: IRC Freenode: #pblcache Google Group: Mail list:

38 FROM THIS...

39 TO THIS

40