Flexible Storage Allocation

Transcription

1 Flexible Storage Allocation A. L. Narasimha Reddy Department of Electrical and Computer Engineering Texas A & M University Students: Sukwoo Kang (now at IBM Almaden) John Garrison

2 Outline Big Picture Part I: Flexible Storage Allocation Introduction and Motivation Design of Virtual Allocation Evaluation Part II: Data Distribution in Networked Storage Systems Introduction and Motivation Design of User-Optimal Data Migration Evaluation Part III: Storage Management across diverse devices Conclusion 2 Texas A&M University Narasimha Reddy

3 Storage Allocation Allocate entire storage space at the time of the file system creation Storage space owned by one operating system cannot be used by another Windows NT (NTFS) Linux (ext2) AIX (JFS) Actual Allocations 30 GB 70 GB 50 GB 50 GB 98 GB Running out of space! 3 Texas A&M University Narasimha Reddy

4 Big Picture Memory systems employ virtual memory for several reasons Current storage systems lack such flexibility Current file systems allocate storage statically at the time of their creation Storage allocation: Space on the disk is not allocated well across multiple file systems 4 Texas A&M University Narasimha Reddy

5 File Systems with Virtual Allocation When a file system is created with X GB, Allows the file system to be created with only Y GB, where Y << X Remaining space used as one common available pool As the file system grows, the storage space can be allocated on demand Windows NT (NTFS) Linux (ext2) AIX (JFS) Actual Allocations 30 GB 10 GB 50 GB 10 GB 98 GB GB Common Storage 40 GB Pool 5 Texas A&M University Narasimha Reddy

6 Our Approach to Design Physical block address Physical Disk Employ Allocate-on-write policy Storage space is allocated when the data is written Writes all data to disk sequentially based on the time at which data is written to the device Once data is written, data can be accessed from the same location, i.e., data is updated in-place 6 Texas A&M University Narasimha Reddy

7 Allocate-on-write Policy Extent Write at t = t Physical Disk Storage space is allocated by the unit of the extent when the data is written Extent is a group of file system blocks Fixed size Retain more spatial locality Reduce information that must be maintained 7 Texas A&M University Narasimha Reddy

8 Allocate-on-write Policy Extent 0 Write at t = t (where t > t ) Extent 1 Write at t = t Physical Disk Data is written to disk sequentially based on write-time Further writes to the same data updated in-place VA (Virtual Allocation) requires additional data structure 8 Texas A&M University Narasimha Reddy

9 Block Map Extent 0 Write at t = t (where t > t ) Extent 1 Extent 2 Write at t = t Physical Disk Block map Block map keeps a mapping of logical storage locations and real (physical) storage locations 9 Texas A&M University Narasimha Reddy

10 VA Metadata VA Meta data Extent 0 Extent 1 Extent 2 Physical Disk Hardening Block map This block map is maintained in memory and regularly written to disk for hardening against system failures VA Metadata represents the on-disk block map 10 Texas A&M University Narasimha Reddy

11 On-disk Layout & Storage Expansion Storage Expansion Threshold VA Meta data FS Meta data Extent 0 Extent 1 Extent 2 Extent 3 Extent 4 Physical Disk Storage Expansion Extent 5 Extent 6 Extent 7 Virtual Disk When the capacity is exhausted or reaches storage expansion threshold, a physical disk can be expanded to other available storage resources File system unaware of the actual space allocation and expansion 11 Texas A&M University Narasimha Reddy

12 Write Operation Application Write Request File System Acknowledgement Buffer/Page Cache Layer Page Search VA block map Block I/O Layer (VA) Hardening Allocate new extent and update mapping information Disk VA Meta data FS Meta data Extent 0 Extent 1 Extent 2 Extent 3 12 Texas A&M University Narasimha Reddy

13 Read Operation Application Read Request File System Buffer/Page Cache Layer Search VA block map Block I/O Layer (VA) Disk VA Meta data FS Meta data Extent 0 Extent 1 Extent 2 Extent 3 13 Texas A&M University Narasimha Reddy

14 Allocate-on-write vs. Other Work Key difference from log-structured file systems (LFS) Only allocation is done at the end of log Updates are done in-place after allocation LVM still ties up storage at the time of file system creation 14 Texas A&M University Narasimha Reddy

15 Design Issues VA Metadata Hardening (File System Integrity) Must keep certain update ordering of VA metadata and FS (meta)data Extent-based Policy Example (with Ext2) I (inode), B (data block), V (VA block map) A B (B is allocated to A) File system-based Policy Example (with Ext3 ordered mode) 15 Texas A&M University Narasimha Reddy

16 Design Issues (cont.) Extent Size Larger extent size: Reduce block map size, retain more spatial locality, cause data fragmentation Reclaiming allocated storage space of deleted files Needed to continue to provide the benefits of virtual allocation Without reclamation, possible to turn virtual allocation into static allocation Interaction with RAID RAID remaps blocks to physical devices to provide device characteristics VA remaps blocks for flexibility Need to resolve performance impact of VA s extent size and RAID s chunk size 16 Texas A&M University Narasimha Reddy

17 Spatial Locality Observations & Issues Metadata and data separation Data clustering: Reduce seek distance Multiple file systems Data placement policy Allocate hot data in a high data region of disk Allocate hot data in the middle of the partition 17 Texas A&M University Narasimha Reddy

18 Implementation & Experimental Setup Virtual allocation prototype Kernel module for Linux Employ a hash table in memory for speeding up VA lookups Setup A 3GHz Pentium 4 processor, 1GB main memory Red Hat Linux 9 with a kernel Ext2 file system and Ext3 file system Workloads Bonnie++ (Large-file workload) Postmark (Small-file workload) TPC-C (Database workload) 18 Texas A&M University Narasimha Reddy

19 VA Metadata Hardening Compare EXT2 and VA-EXT2-EX Compare EXT3 and VA-EXT3-EX, VA-EXT3-FS 19 Texas A&M University Narasimha Reddy

20 Reclaiming Allocated Storage Space Reclaim operation for deleted large files How to keep track of deleted files? Employed stackable file system: Maintain duplicated block bitmap Alternatively, could employ Life or Death at Block-Level (OSDI 04) work 20 Texas A&M University Narasimha Reddy

21 VA with RAID-5 Large-file workload Small-file workload Large-file workload with NVRAM Used Ext2 with software RAID-5 + VA NVRAM-X%: X% of total VA metadata size VA-RAID-5 NO-HARDEN VA-RAID-5 NVRAM-17% VA-RAID-5 NVRAM-4% VA-RAID-5 NVRAM-1% 21 Texas A&M University Narasimha Reddy

22 Data Placement Policy (Postmark) VA NORMAL partition: Same data rate across a partition VA ZCAV partition: Hot data is placed in high data region of a partition VA-NORMAL: start allocation from the outer cylinders VA-MIDDLE: start allocation from the middle of a partition 22 Texas A&M University Narasimha Reddy 16 24

23 Multiple File Systems VA-7GB: 2 x 3.5GB partition, 30% utilization VA-32GB: 2 x 16GB partition, 80% utilization Used Postmark VA-HALF: The 2 nd file system is created after 40% of the 1 st file system is written VA-FULL: 80% 23 Texas A&M University Narasimha Reddy

24 Real-World Deployment of Virtual Allocation Prototype built 24 Texas A&M University Narasimha Reddy

25 VA in Networked Storage Environment Flexible allocation provided by VA leads to Balancing locality vs. load balance issues 25 Texas A&M University Narasimha Reddy

26 Part II: Data Distribution Cold data Locality-based approach Hot data Use data migration (e.g. HP AutoRAID) Employ hot data migration from slower device (remote disk) to faster device (local disk) Load balancing-based approach (Striping) Exploit multiple devices to support the required data rates (e.g. Slice- OSDI 00) 26 Texas A&M University Narasimha Reddy

27 User-Optimal Data Migration data Locality is exploited first Data is migrated from Disk B to Disk A Load balancing is also considered If the load on Disk A is too high, data is migrated from Disk A to Disk B 27 Texas A&M University Narasimha Reddy

28 Migration Decision Issues write write read data Where to migrate: Use I/O request response time When to migrate: Migration threshold Initiate migration from Disk A to Disk B only when How to migrate: Limit number of concurrent migrations (Migration token) What data to migrate: Active data 28 Texas A&M University Narasimha Reddy

29 Design Issues Allocation policy Striping with user-optimal migration: will improve data access locality Sequential allocation with user-optimal migration: will improve load balancing Multi-user environment Each user migrates data in a user-selfish manner Migrations will tend to improve the performance of all users over longer periods of time 29 Texas A&M University Narasimha Reddy

30 Evaluation Implemented as a kernel block device driver Evaluated it using SPECsfs benchmark Configuration SPECsfs Performance Curve Single-User Multi-User 30 Texas A&M University Narasimha Reddy

31 Single-User Environment Configuration: (Allocation Policy)-(Migration Policy) STR (Striping), SEQ (Seq. Alloc.), NOMIG (No migration), MIG (User-Optimal migration) Striping with user-optimal migration Seq. allocation with useroptimal migration 31 Texas A&M University Narasimha Reddy

32 Single-User Environment (cont.) Comparison between migration systems Migration based on locality: hot data (remote local), cold data (local remote) 32 Texas A&M University Narasimha Reddy

33 Multi-User Environment - Striping Server A: Load from 100 to 700 Server B: Load from 50 to Texas A&M University Narasimha Reddy

34 Multi-User Environment Seq. Allocation Server A: Load from 100 to 1100 Server B: Load from 30 to Texas A&M University Narasimha Reddy

35 Storage Management Across Diverse Devices Flash storage becoming widely available More expensive than hard drives Faster random accesses Low Power consumption In Laptops now In hybrid storage systems soon Manage data across Different Devices Match application needs to device characteristics Optimize for performance, power consumption 35 Texas A&M University Narasimha Reddy 8/7/2007

36 Motivation VFS Allows many file systems underneath VFS maintains 1 to 1 mapping from namespace to storage Can we provide different storage options for different files for a single user? /user1/file1 storage system 1, /user2/file2 storage system 2 36 Texas A&M University Narasimha Reddy 8/7/2007

37 Normal File System Architecture Calc Impress Writer WinAmp /user1/file1 /user1/file2 /user2/file3 /user2/file4 User Space VFS Kernel /user1/* Ext2 /user2/* FAT32 Magnetic Disk Flash Drive 37 Texas A&M University Narasimha Reddy 8/7/2007

38 Umbrella File System Calc Impress Writer WinAmp /user1/file1 /user1/file2 /user1/file3 /user1/file4 User Space VFS Kernel UmbrellaFS /FS1/user1/file3 Ext3 Ext2 FAT32 /FS2/user1/file1 /FS3/user1/file4 /FS2/user1/file2 Magnetic Disk Encrypted Magnetic Disk Flash Drive 38 Texas A&M University Narasimha Reddy

39 Example Data Organization User View /usr /usr/dir1 /usr/dir1/foo.jpg /usr/dir1/foo.txt /usr/dir1/foo.avi /images/ usr /images/usr/dir1 /text/usr /text/usr/dir1 /media/ usr /media/usr/dir1 /images/usr/dir1/foo.jpg /text/usr/dir1/foo.txt /media/usr/dir1/foo.avi Underlying data organization 39 Texas A&M University Narasimha Reddy

40 Motivation --Policy Based Storage User or System administrator Choice Allow different types of files on different devices Reliability, performance, power consumption Layered Architecture Leverage benefits of underlying file systems Map applications to file systems and underlying storage Policy decisions can depend on namespace and metadata Example: Files not touched in a week slow storage system 40 Texas A&M University Narasimha Reddy 8/7/2007

41 Rules Structure Provided at mount time User specified Based on inode values (metadata) and filenames (namespace) Provides array of branches 41 Texas A&M University Narasimha Reddy 8/7/2007

42 Umbrella File System Sits under VFS to enforce policy Policy enforced at open and close times Policy also enforced periodically (less often) UmbrellaFS acts as a router for files Not only based on namespace, but also metadata 42 Texas A&M University Narasimha Reddy 8/7/2007

43 Inode Rules Structure Rule Inode/ Filename Field Match Value Branch 1 Inode file permissions = Read Only /fs1, /fs2 2 Filename n/a n/a n/a n/a 3 Inode file creation time >= 8:00 am, August 3 rd, 2007 /fs2 4 Inode file length < 20 KB /fs3 43 Texas A&M University Narasimha Reddy 8/7/2007

44 Inode Rules Provide in order of precedence First match Compare inode value to rule At file creation some inode values indeterminate Pass over those rules 44 Texas A&M University Narasimha Reddy 8/7/2007

45 Filename Rules Structure Rule Match String Branch 1 /*.avi /fs2,/fs1 2 /home/*.txt /fs1 3 /home/jgarrison/* /fs3 45 Texas A&M University Narasimha Reddy 8/7/2007

46 Filename Rules Once first filename rule triggered, all checked Similar to longest prefix matching Double index based on Path matching Filename matching Example: Rules: /home/*/*.bar, /home/jgarrison/foo.bar File: /home/jgarrison/foo.bar File matches second rule more closely (3 path length and 7 characters of file name vs. 3 path length and 4 characters of file name) 46 Texas A&M University Narasimha Reddy 8/7/2007

47 Evaluation Overhead Throughput CPU Limited I/O Limited Example Improvement 47 Texas A&M University Narasimha Reddy 8/7/2007

48 UmbrellaFS Overhead Bonnie Read Overhead Ext2 Inode Rules Filename Rules Throughput (MB/s) Ext Rules 48 Texas A&M University Narasimha Reddy

49 CPU Limited Benchmarks 49 Texas A&M University Narasimha Reddy 8/7/2007

50 I/O Limited Benchmarks 50 Texas A&M University Narasimha Reddy 8/7/2007

51 Flash vs. RAID5 Read Performance 51 Texas A&M University Narasimha Reddy 8/7/2007

52 Flash vs. RAID5 Write Performance Write Performance Throughput (MB/s) RAID 5 Flash SSD File Size (kb) 52 Texas A&M University Narasimha Reddy

53 Flash and Disk Hybrid System 53 Texas A&M University Narasimha Reddy 8/7/2007

54 Disks with Encryption hardware Encryption Example Time (s) Partial Encryption Full Encryption 54 Texas A&M University Narasimha Reddy

55 Conclusion Virtual allocation allows Flexibility Improve the flexibility of managing storage across multiple file systems/platforms Enabled user-optimal migration Balance disk access locality and load balance automatically and transparently Adapt to changes of workloads and loads in each storage device Policy-based storage: Umbrella File System Allows matching application characteristics to devices 55 Texas A&M University Narasimha Reddy