Tape Cloud Scalable and Cost Efficient Big Data Infrastructure for Cloud Computing. Varun S. Prakash with Yuanfeng Wen Weidong Shi

Transcription

1 Tape Cloud Scalable and Cost Efficient Big Data Infrastructure for Cloud Computing Varun S. Prakash with Yuanfeng Wen Weidong Shi University of Houston IEEE Cloud July 1, 2013 I2C Lab

2 Need for Affordable Storage Credit card Transactions per second Walmart: 1 million customer transactions every hour THE AGE OF BIG DATA 247 billion s are sent every Half a Million Virtual Machines in Amazon alone 7/1/2013 Tape Cloud: IEEE Cloud

3 Prevalent Storage Practices Omnipresence of Hard disk based storage system Reduced Costs per unit data stored Reduced size and form factor per unit data stored Acceptable performance for most applications Magnetic Tapes used mainly for archiving and backup applications Hybrid and Automated Infrastructures 7/1/2013 Tape Cloud: IEEE Cloud

4 HP LTO5 Tape Drive Slots to Hold Tapes (12 on each side) Robotic Arm to Load and Unload Tapes Tandberg T24 Tape Library 7/1/2013 Tape Cloud: IEEE Cloud

5 Capacity: 1.5TB Read Rate: 140 MB/s Tracks: 1280 Tandberg T24 Tape Library 7/1/2013 Tape Cloud: IEEE Cloud

6 Energy Used (Joules) A Comparison: HDDs and Tapes (1) Total Operating Costs Idle Standby Tape 430 Hitachi HDS725050KLA IBM IC35L060AVER Fujitsu MPE3084AE Seagate ST380215A Hitachi HDP725025GLA Seagate ST373455LW 7/1/2013 Tape Cloud: IEEE Cloud

7 Temperature (Centigrade) A Comparison: HDDs and Tapes (2) Operating Temperatures Hitachi HDS725050KLA360 Tape Drive IBM IC35L060AVER07 Fujitsu MPE3084AE Seagate ST380215A Hitachi HDP725025GLA380 Seagate ST373455LW Time (Seconds) 7/1/2013 Tape Cloud: IEEE Cloud

8 A Comparison: HDDs and Tapes (3) Total Cost of Ownership for 12 years 7/1/2013 Tape Cloud: IEEE Cloud

9 Commonly Arising Questions Are hard disks a justified choice for storing big data? 90% of the data on networked servers sits untouched (Andrew Leung, Joab Jackson) Can hard disks and related infrastructure profitably serve the big data paradigm? Can TAPES offset its performance lax to compete on the cost front? 7/1/2013 Tape Cloud: IEEE Cloud

10 Cost Efficiency Patrons Media, Maint., Facilities 7/1/2013 Tape Cloud: IEEE Cloud

11 Cost Efficiency Patrons Media, Maint., Facilities Opportunities Capital, Personnel 7/1/2013 Tape Cloud: IEEE Cloud

12 Tape Cloud Cloud based storage IaaS offering nearline and archiving services Hybrid Infrastructure with Magnetic Tape backend and HDD/SSD based buffers Service users exempted from high infrastructure and personnel investments Easy scaling costs for service providers Smaller operating costs 7/1/2013 Tape Cloud: IEEE Cloud

13 High total cost of ownership due to in-house archiving infrastructure investment Infrastructure & Media Operation & Facilities Costs Personnel In-house SAN / NAS archive Company 1 Company 2 Company 3 Host Applications / Servers 7/1/2013 Tape Cloud: IEEE Cloud

14 Company 1 Company 2 Company 3 Eliminated infrastructure, operation, facility and personnel costs 7/1/2013 Tape Cloud: IEEE Cloud

15 Agenda Motivation Introduction Architecture Challenges Ongoing and Future Plans 7/1/2013 Tape Cloud: IEEE Cloud

16 Architecture To design Tape Cloud is to design a Middleware Supports deployment and use of state-of-the-art in tape technology Helps in recognizing and managing performance controlling factors Tasks of Middleware: Data Collection, Data Blocks, Encryption, Policies, I/O Task Management etc. 7/1/2013 Tape Cloud: IEEE Cloud

17 The Data Manager Collect Data from Clients Blocking and Encryption policy enforcement Requests Load Balancing Policies, Rules and Management are the key functions of the Middleware 7/1/2013 Tape Cloud: IEEE Cloud

18 Task Distribution Manage Metadata of Data Blocks Distribution of Blocks Distribution Policies improve overall efficiency 7/1/2013 Tape Cloud: IEEE Cloud

19 Task Queuing Scheduling of IO tasks to optimize performance Client and Workload based queuing policies can be enforced 7/1/2013 Tape Cloud: IEEE Cloud

20 Tape Cloud FUSE Filesystem Userspace Filesystem to manage data and blocks implemented using FUSE 7/1/2013 Tape Cloud: IEEE Cloud

21 Agenda Motivation Introduction Architecture Challenges Ongoing and Future Plans 7/1/2013 Tape Cloud: IEEE Cloud

22 Chief Challenges Business Layer Service Level Agreement for offsite handling sensitive data Limited benefits of Public Clouds over Private Clouds Adaptability to Colocated Infrastructures Technology Layer Data types, formats and workloads Efficient Transfer of Data during Restoration Performance of Tape Library Robotics 7/1/2013 Tape Cloud: IEEE Cloud

23 Agenda Motivation Introduction Architecture Solutions Ongoing and Future Plans 7/1/2013 Tape Cloud: IEEE Cloud

24 Project Advancement Tape Cloud offers exciting research prospects Economics of revisiting Legacy Storage Systems Overhaul of Infrastructure Enhancement of Middleware data distribution and queuing techniques Support for generic and archiving storage workload Extending services to co-located applications 7/1/2013 Tape Cloud: IEEE Cloud

25 Thank you 7/1/2013 Tape Cloud: IEEE Cloud

26 Tape Cloud Scalable and Cost Efficient Big Data Infrastructure for Cloud Computing Varun S. Prakash Backup Slides University of Houston IEEE Cloud July 1, 2013 I2C Lab

27 Closest Process First Scheduling (CPFS) Considers the physical distance of the next process to be executed Suitable due to latencies and architecture of tape libraries 7/1/2013 Tape Cloud: IEEE Cloud

28 Average Response Time(min) Average Response Time (min) Design Assessment (1) FCFS CPF Block Size (GB) # of requests (*1000) Optimizing the Block Size of Sequential Read [1000 Requests, 80% Read] Comparison the performance of CPF and commonly followed FCFS scheduling 7/1/2013 Tape Cloud: IEEE Cloud

29 Average Response Time (min) Average Response Time(min) Design Assessment (2) Partitioned Scheduling Unpartitioned Scheduling Without Global Partition With Global Patition # of Tape Drivers # of request (*1000) The performance of a partitioned tape library infrastructure from the perspective of the no. of Tape Drives The performance of a partitioned tape library infrastructure from the perspective of the no. of Requests 7/1/2013 Tape Cloud: IEEE Cloud