Getting performance & scalability on standard platforms, the Object vs Block storage debate 1
December Webinar Session Getting performance & scalability on standard platforms, the Object vs Block storage debate Speakers William Oppermann, CEO, MPSTOR Mo Hassine, Director of Product and Marketing, MPSTOR 2
Storage requirements in the datacenter Wide range of application workloads provisioned at scale cost effectively Provisioning ALL the Datacenter consumers Opex Capex Virtual machines Tenant spaces Storage centric services Consumer nodes Complexity & management Automation of provisioning Volume services management (snapshot, thin volumes, replication, backup) Use of Open Platforms Proprietary platforms Storage Efficiency (the cost of redundancy in storage) Availibility&Resiliency Component Redundancy Data Consistency& Integrity IDA (Information Dispersal Algorithms) Scalability Scale up Scale scale Reducing the Impact of failures through IDA Performance per workload type BW performance IO performance Caching acceleration Fabrics Tiering SLA/QOS 3
The BIG 6 issues Storage technologies for the cloud struggle in the datacenter because there is a BIG 6 requirements list that is very difficult to fulfil. 1) Storage must be resilient to component failure 2) Storage must be scalable (addition of capacity and amount of data stored) 3) Storage must cater for a wide range of application workloads 4) Storage provisioning to a wide range of storage consumer types Virtual machines Tenant spaces Storage centric services Consumer nodes 5) Storage must deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing. Open platforms Highly automated provisioning End user provisioning tools Simple and easy to administer by the cloud operator 6) Storage must be secure 4
The BIG6 storage challenges RESILIENCY SCALABILITY WORKLOADS CONSUMER TYPES TCO (CAPEX and OPEX) DATA Security 5
Big 6 ranking out of 10 for Block and Object 12 10 8 6 4 2 Block Object 0 6
One paradigm does not fit all ISSUES BLOCK OBJECT? COMMENTS RESILIENCY DIVERSE WORKLOADS TCO (Cloud CAPEX and OPEX) STORAGE SECURITY SCALABILITY MULTIPLE CONSUMER TYPES Object has large windows of failure & poor storage efficiency Object suitable for narrow range of workload types Block storage high cost to Administer Both object and block storage needs managed encryption and security access BLOCK storage very difficult to manage for scale out multi fabric, multi tier storage. Missing paradigm in both Block, File and Object storage for many of the datacenter consumer types Full Support Partial Support Poor Support 7
Object & Block storage Object storage is resilient to component failures, scalable and can be delivered on open hardware platforms cost effectively for some workload types. Strong point is cost, scalability and managing the impact of disk failures but object storage has poor performance in workloads requiring high IO & low latency => It can deliver partially on the BIG 6 issues Block storage is a resilient & somewhat less scalable technology that is usually delivered on proprietary platforms that supports all workload types. Strong point is very good mixed work load performance, support for different media types & fabrics. Scaling can be difficult and the recovery time from failures in large silos can prohibitive. => It can deliver partially on the BIG 6 issues 8
BIG5 Issue#1 (using object storage) Storage needs to be resilient to component failure Object storage keeps multiple copies of data objects within its storage pool (3 copies => storage efficiency = 100/3 = 33%) Each copy is replicated over time T, during this T minutes the data is not protected, T can be long multiples of minutes and is not acceptable in many mission critical enterprise class configurations. If a disk fails the object store notes which objects were dependent on that disk and makes new copies over time of the lost objects. => At the cost of low storage efficiency and a wide window in time of non protection object storage delivers resiliency. For the designed for USE CASE (upload/download outside the datacenter of digital media (photos, videos, documents) this can be acceptable 9
BIG5 Issue#2 (using object storage) Storage needs to be scalable (Scalable in terms of overall capacity and amount of data stored) Adding storage capacity is relatively simple but usually requires a major re-balancing operation when storage is added to the pool (all the data gets reshuffled between its disks) Data objects are stored using data base technology. Each object has a unique ID, the ID is used as a database KEY, as the object store grows the lookup cost of keys grows with the number of objects Object storage has a high overhead every time the object is accessed. This overhead grows with the number of stored objects. Object storage delivers partially the requirement of increasing the pool capacity size at the cost of rebalancing itself when capacity is added. Object storage struggles with the ever increasing number of objects it has to store. 10
BIG5 Issue#3 (using object storage) Storage needs to cater for a wide range of end user workloads The SEMANTICS of Object storage make it suitable for only a restricted (but important) set of workloads, its semantics also make provisioning an end user task which improves the provisioning OPEX cost. Storing photos/videos/documents in object storage works well, taking seconds to load a photo/video is acceptable if once loaded the data can stream. For data processing workloads the object storage look-up costs make it unusable. Adding BLOCK interfaces on OBJECT stores only makes the problems worse in terms of performance but also increases the complexity of the solution. 11
BIG5 Issue#4 (using object storage) Storage delivery to a wide range of storage consumer types Object storage uses a very specific API making it useless at provisioning multiple consumer types which all need block storage to run/boot but can in some cases use object storage when in operation for certain workloads. 12
BIG5 Issue#5 (using object storage) Storage needs to deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing pricing. Storing photos/videos/documents using object storage works well. Simple interface, end user can provision, storage looks like a big pool => Object storage on low cost media & open hardware platforms even with low storage efficiency due to its multiple copies works well for upload/download of large media files. Object storage is poor at transaction data processing, low latency and high IO type workloads Object storage partially delivers on the big5 requirements! 13
BIG5 Issue#1 (using Block storage) Storage needs to be resilient to component failure Block storage uses RAID or ERASURE code technology to store data resiliently. Block storage uses many techniques to improve resiliency to failures such as multipathing, dual controllers, cache coherency techniques which store data with a ZERO time window of non redundancy. Data is secured and consistent, i.e a block storage controller can die mid flight of an IO and the system will store work. This very complex hardware and software is a considerable technical challenge and is reflected in the high cost and proprietary nature of block storage. RAID&ERASURE techniques require special consideration so that the BUILD and REBUILD recovery times stay within defined limits. 14
BIG5 Issue#2 (using Block storage) Storage needs to be scalable (Scalable in terms of overall capacity and amount of data stored) Adding storage usually requires a new RAID, a RAID ADD cost depends on the size of the RAID and the RAID level. Disks sizes are now getting so large the RAID build time is becoming a major problem to admins, during a RAID rebuild the raid is not fully redundant and if enough disks are lost due to failure the entire dataset can be lost. Block storage has been designed for a high setup cost and a very low transaction cost during operation. Unlike object storage the transaction cost of accessing data scales perfectly with the amount of data stored since the transaction cost is a FIXED cost. Block storage works very efficiently when managing a storage SILO (SCALE- UP by adding more storage to the SILO), SCALE-OUT (i,e adding more storage SILOS in parallel) is difficult. This scale out issue in contrast to OBJECT storage puts BLOCK storage into an "all the eggs in one basket" type technology. The basket may be resilient and redundant but there is only one basket. 15
BIG5 Issue#3 (using object storage) Storage needs to cater for a wide range of end user workloads Block storage has inbuilt into its semantics and implementation the ability to cover all workload types. This is BLOCK storages strongest capability and is one of its major advantage over Object storage. Additional advantages are its ability to transition across multiple high speed fabrics and provide the raw building blocks for other protocols such as File and Object storage iself. 16
BIG5 Issue#4 (using Block storage) Storage delivery to a wide range of storage consumer types Block storage does not in it self provision all the consumer types in the datacenter, as a base technology its far easier and higher a performing base to develop tools that can provision all the consumer types of Virtual machines Tenant spaces Storage centric services Consumer nodes 17
BIG5 Issue#5 (using Block storage) Storage needs to deliver 1,2,3,4 at a CAPEX and OPEX compatible with Utility cloud computing pricing. Block storage scales both in capacity and quantity of data stored. Block scales up very easily, scales out with difficulty, manages all workloads very well and is very resilient to failures with a ZERO window of time when the data is not redundant. Block supports wide range of FABRICS and protocols (SAS, FC, IB, FCoE, Eth) in comparison to Object storage (Eth, IP). Block storage has good support for media tiering and accelerated caching using SSD technology. Block storage is complex and in most cases uses proprietary hardware. Block storage is more difficult to administer than object storage. 18
Block? Object, Both? or something else? Is that the end of the debate? Who wins Block or Object? Do we need both? Do we need something else? What are the hard limitations 19
Block v/s Object semantics SCSI CDB LBA@,LEN S3 API ByteArrayInputStream input = new ByteArrayInputStream("Hello World!".getBytes()); conn.putobject(bucket.getname(), "hello.txt", input, new ObjectMetadata()); Volume 20
Block, Object, Block over Object 3 Block over Object Layer LBA Map Obj App S3 Backer Rados Block Device (RDB) 2 Object Layer API Head DB+Files User Land Mgt 1 Block Layer FS HD driver Kernel disk 21
Semantic cost & Performance 50K IO/sec @8K 300K IO/sec @8K 100% CPU Utilisation IO/sec Referece: http://www.mellanox.com/relateddocs/whitepapers/wp_deploying_ceph_over_high_performance_networks.pdf 22
Scale out storage 40G/100G 8PB 8PB 8PB 8PB 10G/16G/40G Scale out Cluster up to 10 nodes Exporting BLOCK FILE OBJECT 1 192TB Scale out Storage Nodes Up to 128 nodes 128 23
Scale Out Scale out allows a storage service to scale in real time without service interruption in both capacity and performance Scale out storage can be Block (ScaleIO, Orkestra-IDA (information dispersal) Object (CEPH, SWIFT) File (GPFS, Gluster) 24
Object IDA (Information Dispersal Algorithm) IO_Write D Load Balancer 1 2 Proxy Proxy Ring D#1 C#2 C#3 3 4 5 5 D#1 C#1 C#2 Zone1 Zone2 Zone3 Zone4 FIG 2 25
Block IDA (Information Dispersal Algorithm) IO_Write D VBD 1 2 VBS VBS Group D/3 D/3 D/3 P 3 3 3 4 5 D/3 D/3 D/3 P Zone1 Zone2 Zone3 Zone4 FIG 2 26
Storage Resiliency Object storage makes copies over (large window of non protection)across multiple storage arrays Weak real time consistent data Scale out block storage stores data redundantly in real time across multiple storage arrays Strong real time consistent data 27
IDA Volume create VBS Node IDA VOL IDA RAID VBD Space is reserved according to the RAID QOS on independent Arrays and built into an RAID on the VBS layer RG1 RG1 RG1 RG1 QoS1 Storage Array Nodes RG2 RG3 RG2 RG3 RG2 RG3 RG2 RG3 QoS2 QoS3 RG4 RG4 RG4 RG4 Array 1 Array 2 Array 3 Array 4 28
Managed pools is not Scale-Out Compute Containers Orkestra SDS Automation Fabrics 1G iscsi Storage Containers Scale out Compute Groups BRONZE compute GROUP SILVER compute GROUP BASIC STANDARD PREMIUM SDS automates the provisioning of storage per group Each GROUP has a configurable QoS & SLA 10G iscsi 6G SAS 4/8/16G FC Storage Pools GOLD compute GROUP 29
Block & Object Storage efficiency 120% Storage Efficiency 100% 80% 60% 40% 100% 94% 50% 88% Object Series1 Block Series2 20% 33% 0% 1 2 3 # parity drives/copies 30
Multi-Tenancy using throttling MPSTOR Data Center Storage Automation SOS supports Bandwidth Throttling (both IOPS and MB/s) 31
Storage Features in the data-center Feature Snapshot Thin Provisioning Rate limiting Replication Tiered Management Solid State Disk Caching High Availibility FC/FCoE SAN SAS SAN iscsi SAN NFS protocol CIFS protocol FC SAS iscsi NFS CIFS Benefits Allows users to take point of time copies Allows capacity to be added as demand increases Allows multi tenancy of high speed media Resilient to failure Allows wide range of workloads Speeds up IO by caching No single point of failure, no loss of functionality or data Automated management of high speed fabric Automated management of high speed fabric TAutomated management of high speed fabric File access protocol for LINUX File access protocol for Windows 32
Storage categories & Contenders Converged Storage Block File Object Server SAN BIG IRON Converged Infrastructure VSAN SOFTWARE DEFINED STORAGE Evolving set of terms and definitions which are frequently a source of confusion 33
One paradigm does not fit all ISSUES BLOCK OBJECT? COMMENTS RESILIENCY DIVERSE WORKLOADS TCO (Cloud CAPEX and OPEX) STORAGE SECURITY SCALABILITY MULTIPLE CONSUMER TYPES Strong or weak real time consistency Block performance Use standard platforms for BLOCK or OBJECT with SDS automation? Gaps exist? Scale out technologies exist for both Block and Object storage Missing paradigm in both block, file and object storage for many of the datacenter consumer types Full Support Partial Support Poor Support 34
Thank You 35