A BEGINNER S GUIDE TO NEXT GENERATION OBJECT STORAGE WHITEPAPER INTRODUCTION Object Storage is the new storage paradigm. There is a high level of interest from organizations, as this new approach resolves the challenges of efficiently storing massive volumes of unstructured data - Big Unstructured Data. This paper addresses the why, what and how of object storage. WHY SHOULD COMPANIES USE OBJECT STORAGE FOR UNSTRUCTURED DATA AND HOW IS IT DIFFERENT FROM NAS OR SAN? The biggest problem with traditional approaches is scalability. NAS lacks the ability to scale as a single system, especially in Petabyte environments. Today s SANs are already complex, when deployed with a file system layer on top. Scaling-out makes the problem a lot worse. Object Storage is essentially just a different way of storing, organizing and accessing data on disk. An Object Storage platform provides a storage infrastructure to store files with lots of metadata added to them referred to as objects. The backend architecture of an object storage platform is designed to present all the storage nodes as one single pool. With Object Storage, there is no file system hierarchy. The architecture of the platform, and its new data protection schemes (vs. RAID, the de-facto data protection scheme for SAN and NAS), allow this pool to scale virtually to an unlimited size, while keeping the system simple to manage. Users access object storage through applications that typically use a REST API (an internet protocol, optimized for online applications). This makes object storage ideal for all online, cloud, environments. When objects are stored, an identifier is created to locate the object in the pool. Applications can very quickly retrieve the right data for the users through the object identifier or by querying the metadata (information about the objects, like the name, when it was created, by who etc.). This approach enables significantly faster access and much less overhead than locating a file through a traditional file system. DDN WOS is a true object storage platform, designed to scale beyond petabytes as a single system, optimizing TCO without compromising performance or durability. This makes WOS a perfect platform for a variety of storage cloud solutions, including online collaboration, active archives, cloud backup and worldwide data distribution.
TABLE OF CONTENTS INTRODUCTION...2 Why should companies use Object Storage for unstructured data and how is it different from NAS or SAN?... 2 HISTORY OF OBJECT STORAGE...3 WHAT IS OBJECT STORAGE?...3 SAN vs NAS... 4 Object Storage, The Third Paradigm... 4 Cloud Storage, Storage Clouds, Object Storage... 6 REST API s... 7 DirectProtect Real-Time Error Detection & Correction... 7 Object Storage Summary... 7 WHY OBJECT STORAGE?...7 Massive Data Growth... 7 Always Online... 7 Power to the Applications... 8 The Big Data Explosion... 8 We All Use Object Storage Everyday... 8 Use Cases... 9 HOW DOES OBJECT STORAGE WORK?...9 Issues with File Storage... 9 Data Protection: Erasure Coding or Not?... 12 WOS : WEB OBJECT SCALER...13 True Object Storage Platform... 13 Optimized for Small and Large Files... 13 Choice of Data Protection Schemes... 13 Self-healing Architecture... 13 Single Storage Infrastructure... 13 Widest Selection of Interfaces; Out of the Box Applications... 14 Enterprise-grade Platform... 14 WOS Benefits... 14 ECOSYSTEM...14 RESOURCES... 16
HISTORY OF OBJECT STORAGE Object Storage is not a new concept, it even predates the 2006 launch of Amazon s S3. Much advancement have been made, and today s current generation of Object Storage platforms cannot be compared to the earlier generations - which were merely black boxes designed to store immutable copies of documents, mostly for compliance environments. EMC Centera, based on Content Addressed Storage (CAS) innovator Filepool, was one such early implementation of an object-based construct. Today, Centera users face a big challenge to move to newer, faster storage infrastructures: Centera provides no interfaces to migrate data to other platforms. THE INDUSTRY HAS LEARNED SOME TOUGH LESSONS USING PROPRIETARY SYSTEMS. The current generation of object storage platforms is designed with this openness & flexibility in mind. Most platforms support a subset of Amazon s REST API and some platforms are designed to be independent of the hardware platform. The industry has learned some tough lessons from using proprietary systems. One initiative to prevent Vendor Lock-in, is SNIA s Cloud Data Management Interface (CDMI). This is a set of pre-defined RESTful HTTP operations for assessing the capabilities of the cloud storage system, allocating and accessing containers and objects, managing users and groups, implementing access control, attaching metadata, billing, moving data between cloud systems, exporting data, etc. 1 Figure 1 Object Storage Archit ecture WHAT IS OBJECT STORAGE? Object Storage is essentially just a different way of storing, organizing and accessing data on disk. To really understand how object storage is different from traditional storage platforms, it is important to understand the what and how of traditional storage, and what the challenges are 2. 1 From Wikipedia: http://en.wikipedia.org/wiki/cloud_data_management_interface 2 For the sake of briefness, we will stick to the very basics. There are hundreds, if not thousands of blog articles and papers about this topic available online. 3
SAN VS NAS A SAN is block storage device, not that different from an external USB disk drive, just bigger. Systems connect to a SAN with a block interface; common protocols for block storage include iscsi, Fibre Channel, Fibre Channel over Ethernet (FCoE), etc. A device attaching to SAN will see the storage presented as a disk drive. SANs allow multiple servers to share a pool of storage that cannot be accessed by individual users. This is to prevent the overwriting each other s data. SANs are typically used by large applications, such as enterprise databases, that handle data locking through the application. SAN storage can be presented as a file system (by putting a file system layer on top), which is generally referred to as a clustered file system. As we will explain later in this document, SANs are complex systems to manage, especially when used for file storage. A NAS is a file storage device. NAS exposes its storage as a network file system. Devices that attach to a NAS see a mountable file system. Common protocols for file storage devices include, NFS and SMB / CIFS. A NAS operates at the file level and is accessible to users with proper access rights - so it needs to manage user privileges, file locking and other security measures. A NAS environment is a much better fit than SANs for to store files. THE BIGGEST CHALLENGE WITH NAS AND SAN IS SYSTEM SCALABILITY. Figure 2 - Simplified SAN infrastructure with Clustered file system and enterprise applications OBJECT STORAGE, THE THIRD PARADIGM So, if NAS & SAN can store files, then, why Object Storage? How is it different? As we will explain in greater detail to follow, the biggest problem with both systems is scalability. NAS cannot scale as a single system in petabyte-size environments. To scale-out a NAS environment requires a combination of multiple systems (management!) or forklift upgrades, with labor-intensive data migration projects. As we mentioned before, SANs are pretty complex when deployed with a file system layer on top. Scaling-out makes the problem a lot worse. Again, with lots of management! Also, most of the unstructured data that is stored online (or in active archives) is immutable data meaning the file will not be modified. Much of the functionality built-in to traditional file systems addresses user access rights and permissions for appending and amending files. These complex functions create a lot of overhead in terms of performance, IOs required to access data and the 4
OBJECT STORAGE IS A MUCH SIMPLER ARCHITECTURE THAN TRADITIONAL FILE SYSTEMS. Figure 3 - Object Storage Timeline ability to scale. Object Storage does not have this functionality. If a user modifies a file, the new version is simply stored as a new object. This results in a much simpler architecture than traditional file systems have. An Object Storage platform is a storage infrastructure to store objects. For now we will refer to objects as similar to files (collection of data blocks with metadata), later in this document we will explain how this is actually only partly true. The backend architecture of an object storage platform is designed so that all the storage nodes are presented as one single pool. There is no file system Figure 4 - Scale out object storage with simple REST API for applications 5
APPLICATIONS CAN RETRIEVE OBJECTS MUCH FASTER THAN FILES. Figure 5 - NAS Storage is presented as a file system to the clients REST API S HAVE BECOME THE PREDOMINANT INTERFACE FOR CLOUD APPLICATIONS. hierarchy. The architecture of the platform, and new data protection schemes (vs. RAID, the defacto data protection scheme for SAN and NAS) allow this pool to scale to virtually unlimited capacities, while keeping the system simple to manage. Users access object storage through applications that will typically use a REST API. They use a set of simple commands: GET (read), PUT (save) and DELETE. REST is an internet protocol, optimized for online applications. This makes object storage ideal for all online, Cloud, environments. When objects are stored, an identifier is created to locate the object in the pool. Applications can very quickly retrieve the right data for the users through the object identifier - or by querying the metadata (information about objects: name, when it was created, by who, etc.). This is much faster than attempting to locate a file through a traditional file system. Applications also handle user access management. Each time a file (object) is changed, it is stored as a new object. This prevents corruption through simultaneous changes. CLOUD STORAGE, STORAGE CLOUDS, OBJECT STORAGE What is the difference between Cloud Storage and Storage Clouds? How does Object Storage fit? The answer is pretty straightforward. Cloud Storage is the storage used for Compute Cloud infrastructures - in other words: to run VM s on. Compute Clouds are very IOPS intensive and usually block storage is used in these applications. Storage Clouds are storage in the cloud, whether public or private. So, Storage Clouds are simply storage capacity that is made available through the Internet. Most of today s storage clouds use object storage technologies. 6
LITTLE PROGRESS HAS BEEN MADE ON API STANDARDIZATION. REST API S REST stands for Representational State Transfer. It is a software architecture that is used for distributed application environments, such as the internet. An API, short for Application Programming Interface, is an interface used for an application (client) to talk to its environment (backend servers, storage, databases etc.). With the success of cloud-style computing (running applications in the cloud, rather than on the user s computer), REST API s have become the predominant interface for cloud applications to connect to the cloud. For storage-centric cloud applications, a REST API is the interface between the application and the object storage platform. DIRECTPROTECT REAL-TIME ERROR DETECTION & CORRECTION The three most common commands in REST API s for object storage environments are GET, PUT and DELETE, which are the equivalents of reading a file, saving a file (technically save as because object storage does not allow you to update an object), or deleting a file. Since the early days of Cloud Computing, there s been a lot of discussion about standardizing on a specific REST API to avoid vendor lock-in. The general idea behind this is, if all vendors (of applications, cloud infrastructures, object storage platforms etc.) use a standard API, users will never be locked-in to a specific environment. Without having to reprogram their applications, they would be able to freely move their data from one platform to another - or keep it on more than one platform. Little progress has been made on the standardization front however, and the result is that object storage platforms will either support the Amazon S3 API, the OpenStack API or a native API (i.e. an API of their own, typically a very easy to use, lightweight interface). OBJECT STORAGE SUMMARY Data is stored as objects in one large, scalable pool of storage Objects are stored with metadata information about the object An Object ID is stored, to locate the data REST is the standard interface, simple commands used by applications Objects are immutable; edits are saved as a new object WHY OBJECT STORAGE? IN 10 YEARS, THE AVERAGE STORAGE OPERATOR WILL HAVE TO MANAGE 15-20 TIMES MORE STORAGE THAN TODAY. MASSIVE DATA GROWTH Depending on which analyst firm you talk to, you will hear storage growth predictions that vary between 30x-40x for the next decade. That means we will all be storing 30 to 40 times as much digital data ten years from now, compared to today. At the same time, companies will only invest an additional 50% in personnel to manage their storage infrastructures. This means that the average storage operator will have to manage 15-20 times as much storage a decade from now. This will drive the need for storage platforms that require little management effort and scale out to virtually unlimited capacities. ALWAYS ONLINE Much of that data growth is driven by the recent innovations in cloud and mobile computing. We already mentioned Amazon S3, but there are also Google, Facebook, Apple and several smaller public storage cloud offerings that set a new level of expectations where all data needs to be available anywhere at anytime. 7
80% OF DATA GROWTH WILL BE UNSTRUCTURED DATA POWER TO THE APPLICATIONS File-based storage platforms not only fail to scale sufficiently, they also become obsolete as more and more applications are designed to use REST API s (the default interface for object storage platforms) to talk directly to the storage, without additional (file system) layers in between. This greatly simplifies architectures and delivers significant performance gains. THE BIG DATA EXPLOSION Essentially, there are three types of Big Data: Big Structured Data, Big Semi-structured Data (Big Data Analytics) and Big Unstructured Data. All three require one or more of the three V s, the commonly accepted definition of Big Data: Big Data refers to any set of data that comes in great Volumes, has a large Variety of information and/or is consumed at high Velocity. Big Structured Data refers to large enterprise databases. Velocity is key here, hence the success of the superfast SSD drives. Big Semistructured Data refers to massive volumes of small log files (often sensor information), that is collected for analytics. Therefore, we also talk about Big Data Analytics. This data is stored in distributed frameworks that support distributed processing. Think of Hadoop and MapReduce. Object Storage is not commonly used for structured or semi-structured Big Data, unless it is for archival purposes. The sweet spot for object storage is Big Unstructured Data, which refers to all data that users best understand as files. Think of image and movie data always growing and always in higher resolution music files, office documents etc. Analysts believe that 80% or more of the expected data growth will be unstructured data. File based storage platforms cannot support this growth. This is the problem Object Storage solves. AMAZON ADVERTISES VERY LOW COSTS TO STORE DATA, BUT THERE ARE HIDDEN COSTS. WE ALL USE OBJECT STORAGE EVERYDAY Possibly the largest object storage infrastructure, and one of the drivers for the adoption of object storage is Amazon s S3. This public storage cloud service was launched in 2006 and has stimulated many application developers to have their applications use S3 as backend storage. The benefits were clear: no hassle with a private infrastructure, relatively low cost, pay as you go, scale as needed and a very simple interface. However, while Amazon advertises very low cost to store data on their infrastructure, there are some hidden costs such as network traffic. At a certain volume of data, there is a point of cost inflection. Many of the startups that launched on Amazon over the past years and who clearly see the benefits of object storage, are now deploying their own infrastructure using object storage platforms. Also, not everyone wants their data in a public environment with debatable SLA s and moderate security at best. More and more enterprises are choosing to deploy their own internal storage clouds to facilitate cloud-based applications. These infrastructures need similar or better availability and durability than the available public services. 8
USE CASES Object Storage is more than a smarter paradigm that allows you to store large volumes of unstructured data. Features like massive scalability, REST APIs, geographic distribution, enable a series of compelling use cases. An interesting side effect is that solutions tend to overlap. Dropbox is not just file sharing, it s backup, collaboration, archiving and mobile storage. Here are a few popular use cases: Online Web Services: As we mentioned earlier, one of the drivers for object storage is the trend to use more and more online cloud applications. Previously, without Amazon s S3, none of this would have been possible. The more successful web services companies are now gradually making the move to in-house infrastructures. Also, with corporate security policies, IP and compliance considerations, most enterprises prefer to run cloud applications on private storage infrastructures. File Sharing is by far the most popular object storage use case. Dropbox offered a solution for a need that most of us did not know we had. Today, service providers are now deploying similar services and enterprises are deploying private file sharing services as people utilize a variety of devices at home, at work and on-the-go. They collaborate with people across the office or around the world. Cloud Backup is increasingly popular. There are dozens and dozens of online services for backup. For enterprises, the idea of backing up to low cost, highly scalable disk infrastructures - rather than tape, which can be cumbersome for recovery - is also very compelling Cloud Archives: Data archiving decisions used to be very simple: data that was infrequently accessed was moved off disk to tape. Very few arguments could beat the low TCO of tape. Disk archives were hard to justify and reserved for those exceptional use cases where latency outweighed the huge cost of disk archives. With object storage, it is now possible to deploy disk archives at an acquisition cost and TCO close to that of tape. Many organizations are opting for hybrid environments - with a really, really superfast hot disk tier and a very cheap cold tape tier. Worldwide Collaboration: Globally distributed teams have become standard practice. Think of researchers from different institutions working on the same project. Think of a movie being shot in New Zealand and produced in Los Angeles - or software being developed in California and then tested in India. Geographically distributed storage pools enable teams to work in real-time on the same datasets. HOW DOES OBJECT STORAGE WORK? ISSUES WITH FILE STORAGE As we explained earlier in this document, file based storage is a great concept: users can access the same resources through a corporate network. The file system takes care of permissions, access rights and avoids users overwriting each other s data. File systems can even present data in a hierarchical Directory structure, which until now has been a very useful tool to keep data organized. The underlying software for such file systems contains a lot of ingenuity, which rapidly becomes complexity when scaling-out the infrastructure. 9
FILES ARE ABSTRACTIONS OF DATA IN A COMPUTER SYSTEM AND ARE REALLY STORED IN FIXED SIZE BLOCKS WHICH ARE ADDRESSED WITH A NUMBER. The concept of a file on a computer system is so well ingrained that it is often difficult to think of computer storage in different terms. It is clearly a very powerful and natural way to think of data. Object Storage is distinct from file storage, but in some ways is even a more natural and powerful way to organize data. The file concept is an abstraction. In actuality, data in a computer system is stored in fixed size blocks which are addressed with a number which ultimately is a physical location in a storage device. This is the case for data stored in a NAS, a SAN or when using Object Storage. The system presents those blocks to the user or application in a form that is useful. For nontransactional data, that form is usually a file. File storage systems will also store a small amount of information that tells which of those data blocks make up the file, in which order, and finally what name has been applied to the collection of blocks. This additional data about data is called file system metadata. Keeping track of the file system metadata is the responsibility of the file system. To keep even more than a few files organized, a file system imposes a hierarchy on the metadata in the form of directory structure. A key concept of the file system is the notion that the files themselves have relationships to one another as one could think of files being co- located in a directory. When the system is instructed to read a file, the repository of file system metadata is consulted and the required data blocks are retrieved from the storage device. Writing data into a file system has the additional complexity of requiring that the file system metadata must be written or updated - potentially by several users or processes simultaneously. Numerous techniques and designs exist that attempt to minimize the impact of dealing with file system metadata, and the locking problem associated with simultaneous access. Unfortunately, as the number of files in the system grows large, keeping the file system metadata correctly organized (so that the names and the data blocks that make up files can be found) becomes increasingly complex. As this requirement increases, keeping track of billions of files (which may be distributed across a number of network connected computer systems), the abstraction of the file system begins to breakdown. Moreover, the hierarchical structure of the file system is insufficient to adequately categorize the data in the system. File systems require at least three layers of software constructs to execute any file operation. As they allow files to be amended by multiple users, they must maintain complex lock structures with OPEN and CLOSE semantics. These lock structures must be distributed coherently to all of the servers used for access. AS FILE SYSTEMS GROW, THE ABSTRACTION OF THE FILE SYSTEM BEGINS TO BREAKDOWN AND INSUFFICIENTLY CATEGORIZES DATA. 10
IN AN OBJECT STORAGE SYSTEM, THE PHYSICAL STORAGE BLOCKS ARE ORGANIZED INTO OBJECTS WHICH ARE COLLECTIONS OF DATA BLOCKS REPRESENTED BY AN IDENTIFIER. Also, as data is placed (based on random block availability), traditional file systems are always fragmented. This is especially true in environments where the data is unstructured and it is not uncommon to write widely varied file sizes. Using a traditional file system designed for amendable data, storing immutable data constitutes an inappropriate and wasteful use of bandwidth and compute resources. This highly inefficient approach requires a great deal of additional hardware and network resources to achieve data distribution goals. These systems now become exponentially more complex as they are scaled-out. Object storage systems dispense with the overburdened concept of file system metadata. This approach allows the system to separate the storage of data from the relationship that the individual data items have to each other. In an object storage system, the physical storage blocks are organized into objects which are collections of data blocks represented by an identifier. There is no hierarchy imposed on the data and no repository of the objects metadata to be consulted when reads or writes are requested. This approach allows an object storage system to scale with both the requirements and size of the system, well beyond the technical & practical boundaries of traditional file systems. While Object Storage systems do not use file system metadata, they do employ object metadata (customizable information about the objects). This information can later be used to query or analyze the information stored. Object metadata for a photo could be the day it was taken, the last time it was modified, the type of camera that was used, whether a flash was used, where it was taken, etc. Object metadata will play an increasingly important role as we store more and more information, but it does not add complexity to the system like file system metadata does. At the highest level, storage servers are, like NAS and SAN, simply boxes with a lot of disks in there. Typically, object storage vendors will use SATA disks in their systems, and may include SSDs for caching. Some platforms opt for separate controllers, but in essence that does not make a difference, as the storage is presented as one pool (namespace). When choosing an object storage platform, it s important to understand the limitations of the namespace and how the system combines different pools or namespaces. Many vendors claim infinite scalability, but there is no such thing. The important thing is to understand how namespaces are combined, presented and managed. How many such namespaces can be combined? Are they managed as one system? The system software manages most of that. The actual software layer is where vendors can differentiate. The list of possible features is endless. A single management interface is always great. Self-healing capabilities are a must for environments that will scale into the hundreds of petabytes. The software layer also provides data protection mechanisms, which we will cover in the next section.an Object ID is stored, to locate the data The standard interface to access data in an object storage platform is a RESTful interface or REST API. This is a set of simple commands that application developers use in their code to let the application access the data. The basic REST commands are LIST, GET, PUT and DELETE, which are used to list (a selection of) objects, read an object, store an object or delete it. There is no standard for REST yet, but the so called Amazon API is by far the most popular amongst developers. Hence, most object storage providers will provide an Amazon compatible API, which is typically a subset of the commands that are supported by Amazon S3. As most legacy applications were designed to interface with a file system, most object storage platforms will also provide one or more file interfaces (a file system layer on top of the object storage pool also called a file system gateway) and often a selection of programming language-specific API s will be provided as well. DDN s WOS has the widest selection of interfaces on the market. 11
THE PROBLEM WITH RAID IS THAT IT WAS ORIGINALLY DESIGNED FOR SMALL CAPACITY DISKS. WOS WOS HAS THE WIDEST SELECTION OF INTERFACES ON THE MARKET. WOS WAS DESIGNED AS A TRUE OBJECT STORAGE PLATFORM. DATA PROTECTION: ERASURE CODING OR NOT? Several data protection mechanisms are used for object storage. The one method that is being abandoned, however, is RAID (which has been the de facto data protection scheme for SAN and NAS for the past two decades). The problem with RAID is that it was originally designed for small capacity disks. The larger the drive capacity, the longer it takes to restore a failed drive. During this restore, the data is less protected. If you are on RAID 5, and a second disk fails in your RAID group during the lengthy restore, then data loss will occur. Also, as all processing capacity is used for the restore, users will experience severe performance drops as data is being written to the replacement disk. Large systems with hundreds of TBs or Petabytes, will routinely be in constant rebuild mode, as drives routinely fail. In an effort to shorten these longer rebuild cycles, RAID systems ship with faster processors, which also consume more energy, but this only masks the problem at best. The simplest way to protect data is to make several copies, replication. A popular concept is called three copies in the cloud, promoted mostly by public cloud platforms like Amazon S3 and Rackspace. While three copies in the cloud provides acceptable data protection, it is also very lucrative for the cloud provider as they are in the business of selling more storage capacity. Swift, the object storage component of Rackspace s open source cloud infrastructure also uses pure replication. A more efficient data protection mechanism is erasure coding. Today, there are several flavors of erasure coding, each one with its own benefits. Erasure coding s key advantage is that you can break up your data into n fragments, add m additional fragments, store the fragments across n+m devices, and then recover the original data from any n of the devices. Survive 4 failures? 10? Pick a number! Also when a disk is lost, the system only has to calculate new fragments, to be spread over any selection of disks with available capacity. This is a lot faster and more efficient than restoring an entire RAID-based disk, even if it was only 20% full. Erasure coding can be implemented locally or distributed, which means that fragments are spread over multiple data centers (at least three) and the system can survive failure of a full datacenter. Distributed erasure coding drastically reduces the overhead (the extra storage that is needed to protect the data). Five 9 s or more are guaranteed with overhead numbers as low as 20% - as opposed to 3 copies requiring 200% overhead. The problem with distributed erasure coding is that it creates a huge WAN cost, as rebuilds require data transfer between the data centers. Also, the availability highly depends on the WAN connectivity. Each data read from a distributed erasure coding pool, requires data to be read from three data centers. If any of these connections have high latency, the user will notice the delayed response. An interesting, best of both worlds solution is local erasure coding with replication. Such architecture combines the benefits of erasure coding with those of replication (a full copy is present in each datacenter). While such a setup requires more overhead than the distributed alternative, the TCO is typically a lot lower due to the reduced WAN traffic. 12
WEB OBJECT SCALER DDN s legacy is designing high-performance storage systems, but without making things more complex than they need to be. WOS is the perfect example of achieving operational excellence through reverse engineering - stripping the architecture down to the very basics. The architecture of WOS consists of three components: WOS building blocks, WOS Core software and a choice of simple interfaces. The backend of a WOS storage infrastructure are the WOS storage nodes. The storage nodes are essentially 4U servers filled with 60 SATA disks. A WOS infrastructure can contain as few as 3 nodes and scales to virtually unlimited capacity by adding more nodes. Smart storage requires intelligent software. WOS Core has a single, straightforward management console for the entire infrastructure - even when distributed across multiple sites. WOS Core s self-healing capabilities and other features drastically reduce operator-driven maintenance interventions. WOS provides the most complete choice of interfaces, including a set of native API s, file access interfaces and S3 REST. TRUE OBJECT STORAGE PLATFORM Most object storage platforms still have a POSIX file system layer on the disk level. WOS, however, was designed as a true object storage platform, a flat, single layer, address structure where objects are stored in a contiguous group of blocks so that disk operations are minimized (singledisk-operation reads dual-operation writes), performance is maximized and disks are used at full capacity. OPTIMIZED FOR SMALL AND LARGE FILES WOS is the only object storage system that is optimized for high-speed throughput of large data volumes and super-fast I/O operations for small files. For multi-site deployments, the built-in WOS Latency-Aware Access Manager will automatically address data access requests to the location with the lowest latency. WOS IS THE ONLY OBJECT STORAGE PLATFORM THAT SEAMLESSLY INTEGRATES WITH OTHER STORAGE TIERS. CHOICE OF DATA PROTECTION SCHEMES WOS offers a choice of data protection mechanisms to ensure the highest data durability AND availability. Reduce your storage overhead while maximizing durability for single site deployments with local ObjectAssure, DDN s implementation of Erasure Coding. Alternatively, you can choose Replicated ObjectAssure, to improve availability without increasing WAN costs. Finally, ObjectAssure can be implemented in a distributed way to ensure higher durability, at a lower cost. SELF-HEALING ARCHITECTURE Keeping traditional storage infrastructures healthy is management-intensive. Disks need to be replaced and restored. Rebuild windows need to be kept to a minimum to avoid data loss and preserve application performance. This is not the case with WOS. The built-in data protection algorithm, ObjectAssure has unique self-healing capabilities that further reduce the management effort. Also, in case of a broken disk, ObjectAssure only has to reconstruct the actual data that was lost - as opposed to the entire disk. This dramatically reduces the rebuild window. SINGLE STORAGE INFRASTRUCTURE WOS is the only object storage platform that seamlessly integrates with other storage tiers. It has one management interface for the entire infrastructure and supports easy data movement between different tiers, e.g. from GPFS to WOS and back, or from and to Lustre environments. 13
UNLIKE COMMODITY HARDWARE, WOS WAS DESIGNED TO REDUCE TCO. WOS SCALES IN CLUSTERS AS LARGE AS 30PB WIDEST SELECTION OF INTERFACES; OUT OF THE BOX APPLICATIONS WOS provides the most complete choice of interfaces, including a set of native API s, file access interfaces and S3 REST. In addition, WOS can be configured with preinstalled applications such as irods for data management or WOS Share for secure global file sharing. ENTERPRISE-GRADE PLATFORM Most vendors recommend commodity hardware for their object storage platforms. In the short term, this could mean initial CAPEX savings, but as such devices typically have shorter replacement cycles, this highly impacts the OPEX further down the road. This is especially so for multi petabyte deployments. While WOS was designed to be hardware agnostic, we designed the WOS 7000 hardware to reduce TCO. Unlike commodity hardware, the WOS 7000 has an ultra dense form factor, so there are fewer systems to house, manage, power, cool and maintain. Leveraging over 15 years of hardware design for the most demanding HPC environments, WOS 7000 was built to run many more years than cheaper commodity hardware. WOS BENEFITS Lowest Global Access Latency WOS was designed with the intent of maximizing performance for storage of massive volumes of immutable data. Scales with All Varieties of Applications WOS scales virtually unlimited in clusters as large as 30PB. Those clusters can consist of any mix of small (kilobytes) or large (terabytes) files. Best Durability & System Availability WOS choice of data protection schemes allows the customer to deploy object storage that combines durability with availability. Lowest Administration Overhead, Lowest TCO Through automated management, lower hardware costs, less power usage, simple architecture, optimized disk usage and reduced WAN bandwidth usage; WOS enables organizations to store more data at a much lower cost. Simple Integration Integrate WOS with your GRIDScaler GPFS storage or your EXAScaler Lustre platform. Use WOS as an archive for your HScaler Big Data Storage, or build an Active Archive of WOS with a tape library for offline cold archiving. Maximum Portability WOS features the most complete set of interfaces to facilitate your application integration, including C++ and Java APIs for direct application integration, REST for web applications (S3 or not) and file gateways to support file-based workflows. Best Data Center Density Designed for massive HPC deployments, WOS 7000 provides the highest data center density possible. ECOSYSTEM Object Storage is clearly the hot space in the storage industry, with offerings from both startups and established storage solution providers. But, there is more than just object storage on the market: object storage has fostered a wave of innovation that enables or leverages the paradigm. The list of Tier 2 object storage players can be endless, especially when including the application providers. Here is a short selection of popular gateways, WAN optimizers, collaboration platforms and other applications. This should help to provide a better understanding of the object storage ecosystem and the opportunities and use cases. 14
THERE ARE MANY DIFFERENT FLAVORS OF OBJECT STORAGE ON THE MARKET. Ctera CTERA leverages object storage to offer a range of solutions for SMBs, enterprise branch offices and remote users, including: data backup and recovery, file-based collaboration and mobile access. Mezeo also provides a number of storage solutions that leverage object storage, including: an AWS compatible REST API and a number of file sync and share clients that give users access and collaboration capabilities from their PC/Mac, smartphone, tablet or browser interface. Panzura built a NAS gateway for storage clouds. The gateway enables enterprises to combine multiple storage (cloud) resources and make them accessible to multiple locations, presented as a unified global file system. Aspera both leverages and facilitates object storage. On the one hand, they have a number of applications for collaboration, distribution etc., but the core of their technology is a protocol that optimizes how data is sent from the object storage pool, over the WAN to a user application - or between sites, if an object storage infrastructure is distributed over multiple locations. Bitspeed and Silverpeak are active in the same space: WAN optimization, which enables faster, more reliable and] secure data transfer between storage sites - or between the object storage pool and the application. These technologies are becoming increasingly important in the deployment of object storage based storage clouds. Dropbox is probably the best-know object storage success case. This early AWS S3 customer launched a file-sharing application when no one even knew they needed one. The power of Dropbox lies in their use of deduplication (when multiple users store the same file in their Dropbox, only one copy is kept). This way, Dropbox saves a lot on storage costs. Deduplication is not new, but Dropbox pioneered its use in an online, object storage based application. This also allowed them to quickly gain a large user base through a fermium model, which would have been unaffordable otherwise. Box(.net) also started as an online file sharing application but with some very important differences. Box runs on their own (object storage) infrastructure, which gave them more control over security, data integrity etc. (as compared to using S3). This allowed them to bring their solution to the SMB and Enterprise markets. Today, Box.net grew to what can probably best be described as a storage-centric Platform as a Service, enabling organizations to customize apps, integrate with their own applications etc. Netflix, which launched as a DVD rental by mail is an early adopter of object storage: in 2007 it launched a movie streaming service which would disrupt the market. Well before Apple added movies and tv shows to their store, Netflix leveraged S3 to offer movies in an online format. Apple, Google and Facebook also have massive object storage deployments, but little is known about their architectures. Apple and Google are going after the S3 end users with document sharing and other storage in the cloud services. With this, they compete both with Amazon and the applications that use S3 such as Dropbox and Evernote. 15
RESOURCES http://knowledgelayer.softlayer.com/learning/introduction-object-storage http://docs.openstack.org/trunk/openstack-object-storage/admin/content/ch_introduction-toopenstack-object-storage.html http://cloudarchitect.att.com/articles/introduction-object-based-storage http://www.conres.com/hitachi-hds-object-storage-content-platform http://www.cs.cmu.edu/~garth/raidpaper/patterson88.pdf http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 https://en.wikipedia.org/wiki/representational_state_transfer#restful_web_apis FOR MORE INFORMATION VISIT: http://www.ddn.com/products/object-storage-web-object-scaler-wos/ ABOUT DDN DataDirect Networks (DDN) is the world s leading big data storage supplier to data-intensive, global organizations. For more than 15 years, DDN has designed, developed, deployed and optimized systems, software and solutions that enable enterprises, service providers, universities and government agencies to generate more value and to accelerate time to insight from their data and information, on premise and in the cloud. Organizations leverage the power of DDN technology and the deep technical expertise of its team to capture, store, process, analyze, collaborate and distribute data, information and content at largest scale in the most efficient, reliable and cost effective manner. DDN customers include many of the world s leading financial services firms and banks, healthcare and life science organizations, manufacturing and energy companies, government and research facilities, and web and cloud service providers. For more information, visit our website www.ddn.com or call 1-800-837-2298. SALES@DDN.COM +1.800.837.2298 2015 DataDirect Networks, Inc. All Rights Reserved. DataDirect Networks, EXAScaler, GRIDScaler, ReACT, SFA12K, SFA, SFX, Storage Fusion Xceleration, Web Object Storage, WOS are trademarks of DataDirect Networks. Other Names and Brands May Be Claimed as the Property of Others. v2 (5/15)