Object Storage A Dell Point of View

Transcription

1 Object Storage A Dell Point of View Dell Product Group 1

2 THIS POINT OF VIEW PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell, the DELL logo, and the DELL badge are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own. December

3 Object Storage Introduction A changing data landscape There is a paradigm shift occurring in digital data generation. It used to be companies generated relatively small amounts of structured digital data (for example, to keep track of customer, financial or supply chain information). This data was usually generated - and consumed - by a relative few in the company, e.g. order entry clerks, or supply chain managers. Today, however: 1. Companies also generate large amounts of unstructured data due to Web 2.0 applications, the universal use of , instant messaging, office productivity applications, as well as digital cameras and mobile phones 2. An increasing number of business processes that used to be paper based e.g. filing an expense report -- are moving online 3. Much of team collaboration now occurs electronically -- preparing for a presentation, for example, a team will often share ideas over and collaborate on the same Microsoft PowerPoint document 4. Digital data is now used by everyone in a company. It is often the primary way in which they get information via, for example, Internet and Intranet web pages, podcasts, video and.pdf documents The result of this paradigm shift is a changing digital landscape with: A significant acceleration in data growth. It is estimated that the amount of digital information will grow to an astounding 35 zettabytes (Zettabyte = 1 trillion gigabytes) over the next 10 years. 1 Increasing storage of a company s business information -- including its intellectual property -- in unstructured data files such as and Microsoft Word, Excel and PowerPoint documents. It is generally accepted that 70% - 90% of new data generated is unstructured in nature A large increase in fixed digital content, e.g. photographic images, that once created, remains unchanged thereafter. This content is usually considered read only, and there is no need for write access Data that is active as teams collaborate on it, but becomes inactive once this collaboration ends Read only data stored in Internet and Intranet production systems for long periods of time The new challenges in storing digital data For storage managers, the changing data landscape is creating a new set of challenges. These include: Developing effective cost management strategies given the new data growth rates and the fact that IT budgets are only growing at a small fraction of the data rate Minimizing the risk of data loss given the amount of data being generated. This is important to meet regulatory requirements, but it is also important to preserve a company s intellectual property. Understanding how to effectively manage the different data types, i.e. structured, unstructured, active, inactive and fixed so costs are minimized and data is stored on the right tiers at all times Making data easily searchable and ediscoverable, both to find business information and to meet the information requirements of a compliance audit or a lawsuit These challenges are difficult to meet just given the volume of data being generated, but are particularly so now that most of this data is unstructured, and has properties that make it difficult to manage. 3

4 The Additional Challenges of Unstructured Data Unstructured data has two properties that make it hard for storage managers to determine the information value and relative importance of each file, and therefore how each file should be managed: First, it is hard to understand the context of unstructured data. Unstructured data files are often created for a particular event (e.g. a presentation) and, with the passage of time, can lose its association to that event and therefore its context. Also, an unstructured data file may be part of a collection of files generated for a particular event e.g. a set of files generated when diagnosing a patient (doctor s notes, lab results, x-ray images, etc). If a file, say an x-ray, needs to be managed without the context of this collection, it can be hard to understand its importance and how it should be managed Second, it is difficult to understand the content within an unstructured data file, and therefore its relevance. For example, a storage manager may come across two video.wav files. One of these may be a humorous video being ed around by colleagues; the other may be a product demo video. Clearly, the storage manager needs to store the product demo and not the humorous video; however understanding which is which can be hard to know without viewing the videos Why it is difficult to meet these new challenges with traditional file storage approaches The current dominant approach to file storage is Network Attached Storage (NAS). NAS was invented approximately 30 years ago with the purpose of consolidating file storage across several servers. However, NAS has three characteristics that make it less suitable for many of today s storage challenges: 1. NAS is designed to allow concurrent read & write access among users. Consequently, NAS includes capabilities for managing permissions and operations such as file locking. This is a useful feature for active data. However, it is less useful for fixed, inactive, and read only static web page content. In these cases, NAS simply adds unnecessary cost and complexity. 2. NAS stores data in hierarchical structures ( trees ) consisting of directories, folders, subfolders and files. This means as the amount of data grows, so do the number of hierarchical structures. Given the amount of data being generated today, the consequence is that design limits can soon be reached, the number of folders can become unwieldy, backups can be difficult and performance can degrade. Additionally, finding a file requires knowing the exact directory structure; a task that becomes increasingly difficult as the amount of data grows. 3. NAS file systems were designed to manage the location of data according to a sequence of nested folders. This means that only basic metadata such as file name, creation date, last modification date, and the file type is necessary to identify the file. The context and content of the file data is not required. The implication is that as files get stored in NAS, they do not carry the context and content information storage managers need to efficiently manage data in their organization. They cannot, for example efficiently tier data, or know whether a file is important to back up or maintain for compliance reasons. Instead they may overcompensate with blanket policies that apply across file types (e.g. all JPEG files, even if they are of photo images from birth announcements, must be stored indefinitely). 4

5 How can Dell help? Dell has developed a customer centric strategy and a set of solutions and services designed to help customers meet the new challenges created by the amount and type of data being generated today. This strategy is based on Dell s Point of View (POV) with respect to Object Storage: Dell s Object Storage POV Companies are confronting an explosion in data due to Web 2.0, the pervasive use of applications such as , instant messaging, Microsoft Office applications, the increasing use of devices such as digital cameras and MP3 players, as well as the increased digitization of business processes. This tremendous growth in data is creating cost and data protection challenges for storage managers, particularly since IT budgets are increasing at only a fraction of the data growth rate. Compounding the problem is that most of the data being generated today is unstructured. This makes it difficult for storage managers to understand the context, content and therefore the informational value of the data. Without this information, it is difficult for storage managers to: o Effectively align storage to the value of the data o Apply the right policies so that business, compliance, and protection requirements can be met o Make data easily accessible so the intellectual property contained within can be leveraged o Ensure data is ediscoverable for litigation or audit purposes Dell believes Object Storage is an important approach to managing the explosion in data confronting companies today. In particular, Object Storage: o o Stores data in a flat address space, enabling data to easily scale to match data growth Enables the attachment of rich metadata to each data file, allowing context and content to be captured, and giving storage managers the information they need to efficiently manage unstructured data o Assigns a unique ID to each object eliminating the need to know the physical location of where a data object is stored Dell believes that traditional file storage approaches such as NAS are less suited to meeting the challenges faced by storage managers today. This is because NAS: o Primarily was designed to manage the location of files within a hierarchy of directories and folders; as such it is not designed to preserve the context of data files. o Becomes unwieldy as the amount of data increases and limits start to be reached on the number of hierarchical structures and folders that can be created However, Dell believes that Object Storage and NAS both have strong roles to play in an organization. In particular, Dell believes NAS is a preferred approach to storing active or frequently accessed data that requires concurrent read/write access among users and high performance. Object Storage, on the other hand, is well suited for storing semi-active, static and inactive data as a secondary file storage repository as a complement to NAS storage. Given its unique properties, Dell believes that Object Storage is a strong platform for several important storage applications, including: o Archiving: Object Storage provides a highly cost effective, scalable disk based approach to archiving that easily enables important requirements such as ediscovery, searchability and the ability to enable compliance through retention policies, data integrity, protection and authenticity checks o Cloud Storage: Object Storage provides several capabilities that make it ideal for cloud storage, including high scalability, cost effectiveness and the use of HTTP as the native communication protocol Dell believes that Object Storage is an integral part of an Intelligent Data Management strategy for storage management, and is particularly useful in enabling storage managers to automatically route 5

6 data especially unstructured data based on policies to the right storage site(s) with the right amount of protection so that storage efficiencies are maximized Dell s believes, however, that customer needs will also be specific to their industry. Consequently, Dell s Object Storage strategy is to provide an open platform based on industry standards and to build a strong ecosystem of partners that can provide effective storage management capabilities through cohesive vertical solutions. In summary, Dell believes Object Storage can help customers: o Meet the new challenges storage managers face given the extraordinary explosion in digital content today, the type of data being generated, and a regulatory environment that requires storing data for longer periods of time o Reduce storage costs by aligning storage with the value of data and by increasing storage management efficiency o Scale storage cost effectively from terabytes to petabytes with minimal downtime o Provide investment protection, allowing for upgrades without costly data migrations o Archive data and ensure ediscovery, searchability, compliance and protection o Build a platform for cloud storage, especially for the storage and distribution of unstructured content such as images, videos and audio files A mini tutorial on Object Storage Object storage is a new file storage approach with properties that make it well suited to today s storage management challenges: 1. Object Storage combines data with rich metadata to create an object. This enables both content and context metadata to be attached to the data and therefore to carry information about the information. To illustrate the difference between Object Storage and File Storage, consider the example of an MRI scan stored as a file versus as an Object, as shown in Figure 1. Figure 1: Example contrasting the amount of metadata associated with an Object vs. a File 6

7 When the MRI is stored as a file in a NAS system, the metadata is basic and may include information such as file name, creation data, creator, and file type. When stored as an Object, on the other hand, the MRI scan can have the basic metadata plus additional metadata such as the patient s name, the patient s ID, the procedure date, the attending physician s name, as well as provide pointers to files that contain the physician s notes. In this way, objects include both content as well as context information 2. Object Storage stores data as in a flat address space. There are no hierarchies, or nested directories. As a result, there are no limits on the number of files that can be stored and storage capacity can be easily scaled from terabytes to petabytes 3. Each object in Object Storage is identified by a unique ID. This ID is generated using a hash function and guarantees that every object is uniquely identified. It also serves as a unique pointer to an object similar to the way URLs point to unique files in the Internet. Consequently, files can be retrieved without knowing their specific location or having to specify a directory path 4. In contrast to NAS, Object Storage does not have the overhead required to manage concurrent read/writes, file locks and permissions. This reduces the performance cost of storing data, where read / write permissions and file locking is not a requirement 5. Object Storage uses an HTTP communication as its standard interface. This makes it ideal for communicating data transfers with storage locations via an intranet and the Internet a prerequisite for cloud storage. Benefits of Object Storage Object Storage gives storage managers the tools they need to meet today s data storage challenges: 1. Align storage with data By allowing storage managers to understand a file s context and content and therefore its information value, Object Storage helps ensure data can be stored on the right storage tier, with the right protection levels, or deleted if the data has no value to the company. 2. Automate and simplify storage management Reading the metadata attached to each object, storage managers can automatically apply policies for storage management according to the context and content of the file. Further Object Storage s flat address space negates the need for managing LUNs and RAID groups, as happens with NAS. 3. Keep related information together by enabling multiple file types to be grouped together. Storage managers can add file pointers to associated files in the metadata attached to each object. This allows files (including those of different file types) to be grouped together to provide a complete picture in much the same way as information might be collected in a single file associated with a person or event in the paper based world. 4. Increase the business intelligence available from stored data Storage managers can apply business intelligence tools to object metadata to provide insights based on the data contained in the files. 5. Increase searchability & ediscovery of data 7

8 Object Storage enables storage managers to more easily search for data without knowing specific filenames, dates or traditional file designations. This benefit is especially useful in archiving applications, where data is often untouched for significant amounts of time, but must be easily discoverable when required. 6. Intelligent replication Instead of RAID systems for backup protection, storage managers can leverage Object Storage to easily create multiple replicas, as necessary, with each replica identified with the same object storage ID. These replicas can further be distributed geographically, increasing the protection provided by replication. 7. Ensure data integrity compliance, and auditability A hash function serves as a powerful mechanism for compliance and auditability. This is because the resulting digest of a hash function, its digital signature, will change if the data file is changed; consequently an unchanged signature provides proof of data integrity as well as its authenticity. This makes Object Storage useful for protecting archived data, meeting regulatory requirements, and use for evidentiary purposes particularly for data that has high legal or compliance risk. 8. Enable easy data migrations, including across hardware upgrades Leveraging each object s unique ID, storage managers can easily migrate files from one storage system to another as required by, say, a hardware upgrade. This is because all that is required to retrieve an object is its ID; its particular hardware location is not relevant. This makes it easy for storage managers to future proof their data as well as manage transitions in technology. 9. Enable self healing By monitoring the state of each signature, an Object Storage system can tell if a file has been corrupted through a change in its signature. With this knowledge, the system can self heal and replace any corrupted files. As a clustered architecture it also provides automated high-availability due to a drive/node failure and automated recovery to ensure data protection (replica) policies are maintained. 10. Enable flexible scalability Through its use of a flat address space and scalable cluster architecture, Object Storage enables flexible scalability in terms of capacity as well as numbers of objects. That is seamless scale from a few terabytes to several petabytes and millions to billions of individual file objects. This benefit is particularly valuable in cases where the exact storage requirements e.g. in Cloud Storage are unknown. 11. Cost effectively manage storage growth One of the key benefits of Object Storage is to help storage managers cost effectively store large amounts of data. This is accomplished through several mechanisms: Efficient data tiering to ensure that data is not incorrectly stored on more expensive storage tiers Automation of storage management processes through the ability to apply policies using object meta- data Lower costs relative to NAS since Object Storage does not require as much administrative effort to manage storage at scale 8

9 Applications of Object Storage Object Storage s properties make it an ideal platform for several important storage applications, including: 1. Archiving Archiving the long term preservation of data - is often mission critical for companies given compliance or regulatory requirements. It has also become seen as delivering additional value as corporate information assets in an archive can be reused or re-purposed for current benefit and/or to enable new revenue opportunities. However, it is also difficult: it requires storage managers to (a) identify data to be archived vs. deleted (b) cost effectively store this data (c) ensure its integrity and (d) make it easily searchable. Object Storage provides an ideal Archiving platform. Leveraging Object Storage, storage managers can: Read each object s metadata to easily identify which data files need to be archived vs. deleted. Take advantage of Object Storage s cost advantages to provide a low cost archiving infrastructure. Track each unique object signature and show it has not changed to provide proof of data integrity and authenticity. Leverage the context and content information available in metadata to make data available, discoverable and accessible well into the future 2. Cloud Storage To meet the needs of cloud service providers, cloud storage has to be easily and cost effectively scalable, with continued ease of management as storage capacity increases. Object Storage given its properties - is an ideal cloud storage platform as it provides cost efficient, non disruptive and flexible scaling to petabytes of data, as well as enables intelligent data replication across geographically distributed sites Object Storage s role in Intelligent Data Management One of the primary benefits of Object Storage is the role it can play in Dell s Intelligent Data Management (IDM) strategy. In IDM, data is routed to the right storage tier and protection levels according to its stage in the data lifecycle. Through this approach, the most expensive high performance systems are reserved for frequently accessed primary data while infrequently accessed secondary data is routed to less expensive storage arrays. This ensures costs are minimized. Object Storage, with its rich metadata, can help automate IDM for unstructured data by automatically applying policies for routing data to the right tiers. In this way, Object Storage provides a powerful tool for IDM and increasing storage efficiency. Object Storage in relation to traditional NAS Object Storage is not a one-sized fits all solution. Instead, it is Dell s point of view that both Object Storage and traditional NAS have a strong role to play in a company s storage infrastructure. Object Storage works best when large numbers (millions or billions) of similar objects need to be stored as well as when data is relatively static and not frequently accessed. This isn t a restrictive criterion. By some estimates, 90% of data remains untouched after it is created, leaving 10% as active data. 2 For this data, I/O performance as well as concurrent access and read/write access protection is likely important, making NAS the better choice. Consequently, as shown in Figure 2, the frequency of data usage is a driver for multiple storage system types within a storage environment and demonstrates that 9

10 both Object Storage and traditional block based file storage systems such as NAS have strong roles to play within an organization. Figure 2: The frequency of data usage is a factor in using Object vs. traditional file storage Dell s Object Storage Solution Offerings 1. Services: For customers who can benefit from technical expertise to determine how best to leverage Object Storage within their storage environment, Dell has developed a rich set of consulting services: Expert advice for crafting a customized IDM and Object Storage strategy Design services to develop and implement smart, policy-driven processes that can help classify and prioritize data based on its value to the customer Specific solution recommendations, including partner integrations 2. The Dell DX Object Storage Platform The Dell DX Object Storage Platform is a comprehensive, integrated hardware and software solution designed to optimize the management, storage and preservation of unstructured file data. The solution is object-based, metadata aware, and is ideally suited to meeting today s data storage changes. In addition, the DX solution has several key features that make it a leading Object Storage platform in the market: Standards based x86 hardware to further help reduce the cost of storage End to end solution with fully integrated hardware and software Seamless upgrades and a modular design to easily add, replace or retire nodes across multiple hardware generations without any manual migration requirements Power optimization with features such as drive spin down to meet cost priorities Easy partner integration via a native API and a simple HTTP interface to an ever growing and comprehensive ecosystem of independent software vendors (ISVs) who can maximize the investment in Object Storage. Dell s ecosystem includes partners in areas such as: o Medical archiving 10

11 o File, and Sharepoint archiving o ediscovery and enterprise content management The complete infrastructure needed to support Cloud Storage In summary, the Dell DX Object Storage Platform is ideal for not only solving today s storage challenges, but also ideal for the next frontier Cloud Storage in storage management. Footnotes 1. IDC IVIEW, Sponsored by EMC, The Digital Universe Decade Are You Ready? Doc.#IDC_925, May Measurement and Analysis of Large-Scale Network File System Workloads, UC Santa Cruz. 11