Meeting Unstructured Data Storage Requirements Across the Data Lifecycle with Scale-Out Storage

Transcription

1 White Paper Quantum StorNext Meeting Unstructured Data Storage Requirements Across the Data Lifecycle with Scale-Out Storage By Terri McClure December, 2010 This ESG White Paper was commissioned by Quantum and is distributed under license from ESG. 2010, Enterprise Strategy Group, Inc. All Rights Reserved

2 White Paper: Quantum StorNext 2 Contents Torrents of Data Wash over IT... 3 No More Business as Usual... 4 Scale-out Storage... 4 Building a Shared Storage Backbone... 4 Scale-out Economics... 4 Quantum StorNext... 6 Scaling... 7 Scale with Flexibility... 7 Scale for Capacity... 8 Scale with Efficiency... 8 Scale Securely... 8 Driving Efficiency... 9 The Bigger Truth... 9 All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of the Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at (508)

3 White Paper: Quantum StorNext 3 Torrents of Data Wash over IT Data growth continues to be IT s constant companion. Industry analysts begin to sound like broken records data, particularly file-based or unstructured data, continues to grow not only exponentially, but at an accelerating rate. We harp on this point for one reason: it is a source of constant pain for IT as every endpoint device becomes a tool for capturing and creating content. It s not that there aren t solutions out there; it s that nothing is really capable of stemming the tide. Web 2.0 applications boast thousands of concurrent users and ever larger files are being created for high-performance computing, medical imaging (with high definition and 4-D files reaching hundreds of MB), oil and gas exploration (relying on 3-D models that can reach 100 TB), and digital media (a two-hour highdefinition video consumes multiple TBs). This torrent of data threatens to drown those organizations that don t find a way to keep up. Recent ESG research confirms this point. A recent survey of digital archiving users took a look at current and expected worldwide digital archiving capacity (see Figure 1). 1 File-based data far outweighed database and data, making up 75% of the worldwide digital archive in That trend continues for the foreseeable future. Imagine the year 2015, when file data consumes more than 200,000 petabytes of capacity how will we manage it? Figure 1. Growth of File-based Data, ,000 Total Worldwide Digital Archive Capacity, by Content Type, (Petabytes) 300, , , , ,000 50, File 25,127 39,237 59,600 92, , ,716 Database 4,065 6,179 9,140 13,824 21,532 32,188 4,025 6,575 10,411 16,796 27,817 44,091 Source: Enterprise Strategy Group, This torrent of data means higher costs for storage capacity, data center space, power and cooling, and the staff to manage it all. Spreadsheet management systems are incapable of tracking so much data. Throwing capacity at each problem results in standalone storage silos, which are costly and result in low utilization rates (often less than 50% of capacity) because they are unable to share available storage with other silos in need. Migrating aging data to archival storage (such as less expensive, higher capacity drives) is helpful, but IT is rarely able to keep up with the job because of sheer volume. Plus, this ceaseless challenge is complicated by the remnants of a global recession and the need to reduce costs in every possible way. IT is stuck in the middle, being asked by application owners to accommodate continual data growth and meet SLAs while simultaneously reducing operational costs to meet budget constraints. 1 Source: ESG Research Report, Digital Archive Market Forecast , July 2010.

4 White Paper: Quantum StorNext 4 No More Business as Usual IT organizations cannot continue trying to deal with massive data growth by addressing problems as they arise. For many, relief only comes from overhauling the storage paradigm and working to meet the demands of the business with a different type of storage infrastructure. Business today needs a virtual, scalable, shared pool of storage that enables resource allocation where and when it is needed. To provide this kind of support, a storage pool should be: Virtual, to mask complexity and physical infrastructure management Scalable, to transparently increase capacity and/or performance as needed Shared, to optimize utilization no matter where capacity is needed Policy-driven, to minimize management effort and operate efficiently Adaptable, to meet changing storage requirements and deliver efficient long-term archiving Scale-out Storage Building a Shared Storage Backbone Scale-out storage can help build this new paradigm, creating a storage backbone that can handle the tremendous challenges of unstructured data growth. Scale-out storage systems, which began in the NAS space, can independently scale both throughput and capacity by adding nodes that work in parallel and are managed as one. These systems can scale to multiple petabytes under a single system image, making them an ideal consolidation platform. Scale-out storage pools virtualize the underlying storage, creating a resource that can adapt as business needs change; bandwidth, processing power, and storage capacity can be independently tuned and scaled on the fly. This resource abstraction is critical to enabling today s nonstop enterprise infrastructure with its higher availability and reliability. Scale-out storage minimizes management costs and data center floor space, power, and cooling needs. The shared resource pool delivers much higher utilization rates to reduce waste. Scale-out Economics Based on ESG research, there are many reasons to deploy scale-out storage. When asked about the concerns that lead to scale-out NAS deployments (or consideration thereof), 57% of survey respondents mentioned faster provisioning times. 2 In addition, 51% cited scalability, easier management, and improved data availability, and close to 50% mentioned performance improvements (see Figure 2). 2 Source: ESG 2008 Enterprise Storage Systems Survey, November 2008.

5 White Paper: Quantum StorNext 5 Figure 2. Scale-out Market Drivers Considerations that led to the deployment (current users) or consideration (planned or interested users) of scale-out NAS solutions. (Percent of respondents, multiple responses accepted) Faster storage provisioning times Improved scalability Easier to manage Improved data availability Improved performance (I/Os) Improved performance (throughput) Need to support specific applications Lower cost of infrastructure 21% 27% 38% 51% 46% 51% 42% 51% 42% 46% 41% 43% 42% 35% 42% 57% Current users (N=37) Planned or potential users N=253) 0% 10% 20% 30% 40% 50% 60% Source: Enterprise Strategy Group, These market drivers make sense as the economics of scale-out storage tell a compelling story of improved scalability, faster provisioning, better performance, and easier management. It starts with: Just-in-time scalability. The modular nature of scale-out environments enables organizations to begin slowly and add capacity and/or performance as needed. They can start with as few as two nodes and scale just-in-time avoiding a large capital investment based on expectations by instead purchasing based on actual need. This eliminates the waste of buying, powering, and cooling empty frames in mostly empty cabinets, waiting for the day when resources are required. By scaling when needed, users can also ride the commodity curve to take advantage of declining prices of disk capacity and processors. Easy, fast provisioning. Adding capacity is simple and done on the fly. A sudden burst in application activity that requires additional storage can be accommodated quickly and easily, without waiting weeks for the procurement, provisioning, and installation processes to be completed. The economic benefits? Reducing costly downtime and getting to revenue-generating operations faster. Manageability at scale. Many low-level storage management tasks are necessary when you grow, such as expanding file systems and load balancing performance to accommodate additional capacity. In scale-out systems, these tasks are automated, helping to keep management costs low. Also, since scale-out systems can scale to multiple petabytes while still being managed as a single system, the management penalty that comes from growth in monolithic, scale-up environments is avoided. It doesn t matter how many nodes make up the storage pool it s still managed as a single unit so no additional headcount should be needed. Increased utilization. Scale-out systems address all usable capacity in the storage pool even as it grows, so there is no unused capacity locked behind individual processors, unable to be used by applications in need. As a result, utilization rates often go from 30% to more than 60% in scale-out environments. As they evolved, scale-out systems developed the ability to create multiple shared storage pools with different price, performance, and protection profiles. For example, a pool using 15K RPM drives configured with RAID 10 offers fast performance with mirrored writes striped across disks for protection. This level of performance and protection is more expensive, but can be applied only for those data sets that require it. A less costly pool may offer

6 White Paper: Quantum StorNext 6 slower disks and parity-only protection for other applications. The benefit is that organizations can divide up shared storage pools into performance and protection profiles and match costs to needs. So how can an organization stop business as usual and instead start building a shared storage backbone that leverages scale-out economics? Quantum StorNext can facilitate that transition. Quantum StorNext Finding a solution to data growth difficulties is uppermost in the minds of IT managers. Proof of this point is demonstrated in ESG research: when asked about their biggest challenges in scaling the storage environments to support applications, survey respondents said their two scaling challenges for primary storage were keeping pace with overall data growth and storage system costs (see Figure 3). 3 Figure 3. Top Challenges of Scaling Storage for Applications In general, what are your organization s greatest challenges with respect to its storage environment? (Percent of respondents, N=504, multiple responses accepted) Keeping pace with overall data growth Storage system costs 36% 35% Securing confidential data Need to improve backup and recovery processes Power and cooling costs General increase in complexity of IT environment, number of servers, etc. Running out of physical space Storage requirements/complexity related to specific applications Insufficient capital budget Rapid growth in unstructured, file-based content Supporting storage and data protection requirements at remote/branch offices Meeting regulatory compliance requirements General complexity of managing external storage systems Lack of qualified IT staff and/or skills Poor storage utilization Poor performance of existing systems Meeting e-discovery demands (i.e., litigation support) Poor search tools/general difficulty finding relevant content across the enterprise Poor service and support from vendor(s) 31% 31% 28% 28% 26% 25% 24% 22% 22% 21% 18% 16% 16% 14% 12% 10% 9% 0% 5% 10% 15% 20% 25% 30% 35% 40% Source: Enterprise Strategy Group, Source: ESG Research, ESG 2008 Enterprise Storage Systems Survey, November 2008.

7 White Paper: Quantum StorNext 7 Quantum StorNext can change the storage paradigm by delivering a next-generation file system designed specifically for scale and efficiency. It offers high-performance file management, massive scalability, long-term online archiving, global protection and distribution, and deduplication. Because it is a virtual platform, it can address whatever price/performance/protection profiles the business needs, even as they change over time. As data sets age and application use ebbs and flows, IT can create new storage pools with different profiles and shift data between pools. This design delivers the highest cost efficiency over the data lifecycle. Scaling First, the basics: the StorNext platform easily scales as businesses grow to help meet the enormous challenge of data growth. Users can add storage nodes and processors transparently and non-disruptively to the shared storage pool when needed; there are no silos or array limitations requiring volume placement and management. Massive scalability can be handled with StorNext and organizations can provide fast access to millions of shared files and petabytes of storage. The virtualized storage pools offer a consolidated data repository across vendor platforms, so organizations can leverage both new technologies and existing investments. StorNext operates on a wide variety of storage devices including high speed disk arrays, value disk arrays, tape libraries, and network attached storage so organizations can deploy customized storage tiers. Scale with Flexibility Organizations are typically forced to choose between the simplicity of shared storage (using NAS) and the high performance scalability of a SAN. StorNext delivers the best of both of these shared storage types, enabling users to create very large file systems with hundreds of millions of files that are accessible to thousands of users on industry-standard servers. Providing direct, native access to storage over a SAN or LAN, StorNext rapidly stores information and simultaneously shares it across multiple operating systems using a single, shared file system instance. While files remain on SAN storage, they can be presented to various applications running Linux, Windows, UNIX, and Mac OS servers simultaneously. This saves time and accelerates business processes because files do not need to be physically moved or converted in order to accommodate various filesystems. SAN and LAN access methods can be used simultaneously on the same file system to support different workflow types: workflows requiring high throughput by few content creators (SAN clients) and workflows requiring scalable access by many users (distributed LAN clients). StorNext SAN File System With the StorNext SAN file system, organizations get the manageability of NAS systems while leveraging the extra bandwidth of a SAN: excellent for applications that need extremely fast data access with low latency such as broadcast content and rich media. It is unique in providing a heterogeneous shared file system that enables servers running any operating system to access a common storage pool. SAN clients allow users to take advantage of Fibre Channel (FC) or Infiniband bandwidth; this ability to leverage higher bandwidth data paths reduces processing overhead and virtually eliminates latency. Servers and clients can be attached to FC or iscsi SAN storage to directly access the same file system, ensuring maximum throughput and minimum latency in mixed operating system environments. Key components. The StorNext File System is enabled by SAN clients residing on heterogeneous servers connected to SAN storage. The StorNext Meta Data Controller (MDC), hosted on a Windows or Linux server, manages shared server access to the storage resource pool to prevent multiple applications from writing to the same disk location. In addition, cache coherency mechanisms ensure that data is correct and up to date even when multiple clients are writing to it (the StorNext Storage Manager is an optional software component that resides on the MDC for automated, policy-based data movement more on that later). Performance. With one or more FC connections on each server to access disk resources, single server performance can be scaled from hundreds of megabytes to multiple gigabytes just by adding FC HBAs. The StorNext file system provides volume management of multiple disk LUNs and multiple arrays for optimal

8 White Paper: Quantum StorNext 8 performance; administrators can create a single volume from multiple disk devices of varying RAID levels. Metadata location also impacts performance, so StorNext allows metadata to be written to a different disk volume, isolating sequential data writes from the more random metadata writes for optimal throughput. StorNext Distributed LAN Client (DLC) The StorNext Distributed LAN Client (DLC) is better for CPU-intensive computational operations (e.g., animation rendering and high-performance computing) for which data access performance is less important. Instead of accessing the file system via FC HBAs and switches, servers can access it over an IP network. SAN clients are designated as clustered gateway servers; distributed servers on an IP network connect to StorNext volumes through these gateways. While the gateways are directly connected to the shared storage pool over Fibre Channel or iscsi, they don t run customer applications. They only service DLC IO. The DLC uses a specialized, block-based IP protocol designed for higher per-stream performance and resilient communication. This enables StorNext to achieve near line-rate throughputs over standard IP connections. Each DLC automatically attaches to multiple clustered gateways, so LAN clients gain the reliability of transparent failover and a load-balanced performance boost. The DLC provides affordable fan-out and fault tolerant connectivity for lower cost LAN-attached clients and high speed SAN access for throughput-intensive applications. Scale for Capacity Users can scale dramatically, enabling them to manage petabytes of data from a single file system instance. The obvious benefit of this is that no matter how much you grow, management efforts and costs should not escalate. This critical component in today s ever-growing data infrastructure addresses a continuing problem for IT managers. Scale with Efficiency StorNext provides a platform on which to consolidate storage infrastructure key to making an environment more efficient. It offers not only extreme scalability and simple management, but simultaneous support of SAN and NAS, FC and iscsi, and heterogeneous operating systems and storage devices. This greater efficiency fits in with every organization s current goals of improving business processes and reducing operational costs. In addition, StorNext includes file system data deduplication for maximum primary storage capacity efficiency. The deduplication algorithm builds its redundancy index on all the data being reviewed, not just the individual file being processed at a given moment. This allows it to find more duplicate content to minimize storage capacity. In addition, it will identify redundancies within a file. As the StorNext platform is particularly useful for large files such as images and video, deduplication is a critical cost saving measure; StorNext deduplication has saved some customers up to 90% of storage space. In addition, StorNext deduplication can take advantage of WAN acceleration techniques because its replication engine can identify duplicate data at a target site and then send only unique file bits across the WAN, greatly reducing bandwidth needs and transfer time. Scale Securely Only recently have enterprise-level features begun to infiltrate scale-out storage solutions. Features such as replication are useful for disaster recovery, making data copies for multi-purposing, and other storage management tasks. The StorNext platform includes asynchronous, host-based replication that is policy-driven and implemented at the directory level. The file system can be configured to replicate to multiple target StorNext file systems in a data distribution model or replicate from multiple StorNext file systems to a single consolidated StorNext file system in a data consolidation model. Replication policies can be assigned to multiple directories for easy management and cascading replication is supported, making any directory a source and/or target for replication.

9 Driving Efficiency White Paper: Quantum StorNext 9 Reducing capital and operational costs is a key benefit of the StorNext platform. It is abundantly clear to IT managers that keeping all data on tier-1 storage is the most expensive, least efficient way to manage it. Virtually no organization needs all of its data stored on the fastest and most accessible types of storage they archive to save money. However, for many organizations, implementing that is not easy. In a standard SAN infrastructure, building storage tiers means purchasing archiving software, buying a server to run it, and creating a separate storage platform to which data can be moved. All of this is costly in terms of management in addition to the licensing and hardware costs. Many begin to wonder if the cure might be worse than the problem. It can be done but at what cost? In contrast, the StorNext platform provides an integrated tiering and archiving workflow. Using the optional Storage Manager software component, policy-based storage tiering and archiving can be performed simply and cost efficiently as the storage profile of certain data changes over time. This is accomplished transparently, without interrupting file access, and eliminates the need for third-party applications to move files between storage tiers. This automated, policy-based data movement engine is easily customized. Policies are assigned at the directory level and allow users to create different storage pools for different file and project requirements within the namespace. These policies are then automatically executed by the Storage Manager. For example, while files might begin on FC disk, after 30 days of inactivity they could be automatically copied to less costly SATA disk or even tape and then removed from the primary storage tier altogether. If necessary, files that are continually in use could be retained constantly on tier-1 storage while temporary files might have a policy indicating they are never to be copied and should be eliminated as needed. All these flexible features improve efficiency by delivering data protection and facilitating long-term archiving while still minimizing cost. Storage Manager uses Distributed Data Movers (DDMs) to move data between storage tiers; adding DDMs can scale system performance. The Bigger Truth Business responsiveness has never been more essential. Business managers want to be able to add applications, scale them as needed, and retain performance no matter how fast data needs grow and change. To operate this way, the concept of IT must be transformed: instead of providing infrastructure, IT must think of itself as a provider of application and data services. As a result, the old ways of implementing IT no longer suffice. Only a virtual, services-oriented infrastructure platform can deliver on today's demands and adapt as the business changes over time. The storage environment must become as dynamic as today s server environments are a transformation that is slowly occurring. Quantum StorNext is a virtual platform delivering scale-out storage that fits this new concept. It enables organizations to create a fluid file storage infrastructure that meets a multitude of performance and long-term archiving requirements. In addition to massive scalability and efficiency, flexibility is a StorNext hallmark it supports heterogeneous environments of multiple operating systems, storage device types, vendors, and storage network protocols. It works for applications that need fast data access and low latency using the SAN File System as well as CPU-intensive applications using the Distributed LAN Client. StorNext also includes enterprise level features such as replication and data deduplication. To support today s nonstop, always-changing business environment, IT administrators must evolve to deliver application and data services. With a virtual, scalable storage platform such as StorNext, that transition can actually take place.

10 20 Asylum Street Milford, MA Tel: Fax: