WOS for Research. ddn.com. DDN Whitepaper. Utilizing irods to manage collaborative research DataDirect Networks. All Rights Reserved.

Transcription

1 DDN Whitepaper WOS for Research Utilizing irods to manage collaborative research DataDirect Networks. All Rights Reserved.

2 irods and the DDN Web Object Scalar (WOS) Integration irods, an open source rule- oriented data management system that helps data grids, digital libraries, persistent archives, and real- time data systems organize and manage large collections of distributed digital data, has been integrated with the DDN Web Object Scaler (WOS), a sophisticated cloud-based file storage technology that radically simplifies and improves how content is stored, distributed, and accessed across multiple geographically dispersed sites. irods helps to mask the complexity of disparate distributed storage silos from users, but the challenge from an IT management perspective is daunting. IT professionals must deal with inordinate number of elements to manage, a maze of configuration parameters to tune, dozens to hundreds of individual file systems to track and load balance between, very high acquisition costs for equipment and software, and high maintenance costs. WOS simplifies an IT professional s life. The elegant simplicity of the WOS overcomes the underlying pitfalls and complications surrounding multi-site scalable content storage and delivery. With WOS, simplicity is designed in from the ground up. The ease of provisioning storage, configuring replicas and replication policies, and scaling out the system combined with the automatic detection of and recovery from component failures dramatically reduces WOS management effort and cost as compared to traditional NAS and SAN storage systems. In fact, you can manage large scale WOS deployments using the simple, easy to use management UI directly from your iphone. About irods irods, which stands for integrated Rule- Oriented Data -management System, is an open source, collaborative effort of the Data Intensive Cyber Environments Center at University of North Carolina at Chapel Hill and the University of California, San Diego, and is funded by the National Science Foundation and the National Archives and Records Administration. Major projects using irods have included the National Archives and Records Administration (NARA), Transcontinental Persistent Archives Prototype (TPAP), the Ocean Observatories Initiative (OOI), the National Optical Astronomy Observatory (NOAO), the Southern California Earthquake Center (SCEC), the French National Library, the United Kingdom e-science program, and many others. About Data Direct Networks Web Object Scaler (WOS) WOS is a revolutionary object- based, cloud storage system that addresses the needs of content scale- out and global distribution. At its core is the WOS object clustering system - intelligent software that allows a massively scalable content delivery platform to be created out of small building blocks, enabling the system to start small and easily grow to a multi -Petabyte scale. WOS is a fully distributed system, meaning there are no single points of failure or bottlenecks. Allows the system to scale with each new cloud building block (called nodes) linearly adding to the system s performance capabilities and storage capacity. WOS nodes are self-contained appliances configured with disk storage and CPU and memory resources. Each node is pre-configured with the WOS software and Gigabit Ethernet network interfaces. A cloud can be created out of any nodes that have Internet Protocol (IP) connectivity to each other, regardless of their physical location. Various capacities and performance configurations of WOS nodes are available. Clouds support heterogeneous node types, allowing tailoring of performance and scale to the needs of the environment DataDirect Networks. All Rights Reserved. 2

3 Rules Oriented Application Meets Policy-Based Object Storage A Marriage Made in Heaven irods is software that can be used to establish a data grid and to establish and enforce rules regarding how digital objects are managed in the grid in order to ensure their preservation as a persistent digital archives. Objects (files or folders) can be moved into storage locations and managed, using unix like icommands, or client software (including a web browser, if some additional software is established on the server). It includes a built in catalog (icat) for managing entries; icat is installed as a postgresql database during install, or you can connect to a preexisting database installation if some additional configuration is completed during and after installation. As shown in the irods Data System Components figure above, the irods server manages objects on disk, and is designed to work with generic storage devices. However, as irods implementations scale to hundreds of millions objects and many petabytes of storage, serious limitations of traditional storage quickly become apparent, specifically in terms of performance & latency, scalability, collaboration, and especially total cost of ownership (TCO). The DDN Web Object Scaler is uniquely architected to address these limitations. Performance and Latency WOS clusters are capable of serving in excess of one million random files per second because as shown in the figure below, the unique architecture of the WOS requires only a single disk seek to the start of the object in order to return it to the requesting application. Generic irods storage devices expose a file system interface which requires on average, ten disk accesses to retrieve a single file. The ability to perform in-memory looksup and single disk seek retrievals for all object locations in the cloud enables WOS to operate 10x faster than other file system technologies that must perform several disk based metadata operations followed by several diskbased block storage operations in order to retrieve information DataDirect Networks. All Rights Reserved. 3

4 Data Locality and User Collaboration irods is designed to operate in massively parallel configurations with geographically dispersed servers. From a storage perspective, a geographically dispersed WOS cloud creates a common storage pool for content ingest, storage, and delivery. The cloud is centrally managed and content distribution within the cloud is automated. Regardless of the number of copies of an object that exist across the cloud, a single common Object Identifier (OID) is used and an irods reference to any OID will retrieve it from the location with the lowest latency, and any new irods objects created at any locality will automatically be copied to other locations based on the replication policy. Thus there is no need to track multiple file names or paths, or to maintain a lookup table pointing to the correct location for an object. A single WOS cloud acts as the global data repository, eliminating the need for multiple storage and file systems, replication software to tie them together, and custom software development to track file locations across the disparate systems. Storage and Administrative Efficiency Large numbers of small files cause performance and scalability problems with traditional file systems, forcing customers to create multiple file systems when they really want to have just one. This requires irods system complexity and processing / database overhead to create and manage multiple file systems (hundreds or even thousands of file systems are not uncommon), as well as track where files are stored among them. In addition, traditional file systems can waste large amounts of storage capacity when the file size is smaller than the logical disk block size. WOS eliminates the wasted capacity common with conventional NAS storage. WOS clouds scale to store literally hundreds of billions of files regardless of size, and efficiently allocates space for small files, eliminating the stranded capacity problem exhibited in standard file systems. WOS "BUCKETS" Contain objects of similar size to optimize placement. WOS OBJECT SLOTS Contain objects of similar size Different sized objects are written in slots contiguously. Slots can be as small as 512B to efficiently support the smallest of files DataDirect Networks. All Rights Reserved. 4

5 WOS Total Cost of Ownership - Best TCO in the industry As irods deployments utilizing standard JBOD or NAS/SAN storage scales out, significant limitations with the storage infrastructure begin to emerge. Imagine trying to scale a system based on these limitations to multiple active sites serving hundreds of millions of files. You would be faced with an inordinate number of elements to manage, an unsolvable puzzle of configuration parameters to tune, dozens to hundreds of individual file systems to track and load balance between, very high acquisition costs for equipment and software, and high maintenance costs. Not to mention the amount of time and engineering work it would take to architect and maintain the system, assuming it could ever be made to function properly. WOS and irods overcome the pitfalls and complications surrounding multi- site scalable content storage and delivery. This enables organizations to focus on their core business, instead of the tedious tasks associated with storage engineering and management. With WOS, DataDirect Networks is ushering in a new era of storage systems design, one where irods storage clouds can be quickly and easily set up, yet scale to service the needs of millions of users accessing multiple pedabytes of information spread around the world, while at the same time delivering the lowest TCO in the industry. Key WOS Metrics Number of Objects Total Raw Capacity Data Replication User-Definied Metadata API Commands Supported API Languages Read/Write Performance (files/sec) 256+ Billion Petabytes Per Cluster Synchronous or Asynchronous by policy From 0 to 64 MB PUT object, GET object, DELETE object, RESERVE ObjectID RESTful, C++, Python, Java, PHP, HTTP Up to 10x NAS/SAN file system performance The WOS / irods Advantage WOS systems deliver radically different capabilities for irods environments as opposed to conventional file storage systems. As an object oriented storage system, WOS has been designed from the ground-up for the Web2.0 and petabyte-scale internet-scale content storage deployments. Petabyte Scalability: Scale out to many of petabytes by simply adding storage modules Unrivaled Simplicity: Management simplicity translates directly to lower cost of ownership Self-Healing: Zero intervention required for failures, and automatically recovers from lost drives Rapid Rebuilds: Fully recover from lost drives in moments Replication Ready: Ingest & distribute data globally Disasters Recoverable: For uninterrupted transactions no matter what type of disaster occurs File Layout: Capacity and performance optimized Object Metadata: User-defined metadata makes files smarter 2012 DataDirect Networks. All Rights Reserved. 5

6 DDN About Us DataDirect Networks (DDN) is the world leader in massively scalable storage. We are the leading provider of data storage and processing solutions and professional services that enable contentrich and high-growth IT environments to achieve the highest levels of systems scalability, efficiency and simplicity. DDN enables enterprises to extract value and deliver results from their information. Our customers include the world s leading online content and social networking providers, high-performance cloud and grid computing, life sciences, media production organizations and security and intelligence organizations. Deployed in thousands of mission-critical environments, worldwide, DDN s solutions have been designed, engineered and proven in the world s most scalable data centers to ensure competitive business advantage for today s information-powered enterprise. For more information, go to www. or call , DataDirect Networks, Inc. All Rights Reserved. DataDirect Networks,, Silicon Storage Architecture, S2A, Web Object Scaler and Storage Fusion Architecture are trademarks of DataDirect Networks. All other trademarks are the property of their respective owners. Version DataDirect Networks. All Rights Reserved. 6