STORIANT TECHNOLOGY DEEP DIVE Data Storage for a New Generation
Storiant / Storiant Technology Deep Dive 2 Table of Contents Introduction 3 Key Features of Storiant Software 4 System Overview 5 Achieving Lowest Total Cost of Ownership 10 Delivering Exabyte Scalability 12 Providing Eighteen Nines of Reliability 13 Making Integration Easy 15 Ensuring Regulatory Compliance SEC 17a-4(f) 17 Enabling Storage-as-a-Service 18 Data Storage for a New Generation 19
Introduction Introduction Digital data growth is unprecedented and accelerating. New sources of data, and requirements to keep it securely, pose logistical challenges, but the growing data lake also offers significant analytical opportunities. Storiant software allows organizations to store and take advantage of their growing data assets at a fraction of the cost of traditional or public cloud storage solutions. Legacy storage solutions were not designed for emerging needs. Long-term media like tape archives offer low-cost storage, but compromise data access and long-term reliability. Public clouds offer seemingly affordable long-term storage, but charge extra for data retrieval, sacrifice quick access, and can put sensitive data at risk. Storiant software delivers data storage for a new generation, combining the best aspects of private data ownership and cloud scale. This paper describes Storiant software and its object storage system. It explains how users can achieve the lowest total cost of ownership for long-term data storage, cloud-level scalability, maximum reliability, and adaptable integration with existing data sources and infrastructure.
Storiant / Storiant Technology Deep Dive 4 Key Features of Storiant Software LOWEST TCO EXABYTE SCALABILITY RELIABLE COMPLIANT STORIANT LONG-TERM STORAGE SOFTWARE PERFORMANCE INTEROPERABLE LIFECYCLE MANAGEMENT Object storage software for private clouds ideal for unstructured data including images, video, and audio; and for industry-specific geospatial, seismic, scientific, medical, and machine data Exabyte scalability Lowest total cost of ownership Less than $.01/GB/month (TCO) Power-down capability uses 80% less power, doubles disk life Runs on low-cost, cloud drives Runs on open-standards hardware Reduced support costs via automatic drive replacement Meets SEC & regulatory requirements Immutable / WORM SEC 17a-4(f) compliant High throughput for fast migration and big data analytics Wire speed on 10 Gb Ethernet Dynamic striping Choice of application interfaces Swift API Amazon S3 API Storiant Link: NFS/CIFS Client libraries Java / Python /.Net Multi-tenant (for infrastructure-as-a-service providers) Highly reliable Eighteen nines of reliability with two copies Continuous data checking
Storiant / Storiant Technology Deep Dive 5 System Overview The architecture of the Storiant storage system addresses the challenges and opportunities presented by exponentially accelerating data growth. A Storiant system functions as private cloud storage, offering the scalability of public cloud storage with the security, accessibility, and performance of an on-premise solution. Intelligent hardware management, ease of maintenance, and open-standards hardware support reduce the total cost of ownership of cloud-scale data. Storiant software also allows organizations to consolidate disparate data sources. It unifies information from distributed locations and organizational divisions, and from differing source technologies, enabling central access and analysis. A unified data lake makes information consistent, error-checking reliable, and regulatory compliance easier to ensure. A self-regulating network of independent servers makes up a Storiant storage system. Storiant software for each server includes industry-adopted and proven open-source components, is open-standards compliant, and supports hardware chosen by the organizations that use it. There are three types of servers: Storage servers manage disk drive hardware, maintain data integrity, and provide access to data through flexible interfaces. Storage servers control multiple disk drive enclosures that provide the data storage capacity of the system. Database servers manage rich metadata about stored data and system configuration. Management servers monitor the status of a Storiant storage system. Management servers collect information about data storage capacity and the operational state of each system component. Object Storage Data objects are the fundamental unit of information storage. Objects may be any size. For example, images and videos, files from a file system, or complete databases may be stored as data objects. Data objects have a flat hierarchy, unlike directory-based file systems. Each object is stored alongside the others, with a single grouping layer of logical containers. Extensible metadata holds information about each data object including retention properties for regulatory compliance. To ensure data integrity and peak network bandwidth speed, Storiant software uses distributed instances of the OpenZFS file system to store data objects across large pools of disk drives. OpenZFS manages virtual storage volumes that span multiple disks. Dynamic array striping places object data on separate disks for concurrent access and high throughput. Erasure coded RAID-Z3 configuration ensures that data objects are stored redundantly and allows constant error-checking and data repair as needed. In addition to the redundancy built into the OpenZFS file system, a Storiant system maintains a configurable number of data copies, which are complete duplications of all stored data objects.
Storiant / Storiant Technology Deep Dive 6 Although the storage capacity of a Storiant system is made up of virtual volumes and may be physically stored on separate server hardware, object storage behaves in a unified manner. Users do not need to keep track of volumes or which volume holds specific objects. Data object metadata is crucial to maintaining the information stored in a Storiant system. Database servers use the Apache Cassandra distributed database management system to hold that object metadata. Cassandra database nodes ensure high-availability by replicating and distributing data automatically. All metadata stored in a cluster of Cassandra database nodes is available from any individual node. Storage capacity may be divided into accounts and containers. Accounts make it possible to store divisions of data separately, in multi-tenant fashion, accessible by different sets of users. Within an account, containers divide stored objects for organizational purposes. System Architecture A Storiant storage system is made up of individual server nodes. Storage servers are attached to one or more disk drive enclosures that supply the data capacity of the system. Each storage server makes data input and output APIs available to system users. Database servers maintain the metadata associated with stored data objects and the system configuration. Management servers provide central monitoring of the entire system in a browser-based operator console and make an administration API available for configuring accounts and users. Storiant software is open-standards compliant and operates on any server supported by the Ubuntu Linux operating system. The physical design of a Storiant storage system depends on the hardware chosen by individual organizations and internal IT practices. The number of storage servers and database servers in a system depend on the data capacity required. Storage servers exchange data with each other and with external users. For best data throughput performance, the REST data API and internal data transfer connections between storage servers use separate 10 gigabit per second network interfaces. Like other aspects of the hardware design, network configuration is flexible and conforms to individual organizations IT practices.
Storiant / Storiant Technology Deep Dive 7 Logical topology of a Storiant system. CUSTOMER DATA STORIANT LINK GATEWAY, INTEGRATIONS AND API S EXTERNAL 10 GbE DATA NETWORK STORAGE SERVER PRIVATE 10 GbE DATA NETWORK STORAGE SERVER 6 GB/S SAS 6 GB/S SAS JBODs JBODs PRIVATE 1 GbE MANAGEMENT NETWORK OPERATOR CONSOLE MANAGEMENT SERVER MANAGEMENT SERVER CASSANDRA METADATA NODES CASSANDRA METADATA NODES STORAGE SERVERS AND DISK ENCLOSURES Storage servers handle the input and output operations, and the physical storage media. The Storiant software for a storage server uses two, independent, 10 gigabit per second network interfaces to exchange data at high throughput with external users and simultaneously with other storage servers. Storage servers are attached to one or more disk drive enclosures that provide the data capacity of the OpenZFS virtual volumes used by the Storiant system. Disk drive enclosures, using JBOD (from Just a Bunch of Disks) architecture, present a large number of disk drive devices to the storage server. For example, specific enclosures supplied by AIC i and EchoStreams ii manage 60 and 96 drives. Enclosures handle the power and cooling requirements of the drives and report drive status to attached servers. Storiant software is aware of the physical position of individual drives in the enclosure and controls power to drives separately. Storage servers build OpenZFS virtual volumes from the drives in attached enclosures, optimizing the data striping of volumes across drives to maximize data throughput and reduce drive vibration. During operation, Storiant software spins individual drives down when they are not in use to reduce power consumption and drive wear.
Storiant / Storiant Technology Deep Dive 8 Like other aspects of the hardware for a Storiant storage system, the choice of specific hard disk drives for JBOD enclosures depends on the local environment and IT practices. Because the data reliability and low-maintenance costs of a Storiant system are based on redundant storage architecture and data validation, the attributes of individual drive devices are not critical. Standard cloud or desktop drives provide suitable, affordable capacity. METADATA DATABASE SERVERS Object metadata is critical for accessing the data in a Storiant system. The database servers use the Apache Cassandra distributed database to maintain object metadata and system configuration. The architecture of the Cassandra database provides scalability, high-availability, and proven performance in cloud-scale enterprises iiị The database nodes form a cluster that replicates data within itself, using native Cassandra functionality. By default, the cluster maintains three replicas of each piece of object metadata. As more database servers join the cluster, each node holds a smaller percentage of the total metadata, improving data access performance and highavailability as the system grows. MANAGEMENT SERVERS Management servers provide tools to monitor the capacity, performance, and status of all the servers in the system. The browser-based operator console makes it easy to configure users and accounts and to get detailed system information. Management servers include an administration API for automated configuration. The operator console dashboard displays crucial system metrics. The log monitoring interface collects messages from each server for easy access and fl exible querying.
Storiant / Storiant Technology Deep Dive 9 Flexible Integration Interfaces The Storiant storage system provides users a choice of data interfaces to store, retrieve, and maintain data objects. Each storage server provides a point of access to the complete data object store. Storage servers support: OpenStack Object Storage (Swift) REST API Amazon Simple Storage Service (Amazon S3) REST API Storiant Link: NFS/CIFS a Network Attached Storage (NAS) interface supporting NFS and CIFS protocols Client library APIs: Java, Python, and.net Management servers provide an administration API and a browser-based operator console. The administration API configures accounts, users, and permissions. It includes reporting resources for storage and network usage. Storiant software includes a reference web application that invokes the administration API for use by storageas-a-service providers. The operator console is an internal administration tool that monitors the state of the entire system, broadcasts configurable alerts, and aggregates log messages.
Storiant / Storiant Technology Deep Dive 10 Achieving Lowest Total Cost of Ownership Storiant software is purpose-designed to reduce the total cost of storing cloud-scale data. Support for openstandards allows implementation with hardware of choice. Adjustable, intelligent drive management reduces power consumption, cooling requirements, and drive wear. Continuous error-checking, data repair, and OpenZFS redundant storage permit the reliable use of standard cloud drives for critical data. Automatic drive sparing eliminates emergency disk replacement. Intelligent Drive Management During operation, a Power Manager monitors data input and output operations and determines which disk drives are required. Whenever possible, the Power Manager turns off a drive to conserve resources. The Power Manager spins up and spins down volumes selectively as they are required. When storing data objects, the system selects one of the virtual OpenZFS volumes for the new data. It keeps this tip volume spun up to receive incoming objects. If throughput requirements increase, the system may select additional tip volumes to meet the need. When retrieving data objects, the system spins up the volumes that contain the required information. The latency introduced by the spin-up delay is less than 15 seconds, much closer to warm storage, and vastly more responsive than traditional data archive technology. If lower latency is a higher priority than cost-reduction, Storiant software may be configured to keep a copy of stored data spun up and ready for access more quickly. Storage servers select specific drives to make up a volume in order to reduce cross-vibration. Reducing the time each drive remains active dramatically extends their lifespans. A reduction in drive activity from 14% to 4% can reduce its seven-year failure rate from 5.76% to 1.45%. Lower resource consumption, fewer replacement drives, and lower maintenance costs reduce the total cost of ownership for data in a Storiant storage system. Lower-Cost Disk Drives Historically, the only reliable disk-based media for long-term data storage were enterprise-class drives. Enterprise drives are designed for greater durability and include features that minimize write errors, but are typically five times more expensive than standard cloud disk drives. The architecture of the Storiant storage system delivers greater reliability than enterprise drives, using cloud drives that are one-fifth the cost. Maximum data reliability is a fundamental design principle of the Storiant system. Although wear on drives is reduced, the architecture assumes that individual devices can and will fail. Data redundancy built into the RAID-Z3 storage of the OpenZFS file system, maintenance job functionality that checks and repairs data, and automatic drive sparing ensure that when a low-cost drive does fail, the data stored in the system will not be compromised.
Storiant / Storiant Technology Deep Dive 11 The production scale of cloud drives lowers cost and increases the available selection. Conforming to open standards, Storiant software is compatible with drives that suit individual organizations, rather than locking in a particular manufacturer or model. Without sacrificing performance and reliability, using low-cost drives reduces the expense of storing cloud-scale data. Automatic Drive Sparing Emergency disk drive maintenance incurs overtime pay and a disproportionate amount of time spent replacing individual devices as they fail. By managing spare drives that are automatically activated when a production drive fails, Storiant software allows long Mean Time To Repair (MTTR) and lowers the maintenance cost of the hardware. Storage servers configure virtual OpenZFS volumes that consist of 19 disk drives by default. Additional drives in a JBOD enclosure that are not included in a volume are reserved as spares. For example, if an enclosure contains 60 drives, three 19-drive volumes will leave three spare drives. Storiant software monitors the status of individual drives as reported by the OpenZFS file system. When a drive fails, the software removes it from the volume, adds a spare drive to the volume, and immediately begins restoring redundant copies of data to the new drive. The browser-based operator console interface uses OpenZFS data to report the status of the disk drives in a volume. At any time, users can determine how many drives are operating correctly, how many have failed and been replaced, and how many spares are remaining. Allowing users to predictably schedule drive replacement during non-emergency hours and without repeated, individual repairs lowers the cost of maintaining data storage. The operator console displays the number of functioning drives and the number of spare drives in each JBOD enclosure.
Storiant / Storiant Technology Deep Dive 12 Delivering Exabyte Scalability The architecture of independent, self-coordinating servers that make up a Storiant storage system is optimized for exabyte-level scalability. Adding storage server and database server nodes linearly increases the capacity and data throughput performance. Each storage server is a data interface access point for the entire object store. Although the data objects in the system are distributed across the JBOD enclosures attached to each server, any storage server will return any stored data object that a user requests. For best performance, storage servers will attempt to use HTTP redirection to directly connect client software to the server that holds the data requested. If a client is not configured to or capable of following HTTP redirection, storage servers are connected by an internal, high-throughput network that allows them to transfer data from one server to another, and then to the requesting client. The data capacity of the storage server requires proportional metadata capacity in its database servers. Database servers use the Apache Cassandra distributed database to maintain object metadata. The Cassandra database is designed to operate as a cluster of separate nodes and is optimized for linear scalability iv. Database nodes added to the cluster use native Cassandra functionality to automatically redistribute replicas of the metadata. Adding nodes improves high-availability and adds multiple points of access for the metadata for any object. Geographically distributed server locations may obstruct a high-throughput network connection between groups of storage servers in a Storiant system. In this situation, the storage servers in a system may be divided into separate sites. Separate sites allow the system to replicate copies of stored objects asynchronously, making sure that duplicate data is stored on the servers for each site without compromising data throughput on either. As capacity requirements increase, additional storage and database servers proportionally increase the throughput performance and availability. Adding storage servers, JBOD enclosures, and database servers will support cloud-scale storage capacity without diminishing functionality.
Storiant / Storiant Technology Deep Dive 13 Providing Eighteen Nines of Reliability The design and architecture of the Storiant storage system emphasizes extreme security and high-availability of the data objects it contains. The requirements of data retention regulations and accurate historical analytics made data loss or corruption unacceptable. As the scale of data storage grows, even minute rates of data degradation represent significant information loss. A Storiant system provides eighteen nines of reliability when it maintains two copies of stored data. Storiant software uses a combination of redundant storage and active validation to ensure reliability and longterm data fidelity. Loosely-coupled clusters of storage and metadata nodes eliminate bottlenecks and single points of failure for continuous access to stored objects. Purposeful, multi-faceted design for maximum reliability makes the system significantly less vulnerable to data loss compared to legacy, disk-based storage and tape storage. In addition to protection from media-related data loss, Storiant software includes features to protect sensitive data from accidental or unauthorized alteration or removal. Regulatory compliance features set an optional, configurable retention period during which objects may not be deleted. Retention holds protect individual objects until the hold properties are explicitly removed. Continuous Data Validation Storiant software performs continuous maintenance to ensure that data objects are stored correctly and remain accurate during long-term storage. Detailed object metadata and redundant data storage allow maintenance functions to detect and repair errors and data loss when it occurs. Maintenance functions run regularly to ensure data validity and do the following: Verify that the OpenZFS files for each data object can be read and that they match the length and checksum information recorded in the metadata. Verify that the OpenZFS file system files for stored data objects are free of errors, and schedule repair operations if needed. Review the metadata for each stored data object on a periodic basis and verify that a corresponding OpenZFS file system file is present and of the correct length. If a file is missing or stored incorrectly, replace it with a file from a redundant copy. Ensure that the system contains the correct number of copies of stored data objects. Verify that each copy of stored data contains identical data objects.
Storiant / Storiant Technology Deep Dive 14 RAID-Z3 Redundant Storage OpenZFS virtual volumes hold the underlying data for stored objects in a Storiant system. OpenZFS (based on Oracle ZFS) is well regarded for its robust reliability and performance at cloud-scale v. Native OpenZFS functionality checks and repairs data and uses RAID-Z3 to place redundant copies of data on three physical disk drives. Up to three drives in each volume may fail without losing stored data. RAID-Z3 redundancy significantly reduces the risk of data loss from physical disk drive device failure. Storiant software automatically replaces failed drives in an OpenZFS volume with spare drives in the JBOD enclosure. Replacing drives immediately allows the data repair functionality of the OpenZFS file system to restore lost redundant data as quickly as possible. Data Copies and Sites Storiant software maintains a configurable number of redundant copies of stored objects. Beyond the redundancy built into OpenZFS RAID-Z3 volumes, data copies reduce the possibility of data loss by multiplying the number of devices that must fail before that risk is incurred. Geographically placing data copies in separate sites within a Storiant system guards against local disasters as well as improving system performance for distributed users. Sites within a Storiant system delineate groups of storage servers that share access to a high-throughput (10 gigabits per second) internal network. When network constraints make sharing that high-speed connection impossible, separating the Storiant system into sites allows each site to maintain high input and output performance independently. Typically, a system with multiple sites is also configured to maintain multiple data copies. Maintenance jobs compare and normalize copies in each site asynchronously. A Storiant site will maintain as many copies of stored data as are required (limited by the total capacity of the system). Systems that have only one site may maintain multiple copies and the number of copies is adjustable during operation. High-Availability The loosely-coupled, independent servers in a Storiant storage system are highly available by nature. Storage servers exchange data internally to respond to requests at any node. Database servers maximize data reliability and scalable performance using Apache Cassandra distributed database software. As a Storiant system increases in scale, its tolerance for faults and failures increases as well. Geographically separate sites and multiple data copies provide access to data even in the event of a local disaster. Storiant storage systems may include optional, standby management servers in case the active server becomes unavailable. The active and standby management servers continuously mirror each other s state and monitor availability through Intelligent Platform Management Interface (IPMI) or equivalent connections. If the active management server fails to respond, the standby server will automatically take its place.
Storiant / Storiant Technology Deep Dive 15 Making Integration Easy Storage servers support several interfaces for exchanging data with the client software. Flexible support for interface protocols makes it easy to implement a Storiant system and to consolidate data from disparate sources within an organization. Management servers provide a browser-based operator console interface for configuring accounts, access permissions, and users as well as a REST API. Manage users, accounts, and access permissions in the operator console. Each storage server functions as a point of access to the complete object store. Although data objects are distributed across the system, any storage server can retrieve objects from any other. During operation, all data interface protocols are available for input and output.
Storiant / Storiant Technology Deep Dive 16 Interface Swift API Description The OpenStack Object Storage (Swift) interface is a REST API. Client software sends HTTP messages to the interface to store, retrieve, and manage data objects. Amazon S3 API The Amazon S3 interface is a REST API. Client software sends HTTP messages to the interface to store, retrieve, and manage data objects. Storiant Link Storiant Link: NFS/CIFS is a Network Attached Storage (NAS) interface supporting NFS and CIFS protocols. Storiant Link runs outside of a Storiant storage system functioning as a traditional file system. Files are stored as data objects in Storiant software. Client Libraries Client library APIs allow developers to create applications that run outside of a Storiant storage system and that store and retrieve data objects. Storiant software includes libraries for the Java, Python, and.net languages. Flexible data interfaces make it possible to perform big data analytics on the information stored in a Storiant system. The data interface access points on multiple storage servers can deliver and receive data in parallel and at high throughput for Apache Hadoop-style queries. Storiant client libraries allow developers to integrate native Storiant data access into their analytics tools.
Storiant / Storiant Technology Deep Dive 17 Ensuring Regulatory Compliance SEC 17a-4(f) Storiant software includes functionality that ensures compliance with data retention requirements, including those of the United States Securities and Exchange Commission regulation SEC 17a-4(f). Storiant software enforces an optional and configurable retention period for all data objects stored. No users may delete or alter stored objects until the retention period has passed. Users may increase the retention period for specific objects but are never permitted to decrease it. Organizations that are not subject to retention requirements can configure a zero-length retention period. Users designated to hold regulatory compliance access permissions may set retention properties for stored data objects. In addition to extending the retention period for data objects, users authorized to set regulatory compliance properties may set two types of hold for specific stored objects. Compliance holds and legal holds allow organizations to ensure that critical data is never lost. Only users authorized to set regulatory compliance properties may remove compliance and legal holds. A Cohasset Associates SEC 17a-4(f) & CFTC 1.31(b)-(c) Compliance Assessment certification of Storiant software is available in a report supplemental to this paper.
Storiant / Storiant Technology Deep Dive 18 Enabling Storage-as-a-Service Organizations that deliver storage as a service can configure a Storiant storage system for multiple data storage customers. Accounts are divisions of the storage capacity that require separate access control. Providing separate accounts for storage-as-a-service customers ensures that the stored data for an individual customer is not accessible by any other. Storage-as-a-service customers can use self-service functionality in the administration API to create their own users and roles. The REST administration API allows technically skilled customers to exercise fine-grained control over the specific users who are able to store data, retrieve data, and set data retention properties. Storiant software includes a user portal web application that presents a user-friendly, browser-based interface for configuring the users and access permissions for an account. The user portal web application allows storage customers to confi gure their own accounts. The REST administration API provides reporting resources for individual accounts and for the entire Storiant storage system. Reporting metrics include the amount of data in storage, the amount of incoming and outgoing network activity, and other data operations performed. Storage-as-a-service providers and customers can use the administration API reports to quantify the resources that have been consumed by data storage.
Storiant / Storiant Technology Deep Dive 19 Data Storage for a New Generation Rapid data growth offers logistical challenges as well as promising opportunities. Previous long-term data storage solutions such as the public cloud and legacy media make significant compromises in cost, security, data reliability, and ready access. Exabyte-scale data requires a new generation of storage solution. Storiant software supports private cloud storage, with the advantages of private data ownership and cloud scale. Designed for both the challenges and opportunities of exponential data growth, the Storiant storage system provides the lowest total cost of ownership, cloud-level scalability, maximum reliability, and adaptable integration with information technology. i http://www.aicipc.com/ ii http://www.echostreams.com/ iii http://planetcassandra.org/companies/ iv http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html v Kadav, A & Rajimwale, A., Reliability Analysis of ZFS. University of Wisconsin, Madison. Retrieved from http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf 2 Oliver Street, Suite 702, Boston, MA 02109 USA +1.617.431.8000 www.storiant.com Copyright 2014. All rights reserved. Storiant is a trademark of Storiant Inc. All other company and product names are trademarks or registered trademarks of the respective companies.