WARP Mechanics PetaPod and TeraPod Series Analytics Engines

Transcription

1 WARP Mechanics PetaPod and TeraPod Series Analytics Engines Features & Benefits Line of turnkey Hadoop analytics appliances Ease of deployment and management at all layers Super computer (HPC) class storage performance and scale PetaPod System S: Max storage performance and super computer scale PetaPod System C: Max CPU/RAM performance and super computer scale TeraPod System S: Good storage performance and low scale price optimized TeraPod System C: Good CPU/RAM performance and low scale price optimized Enter the Analytics Age Hadoop is an exciting, prevailing option for organizing and processing unstructured data. It allows data mining and analytics techniques to get the most value from data assets. It also provides solutions to scalability- related performance and management issues, and a framework for handling looming big data content management problems. But as promising as it may be, Hadoop comes with its own challenges. Hadoop can run distributed applications across clusters of servers by breaking up (mapping) big tasks into smaller jobs that can be spread over many computers, then consolidating (reducing) the results from each member into a result. This requires a suite of applications. Getting Hadoop running with a useful application can take many months of staff time. In addition to Hadoop itself, a useful analytics system also requires integration work to combine hardware, management tools, network connectivity shaping to the source of the data, third party software the list goes on. Hadoop is a powerful tool, but the power comes at the cost of complexity. It often takes the better part of a year for organizations to get from proof of concept (PoC) green light, to running a viable PoC system. Then business processes are rewritten, database query code is changed, IT personnel retrained, etc. While the benefits are compelling, the barriers are also formidable. WARP Mechanics simplified the deployment and management of Hadoop with its line of Analytics Engines. Each appliance includes the enterprise- class servers, storage, cluster interconnect, management tools, and IP networking. These have been pre- configured and tuned at the factory. Just cable it to your infrastructure and power the system on. Set site- specific parameters such as passwords and IP addresses for the management network, and you will be ready to run in minutes. The Analytics Engines are available as a full- scale Hadoop system, with petabytes of storage and thousands of CPU cores. This leverages the WARP Mechanics HPC- class PetaPod TM architecture, originally developed as a turn- key midrange super computer. Compared to WARP Mechanics background in super computing, even large scale Hadoop analytics systems are not considered large at all. But the real business value of the WARP approach is that it also scales down. The SME- optimized TeraPod TM appliances are, bar none, the simplest and most cost- effective way for major organizations to roll out a proof of concept or development system, or for smaller groups to roll out a full production system. WARP Mechanics Super Storage Solutions TM WARP.MECH ( ) info@warpmech.com

2 The right size system PetaPod Turn- Key Super Computers It used to be called super computing. Now it is more widely known as High Performance Computing (HPC). But whichever name you prefer, the organizations that build such systems have quietly been advancing technology in big science laboratories and universities around the world. They have developed the fastest, most scalable computing systems in history, with storage to match. Now, WARP Mechanics has made the same HPC hardware, software, and system architecture available at a price point that enterprises can afford. Three years in the making, and announced in 2012, the WARP Mechanics PetaPod attaches over two 2,300 CPU cores and 36TB of RAM, to a petabyte of high speed storage. The PetaPod design was enhanced to include integrated Hadoop and associated software. The result was the PetaPod Analytics System C: 40 TeraFLOPS of computational horsepower, and 28 Gigabytes per second of throughput (~280 Gbps) to the storage layer. (Figure 1) This system also, for the first time, combined the power of Hadoop with the world s fastest filesystem: the WARPfs edition of Intel/WhamCloud s Lustre. This delivers the search optimization of map/reduce with the performance of Lustre. TeraPod Turn- Key Departmental Systems Figure 1 - PetaPod System C HPC- class systems contain considerable overhead. Floor tile footprint, cluster interconnect, power, and cooling are all proportionate to their ultimate size even if they are initially deployed partially. This is necessary at the high end if a system is intended to scale seamlessly from an initial small deployment up to an HPC size. But in small solutions that are intended to remain small, taking an HPC approach to architecture adds cost without adding value. Figure 2 - TeraPod System S (left) and System C (right) Even if a customer wants to get to that scale eventually, it is undesirable to burden the PoC stage with a multimillion dollar price tag. That is why WARP Mechanics offers systems that are optimized for low scale. The TeraPod systems can be purchased at price points which may allow circumventing long budget cycles and, at about 10U, can fit into existing racks instead of taking up dedicated floor space. with the right combination of hardware Aside from optimally supporting different system scales, the Analytics Engines are available in two variations: optimized for maximum storage performance and capacity (System S), or for maximum CPU and RAM (System C). They each include everything required to start a project: storage, servers, network, and software pre- installed at the factory. In each case, the analytics systems are connected to very high speed storage via a 10Gbps/40Gbps low latency Ethernet fabric. Page 2 of 8

3 System S: Storage Scale and Performance System S engines are ideal for analysis scenarios which require storing considerable amounts of data, but typically analyze only a portion of it at a time: when the data coming into the system must be kept online for extended periods, but only the recent data is mined. A few examples include sorting and matching DNA sequence images, looking at long term market trends, searching through satellite images taken of numerous areas over the course of months or years, or finding relationships between macro- climate effects and current weather. TeraPod System S: Four application servers in 2U, each with two Intel Xeon E core processors and 128GB RAM, yielding a respectable 64 high- frequency CPU cores with 512GB of total RAM. The storage layer has a usable capacity of about 200TB enough to run a PoC or perform production analytics tasks for a smaller organization. PetaPod System S: 72 application servers, each with two Intel Xeon E core processors and 128GB RAM. This yields 1,152 high- frequency cores and 9TB of total RAM. The storage layer has a usable capacity of about 2PB and a practical throughput of more than 50 Gigabytes per second (>500 Gbps). System C: CPU/RAM Scale and Performance System C engines are ideal for analytics problems requiring just a moderate amount of online data, and typically analyze all or most of it at a time. If the data coming into the system is archived off once it becomes irrelevant to the analysis process, then the cluster storage only needs to contain the active data set. Useful areas for System C are recent stock market data, tactical troop movements, market trend analysis for short- term products, monitoring athletes in real- time during games to adjust strategy, or disease vector analysis for seasonal outbreaks. TeraPod System C: 12 application servers, each with two Intel Xeon E core processors and 256GB RAM. This yields 192 high- frequency cores and 3TB of total RAM. These systems are connected to about 32TB of high- speed storage. PetaPod System C: 144 application servers, each with two Intel Xeon E core processors and 256GB RAM. This yields 2,304 high frequency cores and 36TB of total RAM. The storage layer has a usable capacity of about 1PB and a practical throughput of about 28 Gigabytes per second (~280Gbps). Storage Performance, Density, and Reliability Each Analytics Engine contains appropriate WARP storage hardware. All contain redundant controllers, PSUs, and fans; all support hot swap drives and FRUs. They operate at 40Gbps, which puts them at the highest end of the storage performance market. Internally, a hybrid combination of HDDs and fast SSDs allows an optimal price- to- performance ratio. These systems are also available in pure SSD configurations and even as NVRAM/SSD hybrids, for customers who operate at the absolute limits of performance. Figure 3 - WDS 8460 Array These systems can scale seamlessly from 10s of TBs to 10s of PBs. Figure 4 demonstrates different approaches to scalability for increasing capacity or performance, demonstrating an advantage of the PetaPod appliance compared to legacy scale- out storage systems: IO is striped across all WARPfs Object Storage Servers, so performance increases as capacity is added. A single file is divided, and each chunk is stored on a different controller in parallel. With many clients following that algorithm, this results in even load sharing across the cluster. Page 3 of 8

4 Figure 4 File striping in scale up and/or scale out scenarios with WARPraid and WARPfs Customers who want to scale up can add more storage targets off of the OSS controller head. With this method, a single HA head can serve nearly two petabytes of storage (raw) in one 42U rack. However, that design does limit the performance of one HA head to about 7GBytes/sec. (~70Gbps). Granted, this probably seems fast to most IT storage customers, but for analytics and similar applications it may not be sufficient. Customers who want to scale out can simply deploy numerous blocks of Object Storage Servers and targets. Adding a building block also adds another 7GBytes/sec (~70Gbps) of throughput to the system. With this method, the capacity density is lower. However, the performance density is significantly higher: ~42GBytes/sec (~420Gbps) per rack. Whichever method of scaling is chosen, the WARP Mechanics storage building blocks connect to the Linux clients via a high- speed network handled by Brocade Ethernet switches. For small environments, the TeraPod System S and System C utilize the Brocade VDX to provide an aggregate of 480Gbps throughput between compute nodes and storage. For large environments, the PetaPod contains a Brocade VDX 8770 chassis to connect to the storage servers at 40Gbps, and Lustre- native clients at either 40Gbps or 10Gbps. NFS and SMB clients may connect via a larger network, and may use any speed of connection. For both large and small systems, the Brocade ICX provides management access to all components. and the right combination of software. Zettaset makes Hadoop manageable and supportable Once you ve collected your Hadoop s core specifications are developed by the Apache community with a focus on web services. Thus, Hadoop is hard to manage, requires a high level of expertise to deploy and maintain, and does not meet requirements for enterprise systems. Hadoop distributions face constraints such as: Limited support for security, compliance, encryption, policy- enablement, or risk management Does not address high availability (HA), service management, or monitoring Not designed with business analytics system performance in mind Page 4 of 8

5 Zettaset Orchestrator automates Hadoop management, and is designed to meet the real world needs of enterprise- class customers. The software addresses the gaps in the Hadoop architecture, adding HA, security protocols such as LDAP and AD, cluster- aware performance optimization, and provides the following enhancements: Hardened to meet enterprise security, high availability, and performance requirements Software automation simplifies Hadoop deployment and eliminates unnecessary dependencies on professional services Dramatically lowers operational expenses by reducing IT resource requirements Accelerates time to value from weeks to hours Designed to manage the WARP Mechanics Hadoop cluster environment This is not a new Hadoop distribution; rather it operates on top of the included standard Hadoop distribution on the Analytics Engine. It fills gaps between Hadoop and the analytics software with which customers need to interact, providing security, data protection, high availability, and health management, all in an intuitive user interface. WARPraid, WARPfs, and WARPnas make Hadoop fly The WARP Mechanics storage software sits completely on the other side of Hadoop: below the database layer. Like Zettaset, it fills gaps in the standard Hadoop model. The WARPraid modules support advanced features such as thin provisioning, de- duplication, block- level checksums, copy- on- write, snapshots, replication, and many more. It improves performance by many orders of magnitude, with 1000x or even greater improvements in real- world scenarios. This is accomplished by using super- fast SSDs or non- volatile shared DRAM devices to log writes, allowing small random patterns to become large sequential operations. For reads, options include adding multiple terabytes of high- capacity secondary read cache SSDs, or even making the entire primary storage layer out of these devices. It also ensures that data is recorded and recovered accurately, with block- level check sums complementing RAID and other integrity features. Write holes, bit rot, silent data corruption In big data systems, these are not theoretical issues, but rather daily occurrences. The WARPraid architecture eliminates all of these problems and more. For the TeraPod systems, WARPnas provides high performance cluster network access between the array and the analytics servers. Typical real world cases can exceed 3 gigabytes per second (30 Gbps) per NAS unit, with at least two units per TeraPod. With the correct accelerator options added to the stack, IOPS typically can exceed 100,000, even in these small- scale, inexpensive systems. The PetaPod series of systems layer WARPfs on top of the RAID layer instead of using a NAS protocol. This is based on Intel/WhamCloud Lustre: the file system which powers the largest, fastest super computers in the world. This combines with Hadoop to allow sustained writes in the 10s of gigabytes per second, while also supporting screaming fast, IOPS intensive read workloads for the analysis. Page 5 of 8

6 Dataflow Example Figure 5 WARP Analytics Dataflowshows how data flows through the WARP Analytics system. The data source feeds information across your network into the WARP at a constant stream through high speed 10GbE network switches. Ingest nodes receive the input and push it to the storage subsystem at 10Gbps, writing data to the WARPfs Object Storage Servers via NFS. The OSS blasts input data to the WARPraid storage pool over 40Gb Infiniband links where it lands on a superfast NVRAM write accelerator to send an immediate acknowledgment that the data is written to stable storage. This instant ack tells the ingest nodes to keep sending data while the WARPraid coalesces write operations into large transaction groups and flushes them to disk. Performance by design The infrastructure is built to allow continuous capture of critical data. Meanwhile, your application can fire off an analysis of the new data as it is being written. The WARP cache handler is not a simple first in- first out algorithm, but looks at Figure 5 WARP Analytics Dataflow both how recent and how frequent data blocks are accessed. This means the system can keep working on cached data at RAM speeds the new data will not push out the more frequently used data. In addition, the WARPraid hybrid storage system incorporates read- optimized SSDs to extend the cache capacity beyond RAM maximums. As data is read by one analytics node, the SSD accelerators store the information allowing increased throughput of subsequent reads from other nodes at reduced latency. Optionally, you can choose an all- SSD configuration that eliminates mechanical disk bottlenecks entirely. Your application returns the results as they come in, without waiting for the entire capture to complete and without missing any samples. Unobtrusive management The management layer is a sidecar interaction to the data stream. This includes a web GUI, a CLI, wizards, the ability to record/replay commands across all WARP appliances, and a REST API. System authentication and policies are setup in advance to allow direct access to the analytics system without interfering with traffic. Each key service in the system is monitored and mirrored for consistency and redundancy while status information from each node is collected and aggregated. Intuitive interfaces minimize user error by providing a high level dashboard view of the cluster. Kerberos authentication and group and user access controls strengthen the security offering a secure environment that can scale with business needs. The management stack has been fully integrated with the WARP hardware and software layers, and the combined turnkey appliance is fully supported by WARP. The software supports many other features including: automatic generation of reports, virtually unlimited incremental snapshots, synchronous mirroring for disaster recovery and remote site backups, thin provisioning, and more. For a complete list, or to schedule an evaluation, contact your WARP Mechanics authorized reseller. Page 6 of 8

7 WARPraid inner- workings At a lower level, the appliance gives the benefits of ZFS that provide scalability, performance, and enterprise storage features. Of particular interest in mission-critical configurations, the WARP appliance has integrity features to avoid accidental and silent data corruption. ZFS provides end-to-end checksums and transactional copy-on-write IO operations. It eliminates 'write-holes' and silent data corruption that have plagued storage solutions for decades. While such features do lower IOPS compared to entirely raw IO, this is an acceptable trade-off in applications where every bit-value matters. Delving deeper into the IO process, we see in Figure 7 WARPraid IO Process HDD Poolhow the different accelerator technologies work to keep data flowing. First, write operations go directly from RAM to NVRAM so that instant ack can take place. Next, WARPraid coalescing algorithm combines a set of write operations into a single transaction group and writes it to disk. Coalesced transactions mean that all writes are sequential writes, increasing throughput and reducing latency. As data is written, each block is given a checksum for integrity purposes. Every time data is read, each block is compared with its checksum to validate the data. If the checksum fails, WARPraid automatically repairs the data from RAID or another copy (if enabled). For those occasions where extreme data integrity is needed, an administrator can enable multiple copies for a given pool which automatically duplicates the blocks and metadata for another layer of redundancy. For read operations, WARPraid first checks its RAM for the requested blocks, and if it exists fulfills the request immediately. As data comes into RAM, advanced cache algorithms pull the recently or frequently used data from RAM directly to the SSD so it s available for future reads almost instantly. In addition, as read operations pull non- cached data from disk, a specialized fill operation will copy the blocks into the SSD to accelerate subsequent reads and minimize the overhead of going to the hard drive pool. Of course, the all- SSD configurations shown in Figure 6 WARPraid IO Process - SSD Pool eliminate hard drive latency altogether. Figure 7 WARPraid IO Process HDD Pool Figure 6 WARPraid IO Process - SSD Pool Value The WARP Mechanics Analytics Engines delivers the following value: Purpose- built appliance architecture Designed as a storage appliance, with all components pre- configured and tested. Guarantees compatibility, plug- and- play deployment, and ongoing manageability. High- performance design WARPfs delivers the world s highest throughput at the parallel file system level with 10Gbps and 40Gbps controllers, and a hybrid SSD- HDD software design. Multiple data protection mechanisms Data is protected in flight and at rest with reliable integrity features. WARPraid eliminates write holes, silent corruption, bit rot, and similar problems. Virtually unlimited scalability With the Zettabyte File System sitting under each storage subsystem, and one file system reasonably containing dozens or hundreds of subsystems, capacity can scale to astronomical levels. Enterprise RAS (Reliability, Availability, and Serviceability) Each system is designed to eliminate single points of failure, maximize management control, and isolate faults. Hot pluggable FRUs are used throughout. Page 7 of 8

8 Simplified management and security with integrated Zettaset Orchestrator lowers operational expenses by reducing overhead and simplifying otherwise complex tasks to accelerate deployment and maintenance jobs. The hardware in a storage system consists of disks inside storage shelves, controllers, and network connectivity from controllers to hosts. These days, conventional storage servers are constructed with commodity components at all of those layers, even when purchased from firms that historically delivered value- added hardware. When purchased via some routes to market, this can result in a cost savings. Most often, though, it just results in lower reliability and higher administrative overhead. WARP Mechanics took a different approach, constructing every layer with top- quality hardware. The cost savings come from WARP Mechanics more efficient business model. Examining the architecture of both PetaPod and TeraPod Anaytics Engines, the benefits of WARP Mechanics overall solution compares well to legacy storage designs: Optimal Analytics for Everyone Figure 8 - Legacy storage vs. WARP Mechanics appliance *Single chassis with 4 separate embedded server modules, each with 512GB. By selecting the correct Analytics Engine from WARP Mechanics, a project can be optimized in terms of CapEx cost, staff time, project completion time, and ongoing OpEx cost as well. Contact WARP Mechanics today to be connected with the authorized reseller nearest you. Copyright 2013 WARP Mechanics Ltd. All Rights Reserved WARP Mechanics, WARPware, the WARP Mechanics logo, the WARP Mechanics icon, and SmartStorage System are trademarks of WARP Mechanics Ltd. in the United States and other countries. Other brand, product, or service names may be trademarks or service marks of, and are used to identify, products or services of their respective owners. This document is supplied AS IS for information only, without warranty of any kind, expressed or implied. WARP Mechanics reserves the right to change this document at any time, without notice.