" " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " "

Transcription

1 ! WHITE PAPER! The Evolution of High-Performance Computing Storage Architectures in Commercial Environments! Prepared by: Eric Slack, Senior Analyst! May 2014

2 The Evolution of HPC Storage Architectures in Commercial Environments Executive Summary High-performance computing (HPC) environments have unique characteristics and challenges that make them different from traditional IT environments. HPC usually means big analytics projects that involve lots of data, high-speed processing and frequently a compute infrastructure that s been specifically designed for the project, often using open-source software. HPC typically uses a divide and conquer approach that leverages a parallel-processing architecture where very large analytics problems are parsed into many smaller processing jobs and run on individual compute nodes, simultaneously. High-performance computing is very different from traditional IT processes, as is the HPC infrastructure itself, which is often physically separate from the mainstream IT infrastructure. Its clustered compute nodes and supporting storage systems are designed to process the raw data it s given and export the solution set as fast as possible, often using a distributed or parallel file system as opposed to NFS. At this point, these temporary data storage pools are merged into the persistent data storage that makes up the mainstream IT environment, for use in business intelligence (BI) applications, research, and so on. For corporate IT, creating a separate HPC environment is inefficient and dealing with opensource computing systems can be unfamiliar. But given the growing importance of HPC and the HPC-like architectures that often come with projects such as big data, IT organizations are being forced to get familiar with HPC and learn how to manage its unique data storage requirements. Like it or not, HPC is making its way into mainstream IT and becoming a critical tool as companies strive to improve their competitiveness. This paper will examine the challenges of running an HPC project in an enterprise IT environment and how the storage infrastructure can evolve to help address those challenges. What s Special About HPC? HPC is common in industries such as biosciences, oil and gas, design and manufacturing, and media and entertainment, to name a few. It s also being used to develop simulations and run real-time analytics, complex use cases that create challenges for the IT infrastructure, specifically storage. For enterprise IT organizations, these challenges are due to the diversity and size of the data sets involved and the high performance required. Another issue is that HPC is typically run separately from the mainstream IT environment, using different data and outputting different results that must be incorporated back into the mainstream IT workflow. For this reason, the data generated in the high-performance compute process is often considered temporary data, since its function is to generate the solution set that s ultimately exported. Ingest, Conversion and Output HPC analyzes lots of data of varying types, data that must be imported from multiple sources such as sensors, mainframes, databases, file systems, automated systems, and so on. These disparate data sets need to be normalized so they can be compared, combined and analyzed against each other, a process that can be complex and compute intensive in its own right. After the analytics are complete, the results need to be exported from the HPC environment back into the IT environment, where they can be used for decision support and other business intelligence. May 2014 Page! 2 of! 12

3 Storage Switzerland, LLC Distributed Compute Another storage challenge is created by the high performance that HPC requires: more specifically, by the distributed compute architectures that are typically used to generate that performance. Simply getting data from a shared storage infrastructure to the compute engines themselves can create too much latency. For this reason, many HPC projects use distributed architectures, such as classical compute clusters using MPI. More recently, there is increasing use of Hadoop. Hadoop Hadoop is an open-source software framework that was originally derived from Google s MapReduce and Google File System projects. It s composed of several modules, the most important of which are MapReduce, the programming model for large-scale data processing, and the Hadoop Distributed File System (HDFS). HDFS supports the Hadoop distributed architecture that puts compute engines into the same physical nodes that store the data. This brings the compute to the data instead of the traditional method of moving data from storage areas into the processing engine. Hadoop does this by parsing very large projects into manageable subsets that are each applied (mapped) to a processing engine, then processed in parallel on different nodes. The results are then combined (reduced) into a solution set that s typically used in other business processes. Hadoop is often used for the ingest process, in addition to the analytics itself, because of its ability to handle large data sets efficiently. Hadoop not only brings some unique abilities to the HPC environment, but also presents some challenges of its own. Hadoop is designed for what s often called the three Vs of big data : variety, volume and velocity. This creates some tremendous requirements, one of which is for a file system that can provide the requisite performance and maintain that performance while it scales. HDFS has some limitations in scale (and performance as it scales), due to its single namespace server, so parallel file systems like Lustre are also popular in HPC environments. Lustre Lustre (which comes from combining the words Linux and Cluster ) is a POSIX-compliant, parallel file system used in large-scale compute projects, such as HPC. Originally developed in the educational community with support from the U.S. government, Lustre currently is an opensource project that s often used in conjunction with Hadoop to provide a distributed, scalable, high-performance file system in place of HDFS. Lustre enables HPC engines to maintain performance by aggregating multiple I/O paths to multiple servers in the compute cluster. HPC s parallel architecture is used to knock huge compute jobs down to size and Lustre supports this well. Storage Challenges with Hadoop, Lustre and HPC Environments Distributed Storage Hadoop and HPC environments are typically designed to use a distributed storage topology, like clusters of scale-out storage nodes, which are often composed of low-cost (commodity) x86 server hardware with internal disk drives. This can result in a more complex infrastructure that s less reliable than what enterprise IT organizations are accustomed to. The distributed file systems used, like HDFS or Lustre, aren t compatible with traditional NFS storage, creating the need for a separate storage infrastructure to support HPC. But there is another challenge for May 2014 Page! 3 of! 12

4 The Evolution of HPC Storage Architectures in Commercial Environments traditional enterprise IT: the fact that many HPC environments rely heavily on open-source software. Open-Source Software As is often the case with the software used in HPC, Hadoop and Lustre are open-source offerings. The requirements these kinds of advanced-technology projects have are often quite unique, making traditional software solutions less than ideal. This can create a do-it-yourself (DIY) experience for users as they are required to not only run the analytics but know enough to essentially design the platform as well. Many of the first Hadoop implementations were in education and laboratory environments where system users were accustomed to building what they needed for a specific project. For IT managers in enterprise environments, however, running a science experiment infrastructure is quite different (and more difficult) than working with traditional IT products. Not a Comprehensive Solution Hadoop also requires other tools to create a working environment, typically other open-source tools. There is an alphabet soup of product names, each representing a different capability one could use to successfully deploy and run a Hadoop environment, such as Hive, Sqoop, Oozie and Pig. In fact, there s an entire ecosystem of companies in business to provide support for Hadoop users who don t want to do it all themselves. Not a Proprietary Solution Enterprise IT typically requires secure and fully supported products from trusted IT suppliers. Open-source components can present support problems and leave users on their own to resolve issues. Without a single vendor providing the solution, users can run into security problems as well. Instead of relying on tested configurations, users are often cobbling together hardware and software components in order to do the analysis that s needed. This can cause reliability issues, especially since no single vendor has the responsibility to make sure it works. Hadoop is designed to be resilient enough to handle unpredictable jobs that stop and restart successfully and not risk data loss. But the storage infrastructure must be able to hold up its end as well. HPC Means a Separate Storage Environment In addition to the special needs that high-performance computing brings, its supporting storage and compute systems are also very different from traditional IT infrastructures. This means that companies running HPC computing projects can have essentially two storage infrastructures that aren t particularly compatible from a data perspective. One is the mainstream IT environment that typically supports the company s persistent unstructured business data on NFS-based file storage and NAS systems from traditional IT vendors. The other is this HPC environment that s often composed of scale-out storage and compute modules and open-source software running a parallel file system. Besides simply managing these very different infrastructures, the challenge for enterprise IT is simply getting the source data into the HPC cluster and then getting the results back into the mainstream IT environment where it can be used by traditional business analytics and decision support tools. This approach means transferring the temporary data that s been output in multiple, parallel streams from the clustered storage systems into the single-stream, persistent NFS data sets that are compatible with the rest of the corporate IT environment. May 2014 Page! 4 of! 12

5 Storage Switzerland, LLC HPC in the Real World Manufacturing One example of an HPC use case is in complex manufacturing, like the automobile industry. These companies do enormous amounts of product testing, from the component level all the way up to road testing the finished vehicles. This testing includes destructive testing (crash tests) where fully instrumented cars are sent speeding into cement walls. This process is obviously very expensive, so manufacturers use computer modeling and simulations to conduct many of these tests. They still run the physical tests at the end, but simulations can give them a lot of information about the designs before they actually crash a car (see Figure 1). High-performance computing is being used to process the data points required to build these sophisticated computer models and to run these simulations. In order to predict the behavior of a large system, such as a vehicle, programs compile the impact of an event on very large numbers of very small elements that make up that large system. Like the pixels in a highresolution image, the model needs to draw on a database of how each element will behave, given a specific stimulus. It then combines the behaviors of these pixel elements to extrapolate how the system will behave as a whole.! Figure 1. Computer Modeling Used in Crash Test Simulations Clearly, this requires an enormous amount of data and the processing power to calculate the affects of each stimulus, plus the secondary affects from other elements. From a storage perspective, HPC requires the capacity to hold all of these data sets and the performance to make that data available to the processing engines in real time. Feeding these distributed processing engines often requires parallel file systems, such as Lustre, which can manage multiple data streams simultaneously. HPC Moves Out of the Lab Many companies in industries that do this type of testing have begun to use HPC in lab environments where they develop these computer simulations. But market pressures and cost pressures are forcing them to accelerate these projects and speed up the design and implementation of product simulations in order to support shorter development cycles. This is pushing computer modeling into the mainstream and moving HPC into the critical path for product development. The challenges are finding ways to integrate high-performance computing May 2014 Page! 5 of! 12

6 ! The Evolution of HPC Storage Architectures in Commercial Environments into their traditional data center environments, using storage infrastructure and suppliers familiar to enterprise IT. Oil and Gas The oil and gas industry has always had a big data problem. Drilling is very expensive, so almost any front-end work that can improve the chances of finding oil or gas is worth the expense. The processes involved typically mean big files, lots of capacity and the horsepower to get the raw data into the compute engine and the results back to the people who need to see it. Data is of course growing as the resolution (or fidelity) of these images improves, but on an even steeper curve. Due to the extensive use of 3-D modeling, every increase in resolution means the total data set is cubed (increased by a power of three). The workflow in the oil and gas industry has remained somewhat consistent over the years. Seismic data was historically processed in large compute clusters and then viewed on high-end workstations where analysts would interpret the results and typically adjust some of the parameters before rerunning the data. This iterative workflow has been improved with distributed HPC technologies that allow more people to view and interpret that data during the process and shorten the feedback loop. The objective now is to develop a workflow where multiple groups of people can quickly visualize and create a more interactive process that minimizes cycle times (see Figure 2). Figure 2. Oil and Gas Application Using Industry Geospatial Visualization The storage infrastructures required to support these challenging workflows must have some unique characteristics. For the brute force front-end processing, parallel file systems such as Lustre are being implemented to support distributed HPC clusters, that now also run Hadoop, MongoDB, NoSQL, and so on. Then, they have to manage the output, providing access to a diverse set of consumers who are driving the interpretation of that data, often using different applications and different storage platforms. Given the scale of these data sets, the storage infrastructure must efficiently handle large quantities of large files and have the performance to distribute the output users need in the shortest possible time frame. May 2014 Page! 6 of! 12

7 ! Storage Switzerland, LLC HPC Storage Needs to Evolve Both the manufacturing and oil and gas use cases exemplify the need for an evolution in storage infrastructures that are not only more powerful but can also handle the data requirements of both the HPC cluster and the enterprise file system environment. These new infrastructures must be able to support the unique characteristics of a parallel file system, such as Lustre, and have the horsepower to keep HPC compute nodes running at full speed. The evolved infrastructure needs to support the integration of HPC results into the mainstream IT environment, handling the crossover between a parallel file system and NFS, or SMB in Microsoft Windows environments. To do this, it must be able to manage multiple legacy storage platforms and merge data from 3rd-party NAS appliances into a single infrastructure that supports a global namespace. This infrastructure should also provide the enterprise-class storage performance, reliability and services that IT organizations expect. Figure 3 shows how the HPC data infrastructure with its parallel file system fits into the larger IT environment. Figure 3. How HPC Fits Into a Larger IT Environment Lustre Solutions From Hitachi Data Systems Lustre is a distributed, parallel file system that s becoming increasingly common in HPC environments. It can scale easily to over 50PB in a single file system and provide up to 1TB/sec or more of throughput. Its ability to handle thousands of users and hundreds of storage servers is making it ideal for the scale-out compute clusters typically used in high-performance computing. Lustre solutions are used when NFS can t support the throughput, file system size or number of concurrent users that must access data. As an open-source solution that runs on Linux servers and scale-out block storage, Lustre can scale in concert with the HPC compute clusters that typically leverage open-source software and Linux as well, which is also open-source software. But companies looking to integrate HPC into their IT environments need a storage solution that can support the unique characteristics of Lustre, while providing the enterprise features and functionality they ve become accustomed to. Real-time monitoring and management, enterprise May 2014 Page! 7 of! 12

8 ! The Evolution of HPC Storage Architectures in Commercial Environments scalability and performance, Tier 1 uptime and reliability and global support are examples of these features. Hitachi Data Systems (HDS) has developed a set of preconfigured building blocks with which to construct HPC solutions that feature Lustre throughout (see Figure 4). These can be modified to suit different requirements, but currently include the following in their base configurations: Common Solution Framework, which consists of network switches, Fibre Channel infrastructure and a dedicated management node running Intel Manager for Lustre software and Hitachi Command Suite, all housed in an industry-standard 42U rack. High-Availability Metadata Server (MDS), which includes a pair of 1U x86 Hitachi servers and one Hitachi Unified Storage (HUS) 110 with 900GB of capacity on 10K RPM disk drives. High-Availability Object Storage Server (OSS), which includes a pair of 2U Hitachi x86 servers, one HUS 150 and two 5U disk trays with 84 x 3TB nearline serial-attached SCSI (NL-SAS) high-capacity disk drives. Figure 4. Lustre Solution From Hitachi Data Systems The modular architecture allows configurations to be built specifically for the HPC application, using the open-systems platforms that HPC industry is accustomed to using, but is managed from the Hitachi Command Suite framework. Components are built for high capacity, high performance, high reliability and maximum uptime, and all are available from the same enterprise supplier. Consistent with its enterprise-class focus, the Lustre solution from HDS includes the Intel Enterprise Edition for Lustre, a distribution of the open-source Lustre software that is hardened for enterprise environments. It also includes the Intel Manager for Lustre, a point-and-click management tool. May 2014 Page! 8 of! 12

9 Storage Switzerland, LLC High performance computing is becoming part of the critical path for more companies. IT organizations can t be forced to trade the enterprise features they need to do their jobs, like virtualization, high availability and integrated data protection, for the storage performance HPC requires. As a Tier 1 supplier, HDS can provide solutions that have both the performance that HPC demands and feature/functionality that IT managers need. The modular Hitachi architecture provides maximum flexibility to handle dynamic workloads common in HPC and maximum scalability to keep up with big data, including server hardware. Hitachi NAS Platform (HNAS) for Enterprise Environment That Includes HPC Hitachi NAS Platform HNAS is a clustered, network attached storage (NAS) platform that connects to Hitachi Unified Storage (HUS) and Hitachi Virtual Storage Platform (VSP) family of high-end storage systems. It can also connect with other manufacturers arrays via NFS, to provide a consolidated file space designed for an evolved HPC storage environment. It supports all major file and network protocols on the front end, connecting to storage systems via Fibre Channel on the back end. HNAS leverages an intelligent primary storage deduplication process that can reduce redundant data by up to 90% while maintaining system performance. The unique hardware-accelerated architecture of HNAS combined with the full range of Hitachi enterprise storage covers an important middle ground where HNAS can provide more robust performance than traditional NAS systems. This middle ground enables an organization to leverage HNAS for moderately demanding environments where it would otherwise have had no choice but to rely on parallel file system solutions for more performance. However, organizations can still leverage the HDS portfolio of enterprise storage for solutions with Lustre for more demanding environments that require both the scalable performance of a parallel file system and the reliability inherent in the enterprise storage systems. HNAS also works as a file-based gateway to the latest Hitachi VSP family offering, VSP G1000 enterprise storage systems, pushing the combination of performance, flexibility and enterprise features even further. Hitachi VSP G1000 provides industry-leading performance and scalability with advanced enterprise-class management and virtualization functionality. VSP G1000 can be configured for advanced disaster recovery across geographical distances. It also provides unmatched virtualization functions that can integrate other vendor s block storage. By providing a unified view and a layer of isolation, VSP G1000 enables seamless, advanced data management without impacting users or applications. This approach is especially apparent with the elimination of downtime during technology transitions. Cluster Namespace HNAS creates a single namespace that spans all storage nodes, server clusters (physical and virtual) and external storage systems. It provides a directory structure that s independent of physical storage and a common point of integration and management for all unstructured data storage in the enterprise. Data residing on different storage platforms can be virtualized by HNAS, managed from a single control point and used to support both traditional IT applications and HPC workflows. These data can also be physically migrated into the Hitachi storage infrastructure to take advantage of its performance, data protection and data management features, without disruption to existing workflows. May 2014 Page! 9 of! 12

10 ! The Evolution of HPC Storage Architectures in Commercial Environments Hardware Acceleration Using field-programmable gate arrays (FPGAs) and advanced metadata handling, HNAS clusters can provide NFS throughput of up to 2GB/sec and over 293,000 IOPS. This architecture has the horsepower to support multiplatform file virtualization and the unpredictable workloads that are common shared HPC environments, while scaling up to 32PB in a single namespace. The robust performance delivered through this hardware-accelerated architecture dynamically handles both high-throughput for sequential workloads as well as highly random, IOPS-oriented workloads, simultaneously. Universal Migrator This option allows organizations to virtualize 3rd-party storage systems that use the NFS protocol and manage them with the HNAS system (see Figure 5). HNAS assumes the personality of that legacy system such that, from the system users points of view, their data still looks like it s still on the same system. They can continue to access it via the same paths without disruption until at some point IT initiates a migration of the data from the existing storage systems into the HNAS system, transparent to the system users. Figure 5 shows how the universal migrator feature virtualizes file access on an existing storage system and then migrates those files into an HDS environment with no impact on workflows or data access. The Evolved HPC Storage Environment Figure 5. Universal Migrator for Hitachi NAS Platform For companies faced with the requirement to bring an HPC cluster into their production environments, a number of storage-related challenges can come up. To begin with they need a reliable, powerful compute infrastructure that can be configured to the specific project and a data infrastructure that can keep this dynamic, HPC distributed environment supplied with raw data. This can mean pulling data from many sources, often on different storage platforms, using NFS, and presenting it to the HPC compute engines in a parallel file system such as Lustre. May 2014 Page! 10 of! 12

11 Storage Switzerland, LLC When the analytics are complete, the resultant data must be moved back into the persistent, NFS-based data store so it can be used to provide the business intelligence for the HPC project. Eventually, this data can be moved onto lower storage tiers or into an archive as needed. The storage infrastructure that supports the HPC environment should also include the features and functionality that enterprise IT organizations are accustomed to having, and come from a manufacturer they can rely on. HNAS simplifies the traditional heterogeneous enterprise storage environment by virtualizing existing storage platforms and merging existing file servers into a single namespace. At that point, these data can be migrated onto HDS storage transparently, to fully leverage enterprisegrade Hitachi performance, efficiency, management, and so on. From a single control point, HNAS can manage the NFS data sets that provide the raw material for HPC, staging them on dedicated HNAS storage or simply virtualizing a 3rd-party array that already contains this data. Then, they can be moved into the parallel file system environment as needed to support HPC processes and back out when the process is complete. The consolidated infrastructure from HDS provides several compelling benefits that help meet the challenges of bringing HPC into an enterprise environment. Consolidation By combining existing NAS and block storage systems behind a common global namespace, HNAS enables enterprise IT organizations to consolidate their conventional data storage environment and their HPC data storage as well. This approach simplifies overall data management but also enables the smooth transfer of raw data into the HPC cluster and the results back out. Performance The Lustre solution leverages the Hitachi legacy of extreme performance to provide a storage platform that can keep up with any HPC project. As a turnkey system, this solution can be implemented easily and configured with the performance and capacity to support the dynamic HPC workloads being run. HNAS uses hardware acceleration and an advanced metadata architecture to provide maximum throughput and IOPS via standard file protocols to this consolidated file environment. Flexibility With the ability to virtualize multiple storage platforms (from multiple vendors) and leverage storage technologies from SSDs to capacity-centric disk arrays, HNAS with HUS and VSP family systems can be configured to integrate with almost any existing enterprise storage infrastructure. It can also support almost any HPC project that requires standard file protocol access. By incorporating 3rd-party NAS systems in its common namespace, HNAS can smooth the transition to a new infrastructure that supports HPC. A modular architecture also enables the HPC Lustre solution to be configured to meet the higher capacity and greater performance often demanded by HPC projects. Scalability HPC data sets can get very large, very quickly. HDS storage systems are designed with the ability to grow into the hyper-scale range and have the performance to keep HPC workloads running as usual. The HNAS architecture makes it easy to incorporate multiple storage tiers to make that scalability affordable. May 2014 Page! 11 of! 12

12 The Evolution of HPC Storage Architectures in Commercial Environments Control Hitachi Command Suite provides a unified management framework across all storage and compute platforms, including all the HDS and 3rd-party storage systems in the environment. This GUI-based tool gives IT enterprise-grade control over the combined storage and compute infrastructure. An HDS Solution And, since the entire hardware and software stack is sourced from HDS, enterprise users get the comprehensive support, Tier 1 reliability and high-availability operation that is expected from this front-line supplier. A comprehensive solution from HDS, can help make HPC more like a real enterprise IT infrastructure and less like a science project. Conclusion High performance computing is making its way into mainstream IT environments. Companies in industries that routinely perform big data analytics, complex testing simulations and sophisticated image-based analysis, among other use cases, are pushing for this evolution. But in order to make it work, the HPC storage architecture must evolve as well. The problem for enterprise IT organizations is figuring out how to support these HPC infrastructures, which frequently use open-source software, parallel file systems and scale-out compute architectures, within their existing IT environments. They also have to integrate the storage of raw data that goes into the HPC process and the solution sets that come out, often managing multiple storage systems from different vendors. Adding to this challenge is the fact that many of the hardware and software components used in HPC projects are not supplied by the mainstream infrastructure manufacturers that enterprise IT is used to working with. Hitachi is addressing these concerns with comprehensive hardware and software solutions that can support the unique requirements of HPC around scale, performance and flexibility. And it does this while also solving for the reliability, protection and management aspects associated with enterprise-grade storage solutions. These solutions include turnkey systems that feature the Lustre file system for parallel performance and Hitachi NAS Platform for robust, hardwareaccelerated performance. HNAS can then be integrated into a virtualized NAS environment through a combined global namespace. This enables IT organizations to manage the HPC process from beginning to end using enterprise-grade tools and infrastructure from a true enterprise supplier. May 2014 Page! 12 of! 12