MAKING THE BUSINESS CASE

Transcription

1 MAKING THE BUSINESS CASE LUSTRE FILE SYSTEMS ARE POISED TO PENETRATE COMMERCIAL MARKETS table of contents + Considerations in Building the A TechTarget White Paper by

2 Long the de facto parallel file system for science and research, Lustre is poised to penetrate the center of the enterprise for big data and high-performance computing. But buyer take note: Lustre implementations vary widely. As commercial enterprises outgrow network-attached storage (NAS) for performance, IT leaders are considering and deploying scalable parallel file systems. Big data workloads requiring high bandwidth and capacity may be an ideal fit for the capabilities of parallel file systems such as Lustre. Lustre is well suited to applications and workloads requiring large-block sequential I/O or massively concurrent read/write to a single file system. About two-thirds of the world s fastest supercomputers and compute clusters rely on Lustre. Until recently, its strengths and technical merits received little attention among commercial storage buyers, historically served by NAS and an assortment of clustered file systems deployed in front of enterprise storage area networks (SANs). But that s changing. Parallel file systems are maturing and big data workloads for a broad range of use cases are pushing beyond the limits of conventional NAS and SAN, forcing IT leaders and research teams to investigate Lustre and other parallel file systems for commercial and technical computing applications. Still, in order to be accepted by skeptical commercial customers, Lustre requires the backing of leading compute and storage vendors. Support for Lustre in parallel applications took a big step forward with the creation of OpenSFS, a consortium working to advance Lustre and innovate through collaboration. The level of Lustre technical expertise, guidance and support provided by storage vendors varies greatly, to say the least the majority of established storage vendors are in the business of shipping boxes and gear, not configuring and innovating on Lustre. Traditional storage vendors will deliver products and, in some cases, install systems, but they lack expertise in deploying Lustre and typically offer single-vendor proprietary solutions tied to a block storage platform or vendor-specific system. Others, however, provide complete, open Lustre solutions, from the storage disks across the network to the client. Considerations in Building the Business Case for Lustre Lustre is open source, and innovation is community-driven across numerous companies and organizations, such as OpenSFS. That means innovation, architecture and capability are not tied to a single vendor. Considering the growing popularity of open source 1

3 software, open systems and specific application frameworks like MapReduce on Hadoop for big data applications, Lustre is poised to complement much of what traditionally was done on file systems like the Hadoop Distributed File System. For Lustre to gain traction in big data and widen its aperture of commercial applicability, successful technology vendors with established track records in highperformance computing (HPC) and big data need to stand behind Lustre and support it. A key goal should be to support open and interoperable multivendor solutions that offer storage choice and perform optimally and attach to common Linux clusters. Some vendors may claim to support Lustre but will not validate all the components in a given solution. The vendor should support the entire configuration from a Lustre client across the network and back to the disks storing the data. Consider the depth of expertise in supporting end-to-end multivendor solutions before deciding on Lustre. Does the vendor support open systems, without locking you in to a single-vendor storage platform? Also, Lustre should be considered if your organization has a big data initiative or workloads requiring large-block sequential I/O. High availability and performance require expertise, and that depends greatly on the vendor supplying Lustre. If you require solutions that can be uniquely adapted or optimized for performance or capacity, consider how important it is to have the flexibility of choice when choosing the storage platform. If you require petascale systems for I/O- or bandwidth-intensive applications, where capacity and performance can be scaled in balanced ratios, keep in mind how important integration and floor space reduction are to your business. When scalability is important where the system grows and changes over time choose your Lustre vendor wisely. The ability to scale capacity from tens of terabytes to multiple petabytes is a big selling point among many providers, but what does that really mean? Reducing complexity and scaling the system consistently and predictably are key tenets of good Lustre file system design. The system should perform optimally and be available, and take full advantage of the underlying storage platform you choose to deploy. Intimate knowledge of high-bandwidth applications is another important requirement, which underscores the importance of selecting a partner with expertise in scaling applications. Specifically, your vendor needs to take a holistic, systemic approach to scalability, rather than a myopic, storage-only focus. Having an understanding of application requirements, performance tuning and system design to achieve optimal results, and having systems and software knowledge particularly in the open source community ecosystem are important benefits in working with a proven, experienced Lustre solutions provider. Not every supplier will have the full understanding of the end-to-end system, supported by a fully optimized I/O delivery system, to ensure systems perform as stated. 2

4 Density is often touted as a benefit among many storage platforms. Yet, with Lustre, this needs to include all componentry, from cards and cables to ports, servers and controllers. Density does not always equate to easier. Take care to choose solutions from vendors offering embedded, integrated systems (rather than what marketing calls appliances ) that actually reduce the number of managed components. This should be measured by the number of cables, ports and other components, as well as by usable square feet of capacity and performance. Some vendors measure raw capacity, while others measure usable space quoting usable capacity. How much space does the entire system utilize for large performance and capacity configurations? Finally, resiliency, robustness, stability and data integrity need to be considered as part of a complete Lustre solution. Big data and HPC infrastructure leaders should take care to understand how solutions are tested, qualified and supported. Are systems robust, resilient and highly available? Is performance measured using real-world test cases and application workloads, or through smoke-and-mirrors marketing benchmarks or marketing claims? At the end of the day, your business counts on the entire system performing for specific applications. Often, this is best measured by understanding the applications and workloads, and by matching the capabilities of the system to the application. Cray s Approach to Few companies can claim as long a commitment to developing and deploying holistic, systemwide Lustre-based solutions as Cray, which builds on its established reputation for scaling systems and its leadership position in supercomputing. Cray deploys Lustrebased storage and data management solutions for big data and HPC applications demanding performance, and for users demanding an optimal experience. Cray offers complete Lustre solutions that are supported from the disk to the client deployed on the compute cluster. For customers that prefer preintegrated systems that come fully Lustre-ready, including all components and software preinstalled and preconfigured, the Cray Sonexion family features modular, scale-out technology for simplified deployment and balanced scalability of performance and capacity. Sonexion offers the ability to scale in a near-linear fashion, supporting massive capacity and performance in a single Lustre file system. Cray Sonexion scales large-block throughput from 5GB per second to more than 1TB per second in a single file system. For customers requiring choice and flexibility combined with Cray s Lustre expertise, the company offers component-based Lustre solutions. Cray s solutions provide a high degree of configurability to deliver performance and capacity in optimal configurations and a range of sizes. Cray s Lustre solutions are also optimized 3

5 for DataDirect Networks (DDN) and NetApp block storage systems, which Cray both sells and supports. Systems range from small to large, from gigabytes to over a terabyte of bandwidth, and from terabytes to petabytes of capacity. Cray s leadership position in HPC continues to broaden to big data and is taking Lustre to commercial enterprises that demand trust, experience and a single point of support for open, multivendor solutions. Through its leadership and involvement with OpenSFS, Cray advocates for the development of features that drive efficient performance at scale, notes Intersect360 Research s Addison Snell in his April 2013 white paper Cray Brings Leadership and Customer Focus to Lustre Scalability. He adds that Cray s leadership role in OpenSFS has allowed it to work closely with many leading Lustre hardware and software providers, and that this technology-inclusive view puts Cray s board-level membership in a position to help steer Lustre in a direction that benefits HPC customers without lock-in to a particular storage solution. Finally, Cray s long history in supercomputing-class applications with tremendous I/O requirements means it understands how Lustre is deployed in real-world applications. Cray s staff of systems engineers and application consultants is well versed in the requirements of customers that count on Lustre to deliver both the capacity and performance levels to make a difference in data-intensive workloads, now including big data and HPC/high-availability computing. Summary Any organization looking for scalable storage solutions in support of big data or HPC should be evaluating Lustre and talking with Cray. With vendors like Cray behind Lustre, open and scalable file systems are making their way into commercial applications. Choose your Lustre vendor wisely. Look for Lustre implementations that offer flexible storage options and avoid vendor lock-in by supporting multiple storage platforms and Linux compute clusters of any type. Those solutions also need to be backed by experts in application scalability and built on the right compute, network and storage solutions. Cray s suite of Lustre solutions offers choice and flexibility, backed by Cray. Commercial organizations looking to deploy complete, end-to-end Lustre solutions for big data and HPC should consider solutions from Cray. 4