Life Sciences Opening the pipe to faster research, discovery, computation and resource sharing

Transcription

1 Solution Brief: Life Sciences Opening the pipe to faster research, discovery, computation and resource sharing Abstract Advances in Information Technology (IT) are significantly improving the speed at which organizations dedicated to improving the lives of many through research and development conduct business today. In order to dramatically improve the efficiency of their computing infrastructure, life sciences organizations and educational institutions helping with scientific research, computational chemistry, systems biology, and chemical mixing and analysis are turning to Voltaire s InfiniBand-based solutions. The Life Sciences Challenge Contents The Life Sciences Challenge... 1 Today s Solutions... 2 Computational Chemistry Systems Biology... 2 Chemical Mixing and Material Analysis A Better Way... 3 Building High Performance Clusters High-performance InfiniBand Switches... 4 Fast Storage Access... 7 Visualization Solutions Putting It All Together... 8 Key Features & Benefits... 8 Tested & Certified with Leading Applications... 9 Customer Success Stories... 9 Swiss Institute of BioInformatics (SIB)... 9 Tokyo Institute of Technology (TiTech) About Voltaire There are many focus areas for research and development in the life sciences. Life science IT managers are challenged to provide the right solutions for computational chemistry, systems biology, and chemical mixing and analysis. These areas share several challenges: 1. Conducting more simulations per day 2. Alleviating storage bottlenecks associated with exponential data growth 3. Implementing cost effective solutions 4. Allocating compute and storage resources dynamically to meet researchers needs Complex simulations can take days or weeks to run. When simulations take longer, scientific discoveries are delayed, slowing commercialization and increasing competitive threats. With faster simulations, more complex models can be analyzed, additional assumptions can be tested and further modifications become possible leading to more efficient development cycles. Accelerating simulations has clear benefits. The key to accelerating life science application performance is to select high performance computing systems that eliminate bottlenecks associated with Inter-Processor Communications (IPC) and storage connectivity. Large symmetric multi-processing-machines (SMPs) have been used in the past as an answer for generating massive compute power in data centers. However, these proprietary, expensive systems have given way to cluster and grid architectures built from lower-cost commodity elements that offer incredible performance at a significantly lower cost. Because of the ready-availability of Ethernet, many of today s clusters and grids are built with Ethernet as the interconnect. While Gigabit Ethernet-based clustering is less expensive than SMP-based architectures, it tends to be very inefficient. For applications that rely on bandwidth or memory sharing, or require large amounts of data transfer, InfiniBand-based interconnects significantly improve the efficiency of clusters by providing high bandwidths and low latency without increasing CPU utilization.

2 Solution Brief: Higher Education and Research When clusters are built using high performance servers, storage and interconnects, organizations can experience drastically faster simulations and modeling while continuing to decrease the cost of providing computing infrastructure. Today s Solutions Many life sciences organizations employ last generation, less-efficient platforms that use proprietary- or Ethernet-based server interconnects. This approach does not provide the necessary bandwidth for complex simulations or data transfer that are so common with life sciences applications today. In the past, applications were embarrassingly parallel and were thought to not require a high speed inteconnect such as InfiniBand. The latest commerical, open source and university-created applications have been optimized to take advantage of InfiniBand s improved performance and data transfer speeds. Computational Chemistry Life science organizations in the area of computational chemistry conduct drug research and discovery as well as biochemical analysis and modeling. These areas share five primary challenges: Growing size of formulas and problems Need to accelerate development of new life-saving drugs More comprehensive safety and drug interaction identification requirements Applications that are not optimized for today s powerful CPUs Severe price sensitivity and cost pressure from various groups Simulations and modeling for chemical analysis involves huge formulas and constant number crunching. Until recently, most solutions used Ethernet as an interconnect. Because of the high CPU overhead related to the handling of communication requests, servers spend more cycles managing inter-processor communications than actually solving computational tasks. Systems Biology Life science organizations conducting analyses and modeling in many different chemical and molecular areas share three primary challenges: Simulations of larger, more complex systems and modeling of cell behaviors require lots of bandwidth, low latency, and fast storage access File size and quantity is growing exponentially leading to I/O bottlenecks Grids are used to share data, discoveries and models Systems biology involves many researchers that constantly retrieve, change, and replace extremely large files. This causes a tremendous amount of bottlenecks in the network. Because so much data is shared across organizations, life sciences organizations use large-scale grids. Chemical Mixing and Material Analysis Life science organizations conducting research and discovery in chemical analysis, material, adhesive, flavors and scent modeling share three primary challenges: Growing size of formulas and problems cause I/O bottlenecks The need to accelerate product development Safety and quality issues need to be identified sooner in the process 2

3 Similar to computational chemistry, the size of formulas can be quite large and cause huge bottlenecks. If these bottlenecks can be resolved, more analyses can be done faster, accelerating products to market. Additionally, issues and safety hazzards can be caught sooner in the product cycle, saving organizations thousands of dollars. A Better Way Benchmark testing has found that Voltaire interconnect solutions reduce runtime by as much as % and increase application and file-system performance 10x. To improve the speed of life sciences applications, engineers need to optimize the design of high performance computing systems. IT managers spend a lot of time determining the server CPU to be used in clusters and grids, but the interconnect that is deployed to transport information between the servers is often ignored. This is a missed opportunity as less efficient interconnects cause significant degradation in application performance. Voltaire s InfiniBand solutions accelerate application performance. Voltaire offers high-performance, lowlatency solutions that enable applications to reach their full performance potential. Benchmark testing has found that Voltaire interconnect solutions reduce runtime by as much as percent. Voltaire offers highperformance (10, 20 and 40 Gbps), low-latency (< 2 microseconds) interconnect solutions used in the world s highest performance supercomputers and data centers. InfiniBand is an industry-standard interconnect for high-performance computing (HPC) and enterprise applications. The combination of high bandwidth, low latency, and scalability with high performance storage makes InfiniBand the interconnectof choice to power many of the world s largest and fastest computer systems and commercial data centers. Voltaire solutions support most major server vendors, operating systems, storage solutions and chip manufacturers. 1 Gb Ethernet 10 Gb Ethernet Myrinet InfiniBand Bandwidth 1 Gb/sec 10 Gb/sec 2.5 Gb/sec 10, 20 & 40 Gb/sec Latency ~10 us us < 2 us Average Efficiency 53% No Entries 68% 74% Price Per Gig/Port ~$ >~$ ~$ <$ Table 1: Price/performance advantages for InfiniBand In addition, Voltaire works with leading storage and application vendors to optimize their solutions to alleviate IPC and file I/O bottlenecks. By combining leading storage technologies with InfiniBand and Voltaire s Grid Director family of switch products, life sciences organizations can conduct research faster and more efficiently to gain a clear competitive advantage. 3

4 Solution Brief: Higher Education and Research 70 Parallel Speedup Parallel Speedup # Cores % Efficiency % Efficiency Linear Scaling GbE InfiniBand \ Figure 1. 75% Parallel Speed Up with Voltaire vs. only 50% with GbE Building High Performance Clusters Voltaire offers complete end-to-end server interconnect solutions for speeding life sciences applications. The three major elements of the solution include: High-speed, low latency InfiniBand switches Fast Storage Access and Scalable File Systems Visualization Solutions High-performance InfiniBand Switches Voltaire s InfiniBand-based solutions deliver high performance and scalability to compute clusters. Voltaire offers a complete portfolio of products including a scalable line of InfiniBand switches, high performance I/O gateways (for seamless connectivity to Ethernet and Fibre Channel networks) and fabric management software. Voltaire solutions use the Open Fabric Alliance s OFED drivers and the Open MPI (Message Passing Interface) libraries to optimize application performance for both MPI-based and socket-based applications. Figure 2. Voltaire Grid Director 9024 for small-to-medium sized clusters ranging from 16 to 24 nodes 4

5 For small-to-medium sized clusters, Voltaire offers the Voltaire Grid Director It is a 1U device with twenty-four 10 Gbps (SDR) or 20 Gbps (DDR) InfiniBand ports. The switch is a high performance, low latency, fully non-blocking edge or leaf-switch with a throughput of 480 Gbps. The Grid Director 9024 is well-suited for small InfiniBand fabrics with up to 24 nodes because it includes all of the necessary management capabilities to function as a stand-alone switch. The Grid Director 9024 is internally managed and offers comprehensive device and fabric management capabilities. Designed for high-availability (high MBTF) and easy maintenance, the switch is simple to install and features straightforward initialization. The solution is scalable as additional switches can be added to support additional nodes. Figure 3. Voltaire Grid Director 2004 for scalable clusters ranging from compute nodes. For larger clusters ranging from compute nodes, Voltaire offers the Grid Director 2004 and 2012 multi-service switch the industry s highest performing multi-service switches for medium-to-large clusters and grids. The switch enables high performance non-blocking configurations and features an enterprise-level, high availability design. The Grid Director 2004 supports up to 96 InfiniBand 4X ports (20 Gbps) and the Grid Director 2012 supports up to 288 InfiniBand 4X ports (20 Gbps). Voltaire Grid Director switches are scalable through the use of modular line boards and they feature 10 GbE and Fibre Channel capabilities so the solution can provide high-performance, integrated SAN and LAN connectivity. Voltaire has also defined scalable units for deploying larger, scalable clusters. Scalable units are ideal for constructing large clusters that deliver unparalleled performance to applications. Scalable units combine compute, interconnect and storage capabilities with scalable file systems. At the heart of the solution is the Voltaire Grid Director 2012 multi-service switch. Voltaire s director-class, multi-service switches offer integrated InfiniBand, GbE and Fibre Channel connectivity in a single chassis. This enables MPI and storage traffic to run on the same network, a capability that is not available with Ethernet or proprietary fabrics. By enabling IPC and high performance storage on a single network, Voltaire solutions enable far greater scalability. 5

6 sfu-8 sfu-8 sfu-8 sfu-8 Solution Brief: Higher Education and Research Figure 3. A scalable unit of 200 nodes powered by a Voltaire Grid Director 2012 Scaling out further is made easy by using Voltaire Grid Director switches as core switches to interconnect multiple scalable units. Such connectivity can be implemented as fully non-blocking or as partially blocking depending on application requirements or budget constraints. Figure 4. Multiple scalable units interconnected using a Voltaire Grid Director

7 Fast Storage Access For companies looking to incorporate storage into their InfiniBand cluster, Voltaire offers fast I/O capabilities for storage. Voltaire solutions combine scalable compute and storage capabilities with parallel file systems. By using InfiniBand with parallel file systems, the server s CPU overhead is reduced, freeing up CPU cycles for your application. Before Now Server CPU Utilization Compute IPC Storage Compute IPC Storage Software MPI NFS MPI Parallel File System Network Proprietary Interconnect GbE InfiniBand - Low performance - High overhead on CPU - No scalability - High performance - CPU available for applications - Scalability to thousands of nodes At the heart of the solution is the Voltaire Grid Director 2004 multi-service switch (described above). Voltaire s director-class, multi- service switches offer seamless InfiniBand, GbE and Fibre Channel connectivity. This enables MPI and storage traffic to run on the same network, a capability Ethernet and proprietary fabrics do not offer. By enabling IPC and high-performance storage on a single network, Voltaire solutions allow companies to leave behind the limitations of network file systems (NFS) and move to parallel file systems over InfiniBand. This provides far-greater scalability. Applications can now have effective file I/O rates of 350MB/s compared with the 50MB/s previously available by using NFS. Additionally, the size of compute clusters is no longer limited by the limitations imposed by NFS. Scalable File Systems Running scalable file systems over Voltaire InfiniBand solutions creates the most scalable solution in the industry with more than 1,000 nodes on a single name space, and delivers high performance connectivity for the storage and client nodes. Voltaire has significant experience and expertise in enabling large-scale parallel file system deployments. Such solutions include: Lustre, HP SFS, IBM GPFS, Panasas and PVFS. These solutions, when combined with InfiniBand, solve two critical problems that NFS creates: limited throughput and limited scalability. The diagram (Figure 6) below outlines a Voltaire deployment with HP SFS (Lustre) with 1,100 nodes (2,200 cores) all accessing a single file system. 7

8 sctrl Solution Brief: Higher Education and Research OSS Sustained FS Performance 900 MB/s Client Side Performance 350 MB/s Voltaire ISR 9288 DOD T1-05 Voltaire Grid Visualization Nodes Compute Nodes Storage Nodes Multi-Panel Displays Lustre Clients Lustre Servers (OSS, MDS) Figure 6: Voltaire s TI-05 installation at the D.O.D 1,100 nodes on a single file system InfiniBand is used for both MPI and Lustre over the same wires. Visualization Solutions Customers in a variety of industry sectors including life sciences, energy, automotive, aerospace, government and military use Voltaire solutions for visualization on clusters and grids ranging from dozens to hundreds of nodes. Visualization clusters typically require very high bandwidth and Voltaire s InfiniBand solutions with up to 20Gbps per host link are ideal for this. In addition, the low overhead on the CPU frees it to carry out the image processing faster. The result is a powerful solution that delivers higher levels of resolutions and faster image processing. Figure 7: 3D visualization is commonly used in Life Sciences. Putting It All Together Key Features & Benefits Voltaire solutions for life sciences offer many compelling benefits to users: High Bandwidth: Voltaire solutions provide bandwidth of 20 Gbps to allow for faster and more frequent analysis by life sciences applications. Lower Latency: Voltaire solutions provide latency as low as 1.3 microseconds. Moreover, Voltaire s InfiniBand-based solutions leverage Remote Direct Memory Access (RDMA) with CPU and OS bypass-technologies that greatly reduce memory-copy overheads and associated CPU utilization. Standards-Based: Voltaire solutions are based on InfiniBand: the only industry-standard, highperformance interconnect Flexibility: Grids and clusters that use Voltaire solutions can be built as a fully non-blocking 20 Gbps fabric or as a lower-bandwidth fabric based on the needs of the application. Moreover, Voltaire switches are upgradeable in a non-disruptive, hot-pluggable manner. 8

9 Fast I/O for Storage: Voltaire solutions enable parallel file systems over InfiniBand, which offers far-greater performance and scalability than NFS solutions. Tested & Certified with Leading Applications By working closely with leading server and software vendors on integration and testing, Voltaire offers the fastest and most efficient high-speed interconnect solutions for the life sciences market. OS Support Linux Enterprise Edition from Novell (SUSE SLES) and Red Hat (EL/AS) Voltaire solutions support many leading life sciences applications. Supported Parallel File Systems Applications Lustre, HP SFS, IBM GPFS, Panasas, IBRIX, TerraScale s TeraGrid Accelrys, Gaussian, AMBER, BLAST, FASTA, GlimmerM, Wise2, ACT, ClustalW, EMBOSS, HMMER, Image, T-Coffee, Artermis, CHARMm, Cn3D, GAMESS, GROMACS, RasMol, ReadSeq, TribeMCL, NAMD, NMRView Systems & Platform Partners HP, IBM, SUN, NEC, SGI, NEC, Intel, AMD Customer Success Stories Customers in a variety of industries that rely on modeling, simulation and analysis leverage Voltaire solutions for their cluster interconnects. Companies include consumer-product manufacturers, research orgnizations, pharmaceutical companies, and university development labs throughout the world. Swiss Institute of BioInformatics (SIB) The Swiss Institute of BioInformatics (SIB) needed a system that could help them speed up discovery of research for their humanitarian efforts around diseases like mad cow and breast cancer research. They turned to HP and Voltaire to supply a system that could grow with their needs, but also help speed up the life science applications running on the cluster. The Swiss Institute of BioInformatics, through its VitaI IT Computing Institution, created a joint venture between Oracle, HP and Intel using Voltaire Grid Director switches to enable the modeling and analysis. SIB first built the Vital-IT Computer Center in 2003 using a configuration relying on SAN storage and NFS services, using gigabit Ethernet for communication among servers. But 18 months after the initial deployment, SIB realized two major problems. With SAN storage, a limited number of servers can be connected to the SAN, said Dr. Victor Jongeneel of SIB. Our servers send and receive data from the compute nodes using NFS, and this turned out to be a major performance bottleneck. With 64 clients running significant I/O, we needed a faster way than gigabit Ethernet for all of our compute nodes to share common file space. 9

10 sfu-8 Solution Brief: Higher Education and Research Voltaire Grid Director Switch Storage HP SFS Test Cluster #1 (8 nodes) Test Cluster #2 (4 nodes) Production Cluster (68 nodes) Figure 8: The system configuration as deployed by Swiss Institute of BioInformatics (SIB) SIB also has many computing jobs that read large amounts of data into memory, many of which are I/O bound. Because of the load on the file servers, they would sometimes crash, and jobs could be aborted before completion. Because of this, we could not run as many jobs per unit time as we wanted, Jongeneel said. The Voltaire interconnect between compute nodes provides much better performance both in terms of bandwidth and latency, Jongeneel said. Computing jobs using HP-MPI, Voltaire MPI and LAM MPI now run much faster than they used to, and more importantly, our server I/O capacity is no longer a bottleneck for running any size job. SIB selected a configuration of one large cluster consisting of 80 HP Servers with a mix of Itanium2 and EM64T processors connected by the Voltaire Grid Director switch. The solution leverages the multi-service capabilities of Voltaire s Grid Director switches to enable Fast I/O with storage connectivity. The clusters run multiple life science applications such as Blast, Gromacs, CHARM and Platform Computing s scheduler LSF. This customer is extremely satisfied with the solution because the combination of the Voltaire Grid Director switch, HP servers, MPI software and File I/O for larger jobs makes for easier file manipulation at high performance. 10

11 sctrl sctrl sctrl sctrl sctrl sctrl sctrl sctrl Tokyo Institute of Technology (TiTech) The Tokyo Institute of Technology (TiTech), located in Tokyo, Japan, is one of the leading technical universities in the world. TiTech has a long history as a world leader in high performance and GRID computing, and houses one of the world s largest supercomputers. Server Core Storage The TiTech system serves as an example of how scalable and flexible the Voltaire family of products is. Used by a wide range of researchers across the university and collaborators across Japan and around the globe, the solution delivers more than 40 trillion floating point operations per second (TFlops). The solution tackles computationally difficult problems ranging from: Analysis of how Avian Flu mutates and is transmitted from birds to humans Structural analysis of new materials Making buildings more resistant to earthquakes More accurate prediction of the earth s climate Visit for more information on this solution. 11

12 About Voltaire Voltaire (NASDAQ: VOLT) designs and develops server and storage switching and software solutions that enable high-performance grid computing within the data center. Voltaire refers to its server and storage switching and software solutions as the Voltaire Grid Backbone. Voltaire s products leverage InfiniBand technology and include director-class switches, multi-service switches, fixed-port configuration switches, Ethernet and Fibre Channel routers and standards-based driver and management software. Voltaire s solutions have been sold to a wide range of end customers including governmental, research and educational organizations, as well as market-leading enterprises in the manufacturing, oil and gas, entertainment, life sciences and financial services industries. More information about Voltaire is available at or by calling Notice Reproduction of this publication in any form without prior written permission is not allowed. The information in this publication is subject to change without notice and is provided AS IS WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OR INTERPRETATIONS OF THIS INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL VOLTAIRE BE LIABLE FOR ANY DIRECT, SPECIAL, PUNITIVE OR OTHER DAMAGES. Performance results will vary based upon a number of system factors. Some of these include: server configuration of the processor, chip set, memory size, firmware and driver release versions, MPI version and OS kernel version. The configuration or configurations tested or described may or may not be the only available solution. These tests are not a determination of product quality or correctness, nor does it ensure compliance with any federal state or local requirements. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. Contact Voltaire to Learn More info@voltaire.com Voltaire Inc. All rights reserved. Voltaire and the Voltaire logo are registered trademarks of Voltaire Inc. Grid Director is a trademark of Voltaire Inc. Other company, product, or service names are the property of their respective owners.