Accelerate > Converged Storage Infrastructure. DDN Case Study. ddn.com DataDirect Networks. All Rights Reserved

Transcription

1 DDN Case Study Accelerate > Converged Storage Infrastructure 2013 DataDirect Networks. All Rights Reserved

2 The University of Florida s (ICBR) offers access to cutting-edge technologies designed to enable university faculty, staff and students, as well as research and commercial partners worldwide with the tools and resources needed to advance scientific research. The center provides more than 400 services, ranging from custom and high-throughput DNA sequencing to electron microscopy and antibody development and production. Founded in 1987, ICBR has a rich history of helping life science researchers focus on research without being burdened with developing and maintaining the infrastructure and expertise to support ever-changing technologies. As part of its mission, the center champions the growth and development of research while jump-starting technology transfer to accelerate molecular biology research from the lab to the marketplace. Over the years, ICBR projects have ranged from the discovery of new cultivars of plants, mapping the genome of ants, tracing DNA of people across continents as well as developing cures for rare and cancerous diseases. A six-person cyberinfrastructure team provides ICBR with a wide range of support, encompassing information management, research computing, instrument support, business productivity, desktop computing and mobile computing. By providing an integrated environment where instrumentation, infrastructure and interfaces function seamlessly, ICBR can help researchers accelerate time-to-discovery. According to, cyberinfrastructure section director for the ICBR, the team supports dozens to hundreds of concurrent projects requiring researchspecific computing resources. As a smaller center, we move fast to open computational environments in support of research advancements, he explains. This agility makes it easier for us to develop new platforms and approaches that differentiate us from the rest of the life sciences community. The Challenge: ICBR s nimble, holistic approach to computational, life sciences research has proven to be highly effective, especially as the cyberinfrastructure team began supporting bioinformatics as a service as well as next-generation sequencing (NGS), which increased the scale and complexity of data and computational requirements significantly. As one of the top cross-disciplinary life sciences academic research facilities, ICBR needs to be able to accommodate rapid data growth as the Center adds more and more data generation load with every new high-speed device investment, including next generation sequencers and Cryo-electron microscopy (cryo-em) instruments which are capable of generating TBs of data per day DataDirect Networks. All Rights Reserved 2

3 Challenges: Inability to accommodate rapid data growth as the Center adds more and more data generation load with every new high-speed device investment, including next generation sequencers and Cryo-electron microscopy (cryo- EM) instruments Receiving a surge of Big Data from volumes of data generated stored, shared and archived Facing a variety of logistical issues, including lengthy setup times and cross-vendor integration problems, which were exacerbated by limited staff resources Hitting the limits of the data center s space, power, and cooling Solution: DDN s SFA high-performance converged infrastructure with embedded GRIDScaler and irods technology The sheer volumes of data being generated, stored, shared and archived created a surge of Big Data, which required ICBR to redefine the point where it was economical or scientifically sound to store data. Big Data growth also propelled the center from the Information Age into the Analytics Age, as the need for data collection to support predictive analytics grew. NGS, Big Data and analytics drove our push for a converged on-demand infrastructure that could accommodate other life sciences projects involving both internal and external tenants, Gardner adds. Other problems facing ICBR involved a variety of logistical issues, including lengthy setup times and cross-vendor integration problems, which were exacerbated by limited staff resources. Additionally, dataset issues emerged from the increase of heterogeneous data, the need to normalize the pipeline from different instrument vendors, and rapid changes in data structures. A steady stream of technical issues also surfaced, including the breakdown of network protocols and persistent performance issues with small-file I/O. To top it all off, they were hitting the limits of their data center s space, power, cooling. In the past, ICBR had built out its storage resources using commodity hardware, which wasn t very dense or scalable. We realized growing storage in this way would take up 80 percent of our data center, Gardner says. That would have severely limited our ability to handle informatics, which wouldn t fit in the remaining 20 percent. It was very clear we needed an entirely new approach. The Solution: In seeking a distributed storage platform to anchor its burgeoning NGS efforts, the cyberinfrastructure team wanted an open, flexible solution to handle ever-changing application loads. Open-source technology also was important, as the team didn t want to get locked into any system that necessitated a forklift upgrade or replacement if their Big Data demands continued at an unprecedented rate DataDirect Networks. All Rights Reserved 3

4 Our goal was to develop storage building blocks that would enable us to assemble any kind of storage researchers may need in a simple and secure shared-storage model. Business Benefits: Ability to focus on core scientific tasks, rather than multiple storage environments administration Increased freedom to interact more freely with research data Customizable architecture for analyzing large collections of distributed data Lower management overhead In particular, ICBR wanted to take advantage of a converged infrastructure designed to help them lower both their storage and application footprint. Our goal was to develop storage building blocks that would enable us to assemble any kind of storage researchers may need in a simple and secure shared-storage model, notes Gardner. ICBR wanted to embed the integrated Rule-Oriented Data management System (irods), which is a community-driven, opensource, data grid software solution, within their converged storage infrastructure. With irods, the center would be well-positioned to manage large collections of distributed data totaling billions of files and petabytes of data across tens of federated data grids. An irods data grid, installed as a layer over ICBR s storage system, would make it easier for anyone involved in a research project to view, manage, access, add and share some or all of their data as part of a unified collection. ICBR determined that DDN s SFA unified virtual server and storage appliance with embedded GRIDScaler parallel file system and irods would address the flexibility, density and simplicity the center needed in an architecture that could scale as needed. We really liked the ability to run our own virtual machines and application stack, which no one else was doing. DDN s converged infrastructure model would enable ICBR to give internal and external research teams shared access to highperformance compute and storage resources. With DDN, we could take advantage of an integrated appliance to offer dispersed departments, research groups and lab staff the resources they needed without restrictions, Gardner adds. As a result, ICBR entered into a proof-of-concept to validate the features and benefits of a converged, on-demand infrastructure for life sciences research, including DDN s SFA high-performance storage system with embedded GRIDScaler and irods technology. Not only did the storage platform perform as expected, ICBR confirmed that DDN could meet its density, simplicity, flexibility and scalability requirements DataDirect Networks. All Rights Reserved 4

5 We now have the opportunity to build an authoritative, immutable data warehouse that provides a safe harbor in the middle of the Wild Wild West of scientific research. Based on that, as well as the ability to increase performance with smaller footprint, fewer hardware costs, lower management overhead and latency makes me confident we are headed in the right direction. Technical Benefits: Seamlessly scalable solution from 180TB of raw storage to multiple petabytes idrop (file repository) alleviates security concerns over the use of privileged resources Increased performance with a smaller footprint, fewer hardware costs, and lower latency The Benefits: With DDN s storage and embedded distributed file system out of the box, ICBR was able to expedite the deployment of a new converged infrastructure into its production environment. We were able to take advantage of DDN s embedded platform to take in irods with distributed storage, which hasn t been done before, Gardner says. We have the benefit of pairing the two in a way that s easy to manage. Embedding irods within DDN SFA systems has been straightforward and performance has met all expectations. The combination of fast performance with low latency means that accessing and aligning storage resources don t distract from the science, says Gardner. In scaling its storage platform, ICBR also proved transaction performance demands could be met with irods. We scale to a few thousand files a second with a couple hundred simultaneous read/write transactions easily, he adds. With a little more tuning, we ll be in line with the transaction performance required by larger irods systems. With a current capacity of 180TBs of raw storage, which can be scaled seamlessly to multiple petabytes, DDN s embedded solution meets ICBR s varied life sciences workloads. As a result, the converged infrastructure concept continues to gain traction for both staff and clients as users can take advantage of flexible features and increased freedom to interact more freely with research data. Using DDN SFA high-performance storage engine with In-Storage Processing converged compute capabilities and DDN s GRIDScaler parallel file system, ICBR was able to deploy an adaptive and customizable architecture for storing, managing and analyzing large collections of distributed data totaling billions of files and petabytes of storage across tens of federated data grids DataDirect Networks. All Rights Reserved 5

6 We were able to take advantage of DDN s embedded platform to take in irods with distributed storage, which hasn t been done before. We have the benefit of pairing the two in a way that s easy to manage. An example of ICBR s ability to customize its server storage architecture is the way the organization can offer flexible user checkin/check-out features and a scientific file repository, which reinforces the organization s goal of empowering its users to do more. The idea of establishing a file repository called idrop is to alleviate varied security concerns over the use of staff or client-owned privileged resources that reside on third-party lab computers and mobile devices. Instead of having to store and protect all types of research data on dispersed hard drives, USBs, tablets and smartphones, the cyberinfrastructure team is creating a centralized, unified solution for storing and archiving this information. The goal is a data folder that users can synchronize with irods or idrop for fast, easy and secure access, sharing and archiving of critical research data DataDirect Networks. All Rights Reserved 6

7 The combination of fast performance with low latency means that accessing and aligning storage resources don t distract from the science. We can scale to a few thousand files a second with a couple hundred simultaneous read/ write transactions easily. Since researchers can access as little or as much data as they choose, ICBR researchers and partner institutions, can remain focused on scientific advances such as innovation in antibody development and production, rather than the IT infrastructure. Thanks to its converged, on-demand infrastructure, ICBR also is well positioned to deploy Infrastructure as a Service (IaaS) for handling Big Data and other emerging demands. In the future, ICBR also plans to leverage its converged infrastructure and in-storage processing to offer Big Data performance for even small and midsize infrastructures while further minimizing latency in the stack. We now have the opportunity to build an authoritative, immutable data warehouse that provides a safe harbor in the middle of the Wild Wild West of scientific research, concludes Gardner. Based on that, as well as the ability to increase performance with smaller footprint, fewer hardware costs, lower management overhead and latency makes me confident we are headed in the right direction. DDN About Us DataDirect Networks (DDN) is the world leader in massively scalable storage. Our data storage and processing solutions and professional services enable content-rich and high growth IT environments to achieve the highest levels of systems scalability, efficiency and simplicity. DDN enables enterprises to extract value and deliver business results from their information. Our customers include the world's leading online content and social networking providers, high performance cloud and grid computing, life sciences, media production, and security and intelligence organizations. Deployed in thousands of mission critical environments worldwide, DDN's solutions have been designed, engineered and proven in the world's most scalable data centers to ensure competitive business advantage for today's information powered enterprise. For more information, go to www. or call , DataDirect Networks, Inc. All Rights Reserved. DataDirect Networks, the DataDirect Networks logo, Storage Fusion Architecture & SFA are trademarks of DataDirect Networks. All other trademarks are the property of their respective owners. V1 11/ DataDirect Networks. All Rights Reserved 7