Simplifying Big Data Deployments in Cloud Environments with Mellanox Interconnects and QualiSystems Orchestration Solutions

Transcription

1 Simplifying Big Data Deployments in Cloud Environments with Mellanox Interconnects and QualiSystems Orchestration Solutions

2 64% of organizations were investing or planning to invest on Big Data technology Gartner, 2013 Executive Summary New advances in Big Data applications are enabling analysts, researchers, scientists and engineers to run more complex and detailed simulations and analyses than ever before. These applications deliver game-changing insights, bring new products to market and place greater demand on existing IT infrastructures. This ever-growing demand drives the need for instant access to resources compute, storage, and network. Yet one of the challenges for engineering teams is to achieve agility and user empowerment in the deployment of Big Data infrastructure. Mellanox and QualiSystems have partnered to offer a joint solution leveraging Mellanox s CloudX architecture and QualiSystems CloudShell infrastructure orchestration solution. This collaboration delivers provisioning and adaptive web catalog-driven automation that delivers faster validate deployments and easier, more user-empowered Big Data selfservice offerings. The Need for Speed - in Both Performance and in Ease of Deployment Quali's orchestration and automation solutions ease the deployment of complex Big Data infrastructure and applications. According to the Gartner 2013 Big Data Adoption Survey, 64% of organizations were investing or planning to invest in big data technology, led by media and communications, banking and services organizations. Big data investments are being prioritized to enhance customer experiences and the increase process efficiency. These priorities are driving the need for speedy performance. One of the key challenges in deploying this infrastructure is that it requires a lot of expertise to integrate and validate the various components to ensure the desired performance outcome. In addition, customers are looking to deploy their in private and hybrid cloud environment to take advantage of resources capacity planning and high availability. The need for speed and ease of deployment on top of top of the line infrastructure performance is the reason for Mellanox s collaboration with QualiSystems. Mellanox s CloudX architecture has been designed to deliver the required networking and system efficiency capabilities related to bandwidth, latency, offloads, and CPU utilization for Big Data analytics and unparalleled performance capabilities for cloud overlay networks. QualiSystems orchestration and automation solutions ease the deployment of complex, Big Data infrastructure and provide a platform for user-friendly infrastructure selfservice. Quali s non-programmer oriented automation design tools make it possible for the whole IT and engineering team to effectively build and sustain an automation practice, providing the agility to meet ever growing Big Data and cloud infrastructure requirements.

3 The Mellanox CloudX Solution The CloudX OpenCloud architecture helps to reduce the cost of using compute and storage infrastructures CloudX is a recipe, based off the Mellanox OpenCloud architecture, for building an efficient cloud platform using off-the-shelf building blocks: servers, storage, interconnect and software to form flexible and cost-effective private and public clouds. Limitless cloud performance On-demand compute, storage and networking OpenCloud Architecture: Open Platform / Open Source CloudX can be based on top of an open-source environment, such as OpenStack, that enables the installation, provisioning, diagnostics, and maintenance of the cloud in the data center. CloudX helps to reduce the cost of using a compute and storage infrastructure and it enables applications to use much more data at real-time to get conclusions and make business decisions faster. Mellanox adapters, switches, cables, software and specific acceleration technologies implement the world s fastest and most robust end-to-end networking solutions for a complete, high-performance Cloud and Big Data analytics infrastructure. These capabilities ensure optimum data center performance with: Up to 56Gb/s fabric speed to support servers (or blade servers) and storage connectivity in any configuration As low as one microsecond application latency, leveraging remote direct memory access (RDMA) between servers and storage RDMA-based solutions that speed transaction processing by offloading significant portions of a communication job from the server CPU and operating system to the IO controller The QualiSystems Solution CloudShell DevOps self-service orchestration and automation platform helps to deliver agile IaaS Quali s lead solution, CloudShell, is a DevOps self-service orchestration and automation software platform for heterogeneous, multi-generational IT infrastructures and networks. CloudShell helps infrastructure teams to deliver agile, end-to-end infrastructure and platform services to end users with much higher ease of use and speed of deployment. CloudShell/CloudX Portal Login

4 CloudShell implements the following key characteristics Centralized inventory management: Engineers have visibility to any components needed to design and publish Big data infrastructure stacks and networked topologies. OOTB libraries and open resource interface creation: CloudShell provides out of the box libraries for many common infrastructure components, but also provides an easy to use tool to create new drivers for interfaces that require customization. CloudShell provides a robust platform for orchestrating diverse infrastructure Object-based automation paired with GUI tools CloudShell implements an object-oriented approach to automation that contrasts with creating fragile, monolithic automation scripts. All automation elements including resources, provisioning interfaces, and automation tasks (such as running a performance test) are captured as small-scope objects. CloudShell s object-oriented approach offers a number of advantages: The limited scope of automation objects means that they are easy to capture, maintain, and refactor to meet the requirements of a changing network environment. A shared library of resource, provisioning and testing objects can be maintained in a systematic fashion. Automation objects can be tagged with arbitrary labels so that they can be easily searched and leveraged by many users from a shared library. An object library optimizes the skills of programmers, who can maintain the shared library as a high quality service to the rest of the network engineering team The object library can be leveraged by non-programmers using GUI-based, drag and drop-style network topology design and automation workflow tools. This maximizes the productivity of the whole network engineering team, especially as topologies and workflows are shared and reused by multiple users.

5 Mellanox s CloudX architecture delivers the required networking and system efficiency capabilities related to bandwidth, latency, off-loads, and CPU utilization for Big Data analytics. CloudShell offers non-programmer friendly GUI automation authoring tools CloudShell s self-service portal can automate validation such as the Flexible IO (FIO) benchmark tests GUI tools such as an automation workflow authoring tool allow non-programmers to assemble objects into a workflow that can be packaged and published to serve as autoprovisioning and auto-reclamation sequences for infrastructure environments. Self-Service Portal CloudShell provides a powerful self service catalog structure for creating, publishing and consuming infrastructure environments CloudShell provides a powerful web portal that offers an adaptive self-service catalog structure for creating, publishing and consuming infrastructure environments. Template-based environments are adaptable so that a single environment definition can accept a wide variety of user-defined parameters. These parameters can drive autoprovisioning and reclamation workflows that can include installing and configuring a variety of Big Data applications and services. Per-environment automation commands can also be published to allow users to validate performance before deploying and running workloads. CloudX Infrastructure Orchestration For Big Data Applications The following section provides the IT manager with a reference architecture for building a private cloud solution for Big Data. The solution utilizes white box hardware built around the fastest interconnect technology of Mellanox 10Gb/s, 40Gb/s and 56Gb/s Ethernet, in combination with cloud orchestration software to provide a pre-integrated, optimized and certified configuration that is ready for production deployment.

6 Mellanox s Openstack Neutron plug in allows hardware to have unique connectivity, security and QoS attributes (4) Management and control Nodes Single CPU, 8+ cores 32GB DRAM 4 HDDs, 2TB each At least (5) Compute Nodes Dual CPU, 8+ cores 128GB DRAM GB SSD At least (5) Storage Nodes Dual CPU, 8+ cores 64GB DRAM 12+ 1TB 10K RPM HDDs GB SSD (Two SSD drives will be used for Boot/Swap) It is recommended to keep the HDDs to SSDs ratio at or lower than 4:1, SSD are used as the write caching layer for the storage solution and provide the needed transaction and throughput capabilities to the storage solution. Networks Hardware-based switching provides better performance, functionality and security of virtual cloud environments Mellanox SX1036B 10/40/56Gb Ethernet switch Mellanox ConnectX-3 Pro Dual Port, 10/40/56Gb Ethernet NIC, MCX314A-BCCT For Cloud orchestration use your choice of hypervisor and management software. Big Data applications, such as Open Source Hadoop, can utilize guest OS based on Linux to help controlling CAPEX and OPEX. Mellanox s Openstack Neutron Agent Mellanox s OpenStack Neutron plugin supports Mellanox s embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox s Neutron Plugin allows hardware vnics (based on SR-IOV virtual functions) per each Virtual Machine vnic to have its unique connectivity, security, and QoS attributes. Hardware vnics can be mapped to the guest VMs through para-virtualization (using a Tap device), or directly as a Virtual PCI device to the guest, allowing higher performance and advanced features such as RDMA. Hardware-based switching, provides better performance, functionality, and security/ isolation for virtual cloud environments. Future versions of the plug-in will include OpenFlow API to control and monitor the embedded switch and vnics functionality This plugin is implemented according to Plugin-Agent pattern. Mellanox s OpenStack Neutron Plugin implements the Neutron v2.0 API. Mellanox s Neutron Plugin processes the Neutron API calls and manages network segmentation ID allocation. The plugin uses databases to store configuration and allocation mapping.

7 The need for speed and ease of deployment on top of the line infrastructure performance is the reason for Mellanox s collaboration with QualiSystems The plugin maintains compatibility to Linux Bridge Plugin, supports DHCP and L3 Agents by running L2 Linux Bridge Agent on Network Node. Mellanox s OpenStack Neutron Agent (L2 Agent) runs on each compute node. Agent should apply VIF connectivity based on mapping between a VIF (VM vnic) and Embedded Switch port. Mellanox s ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. This driver supports Mellanox s embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox s ML2 Mechanism Driver provides functional parity with Mellanox s Neutron plugin. Mellanox s ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. Hardware vnics mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access). Conclusion Deploying a cloud infrastructure with QualiSystems Cloudshell and Mellanox interconnect provides the IT manager with a smooth installation and maintenance tools. Application deployments and testing is simplified with QualiSystems self service portal and centralized management solutions. With Mellanox s high-performance interconnect, customers enjoy the highest performing cloud deployment. Mellanox s offload engines guarantee customers will be able to take advantage of their investment in highly performing CPUs and vast amounts of storage and memory.