Virtual InfiniBand Clusters for HPC Clouds April 10, 2012 Marius Hillenbrand, Viktor Mauch, Jan Stoess, Konrad Miller, Frank Bellosa SYSTEM ARCHITECTURE GROUP, 1 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP, KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association www.kit.edu
High Performance Computing + Clouds? HPC Applications Weather forecast, crash test simulations Today in use in all scientific disciplines Supercomputers / HPC Clusters Owned and operated by single institutions Fixed and inflexible run-time environments Cloud Promise: Infrastructure-as-a-Service Rent a service instead of buying and operating HW Pay and use capacity adapting to current demand Cloud Reality: No viable choice for HPC today 2 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Analysis: Clouds Today Contemporary clouds not viable for HPC High communication latency and jitter Performance acceptable for loosely-coupled applications [1,2] Communication-intensive workloads do not scale [3,4] Only premium offers compete with small commodity clusters (EC2 cluster compute instances) [5] Existing clouds cannot run communication-intensive applications which are crucial for HPC [1] Juve et al.: Scientific workflow applications on amazon EC2, 2010. [2] Montero et al.: An elasticity model for HTC clusters, 2011. [3] Napper and Bientinesi: Can cloud computing reach the top500? 2009. [4] Gupta and Milojicic: Evaluation of hpc applications on cloud, 2011. [5] Church et al.: Iaas clouds vs. clusters for hpc: A performance study, 2010. 3 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Proposal: Clouds on HPC Base Cloud environment on HPC infrastructure InfiniBand clusters BlueGene supercomputers Future PCI Express interconnects 4 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Proposal: Clouds on HPC Base Cloud environment on HPC infrastructure InfiniBand clusters BlueGene supercomputers Future PCI Express interconnects 4 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Differences of HPC and Clouds Clouds HPC Network Gigabit/10G Ethernet InfiniBand, BlueGene torus, PCI Express Network QoS Flexibility 77.5 µs in EC2 premium VMs Best effort on-demand (re)configuration custom OS image exchangeable SW layers 2 4 µs with InfiniBand QoS features in HW months for installation, weeks for re-partitioning fixed userbase, applications HW constraints are fixed, e.g. isolation/qos per node 5 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
HPC Cloud Architecture 6 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
HPC Cloud Architecture 6 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
HPC Cloud Architecture 6 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
HPC Cloud Architecture 6 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
HPC Cloud Architecture 6 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Network Isolation Goal: Prevent illegitimate traffic between virtual clusters Base: InfiniBand Partitions Membership per node, not per VM Applications freely choose partition to use Our extension: Transparent enforcement of partitions per VM 7 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Network Performance Isolation Goal: Ensure bandwidth and latency SLAs Base: InfiniBand Virtual Lanes Configurable traffic scheduling Known policies for QoS [6] Applications freely choose traffic class Our extension: Transparent enforcement of traffic classes per VM [6] Alfaro et al.: A formal model to manage the InfiniBand arbitration tables providing QoS, 2007. 8 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Implementation: Intercept Commands Base: HPC network virtualization Proposed by Liu et al. [7] Apps issue send/receive operations directly to HW Connection establishment via host OS Applied with SR-IOV Our extension: Intercept connection management in the host Map users partitions and traffic classes Protect physical network configuration Enforce isolation transparently to user [7] Liu et al.: High performance vmm-bypass i/o in virtual machines, 2006. 9 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Virtual HPC Network View Impression of a dedicated HPC network Behaving like physical network for user apps and config tools Custom node addresses, isolation and QoS Routing customized for communication pattern Topology state machine per virtual cluster Simulate configuration interface Redirect users accesses Repurpose debug tool ibsim for InfiniBand Cloud provider s challenge Virtual cluster placement according to constraints Merging virtual configuration of users 10 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Results Prototype VMs with InfiniBand access Automated isolation setup (partitions) Measurements cannot be published SR-IOV drivers in non-public beta PCI passthrough as substitute MPI application latency (SKaMPI) 77.5 µs in premium cloud offering (10GE) 3.4 µs in our prototype (IB @ 10 Gbit/s) Conceptual evaluation with published pre-alpha SR-IOV drivers Transparent enforcement of isolation works Protection of network configuration is inherent 11 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Future Work Transparent Live Migration on HPC Networks protocol state in hardware node addresses bound to physical nodes Low-Latency-Clouds for non-hpc workloads scale-out workloads bound by latency future tightly-coupled cloud environments adapt workloads to new communication primitives 12 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,
Conclusion Architecture for HPC Cloud Computing InfiniBand virtualization Network and performance isolation Transparent enforcement of isolation Virtual HPC network view Impression of exclusive use Behavior of a physical cluster Physical network configuration is protected Next step: Evaluation with SR-IOV 13 10.04.2012 Marius Hillenbrand - Virtual InfiniBand Clusters for HPC Clouds SYSTEM ARCHITECTURE GROUP,