HPC over Cloud July 16 th, 2015 HPC Summer School @ GIST (Super Computing CENTer) GIST (Gwangju Institute of Science & Technology) Dr. JongWon Kim jongwon@nm.gist.ac.kr
Interplay between Theory, Simulation, and Experiment
Cloud & Software-Defined Infrastructure - 3 -
Open Innovation Platform Open Networking T-Shaped vs I-Shaped Development Fast, Cheap, Performance Fast, Fast, Fast Software-defined X (Infrastructure, Interface, Economy,..)
Cloud & Software-Defined Infrastructure (SDN/NFV/Cloud Integration) Cloud DC Traffic Amazon AWS Microsoft Azure Google Cloud Engine VMware Hybrid OpenStack (IBM, HP, Cisco, RackSpace, ) + Network Operator Cloud (AT&T. Verizon, NTT, )
Human-Defined Services over Software-Defined Infrastructure Open Data Cloud Big Data Industrial Internet of Things Open Software (Services) Open API Software-Defined Infrastructure Open Platforms (Tools) Open Infrastructure (Resources) (SDN/NFV integrated Mobile+Cloud) 6
Converged Software-Defined Infrastructure (SDN/NFV/Cloud Integrated) 7
Nutanix, Simplivity, Hyper-Convergence Boxes Pluribus Freedom Server-Switch VMware Project Mystic Evo: Rails, Evo: Racks
SmartX Box: Inter-Connected Functions inside Boxes/Sites Resources Workloads - Services Box Functions Site Inter-Connect
SmartX Box: Inter-Connected Functions inside Boxes/Sites Configuration Control Visibility P+M, C, D ports Secured Boxes Secured VMs, Containers, VNFs Functions Site Inter-Connect Discover & Connect Normal Secured Tag/Steer/Mapped
SDI & Open- Source SW/HW Cloud SDN NFV IoT 11
SmartX Provisioning & Orchestration: Converged Resources - Workloads Diversified Functions Auto Scaling Continuous Integration Instant Visibility Flexible Control Zero-touch Configuration Service Chaining
Realize Smart Services with Open APIs enabled by SmartX Boxes Build Open APIs with Inter-connected Functions Prepare Programma ble & Virtualized Resources with Hyperconverged SmartX Box Architecture Your Smart Things with API Tools - 13 -
Cloud & PaaS (Platform as a Service) Model-driven PaaS: domain specific languages PaaS (Products; Frameworks): apaas (App Server aas) and language runtimes / DBaaS Mini-PaaS (Foundational PaaS, IaaS+): App. Containers + α 15 Dzone Cloud Report 2014
HPC over Cloud
HPC over Cloud: Target & Benefits Interfacing With External components Sophisticated Migration policies Higher efficiency HPC over Cloud Benefits Workload bursting Accelerated Collaboration Pay-per-use Super Computing HPC over Cloud Cloud Scalability Simplified self-service Computing Economic (Easy & Improved Sharing), Green (Energy-efficient)
HPC over Cloud: HPC, HSC, High Performance Computing: Single program High Scalability Computing: Throughput focus; Can be distributed; e.g. Physics Simulation http://www.cloudscaling.com/blog/cloud-computing/grid-cloud-hpc-whats-the-diff/
Meta-Operation Technical Computing HPC Services F3 Racing Team = SC Center Technical Computing Academic Computing Dedicated computing Software (Service) HPC over Cloud Platform (Tool) Infrastructure (Resource) Shared HPC Resources 19 Dedicated Resources
HPC over Cloud Institute HPC Centers National HPC Cloud Centers (Shared Infra/Platform) HPC SW Services (Licenses) CPU/GPUintensive Memory-intensive Data-intensive Virtualized HPC Clusters Dedicated HPC Centers
Building Shared National HPC Resources (every 6 years) Coordinated and Operated by National HPC Center HPC Cloud Center #1 HPC Cloud Center #2 국 가 지 방 HPC Cloud Center #3 HPC Cloud Center #4 #5 #6 기 업
Vision for National HPC Services (Human Resources instead of Hardware Resources) Software Software Platform Infrastructure (Hardware) Platform Infrastructure (Hardware) 22
Diversified HPC Services over Integrated & Reconfigurable Super Computers Compute- Intensive Hybrid Data- Intensive Diversified & Portable HPC Software Open & Common API Reconfigurable (Software-Defined) HPC Platforms Open & Convergent HPC Resources 23 Cloud- Ready HPC
DevOps Automation - 24 -
DevOps for Software-Defined Infrastructure Development + Operation OpenStack (Puppet, Chef, )
Computers in Data Centers SDI & Rack Scale Computing - 26 -
Cloud Datacenter as a BIG Computer Luiz André Barroso, Urs Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse- Scale Machines Google Omega & Apache Mesos - 27 -
Distributed Resource Scheduling & Containers Cloud Datacenter as a BIG Computer 28
M&S Service Market Place HPC over Cloud Cloud- Ready HPC 30 Examples
HPC over Cloud 모델에 따른 Technical Computing Services Materials/Biomedical CAE Deep Learning (GPU) Diversified & Portable HPC M&S Services Compute- Intensive Hybrid Data- Intensive Reconfigurable HPC over Cloud Platforms Open & Common API Open & Convergent Shared HPC Resources 31
HPC over Cloud Examples: CERN / Rackspace OpenLab OpenLab: Hybrid Cloud Model based on OpenStack (open-source cloud OS) COMPUTE NETWORKING STORAGE Rackspace Public Cloud Rackspace Private Cloud @ CERN Demonstration of federated identity and aggregated services between a Rackspace Private Cloud at CERN and at least one other cloud. CERN Private Cloud
HPC over Cloud Examples: MegaRun Cycle Computing MegaRun: Schrödinger s quantum chemistry software on 156,000 cores across 8 AWS regions (to sort through 205,000 compounds for solar panel). (Nov. 2013) $33,000 / 18 hours vs $68 million / 18 hours; $132,000 / 10 months (existing internal 300-core cluster)
HPC over Cloud Examples: CloudFlow Integrates computational services in the Cloud into the engineering workflows of manufacturing SMBs
HPC over Cloud Examples: CloudFlow
HPC over Cloud Examples: UberCloud HPC server End-User workstation HPC over Cloud Simulation, CAE Portable Units are like containers https://www.theubercloud.com 36
HPC over Cloud Examples: Academic Community Cloud & Overlay Cloud (Japan)
Public Cloud Services (EU)
- 39 -
Science and Technology R&D Cyber-Lab with SDN-Cloud & HPC/DataFarm Playground Ops Multi-user Support S&T R&D Portal Control Tower & Access Center Resource Provisioning for HPC & DataFarm S&T R&D Automated Experiments A Virtual Playground for SDN-Cloud Dev Configuration, Control, Visibility DevOps Virtual HPC Virtual BigData Virtual Pr esence
SmartX Software Framework SmartX Automation Orchestration Intelligence Policydriven Orchestration Automated Scaling / Chaining Flexible Control Zero-touch Configuration BigData / FastData Intelligence Visibility Multi-level Visibility Provisioning
Thank you! Sent Inquiry to jongwon@nm.gist.ac.kr http://scent.gist.ac.kr