HPC over Cloud. July 16 th, HPC Summer GIST. SCENT (Super Computing CENTer) GIST (Gwangju Institute of Science & Technology)

Transcription

1 HPC over Cloud July 16 th, 2015 HPC Summer GIST (Super Computing CENTer) GIST (Gwangju Institute of Science & Technology) Dr. JongWon Kim jongwon@nm.gist.ac.kr

2 Interplay between Theory, Simulation, and Experiment

3 Cloud & Software-Defined Infrastructure - 3 -

4 Open Innovation Platform Open Networking T-Shaped vs I-Shaped Development Fast, Cheap, Performance Fast, Fast, Fast Software-defined X (Infrastructure, Interface, Economy,..)

5 Cloud & Software-Defined Infrastructure (SDN/NFV/Cloud Integration) Cloud DC Traffic Amazon AWS Microsoft Azure Google Cloud Engine VMware Hybrid OpenStack (IBM, HP, Cisco, RackSpace, ) + Network Operator Cloud (AT&T. Verizon, NTT, )

6 Human-Defined Services over Software-Defined Infrastructure Open Data Cloud Big Data Industrial Internet of Things Open Software (Services) Open API Software-Defined Infrastructure Open Platforms (Tools) Open Infrastructure (Resources) (SDN/NFV integrated Mobile+Cloud) 6

7 Converged Software-Defined Infrastructure (SDN/NFV/Cloud Integrated) 7

8 Nutanix, Simplivity, Hyper-Convergence Boxes Pluribus Freedom Server-Switch VMware Project Mystic Evo: Rails, Evo: Racks

9 SmartX Box: Inter-Connected Functions inside Boxes/Sites Resources Workloads - Services Box Functions Site Inter-Connect

10 SmartX Box: Inter-Connected Functions inside Boxes/Sites Configuration Control Visibility P+M, C, D ports Secured Boxes Secured VMs, Containers, VNFs Functions Site Inter-Connect Discover & Connect Normal Secured Tag/Steer/Mapped

11 SDI & Open- Source SW/HW Cloud SDN NFV IoT 11

12 SmartX Provisioning & Orchestration: Converged Resources - Workloads Diversified Functions Auto Scaling Continuous Integration Instant Visibility Flexible Control Zero-touch Configuration Service Chaining

13 Realize Smart Services with Open APIs enabled by SmartX Boxes Build Open APIs with Inter-connected Functions Prepare Programma ble & Virtualized Resources with Hyperconverged SmartX Box Architecture Your Smart Things with API Tools

14

15 Cloud & PaaS (Platform as a Service) Model-driven PaaS: domain specific languages PaaS (Products; Frameworks): apaas (App Server aas) and language runtimes / DBaaS Mini-PaaS (Foundational PaaS, IaaS+): App. Containers + α 15 Dzone Cloud Report 2014

16 HPC over Cloud

17 HPC over Cloud: Target & Benefits Interfacing With External components Sophisticated Migration policies Higher efficiency HPC over Cloud Benefits Workload bursting Accelerated Collaboration Pay-per-use Super Computing HPC over Cloud Cloud Scalability Simplified self-service Computing Economic (Easy & Improved Sharing), Green (Energy-efficient)

18 HPC over Cloud: HPC, HSC, High Performance Computing: Single program High Scalability Computing: Throughput focus; Can be distributed; e.g. Physics Simulation

19 Meta-Operation Technical Computing HPC Services F3 Racing Team = SC Center Technical Computing Academic Computing Dedicated computing Software (Service) HPC over Cloud Platform (Tool) Infrastructure (Resource) Shared HPC Resources 19 Dedicated Resources

20 HPC over Cloud Institute HPC Centers National HPC Cloud Centers (Shared Infra/Platform) HPC SW Services (Licenses) CPU/GPUintensive Memory-intensive Data-intensive Virtualized HPC Clusters Dedicated HPC Centers

21 Building Shared National HPC Resources (every 6 years) Coordinated and Operated by National HPC Center HPC Cloud Center #1 HPC Cloud Center #2 국 가 지 방 HPC Cloud Center #3 HPC Cloud Center #4 #5 #6 기 업

22 Vision for National HPC Services (Human Resources instead of Hardware Resources) Software Software Platform Infrastructure (Hardware) Platform Infrastructure (Hardware) 22

23 Diversified HPC Services over Integrated & Reconfigurable Super Computers Compute- Intensive Hybrid Data- Intensive Diversified & Portable HPC Software Open & Common API Reconfigurable (Software-Defined) HPC Platforms Open & Convergent HPC Resources 23 Cloud- Ready HPC

24 DevOps Automation

25 DevOps for Software-Defined Infrastructure Development + Operation OpenStack (Puppet, Chef, )

26 Computers in Data Centers SDI & Rack Scale Computing

27 Cloud Datacenter as a BIG Computer Luiz André Barroso, Urs Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse- Scale Machines Google Omega & Apache Mesos

28 Distributed Resource Scheduling & Containers Cloud Datacenter as a BIG Computer 28

29 M&S Service Market Place HPC over Cloud Cloud- Ready HPC 30 Examples

30 HPC over Cloud 모델에 따른 Technical Computing Services Materials/Biomedical CAE Deep Learning (GPU) Diversified & Portable HPC M&S Services Compute- Intensive Hybrid Data- Intensive Reconfigurable HPC over Cloud Platforms Open & Common API Open & Convergent Shared HPC Resources 31

31 HPC over Cloud Examples: CERN / Rackspace OpenLab OpenLab: Hybrid Cloud Model based on OpenStack (open-source cloud OS) COMPUTE NETWORKING STORAGE Rackspace Public Cloud Rackspace Private CERN Demonstration of federated identity and aggregated services between a Rackspace Private Cloud at CERN and at least one other cloud. CERN Private Cloud

32 HPC over Cloud Examples: MegaRun Cycle Computing MegaRun: Schrödinger s quantum chemistry software on 156,000 cores across 8 AWS regions (to sort through 205,000 compounds for solar panel). (Nov. 2013) $33,000 / 18 hours vs $68 million / 18 hours; $132,000 / 10 months (existing internal 300-core cluster)

33 HPC over Cloud Examples: CloudFlow Integrates computational services in the Cloud into the engineering workflows of manufacturing SMBs

34 HPC over Cloud Examples: CloudFlow

35 HPC over Cloud Examples: UberCloud HPC server End-User workstation HPC over Cloud Simulation, CAE Portable Units are like containers 36

36 HPC over Cloud Examples: Academic Community Cloud & Overlay Cloud (Japan)

37 Public Cloud Services (EU)

38 - 39 -

39 Science and Technology R&D Cyber-Lab with SDN-Cloud & HPC/DataFarm Playground Ops Multi-user Support S&T R&D Portal Control Tower & Access Center Resource Provisioning for HPC & DataFarm S&T R&D Automated Experiments A Virtual Playground for SDN-Cloud Dev Configuration, Control, Visibility DevOps Virtual HPC Virtual BigData Virtual Pr esence

40 SmartX Software Framework SmartX Automation Orchestration Intelligence Policydriven Orchestration Automated Scaling / Chaining Flexible Control Zero-touch Configuration BigData / FastData Intelligence Visibility Multi-level Visibility Provisioning

41 Thank you! Sent Inquiry to