Welcome to the Jungle

Transcription

1 Welcome to the Jungle Dr. Frank J. Seinstra Jungle Computing Research & Applications Group Department of Computer Science VU University, Amsterdam, The Netherlands

2 SARA: Congratulations! 2

3 A Balloon Race School Anniversary 1970s Get furthest & be found Other end of country? Germany? 3

4 A Balloon Race China! Clever Tricks? Aerodynamics Extra lift Go higher 4

5 A Balloon Race A distance record of sorts: 5 meters approximately Stuck in a tree at school playground For all to see for several weeks Lesson learned: Problem solving using complex means may be more difficult than initially expected There may be a jungle out there Must understand it Must have right tools to conquer it 5

6 Jungle Computing Worst case computing as required by end-users Distributed Heterogeneous Hierarchical (incl. multi-/many-cores) 6

7 Why Jungle Computing? Scientists often forced to use a wide variety of resources simultaneously to solve computational problems Prominent causes: Desire for scalability Distributed nature of (input) data Software heterogeneity (e.g.: mix of C/MPI and CUDA) Ad hoc hardware availability 7

8 Problems in the Jungle Jungle Computing for domain scientists? Hardware heterogeneity Middleware heterogeneity Software heterogeneity Kernels in C, MPI, Fortran, Java, CUDA, scripts, Connectivity problems e.g. firewalls, NAT, Infrastructure often dynamic, faulty. Need for integrated, user-friendly solution/toolbox Focus on problem solving, not system fighting 8

9 The Ibis Software Framework transparently overcome connection setup problems 9

10 Not Alone Jason Maassen Niels Drost Rob van Nieuwpoort Henri Bal (and many others) Maarten van Meersbergen Timo van Kessel Ben van Werkhoven 10

11 Domain Example #1: Computational Astrophysics with: Prof. Simon Portegies Zwart and Inti Pelupessy (Leiden Observatory / Leiden University) 11

12 Domain Example #1: Computational Astrophysics Demonstrated live at SC 11, Nov 12-18, 2011, Seattle, USA (three weeks ago) 12

13 Domain Example #1: Computational Astrophysics The AMUSE system (Leiden University) Early Star Cluster Evolution, including gas gravitational dynamics stellar evolution AMUSE hydrodynamics radiative transport Gravitational dynamics (N-body): GPU / GPU-cluster Stellar evolution: Beowulf cluster / Cloud Hydro-dynamics, Radiative transport: Supercomputer 13

14 Domain Example #1: Computational Astrophysics Demonstrated live at SC 11, Nov 12-18, 2011, Seattle, USA (three weeks ago) 14

15 Domain Example #2: Climate Modeling with: Prof. Henk Dijkstra and Michael Kliphuis (Utrecht University) 15

16 Domain Example #2: Climate Modeling The CPL system (Utrecht University) or: The Community Earth System Model (CESM) atmosphere landvegetation CPL sea-ice ocean Ocean, Sea-ice Atmosphere, Land-vegetation GPU / GPU-cluster cluster / Cloud, or supercomputer 16

17 Enlighten Your Research 3 e-infrastructure competition SARA, SURFnet, BigGrid, NWO Propose innovative ways of using requested e-infrastructure Our proposal High-Performance Distributed Multi- Model / Multi-Kernel Simulations Scale up 1000-fold Winner Sustainability Prize Jury: because of the way it utilizes smart software that makes efficient use of the architecture and the resources 17

18 Going Smart Ibis/Constellation: Generalized programming framework for all Jungle Computing applications Automatically maps any application activity (task) onto any appropriate executor (hardware) Activities in any popular language/tool C, MPI, Fortran, Java, CUDA, Python, Smart a.o. for reduced energy consumption: Executors keeping track of own contribution to the whole Static / run-time selection from multiple equi-kernels 18

19 Conclusions Jungle Computing is hard Ibis provides the basic functionality to efficiently & transparently overcome most Jungle Computing complexities Ibis applied successfully in many domains Astronomy, multimedia analysis, climate modeling, remote sensing, semantic web, medical imaging, Data intensive, compute intensive, real-time Open source, download: 19

20 Thank You 20