Cosmological simulations on High Performance Computers Cosmic Web Morphology and Topology Cosmological workshop meeting Warsaw, 12-17 July 2011 Maciej Cytowski Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw Email: m.cytowski@icm.edu.pl
Topics Introduce POWIEW and PRACE projects Describe the potential collaboration models Show examples of joint work Describe how to access Polish and European HPC infrastructure 2
POWIEW project HPC Infrastructure for Grand Challenges of Science and Engineering Consortium of 3 biggest Polish HPC centers (Warsaw, Poznań, Cracow) Competence centers and working groups for Grand Challenges: Climate modelling Materials sciences Cosmology Visual Analysis Neurobiology An interesting collaboration model 3
POWIEW HPC Infrastructure Massively Parallel Processing (MPP) systems Fat-node systems Versatile Symmetric Multiprocessing systems Hybrid systems (GPUs and FPGA) Selection criteria: large scale and complex scientific computations support for long computations (stable systems) mission critical computations (weather forecasting) application driven specialized architectures 4
IBM Blue Gene/P system (notos) IBM Blue Gene/P 1 rack ~14 Tflop/s High performance dedicated interconnect 4 TB RAM 178 TB storage Installation date: December 2010 5
Example: REBEL model on Blue Gene/P Main objectives of joint work: measure the performance of GADGET-3 code on Blue Gene/P architecture test different BG/P execution modes and their performance (pure MPI, MPI + pthreads) test scalability of computations on Blue Gene/P architecture prepare configuration for the 1024 3 Warsaw Universe simulation parallelize the post-processing application 6
Example: REBEL model on Blue Gene/P Computations does not scale in VN mode (1024+ nodes) Best time on 1024 nodes for SMP mode 7
IBM POWER7 system (boreasz) POWER7 IH 1 rack ~78 Tflop/s ~10.2 TB RAM 153 TB storage Installation: Autumn 2011 1TFlop nodes (4 processors) Very high memory bandwidth High performance dedicated interconnect 8
Examples: POWER7 and cosmology Blue Waters project NCSA (University of Illinois) ENZO code will be ported and optimized on Blue Waters infrastructure by the team lead by Brain O Shea More: http://www.ncsa.illinois.edu/news/stories/dark_days/ Performance measurements and optimization (3x improvement) of the Gadget code on POWER7 architecture 9
More examples: Dark matter and bright minds Collaboration with Computer Science students: Summer Schools Thesis Part-time jobs Tomasz Kłos Gadget-2 optimization on RoadRunner architecture Thesis: Cosmological simulations on hybrid architectures Currently: hired by European Space Agency as software developer Jakub Katarzyński Parallelization of post-processing application for the Gadget-3 results The work will be released and available for public use 10
How to access POWIEW resources and services Computational Grants reviewed by the ICM s scientific board Additional info and help: pomoc@icm.edu.pl Special assistance for cosmological type projects: Maciej Cytowski (m.cytowski@icm.edu.pl) Wojciech A. Hellwing (pchela@icm.edu.pl) 11
PRACE Partnership for Advanced Computing in Europe Vision: Enable and support European global leadership in public and private research and development. High Performance Computing resources and services for public research. European HPC pyramid (ESFRI) Tier-0: 3-6 European Centers for Petaflop Tier-0:? European Centers for Exaflop Tier-1: National Centers Tier-2: Regional/University Centers PRACE POWIEW 12
PRACE Resources 1st PRACE System (JUGENE) BG/P by Gauss Center for Supercomputing at Juelich 294912 CPU cores, 144 TB memory 1 PFlop/s peak performance 825.5 TFlop/s Linpack 2nd PRACE System (2010/2011) GENCI (CURIE) Intel based system by Bull ~90,000 cores 1.6 PFlop/s peak performance 3rd PRACE System 2011 HLRS (HERMIT) 4th PRACE System 2012 LRZ (SUPER-MUC) 5th and 6th Systems 2012/2013 Cineca and BSC 13
How to access PRACE resources and services Project Access: intended for individual researchers and research groups including multi-national research groups has a one year duration calls for Proposals for Project Access are issued twice yearly (May and November) 3 Regular Calls so far Example: Large Scale simulations of Ly-alpha and Ly-break galaxies in the high-z universe: Probing the epoch of reionization. Project leader: Gustavo Yepes, Universidad Autonoma de Madrid, Spain Preparatory Access: intended for resource use required to prepare proposals for Project Access applications for Preparatory Access are accepted at any time Type A: Code Scalability Testing Type B: Code development and optimisation by the applicant Type C: Code development with support from experts 14
Example: ENZO model on JUGENE Internal PRACE Call (similar to Preparatory Access) ICM joined project team as PRACE development staff ENZO model was build and optimized on JUGENE supercomputer INITS code used for creating initial conditions for ENZO runs (part of ENZO distribution) was fully parallelized by ICM Large simulations: 4096^3 particles Follow-up: proposal for 3rd PRACE Regular Call Whitepaper: Large Scale Simulations of the Non-Thermal Universe, Claudio Gheller (CINECA), Graziella Ferini (CINECA), Maciej Cytowski (ICM), Franco Vazza (Jacobs University) (available soon at http://www.prace-ri.eu) 15
Summary We invite you to collaborate with us: Parallelization and optimization New algorithm design and implementation Workflow design and implementation Assistance in getting access to POWIEW and PRACE resources Maciej Cytowski (m.cytowski@icm.edu.pl) Wojciech A. Hellwing (pchela@icm.edu.pl) 16
Links POWIEW project: http://www.wielkiewyzwania.pl PRACE project: http://www.prace-ri.eu Blue Waters project: http://www.ncsa.illinois.edu/bluewaters/ ICM site: http://www.icm.edu.pl 17