Deutscher Wetterdienst

Transcription

1 Deutscher Wetterdienst COSMO A model not only for operational weather prediction Ulrich Schättler Deutscher Wetterdienst Research and Development

2 Contents! Numerical Weather Prediction! The COSMO-Model for NWP! The COSMO-Model and other Communities! The COSMO-Model and High Performance Computing! Some Remarks on Co-Design!-Day 2011 SOS 15 Conference - Engelberg 2

3 Deutscher Wetterdienst Numerical Weather Prediction

4 ! As the national weather service of Germany, DWD tries to provide the best possible forecasts to the public (as does MeteoSwiss for Switzerland, etc)! To this end DWD runs a system of models for numerical weather prediction!-day 2011 SOS 15 Conference - Engelberg 4

5 GME, COSMO-EU, COSMO-DE Since November 2010: COSMO-DE Ensemble System with 20 members COSMO-DE "x = 2.8 km COSMO-EU "x = 7 km GME "x = 40 km COSMO-DE: Resolution: 2.8 km 421*461 * 50 grid points time step: 25 sec. Forecast up to 21 hours in 11 minutes COSMO-EU: Resolution: 7 km 665*657 * 40 grid points time step: 40 sec. Forecast up to 78 hours in about 30 minutes GME: Resolution: 30 km * 60 grid points time step: 100 sec. Forecast up to 7 days in about 60 minutes!-day 2011 SOS 15 Conference - Engelberg 5

6 Upcoming Changes to the NWP System at DWD GME COSMO-EU ICON with local refinement over Europe COSMO-DE COSMO-DE COSMO-DE-EPS 3DVar EnKF / LETKF < 2011 >!-Day 2011 SOS 15 Conference - Engelberg 6

7 Deutscher Wetterdienst The COSMO-Model for NWP

8 The History of the COSMO-Model! DWD developed the nonhydrostatic Local Model during and started operational runs on 1 December 1999.! In Summer 1999, the Consortium for Small-Scale Modelling was founded by the national weather services of Germany, Switzerland, Italy and Greece, which took over the Local Model.! Later on, Poland (in 2001),Romania (in 2007) and Russia (in 2009) joined the Consortium.! In 2006 it was decided to use COSMO as the common model name.!-day 2011 SOS 15 Conference - Engelberg 8

9 The COSMO-Model! The COSMO-Model System is a nonhydrostatic limited-area atmospheric prediction system.! The following components are included! Choice of Leapfrog or Runge-Kutta dynamical core! Physical parameterizations for subgrid scale processes like radiation, microphysics, turbulence, etc.! Data assimilation based on the nudging scheme! Diagnostic package for monitoring the forecast quality! I/O in Grib or NetCDF format! Chemical module for aerosols, reactive tracers and pollen (optional, work in progress)!-day 2011 SOS 15 Conference - Engelberg 9

10 The COSMO-Model Complexity Pressure Adiabatic processes Ozone Winds Diffusion Surface roughness Surface temperature Temperature Momentum flux Radiation Snow temperature Water vapour Sensible heat flux Interception storage Cumulus convection Cloud-water Snow Latent heat flux Cloud-ice Grid scale precipitation Ground humidiy Numerical weather prediction models have to deal with a variety of different processes a wide range of space and time scales Snow melt!-day 2011 SOS 15 Conference - Engelberg 10

11 The COSMO-Model Complexity! From a Software Engineering point of view:! software packages are implemented in Fortran90 (and higher) with O (250000) lines of code! some special tasks (I/O of binary data) are written in C! models are parallelized by using the domain decomposition strategy! up to now only a pure MPI implementation exists! first tests with a mixed implementation MPI OpenMP have been done with the COSMO-Model, but with limited success (loop level vs. block level parallelization; need to restructure the components to get bigger blocks)!-day 2011 SOS 15 Conference - Engelberg 11

12 The COSMO-Model Users (NWP)! 7 National Weather Services (Germany, Greece, Italy, Poland, Romania, Russia, Switzerland)! O(30) Universities and Research Institutes! 3 (by now) Licence Takers!-Day 2011 SOS 15 Conference - Engelberg 12

13 DWD (Offenbach, Germany): NEC SX-8R, SX-9 COSMO NWP-Applications Roshydromet (Moscow, Russia), SGI NMA (Bucharest, Romania): Still in planning / procurement phase MeteoSwiss: Cray XT4: COSMO-7 and COSMO-2 use MPI- Tasks on 246 out of 260 quad core AMD nodes!!!!!!!!!!!!! ARPA-SIM (Bologna, Italy): Linux-Intel x86-64 Cluster for testing (uses 56 of 120 cores)! USAM (Rome, Italy): HP Linux Cluster XEON biproc quadcore System in preparation IMGW (Warsawa, Poland): SGI Origin 3800: uses 88 of 100 nodes ARPA-SIM (Bologna, Italy): IBM pwr5: up to 160 of 512 nodes at CINECA COSMO-LEPS (at ECMWF): running on ECMWF pwr6 as member-state time-critical application HNMS (Athens, Greece): IBM pwr4: 120 of 256 nodes!-day 2011 SOS 15 Conference - Engelberg 13

14 COSMO NWP-Research The following Universities / Research Institutes have an active partnership with DWD: Academy of Sciences, Czech Republic Alfred-Wegener Institut, Bremerhaven, Germany Freie Universität Berlin, Germany Institut für Physik der Atmosphäre, DLR, Oberpfaffenhofen, Germany Institut für Troposphärenforschung, Leipzig, Germany Rheinische Friedrich-Wilhelms-Universität Bonn, Germany University College Dublin, Ireland Johann-Wolfgang-Goethe Universität Frankfurt, Germany Leibniz-Universität-Hannover, Germany Universität Karlsruhe / Karlsruher Institut für Technologie, Germany Universität Köln, Germany Universität Leipzig, Germany Johannes-Gutenberg-Universität Mainz, Germany Ludwig-Maximilians-Universität München, Germany Carl von Ossietzky Universität Oldenburg, Germany Universität Trier, Germany There are also other institutes collaborating with the other National Weather Services of COSMO.!-Day 2011 SOS 15 Conference - Engelberg 14

15 COSMO Licences Since 2010 a COSMO Licence is available to run the COSMO-Model for daily operational productions. Oman Brazil United Arab Emirates!-Day 2011 COSMO Training Abu Dhabi 15

16 Deutscher Wetterdienst The COSMO-Model for other Applications

17 History of the COSMO-CLM (CCLM) As early as 2002, a group of research institutes and universities started to use the COSMO-Model as tool for long-term simulations. In 2006, a common code could be used for both applications. Although different subversions do exist, it is always a common code!!-day 2011 COSMO Training Abu Dhabi 17

18 CLM Community 2010!-Day 2011 COSMO Training Abu Dhabi 18

19 History of the COSMO-ART! ART: Aerosols and Reactive Trace Gases! Up to 2009, only local implementations of COSMO- ART existed at the KIT! In Summer 2009 (Version 4.9), the ART interfaces for the Leapfrog scheme and! in 2010 / 2011 (Version 4.18), the interfaces for the Runge- Kutta scheme have been taken over to the official version!-day 2011 SOS 15 Conference - Engelberg 19

20 COSMO-ART Users COSMO-ART Aerosols and Climate Processes, Institute of Meteorology and Climate Research

21 Application of COSMO-ART for UAE Simulated mass concentration, 4-6 February 2010 COSMO-ART Aerosols and Climate Processes, Institute of Meteorology and Climate Research

22 COSMO-ART Applications at Weather Services! In Switzerland it will be used for operational forecasting of Pollen (is under evaluation at DWD)! In April 2010 DWD and KIT were able to produce simulations of the volcanic ash event of Eyjafjalla pretty quick! The results were used to give guidance to the measurement plane of DLR, Oberpfaffenhofen.! This was a very good example of collaboration (KIT, DLR, DWD) that impressed even our politicians.!-day 2011 SOS 15 Conference - Engelberg 22

23 Deutscher Wetterdienst The COSMO-Model and High Performance Computing

24 Evolving Computer Architectures The COSMO-Model has seen some computer architectures up to now, but still uses the same programming model as 15 years ago.? Cray YMP (up to 1997) Cray T3E (1997) NEC SX-9 (2008/09) IBM pwr5 (2002) What is increasing, is the number of cores: But this does not mean, that scientists can run the same code once and forever on all available computer platforms and get an ever increasing performance!!-day 2011 SOS 15 Conference - Engelberg 24

25 The COSMO-Model on existing Computers! 21 hours COSMO-DE forecast NEC SX-9 8 Procs IBM pwr6 256 Procs Computations Dynamics Computations Physics Communications I/O % of I/O and Comm ! Code efficiency! NEC SX-9: 13 % of peak! IBM pwr6: about 5-6 % of peak! Cray XT4: about 2-3 % of peak!-day 2011 SOS 15 Conference - Engelberg 25

26 Scalability of COSMO-Europe on IBM BlueGene: 1500 "1500 " 50, 2.8 km, 3 h, no output Today this is an artificial application Results are Courtesy of Fraunhofer!-Day 2011 SOS 15 Conference - Engelberg 26

27 Problems of the COSMO-Code! I/O: Accessing the disks and the global communication involved heavily disturb scalability! The message passing within a time step includes one global communication. This might be the reason of the degradation on more than processors! The efficiency of the code is not satisfying, due to the memory boundedness of the code! NEC SX-9: 13 % of peak! IBM pwr6: about 5-6 % of peak! Cray XT4: about 2-3 % of peak!-day 2011 SOS 15 Conference - Engelberg 27

28 Tackling the Problems! In Switzerland, a national initiative has been started, to prepare scientific software for emerging massively parallel architectures. One subproject deals with the COSMO-CLM Regional Climate and Weather Modeling on the Next Generations High-Performance Computers: Towards Cloud-Resolving Simulations! COSMO accompanies this project with a Priority Project Performance on Massively Parallel Architectures! Until end of 2012 we hope to improve not only the scalability of the model but also work on the memory bandwidth problem!-day 2011 SOS 15 Conference - Engelberg 28

29 Deutscher Wetterdienst Some Remarks on Co-Design

30 How to start Co-Design! Two decades ago DWD used codes that were tailored to a special architecture / computer.! Now we are using portable codes. For operational purposes we try to find the best (general-purpose) computers to run our codes.! We are aware of the fact, that there will be (dramatic?) changes in the near future! Talking to the vendors, computer architects, software engineers, scientists with similar applications will be a start of co-design!-day 2011 SOS 15 Conference - Engelberg 30

31 Some (COSMO) Constraints! Regarding the big User Group of the COSMO-Model, the use of a portable programming model is very desirable. How can we handle architecture specific implementations? Without destroying ease-of-use too much?! The development group of the COSMO-Model contains mainly Meteorologists, Physicists and Mathematicians. Computational Scientists are not part of the core team. And also people at the universities should still be able to work on the code (although they mostly have only little experience in high performance computing).! Time that can be invested in Co-Design activities is very limited. Daily operations and user support have a high priority. This is a very critical issue!! Will it be possible to incorporate the code modifications from into the operational code?!-day 2011 SOS 15 Conference - Engelberg 31

32 Conclusions! Not only model systems, but also HPC architectures are changing! The real challenge always was and still will be to bring both together!-day 2011 SOS 15 Conference - Engelberg 32

33 Thank you very much for your attention!-day 2011 SOS 15 Conference - Engelberg 33