Infrastructure for High Performance Computing at DWD

Transcription

1 Infrastructure for High Performance Computing at DWD Dr. Henning Weber Deutscher Wetterdienst Department Systems and Operations Offenbach, Germany CAS2K7 CAS2K7-1-

2 Topics Current Status Strategy for Current Procurement Challenges of 2008 CAS2K7-2-

3 QM-Certification ISO 9001:2000 German Meteorological Service National meteorological service of Germany with headquarters at Offenbach near Frankfurt and 6 regional centers Regional Telecommunication Hub (RTH) on the Main Telecommunication Network (MTN) of the WMO Global Telecommunication System (GTS) High Performance Computing (no 25 in Germany) Member of ECMWF (Reading/UK) and EUMETSAT (Darmstadt/Germany) CAS2K7-3-

4 Deutscher Wetterdienst Administrative Advisory Board Scientific Advisory Board BI German Meteorological Service Office of the President and P PS International Affairs Press Office Detlev Frömming President Uwe Kirsche IP ST Internal Audit Wolfgang Kusch Strategy G. Schlachter K. W. Hoffmann GB PB GB TI GB FE GB WV GB KU Personnel and Technical Research and Weather Forecasting Climate and Business Infrastructure Development Environment Administration Hans-Gerd Nitz Prof. Geerd-R. Hoffmann Prof. Dr. Gerhard Adrian Dr. Gerhard Steinhorst Dr. Wilfried Thommes KU PK PB CO TI PK FE PK Wv PK Planing, Co-ordination Controlling Planning, Co-ordination Research Planning and Co-ordination and Marketing Thoma-Andrea Zingraf and Quality Assurance Co-ordination Harald Wollk opf U. Rupprecht Dr. Labendz Harry Wullbrandt PB FB WV FK KU 1 Technical Information Centre TI 1 FE ZE Remote Sensing Climate and Environand Library Systems Central Wolfgang Benesch ment Consultancy and Operations Development Dr. Paul Becker PB JU Dr. Peyinghaus WV DS Legal Affairs Prof. G.-R. Hoffmann Data Service KU 2 Gabriele Geisler FE LG Lindenberg Klaus Haderlein Climate Monitoring TI 2 Observatory LG Ulrich Otte PB 1 Observing Networks Dr. F. Berger WV 1 Personnel and Data Basic Forecasts KU 3 Freiburg and Finances FE HP Hohenpeissenberg Human Dr. V. Kurz Observatory HP Harald Weingärtner Biometeorology Norbert Wetter Dr. Fricke Dr. Paul Becker TI 3 WV 2 Service Aviation Meteorology KU 4 and Logistics FE 1 Sturm Hydrometeorology GB = Business Area Meteorological Analysis Dr. Bruno Rudolf GF = Business Unit Hans-J. Sievers and Modelling WV 3 Hamburg Board of Directors Marine Shipping KU 5 Departments H.-J. Koppert Hill Agrometeorology Divisions HEADQUARTERS in: Ulrich Otte OFFENBACH Essen Hamburg Leipzig München Potsdam Stuttgart CAS2K7-4-

5 DWD in Germany Offices and Stations 200 local offices 3760 stations Total staff of about 2500 civil servants 1.9% annual staff reduction 232 mio. expenses, 43 mio. income CAS2K7-5-

6 Meteorological Input/Output Data Observational networks (temp., pressure, wind, weather...) Remote sensing (pictures...) Radioactivity measurements Products Forecasts (maps, reports...) Warnings (severe weather...) Studies (climate, environment...) Special services Advice, Consulting 6000 products, 5000 customers, 2,7 mio. files per day CAS2K7-6-

7 DWD Sat (as part of EUMETCAST) CAS2K7-7-

8 News since 2005 GME FE13, DWD COSMO-EU (LM-E) COSMO-DE (LM-K) convection-resolving Model configuration: grid spacing: Δx 2.8 km 50 vertical levels Δt = 25 s Boundary conditions interpolated COSMO-EU forecasts Data assimilation same as COSMO-EU including Latent Heat Nudging for radar reflectivities Cloud microphysics include graupel, snow and rain Very short-range forecasts up to 21 hours Operational at DWD since April 2007 (21 hours forecast, started every 3 hours) CAS2K7-8-

9 COSMO-DE Precipitation 1st Oct. 2006, h Combination of 0 UTC & 12 UTC run accumulated precipitation in mm COSMO-EU Observation Radar FE13, DWD CAS2K7-9-

10 Operational Timetable of the DWD Forecast Models development, research maintenance system load FE13, DWD Twin-system needed! CAS2K7-10-

11 Compute-Server: Twin-System Upgrade 2006: second identical POWER5 two independent Compute Servers Each compute server with 95% availibility per month one system for operation, one for back-up and research take over time approx. 15 min. operational availability of 99.8% ( 1,5 h per month) no job preemption anymore! AIX 5.3 Nodes: 8 Proc. Power5 1,9 GHz Number : 52 Peak: 3161 Gflops/s AIX 5.3 Nodes: 8 Proc. Power5 1,9 GHz Number : 52 Peak: 3161 Gflops/s Production Server User Server CAS2K7-11-

12 PACE and the HPC-Pyramid in Germany PACE: Partnership for Advanced Computing in Europe Memorandum of Understanding concerning the establishment of a European Tier 0 HPC-Service Signed on 17th of April in Berlin Tier 0 German Tier 0 organization: HLRS+LRZ+ZAM= Gauss Centre for Supercomputing (GCS) Tier 1 Tier CAS2K7-12- moneystream: 270 Mio. 270 Mio. 270 Mio.

13 Issues of NWP-Strategy for Options: Higher resolution: DWD: Yes, but only moderate GME 40 km 30 km 20 km Bigger domains No More sophisticated algorithms Definitely Ensemble prediction systems Run an operational Short Range EPS by 2009 CAS2K7-13-

14 Development of the EPS Work has started in COSMO Scientific question: how to make the members different? Estimation of model uncertainty perturbation of model physics (e.g. cloud microphysics, turbulence, boundary layer processes, vegetation) each ensemble member is defined by one perturbed parameter Estimation of uncertainty introduced by data Disturbing initial and boundary conditions transfer of uncertainty across scales in an ensemble chain Case Study Experimental SREPS FE15, DWD CAS2K7-14-

15 Case Study for Perturbation of Physics FE15, DWD CAS2K7-15-

16 Radar Det. FE15, DWD The forecast quality remains high for every member CAS2K7-16-

17 HIRL HRM MM5 LM UM FE15, DWD Perturbation of Boundaries AVN GME IFS UM COSMO-SREPS: 4 parameter perturbations 16 members 10 km res. INM-Ensemble: Multi-Model Multi-Boundaries 20 members 25 km res. p1 p2 p3 p4 16 LM runs at 10 km CAS2K7-17-

18 INM-Ensemble (LM: 25 km) Ensemble Chain COSMO-SREPS (LM: 10 km) CAS2K7-18- COSMO- SREPS COSMO-DE-Ensemble (2,8 km) COSMO-DE Ensemble still experimental not reliable enough for operational use need GRID? FE15, DWD INM-Ensemble

19 Build a High Resolution EPS E.g. Ensemble with 23 members: control Problem: Want to remain in the convective scale FE15, DWD Every ensemble member is a full COSMO-DE with 2.8 km! We will need 23 times more - Compute power - Online storage - Mass storage - Bandwidth CAS2K7-19-

20 Data Center Replacement Sustained TFlops at DWD 1,E+03 1,E+02 1,E+01 1,E+00 1,E-01 1,E-02 1,E-03 1,E-04 1,E-05 Performance enhancement by 15 (ca. 10 Teraflops sustained performance) additional development machine needed DMRZ_E2 (dev) DMRZ_E2 (prod) DMRZ_E1 (dev) DMRZ_E1 (prod) IBM p5-575 (dev) IBM p5-575 (prod) IBM RS/6000 SP IBM RS/6000 SP Cray T3E/1200 Cray T3E/1200 Cray-C90/8 Cray-YMP/4 Cyber 76 CDC-3800 Moore's Law 1,E-06 1,E CAS2K7-20-

21 Procurement: Offenes Verfahren V005/07 Installment purchase with service contract over 4¾ years Use of hardware with guaranteed performance Software, licenses and updates Maintenance of hard- and software On-site analyst Training and support during migration Availability of 99,8% for production 12,00 10,00 Money Stream until 2012: DWD: 31.0 mio. Federal Armed Forces: 7.8 mio. 8,00 6,00 4,00 2,00 0, CAS2K7-21-

22 Procurement Schedule 3/2007 invitation to tender 01/2008 installation /2007 signing of contract 04/2008 start of operation 07/2008 shut-down of old systems CAS2K7-22-

23 Performance Requirements Phase 1 At least factor of 15 compared to today (2x 5 TF sustained) time frame for upgrade Phase 2 Time frame for upgrade: 07/ /2010 Factor to two Increase in performance, bandwidth and storage min. 2x 5 TF Computeserver 1 Computeserver CAS2K7-23-

24 Compute Server Facts 5 TF sustained 97% operation + 3 TF peak 5 TF sustained 97% research + 3 TF peak Two independent compute servers (one for operations, one for research) Research machine is backup for the operational server Each 5 Teraflop/s sustained performance (measured with LM-RAPS benchmark) Additional 3 TF peak for serial and interactive applications 200 TB storage (I/O: 2 GB/s in, 2 GB/s out) 80 TB 120 TB Availability of one operational server: at least 99.9% per month (45 downtime) CAS2K7-24-

25 Database Server Facts Main DB-Server 99.3% Backup 80 TB Metadata handled by ORACLE User data is stored in BFILEs on the file system Has to handle up to 290 TB data and up to 30 TB metadata I/O performance: 2 GB/s read/write Balanced performance of compute- and database-servers required! 320 TB Availability of one database server: more than 99.9% per month CAS2K7-25-

26 Login Computenodes Computenodes Login Main DB Global FC SAN Mirror DB CAS2K7-26-

27 Dez 14 Mass Storage Projection Phase 1 Phase 2 Beyond Speicherplatz [TB] optional compression of gridded data Jun 07 Sep 07 Dez 07 Mrz 08 Jun 08 Sep 08 Dez 08 Mrz 09 Jun 09 Sep 09 Dez 09 Mrz 10 Jun 10 Sep 10 Dez 10 Mrz 11 Jun 11 Sep 11 Dez 11 Mrz 12 Jun 12 Sep 12 Dez 12 Mrz 13 Jun 13 Sep 13 Dez 13 Mrz 14 Jun 14 Sep 14 CAS2K7-27-

28 German Meteorological Computing Centre 2007 (DMRZ) Status: September 2007 Keys: Number of Processors/Memory in GB/disc space in GB DWD (Offenbach ) Bundeswehr (Traben-Trarbach) Computeserver 2 IBM p p5/1472/14386 Computeserver IBM p p4/512/2071 GB Ethernet Switch Switch GB Ethernet Archive SUN SF p/64/19138 Routine IBM- p690 9p4/20/ESS 9p4/20/ESS Database IBM- p690 20p4/90/ESS 20p4/90/ESS Test IBM- p690 2p4/8/ESS 2p4/8/ESS Database IBM-p655 8p4/32/ESS 2p4/16/ESS Routine IBM-p655 8p4/16/ESS 6p4/16/ESS Devel/Test IBM-p655 8p4/16/ESS FC SAN Silo 1.5 PB ACS 9310 Powderhorn FC SAN ESS IBM * 13,5 TB Switch Router Switch Router 155 MBit/s Main Line BVBW-WAN Switch Router MPLS Backbone 155 Mbit/s Switch Router Site will be closed by 2009 Computeserver 1 IBM p p5/1472/14386 Fiberchannel SAN ESS IBM ,5 TB MPLS Backbone 155 MBit/s Silo 14 TB IBM 3583-L72 CAS2K7-28-

29 New Backup Site Needed DWD ECMWF Reading Potsdam Germany can use up to 4% of ECMWF computing resources for national purposes ECMWF will be backup site for the numerical models only A small data center will be set up in Potsdam Backup servers for product generation and data handling will run in Potsdam CAS2K7-29- DWD Offenbach

30 WAN of the Ministry of Transport, Building and Housing (BVBS-WAN) Establishment and Operation of a special IP WAN: Core Cooperation Networkwith 155 Deutsches Mbps, 99% Forschungsnetz availability Circular Core Network topology 1 Gbps with MPLS technology transparent back-up on ISDN dial-in lines Access Network, 98,5% availability 64, 128, 256, Kbps PPP technology ISDN dial-in as back-up Cisco components (routers, switches) SLA with Service Provider guaranteed bandwith Network Competence Center (TI16) at DWD for IVBV (Federal Government) CAS2K7-30- WSA-Cuxhaven WSA WSD WSA Nordwest, B'haven W'haven Aurich Radar Emden WSA Emden WSA Bremen FWW Bremen WSA-Meppen.... WSA-Rheine WSD-West FWW Münster/ Osnabrück... Nordrhein-Westfalen KBA Flensburg Schleswig WSÄ-Kiel (2) Kiel WSD-Nord WSA-Stralsund.. WSA-Rendsburg. Schleswig-Holstein Rostock-RGB WSA-Brunsbüttel Lübeck Rostock-Radar Mecklenburg-Vorpommern Radar Hamburg BSH BAW WSA-Lübeck Hamburg.. LBZ Hamburg. WSA-HH Schwerin SL/Nord. EBA Schwerin. RZ Hamburg. EBA HH. WSA Verden WSA Uelzen.. Niedersachsen Buch FWW Tegel BBR-Berlin LBZ Tempelhof ẸBA Berlin FWW Hannover RZ Potsdam WSD-Ost FWW. Schönefeld EBA Hannover SAF Hannover WSA Helmstedt WSA Brandenburg MO PotsdamSOR. BBZ Kleinmachnow Brandenburg Braunschweig WNA Magdeburg NBA Hannover Magdeburg. WSA Braunschweig Radar Ummendorf LRSt Magdeburg WSD Mitte Hannover,. Sachsen-Anhalt Halle Kassel EBA WSA Hann.-Münden FWW Leipzig Sachsen Halle Radar Flechtdorf. RZ Leipzig WSADresden Dresden FWW Erfurt FWW Dresden. EBA Erfurt EBA Dresden BAG ErfurtWeimar Thüringen Gera Chemnitz. BAW Zwickau Hessen WSA-Tönning WSA EBA Minden..... WSÄ-Duisburg (2) WNA-Datteln NSt Hamm RZ Essen Mülheim EBA Essen FWW Düsseldorf WSA-Köln EBA Köln BAG-Köln LBZ Köln/ Bonn. BASt, Bergisch Gl. EBA Bonn. Bonn BMVBW Bonn Ilmenau WSA Koblenz Radar Neuhaus WSA Bingen EBA Frankfurt Geisenheim Frankfurt Rheinland-Pfalz Traben-Trarbach WSDSüdwest, Mainz Radar AMilGeophys Z Offenbach Neuheilenbach Radar/FWW BTZ Langen WSA-Aschaffenburg Frankfurt. Trier ẈSA-Trier Radar Eisberg FWW Nürnberg EBA WSA Mannheim Nürnberg EBA Saarbrücken FWW Saarbrücken Heidelberg.. WSA Heidelberg. EBA. WSA-Saarbrücken Karlsruhe Bayern BAW Radar Türkheim Karlsruhe EBA Stuttgart LBZ Stuttgart RZ Stuttgart Zolling. SU WSA Archivstraße Stuttgart Augsburg Radar Führholzen. Baden-Würrtemberg S/L Süd FWW München EBA München München WSA Freiburg Freiburg MO Hohenpeißenberg Radar Feldberg MO Lindenberg BAG Dresden KBA Dresden (9/99)

31 DWD s New Building at Offenbach Building Costs: 70 Mio. Euro CAS2K7-31-

32 Infrastructure: DWD s new building at Offenbach m² 391 m² new central computer facilities CAS2K7-32-

33 New Central Computing Facilities Computer area 1100 m² 2 m high false floor Feed-in over power bus system CAS2K7-33-

34 Power up to 3 independent dynamical uninterruptible power supply systems each with 800 kw (protection against power failure) Diesel generators with l diesel with endurance run of 48 hours Feed-in over power bus system (decrease of fire load) Cooling 650 kw cooling capacity in each fire area (expandable) water cooling optional (special chillers have to be part of the tender) CAS2K7-34-

35 Only 1100 meters, but our challenge for 2008: 800 PCs 400 servers and HA Cluster 10 Gbps link for 12 month Tape library with tapes 200 TB online storage New HPC 50% new servers New central SAN Move of redundant systems one by one CAS2K7-35-

36 Summary / Challenges for 2008/09 Running ensemble predictions in early 2009 work has been started Increase of compute power by factor of 15 procurement running Merge both parts of DMRZ in Offenbach we are working on a new backup concept Need more Space, Power and Cooling new building ready in 2008 Move operations to new facility in 2008 mixture of new components and use of redundancies CAS2K7-36-