MANUFACTURING WEATHER FORECASTING SIMULATIONS ON HPC INFRASTRUCTURES

Transcription

1 MANUFACTURING WEATHER FORECASTING SIMULATIONS ON HPC INFRASTRUCTURES LADISLAV HLUCHÝ V. ŠIPKOVÁ, M. DOBRUCKÝ, J. BARTOK, B.M. NGUYEN INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES ECW ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016

2 PARTNERS IISAS: INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES (ACADEMIC SECTOR) MICROSTEP-MIS: MONITORING AND INFORMATION SYSTEMS (COMMERCIAL SECTOR) IMS MODEL SUITE: COMPLEX SOFTWARE SYSTEM FOR METEOROLOGY AND CRISIS MANAGEMENT THIS PAPER PRESENTS A PART OF MANUFACTURING WRF ON HPC INFRASTRUCTURE FOR IMS MODEL SUITE

3 WRF - WEATHER RESEARCH AND FORECASTING DESIGNED FOR RESEARCH AND OPERATIONAL PURPOSES NUMERICAL WEATHER PREDICTION ATMOSPHERIC SIMULATION TWO DYNAMIC SOLVERS ARW: ADVANCE RESEARCH WRF NMM: NON-HYDROSTATIC MESOSCALE MODEL FLEXIBLE AND PORTABLE CODE SEQUENTIAL PARALLEL (MPI) WITH OR WITHOUT MULTI-THREADING SUPPORTS A TWO-LEVEL DOMAIN DECOMPOSITION AT FIRST INTO PATCHES FOR DISTRIBUTED MEMORY, THEN WITHIN EACH PATCH MULTI-THREADING IS APPLIED FOR SHARED MEMORY

4 OBJECTIVES DEVELOPMENT OF MANAGEMENT TOOLS TO FACILITATE THE EXECUTION OF THE WRF SIMULATION PROCESS ON HPC INFRASTRUCTURES LOCAL HPC CLUSTER GRID INFRASTRUCTURE (EGI) PERFORMANCE INVESTIGATION OF PARALLEL WRF MODELS TO FIND OUT THE MOST SUITABLE CONFIGURATION WITH THE GIVEN INPUT SCENARIO FOR 3D METEOROLOGICAL MODELLING MPI MPI + OPENMP THE NUMBER OF COMPUTE NODES, CORES, MPI PROCESSES, OPENMP THREADS THE MANAGEMENT TOOLS ARE ALSO USED FOR PARAMETER TUNING OF THE MODELS (FOR IMS BY MICROSTEP-MIS) THAT REQUIRES TENS OF EVALUATIONS OF THE PARAMETERIZED MODEL ACCURACY EACH EVALUATION OF THE MODEL PARAMETERS REQUIRES RE-RUNNING OF THE HUNDREDS OF METEOROLOGICAL SITUATIONS COLLECTED OVER THE YEARS AND COMPARISON OF THE MODEL OUTPUT WITH THE OBSERVED DATA

5 3D METEOROLOGICAL MODELLING DOMAINS - WEATHER MODELLING HORIZONTAL, VERTICAL AND TIME RESOLUTION, SO THE MODEL CAN CATCH LOCAL CONDITIONS METEOROLOGICAL INITIAL AND BOUNDARY CONDITIONS FROM THE GLOBAL MODEL GFS (GLOBAL FORECASTING SYSTEM) OF US NATIONAL WEATHER SERVICE THE SETTING ENABLED TO MODEL THE ARABIAN PENINSULA WEATHER THE UPPERMOST DOMAIN WITH THE RESOLUTION 50X50 KM THE FINAL DOMAIN WITH THE RESOLUTION 1.8 KM, AROUND DUBAI AND ABU DHABI

6 WRF SIMULATION Pi MPI process Tj OpenMP thread WRF SIMULATION CONSISTS OF MANY EXECUTABLE PROGRAMS VARIOUS TYPE AND COMPLEXITY, SEQUENTIAL AND PARALLEL TAKING A DIFFERENT NUMBER OF PROCESSOR CORES FOR EXECUTION WRF WORKFLOW - DAG GRAPH (JOB 1) WPS PREPROCESSING, (JOB 2) WRF MODELING, (JOB 3) UPP POST-PROCESSING

7 WRF WORKFLOW MORE DETAILS JOB 1 - WPS PREPROCESSING: CONVERSION OF INPUTS FROM GRIB TO NETCDF FORMAT USING GEOGRID.EXE (SERIAL/MPI) UNGRIB.EXE (SERIAL) METGRID.EXE (SERIAL/MPI) JOB 2 - WRF MODELING - NUMERICAL MODELING USING REAL.EXE INITIALIZATION - REAL DATA PREPROCESSOR (MPI/MPI+OPENMP) WRF.EXE NUMERICAL INTEGRATION - ARW SOLVER (MPI/MPI+OPENMP) JOB 3 - UPP POST-PROCESSING CONVERSION OF OUTPUTS FROM NETCDF TO GRIB FORMAT USING UNIPOST.EXE (SERIAL/MPI) IN A NESTED CYCLE FOR ALL HOURS OF THE PREDICTED TIME PERIOD THERE IS NO DEPENDENCY BETWEEN PROCESSING DATA OF INDIVIDUAL HOURS, SO, THE JOB CAN BE STRUCTURED AS A PARAMETRIC STUDY (PS), WHERE EACH SUB-JOB HANDLES A SECTION OF THE TIME PERIOD

8 WRF WORKFLOW EXECUTION STARTS ON THE UI MACHINE THROUGH THE INVOCATION OF THE WRF WORKFLOW-MANAGER ENCOMPASSED WITH NEEDED INPUT PARAMETERS IS REALIZED WITHIN THE RUNNING-ENVIRONMENT LOCATED IN THE SHARED ADDRESS SPACE WHICH HAS THE DIRECTORY STRUCTURE GEOG CFG PARM BIN INPUT_ARCH OUTPUT_ARCH WPS_RUN MODEL_RUN POSTPR_RUN GEOGRAPHICAL DATA, SEVERAL GEO-TABLES CONFIGURATION FILES FOR INPUT SCENARIO AND SIMULATION OPTIONS UPP POST-PROCESSING PARAMETERS RUN-SCRIPTS AND EXECUTABLES INPUT DATA FILES OUTPUT DATA FILES WPS PREPROCESSING WRF MODELING UPP POST-PROCESSING

9 IISAS HPC CLUSTER v HARDWARE CONFIGURATION 52X IBM DX360 M3 (2X INTEL 48 GB RAM, 2X 500 GB SCRATCH DISK), 2X IBM DX360 M3 (2X INTEL 48 GB RAM, 2X 500 GB SCRATCH DISK, NVIDIA TESLA M2070: 6 GB RAM CUDA CORES), 2X X3650 M3 MANAGING SERVERS (2X INTEL 48 GB RAM, 6X 500 GB DISKS), 4X X3650 M3 DATA- MANAGING SERVERS (2X INTEL 48 GB RAM, 2X 500 GB DISKS, 2X 8 GBPS FC), 1X X3550 M4 SERVER (1X INTEL 8 GB RAM, 2X 500 GB DISKS), INFINIBAND 2X 40 GBPS (IN NODES), 2X DS3512 WITH 72TB DISKS v SOFTWARE INSTALATION WRF PACKAGE VERSION (WRF, WPS, TERRESTRIAL DATASETS), UPP VERSION 3.0, LIBRARIES NETCDF 4, JASPER 1.7, GNU COMPILERS VERSION (GFORTRAN, GCC, OPENMP LIBRARY), OPEN MPI VERSION

10 PERFORMANCE RESULTS WRF MODEL: SEQUENTIAL ON THE LOCAL CLUSTER PREDICTION TIME PERIOD 3 HOURS IN THIS PAPER FOR SCALING WRF SIMULATIONS FOR TESTING PURPOSE WITH GIVEN HW/SW CONFIGURATIONS 48 HOURS IN REAL SIMULATIONS (MICROSTEP-MIS) TO MODEL THE ARABIAN PENINSULA WEATHER THE NEED OF HPC TO ACCELERATE SIMULATIONS Number of nodes Number of cores per node Execution time hh:mm:ss WPS :39:54 WRF :57:53 UPP (2 jobs) :03:48 Complete simulation process 16:41:35

11 PERFORMANCE RESULTS WRF MODEL: MPI ON LOCAL CLUSTER FIXED NUMBER OF CORES PER NODE Number of nodes Number of cores per node Number of MPI processes Execution time hh:mm:ss WPS :04:22 WRF :36:33 WRF :27:01 WRF :49:03 WRF :30:13 WRF :20:47 WRF :13:57 UPP (2 jobs) :01:44 Complete simulation process (best) 00:20:03

12 PERFORMANCE RESULTS WRF MODEL: MPI + OPENMP ON LOCAL CLUSTER FIXED MPI PROCESSES Number of nodes x cores Number of MPI processes ( per node) Number of OpenMP threads Execution time hh:mm:ss WRF 8x12 32 (4) 2 00:31:31 WRF 16x12 32 (2) 4 00:20:52 WRF 16x12 32 (2) 6 00:17:21 WRF 32x12 32 (1) 8 00:15:47 WRF 32x12 32 (1) 10 00:15:15 WRF 32x12 32 (1) 12 00:15:20

13 PERFORMANCE RESULTS WRF MODEL: MPI + OPENMP ON LOCAL CLUSTER FIXED NUMBER OF OPENMP THREADS Number of nodes x cores Number of MPI processes (per node) Number of OpenMP threads Execution time hh:mm:ss WRF 8x12 32 (4) 3 00:24:44 WRF 12x12 48 (4) 3 00:19:28 WRF 16x12 64 (4) 3 00:17:27 WRF 24x12 96 (4) 3 00:13:49 WRF 32x (4) 3 00:12:24 WRF 16x12 32 (2) 6 00:17:21 WRF 24x12 48 (2) 6 00:14:31 WRF 32x12 64 (2) 6 00:12:09 WRF 40x12 80 (2) 6 00:12:01

14 WRF MODEL MPI ON GRID INFRASTRUCTURE EGI WRF RUNNING-ENVIRONMENT IN ITS INITIAL STATE, ALL EXECUTABLES AND INPUT FILES ARE STORED IN GRID STORAGE ELEMENT (SE), FROM WHICH THEY ARE DOWNLOADED GEOGRAPHICAL DATASETS (174 GB) ARE LOCATED IN CLUSTER SHARED ADDRESS SPACE, THEY DO NOT PARTICIPATE ON THE DATA TRANSFER GRID WRF WORKFLOW IS DESIGNED AS ONE GRID JOB ENCAPSULATING ALL TASKS: WPS+WRF+UPP MPI PROGRAMS ARE EXECUTED USING MPI-START OUTPUT OF SIMULATION IS UPLOADED TO STORAGE ELEMENT (SE) TIME OVERHEAD BY DATA TRANSFERS BETWEEN CE AND SE: 2 MINUTES Grid UI Grid User Interface WMS Workload Management System VO Virtual Organization CE Computing Elements GG Grid Gate LRMS Local Resource Management System WN Working Node SE Storage Element PBS Portable Batch System

15 CONCLUSION MANAGEMENT TOOLS ARE BUILT AND FULFILL DESIGNED PURPOSES TO LOCATE THE OPTIMAL CONFIGURATION WITH GIVEN SCENARIO FOR IMS MODEL PARAMETER TUNING (MICROSTEP-MIS) HYBRID PROGRAMMING MODEL (MPI + OPENMP) SEEMS A NATURAL FIT FOR THE WAY MOST CLUSTERS ARE BUILT TODAY THE GRID OVERHEAD IS CAUSED MAINLY BY THE TRANSFER OF BIG FILES BETWEEN THE SE AND CE

16 FUTURE DIRECTIONS Ø GRID AT THE MOMENT, IN EUROPEAN GRID INFRASTRUCTURE (EGI), ONLY A FEW GRID SITES AND VIRTUAL ORGANIZATIONS (VO) ARE SUPPORTING MPI AND OPENMP APPLICATIONS Ø CLOUD PERFORMANCE OVERHEAD ASSOCIATED WITH VIRTUALIZATION OF INTERCONNECTION NETWORK WRF IS REPORTED TO RUN ON VIRTUALIZED INFINIBAND INTERCONNECT WITH ONLY 15% OVERHEAD WHICH MAKES FULLY VIRTUALIZED HPC CLUSTERS VIABLE SOLUTION Ø ACCELERATORS PARTS OF WRF WERE PORTED TO NVIDIA GPU AND INTEL XEON PHI WITH PROMISING RESULTS

17 THANK YOU FOR YOUR ATTENTION MANUFACTURING WEATHER FORECASTING SIMULATIONS ON HPC INFRASTRUCTURES INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES ECW ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016