Simplified picture of FMI Information Management System. Finnish Meteorological Institute Roope Tervo

Transcription

1 Simplified picture f FMI Infrmatin Management System Finnish Meterlgical Institute Rpe Terv

2 Finnish Meterlgical Institute Finnish Meterlgical Institute 2

3 FMI Infrmatin Management System 09/10/2015 3

4 FMI Infrmatin Management System 09/10/2015 4

5 Observatins FMI perated Surface bservatin statins 181 Rainfall measurement sites 91 Sunding statins 3 Air quality statins 29 Buys 13 Antennas fr lightning detectin 8 Maregaphs 14 Slar 8 Ttal 481 Autmatin 95% 09/10/2015 Finnish Meterlgical Institute 5

6 Observatins External External air quality 83 Skandinavian lightning detectin 29 Baltic maregraphs 24 Rad weather statins 29 Runway weather statins 63 Freign surface weather statins External water measurement statins 63 Rad weather statins /10/2015 Finnish Meterlgical Institute 6

7 Observatins Technlgy Weather statin stres the measurements lcally and calculate sme aggregatin values Sends data t quality check LAN / 3G 10 minutes instant values, 1 h aggregate values, stream Stred t Oracle database 1 master db and 1 read-nly replica Distributed t grid fr NWPs and visualizatin purpses 09/10/2015 Finnish Meterlgical Institute 7

8 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 8

9 Radars 9 radars Range km Typical file frmats HDF5, GeTiff, PNG Sending pwer W Receiving pwer 0, , W 09/10/2015 Finnish Meterlgical Institute 9

10 Radars Prducts Single radars Reflectivity Cappi ( rain intensity ) Dppler speed ( wind ) Echtp ( clud base ) Echtp ( thunder risk ) Cmpsite Reflectivity Rain intensity Rain accumulatin (1h, 3h, 6h ) Crss sectins 09/10/2015 Finnish Meterlgical Institute 10

11 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 11

12 Satellite Earth Observatin In weather, used t bserve: cluds, lw-pressure areas, winds, temperature... Operated by EUMETSAT and NOAA Receptin, prcessing and filing f data Gestatinary satellites Received in Germany Delivered t Finland via telecmmunicatin satellite Plar satellites Received in Helsinki and Sdankylä MSG DATA FLOW RATE ~ 1.5 GB/hur/satellite = 13 TB/year/satellite 09/10/2015 Finnish Meterlgical Institute 12

13 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 13

14 Lightning detectin 9 sensr in Finland, 34 in NORDLIS Practically all grund strikes detected with < 1km accuracy Als part f clud strikes Stred t climate database Can be delivered t clients in apprximately 20 secnds 09/10/2015 Finnish Meterlgical Institute 14

15 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 15

16 Numerical Weather Predictin (NWP) Physical mdelling f atmspheric Mdels develped in internatinal cnsrtiums Typical prgramming languages: Frtran, C++ Input: bservatins, bundaries frm ther mdel, satellites Output: grid frm several surfaces Surface, pressure levels, mdel levels Typical data frmats: GRIB1, GRIB2 (and NetCDF in marine mdels) 09/10/2015 Finnish Meterlgical Institute 16

17 Numerical Weather Predictin (NWP) Mdel Hirlam Harmnie Spatial Reslutin 7,5 km 2,5 km Cverage Eurpe Skandinavia Tempral Reslutin 1h 15 min Time range 48h 48h Update interval Output data 6h Surface, pressure levels, mdel levels 6h Surface, pressure levels, mdel levels 09/10/

18 Numerical Weather Predictin (NWP) ECMWF Mdel ECMWF ECMWF EPS Spatial Reslutin 15km 15 km Cverage Glbal Glbal Tempral Reslutin 1-3h 1-3h Time range 10 days 10 days Output data Surface, pressure levels Surface 51 members Eurpean Centre fr Medium-Range Weather Frecasts (ECMWF) t create prducts that supprt weather service ECMWF mdel gridded data in Meterlgical Archival and Retrieval System (MARS). It is the repsitry f meterlgical data at ECMWF (strage size f PBs f data) Update interval 6h 6h 09/10/

19 Manual Editing Meterlgists selects the best surce mdel and refines frecast The frecast is updated as ften as necessary Output is used as primary data fr Scandinavian area Sftware SmartMet is dne at FMI (C++) The data is handled in grid The sftware is installed t several freign cuntries Meterlgists draws an analysis maps Frnts, jets Sftware Mirwa is dne at FMI (Java) The data is handled as bjects Meterlgists generates warnings Dne with Mirwa Meterlgists des aviatin frecasts METARS, TAFs, SWC charts Several ther tasks Writing texts, generating ice advisries, cnsulting abut weather 09/10/2015 Finnish Meterlgical Institute 19

20 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 20

21 String Data Ttal stred data > 800 TB Operative disk strage 400 TB real time strage Archive system ~400 TB Majrity f stred data is radar, satellite and NWP data Operatinal databases Oracle, PstgreSQL, MySQL and Redis Usability requirement > 99,5 % Data size grws expnentially in near future FMI utilizes ECMWF data strage in daily peratins Finnish Meterlgical Institute 21

22 String Data In-situ weather measurement 1 TB Weather radar data 50 TB NWP mdel gridded data (FMI) HIRLAM 120 TB HARMONIE 230 TB Satellite image data (FMI) Glbsnw 32 TB Other 50 TB Climate mdel data (FMI) 29 TB Other mdels SILAM 100 TB Tuuliatlas 21 TB Other 300 TB In-situ weather measurement Weather radar data HIRLAM 120 TB Harmnie Glbsnw (Satellite stred at FMI) Other satellites stred at FMI Climate mdel data (FMI) Finnish Meterlgical Institute 22

23 FMI Infrmatin Management System 09/10/2015 Finnish Meterlgical Institute 23

24 Data Server Brainstrm Data and prduct server fr MetOcean data Prduced by FMI t meet requirements f web based services High capacity & availability Data is extracted and prducts generated always n-demand Several utput frmats WMS, WFS 2.0 JSON, XML, ASCII, HTML, SERIAL INSPIRE Cmpliant Operative since 2008 FMI client services (since 2008) Finnish Meterlgical Institute (FMI) Open Data Prtal (since 2013) Finnish Meterlgical Institute 24

25 Data Server Usage Basis f FMI prduct generatin FMI Open Data Backend fr Prtal & FMI clients web INSPIRE Data services Services Integratin t clients Backend fr systems mbile applicatins Backend fr FMI public Backend fr pages FMI Client Services Finnish Meterlgical Institute 25

26 Data Server Architecture Frntend Lad balancing with service discvery Prduct cache Backend Different backends may cntain different services Plugin-based architecture Engines prvide shared access t the data Plugins prvide different APIs based n data prvided by the engines Finnish Meterlgical Institute 26

27 Data Server FMI Setup 2 frntends RAM: 256G CPU: 24x 2.10GHz OS: RHEL7 5 backends RAM: 12G CPU: 24x2.50GHz OS: RHEL7 Lad Balancer F5 BIG IP 11 Databases Master Observatin Database: Oracle Lcatin Database: PstGIS Finnish Meterlgical Institute 27

28 Data Server Perfrmance FMI setup > req/day Peaks ver 650 req/s Average respnse times varies depending n the request: WFS: 140 ms/req, WMS: 130 ms/req, Timeseries: 30 ms/req, Autcmplete: 4 ms/req Overall average: 70 ms/req Over 99,95% availability N limits have been fund yet Slw query queue easy t DS due t pssibility t large data requests and heavy peratins Dn t affect t fast query queue Finnish Meterlgical Institute 28

29 FMI Open Data Finnish Meterlgical Institute pened its data in Basically everything that FMI has prperty rights was pened. Data is prvided in freely in machine readable frmat Finnish Meterlgical Institute 29

30 FMI Open Data Prtal FMI Open Data Prtal fllws INSPIRE requirements. Meta data Services Data Mdels The very same data prtal wrks as Open Data and INSPIRE prtal Finnish Meterlgical Institute 30

31 Example f Data Sets Data set Descriptin Time Interval Weather Observatins Sun Radiatin Marine Observatins Weather Radars Lightning Temperature, Wind, Humidity, Grund Temperature UV, Shrt and Lng Term Radiatin Waves, Sea Temperature, Sea Level Estimated publish date 10 min Open, lder data t be added 1 min Open 1 h Open Precipitatin Rate, 5 min Open, Precipitatin Amunt lder data t be added Thunder Strikes in Finland 5 min Open Finnish Meterlgical Institute

32 Example f Data Sets Data set Descriptin Time Interval Estimated publish date Real Time Observatins Climatlgical Observatins Climatlgical Observatins Climatlgical Reference Real Time Observatins frm specific lcatin(s) Dayly and mnthly temperature mean and extreme values frm weather statins Mnthly temperature and precipitatin rate mean values interplated t grid Climatlgical Reference. Temperature, humidity, pressure, precipitatin amunt and snw depth. AWS 2010 Sundings 1959 Flashes 1998 Sea Level 1971 Waves Open Open Reference seasns: Open lder data will be added Open Finnish Meterlgical Institute

33 Example f Data Sets Data set Descriptin Time Interval Estimated publish date Weather frecast mdel HIRLAM RCR Sea frecast mdels Envirnmental Mnitring Facilities Aviatin Observatins Grund & mast bservatins Pint frecasts and grid data Sea level pint frecasts, Wave (WAM) and current (HBM) as grid data Weather bservatin statins, radars Latest mdel run (4 times a day) 0 54 h Latest mdel run (4 times a day) h Open Open 2015 METAR 30 min pen Special bservatins frm grund and masts 2016 /Open Finnish Meterlgical Institute

34 Example f Data Sets Data set Descriptin Time Interval Estimated publish date Air Quality Observatins Silam Mdel Air Quality Observatins 1h Dispersin Mdel fr Air Quality, Frest Fire and Pllen Latest mdel run (nce a day) 0 96h HELMI Ice Mdel Ice frecast mdel Latest mdel run (4 times a day) h 2015 pen Sundings Rad Weather Observatins (LIVI) Temperature, Humidity, Pressure, Wind frm grund t 25 km height Rad Weather Observatins 2 times a day min pen Finnish Meterlgical Institute

35 Registratin Registratin is required t use View and Dwnlad Services Wrking address is the nly mandatry infrmatin After registratin the user gets an API key which have t be added int all requests GET parameter fmi-apikey= & Header fmi-apikey; Part f url /wms? One can create several API keys with ne Finnish Meterlgical Institute

36 Usage Limits With ne API key it s allwed t d at mst requests per day t Dwnlad Service d at mst requests per day t View Service d at mst 600 requests per 5 minutes t bth services If all bservatins frm ne time step is calculated t as ne, little ver new data sets are published daily S, with ne API key it s allwed lad everything nce View service can be used fr testing but can nt be used as a back end fr ppular clients Finnish Meterlgical Institute

37 MetLib Open surce JavaScript library prduced by Finnish Meterlgical Institute Helps users t lad and use the data Easy requests Cache Parse the data t as JSON Supprts multi pint cverage data frmat Pythn versin is n the list Finnish Meterlgical Institute

38 View Service (WMS) Based n GeServer Only the mst cmmn layers published Catalg Service (CSW) Based n GeNetwrk Finnish Meterlgical Institute

39 DMZ Lad Balancer Server 1 Server 2 Server 3 GS 1 GS 2 GS 3 GS 1 GS 2 GS 3 GS 1 GS 2 GS 3 Intranet Database Data (NFS) Cnfiguratin (NFS) Cnfiguratin GeServer

40 Dwnlad Service (WFS 2.0) Web Feature Service (WFS) 2.0 Simple Prfile Based n stred queries Predefined data sets with pssibility fr additinal parameters (i.e. time and area) Based n Brainstrm data server Finnish Meterlgical Institute

41 Data Mdels Observatins and pint frecasts as GML The same data is published in: MultiPintCverage MeasurementTimeSeries SimpleFeature Gridded data is prvided in apprpriate binary frmat (Grib, NetCDF, GeTiff ) WFS members cntains the metadata envelpe with a link t a actual data Finnish Meterlgical Institute

42 Data Mdels aviatin bservatins IWXXM / METARS Aviatin weather repsts are delivered as IWXXM New data mdel cming int use in aviatin Cnsists f the same elements than ther messages m:phenmenntime, m:prcedure, m:featureofinterest, m:result Cntent f the METAR is in m:result part as extracted int XML elements riginal, ld fashin, METAR string Finnish Meterlgical Institute

43 Weather data is it big? Snijders, C.; Matzat, U.; Reips, U.-D. (2012). "'Big Data': Big gaps f knwledge in the field f Internet". Internatinal Jurnal f Internet Science 7: Finnish Meterlgical Institute 43

44 Meterlgical Data as Part f Big Data Imprtant cnstituent Weather impacts t everything and each ne f us. New technlgy makes us even mre fragile f weather. Retail trade Advertising Traffic Outdr activities Energy Finnish Meterlgical Institute 44

45 Resurces