Big Data at ECMWF Providing access to multi-petabyte datasets Past, present and future

Transcription

1 Big Data at ECMWF Providing access to multi-petabyte datasets Past, present and future Baudouin Raoult Principal Software Strategist ECMWF Slide 1

2 ECMWF An independent intergovernmental organisation established in 1975 with 20 Member States 14 Co-operating States Slide 2

3 Slide 3

4 Slide 4

5 Major assimilated datasets Surface stations Polar, infrared Radiosonde balloons Polar, microwave Aircraft Geostationary, IR Slide 5

6 The forecast process Slide 6

7 ERA-20C completed: Climate monitoring of the 20th Century Using >5% of ECMWF s computing power Assimilating billions of observations Producing 2,400 global forecasts per day Generating 1 PB of reanalysis data in 200 days Currently serving reanalysis products to 20,000 users Slide 7

8 Online catalogue, quick-looks and data European Air Quality Global atmospheric composition Surface fluxes: greenhouse gases, fires, emissions Radiation and ozone layer Slide 8

9 ECMWF products ECMWF currently receives 300 million observation from 130 sources daily. ECMWF operational models produce 13 millions fields daily, for a total of around 8 TB. 77 million products disseminated ever day, for a total of 6 TB. Slide 9

10 What is Big Data? Wikipedia: Big Data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization. Gartner: Big Data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization. (The 3 V of Big Data). I would like to add two S : Scalability and Sustainability. Slide 10

11 V is for Volume (or coping with exponential growth) Deletion of 1 PB Slide 11

12 V is for Velocity (or coping with exponential growth, part 2) ECMWF s archive grows exponentially: At ECMWF, r is around 0.5, which is a 50% increase per year The daily amount of data added to the archive grows exponentially at the same rate! In 1995, the size of the archive was increasing at a rate of 14 TB/year. In 2014, the size of the archive increases at a rate higher than 60 TB/day (with peaks at 100 TB). Slide 12

13 V is for Variety (or coping with complexity) 10T 1T X-MP/4 Y-MP/8 C90/12 C90/16 T106L19 T213L31 T106L16 VPP VPP T319L50 T319L31 VPP5000 IBM-P4 IBM-P5 IBM-P5+ IBM-P6 T319L60 T511L60 00Z EPS T799L91 Weekly Monthly T1279L91 Ensemble data assimilation 00Z 10 day FC Overlap, CalVal Vareps/Monthy 4d-Var Model errors 100G 4D-var increments SCDA 4D-Var DCDA Wave EFIs EPS PT levels DCDA Wave 10G Weekend EPS Extra fields, new gaussian grid End sensitivity Wave proba. SCDA Waves SCDA Forecast Wave 4V PT and PV levels SCDA Forecast EFIs Multi-Analysis 4D-Var Wave EPS SCDA Analysis Errors if FG, surface 1G 00Z Run 00Z Run Sensitivity Other centers OI Tubes Errors in AN and FG NCEP EPS Errors in FG Clusters EPS Waves FG Probabilities Ensemble means & stdev 100M FC Model levels SSTs Waves Chernobyl TOGA FC 3DVar 4DVar 12 Hour 4DVar DCDA Vareps/Monthy EDA FC Pressure levels 50 Members EPS EPS 15 days 10M Slide 13

14 ECMWF s Meteorological Archival and Retrieval System 28 years in existence A managed archive MARS is not a file system Users are not aware of the location of the data Retrievals are expressed in meteorological terms An archive, not a database Metadata online Data offline (automated tape library) Slide 14

15 ECMWF s Meteorological Archival and Retrieval System directly addressable objects Unique hypercube-based indexing Data is kept forever: For many studies, a dataset becomes useful once enough data has been accumulated Deleting old data in an exponentially growing archive is meaningless 200 million objects/65 TB added daily 7000 registered users 650 active users, 100 TB retrieved per day, in 1.5 million requests Slide 15

16 Scalability and sustainability Distributed service-oriented architecture, to scale out Queues and priorities to ensure quality of service and scaling with the demand Indirection is the key to scalability: Allow services to be modified/redeployed Allow data to be moved to different media/storage without any impact on users It has allowed us to migrate several times during the past decades: MVS to AIX, AIX to Linux, single server to clusters CFS to TSM to HPSS PL/I and Fortran to C/C++ Slide 16

17 Continuous evolution Change is driven by new needs and expectations Users are not domain experts anymore users expect information at Google speed ECMWF is continuously evolving its data delivery methods and services to cater for the new user requirements Interoperability is key Follow standards and governance to enable interoperability OGC, INSPIRE, ISO 19xxx series, NetCDF-CF, WMO Information System, GEOSS Provide high-level services on the data Data portals for data discovery Web services, REST APIs for data retrieval and manipulation Close-to-the-data processes Slide 17

18 Data exploring tools Slide 18

19 Interoperability Slide 19

20 The next steps The era of pushing the data to the user is coming to an end The volumes involved are too large We live in a post-pc era Users want to access data from smart devices anywhere, anytime, any device and share their results. We need to bring the user processing to the data: Cloud Computing is now mature enough to implement operational services We need to build a platform as a service (PaaS), on which to software as a service (SaaS) solutions for environmental data and products Slide 20

21 Thank you Slide 21 21