Integrating Big Data for Environmental Intelligence. Dr Sue Barrell Deputy Director (Observations and Infrastructure)

Transcription

1 Integrating Big Data for Environmental Intelligence Dr Sue Barrell Deputy Director (Observations and Infrastructure)

2 Public Safety Societal Wellbeing National Security ENVIRONMENTAL INTELLIGENCE Conclusions drawn from environmental observations and models to guide decisions and actions by governments, businesses and individuals. Economic Prosperity Environmental Sustainability

3 Reliable, resilient, national capability National/Global observing system: atmosphere, marine, water, land, space 24/7 Operational forecasting systems for weather, climate, oceans and flooding Supercomputing and massive data storage High uptime internet communications and disaster recovery Professional forecasting capability across multiple disciplines Experts out posted in the Australian Defence Force, State Emergency Centres and Aviation Operation Centres

4 Big Data Collection Where does the data come from?

5 The Bureau tracks many aspects of earth system science Weather Climate Air quality Ash dispersion Radiation dispersion Ocean dynamics Tides Storm surges Tsunamis River flood heights River flow volumes Groundwater levels Soil moisture Water storages Water quality Space weather

6 We are a part of the global observing system Each day the Bureau collects: ~ 10 million atmosphere observations suitable for ingest into our operational NWP systems. ~ 5.5 million ocean observations suitable for ingest into our operational Ocean systems. And an additional ~ 1 billion observations, per day, already arriving from the next generation of satellites such as Himawari- 8/9 from Japan Mostly received, ingested, visualised, assimilated, applied etc in (near) real time

7 One day s collection, just for weather forecasting (15M observations) Shared globally and freely under the WMO Convention SYNOPS AND SHIPS BUOYS RADIOSONDES IR AND MW SOUNDERS PILOTS AND PROFILERS AIRCRAFT WATER-VAPOUR RADIANCES OCEAN WINDS (SSM/I) SATELLITE WINDS OZONE SCATTEROMETER

8 Big Data Analysis and Processing What do we do with all that data?

9 Global Circulation Modelling in the Bureau Scenarios Climate Change Decades Centuries Forecast uncertainty Climate variability Years Outlook Predictions Guidance Months Seasons Forecasts Weather Days Week Weeks Alerts Watches Warnings Minutes Hours Emergency response Sectoral preparedness planning Strategic planning International policy negotiation

10 Supercomputer capacity and usage Numerical Weather Prediction ACCESS G, R, C, TC, Fog Emergency Environmental Response Volcanic ash Nuclear and chemical dispersion Smoke Fire Weather Fire Dangers (grassland and forest) Fire weather Special fire weather (for individual fires) Verification Hindcasts Model evaluation Bias Error Current system: Ngamai, 2013: 104 teraflops New system: Australis, 2016: 1660 teraflops Future system: Australis+, 2019: over 5000 teraflops Seasonal Climate Analysis and Prediction Multiweek forecasts Seasonal (3 monthly) predictions Ocean and Marine Analysis and Prediction BlueLink Sea surface and sub-surface temperatures Waves Currents Salinity Research BOM CSIRO (CAWCR, WIRADA) Externals (Universities) UV Index Forecast

11 Numerical Weather Prediction (NWP) Complex physical process in each cell: heat transfer solar and terrestrial radiation water vapour vertical and horizontal motion (wind) ocean temperatures

12

13 The weather model improvements roadmap to Dec 2020 D=Deterministic (one outcome) E= Ensemble (multiple outcomes) Now Grid scale (frequency) This year Mid 2018 End 2020 D-Global (G) 40km (6hrly) 25km(6hrly) 12km (6hrly) 12km (6hrly) D-Regional (R) 12km (6hrly) 12km (6hrly) 8km(6hrly) 4.5km (6hrly) D-Cyclone (TC) 12km(12hrly) 12km 12hrly) 4.5km (6hrly) 4.5km(6hrly) E-Global (GE) - 60km (6hrly) 30km (6hrly) 30km (6hrly) D-City (C) 4km (6hrly) 1.5km (6hrly) 1.5km (3hrly) 1.5km (3hrly) E-City (CE) km (3hrly) 1.5km (3hrly) D-Relocatable (X) km (3hrly) 1.5km (hourly) E-Relocatable (XE) km (hourly)

14 Big Data Delivery Where can you get your hands on our data and products?

15 Website

16 Operational continental water balance model AWRA: A national, daily time-step, 5 km resolution model Rainfall Evapotranspiration Deep Drainage Climatology Runoff Soil Moisture Inputs Model Analysis Outputs

17 Informing through mobile, video and TV

18 Big Data The Future? This is just the beginning. There is much more to come, both challenges and opportunities!

19 Balancing cost and value Meteorology has always been in 'big data' space Satellites and supercomputers great equaliser Data Systems Science Services Value through alignment of data, systems, science and service Challenge to continue to optimise balance and value

20 When does 'big data' become too big? When costs exceed value derived Assess cost across data value chain End-to-end, user-focussed

21 New, more and/or diverse sources of (big) data? Value lies in fitness-for-purpose To meet new service needs products, scale, timeliness, detail To support new research goals To feed new models and forecast systems What are the new and emerging needs and opportunities? Space-based observations Ground-based high density, ground-truth, calibration Chemistry, physics, oceans, ice, land, interfaces. Crowd-sourced data fit for what purpose? Verification, phenomena, situation awareness, rapid response Model data assimilation?

22 The challenge of supply and demand Maintaining pace with user expectations and capabilities Demand for more timely, more detailed, more interactive information and products Changing paradigms Collaboration and partnership Evolving from a linear production model to a more interdependent relationship Science Data Science Service Data Service

23 Internet of Things next stop for BoM? Enhanced environmental intelligence outcomes timely, relevant, integrated Exploring a completely new data flow and technology stack based on PUB/SUB technology Deliver truly live data from a remote site to the user via mobile or web Massively scalable, highly secure, geolocatable, mobile at its core POCs including dashboards, 3 rd party data integration Potential to deliver value to agriculture on many scales Eg Italian Institute of Biometeorology Strategies, technologies and practical solutions for resilience and sustainability related to food, climatic, meteorological and energy emergencies

24 Thank you Please visit our stand for more information!