The Digicene: the Age of Big Data in the Geosciences

Transcription

1 The Digicene: the Age of Big Data in the Geosciences Lee Allison Arizona Geological Survey USGIN Foundation, Inc. Earth Data Science in the Era of Big Data and Compute April 30, 2015

2 A system that works for one geologist, in the field and at their desk and for users of streaming, High Performance Computing

3 Data Long tail of science Pb Tb ~20% of earth scientists use/need HPC capabilities; 80% rely dominantly on desktop software and data sets they collect themselves Gb Mb Mainstream Scientists

4 Big Data: Data Integration ( interoperability ) Capabilities for moving data between data warehousing, business analytics, master data management, enterprise applications, and custom applications. Integrating structured data in relational databases with social media data, weblogs, and various unstructured data Dain Hansen, Oracle Corp 2012

5 Big Data is not just large data sets Digitizing, organizing, analyzing, modeling, visualizing [small] data from disparate sources

6 National Geothermal Data System Oregon Institute of Technology Geo Heat Center Boise State Free online access to: Stanford Reservoir Engineering AZGS: 50 State Geological Surveys U of Nevada Reno University of Utah Energy & Geosciences Institute U.S. DOE Geothermal Data Repository hosted at NREL U.S. Geological Survey Maps, data, & documents from 65+ providers nationwide Southern Methodist University Distributed network >10 million data records >3 million oil & gas wells >47,000 maps & reports Powered by USGIN

7 For mainstream science Not centralized big iron, but decentralized data wrangling Not one ring to rule them all but small pieces loosely joined Mass democratisation of the means of access, storage & processing of data A distributed ecosystem of information, an ecosystem of small data Distributed models not centralized ones Collaboration not control, and small data not big data Rufus Pollock Open Knowledge Foundation April 2013

8 Small data culture Big data environment Barriers to discovery, access, and integration of data have shaped the scientific practice since it s beginnings Most of us work in small data because there is no viable alternative Tackling interdisciplinary/multidisciplinary problems was not realistic Many large geoscience databases are not really big data Open data access and interoperability are changing the paradigm Digitization, integration, and modeling are creating unprecedented opportunities and challenges The new generation of geoscientists increasingly will need data science skills

9 Turning legacy data.

10 .into this NERC 2008

11 Creating common standards and protocols Engaging the vast number of distributed data resources Establishing practices for recognition of and respect for intellectual property Developing simple data and resource discovery and access systems Building mechanisms to encourage development of web service tools and workflows for data analysis Brokering the diverse disciplinary service buses Creating sustainable business models for maintenance and evolution of information resources Integrating the data management life-cycle into the practice of science

12 Big Data and the Geoscience Profession There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions. McKinsey Global Institute, 2011

13 Global Cyberinfrastucture and Data Management Initiatives Environment Earth Observation Geoscience Marine Petroleum/ Energy All FOCUS EarthCube (US only) Earth Science Information Partners (ESIP) (US) GeoSeas (EU) Renewable Energy Agency US Inspire (EU only) National/ Continental IGSN (EU, US, AUS) ODIP Oceans Data Interoperability Platform (US, EU, AUS) CoopEUS (EU, US) Cross Continental Belmont e- Infrastructure GEO/GEOSS Geoscience Information Council OneGeology Energistics National Data Repositories CODATA ICSU World data System Research Data Alliance Global USGIN involvement Ones GA is actively involved in Ones GA follows Ones mentioned in OneGeology agenda Ones not on OneGeology Agenda Geoscience Australia

14 Geothermal Petroleum Mining Geological Surveys Academia Professional Organizations National Geothermal Data System (NGDS) IRENA Global Atlas African Rift Geotherm al (ARGeo) Standards Leadership Council National Data Repositories (NDR) African Minerals Geoscience Initiative OneGeology USGIN USGS Community for Data Integration EarthCube Belmont Forum Intl Geo Sample Numbering (IGSN) IUGS CGI GeoSciML Coalition for Publishing Data in the Earth and Space Sciences Geoscience Cyberinfrastructure

15 Everything digital, online, & interoperable Productivity increase Enabling inter- and multidisciplinary research Enabling unheard-of analytical, computational, visualization, and modeling

16 Digitization Open access Integration/Interoperability Modeling, visualization, analytical capacity Capacity building Social construct

17 GLOBAL CONVERGENCE IN GEOSCIENCE CYBERINFRASTRUCTURE DAWN OF THE DIGICENE