Open Data and Big Data at NOAA

Transcription

1 Open and Big at NOAA Derek Hanson International Relations Specialist on behalf of Jeff de La Beaujardière, PhD NOAA Management Architect

2 NOAA has "Big " 10 satellites 120+ weather radars 3 buoy networks 200+ tide gauges human observers animal telemetry 17 ships 10 aircraft Many numerical models NOAA data are unique, valuable, irreplaceable, and collected at public expense (slide adapted from "NOAA 101" briefing) 2

3 Premise of NOAA Big Project There is additional value in NOAA data that has not yet been realized because of access & infrastructure difficulties If NOAA data were accessible in the Cloud, alongside computing capability, private enterprise might generate new value-added products, services, and lines of business. Private enterprise might be willing to support the cost of transferring and storing large datasets because of potential for new markets Existing example: NWS "Family of Services" providing enhanced access at cost to commercial weather industry 3

4 4 Big RFI (2014) Request for Information (RFI) issued Feb 2014 Positive, enthusiastic, cross-industry response 70+ responses to RFI 300+ people from 200 companies at subsequent Industry Day (Oct 2014)

5 5 Challenges Recognized need for a research and prototyping phase too many unknowns Explore technologies and methodologies Incubate ideas for business models Prove self-sustainability Reduce risk and initial capital investment Also needed to establish key principles Equal access to data by all stakeholders No impact on NOAA operational mission No net additional cost to NOAA

6 6 Big CRADA (2015) Signed Cooperative Research and Development Agreements (CRADA) with 5 Infrastructure-as-a- Service (IaaS) providers in April 2015 Amazon, Google, IBM, Microsoft, Open Cloud Consortium Interested organizations and entrepreneurs can form alliances with one or more of these Web site launched: Working on first data test case: NEXRAD radar (historical and real-time)

7 Architectural Concept of NOAA Big Project (BDP) new customers & lines of business Customer 1 Customer 2 Customer 3 application & product providers Cloud IaaS provider(s) [Infrastructure as a Service] Custom Product/ App #1 integration functions Custom Product/ App #2 Custom Product/ App #3 analysis functions working copy of data 5 IaaS collaborators announced April 2015 data-alliance.noaa.gov agency security boundary agency-provided services master copy of data Agency Service Tier Earth Observations Catalog Metadata Earth Observations Access Services Formatting Model Outputs 7

8 Thank You! Jeff de La Beaujardière

9 Vision for NOAA Management Discoverable All NOAA environmental data shall be Accessible Usable Preserved for all types of users and applications. 9

10 NOAA Environmental Management Framework 10 Management Framework Lifecycle Principles Governance Resources Standards Architecture Assessment Lifecycle Lifecycle Lifecycle Lifecycle Purpose: To organize, guide and support NOAA environmental data management activities. Mandate: Science Advisory Board (SAB) recommendation to NOAA. MC/framework.php Issued March /8/2015

11 Federal Open Mandates Public Access to Research Results (OSTP memo ) osites/ostp/ostp_public_access_memo_2013.pdf Grantees Citation Open Access Research DM Planning Open Policy (Exec. Order ) es/omb/memoranda/2013/m pdf Metadata Online Services Cataloging Standard formats Publications Business 11

12 Conceptual Description of NOAA PARR Plan Policy documents doi.org/ /v5f47m2h federal funding $ funding acknowledgement via FundRef Publications Policy Sharing by Grantees Access Citation Metadata Mgmt Planning Archive Approval Researchers Catalog Access Services Metadata link publication to data via DOI (Digital Object Identifier) Research Articles Institutional Repository (documents) submit final manuscript to NOAA Library Catalog Link to pub. version expose ms. after embargo 12

13 13 NOAA Catalog data.noaa.gov

14 set Identifier Project 14 used in & Metadata Published Paper or other work cites National Centers for Environmental Information (NCEI) DOI (Digital Object Identifier) resolves to submitted to assigns links to landing page

15 NOAA Big Earth Initiative (BEDI) Inter-agency activity coordinated through US Group on Earth Observations (USGEO) Management Working Group Goal: Improve discoverability, accessibility, & usability of data Earth Observations Common Framework under development NOAA: ~$750k internal funding allocated for improving datasets Agency Service Tier agency-provided services master copy of data Earth Observations Catalog Metadata Earth Observations Access Services Formatting Model Outputs 15