Global Earth Observation Integrated Data Environment (GEO-IDE) Presentation to the Data Archiving and Access Requirements Working Group (DAARWG) Ken McDonald Data Management Integration Architect National Oceanic and Atmospheric Administration May 24, 2007
Issue: GEO-IDE (from DAARWG, 12/2006) The Global Earth Observation Integrated Data Environment is supposed to provide integrated observing, data processing, and information management systems It is intended to support the International Global Earth Observing System of Systems (GEOSS) What are the linkages with other programs, in NOAA, with other agencies and internationally? What resources are available to implement GEO- IDE? Is the approach being taken likely to result in effective integration? 2
Important societal issues require data from many observation systems Discipline Specific View Whole System View Atmospheric Observations Land Surface Observations Ocean Observations Space Observations Data Systems Current systems are program specific, focused, individually efficient. But incompatible, not integrated, isolated from one another and from wider environmental community Coordinated, efficient, integrated, interoperable 3
Technical Challenges New observation systems will lead to 100-fold increase in data volume Increasing need for interdisciplinary use of data Integration of data among systems is needed to answer questions that address diverse societal benefits Current data management systems already face challenges Societal Benefits Improve weather forecasting Reduce Loss of Life and Property from Disasters Protect and Monitor our Ocean Resources Understand, Assess, Predict, Mitigate and Adapt to Climate Variability and Change Support Sustainable Agriculture and Combat Land Degradation Understand the Effect of Environmental Factors on Human Health & Well Being Develop the Capacity to Make Ecological Forecasts Protect and Monitor Water Resources Monitor and Manage Energy Resources 4
What is GEO-IDE? Concept Services-oriented architecture for NOAA data management systems, providing common services and leveraging the benefits of existing data management systems Developed by the Data Management Integration Team (DMIT) Vision A system of systems a framework that provides effective and efficient integration of many quasi-independent systems Foundation NOAA adopted data standards, principles and guidelines Result An integrated system of systems (user perspective) that is used to access the information needed to address significant societal questions 5
Target Architecture Principles: Utility Interoperability Flexibility Sustainability Affordability NOAA s Observation System Target Architecture Coordinated with all appropriate partners (international & national) 6
Services-Oriented Architecture Construct for Establishing GEO-IDE 7
Initial Steps Toward GEO-IDE Developed GEO-IDE Concept of Operations Developed Baseline Data Management Architecture Established Data Standards Working Group and Proposed Data Standards Adoption Process Submitted OMB Exhibit 300 for GEO-IDE 8
Initial Inventory of NOAA Data Management Systems NOAA Observing System Council Weather & Water Commerce & Transportation Ecosystems Advanced Weather Interactive Processing System (AWIPS) Geodetic Support System, Continuously Operating Reference Stations (GSS) Northeast Fisheries Data and Information Management System (NE-FDIMS) National Weather Service Telecommunication Gateway (NWSTG) Nautical Charting System (NCS) National Marine Sanctuaries System Wide Monitoring (SWIM-IMS) NWS-NCEP Space Weather Operations (3210) (NCEP- National Geophysical Data Center Data & Information Water, Weather, Nutrients Data Repository SWO) Management System (NGDC&IMS) (WWNDR) NCEP Central Computing System (CCS) Physical Oceanographic Real-Time System & Alaska Fisheries Information Systems National Water Level Observation Network (PORTS & Network (AFISN) NWLON) Major Application Real-time weather data product dissemination system (NWWS) GEOphysical DAta System (GEODAS) NMFS-West Coast Regional Information Management System (WC-RIMS) All Hazards Warning Collection System (HazCollect) Space Physics Interactive Data Resource (SPIDR) National Oceanographic Data Center's Data and Information Management System (NODC IMS) NWS Ocean Data Quality Analysis and Management Coastal Ecosystem Data Access and Delivery Climate (NODQAM) Services (CEDADS) Hydrometeorological Automated Data System (HADS) NOAA Operational Model Archive and Distribution System (NOMADS) Pacific Islands Regional Information Management System (PIRIMS) Next Generation Weather Radar (NEXRAD Archive) Comprehensive Large Array Stewardship System (CLASS) Fisheries National Information Management System (FNMS) OAR-Airborne Ozone Lidar Data System International Comprehensive Ocean-Atmosphere Data Integrated Coral Observing Network (ICON) Set (ICOADS) High Performance Computing System (HPCS) Integrated Global Radiosonde Archive (old CARDS) (IGRA) Coastal Change Analysis Program (C-CAP IMS) Centralized Automated Data Acquisition System Integrated Surface Data Processing System (ISD) Coral Reef Information System (CORIS) (CADAS) Local Data Acquisition and Dissemination system (LDAD) Tropospheric Chemistry Data Management System Great Lakes Fisheries Data and Information Management System (GL-FDIMS) MDCRS-ARINC Meteorological Assimilation Data Ingest System (MADIS) Gulf of Mexico Fisheries Data and Information Management System (GM-FDIMS) NWS-NOHRSC-National Snow Analyses ESRL/PSD Data Management South Atlantic Fisheries Data and Information Management System (SA-FDIMS) Atmospheric Integrated Research Monitoring Network (AIRMoN) GSD High Performance Computing Caribbean Fisheries Data and Information Management System (C-FDIMS) Surface Energy Budget Network (SEBN) NS&T IMS USGODAE Argo Global Data Assembly Center (GDAC) Fish Survey Data-varied Mission Support Environmental Satellite Processing Center (ESPC) Okeanos NPOESS Interface Data Processor Segment (IDPS) SARSAT Satellite Operations Control Center and Command and Data Acquisition (SOCC/CDA) NOAA - DMS contributing To IOOS 9
GEO IDE Relies Upon Standards Archive system standards ISO Open Archival Information Systems Metadata standards FGDC and ISO 19115 w/ remote sensing extensions Format standards XML Schemas, Spatial Databases (SQL), data formats (WMO, NetCDF, HDF, etc.) Standard names and terminology Open Geospatial Consortium (OGC) standards: Features, Coverage (data), Geographic Markup Language Web Services Standards (World Wide Web Consortium) Process for evaluating and adopting standards within NOAA 10
FY 07 Priorities Develop GEO IDE Implementation Plan Alternatives Analysis Acquisition Plan Risk Management Plan Sustained funding Approve and implement data standards adoption process Philosophy of adopt, adapt, and only as a last resort create standards For legacy systems, only make changes justified on cost, performance and schedule bases Evaluate, select and advance initial pilot projects Web portal Registries 11
Issue: GEO-IDE (from DAARWG, 12/2006) The Global Earth Observation Integrated Data Environment is supposed to provide integrated observing, data processing, and information management systems It is intended to support the International Global Earth Observing System of Systems (GEOSS) What are the linkages with other programs, in NOAA, with other agencies and internationally? What resources are available to implement GEO- IDE? Is the approach being taken likely to result in effective integration? 12
Project Oversight Undersecretary for Oceans and Atmosphere NOSC NOAA Observing System Council (NOAA CIO Member) All Goals DMC NOAA Data Management Committee (CIO Council Co-Chair) All Line Offices DMIT Data Management Integration Team 13
GEO-IDE Framework to effectively integrate NOAA s many systems (a System of systems ) Minimize impact on legacy systems Utilize standards Work towards a service-oriented architecture 14
Issue: GEO-IDE (from DAARWG, 12/2006) The Global Earth Observation Integrated Data Environment is supposed to provide integrated observing, data processing, and information management systems It is intended to support the International Global Earth Observing System of Systems (GEOSS) What are the linkages with other programs, in NOAA, with other agencies and internationally? What resources are available to implement GEO- IDE? Is the approach being taken likely to result in effective integration? 15
Resources Currently developing budget proposal for FY 10-14. Previous attempts to get dedicated GEO-IDE support have been unsuccessful. All accomplishments to date have relied on funds drawn from other sources and the contributions of the DMIT member organizations. Plan for near-term is to perform pilot projects with funded projects (e.g. IOOS, CLASS) 16
Issue: GEO-IDE (from DAARWG, 12/2006) The Global Earth Observation Integrated Data Environment is supposed to provide integrated observing, data processing, and information management systems It is intended to support the International Global Earth Observing System of Systems (GEOSS) What are the linkages with other programs, in NOAA, with other agencies and internationally? What resources are available to implement GEO- IDE? Is the approach being taken likely to result in effective integration? 17
Development Approach In near and long-term, GEO-IDE approach is to work with and through programs and projects, NOAA-wide. GEO-IDE is not a large development project Effort will require support from the individual projects and continued endorsement from NOAA management GEO-IDE will require some level of dedicated resources to support: Coordination of the standards adoption process Oversight of technology assessments and trade studies Support to NOAA s interface to interagency and international initiatives Possible development of infrastructure elements Assistance to projects implementing the GEO-IDE vision 18
GEO-IDE Challenges Challenges: Data Management not a formal Goal" in NOAA. DM is split into many separate points of decision making. DM decisions are generally made at the project level -- much of this must be preserved because requirements differ sharply across projects and programs. Response: Develop mechanisms to establish agency DM standards and guidelines and apply them where appropriate Promote and support enterprise-aware development 19
Collaboration Model Understand/Reconcile Different Perspectives Project View Enterprise View Focused Goals Broad Objectives Short Term Long Term Costs Investments One time Interoperable, Reusable, Sustainable 20