A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8)

Transcription

1 NATO Science & Technology Organization Centre for Maritime Research and Experimentation (STO-CMRE) Viale San Bartolomeo, La Spezia, Italy A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8) Response Due Date: 17 March 2014 Programmatic POC Alessandro Berni Alessandro.Berni@cmre.nato.int Technical POC Elena Camossi Elena.Camossi@cmre.nato.int

2 STO-CMRE Response to the GEOSS AIP-8 CFP Overview This response is submitted by STO-CMRE ( leveraging resources provided by the partner Terradue Srl ( The purpose of this participation is to continue and extend the scope (adding in-situ data and OGC SOS service) of the work done in AIP-7. Our goal for AIP-8 is to experiment further with the key enablers in the domain of integrated environmental knowledge and decision support tools. The objective will be the exploitation of key ocean environmental variables from in-situ observations, leveraging environmental models and geo-processing, with specific attention to interoperability and connectivity with the comprehensive global network of environmental data collection and delivery. The proposed development is leveraging ocean forecasting data feeds, computeintensive models for data processing, and registration of ocean forecasting products in the GEOSS Common Infrastructure: Results exploitation will be performed in a concrete framework for policy and decision making, where scientific collaboration is having a key role in supporting informed decision; Application development will consist in the continuation/extension of earlier work conducted during AIP-7 on porting an ocean forecasting model into a dedicated Cloud environment, a Platform as a Service (PaaS) that provides parallel processing and standard OGC interfaces to the existing model. Application development will also provide a non-expert (streamlined) user dashboard to visualize and monitor the environment variables involved in the processing; The participation provides benefits for the GEO community in terms of new contributions of research data, in the form of increased availability of ocean in-situ data and added-value outputs. This represents an example of potential GEOSS Future Products, as well as a demonstration of the possible use by the GEO community of Cloud processing services, generating these products from remotely sensed data and near real-time in-situ data feeds. 2

3 Proposed Contributions STO-CMRE started to engage with the GEOSS community during AIP-7, during which some development activities were made using the Regional Ocean Modeling System (ROMS) in conjunction to one sea trial, as an enabler to subsequent model validation activities. Objectives of the continued engagement with GEOSS include linking with an international network and an advanced ocean data management capability, related to research activities in the Maritime domain. Through GEOSS AIP, STO-CMRE is seeking to further develop research collaborations with GEO stakeholders working in the Ocean monitoring and Ocean forecasting fields. Fig. 1: In-situ environmental surveys & collection of ocean data for model validation The new development in AIP-8 will consist in the integration of observational data from ocean gliders using the Sensor Observation Service (SOS) to register in-situ observations such as data from gliders, and then to serve them (after they have been validated) to an App (e.g. Regional Ocean Model System ROMS) running in a PaaS cloud system. 3

4 1.1.1 Key application support STO-CMRE and Terradue are contributing directly to the design and development of a key app listed in the CFP, with the following information on requirements and contributions: Name of the application: Standards-based open source processing chain for ocean modeling Contributing organizations and contact information: STO-CMRE (Alessandro Berni), TERRADUE (Hervé Caumont). SBA(s) supported by the application: the proposed App is supporting the GEO SBAs related to the improvement of weather information, forecasting and warning (Weather SBA) and related to the management and protection of coastal and marine resources (Ecosystems SBA). Names and contact information of end-user(s) who will provide the requirements: several end-user prospects will be addressed by CMRE in the frame of bilateral scientific collaborations, as well as in the domain of commercial activities dependent on ocean data. Overview of the App functionality (use case): the compute-intensive processing functionality is an extension of the Developer Cloud Sandbox service, initially contributed to GEOSS through the EC FP7 GEOWOW project, with a specific application to ocean forecasting models. Application Development Framework to be used: Terradue s Developer Cloud Sandboxes PaaS (Platform as a Service). Deployment: demonstrate a standards-based open source processing chain for ocean modeling, with browser-based access for users, leveraging GEOSS data, and instantiated for Ocean observations and commercial fisheries applications. List of data sources, standards based interface protocols (OGC): o The data sources that will be exploited on the Cloud infrastructure consist in forecast elements that are in turn derived from the following assimilated information: In-Situ data: in situ temperature and salinity profiles acquired with ARGO floats, CTD, XBT, and Glider data EO data: sea level anomaly, sea surface temperature (SST) from atmospheric and ocean models output, and bathymetries o The data products generated and contributed to GEOSS consist in experimental model output products, encoded in OGC Network Common Data Format (NetCDF): Sea surface temperature, Sea level anomaly Salinity, Currents Supporting Technologies This proposal falls in the Model Web thread, exploiting the similarities between the Information to Knowledge theme with CMRE s data to decision vision of establishing information supply chains than span from the collection of observational data, to their curation and processing, to derive useful products to support decision-making processes. This, in turn, requires interoperability with the 4

5 global Earth Observation networks, incorporating feedback between decision-making and observational data requirements. STO-CMRE Data Services (Spatial Data Infrastructure) Spatial Data Infrastructure Standard interfaces: o OGC WPS o OGC WMS o OGC CSW Metadata applied in near real time (ISO 19115, /19139, with NATO Geospatial Metadata Profile); data fed into a OGCcompliant catalogue; o OGC SOS Use of the istsos distribution CKAN user portal: o Periodic harvesting and indexing of catalogue data; THREDDS server (to serve data to the model running on the cloud sandbox) Cloud Infrastructure and Cloud Services (Terradue) Terradue s contribution will leverage the company s experience as an operator of a private Cloud infrastructure, supporting bursting to commercial Clouds via APIs. Terradue s infrastructure is outsourced, rented with a setup fee and exploited through a subscription cost model. Cloud bursting to commercial providers is supporting both pay-as-you-go or subscription costs, offering a fine grain Infrastructure-as-a-Service in Terradue s Cloud Services portfolio. The result is a Cloud infrastructure composed of a set of Virtual Processing and Archiving centers. In the context of AIP-8, the cloud bursting capability will leverage the Amazon Web Services public Cloud to perform compute intensive tasks. Terradue s Cloud platform is operating on a virtualized data center, enabled with the OpenNebula cloud controller, notably featuring Orchestration and Auto-Scaling of Cloud Multi-Tier Applications. The platform s Virtual Hosts are made available with the following default configuration: - 4-Core CPU, 8 GB RAM - 2 TB local disk - OS: CentOS 6 - Other software pre-installed: command line utilities for data staging Building on top of Terradue s Cloud platform, partners can host geospatial data processing, available from and to OGC providers worldwide. To this end, the "Developer Cloud Sandboxes" service builds on technology designed for use cases having data-intensive requirements. Partner organizations can implement processing chains with full control of code, parameters & data flows, in a collaborative way 5

6 within a shared Platform delivered as a Service, and seamlessly exploiting Cloud APIs to stage data and deploy code on ad-hoc computing clusters. Science Application Development Layer Platform Tools Data Repositories Query Lead Scientist Data Access Results Publication & Visualization Scientists Direct Acyclic Graph Compilers MatLab Project Tool Suite Versioning R IDL Ticketing Oozie Workflow Scheduler for Hadoop Distributed Data Storage HDFS Distributed Data Analytics Hadoop Map Reduce Streaming Virtualization Layer Fig. 2: The Developer Cloud Sandboxes service on Terradue s Platform Using the power of 'Hadoop Streaming' for legacy applications integration, processing algorithms are enabled to access distributed data holdings and to scale over computing clusters. The service leverages the power of OpenNebula technology for managing virtualized computing nodes, and of standard Cloud APIs for resources provisioning and appliances deployment. OpenNebula (ONe): OpenNebula.org is an open-source project developing the industry standard solution for building and managing virtualized enterprise data centers and enterprise private clouds. Terradue is a contributor to the OpenNebula open source baseline and it delivers native capabilities for Orchestration and Auto- Scaling of Cloud Multi-Tier Applications. Terradue is a Channel Partner (Value Added Reseller) of the OpenNebula C12G Partner Program and was a speaker at the OpenNebula Global Conference in Berlin, September 2013, offering an overview of Terradue s Cloud Infrastructure and DevOps-enabled Services ( Apache Hadoop: Terradue is operating a Cloud processing service leveraging the Apache Hadoop framework (Cloudera distribution). The WPS-Hadoop Cloud service runs on an OpenNebula-driven private Cloud, with: - One appliance with Terradue s WPS-Hadoop I/F. 6

7 - Several Hadoop pseudo-clusters, configured from the Cloudera's Distribution Including Apache Hadoop (CDH3) baseline Architecture and Interoperability Arrangement Development STO-CMRE and Terradue jointly plans to contribute and support the refinement of the GEOSS AIP Architecture and Interoperability Arrangements, in the following ways: Type of contribution: o Cloud services interoperability (GCI evolutions for Cloud processing workflow management) o Cloud Services architecture (GCI evolutions for cloud interoperability) Data sources supported: o Essential Ocean Variables (netcdf) Standards supported to provide access to data and products: o OGC CSW o OGC CSW 3.0 OpenSearch Geo & Time Extensions o OGC WCS 1.0 o OGC WPS 1.0 o OGC SOS 2.0 7

8 Description of STO-CMRE NATO s Science & Technology Organization (STO) Centre for Maritime Research and Experimentation (CMRE) is an established, world-class scientific research and experimentation facility that organizes and conducts scientific research and technology development, centered on the maritime domain, delivering innovative and field tested Science & Technology (S&T) solutions. Located in La Spezia (Italy), the CMRE is built on more than 50 years of experience and has produced a cadre of leaders in ocean science, modeling and simulation, acoustics and other disciplines, as well as producing critical results and understanding for the benefit of member Nations. CMRE operates two research vessels that enable science and technology solutions to be explored and developed at sea. The largest of these vessels, the NATO Research Vessel Alliance, is an ice-capable global class vessel that is one of the world's quietest vessels, allowing for precision acoustic studies to be conducted at sea. The engagement with AIP-8 is performed within the scope of project Decisions in Uncertain Ocean Environments, funded by Allied Command Transformation as part of the Environmental Knowledge and Operational Effectiveness Programme. The technologies that will be tested and showcased represent some of the enablers of CMRE s data to decision concept, an information supply chain in which data from real world and the model web are combined to provide decision-makers with new and improved supporting tools. STO-CMRE key persons are: Alessandro Berni, Elena Camossi, Raul Vicen-Bueno and Giampaolo Cimino. 8