The CBOS/CBIBS Data Management Framework. December 4, 2007 Tucker Pierce Tellus Applied Sciences

Transcription

1 The CBOS/CBIBS Data Management Framework December 4, 2007 Tucker Pierce Tellus Applied Sciences

2 CBOS/CBIBS Data Management Consolidation We are currently undergoing a CBOS/CBIBS Consolidation to make NCBO s data management activities for both CBIBS and CBOS less redundant, more cost effective The new data management framework will make broad use of state-ofthe-art Web Services standards for both front-end (user side) and back-end (data provider side) interfaces Data-providers, modelers, and end-users will need less system specific knowledge to interact with the system The consolidation of CBIBS/CBOS data management functions will allow us to do more with the available resources

3 A Web Services approach will be employed to reduce the cost associated with NCBO s Data Management Framework Easier incorporation of new data sources within an existing information management framework Quality assurance implementation for new data sources, as required End-user information discovery, sub-setting, and data access tools Metadata definition, implementation and management tools Business logic functions- applications and building blocks for tailored end-user products WWeb Services architectures allow systems and applications to communicate independently of operating system, programming language, database structure, and location

4 System Concept Sensor Systems And Assocaited Databases Data Management Framework Retrieving Archiving Front End Provides for data discovery, access, and supports third party apps development Standards-based (XML and JSON) data interchange format provides for syndicated data feeds such as RSS, Yahoo Pipes Data Quality Control Back End Sensor system interface approach decouples framework from sensor hardware specifics Standards-based (XML) Web Services ingest Metadata Web Client Services (e.g. Models) Syndicated features, tools, and data IVR Web application Browser Developer keyed access Server Web application Browser

5 System Consolidation The old CBOS will become the system s back-end function, responsible for interfaces with current and new data sources and providers The old CBIBS will become the system s front-end function, responsible for interfaces with current and new users, and actively supporting the development of data products This change will be transparent to: End-users directly connected to a particular sensor database Web-services clients of the CBIBS system

6 New CBOS/CBIBS Web Services Framework Overview T Web Services Interfaces Web Services Interfaces DBASE CBIBS buoys Data Archive Handler Public Internet CBOS Sensors DBASE DBASE... DBASE DBASE QA/QC Cache Metadata Storage Business Logic Business Logic Business Logic Data Retrieval Handler Client Data Request Handler Mobile Sources Web Services Interfaces NERRS CDMO National Weather Service National Estuarine Research Reserve External Data Sources

7 Sensor Data Acquisition and Storage T The Data Archive Handler is responsible for archiving the data received from the observing system sensors (or other sources) to the appropriate local CBOS/CBIBS data storage facility (Buoys, sensors, meters, etc) Data Source 1 System DB TCP/IP XML Data Archive Handler QA/QC The DAH - QA/QC module performs the necessary quality checks on the incoming data, and generates the appropriate system messages in the case of anomalies. Data Source 2 System DB Data Source 3 * System DB Transforms Data & Info Storage (To DMS Framework) * Data Source n System DB Metadata The DAH - Transformation module performs any data modifications required for the newly acquired raw data The Metadata module contains data about the data, e.g. sensor type, calibration history, and other information that gives the user confidence in the validity of the data contained in the archive

8 The CBIBS Data QA/QC Function T The QA/QC module is a part of the Data Archive handler

9 Data Retrieval Handler and Business Logic T NDBC (as required) Data Retrieval Handler Construct, send, receive OpenDap commands and data to and from NDBC (from Data Archive Handler) Data Communications Interface OpenDap Connector NT Archive Connector Manages access to the server s local near-term archive to satisfy user service request Data Storage Services connector Manages access and retrieval of data from external sources, e.g. NWS, NERRS. XML This function represents Push based feed from external data or information source, e.g. NWS, NERRS Business Logic Business Logic Business Logic External Data Sources Models, procedures, functions, performing value added services on demand as required to fulfill user needs The Data Retrieval Handler locates and retrieves data and products requested by the internet based front end function. It also initiates business logic processing for either internal or external applications supporting value added data product development for target clients.

10 Client Data Request Handler and front-end Web Services interface The Client Data Request Handler manages all connections and data requests from client (internet browser) applications. Client Data Request Handler Public Internet (to the Data Retrieval Handler) XML Direct Access Mgr SQS (as required) XML Data Comm interface IVR Web application Browser Browser applications use API key for access to framework data and/or services Server Web application Browser The system first checks the Near Term Archive for any data requested. If not found, system reformats request as necessary to retrieve the data from the appropriate source The Near Term Archive is a queue which pushes older data out in lieu of data which is in more frequent demand Note: Provision has been made to accommodate the Amazon Simple Cue Service if and when the need arises to load balance service requests from the connected browser application(s)

11 CBOS/CBIBS Web Services Capabilities Under Development Expansion of the QA/QC service function Build Out of New Interface Types - to support all the popular tools SOAP REST Metadata module expansion Build Out of Discovery/Interoperability Capabilities UDDI/WSIL ( BPEL ) Enterprise System Bus Architectures Build Out of Authentication and User Enrolment Self Service (or near Self Service) Developer Enrolment Provides fine-grained access and usage control, e.g. quotas, throttling, private data (as required) Enhanced Security (when needed) HTTPS/SSL Enhanced Administration Web-based Administration makes it easy to add and integrate data sources

12 SUMMARY The benefits of consolidation and adoption of a Web Services Architecture Provides a current, industry-standard Means of Inter-operating with the CBOS/CBIBS data management framework Encourages the development of third party applications and data products Reduces the cost of maintaining the system Reduces the cost of interacting with the system for providers and users For the Back End (data provider side) ( sets Increases choice of back-end data sources (databases, data Increases options for sensor hardware and software For the Front End (data user side) Increases choice of service requester technologies Any tool conforming to standard may be used to access and retrieve data