OGSA - A Guide to Data Access and Integration in UK

Transcription

1 The OGSA-DAI Project Databases and the Grid Neil Chue Hong Principal Consultant EPCC, Edinburgh

2 What is OGSA-DAI? 4It is a project: OGSA Data Access and Integration: funded by the UK escience Grid Core Programme 4It is a vision: From simple database access to truly virtualised data resources 4It is a standard: The GridDataService Specification from the Data Access and Integration Working Group (DAIS-WG) of the Global Grid Forum (GGF) 4It is software that you can use: Core services now in Release 2

3 OGSA-DAI Objective 4To define: open standards and open source based uniform service interfaces for accessing heterogeneous data sources within the Open Grid Services Architecture (OGSA) framework 4Why? Because we are increasingly wanting to integrate different data sources from different organisations together The Grid, and OGSA, appears to provide a framework for producing software to do this

4 Who are we? Contributing to the global grid computing community IBM USA Belfast Glasgow EPCC & NeSC Newcastle Daresbury Lab Manchester EPCC & NeSC IBM UK IBM USA Manchester e-sc Newcastle e-sc Oracle 373 man months Oxford Oracle RAL Cardiff London IBM Hursley Southampton 3 million, 18 months, started February 2002 Funded by the Grid Core Programme Cambridge Hinxton

5 What are we doing? Monitoring Diagnosis Logging Data Intensive Applications Scientific Data Mining & Integration Technology Scheduling Accounting Grid Plumbing & Security Infrastructure Data & Storage Resources Distributed

6 What are we doing? Data Intensive Applications Scientific Data Mining & Integration Technology Monitoring Diagnosis Logging Data Integration Scheduling Accounting Authorisation Data Access Grid Plumbing & Security Infrastructure Data & Storage Resources Structured Data Distributed

7 What are we doing? Data Intensive Applications App. Developers Scientific Data Mining & Integration Technology Monitoring Diagnosis Logging Data Integration Scheduling Accounting Authorisation Data Access Owners Grid Plumbing & Security Infrastructure Data & Storage Resources Structured Data Operations Team Distributed

8 Data Intensive Application Scientists What are we doing? Monitoring Diagnosis Logging Data Intensive Applications App. Developers Scientific Data Mining & Integration Technology Tech. Developers Data Integration Scheduling Accounting Authorisation Data Access Owners Grid Plumbing & Security Infrastructure Data & Storage Resources Distributed Operations Team Structured Data Data Providers Data Curators

9 Data Intensive Application Scientists What are we doing? Monitoring Diagnosis Logging Scheduling Owners Accounting Data Intensive Applications Scientific Data Mining & Integration Technology Tech. Developers Data Integration Keep all the groups happy Authorisation Grid Plumbing & Security Infrastructure Data & Storage Resources Distributed App. Developers Data Access Operations Team Structured Data Data Providers Data Curators

10 Project Requirements 4Derived from project requirements survey see DAIS WG 4Driven by Technical Authority and Early Adopters AstroGrid MyGrid 4Close relationship with many other projects

11 DAIS WG 4GridDatabaseService Specification DAIS WG of the GGF Will enter documentation track after GGF8 Defines an interface for a GridDatabaseService Primary contributors are OGSA-DAI Project and EuropeanDataGrid OGSA-DAI (the software) seeks to be a reference implementation of this standard But does not necessarily track it exactly just now Requirements and Overview Informational documents also published

12 The OGSA-DAI Approach 4Reuse existing technologies and standards OGSA, Query languages, Java, transport 4Three key new services: GridDataService GridDataServiceFactory GridDataServiceRegistry 4Benefits: Location independence Hides heterogeneity Scalable Flexible Dynamic

13 OGSA-DAI Positioning - Today OGSA-DAI Distributed Query OGSA-DAI Basic Services Delivery Data Format Drivers Query (Create Retrieve Update Delete) Meta Data F OGSA R Notification Lifetime Location Database, Communication, OS Technology

14 OGSA-DAI in one slide Distributed Query Processor Common Infrastructure Lifetime Service Types and Service Implementations st Fst Rst GDTSst GDTDst Context porttypes GS F R NSrc NSnk NSub HR GDT Configuration Logging Exceptions Interface Data GSH GSR GDR -MD -CS F-MD -S -SR F-S GDT-S GDT-SR D Versioning Security Policy XML Utilities File Utilities Audit Accounting porttype Functions SDE (From GS) XML Fragment Repositor y XPath Factory (From F) Repository (From R) Engine (From ) Execute Parameter Log Notify Transaction Cache Translation XSLT Hardcode Statement SQL Xpath XQuery XUpdate Delivery Mechanism: Bulk Stream Mode : Full Block Protocol :- FTP GridFTP SOAP/HTTP Other Management Notification Unique Naming Database Access Management Connection Role Map Internationalization Database Access Implementation JDBC XMLDB OGSA-DAI Grid Data Service Training Course File

15 OGSA-DAI To Date 4Assuming that OGSA becomes the standard framework Have adopted the OGSA approach 4Have first concentrated on data access Released software has only data access so far Distributed query processor prototype due in July 4Implementation provides focus on basic functionality first But architecturally we have tried to answer many pertinent questions Functionality will increase over subsequent releases

16 OGSA-DAI Core Services 4OGSA-DAI Release 2 April 15 th Java, Tomcat, Globus Toolkit 3 Alpha3 Supports MySQL, DB2, Xindice; SQL92, XPath, Xupdate 4Continue to track Globus Toolkit 3 releases Experimental, then production, GT3 grids will help 4Changes expected in next release ServiceGroups, DataResources,

17 a. Request to egistry for ources of data bout x Analyst 3c. Results of query returned to client as XML 1b. Registry responds 2a. Request to Factory for access to with database Factory handle OR 3d. Results of query delivered to consumer as XML Consumer Registry R 2c. Factory returns handle of to client 3a. Client queries with SQL, XPath, XQuery etc in action 2b. Factory creates GridDataService to manage access Grid Data Service 3b. interacts with database Factory F SOAP/HTTP service creation API interactions Database (Xindice MySQL Oracle DB2)

18 Data Resource Implementation Mapping Configuration DataResource ImplementationMap gdsf_config.xml Data Resource Data location Resource location DBMS dataresource Name dataresource Name scheme path path DB DB connection DataResourceImplementaitonMap connection DataResourceImple menation F connection DataResourceImple mentation

19 Activity Mapping gdsf_config.xml Data Resource Data location Resource location DBMS scheme path path DB DB grid_data_service.wsdl F Data resource SDE Data resource SDE grid_data_service_bindings.wsdl grid_data_service_port_type.wsdl activitymap.xml grid_data_service_faults.xml grid_data_service_types.xsd Activity schema SDE Activity schema SDE Activity schema SDE sql_query_statement.xsd sql_update_statement.xsd xpath_statement.xsd xupdate_statement.xsd ActivityMap + config ActivityMap + config ActivityMap + config ActivityMap + config SQLQueryStatement.class SQLUpdateStatement.class XPathStatement.class XUpdateStatement.class

20 4Asynchronous delivery Pull Asynchronous Delivery Client Q 1 R s Instance 2 GSH/R + data id Consumer D + GDH 3 DT R a GDT DB 4Asynchronous delivery Push Client Q + D + GSH/R 1 R s Instance GDT Consumer DT 3 R a 2 GSH/R DB

21 Composition Client Operation DB Operation Operation 4 Client Operation DB Client Operation DB Operation Operation Operation Client Operation 5 Client Operation DB Operation DB Operation Operation

22 4A higher level service: Distributed Query Service Extension of Polar* query processor, partitions and schedules queries Sits on top of OGSA and OGSA-DAI 4Defines new porttypes and services GridDistributedQuery(GDQ) PortType GridDistributedQueryService(GDQS) wraps Polar* GridQueryEvaluatorService(GQES) perform subqueries 4Currently based on OGSA-DAI Release 1.5

23 DQS Architecture

24 DQP in action

25 DQS: the future 4The GridDistributedQueryService is an example of a higher level data integration service which utilises OGSA-DAI core services Assumes that F, GDQS Factory and client live in different containers Really requires a well-defined meta-model for the physical schema of a database Being partially addressed in DAIS WG Shows how a can be both client and service Service hierarchy and composition 4DAIT (proposed follow-on to OGSA-DAI) would produce a robust reference implementation of the DQP components

26 Projects using OGSA-DAI 4Industry: FirstDIG: business process analysis (with First Transport Group) OGSA-DAI with datamining 4Collaborative Bridges: database integration over six geographically distributed genomics research sites (with IBM UK) OGSA-DAI with DiscoveryLink edikt: porting OGSA-DAI to other platforms OGSA-DAI with performance DEISA: linking Europe s HPC centres OGSA-DAI with distributed accounting MS.Net Grid: porting OGSA-DAI to the.net framework (with Microsoft Research UK) OGSA-DAI with.net

27 WS + GSI UK support ( > 100 downloads) XML + OGSA Prototypes for Early Adopters Design Documents & Demos for DAIS GGF5 XML + OGSA Prototype Available RDB + GT2 / OGSA Prototypes Available Project Timeline today GGF6 WG Papers & Prototypes Early Adopters NeSC Ship Release 1 (Jan 15 th 2003) OGSADAI NeSC Release 1.5 (Feb 28 th 2003) GGF7 Release 2 NeSC Release 2.5 Release 3 Feb 02 May 02 Jul 02 Sep 02 Dec 02 Feb 03 May 03 Sep 03 TP4 TP5 GT3 A1 GT3 A3 GT3 Beta Phase 1 Starts Phase 2 Starts GT3 A2 GT3 A4 GT3 Final

28 Further information 4The OGSA-DAI Project Site: 4The DAIS-WG site: 4OGSA-DAI Users Mailing list General discussion on grid data access and integration 4Formal support for OGSA-DAI releases + support@ogsadai.org.uk 4OGSA-DAI training courses