Shuffling Data Around

Transcription

1 Shuffling Data Around An introduction to the keywords in Data Integration, Exchange and Sharing Dr. Anastasios Kementsietsidis Special thanks to Prof. Renée e J. Miller

2 The Cause and Effect Principle Cause: Data sources are autonomous, heterogeneous Different data models, types and schemas Different vocabularies (in data and schemas) Different requirements for what/how data is shared Effect: Integration: Provide uniform access to heterogeneous data Exchange: Move data between heterogeneous sources Sharing: Provide non-uniform access to data through each source s schema and vocabulary 2005 Anastasios Kementsietsidis and Renée e J. Miller 2

3 Data Warehousing Architecture or here? User Query Relational Database (Warehouse) What about updates here? Data Extraction/Forwarding tools GDB Database Swissprot Database Image server Outside Website 2005 Anastasios Kementsietsidis and Renée e J. Miller 3

4 Virtual Integration Architecture User Query Mediated Schema Mediator Reformulation Engine Optimization Engine Execution Engine Metadata Wrapper Wrapper Wrapper Wrapper GDB Database Swissprot Database Image server Outside Website 2005 Anastasios Kementsietsidis and Renée e J. Miller 4

5 Peer-to-Peer Architecture User Query Outside Website User Query GDB Database Image server Swissprot Database User Query 2005 Anastasios Kementsietsidis and Renée e J. Miller 5

6 What are the Metadata? Schemas (models of data) Structured or Semi-structured e.g., relational, object-oriented, XML data, Talk will not cover unstructured data e.g., documents, images, audio files, Data(base) is an instance of a schema Mappings Model relationship between schemas or data Schema mapping (e.g., views) Data mapping (e.g., aliases) Requirements for mapping specifications 2005 Anastasios Kementsietsidis and Renée e J. Miller 6

7 Metadata Lifecycle Creation Automatic discovery or creation Design tools facilitating creation Maintenance Maintain (integrated) schemas as sources change Maintain mappings as schemas change Use Query answering Data exchange (materialization), updates, etc 2005 Anastasios Kementsietsidis and Renée e J. Miller 7

8 Schema Integration Global Integrated Schema G What about schema changes? or source additions? Local Source Schema S 1 Local Source Schema S 2 Local Source Schema S n Data Data Data Schema Design Problem: Create global integrated schema G (and mappings) for a set of independently designed local schemas S i, 1 i n 2005 Anastasios Kementsietsidis and Renée e J. Miller 8

9 Mappings (a.k.a. views) A view is just a query works like a function. It accepts as input the local source instance(s) It outputs an instance of the global (target) schema Global Integrated Schema G Relation: funding (aid, amount, project, date) Relation: finances (aid, date, amount) create view funding (aid, amount,, project, date) as ( select grant.gid gid, grant.amount amount, grant.project project, received.date date from grant, received where grant.gid gid = received.gid gid) This is Global-as-View (GAV) Local Source Schema S Relation: grant (gid, amount, project) Relation: received (gid, date) 2005 Anastasios Kementsietsidis and Renée e J. Miller 9

10 This is Local-as-View (LAV) Global Integrated Schema G Relation: funding (aid, amount, project, date) Relation: finances (aid, date, amount) create view grant (gid, amount,, project) as ( select funding.aid aid, funding.amount amount, funding.project project, from funding) Local Source Schema S Relation: grant (gid, amount, project) Relation: received (gid, date) 2005 Anastasios Kementsietsidis and Renée e J. Miller 10

11 GAV vs. LAV (in plain English) GAV: LAV: Gives direct information about which data satisfy the elements of the global schema Not easily extendible on source schema changes or source additions Query answering is easy Does not give direct information about which data satisfy the global schema Easily extendible on source schema changes or source additions Query answering is hard 2005 Anastasios Kementsietsidis and Renée e J. Miller 11

12 But There Is More Global-and-Local-as-View (GLAV) Global Integrated Schema G Relation: funding (aid, amount, project, date) Relation: finances (aid, date, amount) (select funding.aid aid, funding.amount amount, funding.project project, funding.date date from funding, finances where funding.aid = finances.aid aid) (select grant.gid gid, grant.amount amount, grant.project project, received.date date from grant, received where grant.gid gid = received.gid gid) Local Source Schema S Relation: grant (gid, amount, project) Relation: received (gid, date) 2005 Anastasios Kementsietsidis and Renée e J. Miller 12

13 Creating Mappings with the help of Schema Matching grant gid amount project funding aid amount proj date received gid date financial aid date amount It uses schema-level and/or instance-level information 2005 Anastasios Kementsietsidis and Renée e J. Miller 13

14 Wrapper Database City County Database Database City And Using Them In Data Integration Virtual Integration Architecture In Data Exchange Data Warehousing Architecture User Query User Query Mediated Schema Mediator Reformulation Engine Optimization Engine Execution Engine Metadata Relational Database (Warehouse) Wrapper Wrapper Wrapper Data Extraction/Forwarding tools Image server Outside Website County Database Image server Outside Website 2005 Anastasios Kementsietsidis and Renée J. Miller Anastasios Kementsietsidis and Renée J. Miller Anastasios Kementsietsidis and Renée e J. Miller 14

15 Data Integration vs. Exchange Data Integration Global schema is a reconciled virtual view of heterogeneous sources Uses GAV or LAV mappings No constraints in the global schema are considered Query is answered using source data; integration is virtual Answer is set of tuples in query result on ALL possible target instances: certain answers Data Exchange Global schema is an independently created local source schema Uses GLAV mappings Considers the presence of constraints Query is answered using ONE materialized target Can single target give same information as source(s)? 2005 Anastasios Kementsietsidis and Renée e J. Miller 15

16 Something Slightly Different Data Mappings and Data Sharing Useful in environments where: Sources are unwilling to share schemas The schema of one source cannot be expressed as a view of another There is a need to map (data) vocabularies gid name 001 NF1 002 NGFR 003 NEU1 gene(gid, gid, name) GDB encodes for gid pid protein(pid, pid, name) SwissProt This is a mapping table pid name 101 Neurofibromin 102 p75 ICD 103 Sialidase G9 Sialidase 2005 Anastasios Kementsietsidis and Renée e J. Miller 16

17 Data Sharing Architecture article id UofE Library I have article Mapping I want info for Tables protein AARE Mapping Tables Mapping Tables GDB I have info for gene APEH gene NF1 APEH Mapping Tables Mapping Tables SwissProt protein AARE G9 Sialidase Establish mapping tables between the vocabularies of different sources. Use these tables to translate query requests between the sources 2005 Anastasios Kementsietsidis and Renée e J. Miller 17

18 Management of Tables Source S 1 Mapping Tables Mapping Tables Inferred Tables Network Consistent? Mapping Tables Mapping Tables Source S n Inferred Tables The Consistency and Inference problems, the main vehicles for managing mapping tables. Solving these problems allows us to: Infer new mapping tables from existing ones Augment existing mapping tables with new associations Validate mapping tables 2005 Anastasios Kementsietsidis and Renée e J. Miller 18

19 Closing Remarks and things to remember (other than the keywords): Integration, one of the oldest problems in database research. Research is still going strong in this area Exchange, an interesting and practical problem (e.g. B2B apps) Sharing, the latest twist in the integration problem, also of practical importance (e.g. in P2P apps) Disclaimer: This talk provides only a glimpse of the research issues in these areas. Note: If you find any of these interesting, TALK to us We are in Appleton Tower, 2nd Floor 2005 Anastasios Kementsietsidis and Renée e J. Miller 19

20 Questions!?

21 GAV vs. LAV Local Source Schema S Q S/G Global Integrated Schema G User Query Data Data (?) GAV: Q S (S) R G, where R G is a relation in G, Q S is a query on S LAV: R S Q G (G), where R s is a relation in S, Q G is a query on G 2005 Anastasios Kementsietsidis and Renée e J. Miller 21

22 Query Answering GAV uses Query/View Unfolding select project from funding where amount > $ create view funding (aid, amount,, project, date) as ( select grant.gid gid, grant.amount amount, grant.project project, received.date date from grant, received where grant.gid gid = received.gid gid) What about LAV? It uses a method called select project Query Rewriting from grant, received (not presented here) where grant.gid = received.gid AND grant.amount amount > $ Anastasios Kementsietsidis and Renée e J. Miller 22