Data Integration and Network Marketing

Transcription

1 CID Name Quarter CSE444 Databases fall CSE541 Operating systems winter Data Integration Alon Halevy Google Inc. University of Aalborg September, 2007 Introduction What is Data Integration and Why is it Hard? DBMS: it s all about abstraction Logical vs. Physical; What vs. How. Students: SSN Name Category Charles undergrad Dan grad Courses: CID Name Quarter CSE444 Databases fall CSE541 Operating systems winter Takes: SSN CID CSE142 SELECT C.name FROM Students S, Takes T, Courses C WHERE S.name= Mary and S.ssn = T.ssn and T.cid = C.cid Data Integration: A Higher-level Abstraction Query Independence of: source & location Mediated Schema data model, syntax semantic variations S1 S2 S3 SSN Name Category Charles undergrad Dan grad Semantic Mappings SSN CID CSE142 <cd> <title> The best of </title> </cd> <artist> Carreras </artist> <artist> Pavarotti </artist> <artist> Domingo </artist> <price> </price> Application Area 1: Business Enterprise Databases Legacy Databases Single Mediated View and Applications 50% of all IT $$$ spent here! EII Apps: CRM ERP Portals Phenotype OMIM Application Area 2: Science Gene Gene- Clinics Protein HUGO Sequenceable Entity Locus- Link Nucleotide Sequence Swiss- Prot Structured Vocabulary Entrez GO Hundreds of biomedical data sources available; growing rapidly! Experiment Microarray Experiment GEO 1

2 Application Area 3: The Web FullServe Corporation FullTimeEmp Temp Training Courses Enrollments Products Resumes Interview CV Customers Contracts HelpLine Calls EuroCard Corporation Credit Cards Customer CustDetail Resumes Interview HelpLine Calls Heterogeneity 101 FullTimeEmp ssn, empid, firstname middlename, lastname empid, hiredate, recruiter Temp ssn, hirestart, hireend ID, firstnamemiddleinitial, lastname ID, hiredate, recruiter Find all employees (making over $100K) Customer Call Center Customer: I lost my credit card!! Agent: would you like to buy an espresso machine? Products Customers Contracts Credit Cards Customer CustDetail Challenges & Opportunities Create a (useful) web site for tracking services Collaborate with third parties E.g., create branded services Comply with government regulations Find risky employees Business intelligence What s really wrong with our products? 2

3 The Deep Web Millions of (good) forms out there Each form has its own special interface Hard to explore data across sites. Goal (for some domains): A single interface into a multitude of deepweb sources. The Semantic Web Knowledge sharing at web scale. Web resources are described by ontologies: Rich domain models; allow reasoning. RDF/OWL are the emerging standards OWL-lite may actually be useful. Issues: Too complex for users? Killer apps? Scalability of reasoning? Proposal: let s build the SW bottom up. Goal of Data Integration Uniform query access to a set of data sources Handle: Scale of sources: from tens to millions Heterogeneity Autonomy Semi-structure 3

4 Why is it Hard? Systems-level reasons: Managing different platforms SQL across multiple systems is not so simple Distributed query processing Logical reasons: Schema (and data) heterogeneity Social reasons: Locating and capturing relevant data in the enterprise. Convincing people to share (data fiefdoms) Security, privacy and performance implications. Setting Expectations Data integration is AI-Complete. Completely automated solutions unlikely. Goal 1: Reduce the effort needed to set up an integration application. Goal 2: Enable the system to perform gracefully with uncertainty (e.g., see the web) Relational Terminology Terminology Relations and Queries Relational schemas Tables, attributes Relation instances Sets (or multi-sets) of tuples Integrity constraints Keys, foreign keys, inclusion dependencies Table/relation name Product PName Price Gizmo $19.99 Powergizmo $29.99 SingleTouch $ MultiTouch $ Tuples or rows Attribute names Category Manufacturer Gadgets GizmoWorks Gadgets GizmoWorks Photography Canon Household Hitachi Interview: SQL (very basic) candidate, date, recruiter, hiredecision, grade EmployeePerf: empid, name, reviewquarter, grade, reviewer select recruiter, candidate from Interview, EmployeePerf where recuiter=name AND grade < 2.5 4

5 SQL (w/aggregation) EmployeePerf: empid, name, reviewquarter, grade, reviewer Q(R,C) :- Conjunctive Queries Interview(X,D,Y,H,F), EmployeePerf(E,Y,T,W,Z), W < 2.5. select reviewer, Avg(grade) from EmployeePerf where reviewquarter= 1/2007 select recruiter, candidate from Interview, EmployeePerf where recuiter=name AND grade < 2.5 Unions of Conjunctive Queries Q(R,C) :- Interview(X,D,Y,H,F), EmployeePerf(E,Y,T,W,Z), W < 2.5. Q(R,C) :- Interview(X,D,Y,H,F), EmployeePerf(E,Y,T,W,Z), Manager(y), W > 3.9. Datalog (recursion) Database: edge(x,y) Path(X,Y) :- edge(x,y) Path(X,Y) :- edge(x,z), path(z,y) Warmup Exercise Virtual Data Integration Architecture 5

6 Mediated Schema Movie: Title, director, year, genre Actors: title, actor Plays: movie, location, starttime Reviews: title, rating, description Source Descriptions wrapper wrapper wrapper wrapper wrapper S1 S2 S3 S4 S5 Movies: name, actors, director, genre Cinemas: place, movie, start Cinemas in NYC: cinema, title, starttime Cinemas in SF: location, movie, startingtime Reviews: title, date grade, review Wrappers <cd> <title> The best of </title> <artist> Abiteboul </artist> <artist> Pavarotti </artist> <artist> Domingo </artist> <price> </price> </cd> Mediation Languages Mediated Schema CD: ASIN, Title, Genre, Artist: ASIN, name, logic Lixto [Vienna] Fetch [ISI] XQWrap [Georgia Tech] Wrapper induction [Dublin] CDs Album ASIN Price DiscountPrice Studio CDCategories ASIN Category Books Title ISBN Price DiscountPrice Edition BookCategories ISBN Category Authors ISBN FirstName LastName Artists ASIN ArtistName GroupName Query Query Processing Query reformulation Logical query plan Query optimizer Physical query plan Woody Allen Comedies in NY Movie: Title, director, year, genre Actors: title, actor Plays: movie, location, starttime Reviews: title, rating, description Replanning request wrapper source Execution engine wrapper wrapper source source wrapper source wrapper source select title, starttime from Movie, Plays where Movie.title=Plays.movie AND location= New York AND director= Woody Allen 6

7 Movies: name, actors, director, genre Cinemas: place, movie, start Movie: Title, director, year, genre Actors: title, actor Plays: movie, location, starttime Reviews: title, rating, description select title, starttime from Movie, Plays where Movie.title=Plays.movie AND location= New York AND director= Woody Allen S1 S2 S3 S4 S5 Cinemas in NYC: cinema, title, starttime Cinemas in SF: location, movie, startingtime Reviews: title, date grade, review Outline (1) Mediation languages: theoretical foundations (containment, answering queries using views) Creating schema mappings Query processing Adaptive query processing XML and its role in data integration Outline (2) Other architectures for data integration Warehousing, data exchange, p2p dbms Advanced (i.e., current) topics: Dataspaces The deep web Uncertainty in data integration 7