Data Integration. Helena Galhardas Paulo Carreira DEI IST

Size: px

Start display at page:

Download "Data Integration. Helena Galhardas Paulo Carreira DEI IST"

Lester Fox
8 years ago
Views:

1 Data Integration Helena Galhardas Paulo Carreira DEI IST (based on the slides of the course: CIS 550 Database & Information Systems, Univ. Pennsylvania, Zachary Ives)

2 Agenda Overview of Data Integration

3 Data Integration (same company) Sources D G I T A Integrated NORTH CENTER Analysis SOUTH ~700 Tables

4 How it is done Sources Integrated NORTH CENTER FTP P Analysis SOUTH ~700 Tables

5 Data Integration (across companies) Google Maps Maps Insurance Company National Road Safety Authority Hot Spots Car Accidents

6 Goal of data integration Offer uniform access to a set of data autonomous and heterogeneous data sources

7 Motivation Often people build databases in isolation Different systems within an enterprise Different information brokers on the Web Then they want to share their data To add value to each others data

8 Motivating example (1/4) FullServe: American company that provides internet access to homes, but also sells products to support the home computing infrastructure (ex: modems, wireless routers, etc) FullServe acquires EuroCard, an European company that is mainly a credit card provider. Recently EuroCard started leveraging its customer base to enter the Internet market.

routers, etc) FullServe acquires EuroCard, an European company that is mainly a credit card

9 Motivating Example (2/4) FullServe databases (highly simplified): Employee Database FullTimeEmp(ssn, empid, firstname, middlename, lastname) Hire(empId, hiredate, recruiter) TempEmployees(ssn, hirestart, hireend, name, hourlyrate) Training Database Courses(courseID, name, instructor) Enrollments(courseID, empid, date) Services Database Services(packName, textdescription) Customers(name, id, zipcode, streetadr, phone) Contracts(custID, packname, startdate) Sales Database Products(prodName, prodid) Sales(prodName, customername, address) Resume Database Interview(interviewDate, name, recruiter, hiredecision, hiredate) CV(name, resume) HelpLine Database Calls(date, agent, custid, text, action)

Services(packName, textdescription) Customers(name, id, zipcode, streetadr, phone) Contracts(custID, packname, startdate) Sales Database Products(prodName, prodid)

10 Motivating Example (3/4) EuroCard database: Employee Database Emp(ID, firstnamemiddleinitial, lastname) Hire(ID, hiredate, recruiter) CreditCard Database Customer(CustID, cardnum, expiration, currentbalance) CustDetail(CustID, name, address) Resume Database Interview(ID, date, location, recruiter) CV(name, resume) HelpLine Database Calls(date, agent, custid, description, followup)

expiration, currentbalance) CustDetail(CustID, name, address) Resume Database Interview(ID,

11 Why data resides in multiple DBs in a Rather than in a single well-organized DB? 1 When companies go through internal restructuring, they do not always align their DBs 2 Most DBs are created by a group within the company with a specific need Not all the the future information needs can be anticipated!

1 When companies go through internal restructuring, they do not always

12 Motivating Example (4/4) Some queries employees or managers in FullServe may want to pose: The Human Resources wants to query all of its employees whether in the US or in Europe Require accessing 1 database in the American side and 1 in the European side There is a single customer support hot-line. Customers can call about any service or product, be it internet service, credit card. Require access to 2 databases in the US side and 1 in the European side.

side and 1 in the European side There is a single customer support hot-line.

13 Another example: Searching for a new job (1/2)

14 Another example: searching for a new job (2/2) Each form (site) asks for a slighly different set of attributes (ex: keywords describing job, location and job category or employer and job type) Ideally, would like to have a single web site to pose our queries and have that site integrating data from all relevant sites in the Web,

category or employer and job type) Ideally, would like to have a single web site

15 Why is it so hard? System Challenges Querying and optimizing over disparate sources Even with the same HW and all relational sources, the SQL dialect supported is not always the same Large number of sources Logical Challenges Heterogeneous data sources Structured vs semi-structures vs unstructured Distinct schemas Data Heterogeneities: Different representations for the same data Administrative/social constraints Autonomous data sources We may not have full access to the data or source may not be available all the time.

supported is not always the same Large number of sources Logical Challenges Heterogeneous data sources Structured vs

16 Practical goals of data integration Ideally: data integration system access a set of data sources and automatically configures itself to correctly and efficiently answer queries over multiple sources Actually: Build tools to reduce the effort required to integrate a set of data sources Improve the ability of the system to answer queries in uncertain environments Trade-off: User effort vs accuracy

sources Actually: Build tools to reduce the effort required to integrate a set of data sources

Virtual Data Integration

Virtual Data Integration Helena Galhardas Paulo Carreira DEI IST (based on the slides of the course: CIS 550 Database & Information Systems, Univ. Pennsylvania, Zachary Ives) Agenda Terminology Conjunctive