Avigdor Gal Technion Israel Institute of Technology

Transcription

1 Avigdor Gal Technion Israel Institute of Technology

2 Tutorial Big data integration Applications of big data integration Current challenges and future research directions

3 Big data is a game changer From Theory to Systems: empirical evaluation counts From Systems to : large scale empirical evaluation counts

4 Volume: No Longer the Size of a Teacup Volume Table : Cross Table Big data may be a single dataset with a lot of data

5 Volume: No Longer the Size of a Teacup Table : Cross Table Big data may be a single dataset with a lot of data

6 Velocity: Replacing a Teacup with a Tea Hose Volume Velocity Table : Cross Table Big data may be data that rapidly changes

7 Velocity: Replacing a Teacup with a Tea Hose Table : Cross Table Big data may be data that rapidly changes

8 Velocity: Replacing a Teacup with a Tea Hose Table : Cross Table Big data may be data that rapidly changes

9 Velocity: Replacing a Teacup with a Tea Hose Table : Cross Table Big data may be data that rapidly changes

10 Variety: When One Tea Type is Just not Enough Volume Velocity Variety Table : Cross Table Big data may be a small dataset with many different schemata

11 Variety: When One Tea Type is Just not Enough Table : Cross Table Big data may be a small dataset with many different schemata

12 Veracity: Is it Coffee or Black Tea with Milk? Volume Velocity Variety Veracity Table : Cross Table Big data may be data with varying levels of trustworthiness

13 Veracity: Is it Coffee or Black Tea with Milk? Table : Cross Table Big data may be data with varying levels of trustworthiness

14 Gathering: where and when to expect the fountain to burst Volume Velocity Variety Veracity Gathering Signal and Event Processing Table : Cross Table

15 Gathering: where and when to expect the fountain to burst Table : Cross Table

16 Management: Not your typical DBA anymore Volume Velocity Variety Veracity Gathering Managing Cloud Computing, NoSQL, NewSQL Table : Cross Table

17 Analytics: When Analysis Explodes Multi-Dimensionally Volume Velocity Variety Veracity Gathering Managing Analyzing Table : Cross Table & Process Mining ML, IR, NLP

18 Visualization: The Machine Offering to Mankind Volume Velocity Variety Veracity Gathering Managing Analyzing Visualizing Table : Cross Table User Experience

19 Visualization: The Machine Offering to Mankind Table : Cross Table

20 Cross Table Volume Velocity Variety Gathering Managing Analyzing Visualizing Veracity Table : Cross Table

21 What is? is the task of integrating multiple data sources into a single data source. is a management task in the Cross Table. Two major tasks of data integration are schema matching and entity resolution.

22 Schema Matching What is Schema Matching? Ancient history: heterogeneity of schemata Different DBAs, different names Granularity matters Schema matching is the process of creating attribute correspondences among multiple schemata

23 Schema Matching What is Schema Matching? Ancient history: heterogeneity of schemata Different DBAs, different names Granularity matters Schema matching is the process of creating attribute correspondences among multiple schemata Existing Work Formal Models: uncertain schema matching Algorithmic & Heuristic solutions: string, value, structure-based Empirical benchmarks: University applications, Web forms, Ontology matching competition (OAEI)

24 Textbook Example for Schema Matching Id name ZIP Income r 1 Green K r 2 Green K r 3 Peter K r 4 Peter K Table : SM Simple Example Id firstname lastname Address Salary r 1 John Green CARTER LAKE IA ,000 r 2 Sarah Green CARTER LAKE IA ,000K r 3 Peter Smith CLEVELAND GA ,000 r 4 Peter Smith CLEVELAND GA ,000 Table : SM Simple Example 2

25 Entity Resolution What is Entity Resolution? Real world data is dirty Typographical errors and missing values Different date formats and terminology Multiple representations of the same real-world object Multi-dimensional data aspects: temporal, spatial,... ER is the process of determining when different entity representations refer to the same entity.

26 Entity Resolution What is Entity Resolution? Real world data is dirty Typographical errors and missing values Different date formats and terminology Multiple representations of the same real-world object Multi-dimensional data aspects: temporal, spatial,... ER is the process of determining when different entity representations refer to the same entity. Existing work Formal Models and Languages Algorithmic solutions Comparative empirical analysis of solutions: FEBRL

27 Textbook Example for Entity Resolution Id name ZIP Income r 1 Green K r 2 Green K r 3 Peter K r 4 Peter K r 5 Gtee K r 6 Howard K Table : ER Simple Example

28 + = Volume Velocity Variety Veracity Gathering Managing Analyzing Visualizing ER ER SM SM & ER Table : Cross Table

29 + = Volume Velocity Variety Veracity Gathering Managing Analyzing Visualizing ER ER SM SM & ER

30 : not Your Typical Anymore Urban Traffic Management

31 : not Your Typical Anymore Traffic Flow

32 : not Your Typical Anymore Bus Log Bus Model s ω_2 ω_3 ω_i ω_{n-1} d

33 : not Your Typical Anymore Challenges Volume: 23 Million records per month ( 4GB) Velocity: 770,000 new records per day (an event each 2-6 seconds) Variety: Homogeneous Veracity: GPS locations

34 Big data The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it that s going to be a hugely important skill in the next decades. (Hal Varian, Google s Chief Economist) integration integration has been the basis of data understanding and processing for many years now. With big data joining in, the impact of data integration is not diminishing. Rather, it changes shape while remaining dominant.

35 Challenges Volume Compute data integration faster, by using parallelization. Velocity Create incremental computation methods for data integration. Variety Extend evaluation models to support data integration with minimal or no human input in the loop. Veracity Quantified uncertainty management for data integration.

36 Thank You Avigdor Gal Technion Israel Institute of Technology