# Unique column combinations

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Unique column combinations Arvid Heise Guest lecture in Data Profiling and Data Cleansing Prof. Dr. Felix Naumann

2 Agenda 2 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

3 Agenda 3 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

4 Unique column combinations 4 Relational model Dataset R with schema S Unique column combination K S r, r R : i j r[ K] r [ K] i j i In the following, they are called uniques Examples: all primary keys, all unique constraints j A B C a 1 x b 2 x c 2 y Uniques: {A, AB, AC, BC, ABC} Non-uniques: {B, C}

5 Minimal uniques 5 We are mostly interested in minimal uniques K S K' S : unique( K') K' K Removal of any column leads to non-unique combination For the previous example: {A, BC} Redundant: {AB, AC, ABC} A B C a 1 x b 2 x c 2 y Candidates for primary keys

6 Maximal non-uniques 6 Analogously we can define maximal non-uniques K' S : non unique( K') K K' Adding any column leads to unique combination K S Non-unique: {AB, AC} Redundant: {A, B, C} A B C a 1 x a 2 x a 2 y May be a data quality problem

7 Applications 7 Learning characteristics about a new data set Database management Finding a primary key Finding unique constraints Query optimization Cardinality estimations for joins Finding duplicates / data quality issues If expected unique column combinations are not unique Or with approximate uniques

8 Agenda 8 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

9 Exponential search space 9 ABCDE ABCE ABCD ABDE ACDE BCDE ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

10 Result of algorithm 10 minimal unique ABCDE maximal non-unique unique ABCE ABCD ABDE ACDE BCDE non-unique ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

11 TPCH line item 11 unique non-unique 8 columns 9 columns 10 columns

12 Size of the lattice 12 ABCDE ABCE ABCD ABDE ACDE BCDE ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

13 Computational feasibility 13 For a lattice over n columns n combinations of size k k All combinations: 2 n -1 (let s ignore -1 for the remaining slides) Largest solution set: n n 2 minimal uniques are of size n 2 Verifying minimality, requires to check also all combinations of n size 2 1 Adding a column doubles search space

14 Brute forcing Uniprot 14 Data set about proteins with 223 columns Combinations: ~1.3*10 67 Largest solution: ~7.2*10 65 There are roughly atoms on earth Assuming all uniques are of size ms verification time results in 100ka processing time

15 Agenda 15 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

16 Null values 16 Null values have a wide range of interpretations Unknown (birth day) Non-applicable (driver license number for kids) Undefined (result of integration/outer join) What is the minimal unique for the following data set? A B C D a 1 x 1 b 2 y 2 c 3 z 5 d 3 5 e 5

17 Handling null values #1 17 Depends on the actual application To find primary keys Remove all columns with null values Result: {A} A B C D a 1 x 1 b 2 y 2 c 3 z 5 d 3 5 e 5

18 Handling null values #2 18 Depends on the actual application To define unique constraints SQL defines grouping for null: null!= null Result: {A, C} -> CD unique A column of nulls is unique! A B C D a 1 x 1 b 2 y 2 c 3 z 5 d 3 5 e 5

19 Handling null values #3 19 Depends on the actual application To define unique constraints SQL defines distinctness for null: null = null Result: {A, BC} A B C D a 1 x 1 b 2 y 2 c 3 z 5 d 3 5 e 5

20 Agenda 20 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

21 Pruning with uniques 21 Pruning: inferring the type of a combination without actual verification If A is unique, supersets must be unique

22 Pruning effect of a pair 22 minimal unique ABCDE unique ABCE ABCD ABDE ACDE BCDE ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

23 Pruning with uniques #2 23 Pruning: inferring the type of a combination without actual verification If A is unique, supersets must be unique Finding a unique column prunes half of the lattice Remove column from initial data set and restart Finding a unique column pair removes a quarter of the lattice In general, the lattice over the combination is removed The pruning power of a combination is reduced by prior findings AB prunes a quarter BC additionally prunes only one eighth ABC already pruned one eights

24 Pruning both ways 24 minimal unique ABCDE maximal non-unique unique ABCE ABCD ABDE ACDE BCDE non-unique ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

25 Pruning on-the-fly 25 Materialization of the lattice is infeasible Only possible for few columns Nodes cannot be removed when discovering unique Prune on-the-fly Enumerate nodes as before Skip a node that has been pruned Depending on the approach that might be challenging Might require an efficient index structure Often: candidate generation

26 Agenda 26 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

27 Discovery Algorithms 27 Unique column Combination discovery Column-based Row-based Bottom up Hybrid Top down Apriori HCA DUCC Gordian

28 Column-based algorithms 28 Traverse through lattice Check for uniqueness Different approaches possible Use database back end and distinctness query SELECT COUNT(DISTINCT A, B, C) FROM R Compare with row number Position list indexes (explained later) For now, check is blackbox Prune lattice accordingly

29 Agenda 29 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

30 Apriori-based 30 C. Giannella and C. M. Wyss. "Finding minimal keys in a relation instance." (1999). Actually does not use much of the apriori idea Basic idea: Using the state of combinations of size k We need to visit only unpruned combinations of size k+1 Start with columns Check pairs of non-unique columns Check triples of non-unique pairs Terminate if no new combinations can be enumerated

31 Candidate generation 31 Do not generate too many duplicate combinations ABC, ABD, ACD, and BCD could point to ABCD Apriori: prefix-based generation Generate only combination of size n if prefix n-1 matches Only ABC and ABD can generate ABCD Still redundant verifications

32 Apriori visualized 32 minimal unique ABCDE maximal non-unique unique ABCE ABCD ABDE ACDE BCDE non-unique ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

33 Characteristics of Apriori 33 Works well for small uniques Bottom-up checks columns first Best case: all columns are unique n checks Worst case: no uniques = one duplicate row 2 n checks Apriori is exponential to n

34 Extensions 34 Top-down Start from top and go down Performs better if solution set is high up Candidate pruning becomes more tricky Hybrid Combine bottom-up and top-down Interleave checks Works well if solution set has many small and large columns Worst case: solution set in the middle

35 Agenda 35 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

36 Histogram-Count-based Apriori 36 Ziawasch Abedjan and Felix Naumann. "Advancing the discovery of unique column combinations." Proceedings of the international Conference on Information and Knowledge Management Extension of bottom-up apriori More sophisticated candidate generation Uses histograms for pruning Finds and uses functional dependencies on-the-fly

37 HCA candidate generation 37 Maintains a sorted list of non-uniques Avoids duplicate generation of combinations Prunes non-minimal uniques efficiently ABC unique, ABD is non-unique ABD would generate ABCD HCA performs quick minimality check with bitsets Hybrid approach At least checks if remaining columns contains duplicates

38 Statistics 38 Prunes column combinations that cannot be unique A and B contains the same value for 4/7 of the data C contains the same value for 5/7 of the data AC cannot be unique, AB might (not very likely) Especially viable if there are already indices A B C 1 A U 1 A U 1 A U 1 A U 2 B U 3 C V 4 D W

39 Functional dependencies 39 Functional dependency Value of one column determines value of another Birthday->age Intuition: If A->B and A non-unique, B must be non-unique If A->B and B unique, A must also be unique If A->B and AC non-unique, BC must be non-unique If A->B and BC unique, AC must also be unique FD A->B can be found with histogram of AB and B Histograms of FDs have the same distinctness counts

40 Analysis of HCA 40 Works well on data sets with small numbers of columns Quickly converges for many small combinations Efficient pruning Saves distinctness checks for many pairs For larger combinations statistics become less important At some point has to try all combinations Suffers from the same general complexity of Apriori

41 Agenda 41 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

42 DUCC 42 Arvid Heise, Jorge-Arnulfo Quiané-Ruiz, Ziawasch Abedjan, Anja Jentzsch, and Felix Naumann, Scalable Discovery of Unique Column Combinations, in preparation Done during internship at QCRI Basic idea: random walk through lattice Pick random superset if current combination is non-unique Pick random subset otherwise Lazy prune with previously visited nodes

43 Minimum unique column combination candidate Minimum unique column combination Maximum non-unique column combination candidate Maximum non-unique column combination Pruned ABCDE Visited nodes: 10 out of 26 ABCD ABCE ABDE ACDE BCDE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

44 Position List Index 44 Incorporates row-based pruning Intuition: number of duplicates decrease when going up Many unnecessary rows are checked again and again Keep track of duplicates with inverted index A: a->{r 1, r 2, r 3 }, b->{r 4, r 5 } B: 1->{r 1, r 3 }, 2->{r 2, r 5 } A B We don t need the actual value A: {{r 1, r 2, r 3 }, {r 4, r 5 }} B: {{r 1, r 3 }, {r 2, r 5 }} a 1 a 2 a 1 b 3 b 2

45 PLI Intersection 45 Initial PLIs A = {{r 1, r 2, r 3 }, {r 4, r 5 }} = {A 1, A 2 } B = {{r 1, r 3 }, {r 2, r 5 }} = {B 1, B 2 } Build(A) r 1 A 1 r 2 A 1 r 3 A 1 r 4 A 2 r 5 A 2 Probe(B) (A 1, B 1 ) -> {r 1, r 3 } (A 1, B 2 ) -> {r 2 } (A 2, B 2 ) -> {r 5 } Consolidate(AB) AB = {{r 1, r 3 }}

46 Analysis of PLI 46 n Space complexity: n sizeof(long) sizeof(array) 2 Intersection time complexity: O(n+n) Hash bigger PLI and probe smaller PLI If there is enough main memory Keep PLI of columns in main memory Going up in the lattice requires only to probe the current PLI Becomes increasingly fast when going up <1ms for most combinations Going down Unfortunately, PLI does not help Start from scratch

47 Experiments 47 Uniprot, 100k rows, (DUCC null = null)

48 Analysis of DUCC 48 Runtime mainly depends on size of solution set Worst case: solution set in the middle Aggressive pruning may lead to loss of minimal uniques! Gordian s final step can be used to plug these holes

49 Scaling up and out 49 Scalability is major design goal of DUCC Random walk well suited for parallelization Few coordination overhead Threads/worker share findings through event bus Uniques/non-uniques Holes in graph Lock-free to avoid bottlenecks Only memory barrier in local event bus

50 Agenda 50 Introduction and problem statement Unique column combinations Exponential search space Null values General pruning techniques Discovery algorithms Apriori HCA DUCC Gordian

51 Gordian 51 Yannis Sismanis et al. "GORDIAN: efficient and scalable discovery of composite keys." Proceedings of the international conference on Very Large Data Bases Row-based algorithm Builds prefix tree while reading data Determines maximal non-uniques Compute minimal uniques from maximal non-uniques

52 Prefix tree 52

53 Calculating minimal uniques 53 minimal unique ABCDE maximal non-unique unique ABCE ABCD ABDE ACDE BCDE non-unique ABC ABE ABD ACD ADE ACE BCD BCE BDE CDE AB AC AD AE BC BD BE CD CE DE A B C D E

54 Analysis Gordian 54 According to paper, polynomial in the number of tuples for data with a Zipfian distribution of values Can abort scan as soon as duplicate has been found Worst case Exponential in the number of columns All data needs to be stored in memory Computing minimal uniques from maximal non-uniques O( uniques 2 columns ) Can be sped up with presorted list

55 Outlook 55 Finding primary keys Uniqueness is necessary criteria No null values Include other features Name includes id, number of columns Approximate uniques 99.9% of the data unique Useful to detect data errors Gordian, HCA, and DUCC can be easily modified Heuristics with sampling

### Functional Dependencies: Part 2

Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong In designing a database, for the purpose of minimizing redundancy, we need to collect a set F of functional dependencies

### Data Mining: Partially from: Introduction to Data Mining by Tan, Steinbach, Kumar

Data Mining: Association Analysis Partially from: Introduction to Data Mining by Tan, Steinbach, Kumar Association Rule Mining Given a set of transactions, find rules that will predict the occurrence of

### Data Mining Apriori Algorithm

10 Data Mining Apriori Algorithm Apriori principle Frequent itemsets generation Association rules generation Section 6 of course book TNM033: Introduction to Data Mining 1 Association Rule Mining (ARM)

### Lecture Notes on Database Normalization

Lecture Notes on Database Normalization Chengkai Li Department of Computer Science and Engineering The University of Texas at Arlington April 15, 2012 I decided to write this document, because many students

### Effective Pruning for the Discovery of Conditional Functional Dependencies

Effective Pruning for the Discovery of Conditional Functional Dependencies Jiuyong Li 1, Jiuxue Liu 1, Hannu Toivonen 2, Jianming Yong 3 1 School of Computer and Information Science, University of South

### Association Analysis: Basic Concepts and Algorithms

6 Association Analysis: Basic Concepts and Algorithms Many business enterprises accumulate large quantities of data from their dayto-day operations. For example, huge amounts of customer purchase data

### Databases -Normalization III. (N Spadaccini 2010 and W Liu 2012) Databases - Normalization III 1 / 31

Databases -Normalization III (N Spadaccini 2010 and W Liu 2012) Databases - Normalization III 1 / 31 This lecture This lecture describes 3rd normal form. (N Spadaccini 2010 and W Liu 2012) Databases -

### Lecture 4, Data Cube Computation

CSI 4352, Introduction to Data Mining Lecture 4, Data Cube Computation Young-Rae Cho Associate Professor Department of Computer Science Baylor University A Roadmap for Data Cube Computation Full Cube Full

### Data Mining Association Analysis: Basic Concepts and Algorithms. Lecture Notes for Chapter 6. Introduction to Data Mining

Data Mining Association Analysis: Basic Concepts and Algorithms Lecture Notes for Chapter 6 Introduction to Data Mining by Tan, Steinbach, Kumar Tan,Steinbach, Kumar Introduction to Data Mining 4/8/24

### Discovery of Data Dependencies in Relational Databases

Discovery of Data Dependencies in Relational Databases Siegfried Bell & Peter Brockhausen Informatik VIII, University Dortmund 44221 Dortmund, Germany email: fbell; brockhg@ls8.informatik.uni-dortmund.de

### Frequent item set mining

Frequent item set mining Christian Borgelt Frequent item set mining is one of the best known and most popular data mining methods. Originally developed for market basket analysis, it is used nowadays for

### Benchmark Databases for Testing Big-Data Analytics In Cloud Environments

North Carolina State University Graduate Program in Operations Research Benchmark Databases for Testing Big-Data Analytics In Cloud Environments Rong Huang Rada Chirkova Yahya Fathi ICA CON 2012 April

### Boolean Algebra (cont d) UNIT 3 BOOLEAN ALGEBRA (CONT D) Guidelines for Multiplying Out and Factoring. Objectives. Iris Hui-Ru Jiang Spring 2010

Boolean Algebra (cont d) 2 Contents Multiplying out and factoring expressions Exclusive-OR and Exclusive-NOR operations The consensus theorem Summary of algebraic simplification Proving validity of an

### MAXIMAL FREQUENT ITEMSET GENERATION USING SEGMENTATION APPROACH

MAXIMAL FREQUENT ITEMSET GENERATION USING SEGMENTATION APPROACH M.Rajalakshmi 1, Dr.T.Purusothaman 2, Dr.R.Nedunchezhian 3 1 Assistant Professor (SG), Coimbatore Institute of Technology, India, rajalakshmi@cit.edu.in

### Database Design and Normalization

Database Design and Normalization Chapter 10 (Week 11) EE562 Slides and Modified Slides from Database Management Systems, R. Ramakrishnan 1 Computing Closure F + Example: List all FDs with: - a single

### HOW TO USE MINITAB: DESIGN OF EXPERIMENTS. Noelle M. Richard 08/27/14

HOW TO USE MINITAB: DESIGN OF EXPERIMENTS 1 Noelle M. Richard 08/27/14 CONTENTS 1. Terminology 2. Factorial Designs When to Use? (preliminary experiments) Full Factorial Design General Full Factorial Design

### Efficient Computation of Multiple Group By Queries Zhimin Chen Vivek Narasayya

Efficient Computation of Multiple Group By Queries Zhimin Chen Vivek Narasayya Microsoft Research {zmchen, viveknar}@microsoft.com ABSTRACT Data analysts need to understand the quality of data in the warehouse.

### Functional Dependencies and Normalization

Functional Dependencies and Normalization 5DV119 Introduction to Database Management Umeå University Department of Computing Science Stephen J. Hegner hegner@cs.umu.se http://www.cs.umu.se/~hegner Functional

### Theory behind Normalization & DB Design. Satisfiability: Does an FD hold? Lecture 12

Theory behind Normalization & DB Design Lecture 12 Satisfiability: Does an FD hold? Satisfiability of FDs Given: FD X Y and relation R Output: Does R satisfy X Y? Algorithm: a.sort R on X b.do all the

### Overview of Transaction Management 221

Overview of Transaction Management 221 1. The following schedule results in a write-read conflict: T2:R(X), T2:R(Y), T2:W(X), T1:R(X)... T1:R(X) is a dirty read here. 2. The following schedule results

### Database design theory, Part I

Database design theory, Part I Functional dependencies Introduction As we saw in the last segment, designing a good database is a non trivial matter. The E/R model gives a useful rapid prototyping tool,

### Die Welt Multimedia-Reichweite

Die Welt Multimedia-Reichweite 1) Background The quantification of Die Welt s average daily audience (known as Multimedia-Reichweite, MMR) has been developed by Die Welt management, including the research

### Index support for regular expression search. Alexander Korotkov PGCon 2012, Ottawa

Index support for regular expression search Alexander Korotkov PGCon 2012, Ottawa Introduction What is regular expressions? Regular expressions are: powerful tool for text processing based on formal language

### Association Rule Mining

Association Rule Mining Association Rules and Frequent Patterns Frequent Pattern Mining Algorithms Apriori FP-growth Correlation Analysis Constraint-based Mining Using Frequent Patterns for Classification

### Database Systems Lecture #4 FD. Guifeng Zheng School of Software, SYSU

Database Systems Lecture #4 FD Guifeng Zheng School of Software, SYSU 1 Agenda Last time: relational model This time: 1. Functional dependencies Keys and superkeys in terms of FDs Finding keys for relations

### Homework 3: Normalization, Indexing SOLUTION. AB C ; D B ; AC D. Answer each question below and carefully justify your answer.

CS 461, Database Systems, Spring 2015 Problem 1 (25pts): Normalization Homework 3: Normalization, Indexing SOLUTION Consider relation R (ABCD) together with the following set of FDs: AB C ; D B ; AC D.

### Lecture 3: Data Mining.

Lecture 3: Data Mining http://www.cs.kent.edu/~jin/advdatabases.html 1 1 Roadmap What is data mining? Data Mining Tasks Classification/Decision Tree Clustering Association Mining Data Mining Algorithms

### MINING THE DATA FROM DISTRIBUTED DATABASE USING AN IMPROVED MINING ALGORITHM

MINING THE DATA FROM DISTRIBUTED DATABASE USING AN IMPROVED MINING ALGORITHM J. Arokia Renjit Asst. Professor/ CSE Department, Jeppiaar Engineering College, Chennai, TamilNadu,India 600119. Dr.K.L.Shunmuganathan

### ON THE USE OF ASHENHURST DECOMPOSITION CHART AS AN ALTERNATIVE TO ALGORITHMIC TECHNIQUES IN THE SYNTHESIS OF MULTIPLEXER-BASED LOGIC CIRCUITS

Nigerian Journal of Technology, Vol. 18, No. 1, September, 1997 OSUAGWU 28 ON THE USE OF ASHENHURST DECOMPOSITION CHART AS AN ALTERNATIVE TO ALGORITHMIC TECHNIQUES IN THE SYNTHESIS OF MULTIPLEXER-BASED

### Mining Association Rules. Mining Association Rules. What Is Association Rule Mining? What Is Association Rule Mining? What is Association rule mining

Mining Association Rules What is Association rule mining Mining Association Rules Apriori Algorithm Additional Measures of rule interestingness Advanced Techniques 1 2 What Is Association Rule Mining?

### a. Write a query to generate the number of charter flights to the same destination and shows the results in alphabetic order.

CS 631 _ DBMS Sample Final Exam Questions Question #1: Using the data given in the AviaCo database, perform the following queries and show the expected results: a. Write a query to generate the number

### Big Data Frequent Pattern Mining

Big Data Frequent Pattern Mining David C. Anastasiu and Jeremy Iverson and Shaden Smith and George Karypis Department of Computer Science and Engineering University of Minnesota, Twin Cities, MN 55455,

### How to bet using different NairaBet Bet Combinations (Combo)

How to bet using different NairaBet Bet Combinations (Combo) SINGLES Singles consists of single bets. I.e. it will contain just a single selection of any sport. The bet slip of a singles will look like

### Introduction. The Quine-McCluskey Method Handout 5 January 21, 2016. CSEE E6861y Prof. Steven Nowick

CSEE E6861y Prof. Steven Nowick The Quine-McCluskey Method Handout 5 January 21, 2016 Introduction The Quine-McCluskey method is an exact algorithm which finds a minimum-cost sum-of-products implementation

### (Refer Slide Time: 01:35)

Artificial Intelligence Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture - 6 Informed Search -2 Today we start our 6 th lecture. This

### Chapter 6: Episode discovery process

Chapter 6: Episode discovery process Algorithmic Methods of Data Mining, Fall 2005, Chapter 6: Episode discovery process 1 6. Episode discovery process The knowledge discovery process KDD process of analyzing

### Laboratory Module 8 Mining Frequent Itemsets Apriori Algorithm

Laboratory Module 8 Mining Frequent Itemsets Apriori Algorithm Purpose: key concepts in mining frequent itemsets understand the Apriori algorithm run Apriori in Weka GUI and in programatic way 1 Theoretical

### Schema Refinement & Normalization Theory

Schema Refinement & Normalization Theory Functional Dependencies Week 11-1 1 What s the Problem Consider relation obtained (call it SNLRHW) Hourly_Emps(ssn, name, lot, rating, hrly_wage, hrs_worked) What

### normalisation Goals: Suppose we have a db scheme: is it good? define precise notions of the qualities of a relational database scheme

Goals: Suppose we have a db scheme: is it good? Suppose we have a db scheme derived from an ER diagram: is it good? define precise notions of the qualities of a relational database scheme define algorithms

### Database Management Systems. Redundancy and Other Problems. Redundancy

Database Management Systems Winter 2004 CMPUT 391: Database Design Theory or Relational Normalization Theory Dr. Osmar R. Zaïane Lecture 2 Limitations of Relational Database Designs Provides a set of guidelines,

### ! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions

Chapter 13: Query Processing Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions Basic Steps in Query

### Big Data looks Tiny from the Stratosphere

Volker Markl http://www.user.tu-berlin.de/marklv volker.markl@tu-berlin.de Big Data looks Tiny from the Stratosphere Data and analyses are becoming increasingly complex! Size Freshness Format/Media Type

### Data Warehousing. Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de. Winter 2014/15. Jens Teubner Data Warehousing Winter 2014/15 1

Jens Teubner Data Warehousing Winter 2014/15 1 Data Warehousing Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Winter 2014/15 Jens Teubner Data Warehousing Winter 2014/15 152 Part VI ETL Process

### Relational Database Design

Relational Database Design To generate a set of relation schemas that allows - to store information without unnecessary redundancy - to retrieve desired information easily Approach - design schema in appropriate

### Lecture 3 Monday, August 27, 2007

CS 6604: Data Mining Fall 2007 Lecture 3 Monday, August 27, 2007 Lecture: Naren Ramakrishnan Scribe: Wesley Tansey 1 Overview In the last lecture we covered the basics of itemset mining. Most pertinent

### Schema Refinement and Normalization

Schema Refinement and Normalization Module 5, Lectures 3 and 4 Database Management Systems, R. Ramakrishnan 1 The Evils of Redundancy Redundancy is at the root of several problems associated with relational

### Chapter 13: Query Processing. Basic Steps in Query Processing

Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions 13.1 Basic Steps in Query Processing 1. Parsing

### DBMS Interview Questions

DBMS Interview Questions 1. What is database? A database is a collection of information that is organized. So that it can easily be accessed, managed, and updated. 2. What is DBMS? DBMS stands for Database

### COMPRESSION SPRINGS: STANDARD SERIES (INCH)

: STANDARD SERIES (INCH) LC 014A 01 0.250 6.35 11.25 0.200 0.088 2.24 F F M LC 014A 02 0.313 7.94 8.90 0.159 0.105 2.67 F F M LC 014A 03 0.375 9.52 7.10 0.126 0.122 3.10 F F M LC 014A 04 0.438 11.11 6.00

### Market Basket Analysis and Mining Association Rules

Market Basket Analysis and Mining Association Rules 1 Mining Association Rules Market Basket Analysis What is Association rule mining Apriori Algorithm Measures of rule interestingness 2 Market Basket

### Week 11: Normal Forms. Logical Database Design. Normal Forms and Normalization. Examples of Redundancy

Week 11: Normal Forms Database Design Database Redundancies and Anomalies Functional Dependencies Entailment, Closure and Equivalence Lossless Decompositions The Third Normal Form (3NF) The Boyce-Codd

### ICS 434 Advanced Database Systems

ICS 434 Advanced Database Systems Dr. Abdallah Al-Sukairi sukairi@kfupm.edu.sa Second Semester 2003-2004 (032) King Fahd University of Petroleum & Minerals Information & Computer Science Department Outline

### Limitations of E-R Designs. Relational Normalization Theory. Redundancy and Other Problems. Redundancy. Anomalies. Example

Limitations of E-R Designs Relational Normalization Theory Chapter 6 Provides a set of guidelines, does not result in a unique database schema Does not provide a way of evaluating alternative schemas Normalization

### Online EFFECTIVE AS OF JANUARY 2013

2013 A and C Session Start Dates (A-B Quarter Sequence*) 2013 B and D Session Start Dates (B-A Quarter Sequence*) Quarter 5 2012 1205A&C Begins November 5, 2012 1205A Ends December 9, 2012 Session Break

### Chapter 8. Database Design II: Relational Normalization Theory

Chapter 8 Database Design II: Relational Normalization Theory The E-R approach is a good way to start dealing with the complexity of modeling a real-world enterprise. However, it is only a set of guidelines

### Gerry Hobbs, Department of Statistics, West Virginia University

Decision Trees as a Predictive Modeling Method Gerry Hobbs, Department of Statistics, West Virginia University Abstract Predictive modeling has become an important area of interest in tasks such as credit

### Unit 3 Boolean Algebra (Continued)

Unit 3 Boolean Algebra (Continued) 1. Exclusive-OR Operation 2. Consensus Theorem Department of Communication Engineering, NCTU 1 3.1 Multiplying Out and Factoring Expressions Department of Communication

### New Approach of Computing Data Cubes in Data Warehousing

International Journal of Information & Computation Technology. ISSN 0974-2239 Volume 4, Number 14 (2014), pp. 1411-1417 International Research Publications House http://www. irphouse.com New Approach of

### New Matrix Approach to Improve Apriori Algorithm

New Matrix Approach to Improve Apriori Algorithm A. Rehab H. Alwa, B. Anasuya V Patil Associate Prof., IT Faculty, Majan College-University College Muscat, Oman, rehab.alwan@majancolleg.edu.om Associate

### KEYWORD SEARCH IN RELATIONAL DATABASES

KEYWORD SEARCH IN RELATIONAL DATABASES N.Divya Bharathi 1 1 PG Scholar, Department of Computer Science and Engineering, ABSTRACT Adhiyamaan College of Engineering, Hosur, (India). Data mining refers to

### Discovering Data Quality Rules

Discovering Data Quality Rules Fei Chiang University of Toronto fchiang@cs.toronto.edu Renée J. Miller University of Toronto miller@cs.toronto.edu ABSTRACT Dirty data is a serious problem for businesses

### Introduction to DBMS

CHAPTER 1 Introduction to DBMS In this chapter, you will learn 1.0 Introduction 1.1 History of Database Management System 1.2 Database Architecture 1.3 Database Management System Users 1.4 Role of DBMS

### Graph Database Proof of Concept Report

Objectivity, Inc. Graph Database Proof of Concept Report Managing The Internet of Things Table of Contents Executive Summary 3 Background 3 Proof of Concept 4 Dataset 4 Process 4 Query Catalog 4 Environment

### 澳 門 彩 票 有 限 公 司 SLOT Sociedade de Lotarias e Apostas Mútuas de Macau, Lda. Soccer Bet Types

Soccer Bet Types 1. Asian Handicap Bet on a team to win in a designated match. Bets will be fully refunded in the case of a draw result after calculating handicap-goal*. *Handicap-goal Handicap-goal applies

### Why Is This Important? Schema Refinement and Normal Forms. The Evils of Redundancy. Functional Dependencies (FDs) Example (Contd.)

Why Is This Important? Schema Refinement and Normal Forms Chapter 19 Many ways to model a given scenario in a database How do we find the best one? We will discuss objective criteria for evaluating database

### Query Optimization for Distributed Database Systems Robert Taylor Candidate Number : 933597 Hertford College Supervisor: Dr.

Query Optimization for Distributed Database Systems Robert Taylor Candidate Number : 933597 Hertford College Supervisor: Dr. Dan Olteanu Submitted as part of Master of Computer Science Computing Laboratory

### CSE 544 Principles of Database Management Systems. Magdalena Balazinska Winter 2009 Lecture 15 - Data Warehousing: Cubes

CSE 544 Principles of Database Management Systems Magdalena Balazinska Winter 2009 Lecture 15 - Data Warehousing: Cubes Final Exam Overview Open books and open notes No laptops and no other mobile devices

### Database Design and Normalization

Database Design and Normalization CPS352: Database Systems Simon Miner Gordon College Last Revised: 9/27/12 Agenda Check-in Functional Dependencies (continued) Design Project E-R Diagram Presentations

### Enterprise Discovery Best Practices

Enterprise Discovery Best Practices 1993-2015 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)

### Soccer Bet Types Content

Soccer Bet Types Content Content 1 1. Asian Handicap 2 Examples on Asian Handicap All-Up Win bets 3 2. Win/Draw/Win 5 3. All-Up Win 6 Banker Combo 9 Multiple Combo 12 4. Over/Under 14 5. Correct Scores

### 8. Query Processing. Query Processing & Optimization

ECS-165A WQ 11 136 8. Query Processing Goals: Understand the basic concepts underlying the steps in query processing and optimization and estimating query processing cost; apply query optimization techniques;

### Efficiently Identifying Inclusion Dependencies in RDBMS

Efficiently Identifying Inclusion Dependencies in RDBMS Jana Bauckmann Department for Computer Science, Humboldt-Universität zu Berlin Rudower Chaussee 25, 12489 Berlin, Germany bauckmann@informatik.hu-berlin.de

### Estimating Compilation Time of a Query Optimizer

Estimating Compilation Time of a Query Optimizer Ihab F. Ilyas Jun Rao Guy Lohman Purdue University IBM Almaden Research Center IBM Almaden Research Center ilyas@cs.purdue.edu junrao@almaden.ibm.com lohman@almaden.ibm.com

### Chapter 7: Relational Database Design

Chapter 7: Relational Database Design Database System Concepts, 5th Ed. See www.db book.com for conditions on re use Chapter 7: Relational Database Design Features of Good Relational Design Atomic Domains

### Some issues on Conceptual Modeling and NoSQL/Big Data

Some issues on Conceptual Modeling and NoSQL/Big Data Tok Wang Ling National University of Singapore 1 Database Models File system - field, record, fixed length record Hierarchical Model (IMS) - fixed

### D B M G Data Base and Data Mining Group of Politecnico di Torino

Database Management Data Base and Data Mining Group of tania.cerquitelli@polito.it A.A. 2014-2015 Optimizer objective A SQL statement can be executed in many different ways The query optimizer determines

### Efficient Use of the Query Optimizer for Automated Physical Design

Efficient Use of the Query Optimizer for Automated Physical Design Stratos Papadomanolakis Debabrata Dash Anastasia Ailamaki Carnegie Mellon University Ecole Polytechnique Fédérale de Lausanne {stratos,ddash,natassa}@cs.cmu.edu

### Finding Frequent Itemsets using Apriori Algorihm to Detect Intrusions in Large Dataset

Finding Frequent Itemsets using Apriori Algorihm to Detect Intrusions in Large Dataset Kamini Nalavade1, B.B. Meshram2, Abstract With the growth of hacking and exploiting tools and invention of new ways

### Unit 4.3 - Storage Structures 1. Storage Structures. Unit 4.3

Storage Structures Unit 4.3 Unit 4.3 - Storage Structures 1 The Physical Store Storage Capacity Medium Transfer Rate Seek Time Main Memory 800 MB/s 500 MB Instant Hard Drive 10 MB/s 120 GB 10 ms CD-ROM

### Lecture Notes on Spanning Trees

Lecture Notes on Spanning Trees 15-122: Principles of Imperative Computation Frank Pfenning Lecture 26 April 26, 2011 1 Introduction In this lecture we introduce graphs. Graphs provide a uniform model

### Data Mining: Foundation, Techniques and Applications

Data Mining: Foundation, Techniques and Applications Lesson 1b :A Quick Overview of Data Mining Li Cuiping( 李 翠 平 ) School of Information Renmin University of China Anthony Tung( 鄧 锦 浩 ) School of Computing

### A Fast and Efficient Method to Find the Conditional Functional Dependencies in Databases

International Journal of Engineering Research and Development e-issn: 2278-067X, p-issn: 2278-800X, www.ijerd.com Volume 3, Issue 5 (August 2012), PP. 56-61 A Fast and Efficient Method to Find the Conditional

### Plan for the Query Optimization topic

VICTORIA UNIVERSITY OF WELLINGTON Te Whare Wananga o te Upoko o te Ika a Maui COMP302 Database Systems Plan for the Query Optimization topic COMP302 Database Systems Query Optimisation_04 1 What is Query

### Data Warehousing und Data Mining

Data Warehousing und Data Mining Multidimensionale Indexstrukturen Ulf Leser Wissensmanagement in der Bioinformatik Content of this Lecture Multidimensional Indexing Grid-Files Kd-trees Ulf Leser: Data

### MB2-707: Version: Microsoft Dynamics CRM Customization. and Configuration. Demo

MB2-707: Version: Microsoft Dynamics CRM Customization and Configuration Demo 1. You are a Microsoft Dynamics CRM consultant. You are assigned a new implementation. Before you configure the customer's

### CH3 Boolean Algebra (cont d)

CH3 Boolean Algebra (cont d) Lecturer: 吳 安 宇 Date:2005/10/7 ACCESS IC LAB v Today, you ll know: Introduction 1. Guidelines for multiplying out/factoring expressions 2. Exclusive-OR and Equivalence operations

### RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems CLOUD COMPUTING GROUP - LITAO DENG

1 RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems CLOUD COMPUTING GROUP - LITAO DENG Background 2 Hive is a data warehouse system for Hadoop that facilitates

### SAKey: Scalable Almost Key discovery in RDF data

SAKey: Scalable Almost Key discovery in RDF data Danai Symeonidou 1, Vincent Armant 2, Nathalie Pernelle 1, and Fatiha Saïs 1 1 Laboratoire de Recherche en Informatique, University Paris Sud, France, 2

### Database Design Patterns. Winter 2006-2007 Lecture 24

Database Design Patterns Winter 2006-2007 Lecture 24 Trees and Hierarchies Many schemas need to represent trees or hierarchies of some sort Common way of representing trees: An adjacency list model Each

### Asking Hard Graph Questions. Paul Burkhardt. February 3, 2014

Beyond Watson: Predictive Analytics and Big Data U.S. National Security Agency Research Directorate - R6 Technical Report February 3, 2014 300 years before Watson there was Euler! The first (Jeopardy!)

### The safer, easier way to help you pass any IT exams. SAP Certified Application Associate - SAP HANA 1.0. Title : Version : Demo 1 / 5

Exam : C_HANAIMP_1 Title : SAP Certified Application Associate - SAP HANA 1.0 Version : Demo 1 / 5 1.Which of the following nodes can you use when you create a calculation view with the SAP HANA studio

### Chapter 10, Functional Dependencies and Normalization for Relational Databases

Chapter 10, Functional Dependencies and Normalization for Relational Databases We need some formal measure of why the choice of attributes for a relation schema may be better than another. Functional dependencies

### Normalization in Database Design

in Database Design Marek Rychly mrychly@strathmore.edu Strathmore University, @ilabafrica & Brno University of Technology, Faculty of Information Technology Advanced Databases and Enterprise Systems 14

### Performance Tuning for the Teradata Database

Performance Tuning for the Teradata Database Matthew W Froemsdorf Teradata Partner Engineering and Technical Consulting - i - Document Changes Rev. Date Section Comment 1.0 2010-10-26 All Initial document

### Digital Logic Design

Digital Logic Design ENGG1015 1 st Semester, 2010 Dr. Kenneth Wong Dr. Hayden So Department of Electrical and Electronic Engineering Determining output level from a diagram Implementing Circuits From Boolean

### Parallelogram Bisectors Geometry Final Part B Problem 7

Parallelogram Bisectors Geometry Final Part B Problem 7 By: Douglas A. Ruby Date: 11/10/2002 Class: Geometry Grades: 11/12 Problem 7: When the bisectors of two consecutive angles of a parallelogram intersect

### Integrating Vertical and Horizontal Partitioning into Automated Physical Database Design

Integrating Vertical and Horizontal Partitioning into Automated Physical Database Design Sanjay Agrawal Microsoft Research sagrawal@microsoft.com Vivek Narasayya Microsoft Research viveknar@microsoft.com

### Chapter 10. Functional Dependencies and Normalization for Relational Databases. Copyright 2007 Ramez Elmasri and Shamkant B.

Chapter 10 Functional Dependencies and Normalization for Relational Databases Copyright 2007 Ramez Elmasri and Shamkant B. Navathe Chapter Outline 1 Informal Design Guidelines for Relational Databases