# 8. Query Processing. Query Processing & Optimization

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 ECS-165A WQ Query Processing Goals: Understand the basic concepts underlying the steps in query processing and optimization and estimating query processing cost; apply query optimization techniques; Contents: Overview Catalog Information for Cost Estimation Measures of Query Cost Selection Join Operations Other Operations Evaluation and Transformation of Expressions Query Processing & Optimization Task: Find an efficient physical query plan (aka execution plan) for an SQL query Goal: Minimize the evaluation time for the query, i.e., compute query result as fast as possible Cost Factors: Disk accesses, read/write operations, [I/O, page transfer] (CPU time is typically ignored)

2 ECS-165A WQ Basic Steps in Processing an SQL Query SQL Query Parser & Translator Relational Algebra Expression Statistics Optimizer Query Result Evaluation Engine Execution Plan (System Catalogs) Data Files Parsing and Translating Translate the query into its internal form (parse tree). This is then translated into an expression of the relational algebra. Parser checks syntax, validates relations, attributes and access permissions Evaluation The query execution engine takes a physical query plan (aka execution plan), executes the plan, and returns the result. Optimization: Find the cheapest execution plan for a query

3 ECS-165A WQ A relational algebra expression may have many equivalent expressions, e.g., π CName (σ Price>5000 ((CUSTOMERS ORDERS) OFFERS)) π CName ((CUSTOMERS ORDERS) (σ Price>5000 (OFFERS))) Representation as logical query plan (a tree): CName o Price > 5000 CName CUSTOMERS ORDERS OFFERS CUSTOMERS ORDERS OFFERS o Price > 5000 Non-leaf nodes operations of relational algebra (with parameters); Leaf nodes relations A relational algebra expression can be evaluated in many ways. An annotated expression specifying detailed evaluation strategy is called the execution plan (includes, e.g., whether index is used, join algorithms,... ) Among all semantically equivalent expressions, the one with the least costly evaluation plan is chosen. Cost estimate of a plan is based on statistical information in the system catalogs.

4 ECS-165A WQ Catalog Information for Cost Estimation Information about relations and attributes: N R : number of tuples in the relation R. B R : number of blocks that contain tuples of the relation R. S R : size of a tuple of R. F R : blocking factor; number of tuples from R that fit into one block (F R = N R /B R ) V(A, R): number of distinct values for attribute A in R. SC(A, R): selectivity of attribute A average number of tuples of R that satisfy an equality condition on A. SC(A, R) = N R /V(A, R). Information about indexes: HT I : number of levels in index I (B + -tree). LB I : number of blocks occupied by leaf nodes in index I (first-level blocks). Val I : number of distinct values for the search key. Some relevant tables in the Oracle system catalogs: USER TABLES USER TAB COLUMNS USER INDEXES NUM ROWS NUM DISTINCT BLEVEL BLOCKS LOW VALUE LEAF BLOCKS EMPTY BLOCKS HIGH VALUE DISTINCT KEYS AVG SPACE DENSITY AVG LEAF BLOCKS PER KEY CHAIN CNT NUM BUCKETS AVG ROW LEN LAST ANALYZED

5 ECS-165A WQ Measures of Query Cost There are many possible ways to estimate cost, e.g., based on disk accesses, CPU time, or communication overhead. Disk access is the predominant cost (in terms of time); relatively easy to estimate; therefore, number of block transfers from/to disk is typically used as measure. Simplifying assumption: each block transfer has the same cost. Cost of algorithm (e.g., for join or selection) depends on database buffer size; more memory for DB buffer reduces disk accesses. Thus DB buffer size is a parameter for estimating cost. We refer to the cost estimate of algorithm S as cost(s). We do not consider cost of writing output to disk. Selection Operation σ A=a (R) where a is a constant value, A an attribute of R File Scan search algorithms that locate and retrieve records that satisfy a selection condition S1 Linear search cost(s1)= B R

6 ECS-165A WQ Selection Operation (cont.) S2 Binary search, i.e., the file ordered based on attribute A (primary index) ı SC(A, R) cost(s2) = log 2 (B R ) + 1 log 2 (B R ) cost to locate the first tuple using binary search Second term blocks that contain records satisfying the selection. If A is primary key, then SC(A, R) = 1, hence cost(s2) = log 2 (B R ). F R

7 ECS-165A WQ Example (for Employee DB) F Employee = 10; V(Deptno, Employee) = 50 (different departments) N Employee = 10, 000 (Relation Employee has 10,000 tuples) Assume selection σ Deptno=20 (Employee) and Employee is sorted on search key Deptno : = 10,000/50 = 200 tuples in Employee belong to Deptno 20; (assuming an equal distribution) 200/10 = 20 blocks for these tuples = A binary search finding the first block would require log 2 (1, 000) = 10 block accesses Total cost of binary search is block accesses (versus 1,000 for linear search and Employee not sorted by Deptno).

8 ECS-165A WQ Index scan search algorithms that use an index (here, a B + -tree); selection condition is on search key of index S3 Primary index I for A, A primary key, equality A = a cost(s3) = HT I + 1 (only 1 tuple satisfies condition) S4 Primary index I on non-key A equality A = a ı SC(A, R) cost(s4) = HT I + S5 Non-primary (non-clustered) index on non-key A, equality A = a F R cost(s5) = HT I + SC(A, R) Worst case: each matching record resides in a different block. Example (Cont.): Assume primary (B + -tree) index for attribute Deptno 200/10=20 blocks accesses are required to read Employee tuples If B + -tree index stores 20 pointers per (inner) node, then the B + -tree index must have between 3 and 5 leaf nodes and the entire tree has a depth of 2 = a total of 22 blocks must be read.

9 ECS-165A WQ Selections Involving Comparisons Selections of the form σ A v (R) or σ A v (R) are implemented using a file scan or binary search, or by using either a S6 A primary index on A, or S7 A secondary index on A (in this case, typically a linear file scan may be cheaper; but this depends on the selectivity of A) Complex Selections General pattern: Conjunction σ Θ1... Θn(R) Disjunction σ Θ1... Θn(R) Negation σ Θ (R) The selectivity of a condition Θ i is the probability that a tuple in the relation R satisfies Θ i. If s i is the number of tuples in R that satisfy Θ i, then Θ i s selectivity is estimated as s i /N R.

10 ECS-165A WQ Join Operations There are several different algorithms that can be used to implement joins (natural-, equi-, condition-join) Nested-Loop Join Block Nested-Loop Join Index Nested-Loop Join Sort-Merge Join Hash-Join Choice of a particular algorithm is based on cost estimate For this, join size estimates are required and in particular cost estimates for outer-level operations in a relational algebra expression. Example: Assume the query CUSTOMERS ORDERS (with join attribute only being CName) N CUSTOMERS = 5,000 tuples F CUSTOMERS = 20, i.e., B CUSTOMERS = 5,000/20 = 250 blocks N ORDERS = 10,000 tuples F ORDERS = 25, i.e., B ORDERS = 400 blocks V(CName, ORDERS) = 2,500, meaning that in this relation, on average, each customer has four orders Also assume that CName in ORDERS is a foreign key on CUSTOMERS

11 ECS-165A WQ Estimating the Size of Joins The Cartesian product R S results in N R N S tuples; each tuple requires S R + S S bytes. If schema(r) schema(s) = primary key for R, then a tuple of S will match with at most one tuple from R. Therefore, the number of tuples in R S is not greater than N S If schema(r) schema(s) = foreign key in S referencing R, then the number of tuples in R S is exactly N S. Other cases are symmetric. In the example query CUSTOMERS ORDERS, CName in ORDERS is a foreign key of CUSTOMERS; the result thus has exactly N ORDERS = 10,000 tuples If schema(r) schema(s) = {A} is not a key for R or S; assume that every tuple in R produces tuples in R S. Then N R N S the number of tuples in R S is estimated to be: V(A, S) N R N S If the reverse is true, the estimate is V(A, R) and the lower of the two estimates is probably the more accurate one.

12 ECS-165A WQ Size estimates for CUSTOMERS ORDERS without using information about foreign keys: V(CName, CUSTOMERS) = 5,000, and V(CName, ORDERS) = 2,500 The two estimates are 5,000*10,000/2,500=20,000 and 5,000*10,000/5,000=10,000. We choose the lower estimate, which, in this case, is the same as our earlier computation using foreign key information.

13 ECS-165A WQ Nested-Loop Join Evaluate the condition join R C S for each tuple t R in R do begin for each tuple t S in S do begin check whether pair (t R, t S ) satisfies join condition if they do, add t R t S to the result end end R is called the outer and S the inner relation of the join. Requires no indexes and can be used with any kind of join condition. Worst case: db buffer can only hold one block of each relation = B R + N R B S disk accesses Best case: both relations fit into db buffer = B R + B S disk accesses.

14 ECS-165A WQ An Improvement: Block Nested-Loop Join Evaluate the condition join R C S for each block B R of R do begin for each block B S of S do begin for each tuple t R in B R do for each tuple t S in B S do check whether pair (t R, t S ) satisfies join condition if they do, add t R t S to the result end end end end Also requires no indexes and can be used with any kind of join condition. Worst case: db buffer can only hold one block of each relation = B R + B R B S disk accesses. Best case: both relations fit into db buffer = B R + B S disk accesses. If smaller relation completely fits into db buffer, use that as inner relation. Reduces the cost estimate to B R + B S disk accesses.

15 ECS-165A WQ Block Nested-Loop Join (cont.) Some improvements of block nested-loop algorithm If equi-join attribute is the key on inner relation, stop inner loop with first match Use M 2 disk blocks as blocking unit for outer relation, where M = db buffer size in blocks; use remaining two blocks to buffer inner relation and output. Reduces number of scans of inner relation greatly. Scan inner loop forward and backward alternately, to make use of blocks remaining in buffer (with LRU replacement strategy) Use index on inner relation, if available...

16 ECS-165A WQ Index Nested-Loop Join If an index is available on the inner loop s join attribute and join is an equi-join or natural join, more efficient index lookups can replace file scans. It is even possible (reasonable) to construct index just to compute a join. For each tuple t R in the outer relation R, use the index to lookup tuples in S that satisfy join condition with t R Worst case: db buffer has space for only one page of R and one page of the index associated with S: B R disk accesses to read R, and for each tuple in R, perform index lookup on S. Cost of the join: B R + N R c, where c is the cost of a single selection on S using the join condition. If indexes are available on both R and S, use the one with the fewer tuples as the outer relation.

17 ECS-165A WQ Example: Compute CUSTOMERS ORDERS, with CUSTOMERS as the outer relation. Let ORDERS have a primary B + -tree index on the joinattribute CName, which contains 20 entries per index node Since ORDERS has 10,000 tuples, the height of the tree is 4, and one more access is needed to find the actual data records (based on tuple identifier). Since N CUSTOMERS is 5,000, the total cost is = 25,250 disk accesses. This cost is lower than the 100,250 accesses needed for a block nested-loop join.

18 ECS-165A WQ Sort-Merge Join Basic idea: first sort both relations on join attribute (if not already sorted this way) Join steps are similar to the merge stage in the external sort-merge algorithm (discussed later) Every pair with same value on join attribute must be matched. values of join attributes Relation R Relation S If no repeated join attribute values, each tuple needs to be read only once. As a result, each block is read only once. Thus, the number of block accesses is B R + B S (plus the cost of sorting, if relations are unsorted). Worst case: all join attribute values are the same. Then the number of block accesses is B R + B R B S. If one relation is sorted and the other has a secondary B + -tree index on the join attribute, a hybrid merge-join is possible. The sorted relation is merged with the leaf node entries of the B + -tree. The result is sorted on the addresses (rids) of the unsorted relation s tuples, and then the addresses can be replaced by the actual tuples efficiently.

19 ECS-165A WQ Hash-Join only applicable in case of equi-join or natural join a hash function is used to partition tuples of both relations into sets that have the same hash value on the join attribute Partitioning Phase: 2 (B R + B S ) block accesses Matching Phase: B R + B S block accesses (under the assumption that one partition of each relation fits into the database buffer) Cost Estimates for other Operations Sorting: If whole relation fits into db buffer quick-sort Or, build index on the relation, and use index to read relation in sorted order. Relation that does not fit into db buffer external sort-merge 1. Phase: Create runs by sorting portions of the relation in db buffer 2. Phase: Read runs from disk and merge runs in sort order

20 ECS-165A WQ Duplicate Elimination: Sorting: remove all but one copy of tuples having identical value(s) on projection attribute(s) Hashing: partition relation using hash function on projection attribute(s); then read partitions into buffer and create in-memory hash index; tuple is only inserted into index if not already present Set Operations: Sorting or hashing Hashing: Partition both relations using the same hash function; use in-memory index for partitions R i R S: if tuple in R i or in S i, add tuple to result : if tuple in R i and in S i,... : if tuple in R i and not in S i,... Grouping and aggregation: Compute groups via sorting or hashing. Hashing: while groups (partitions) are built, compute partial aggregate values (for group attribute A, V(A,R) tuples to store values)

21 ECS-165A WQ Evaluation of Expressions Strategy 1: materialization. Evaluate one operation at a time, starting at the lowest level. Use intermediate results materialized in temporary relations to evaluate next level operation(s). CName CUSTOMERS ORDERS OFFERS o Price > 5000 First compute and store σ Price>5000 (OFFERS); then compute and store join of CUSTOMERS and ORDERS; finally, join the two materialized relations and project on to CName. Strategy 2: pipelining. evaluate several operations simultaneously, and pass the result (tuple- or block-wise) on to the next operation. In the example above, once a tuple from OFFERS satisfying selection condition has been found, pass it on to the join. Similarly, don t store result of (final) join, but pass tuples directly to projection. Much cheaper than materialization, because temporary relations are not generated and stored on disk.

22 ECS-165A WQ Evaluation of Expressions (cont.) Pipelining is not always possible, e.g., for all operations that include sorting (blocking operation). Pipelining can be executed in either demand driven or producer driven fashion.

23 ECS-165A WQ Transformation of Relational Expressions Generating a query-evaluation plan for an expression of the relational algebra involves two steps: 1. generate logically equivalent expressions 2. annotate these evaluation plans by specific algorithms and access structures to get alternative query plans Use equivalence rules to transform a relational algebra expression into an equivalent one. Based on estimated cost, the most cost-effective annotated plan is selected for evaluation. The process is called cost-based query optimization. Equivalence of Expressions Result relations generated by two equivalent relational algebra expressions have the same set of attributes and contain the same set of tuples, although their attributes may be ordered differently.

24 ECS-165A WQ Equivalence Rules (for expressions E, E 1, E 2, conditions F i ) Applying distribution and commutativity of relational algebra operations 1. σ F1 (σ F2 (E)) σ F1 F 2 (E) 2. σ F (E 1 [,, ] E 2 ) σ F (E 1 ) [,, ] σ F (E 2 ) 3. σ F (E 1 E 2 ) σ F0 (σ F1 (E 1 ) σ F2 (E 2 )); F F0 F1 F2, Fi contains only attributes of E i, i = 1, σ A=B (E 1 E 2 ) E 1 A=B E 2 5. π A (E 1 [,, ] E 2 ) π A (E 1 ) [,, ] π A (E 2 ) 6. π A (E 1 E 2 ) π A1 (E 1 ) π A2 (E 2 ), with Ai = A { attributes in E i }, i = 1, E 1 [, ] E 2 E 2 [, ] E 1 (E 1 E 2 ) E 3 E 1 (E 2 E 3 ) (the analogous holds for ) 8. E 1 E 2 π A1,A2 (E 2 E 1 ) (E 1 E 2 ) E 3 E 1 (E 2 E 3 ) (E 1 E 2 ) E 3 π((e 1 E 3 ) E 2 ) 9. E 1 E 2 E 2 E 1 (E 1 E 2 ) E 3 E 1 (E 2 E 3 ) The application of equivalence rules to a relational algebra expression is also sometimes called algebraic optimization.

25 ECS-165A WQ Examples: Selection: Find the name of all customers who have ordered a product for more than \$5,000 from a supplier located in Davis. π CName (σ SAddress like %Davis% Price>5000 (CUSTOMERS (ORDERS (OFFERS SUPPLIERS)))) Perform selection as early as possible (but take existing indexes on relations into account) π CName (CUSTOMERS (ORDERS (σ Price>5000 (OFFERS) (σ SAddress like %Davis% (SUPPLIERS))))) Projection: π CName,account (CUSTOMERS σ Prodname= CD ROM (ORDERS)) Reduce the size of argument relation in join π CName,account (CUSTOMERS π CName (σ Prodname= CD ROM (ORDERS))) Projection should not be shifted before selections, because minimizing the number of tuples in general leads to more efficient plans than reducing the size of tuples.

26 ECS-165A WQ Join Ordering For relations R 1, R 2, R 3, (R 1 R 2 ) R 3 R 1 (R 2 R 3 ) If (R 2 R 3 ) is quite large and (R 1 R 2 ) is small, we choose (R 1 R 2 ) R 3 so that a smaller temporary relation is computed and materialized Example: List the name of all customers who have ordered a product from a supplier located in Davis. π CName (σ SAddress like %Davis% (SUPPLIERS ORDERS CUSTOMERS)) ORDERS CUSTOMERS is likely to be a large relation. Because it is likely that only a small fraction of suppliers are from Davis, we compute the join σ SAddress like %Davis% (SUPPLIERS ORDERS) first. Summary of Algebraic Optimization Rules 1. Perform selection as early as possible 2. Replace Cartesian Product by join whenever possible 3. Project out useless attributes early. 4. If there are several joins, perform most restrictive join first

27 ECS-165A WQ Evaluation Plan An evaluation plan for a query exactly defines what algorithm is used for each operation, which access structures are used (tables, indexes, clusters), and how the execution of the operations is coordinated. Example of Annotated Evaluation Plan Query: List the name of all customers who have ordered a product that costs more than \$5,000. Assume that for both CUSTOMERS and ORDERS an index on CName exists: I 1 (CName, CUSTOMERS), I 2 (CName, ORDERS). CName (sort to remove duplicates) block nested loop join get tuples for tids of I2 pipeline index nested loop join pipeline o Price > 5000 full table scan I 1(CName, CUSTOMERS) I 2(CName, ORDERS) ORDERS

28 ECS-165A WQ Choice of an Evaluation Plan Must consider interaction of evaluation techniques when choosing evaluation plan: choosing the algorithm with the least cost for each operation independently may not yield the best overall algorithm. Practical query optimizers incorporate elements of the following two optimization approaches: Cost-based: enumerate all the plans and choose the best plan in a cost-based fashion. Rule-based: Use rules (heuristics) to choose plan. Remarks on cost-based optimization: Finding a join order for R 1 R 2... R n : n! different left-deep join orders For example, for n = 9, the number is use of dynamic programming techniques Heuristic (or rule-based) optimization transforms a given query tree by using a set of rules that typically (but not in all cases) improve execution performance: Perform selection early (reduces number of tuples) Perform projection early (reduces number of attributes) Perform most restrictive selection and join operations before other similar operations.

### ! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions

Chapter 13: Query Processing Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions Basic Steps in Query

### Chapter 13: Query Processing. Basic Steps in Query Processing

Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions 13.1 Basic Steps in Query Processing 1. Parsing

### Query Processing, optimization, and indexing techniques

Query Processing, optimization, and indexing techniques What s s this tutorial about? From here: SELECT C.name AS Course, count(s.students) AS Cnt FROM courses C, subscription S WHERE C.lecturer = Calders

### ICS 434 Advanced Database Systems

ICS 434 Advanced Database Systems Dr. Abdallah Al-Sukairi sukairi@kfupm.edu.sa Second Semester 2003-2004 (032) King Fahd University of Petroleum & Minerals Information & Computer Science Department Outline

### SQL Query Evaluation. Winter 2006-2007 Lecture 23

SQL Query Evaluation Winter 2006-2007 Lecture 23 SQL Query Processing Databases go through three steps: Parse SQL into an execution plan Optimize the execution plan Evaluate the optimized plan Execution

### Query Processing C H A P T E R12. Practice Exercises

C H A P T E R12 Query Processing Practice Exercises 12.1 Assume (for simplicity in this exercise) that only one tuple fits in a block and memory holds at most 3 blocks. Show the runs created on each pass

### Query Processing + Optimization: Outline

Query Processing + Optimization: Outline Operator Evaluation Strategies Query processing in general Selection Join Query Optimization Heuristic query optimization Cost-based query optimization Query Tuning

### Evaluation of Expressions

Query Optimization Evaluation of Expressions Materialization: one operation at a time, materialize intermediate results for subsequent use Good for all situations Sum of costs of individual operations

### Database System Concepts

Chapter 14: Optimization Departamento de Engenharia Informática Instituto Superior Técnico 1 st Semester 2009/2010 Slides (fortemente) baseados nos slides oficiais do livro c Silberschatz, Korth and Sudarshan.

### Chapter 14 Query Optimization

Chapter 4: Query Optimization Chapter 4 Query Optimization Introduction Catalog Information for Cost Estimation Estimation of Statistics Transformation of Relational Expressions Dynamic Programming for

### Plan for the Query Optimization topic

VICTORIA UNIVERSITY OF WELLINGTON Te Whare Wananga o te Upoko o te Ika a Maui COMP302 Database Systems Plan for the Query Optimization topic COMP302 Database Systems Query Optimisation_04 1 What is Query

### CSE 444 Practice Problems. Query Optimization

CSE 444 Practice Problems Query Optimization 1. Query Optimization Given the following SQL query: Student (sid, name, age, address) Book(bid, title, author) Checkout(sid, bid, date) SELECT S.name FROM

### 7. Indexing. Contents: Single-Level Ordered Indexes Multi-Level Indexes B + Tree based Indexes Index Definition in SQL.

ECS-165A WQ 11 123 Contents: Single-Level Ordered Indexes Multi-Level Indexes B + Tree based Indexes Index Definition in SQL 7. Indexing Basic Concepts Indexing mechanisms are used to optimize certain

### OVERVIEW OF QUERY EVALUATION

12 OVERVIEW OF QUERY EVALUATION Exercise 12.1 Briefly answer the following questions: 1. Describe three techniques commonly used when developing algorithms for relational operators. Explain how these techniques

### Comp 5311 Database Management Systems. 16. Review 2 (Physical Level)

Comp 5311 Database Management Systems 16. Review 2 (Physical Level) 1 Main Topics Indexing Join Algorithms Query Processing and Optimization Transactions and Concurrency Control 2 Indexing Used for faster

### Chapter 14: Query Optimization

Chapter 14: Query Optimization Database System Concepts 5 th Ed. See www.db-book.com for conditions on re-use Chapter 14: Query Optimization Introduction Transformation of Relational Expressions Catalog

### Query Optimization. Query Optimization. Optimization considerations. Example. Interaction of algorithm choice and tree arrangement.

COS 597: Principles of Database and Information Systems Query Optimization Query Optimization Query as expression over relational algebraic operations Get evaluation (parse) tree Leaves: base relations

### Chapter 13: Query Optimization

Chapter 13: Query Optimization Database System Concepts, 6 th Ed. See www.db-book.com for conditions on re-use Chapter 13: Query Optimization Introduction Transformation of Relational Expressions Catalog

### Kathleen Durant PhD Lecture 19 CS 3200 Northeastern University. Slide Content Courtesy of R. Ramakrishnan, J. Gehrke, and J.

Relational Query Optimization Kathleen Durant PhD Lecture 19 CS 3200 Northeastern University 1 Slide Content Courtesy of R. Ramakrishnan, J. Gehrke, and J. Hellerstein Overview of Query Evaluation Query

### SQL QUERY EVALUATION. CS121: Introduction to Relational Database Systems Fall 2015 Lecture 12

SQL QUERY EVALUATION CS121: Introduction to Relational Database Systems Fall 2015 Lecture 12 Query Evaluation 2 Last time: Began looking at database implementation details How data is stored and accessed

### Inside the PostgreSQL Query Optimizer

Inside the PostgreSQL Query Optimizer Neil Conway neilc@samurai.com Fujitsu Australia Software Technology PostgreSQL Query Optimizer Internals p. 1 Outline Introduction to query optimization Outline of

### DATABASE DESIGN - 1DL400

DATABASE DESIGN - 1DL400 Spring 2015 A course on modern database systems!! http://www.it.uu.se/research/group/udbl/kurser/dbii_vt15/ Kjell Orsborn! Uppsala Database Laboratory! Department of Information

### Datenbanksysteme II: Implementation of Database Systems Implementing Joins

Datenbanksysteme II: Implementation of Database Systems Implementing Joins Material von Prof. Johann Christoph Freytag Prof. Kai-Uwe Sattler Prof. Alfons Kemper, Dr. Eickler Prof. Hector Garcia-Molina

### CS 525: Advanced Database Organization. Execution Plan. Query Execution. How to estimate costs. 10: Query Execution. Estimating IOs: Boris Glavic

SQL query CS 55: Advanced Database Organization 0: Query Execution Boris Glavic Slides: adapted from a course taught by Hector Garcia-Molina, Stanford InfoLab CS 55 Notes 0 - Query Execution parse parse

### Query Optimization. Coming to Introduction to Database Systems by C. J. Date, he discussed the automatic optimization.

Query Optimization Introduction: Query optimization is a function of many relational database management systems in which multiple query plans for satisfying a query are examined and a good query plan

### External Sorting. Why Sort? 2-Way Sort: Requires 3 Buffers. Chapter 13

External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing

### Record Storage, File Organization, and Indexes

Record Storage, File Organization, and Indexes ISM6217 - Advanced Database Updated October 2005 1 Physical Database Design Phase! Inputs into the Physical Design Phase " Logical (implementation) model

### Database 2 Lecture II. Alessandro Artale

Free University of Bolzano Database 2. Lecture II, 2003/2004 A.Artale (1) Database 2 Lecture II Alessandro Artale Faculty of Computer Science Free University of Bolzano Room: 221 artale@inf.unibz.it http://www.inf.unibz.it/

### Reading Assignment 5 An Overview of Query Optimization in Relational Systems

Reading Assignment 5 An Overview of Query Optimization in Relational Systems José Filipe Barbosa de Carvalho (jose.carvalho@fe.up.pt) 5th December 2007 Advanced Database Systems Technische Universität

### 6. Storage and File Structures

ECS-165A WQ 11 110 6. Storage and File Structures Goals Understand the basic concepts underlying different storage media, buffer management, files structures, and organization of records in files. Contents

### In order to minimize the execution cost is possible: . minimize the size of intermediates results.

Query Optimization The query optimizer is the most important block in DBMS: it generates the strategy called execution plan. This module also defines how reads are performed:. directly;. with indices.

### Chapter 5 More SQL: Complex Queries, Triggers, Views, and Schema Modification

Chapter 5 More SQL: Complex Queries, Triggers, Views, and Schema Modification Copyright 2011 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 5 Outline More Complex SQL Retrieval Queries

### University of Massachusetts Amherst Department of Computer Science Prof. Yanlei Diao

University of Massachusetts Amherst Department of Computer Science Prof. Yanlei Diao CMPSCI 445 Midterm Practice Questions NAME: LOGIN: Write all of your answers directly on this paper. Be sure to clearly

### External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1

External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing

### Lecture 6: Query optimization, query tuning. Rasmus Pagh

Lecture 6: Query optimization, query tuning Rasmus Pagh 1 Today s lecture Only one session (10-13) Query optimization: Overview of query evaluation Estimating sizes of intermediate results A typical query

### Query Processing and Optimization

Query Processing and Optimization Query optimization: finding a good way to evaluate a query Queries are declarative, and can be translated into procedural languages in more than one way Hence one has

### Data Management for Data Science

Data Management for Data Science Database Management Systems: Access file manager and query evaluation Maurizio Lenzerini, Riccardo Rosati Dipartimento di Ingegneria informatica automatica e gestionale

### MS Sql Server Indexes

MS Sql Server Indexes Author: Nemanja Kojic MScEE 1 of 32 Indexing Vital for system performance Improves query execution performance NOT one size fits all trade offs must be made Penalties during INSERT/UPDATE

### Storage and Indexing. DBS Database Systems Implementing and Optimising Query Languages. Differences between disk and main memory

DBS Database Systems Implementing and Optimising Query Languages Peter Buneman 9 November 2010 Reading: R&G Chapters 8, 9 & 10.1 Storage and Indexing We typically store data in external (secondary) storage.

### 3. Relational Model and Relational Algebra

ECS-165A WQ 11 36 3. Relational Model and Relational Algebra Contents Fundamental Concepts of the Relational Model Integrity Constraints Translation ER schema Relational Database Schema Relational Algebra

### Answer Key. UNIVERSITY OF CALIFORNIA College of Engineering Department of EECS, Computer Science Division

Answer Key UNIVERSITY OF CALIFORNIA College of Engineering Department of EECS, Computer Science Division CS186 Fall 2003 Eben Haber Midterm Midterm Exam: Introduction to Database Systems This exam has

### Execution Strategies for SQL Subqueries

Execution Strategies for SQL Subqueries Mostafa Elhemali, César Galindo- Legaria, Torsten Grabs, Milind Joshi Microsoft Corp With additional slides from material in paper, added by S. Sudarshan 1 Motivation

### Performance Tuning for the Teradata Database

Performance Tuning for the Teradata Database Matthew W Froemsdorf Teradata Partner Engineering and Technical Consulting - i - Document Changes Rev. Date Section Comment 1.0 2010-10-26 All Initial document

### 1 11/11/2002. σ P.Dnum=D.Dnum and D.Dmgrssn=E.SSN and P.Plocation= Stafford 3 11/11/ /11/2002

376a. Database Design Dept. of Computer Science Vassar College http://www.cs.vassar.edu/~cs376 Class 17 Query optimization algorithms Query tree A representation of a select statement Canonical form of

### Query Optimization for Distributed Database Systems Robert Taylor Candidate Number : 933597 Hertford College Supervisor: Dr.

Query Optimization for Distributed Database Systems Robert Taylor Candidate Number : 933597 Hertford College Supervisor: Dr. Dan Olteanu Submitted as part of Master of Computer Science Computing Laboratory

### Databases and Information Systems 1 Part 3: Storage Structures and Indices

bases and Information Systems 1 Part 3: Storage Structures and Indices Prof. Dr. Stefan Böttcher Fakultät EIM, Institut für Informatik Universität Paderborn WS 2009 / 2010 Contents: - database buffer -

### Relational Algebra. CSE 544: Relational Operators, Sorting. What about Intersection? 1. Union and 2. Difference. 4. Projection. 3.

Relational Algebra CSE 544: Relational Operators, Sorting Wednesday, 5/12/2004 Operates on relations, i.e. sets Later: we discuss how to extend this to bags Five operators: Union: Difference: - Selection:

### Distributed Database Design and Distributed Query Execution. Designing with distribution in mind: top-down

Distributed Database Design and Distributed Query Execution Designing with distribution in mind: top-down 1 Data Fragmentation and Placement Fragmentation: How to split up the data into smaller fragments?

### Chapter 6: Physical Database Design and Performance. Database Development Process. Physical Design Process. Physical Database Design

Chapter 6: Physical Database Design and Performance Modern Database Management 6 th Edition Jeffrey A. Hoffer, Mary B. Prescott, Fred R. McFadden Robert C. Nickerson ISYS 464 Spring 2003 Topic 23 Database

Chapter 7. Indexes Table of Contents Objectives... 1 Introduction... 2 Context... 2 Review Questions... 3 Single-level Ordered Indexes... 4 Primary Indexes... 4 Clustering Indexes... 8 Secondary Indexes...

### arm DBMS File Organization, Indexes 1. Basics of Hard Disks

DBMS File Organization, Indexes 1. Basics of Hard Disks All data in a DB is stored on hard disks (HD). In fact, all files and the way they are organised (e.g. the familiar tree of folders and sub-folders

### Query processing. Tore Risch Information Technology Uppsala University

Query processing Tore Risch Information Technology Uppsala University 2011-03-08 What is query processing? A given SQL query is translated by the query processor into a low level program called an execution

### Unit 4.3 - Storage Structures 1. Storage Structures. Unit 4.3

Storage Structures Unit 4.3 Unit 4.3 - Storage Structures 1 The Physical Store Storage Capacity Medium Transfer Rate Seek Time Main Memory 800 MB/s 500 MB Instant Hard Drive 10 MB/s 120 GB 10 ms CD-ROM

### Chapter 18 Indexing Structures for Files. Indexes as Access Paths

Chapter 18 Indexing Structures for Files Indexes as Access Paths A single-level index is an auxiliary file that makes it more efficient to search for a record in the data file. The index is usually specified

### Distributed Databases Query Optimisation. TEMPUS June, 2001

Distributed Databases Query Optimisation TEMPUS June, 2001 1 Topics Query processing in distributed databases Localization Distributed query operators Cost-based optimization 2 Decomposition Query Processing

### Query Optimization! Chapter 13: Query Optimization!

Query Optimization 13.1 Chapter 13: Query Optimization Introduction Execution plans Transformation of Relational Expressions Catalog Information for Cost Estimation Statistical Information for Cost Estimation

### SQL Server Query Tuning

SQL Server Query Tuning Klaus Aschenbrenner Independent SQL Server Consultant SQLpassion.at Twitter: @Aschenbrenner About me Independent SQL Server Consultant International Speaker, Author Pro SQL Server

### Big Data and Scripting map/reduce in Hadoop

Big Data and Scripting map/reduce in Hadoop 1, 2, parts of a Hadoop map/reduce implementation core framework provides customization via indivudual map and reduce functions e.g. implementation in mongodb

### Relational Databases

Relational Databases Jan Chomicki University at Buffalo Jan Chomicki () Relational databases 1 / 18 Relational data model Domain domain: predefined set of atomic values: integers, strings,... every attribute

### Fallacies of the Cost Based Optimizer

Fallacies of the Cost Based Optimizer Wolfgang Breitling breitliw@centrexcc.com Who am I Independent consultant since 1996 specializing in Oracle and Peoplesoft setup, administration, and performance tuning

### CIS 631 Database Management Systems Sample Final Exam

CIS 631 Database Management Systems Sample Final Exam 1. (25 points) Match the items from the left column with those in the right and place the letters in the empty slots. k 1. Single-level index files

### 4. SQL. Contents. Example Database. CUSTOMERS(FName, LName, CAddress, Account) PRODUCTS(Prodname, Category) SUPPLIERS(SName, SAddress, Chain)

ECS-165A WQ 11 66 4. SQL Contents Basic Queries in SQL (select statement) Set Operations on Relations Nested Queries Null Values Aggregate Functions and Grouping Data Definition Language Constructs Insert,

### The Relational Algebra

The Relational Algebra The relational algebra is very important for several reasons: 1. it provides a formal foundation for relational model operations. 2. and perhaps more important, it is used as a basis

### Oracle Database 11g: SQL Tuning Workshop Release 2

Oracle University Contact Us: 1 800 005 453 Oracle Database 11g: SQL Tuning Workshop Release 2 Duration: 3 Days What you will learn This course assists database developers, DBAs, and SQL developers to

### Relational Algebra The Relational Algebra and Relational Calculus. Relational Query Languages

The Relational Algebra and Relational Calculus Relational Algebra Slide 6-2 Relational Query Languages Query languages Allow manipulation and retrieval of data Not like programming languages Not intend

### Chapter 5: Overview of Query Processing

Chapter 5: Overview of Query Processing Query Processing Overview Query Optimization Distributed Query Processing Steps Acknowledgements: I am indebted to Arturas Mazeika for providing me his slides of

### Physical Data Organization

Physical Data Organization Database design using logical model of the database - appropriate level for users to focus on - user independence from implementation details Performance - other major factor

### D B M G Data Base and Data Mining Group of Politecnico di Torino

Database Management Data Base and Data Mining Group of tania.cerquitelli@polito.it A.A. 2014-2015 Optimizer objective A SQL statement can be executed in many different ways The query optimizer determines

### Database System Architecture and Implementation

Database System Architecture and Implementation Kristin Tufte Execution Costs 1 Web Forms Orientation Applications SQL Interface SQL Commands Executor Operator Evaluator Parser Optimizer DBMS Transaction

### Query processing using HP NonStop SQL/MP software

Query processing using HP NonStop SQL/MP software Introduction... 2 Query processing components... 2 Query optimization... 3 Generating the plan... 3 Serial execution plans... 4 Parallel execution plans...

### Lecture 1: Data Storage & Index

Lecture 1: Data Storage & Index R&G Chapter 8-11 Concurrency control Query Execution and Optimization Relational Operators File & Access Methods Buffer Management Disk Space Management Recovery Manager

### Why Query Optimization? Access Path Selection in a Relational Database Management System. How to come up with the right query plan?

Why Query Optimization? Access Path Selection in a Relational Database Management System P. Selinger, M. Astrahan, D. Chamberlin, R. Lorie, T. Price Peyman Talebifard Queries must be executed and execution

### Elena Baralis, Silvia Chiusano Politecnico di Torino. Pag. 1. Physical Design. Phases of database design. Physical design: Inputs.

Phases of database design Application requirements Conceptual design Database Management Systems Conceptual schema Logical design ER or UML Physical Design Relational tables Logical schema Physical design

### Query Optimization in a Memory-Resident Domain Relational Calculus Database System

Query Optimization in a Memory-Resident Domain Relational Calculus Database System KYU-YOUNG WHANG and RAVI KRISHNAMURTHY IBM Thomas J. Watson Research Center We present techniques for optimizing queries

### Impala: A Modern, Open-Source SQL Engine for Hadoop. Marcel Kornacker Cloudera, Inc.

Impala: A Modern, Open-Source SQL Engine for Hadoop Marcel Kornacker Cloudera, Inc. Agenda Goals; user view of Impala Impala performance Impala internals Comparing Impala to other systems Impala Overview:

### Chapter 13. Disk Storage, Basic File Structures, and Hashing

Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible Hashing

### a presentation by Kirk Paul Lafler SAS Consultant, Author, and Trainer E-mail: KirkLafler@cs.com

a presentation by Kirk Paul Lafler SAS Consultant, Author, and Trainer E-mail: KirkLafler@cs.com 1 Copyright Kirk Paul Lafler, 1992-2010. All rights reserved. SAS is the registered trademark of SAS Institute

### Big Data and Scripting. Part 4: Memory Hierarchies

1, Big Data and Scripting Part 4: Memory Hierarchies 2, Model and Definitions memory size: M machine words total storage (on disk) of N elements (N is very large) disk size unlimited (for our considerations)

### Storage and File Structure

Storage and File Structure Chapter 10: Storage and File Structure Overview of Physical Storage Media Magnetic Disks RAID Tertiary Storage Storage Access File Organization Organization of Records in Files

### HEURISTIC BASED QUERY OPTIMIZATION

HEURISTIC BASED QUERY OPTIMIZATION Vishal Hatmode 1, Prof. Sonali Rangdale 2 Department of Information Technology, Siddhant College of Engineering, Pune, India 1,2 Abstract: In this paper, we will enlist

### Raima Database Manager Version 14.0 In-memory Database Engine

+ Raima Database Manager Version 14.0 In-memory Database Engine By Jeffrey R. Parsons, Senior Engineer January 2016 Abstract Raima Database Manager (RDM) v14.0 contains an all new data storage engine optimized

### Overview of Storage and Indexing

Overview of Storage and Indexing Chapter 8 How index-learning turns no student pale Yet holds the eel of science by the tail. -- Alexander Pope (1688-1744) Database Management Systems 3ed, R. Ramakrishnan

### 10. What SQL command can be used to add columns to a table? a. MODIFY TABLE TableName ADD COLUMN ColumnName b. MODIFY TABLE TableName ADD ColumnName c

1. The first generation of DBMS is represented by systems a. Hierarchical and CODASYL systems. b. Relational model c. Network model 2. Which of the following SQL statements are helpful in database redesign?

### Q4. What are data model? Explain the different data model with examples. Q8. Differentiate physical and logical data independence data models.

FAQs Introduction to Database Systems and Design Module 1: Introduction Data, Database, DBMS, DBA Q2. What is a catalogue? Explain the use of it in DBMS. Q3. Differentiate File System approach and Database

### Oracle Database 11g: SQL Tuning Workshop

Oracle University Contact Us: + 38516306373 Oracle Database 11g: SQL Tuning Workshop Duration: 3 Days What you will learn This Oracle Database 11g: SQL Tuning Workshop Release 2 training assists database

### Relational Database: Additional Operations on Relations; SQL

Relational Database: Additional Operations on Relations; SQL Greg Plaxton Theory in Programming Practice, Fall 2005 Department of Computer Science University of Texas at Austin Overview The course packet

### Overview of Storage and Indexing. Data on External Storage. Alternative File Organizations. Chapter 8

Overview of Storage and Indexing Chapter 8 How index-learning turns no student pale Yet holds the eel of science by the tail. -- Alexander Pope (1688-1744) Database Management Systems 3ed, R. Ramakrishnan

### In This Lecture. Physical Design. RAID Arrays. RAID Level 0. RAID Level 1. Physical DB Issues, Indexes, Query Optimisation. Physical DB Issues

In This Lecture Physical DB Issues, Indexes, Query Optimisation Database Systems Lecture 13 Natasha Alechina Physical DB Issues RAID arrays for recovery and speed Indexes and query efficiency Query optimisation

### Extended Operators in SQL and Relational Algebra

Extended Operators in SQL and Relational Algebra T. M. Murali September 15, 2010 Bags or Sets? So far, we have said that relational algebra and SQL operate on relations that are sets of tuples. Real RDBMSs

### Index Construction. Merging. Merging. Information Retrieval. ! Simple in-memory indexer. ! Indexing

Information Retrieval INFO 4300 / CS 4300! Indexing Inverted indexes Compression Index construction Ranking model Index Construction! Simple in-memory indexer t! t! t! Merging Merging! Merging addresses

### Partitioning under the hood in MySQL 5.5

Partitioning under the hood in MySQL 5.5 Mattias Jonsson, Partitioning developer Mikael Ronström, Partitioning author Who are we? Mikael is a founder of the technology behind NDB

### ICOM 6005 Database Management Systems Design. Dr. Manuel Rodríguez Martínez Electrical and Computer Engineering Department Lecture 2 August 23, 2001

ICOM 6005 Database Management Systems Design Dr. Manuel Rodríguez Martínez Electrical and Computer Engineering Department Lecture 2 August 23, 2001 Readings Read Chapter 1 of text book ICOM 6005 Dr. Manuel

### Index Selection Techniques in Data Warehouse Systems

Index Selection Techniques in Data Warehouse Systems Aliaksei Holubeu as a part of a Seminar Databases and Data Warehouses. Implementation and usage. Konstanz, June 3, 2005 2 Contents 1 DATA WAREHOUSES

### Understanding SQL Server Execution Plans. Klaus Aschenbrenner Independent SQL Server Consultant SQLpassion.at Twitter: @Aschenbrenner

Understanding SQL Server Execution Plans Klaus Aschenbrenner Independent SQL Server Consultant SQLpassion.at Twitter: @Aschenbrenner About me Independent SQL Server Consultant International Speaker, Author

### Oracle EXAM - 1Z0-117. Oracle Database 11g Release 2: SQL Tuning. Buy Full Product. http://www.examskey.com/1z0-117.html

Oracle EXAM - 1Z0-117 Oracle Database 11g Release 2: SQL Tuning Buy Full Product http://www.examskey.com/1z0-117.html Examskey Oracle 1Z0-117 exam demo product is here for you to test the quality of the

### Roadmap. Caching for Client/Server Database Architectures. Client/Server Database Architectures. Client-side caching. Page caching.

Roadmap Caching for Client/Server Database Architectures CPS 296.1 Topics in Database Systems!Semantic (query) caching Dar et al. Semantic Data Caching and Replacement. VLDB, 1996 Object caching (piggyback

### Suppose you are accessing elements of an array: ... or suppose you are dereferencing pointers:

CSE 100: B-TREE Memory accesses Suppose you are accessing elements of an array: if ( a[i] < a[j] ) {... or suppose you are dereferencing pointers: temp->next->next = elem->prev->prev;... or in general

### In this session, we use the table ZZTELE with approx. 115,000 records for the examples. The primary key is defined on the columns NAME,VORNAME,STR

1 2 2 3 In this session, we use the table ZZTELE with approx. 115,000 records for the examples. The primary key is defined on the columns NAME,VORNAME,STR The uniqueness of the primary key ensures that