Database Internals (Overview)

Transcription

1 Database Internals (Overview) Eduardo Cunha de Almeida

2 Outline of the course Introduction Database Systems (E. Almeida) Distributed Hash Tables and P2P (C. Cassagne) NewSQL (D. Kim and J. Meira) NoSQL (D. Kim) MapReduce (E. Almeida and E. Lucas) Data Streaming (C. Cassagne) Machine Learning (C. Henard) Assignment (E. Almeida, J. Meira and E. Lucas) 2

3 Outline of the course Introduction Database Systems (E. Almeida) Distributed Hash Tables and P2P (C. Cassagne) NewSQL (D. Kim and J. Meira) NoSQL (D. Kim) MapReduce (E. Almeida and E. Lucas) Data Streaming (C. Cassagne) Machine Learning (C. Henard) Assignment (E. Almeida, J. Meira and E. Lucas) 3

4 Readings - Joseph Hellerstein, Michael Stonebraker and James Hamilton. Architecture of a Database System. Material at: - Hector Garcia-Molina, Jeffrey Ullman, Jennifer Widom. Database Systems: The Complete Book. Material at: 4

5 History 1970's INGRES Project at Berkeley -> INGRES corp, Sybase, MS SQL Server, PostgreSQL System-R Project at IBM -> DB2, Non-StopSQL, HP Allbase 5

6 History DB2 Oracle 1980's Informix Teradata Sybase SQL as the standard query language. 6

7 History Postgres MySQL 1990's Illustra (from Postgres) BerkeleyDB (embedded K/V) 7

8 History MonetDB (as open-source) Vertica Greenplum 2000's EnterpriseDB Infobright 8

9 History VoltDB (from H-Store) Facebook Presto Google F1 2010's Google Megastore NewSQL and NoSQL architectures 9

10 Database Genealogy [Felix Naumann, 13] 10

11 General DBMS Architecture SQL Interface, Apps Comm. manager Buffer Access Methods Log Lock Parser Auth. manager Catalog Optimizer Executor Query processor Adm & Monitoring Access control Scheduling Trans. Manager Shared components Proc. Manager DBMS Database Database 11

12 Database Layout 12

13 Database Layout Logic Physical DB Tablespace File Segments Extents DBMS page OS page

14 Tab customers Index Tab products

15 Tab customers Tab products Index on prodid Tablespaces

16 Fixed size pages slot1 Record (rid, attr1, attr2,, attr n) Fixed size{... Slot2 (empty) Slot n N N of slots

17 Variable size pages N-ary Storage Model (NSM) Variable record size rid=(i,...) rid=(i,...) Data Free space n * For write intensive applications Number of entries

18 NSM Example INSERT INTO T VALUES(2, 'MARIA', 18, CURITIBA, F); SELECT * FROM CUSTOMER WHERE ID=2; rid=(i,...) rid=(i,...) Data Free space SELECT AVG(AGE) FROM CUSTOMER; n *

19 NSM Example INSERT INTO T VALUES(2, 'MARIA', 18, CURITIBA, F); SELECT * FROM CUSTOMER WHERE ID=2; rid=(i,...) rid=(i,...) Data Free space SELECT AVG(AGE) FROM CUSTOMER; Few write operations - Read the entire tuple at once - Aggregations may read the entire DB!! n *

20 Columnar page Decomposition Storage Model (DSM) [Ailamaki, VLDB02] Page: Attr1 Page: Attr2 Page: Attr3 (rid1,attr1) (rid1,attr2) (rid1,attr3) (rid2,attr1) (rid2,attr2) (rid2,attr3) (rid_n,attr1) (rid_n,attr2) (rid_n,attr3) Improves aggregation and compression For read intensive applications

21 DSM Example INSERT INTO T VALUES(2, 'MARIA', 18, CURITIBA, F); SELECT * FROM CUSTOMER WHERE ID=2; SELECT AVG(AGE) FROM CUSTOMER; Page: Attr1 Page: Attr2 Page: Attr3 (rid1,attr1) (rid1,attr2) (rid1,attr3) (rid2,attr1) (rid2,attr2) (rid2,attr3) (rid_n,attr1) (rid_n,attr2) (rid_n,attr3)

22 DSM Example INSERT INTO T VALUES(2, 'MARIA', 18, CURITIBA, F); SELECT * FROM CUSTOMER WHERE ID=2; SELECT AVG(AGE) FROM CUSTOMER; Page: Attr1 Page: Attr2 Page: Attr3 (rid1,attr1) (rid1,attr2) (rid1,attr3) (rid2,attr1) (rid2,attr2) (rid2,attr3) (rid_n,attr1) (rid_n,attr2) (rid_n,attr3) - More write operations than NSM - Read one attribute at a time - Aggregations are fast do not read the entire DB!! - Agressive compression

23 Quiz What types of page layout are more likely for the following queries? Why? Q1 Q2 Q3 Q4 SELECT * FROM CUSTOMERS; SELECT A.ID, A.BALANCE, C.NAME FROM CUSTOMERS C, ACCOUNT A WHERE A.ID=C.ID SELECT SUM(BALANCE) FROM ACCOUNT; SELECT C.CITY, SUM(A.BALANCE) FROM CUSTOMERS C, ACCOUNT A WHERE A.ID=C.ID GROUP BY C.CITY;

24 Database Index 24

25 Access method: B-tree Index Adam Betty Smith

26 P0 k1 P1 k2 P km Pm Entrada de índice 17 <17 >17 Paginas não folha (só para direcionar) * 3* 5* 11* 14* Paginas folha (entradas de dados) Arquivo

27 Access method: Hash Index 4* 12* 32* * 21* 10* 11 Directory If insert 5*, then 5=101, bucket '01' (read backwards) Buckets

28 Clustered vs. Non-clustered index Index entries Data Records Non-clustered Clustered Physically sorted: only one clustered index per table

29 Quiz Considering the following queries, what types of index need to be implemented? Why? Q1 Q2 Q3 SELECT NAME, AGE FROM CUSTOMER WHERE AGE BETWEEN 18 AND 25; SELECT NAME, AGE FROM CUSTOMER WHERE CUST_ID= 1234; SELECT NAME, CITY, AGE, SSN FROM CUSTOMER WHERE AGE BETWEEN 18 AND 25 AND CITY ;

30 Overview of Query Processing 30

31 31

32 Query processing Query Query expression BALANCE (σ BALANCE > 2500 (ACCOUNT)) SELECT BALANCE FROM ACCOUNT WHERE BALANCE > 2500; σ BALANCE > 2500 ( BALANCE (ACCOUNT)) 32

33 Query processing Query Query expression BALANCE (σ BALANCE > 2500 (ACCOUNT)) SELECT BALANCE FROM ACCOUNT WHERE BALANCE > 2500; σ BALANCE > 2500 ( BALANCE (ACCOUNT)) May have n plans!!! How does the DBMS rewrite a query? 33

34 Query Transformation Rules 34

35 Query rewrite Rule: Selection operations are commutative 35

36 Query rewrite Rule: Join operations are commutative if the order of the attributes is not taken into account 36

37 Query rewrite Rule: Natural-joins are associative. 37

38 Query rewrite Rule: Natural-joins are associative. For theta-joins, this only holds if the following join involves attributes only from T2 and T3. 38

39 Query rewrite Rule: It distributes when all the attributes in the selection condition involve only the attributes of one of the expressions (say T1) 39

40 Query rewrite Rule: Conjunctive selection operations can be split into a sequence of individual selections 40

41 Further rules can be found in the readings. 41

42 Quiz Please, rewrite the following expressions using the presented rules and the following relations: ACCOUNT(BALANCE, ACC_ID, C_ID, B_ID), CUSTOMER(C_ID), BRANCH(B_ID, B_NAME) Q1 σ BALANCE > 2500 AND C_ID > 1000 (ACCOUNT X CUSTOMER) Q2 Q3 σ BALANCE > 2500 (σ B_ID = 1000 (ACCOUNT X BRANCH)) BALANCE, C_ID (σ BALANCE > 2500 (ACCOUNT X CUSTOMER)) Q4 BALANCE, ACC_ID (CUSTOMER X ACCOUNT) Q5 BALANCE, ACC_ID (CUSTOMER X (ACCOUNT X BRANCH))

43 Quiz Please, rewrite the following expressions using the presented rules and the following relations: ACCOUNT(BALANCE, ACC_ID, C_ID, B_ID), CUSTOMER(C_ID), BRANCH(B_ID, B_NAME) Q1 σ BALANCE > 2500 (σ C_ID > 1000 (ACCOUNT X CUSTOMER)) Q2 σ B_ID = 1000 (σ BALANCE > 2500 (ACCOUNT X BRANCH)) Q3 Q4 Q5 σ BALANCE > 2500 ( BALANCE, C_ID (ACCOUNT X CUSTOMER)) BALANCE, ACC_ID (CUSTOMER X ACCOUNT) BALANCE, ACC_ID (BRANCH x (CUSTOMER X ACCOUNT ))

44 Database Catalog The catalog is a meta-database that stores informations about: DBMS and BD Data structures Buffer and page sizes Indices Views 44

45 Database Catalog The catalog is a meta-database that stores informations about: Statistics Cardinality of tables and indices Domain values Page sizes for tables and indices 45

46 Database Catalog ALL_TABLES view from the Oracle catalog 46

47 Query optimization 47

48 Operators I/O cost for Selection ( ) Scan O(M), where M = number of pages Scan on sorted data O(log M), where M = number of pages Index scan (Clustered index) O(h + 1), where h = height of the index tree Index scan (Non-Clustered index) O(h + n), where n = number of fetched tuples. n = M if the tuples are spread across all pages (worst case). 48

49 Quiz Compute of the cost for a query selecting 10% of a table 'R' based on the following information from the catalog: Relation 'R' Tuples/page ratio 10,000 tuples 100 tuples Scan Scan on sorted Clustered index scan Non-clustered index scan 49

50 Quiz Compute of the I/O cost for a query selecting 10% of a table 'R' based on the following information from the catalog: Relation 'R' Tuples/page ratio 10,000 tuples 100 tuples Scan Scan on sorted Clustered index scan Non-clustered index scan 100 I/O 6.6 I/O 3 I/O 103 I/O 50

51 Transactions 51

52 Transactions Definition: A transaction is an atomic unit of work that is either completed in its entirety or not done at all. [Navathe and Elmasri, 2005] T1 Database r(balance) balance = 200 balance += 100 r(x) w(balance) balance =

53 Transaction State Transition [Elmasri and Navathe, 2005] Read, write Begin active End Partially commited Commit commited abort Failed Terminated 53

54 ACID properties Atomicity: a transaction is either completed in its entirety or not done at all Consistency: a transaction is consistency preserving if its complete execution takes the database from one consistent state to another Isolation: The execution of a transaction should not be interfered Durability: the changes applied by a commit must persist in the database 54

55 Schedules A schedule orders the execution of concurrent transactions in an interleaving fashion. T1 T2 r(x) r(x) w(x) w(x) 55

56 Concurrency Conflicts Concurrency problems: dirty read and lost update T1 T2 T1 T2 r(balance) r(balance) balance += 100 r(x) r(balance) w(balance) w(x) balance += 100 w(x) w(x) r(balance) w(x) balance -= 100 abort balance-=100 w(balance) w(balance) w(balance) 56

57 Concurrency Conflicts Concurrency problems: dirty read and lost update T1 T2 T1 T2 r(balance) r(balance) balance += 100 r(balance) w(balance) balance += 100 r(balance) balance -= 100 abort balance-=100 w(balance) w(balance) w(balance) 57

58 Algorithm: Testing Conflict Serializability Create a node for each transaction T in schedule S; Create an edge Ti -> Tj for each r(x) in Tj after w(x) in Ti Create an edge Ti -> Tj for each w(x) in Tj after r(x) in Ti Create an edge Ti -> Tj for each w(x) in Tj after w(x) in Ti Schedule S is serializable if and only if the precedence graph has no cycles. For example: Not serializable T1 r(x) w(x) T2 r(x) w(x) T1 T2 58

59 Hands-on Which of the following schedules is conflict serializable? S1 - T1 : r(x),t3 : r(x),t1 : w(x),t2 : r(x),t3 : w(x) S2 - T1 : r(x),t2 : r(x),t1 : w(x),t2 : w(x) S3 - T1 : r(x),t2 : r(y),t3 : w(x),t2 : r(x),t1 : r(x) 59