Lecture Data Warehouse Systems

Transcription

1 Lecture Data Warehouse Systems Eva Zangerle SS 2013

2 PART C: Novel Approaches Column-Stores

3 Horizontal/Vertical Partitioning Horizontal Partitions Master Table Vertical Partitions Primary Key 3

4 Motivation Most relational database systems store data row by row, i.e. Disadvantage: OLAP systems like data warehouse systems frequently want to read only a few columns of all rows If the data is stored row by row, they have to read a lot of data the query does not actually need 4

5 Column Stores So: Why not storing data column by column? Column Stores 5

6 Sample OLAP Query Consider the OLAP query SELECT avg(totalprice) FROM Order In a column store, just the data marked red must be read 6

7 Sample OLTP Query Of course, for a OLTP query like SELECT* from Order where orderkey = storing data row by row is better 7

8 OLAP queries usually are OLTP vs. OLAP More exploratory you do not exactly know in advance what some analyst wants to know, so optimizations like introducing indices are rather difficult Longer lasting More read-oriented than write-oriented the data is typically produced in some OLTP system, and then transferred to the OLAP system (=DWH) in batch runs Running OLAP queries on a separate system is typically a good idea for performance reasons More attribute focused than entity focused (e.g. calculate the sum of a column vs. read all columns of one specific row) 8

9 Approaches Building a Column Store Emulate on top of a row store Use a row oriented query executor on top of a column-oriented storage layer Use a column oriented query executor on top of a columnoriented storage layer (column store) 9

10 Approach 1 Called Decomposition Storage Model (DSM) or Vertical Partitioning Emulate by splitting each table T up into two-column tables (primary key, i-th column of T) and storing them in a row store Easy to implement Any current relational database system can be used Additional wrapper is necessary 10

11 Building a Column Store Approach 1 Primary key column must be stored multiple times Overhead because of tuple headers (for each tuple, about 8 bytes of administrative informations are stored) Easily two or three times the disk space of the original table required In summary: easy to implement, but consumes a huge amount of both disk space and I/O bandwidth Scientists tested this approach using the SSBM (star schema benchmark) and reported a performance decrease instead of an increase 11

12 Approach 2 Modify the storage layer of a conventional relational database system to store data column by column instead of row by row Unchanged schema at the logical level Data is stored column by column on storage level, plus tuple headers separately When executing a query Required data (subsets of columns) is fetched from storage layer Tuples containing exactly the required columns are constructed (next slide) Finally a row oriented query executor processes the query 12

13 Building a Column Store Approach 2 Constructing tuples from the individually stored columns Implicit column positions are used Each tuple is assigned an implicit position i In each column, the attribute value of tuple i is stored at the i-th position Construct the i-th tuple by taking the i-th value from each column 13

14 Building a Column Store Approach 2 Modify the storage layer of a conventional relational database system to store data column by column instead of row by row In theory One does not need to write a complex query executor containing a huge amount of optimization logic But can use a mature conventional relational database system for this In practice Integration into the existing DBS can be difficult to impossible 14

15 Approach 3 Rewrite both storage layer and query executor from scratch On storage level, data is stored column by column, maybe redundantly for reasons of efficiency The query executor works in column-oriented fashion Huge implementation effort But many chances for optimizations 15

16 Comparing Column Store Architectures Outline Benchmarks Why you want to use them A concrete example: The Star System Benchmark Materialized views: A row store approach and its limitations Comparing the three approaches from 2.3 using the Star System Benchmark 16

17 Benchmarks Comparing different Implementations or Algorithms Goal: One wants to find out, which implementation, algorithm, etc. performs best for solving some problem Here: Which kind of database system performs best for processing OLAP queries To answer this question, one basically has to execute queries and measure the results, however Which queries should one choose? Are they representative for the daily work with the system? Does the test setup miss some important case? What do the results tell us with respect to the measurements of other people? Solution: Use a standardized benchmark 17

18 The Star Schema Benchmark A data Warehouse Benchmark 18

19 The Star Schema Benchmark Using scalefactor, different sizes of the data warehouse can be simulated Contains 13 OLAP queries like SELECT SUM (o.extendedprice * o.discount) as revenue FROM Order o, Date d WHERE o.date = d.datekey AND d.year = 1993 AND o.discount between 1 and 3 AND o.quantity < 25 19

20 The Star Schema Benchmark Contains 13 OLAP queries like select d.year, s.nation, p.category, sum(o.revenue o.supplycost) as profit1 from date d, customer c, supplier s, part p, order o where o.customerkey = c.customerkey and o.supplierkey = s.supplierkey and o.partkey = p.partkey and o.date = d.datekey and c.region = `AMERICA` and s.region = `AMERICA` and (d.year = 1997 or d.year = 1998) and (p.mfgr = `MFGR#1` or p.mfgr = `MFGR#2`) group by d.year, s.nation, p.category order by d.year asc, s.nation asc, p.category asc 20

21 Materialized Views for OLAP Optimization for conventional relational databases: create materialized views containing only the columns needed for answering the expected queries Of course, the original tables still exist Advantage: No need to fetch data not needed for the query at hand; hence IO is minimzed Disadvantage: Knowledge about expected queries is needed in advance Remember: OLAP queries are often used for analyzing things, which can be a quite creative process Nevertheless, measurements using this approach are useful as reference when evaluating the column store approaches 21

22 Remember Comparing CS Approaches Approach 1: Store data in two-column tables in a Row-Store Approach 2: Row oriented query executor on top of column-oriented storage layer Approach 3: Column oriented query executor on top of column-oriented storage layer 22

23 Comparing CS Approaches About the presented performance measuremens Obtained using the SSBM benchmark Each repeated several times About the evaluated implementations Most OLAP database systems are commercial, so [1] choose one of them for evaluating both the materialized view and the Decomposite Storage Model approach According to [1], mentioning which database system they choose was forbidden for license reasons (approach 1: two column tables, approach 2: row oriented query executor on top of selfwritten column-oriented storage layer, approach 3: everything self-written) 23

24 Comparing CS Approaches About the evaluated implementations Ideally, when comparing implementations, they should only differ in the change or improvement one actually wants to evaluate Hence Approaches 1 and 2 should share the query executor Approaches 2 and 3 should share the storage layer Here Proprietary database system for approach 1 Self written implementation for approaches 2 and 3 (approach 1: two column tables, approach 2: row oriented query executor on top of self-written column-oriented storage layer, approach 3: everything self-written) 24

25 Comparing CS Approaches About the evaluated implementations So: Approaches 1 and 2 did not share the query executor Optimized commercial vs. basic self-written version According to [1], additional experiments showed that the commercial version alone is about two or three times faster than the basic self written version In summary Not a perfect comparison Keep this in mind when having a look on the results 25

26 Comparing CS Approaches Performance comparison using the SSBM between Traditional: Traditional row-oriented implementation Materialized views on a row store Approach 1: Column store using two column row store tables Approach 2: Row store query executor on top of column store storage layer Approach 3: Column store completely self written 26

27 C-Store C-Store: A Column Store implementation Developed in about 2005 within a dissertation So: not a fully developed and optimized database system, rather a proof of concept Its successor today is a commercial database system 27

28 C-Store Architecture Data is physically stored column by column The users interact with a relational interface, using SQL Each table is physically represented by a collection of projections Projection: Subset of a table, containing all rows and some columns There is one projection covering the whole table, as joining them is slow Each column can be part of any number of projections Each projection has its own sort order, shared by all its columns 28

29 C-Store Architecture Advantage: Each column can be stored in multiple sort orders, the query optimizer can choose a projection based on the query at hand and the primary, secondary, etc. sort orders in the projections Disadvantage: Data is stored redundantly, updates are more expensive, more space is required But OLAP systems work mostly read oriented Memory has become cheap Here: new data arrives in a write store and is transferred to the read store e.g. once a day (batch updates) 29

30 C-Store Storage Layer Stores data in 64 KB blocks Indices to blocks If sorted: Sparse index on column value Always: Sparse index on tuple ID Although Column Stores very much work with full table scans, those indices are needed in some situations Some operators can work on position lists (details later) E.g. the input data of such an operator may tell it do something on the tuples with IDs 64, 332, 749, 1212, When filtering for certain tuples 30

31 Compression Reduced disk I/O Reduced amount of storage Seek time reduced locality of data Buffers may hold larger amount of data 31

32 Compression in Column Stores In row stores, often dictionary based schemes are used Encoding multiple values of a column at a time is not easily possible But: The values of a column are usually much more similar than the different attribute values of a tuple In Column Stores More compression algorithms can be used Their compression ratio is often higher Columns hold similar data Iterate over page of values vs. page of tuples: easier and faster Exploit sort order Also operators working directly on compressed data are possible 32

33 Compression in Column Stores Note: The focus of compression in column stores is to maximize query performance, not minimize storage sizes Improved IO performance vs. CPU cost for decompression tradeoff 33

34 Null suppression Compression Schemes Consecutive null values or blanks are deleted Replaced with description how many null values there were and where these occurred E.g. store number of bytes previously occupied by null values variable field size: description of size required 34

35 Compression Schemes Run Length Encoding Replace runs of the same value by a compact description e.g. given a sorted integer column, instead of storing store (e.g. at position 9, a sequence of 10 times 6 starts) However: For a run of length 1, three data items instead of one will be stored So, only useful if there actually are long runs of the same value 35

36 Dictionary encoding Compression Schemes Replace frequent patterns by codes E.g. replace the strings red, blue, yellow and green by the bit sequences 00, 01, 10, 11 Row stores: Values from different tuples usually cannot be mixed Column stores: Mixing is possible, e.g represents the strings red, green, blue, green in the same column of four consecutive rows Decompression using bitshift operations Algorithms like Huffman-encoding based on popularity distribution of characters 36

37 Bit-Vector encoding Compression Schemes Situation: There is a column with very few distinct values, e.g. containing only the two strings Yes, No" Store a bit vector for each string, e.g. for the string sequence Yes Yes Yes No Yes No No No Unknown in subsequent rows of a column, store the bitvectors , and Performance in Column Stores questionable Merge of bitvectors for complete decompression is expensive Algorithms for further compression (e.g. runs of the same value like in ) exist 37

38 Compression Schemes Heavyweight Compression Schemes Algorithms like Lempel-Ziv Lempel-Ziv is the algorithm used for gzip They use a possibly sophisticated algorithm to compress big blocks of data at a time minimize storage size 38

39 Operating on Position Lists Join in a column store can produce position lists instead of tuples (compression-aware) 39

40 Operating on Compressed Data Operating directly on compressed data possible in some situations IF Column c1 is not compressed AND Column c2 is RLE compressed FOR EACH VALUE valc1 WITH POSITION i in c1 DO FOR EACH TRIPLE t with VAL V, STARTPOS j AND RUNLEN k in c2 DO IF joinpredicate(valc1,v) THEN OUTPUT-LEFT: NEW RLE TRIPLE (NULL, i, k) 40 OUTPUT-RIGHT: (j j+k-1)

41 Different Compression Schemes How to deal with different compression algorithms at once Solution: Software design Introduce compression blocks Storage blocks contain any number of compression blocks A compression block hides its contents, provides only access through the methods isonevalue: true if the block contains one value on one/more positions isvaluesorted: true if the block is sorted isposcontig: true if in the block is a consecutive subset of a column getnext: iterator access: return the next value asarray: decompress and return as an array getsize: returns the number of values in the block getstartvalue: returns the first value of the block getendposition: returns the position of the last value in the block 41

42 Compression: Summary According to [1], enabling compression can make a Column Store about two times faster Different techniques Null suppression, Run Length Encoding Dictionary Encoding Bit vector Encoding Heavyweight compression schemes Different techniques more or less useful in different situations E.g. depending on sort order, distribution of data values, etc. Maybe choose compression strategy dynamically based on cost models, heuristics, etc., without explicit administration 42

43 Materialization Strategies A column store stores data column by column Users and applications usually expect row-oriented results Hence: When (during the process of query execution) should the tuples be constructed? 43

44 Early Materialization Query like SELECT a,b,c FROM T WHERE a < const1 AND b < const2 AND c < const3 Early materialization Construct tuples early Pass them between the operators 44

45 Late Materialization Query like SELECT a,b,c FROM T WHERE a < const1 AND b < const2 AND c < const3 Late materialization Construct tuples as late as possible Operate on position information before 45

46 Early vs. Late Materialization Early materialization Certain column values may have to be accessed multiple times in a query plan E.g. (1) filter for a predicate A < 7, (2) return column A Early materialization has less overhead in such situations However, even late materialization shouldn t have multiple disk accesses due to caching either Late materialization Often, operating directly on positions or compressed data is possible Certain tuples need not be constructed at all (e.g. because discarded by predicates or in aggregations) 46

47 Early vs. Late Materialization The choice between the strategies is not always obvious Usually, Late Materialization is better for Aggregative queries Queries with highly selective predicates Queries on compressed data Enabling Late Materialization made the SSBM benchmark queries about two to three times faster [1] 47

48 Results Dictionary compression achieves higher results the more data is stored Generally, dictionary compression and LZ work best in regards to column storage size Query performance directly on compressed data: RLE and dictionary compression work best 48

49 Some Real World Systems Sybase IQ (commercial) Vertica, the commercial successor of C-Store MonetDB (open source) 49

50 Summary Column Stores store data column by column Using them can speed up OLAP applications Three possible alternatives Decomposition Storage Model Replace storage layer of row oriented database system Rewrite from scratch Implementing a Column Store C-Store architecture Compression Early vs. Late Materialization 50

51 10 Literature [1] Daniel J. Abadi: Query execution in column-oriented database systems, Ph.D. thesis, Massachusetts Institute of Technology, [2] M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. J. O Neil, P. E. O Neil, A. Rasin, N. Tran, and S. B. Zdonik, C-Store: A column-oriented DBMS, in Proc. of the 31st Int. Conf. on Very Large Databases (VLDB 05), Trondheim, 30 th August September 2 nd 2005, pp [3] Hasso Plattner. A common database approach for OLTP and OLAP using an in-memory column database, in Proc. of the 35th Int. Conf. on Management of Data, Providence, Rhode Island, USA, June 29 th July 2 nd 2009, pp [4] Daniel Abadi, Samuel Madden, and Miguel Ferreira. Integrating compression and execution in column-oriented database systems. In Proceedings of the 2006 ACM SIGMOD international conference on Management of data (SIGMOD '06). ACM, New York, NY, USA,