Unlock your data for fast insights: dimensionless modeling with in-memory column store. By Vadim Orlov

Transcription

1 Unlock your data for fast insights: dimensionless modeling with in-memory column store By Vadim Orlov

2 I. DIMENSIONAL MODEL Dimensional modeling (also known as star or snowflake schema) was pioneered by Nielsen in 70s, and is considered the standard data modeling technique for business intelligence and analytics with nearly every BI ad reporting tool on the market supporting or requiring it. Both leading data warehousing methodologies, Inmon s Corporate Information Factory and Kimball s dimensional modeling, prescribe to use a dimensional model for exposing data to users. The Dimensional model provides a logical grouping of data by reporting usage (facts/dimensions), delivers good performance, enables cross-system integration and supports MDM (master data management). Why are we challenging something that has been proven to work well for over 40 years? First, let us look at the key drivers or architectural goals for dimensional model. Why does the dimensional model exist? 1. Performance The standard structure for operational data (OLTP) used by majority of applications is 3NF (third normal form) or higher. It eliminates data duplication, provides efficient storage and fast CRUD operations on narrow individual records or small data sets. A typical analytical query has a distinctly different workload; it retrieves a large and wide dataset, often exceeding millions of rows, that is typically aggregated. 3NF structure performs very poorly on an analytics workload because of the need to join several related tables to return all attributes. A Dimensional model improves query performance by minimizing number of joins for each query.

3 In the example below, a classic retail orders table structure shows a 3NF schema of 13 tables, which has been consolidated into a star schema model with 1 fact and 3 dimensions: 3NF Star Schema Customer Tier Customer-Location Customer dim_customer Region Location Orders Order Details fact_orders dim_store Store-Location Store Product dim_product Store Brand Product Subcategory Product Category The query that would have required 13 joins now requires only 3.

4 2. Merging Data from Multiple Sources/Systems & Tracking Historical Changes To support these 2 objectives, the dimensional model uses a concept of surrogate keys. Each dimension has a generic primary key (typically integer) that is generated by the ETL system. Dimensions are joined to the fact table by these surrogate keys. This supports loading data from multiple sources by eliminating potential source system key collision. It also supports tracking of slowly changing dimension attributes. 3. Data navigation 3NF data structures are notoriously complex, with a large number of joins, keys and technical naming conventions. The Dimensional model restructures the data, presenting it in a format easily understandable by a business analyst, with categorical attributes contained in dimensions and numerical measures in fact tables. The Dimensional model delivers all of the above benefits, but there is an associated cost the complexity of ETL code required to populate the star schema is very high. Let s look at typical tasks that each of the ETL components need to perform. Dimension loading package Extract required dimension attributes from each transactional source Identify whether record is new or updated/existing Generate new surrogate key and Insert new record For existing record, identify which attributes have changed and perform appropriate SCD type operation (insert or update). It is common to have multiple instances of dimension processing packages (one per dimension and source system). Fact loading package Extract source data from transactional source Perform dimension key lookups for each dimension. This is typically very resource-intensive and complex phase. Complications involve early- and late-arriving facts. Once all keys are looked up, insert the row into fact table Similar to dimension load, it is common to have multiple instances of fact loading packages (one per fact table and source system)

5 To summarize, the dimensional model delivers multiple benefits that enable effective reporting and analytics at the expense of the high complexity of the ETL code required to load it, associated data latency, high development and maintenance efforts. II. IN-MEMORY COLUMN STORE By modernizing this approach with newly available technology, we are able to address the key deficiencies of the dimensional model. The in-memory column store is a columnar database that keeps compressed data in columns using memory for fast compression and queries. Several leading database vendors have adopted this technology: Oracle and Microsoft SQL Server in on premise world, Amazon Redshift and Azure SQL DW in the cloud. Looking at how the SQL Server column store data is processed, it is very similar to dimensional model ETL: Data is loaded in 1M row segments. Within each segment, compression is applied for each column based on its data type and cardinality. Dictionary compression is typically used for text and low-cardinality numbers and dates. Dictionary compression replicates surrogate key generation and fact-dimension key lookup process in the dimensional model ETL load. The major difference is that each column becomes its own virtual dimension, which has some pros and cons that we will analyzed later. Externally the data is presented as one, single (wide) fact table, as illustrated below:

6 fact_orders product_id product_name product_subcategory product_category cusomer_id customer_name customer_address customer_type store_id store_location store_name store_region quantity price discount extended_price Product attributes Customer attributes Store attributes Measures In this example, attributes are grouped into logical dimension and measure groups similar to dimensional model. Think of this as flat star schema where both measures and dimension attributes reside in the same table. What are the main benefits of this reporting approach based on column store structure? 1. Performance, performance, performance. Microsoft claims 10x query performance improvements. We observed similar or better improvements, by testing on several datasets provided by our clients. What are the main drivers that enable us to achieve such significant increases in performance? Utilizing a Single flat table. If dimensional model improved performance by reducing the number of joins, column store allows for the completely elimination of joins by storing both dimensional attributes and measures in a single, wide compressed table.

7 Performance gains are even more significant as your data volumes grow and you start scaling out using distributed MPP architectures (Hadoop, Amazon Redshift, Azure SQL DW). Storing data in columns works well with a typical analytical query pattern, when just a handful of columns are usually queried; data is only read from pages that contain queried columns. The Column store structure eliminates the need to maintain expensive indexes on the fact table High compression minimizes disk access latency, shifting processing to fast memory. 2. Significant reduction in ETL complexity This structure eliminates dimensional processing, surrogate key generation, and SCD processing. The number of ETL packages is reduced from 1 per source/dimension/fact, to 1 per source/fact. This results in 10-20x reduction in number of ETL components required. Eliminating dimension key lookups in fact loading significantly reduces complexity of the load package and improves performance. 3. Simplified history tracking Historical changes (SCD2) are tracked automatically by inserting the latest values of current data into the fact table. No need to perform complex SCD2 surrogate key processing, comparisons and lookups. 4. Effective storage utilization Column store tables deliver a very high data compression (we observed 100x compression at one client). They are actually more effective in storage utilization than traditional dimensional models. You may think it is counter-intuitive - after all the data is fully de-normalized and dimensional attributes are being repeated multiple times. However, such frequently repeated attributes with low data cardinality compress extremely well which, results in more efficient space utilization. 5. Improved query capability Having all attributes in one flat table simplifies the data model and queries across multiple fact tables with varying grains. This is one area that Kimball s dimensional model doesn t handle very well. An example of this is joining the sales fact at a date/product grain to forecast fact and month/product category grain. The traditional dimensional model would require snowflaking of the month and product category dimensions, which increases the model and ETL complexity and impacts query performance. With column store, data is stored in the fact together and can be simple joined on applicable dimension attributes (month and product category). To summarize: column store delivers better performance, consumes less space, tracks history out of the box and significantly reduces ETL complexity. What is not to like?

8 Below are few scenarios, not necessarily column store drawbacks but something that will require additional consideration when designing reporting data model. Reporting by SCD1 type (latest dimension attribute value) requires extra processing. Potential options are: A. Maintain SCD1 attributes as virtual dimensions via views built on column store table that select latest values for each dimension attribute B. Add set of _latest columns to column store table and update prior values after each load Option A delivers better loading performance while option B brings faster query speed. Complex data structures with multiple fact tables require extra diligence in maintaining copies of dimensional attributes (conformed dimensions) across multiple fact tables. Fact table updates will require index reorganization to support optimal performance. To be fair fact updates don t work well in dimensional model either. Your best approach is to structure the facts as transactional and avoid the updates. Columns with very high cardinality data (e.g. transaction id, order number, guide, etc.) do not compress as well which may impact overall query performance. These columns are rarely used in analytic queries and leaving them out of column store is generally acceptable. If these columns are required, consider storing them in a separate, regular row table. III. IMPLEMENTATION PRIMER We recently implemented in-memory column store reporting model for a digital marketing organization. Implementation replaced legacy platform with over 1000 tables (staging, facts and dimensions) which were sourced from 50+ internal and external sources. Consolidated reporting tables from over a 100 to 4. Development was completed 60% faster compared to our original estimates based on traditional dimensional model methodology Business analysts loved the simplicity of having all data in single fact Everyone was happy with query performance we struggled to come up with a query that would take longer than few seconds Daily data loading time decreased from 6 hours to 1 hour DBAs were amazed by storage compression of 80x that was achieved

9 IV. CONCLUSION In-memory column store technology enables new, more efficient alternative to traditional data modeling methods. It delivers better performance, improves usability and significantly reduces ETL complexity which leads to faster implementation and reduced costs. For more information about modeling data to unlock its full analytic potential, please contact Vadim Orlov,