Business Intelligence. 9. Real Time Data Warehouses OLAP December 2013.

Transcription

1 Business Intelligence 9. Real Time Data Warehouses OLAP December 2013.

2 Multiple currencies (1) A large international company Transactions in N different currencies Standard request: report sales in standard currency (e.g. ) and in local currency Necessary even when we know the location where the transaction took place because the relation location: currency is not always 1:1 dcurrency idcurrency currencyname cracun idlocalcurrency idlocation netamntlocalcurr grosamntlocalcurr discntlocalcurr netamnteuro grosamnteuro discnteuro 2

3 Multiple currencies (2) The requirements can be more complex: Report sales in arbitrary currency, e.g.: We are introducing a conversion table The table should be complete CHF kn dcurrency idcurrency currencyname fsales idlocalcurrency idlocation netamntlocalcurr grosamntlocalcurr discntlocalcurr netamnteuro grosamnteuro discnteuro CurrConvTable idconversiondate idcurrencyfrom idcurrencyto fromtoconvfactor tofromconvfactor Two values - does not have to be symmetrical 3

4 Multiple units of measure Different organizational units within the company want to see the reports expressed in different units of measurement For example: Production wants to see in transport pallets or delivered packages Sales and Marketing wants to see in retail packages or individual packages of products Conversion factor in dimension table Add to the fact table: If the number of measures is small express in all units of measure If not, express in one unit of measure and store N additional conversion factors, e.g. if there are 10 measures and 5 units of measure Ako je veći broj, izraziti u referentnoj mjernoj jedinici i pohraniti N there will be 10 measures and 4 conversion factors in the fact table 4

5 Real Time Data Warehouses Poslovna inteligencija

6 Real-Time Data Warehouse (1) So far, typically: joint (batch) load in a period of DW inactivity The need for up-to-date data (e.g. CRM - Customer Relationship Management) New Paradigm: continuous updating (continuous operation) First of all, we should consider: Do we really need a real time data warehouse? To what extent? It is assumed that data can be incrementally loaded The concept of real time data warehouse refers to an attempt to reduce the DW's delay (with regards to the source data) to a minimum or to an acceptable level. 6

7 Real-Time Data Warehouse (2) Different names or understanding of the term: 1. On time Information provided in accordance with the business needs, not necessarily real time. 2. Simulated real-time Key business analysis are available in real time, otherwise classical ETL (e.g. EII - discussed later). 3. Right time Data is transferred as quickly as possible, given the options. Delays are still present. When using RT DW, we usually refer to this option. 4. Real time Strictest definition, no delays. Very difficult to achieve, "hard" on the source system. 7

8 Real-Time Data Warehouse (3) Real-time ETL "misnomer" : Not really real-time Sometimes not even "ETL" Software that asynchronously copies data into the data warehouse with some urgency (within N minutes of data occurrence) It is possible to simply reduce the interval between two loading data, but that is not considered to be a real time DW, and has a negative impact on the source systems Zero-latency: embedding logic to copy tuples to the data warehouse from within source system transactions - very wrong! 8

9 Real-Time Data Warehouse (4) Three categories of problems: A. CDC - Change Data Capture B. How to continuously input (update) the data warehouse C. Overall ETL strategies: 1. Microbatch ETL 2. EAI - Enterprise Application Integration 3. EII Enterprise Information Integration When implementing C, A and/or B are used 9

10 A. Change Data Capture(1) 1. Timestamps Add a timestamp column to every table (insert, update) Robust and simple Problem with deletes How to distinguish insert and update? Negative impact on the performance of the source system Indexing timestamps (both positive and negative impacts) 2. ETL log tables Triggers - enter the transaction id, status, operation, timestamp and into dedicated ETL log table (each table has it's log table) When transferring data we join the production tables with log tables Processed log records can be flagged or archived (moved) Does not have significant impact on the source system 10

11 A. Change Data Capture(2) 3. DBMS log scrapers Logical logs (already present in RDBMS systems) are used - added value All operations can be reproduced Does not have an imapct on the source system Complex (expensive)? Platform (vendor) specific 4. (Network sniffers) Sniff and filter traffic Useful in web server farms Makes sense whenever there is a data stream which requires the analysis (e.g. telecom routing calls) Complex and rarely applicable 11

12 B. Updating The RT DW(1) How to merge the newly arrived data into the existing DW? (Direct) trickle feed Insert data, as it arrives Interaction with the currently running queries Performance penalites, especially when indexes are present Trickle & flip RT table/partition - duplicate fact table (often all in memory) Small amount of data Minimal or none indexing of the RT table RT table's content is periodically merged (moved) to the "static" table/partition With regards to location of the RT table/partition: real-time partition (same DB) external real-time partition, external real-time data cache (external DB or...) 12

13 B. Updating The RT DW(2) RT partition, example: fsales ddate dclient dstore... trickle frtsales Problem: should be supported by the tool If not external, UNION view can be defined: CREATE VIEW frealtimesales(...) AS SELECT * FROM fsales UNION SELECT * FROM frtsales 13

14 C. ETL strategies: Microbatch ETL (1) Conventional ETL tools: More o en (day hour... ) Less data "Once a day", RT partition data is: a) Moved to the "static" partition or b) deleted (allowing "regular" ETL to refresh the static partition) DB CDC microbatch ETL Dim. mngr. microbatch RT part. b) a) "static" partition 14

15 ETL strategies, digression: Dimensional manager What if new dimension records are created (e.g. new client)? SCDs? Possible compromises: surrogate dimension records Ignoring updates Smart usage of minidimensions Regard as Type 1 in the "real time", and as Type 2 affterwards in the "regular" ETL The role of the DM is to define, maintain and publish conformed dimensions 15

16 C. ETL strategies: EAI - Enterprise Application Integration EAI is the use of software and computer systems architectural principles to integrate a set of enterprise computer applications. Wikipedia Flexible, modular, upgradeable Can be slow Can be complex and hard to implement adapter broker forder adapter DB adapter fcall adapter DB XML dclient DM adapter dproduct DM adapter dclient DM dproduct DM EAI middleware 16

17 C. ETL tools: CTF Capture, Transform and Flow Relatively new category of data-integration tools CTF tool's functionality is second to mature ETL tools (mostly light transformations, e.g. date standardizations, type casts, field lengths, etc.) Application layer is bypassed - direct DB to DB transfer: Data goes into the staging area From there, more complex transformations are preformed on insert triggers or via microbatch strategy u konačnici podaci opet odlaze u RT particije Easier and simpler than EAI, less powerful 17

18 C. ETL tools: : EII Enterprise Information Integration Data is transferred on analytician's demand when needed Virutal integration systems, integrate multiple data sources Instead of DW, the destination is report, file, OLE DB, XML object,... Can interoperate with the DW Limited functionality Good for start, when RT is needed, can serve as a pilot project for the RT DW 18

19 RT DW - in conclusion Still relatively rarely represented Numerous problems: Aggregates Caching (e.g. in web apps) Compliance with the "static ETL"? Tool support Dimension manager Error management Is it really needed and to what extent? Kimball in Chapter 11 ( ) * brings the ETL strategy selection criteria based on user's needs Further reading: ETL.pdf * The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data 19

20 OLAP Poslovna inteligencija Business Intelligence 2013/2014

21 OLAP (engl. On-Line Analytical Processing) (1) OLAP is an approach to data analysis that enables a user to easily and selectively extract and view data from different points of view, based on the multidimensional data structure called a cube. Defined by Edgar F. Codd (via 12 rules and 18 properties) FASMI (N. Pendse, R. Creeth, 1995.): Fast Analysis Shared Multidimensional Information More at: 21

22 Fast Requirement: 1-5 seconds, seldomly over 20 seconds Users consider the 30 + seconds queries as failed How? Special data storage techniques Preaggregation and precalculation Specific hardware solutions Aggregation is the fundamental technique, time-space tradeoff 22

23 How to achieve speed improvements? fsales iddate... ddate iddate year month day... SELECT year, SUM(amount) as sumamount FROM fsales, ddate WHERE fsales.iddate = ddate.iddate GROUP BY year year sumamount

24 Aggregation is the fundamental technique used to achieve speedy responeses in OLAP systems aggsalesyear year sumamount aggsalesmonth year month sumamount aggsalesdayofweek dayofweek sumamount fsales iddate... OLAP AN ddate sifdatum year month day... SELECT year, SUM(amount) as sumamount FROM fsales, ddate WHERE fsales.iddate = ddate.iddate GROUP BY year 24

25 Analysis, Shared ANALYSIS* means that the system can cope with any business logic and statistical analysis that is relevant for the application and the user, and keep it easy enough for the target user. Although some pre-programming may be needed, it is not acceptable if all application definitions have to be done using a professional 4GL. SHARED* means that the system implements all the security requirements for confidentiality (possibly down to cell level) and, if multiple write access is needed, concurrent update locking at an appropriate level. *Taken from: 25

26 Multidimensional, Information The OLAP system must provide a multidimensional* conceptual view of the data, including full support for hierarchies and multiple hierarchies Underlying database technology used to provide multidimensional conceptual view is not specified INFORMATION* is all of the data and derived information needed, wherever it is and however much is relevant for the application. We are measuring the capacity of various products in terms of how much input data they can handle, not how many Gigabytes they take to store it. *Taken from: 26

27 The OLAP system must provide a multidimensional conceptual view of the data, including full support for hierarchies and multiple hierarchies , , ožujak , , , , , , , , , , , , , , , ,00 ožujak Total * 4,16 siječanj 3,71 veljača 3,72 1 Total * 3,74 2 3,62 3 3,14 4 3, Total * 3,52 Year Quarter Month Day Drill Up Drill Down 27

28 OLAP: hierachical multidimensional view and crossjoin 28

29 OLAP terminology: slice & dice Course Time Course Time 29

30 OLAP comment Technology is not prescribed and it is not specified in any way how OLAP achieves the desired properties Vendors put the effort themselves to meet the OLAP requirements 30

31 ROLAP, MOLAP, HOLAP Three models (with regards to aggregate's location): 1. ROLAP (Relational OLAP) aggregates and data in the rel. DB 2. MOLAP (Multidimensional OLAP) aggregates and data on the OLAP server 3. HOLAP (Hybrid OLAP) aggregates on the OLAP server, data in the rel. DB 31

32 Microsoft Analysis Services Poslovna inteligencija Business Intelligence 2013/2014

33 Analysis Services tutorials and links Analysis Services Tutorial: Pripadajući Visual Studio Solutions: Movies: especially: SSAS Creating Dimensions SSAS Creating Hierarchies SSAS Attribute Relationships 33

34 BIDS/SQL Server Data Tools New project In version 2008, VS Shell from VS2008 is used BI Dev Studio In version 2012, VS Shell from VS2010 is used SQL Server Data Tools New project Analysis Services Project Data Sources new data source Data Source Views new data source view 34

35 OLAP structures definition Define the connection, define the Data Source View Dimension wizard first: Define dimensions Define hierarchies, hide (AttributHierarchyVisible=false) hierarchy member attributes Define attribute relationships (Attribute Relationships Tab) instructs OLAP system to the attribute relationships (otherwise every query would have to be resolved to the bottom level). : Enhances query performance Enhances data storage Reduced processig time the single most important thing you can do for performance defining attribute relationships improves dimension, partition, and query processing performance Then, Cube wizard 35

36 Hierarchies In dimensions, data are structured into hierarchies One dim can have N hierarchies Hierarchy can be one level deep E.g.: One level hierarchy (day of week) Three level hierarchy (year, month, day) Two types of hierarchies: 1. Natural, e.g.: year month day 2. The other kind (reporting, ), e.g.: Gender Marital status Income To the user navigating a hierarchy, these two types of user hierarchies are the same We could extract month as a separate hierarchy, what is the difference? (QuickHit#2 video) 36

37 Important dimension attributes Name (programmatic) attribute name, usually the same or similar as the one in the relational table KeyColumns key(s) that uniquely determine the attribute. Can be composite E.g. idstudent for studname Warning: for montname it is not (month) but (year, month) Name Column: that provides the name of the attribute that is displayed to users: If not assigned, KeyColumn is used If KeyColumn is composite(e.g. in montname) then it must be defined Usage: Regular ( common, non-key attribute) Key (key dimension attribute, each dim has exactly one) Parent (used in parent-child dimensions, e.g. (superior)orgunit) OrderBy, OrderByAttribute used for ordering, explained later AttributeHierarchyVisible defines wheteher attribute is visible outside its hierarchical position - as a separate "hierarchy" 37

38 Ordering (OrderBy and OrderByAttribute ) OrderBy OrderByAttribute Order by Key (NA) the defined key for this level: idemployee Name (default) (NA) the defined name for this level : EmployeeFullName Attribute Key Last Name the attribute key defined in the OrderByAttribute, that is Last Name (in this example it is the same attribute Last Name, but in general it can be some other int, ) Attribute Name Last Name the attribute key defined in the OrderByAttribute, that is Last Name (shown in the picture below) Po prezimenu 38

39 Attribute relationships(obligatory!) * Employee is type 2 SCD so the LastName (and full name) are fixec, that is, when they change a new record is created Relationship types: Rigid - unchangeable (better performance if we define the connection as rigid AS can keep previous aggregation when updating cubes) E.g.. date month year Denoted by a black arrow Flexible (default) changable in time E.g. student phone, address, Denoted by a gray arrow 39

40 For the next lecture Define ddate dimension: Open new Analysis Services project (VS2008) Define data source Define data source view Make a new dimension based on the table ddate: Define hierarchy: year-> month -> day Use composite key for month name, order by key Save, process, browse 40