Introduction to OLAP

Transcription

1 Introduction to OLAP Sources: Daniel s intro slide from 2003; Owen s presentation to NRC, 2003.

2 Overview Review of the industry Motivation through example Definitions!!!

3 OLAP is important? 7% growth in 2003 (most recent result) Source: OLAP Report of 8 April (

4 Historical Perspective 1970 Codd proposes relational model 1980 SQL becomes a commercial success (Oracle, IBM) 1993 Codd coined OLAP, Excel offers Pivot Tables 1997 MOLAP vs ROLAP debate 1999 SQL-99 offers some OLAP functionality

5 Industry standards Name Status Platform Proponent OLE DB In use Wintel Microsoft XML Analysis seems stalled? SOAP Microsoft, Hyperion JOLAP finalized Aug Java (J2EE) IBM, Oracle, Hyperion, Sun

6 Who sells OLAP?

7 Open-source OLAP: Mondrian Mondrian ( uses one of several relational databases for data storage. Written in Java. It supports MDX (OLE DB) query language. It has a JOLAP API. It has an XMLA interface. Mondrian installation is not trivial. My experience: the implementation may be incomplete or buggy. Nevertheless, let s try to use it.

8 OLAP: Sales Example Assume one (real-valued) measure value, Sales Amount and dimensions Item, Place, Month. Example fact: Iced Tea was sold in Auckland in January. Measure: $20k Maybe no fact about Iced Tea in Auckland in August. Mapping Item Place Month Sales Amount is our cube.

9 Dimensions Recall that dimension values commonly organized into containment hierarchies. Example: Schema:. Place is organized by City, Province, Country, Instance: Invercargill is in Otago, Christchurch is in Canterbury, Boston is in Massachusetts, Otago and Canterbury are in New Zealand, all places are in...

10 Example OLAP session Show total sales of Iced Tea. Dice by item Iced Tea. Answer $32M. Seems low. Who s not drinking? Next query asks for total sales of Iced Tea by Country. drill down on Place Data returned in a 1-d table. Australia Canada New Zealand... USA What? Only $1M for New Zealand??

11 OLAP session continues OK, show me Iced Tea sales for New Zealand broken down by month. further refining the dice, drilling down on month Data returned in a 1-d table. Jan Feb Mar Apr May Jun Jul... Dec Weird, sales dropping in June. Maybe it s geographic.

12 OLAP session continues drill down some more Show me Iced Tea sales for New Zealand cities, by month.

13 Big Query Answer Data returned in a 2-d table (in $k), Jan Feb Mar Apr May Jun Jul... Dec Auckland Blenheim Christchurch omitting 30 cities... Wellington Lots of variability, and drowning in data, so roll up on Place

14 OLAP session continues Show me Iced Tea sales for N.Z. provinces, by month Data returned in 9-row 2-d table (in $k), Jan Feb Mar Apr May Jun Jul... Dec Auckland Canterbury Otago Aha: Not many sales in Otago in June and July. Guess it s too cold then.

15 OLAP session concludes Decide to run the 10 unconventional uses for Iced Tea Mix ad in Otago.

16 But what is OLAP exactly? Short answer: a marketing term more catchy than analysis using a multidimensional database. Providing OLAP (On-Line Analytical Processing) to User-Analysts: An IT Mandate Old URL:

17 Some of Codd s definining conditions Multidimensional Conceptual View Generic Dimensionality Unlimited Dimensions and Aggregation Levels

18 Some of Codd s definining conditions Unrestricted Cross-Dimensional Operations Consistent Reporting Performance Dynamic Sparse Matrix Hadling

19 Decision Oriented Concepts Data Mining : pattern discovery rules, models rule-based decision Examples: regressions, neural nets, decision trees, clustering OLAP: analysis descriptive knowledge (human) decision

20 Definitions Variable A unit-bearing data type, either measured or derived. Attribute Information associated with an object. Dimension Collection of objects of the same type. For our purposes, Variable = Attribute.

21 Dimension versus Variable weight height John 160lbs 1.8m Maggy 125lbs 1.4m

22 Definitions To Aggregate The process of combining two or more data items into a single item. Measure A unit-bearing data type (and how to aggregate it). Cell A measure associated with one and only one member from each of multiple dimensions. Hypercube or Data Cube A multi-dimensional schema formed from the cross-product of a number of dimensions.

23 MOLAP, ROLAP, and HOLAP One viewpoint: {M,R,H}OLAP are just implementation approaches to provide multidimensional databases. MOLAP: we know how to store big multi-dim arrays! Do this on disk. ROLAP: we re good at storing relational data, so store the cube as a relation. HOLAP: a little of both... more soon.

24 MOLAP arrays (6-d university example) Session Student Course Section Campus ForCredit Grade Over last 25 years: 100 Session values, 100,000 Students, 1000 Courses, (up to) 10 Sections, 2 Campuses, 2 ForCredit values. Big array: = cells. (Maybe 800GB storage)

25 Sparseness So, array has cells. Yet each student would take at most 40 courses. Hence, at most facts. Density of cube = = 0.001%. Our cube is sparse.

26 MOLAP and Its Advantages It s really quick to retrieve an item from an array, if you know its index. direct mapping If the array is dense (not sparse), computer memory is nicely filled with measure values only (and a few blanks). The dimensional values for a fact are implicit in the address. We know array indices are integers. But dimension values are typically strings, Jack, Jill etc. Thus dimension values must be normalized (mapped to integers). eg. Jack 0, Jill 1, etc.

27 MOLAP Example measure value, for fact "Jill got 87 in CS5678" "students" dimension normalization: Jill 0 Jack 1 Al 2 "courses" dimension CS5678 > 0 EE2222 > 1 CS4002 > 2 empty cell, no fact about Jill and CS a[0][0] a[0][1] a[0][2] a[1][0] a[1][1] a[1][2] a[2][0] a[2][1] a[2][2] "Row Major" PHYSICAL LAYOUT (Disk or RAM)

28 MOLAP and Sparseness Scientific computing has dealt with huge sparse 2-d and 3-d arrays for years. Tricks store data in blocks (no more row-major arrays) use data compression carefully some compression techniques look suspiciously like storing relations.

29 Curse of Dimensionality Observation: The Curse of Dimensionality : techniques that work well for 2-d or 3-d fail miserably on 20-d! Sizes and other costs grow exponentially with the number of dimensions.

30 ROLAP and Its Advantages RelationalOLAP stores facts as tuples in relations. We ve already seen the star schema. Relational technology s very mature. Good for sparse cubes: no fact, no storage! Need fancy indexing for speed

31 ROLAP Operations Operations could be supplied in two ways Augment the standard set of relational operators with some OLAP ones, notably CUBE-BY and ROLLUP in SQL. A ROLAP Middleware will accept OLAP queries and reformulate them for a relational engine.

32 HOLAP ROLAP best on sparse data; MOLAP best on dense data HOLAP: do both (A hybrid. Maybe some materialized view are dense, but the base cube is sparse. store the view with MOLAP; store the base cube with ROLAP. Carry idea further: Observe that parts of a cube/view are dense, even if most of it is sparse. Store/process the dense parts like MOLAP; process the sparse parts like ROLAP.

33 HOLAP example eg 6-d registrar s cube: most of the facts about specific CS student James who started in 1995 will relate to CS courses during the sessions from eg There are no facts about the Intro-to-Internet course from sessions before 1995.

34 Schemas for ROLAP Already we ve seen the star schema where a central fact table has foreign keys into the dimension tables. Another variation, with less redundancy, is the snowflake schema.

35 Redundancy in the Star Schema Let s make the Courses dimension more interesting. Now all courses are contained in some Discipline, and all Disciplines are contained in a Faculty. T T Faculty Science Engg Discipline CSCourses EECourses Course CS4002 CS5678 EE2222

36 Redundancy in the Star Schema, cont. Using the star schema, the Courses table is cid Course Discipline Faculty 0 CS5678 CSCourses Science 1 EE2222 EECourses Engg 2 CS4002 CSCourses Science Note the redundancy: every tuple that contains CSCourses for Discipline will necessarily contain Science for Faculty.

37 Star-Schema Redundancy, 2. This sort of redundancy is a usually a sign of bad relational design. But DW is special: usual rules may not apply. Space waste is not significant: the Fact table dwarfs dimension tables. Update anomalies are not a problem: DW assumes infrequent and off-line updates. But, if you want to avoid the redundancy, go to snowflake schema.

38 Snowflake Schema Use one table for every level of every dimension. Note foreign keys KindID in Students. Students sid Student KindID 0 Jill 0 1 Jack 0 2 Al 1 Kinds KindID Kind 0 grad 1 u/g Course tables are on the next slide... Facts cid sid Grade

39 Snowflake Example: Courses Note foreign keys DiscID from Courses into Disciplines, and fid from Disciplines into Faculties. Courses cid Course DiscID 0 CS EE CS Disciplines DiscID Discipline fid 0 CSCourses 0 1 EECourses 1 Star s Courses table can obtained by Faculties fid Faculty 0 Science 1 Engg π cid,discipline,faculty (Courses Disciplines Faculties)

40 Level Indicator Berson and Smith[?] talk about a Level Indicator, allowing data at different granularity to be stored in one table. Probably a bad idea: invitation to mess up your query! Example: Course C Level Student S Level Grade CS Jill CSCourses 2 grad 2??

41 How might the level indicator arise? Perhaps the DW tries to mix aggregated facts with base-data facts. the DW designer is at fault. Or, perhaps the data is naturally available at different granularity. Eg, before 1979, company reported sales only by the week. After 1979, sales reported daily. Do we really want to lose by day info for 1980-now? Or give up pre-1980 sales data?

42 Avoiding the level indicator Basic idea: sweep all facts with same levels into their own table. We could keep computed aggregates as (maybe materialized) views. Snowflake scheme is handy here.

43 Avoiding, cont. Fact-1-1 cid sid Grade Fact-2-2 DiscID Kind Grade 0 0?? Neverthless, many recommend avoiding the Snowflake.

44 References [BS97] Alex Berson and Stephen J. Smith. Data Warehousing, Data Mining, and OLAP. McGraw-Hill, [C + 01] Surajit Chaudhuri et al. Database technology for decision support systems. IEEE Computer, pages 48 55, December 2001.