INFO 321, Database Systems, Semester

Transcription

1 References References INFO 321 Chapter 3: Decision Support Systems Department of Information Science Semester 2, 2012 General Kifer Chapter 17 Silberschatz (5th ed.) Chapter 18 Data Warehousing for Cavemen (see Blackboard Other Documents Decision Support Systems) Coronel, Morris & Rob (9th ed.) Chapter 13 Mannino (3rd ed.) Chapter 16 Oracle11g documentation Data Warehousing Guide OLAP User s Guide OLAP DML Reference Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO What is decision support? (Kifer 1.4, 17.1; Silberschatz 18.1) Decision-making occurs at the operational level (see also Figure 3 1) Decision-making occurs at the tactical level (see also Figure 3 1) Data timely, relevant, well-visualised information. Tune information and presentation to specific purposes. Very short-term. Well-defined inputs. Produced by existing applications or simple front-end tools. Line managers. Short-term. Less well-defined inputs. Middle managers. Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 61

2 Decision-making occurs at the strategic level (see also Figure 3 1) Long-term. Ill-defined inputs. Often cannot use pre-existing applications Decision Support Systems (DSS); or Executive Information Systems (EIS). Senior managers. Operational vs. decision support queries Operational How many brass reciprocating hammers do we have in stock? How much electrical twine did we sell yesterday? Decision support How many brass reciprocating hammers were sold to customers aged in large North Island towns over each of the last six months? If we double the advertising budget for electrical twine, how might that affect revenues for the next six months? There is a strong need for DSS Modern business very complex. Shrinking time frame for decision-making. Data from multiple sources: (see Figure 3 1) internal vs. external formal vs. informal must be sensibly integrated Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Components of a DSS (adapted from Coronel, Morris & Rob, Figure 13.2) There are many types of decision support tool Operational vs. decision support data (Coronel, Morris & Rob, Table 13.4; see also Mannino ) External data Operational data Data extraction, transformation and loading Decision support data Data store Decision Support System Business data analysis models End-user query tool Sales Expenses Profits End-user presentation and visualisation tool Basic Ad hoc query tools (SQL?). Graph and report generators. Spreadsheets (small data sets only!). More advanced Data warehouses. Online analytical processing (OLAP). Characteristic Operational data Decision support data Data currency Current operations Historic data Real-time data Snapshot of company data Time component (week/month/year) Granularity Atomic, detailed data Summarised data Summarisation level Low; some aggregation High; heavily aggregated Data structure Highly normalised Non-normalised Mostly RDBMS Complex structures Some relational; mostly multidimensional Transaction type Mostly updates Mostly queries Transaction volumes High update volume Periodic loads and summary calculations Transaction speed Updates are critical Retrievals are critical Query activity Low to medium High Query scope Narrow range Broad range Query complexity Simple to medium Very complex Data volumes Hundreds of MiB GiB Hundreds of GiB TiB Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 62

3 Timespan is a key difference Granularity is a key difference Dimensionality is a key difference (Kifer 17.2; Silberschatz ) Operational Very short (current transactions). Decision support Long (past and future). Data may not be current. Operational Represent specific transactions (atomic). Decision support Varying levels of aggregation (atomic highly summarised). Drilling down vs. rolling up. Operational Flat (tables of atomic transactions). ** Decision support Many dimensions. View orders by region per quarter (2D). Compare sales of products during the last six months by region, city, store & customer (4D). Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Dimensionality is a key difference (adapted from Coronel, Morris & Rob, Figure 13.13; see also Kifer 17.2 & Silberschatz ) Data warehouses store decision support data (Mannino ; Coronel, Morris & Rob Table 13.7) Data warehouse data are integrated (Coronel, Morris & Rob Table 13.7) Product Ball Bat Club Location AKL CHC DUD April May June Time Conceptual three-dimensional cube of sales by product, location and time. Sales facts are stored in the cells of the intersection of each product, time and location dimension value. Designed and optimised for decision support data. Internal structure quite different from operational databases: aggregated denormalised data from multiple internal/external sources Operational database data Mostly internal sources. Multiple representations. Data warehouse data Both internal and external sources. Transformed, cleaned and summarised during integration. Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 63

4 Data warehouse data are subject-oriented (Coronel, Morris & Rob Table 13.7) Operational database data Functional or process-oriented (invoices, payments, products). Data warehouse data Facts or measures organised by major subject areas (sales, marketing, etc.). Held according to dimensions or variables of interest: product, customer, region,... Aggregated data from many operational tables. Queries tuned to specific decision-making needs. Data warehouse data are time-variant (Coronel, Morris & Rob Table 13.7) Operational database data Current transactions with precise time stamps. Data warehouse data Time an important dimension for almost all subject areas. Data aggregated by time, e.g., sales by week, month, quarter, year... Historical focus (past and future). Data warehouse data are non-volatile (Coronel, Morris & Rob Table 13.7) Operational database data Frequent changes dynamic. Often archived periodically. Data warehouse data Read only (occasional batch updates) static. Historical data retained always growing (GiB... ). Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Defining a data warehouse in more detail (Silberschatz ; Table 3 2) Data marts are small, specialised data warehouses Data warehouse analysis is more demanding (Mannino ) Read-only database optimised for data analysis and query processing. Data from: legacy /archived databases operational databases other sources Optimisation includes: decisions on aggregations important dimensions appropriate indexing and physical design Focused subset of data. Clusters of data marts surrounding central enterprise data warehouse? Some queries may be impossible if not designed for. Not as flexible for ad hoc queries. Users must identify intended use. Data derived from both internal and external sources (e.g., Internet: NZX, Dow Jones, NASDAQ). Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 64

5 The difficulty of data warehouse design (The Standish Group (1997), The Meta Myth ; Facts are a key design aspect (Kifer pp ; Mannino ) Dimensions are a key design aspect (Kifer pp ; Mannino ) Interviewer: How many data warehouses have you had? Data warehouser: We have had eight. Interviewer: To what do you attribute so many warehouses? Data warehouser: Seven mistakes... A value that we are interested in. Examples: revenue, profits, cost, number of sales. Also known as measures. A factor/variable that influences the facts. Examples: time, product, customer, salesrep, location. Each has attributes. Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Time as a dimension (see also Mannino ) Star schemas for relational data warehouses (Kifer pp ; Silberschatz ; Figure 3 3; see also Data Warehousing Guide ch. 2) Star schemas for relational data warehouses (Kifer pp ; Silberschatz ; Figure 3 3; see also Data Warehousing Guide ch. 2) Dimension table Central fact table. Not as simple as it seems! Cluster of related dimension tables. Granularity (unit size): year, month, week, day, hour. Alternate units (periodicity): season, financial year, quarter. Needed because of inadequate physical data independence? (denormalised) Partial normalisation snowflake or starflake structure (also constellation ). Dimension table Fact table Dimension table Dimension table Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 65

6 Three steps to populate a data warehouse (Kifer 17.6; Mannino 16.4) Performance tuning for data warehouses (Kifer 17.5; Data Warehousing Guide ch. 3; see also INFO 321 Chapter 1) Performance tuning for data warehouses (Kifer 17.5; Data Warehousing Guide ch. 3; see also INFO 321 Chapter 1) Extraction: obtaining data from sources. Transformation: altering form of data (includes cleaning). Loading: adding data to warehouse. Possibly intermediate data staging steps. Critical for successful data warehouses. Complex queries denormalisation (fewer joins). Mostly read-only + complex queries index heavily. Other techniques: normalise dimension tables multiple fact tables for different aggregation levels physical tuning: partitioning, replication, etc. B-tree indexes and hashing generally useful. Bitmap indexes particularly for counting by category queries. Integrated indexes for dimension tables? Function-based indexes could be useful? (Time queries?) Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Oracle11g supports data warehouses (Oracle11g Data Warehousing Guide) The simple approach Use distribution and replication services. Scales poorly. Oracle data mart suite Add-on for constructing Oracle data marts. GUI interface; design & ETL modules; third-party tools. Oracle Data Integrator Enterprise Edition Build & manage high-end, complex data warehouses. Combines Oracle Data Integrator and Oracle Warehouse Builder. Oracle11g supports data warehouses (Oracle11g Data Warehousing Guide) Bitmap & function-based indexes, index-organised tables. Bitmap join indexes. Other relevant tools: SQL*Loader (possibly in conjunction with Transparent Gateways) export and import (basic) Also see Oracle s web site (good luck!). OLAP tools enable complex data processing (Silberschatz 18.2; Figure 3 4) Complex analysis of multidimensional data. Spreadsheet-like simplicity. Data stored in warehouse or tool s internal proprietary database. Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 66

7 OLAP tools have many capabilities A simple OLAP example using Excel (Kifer 17.3; see also Silberschatz ) Slice and dice enables dynamic visualisation (adapted from Coronel, Morris & Rob, Figure 13.14; see also Kifer ) Data transformation. Business modelling. Statistical analysis. Powerful GUI query facility. Visualisation (graphics). Sales subject area dimensions: customer, salesreps, product, region, time,... View sales aggregated by dimensions. Dynamically alter presentation: drill down/roll up slice and dice (see next slide) pivot the table (see demo) highlight exceptions (e.g., high loss products) invent new columns (e.g., % sales revenue) Product Ball Bat Club Location AKL CHC April May June DUD Store manager s view of sales data Product manager s view of sales data Time Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Another example: Quicken Another example: Quicken Another example: Quicken Intuit s Quicken provides some simple OLAP-like features. Drill down to expand a summarised category. Drill down to expenses by month for a particular category. Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 67

8 Another example: Quicken OLAP data may be stored in different ways (Kifer 17.4; Silberschatz ) Oracle11g SQL has extensive OLAP support (Oracle11g SQL Language Reference: SELECT; see also Kifer & Silberschatz ) Drill down to individual category transactions within a month. Internal proprietary database (often MDD). Access external databases (data warehouses?): relational (ROLAP) multidimensional (MOLAP) both (HOLAP) GROUP BY CUBE (<columns>). GROUP BY ROLLUP (<columns>). GROUPING SETS (different from SQL:1999 s GROUPING function). MODEL clause. Various analytic functions, including RANK, PARTITION BY. (see Oracle11g SQL Language Reference: Analytic Functions) Crosstabs using PIVOT and UNPIVOT.... Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Data mining may find hidden trends (Kifer 17.7; Silberschatz 18.4) There are many data mining techniques (Kifer ; Silberschatz 18.4) Some examples of data mining OLAP & data warehousing help identify trends and relationships. BUT: Some relationships too complex or subtle to easily notice. Data mining tools claim to sift through databases and find unrecognised relationships and trends. Neural networks. Complex visualisation. Genetic algorithms (evolve a solution). Advanced statistical analysis (traditional). See INFO 331 for many of these. Beer and nappies (probably apocryphal). Fraud detection (phone, credit card). MCI s statistical profiles. Risk assessment for car insurance (FIG). NBA strategy analysis. But data mining is not foolproof... Semester 2, 2012 INFO Semester 2, 2012 INFO Semester 2, 2012 INFO Chapter 3: Decision Support Systems Lecture Slides (1 51) 68

9 Figure 3 1: Sources of information Table 3 2: Twelve rules that define a data warehouse Bill Inmon is widely referred to as the father of data warehouses. In 1994, he and Chuck Kelley defined a list of twelve rules defining a data warehouse. 1. The data warehouse and operational databases are separated. 2. Data warehouse data are integrated. 3. A data warehouse contains historical data over a long time horizon. 4. Data warehouse data are a snapshot captured at a particular point in time. 5. Data warehouse data are subject-oriented. 6. A data warehouse is mainly read-only with periodic batch updates from operational data. No online updates are allowed. 7. The data warehouse development cycle is data-driven, whereas the classical systems development approach is process-driven A data warehouse contains data at several levels of detail: current detail data, old detail data, lightly summarised and highly summarised data. 9. Database operations in a data warehouse are typically read-only transactions on very large data sets, whereas in an operational database, there are typically many update transactions to a few data entities at a time. 10. A data warehouse has a system that tracks data sources, transformations and storage. SOURCE: Unknown 11. Metadata are critical for a data warehouse, as they identify and define all data elements. Metadata provide the source, transformation, integration, storage, usage, relationships and history of each data element A data warehouse contains a charge-back mechanism for resource usage, in order to enforce optimal use of data by end users. 3 INFO 321 Chapter 3: Decision Support Systems F-1 1 There are many who would argue with the latter claim. 2 This sounds suspiciously similar to number This one seems somewhat out of place, leaving one to wonder whether it was included simply to ensure that there were twelve rules! INFO 321 Chapter 3: Decision Support Systems F-2 Chapter 3: Decision Support Systems Figures & Examples ( ) 69

10 Figure 3 3: Star schemas Figure 3 4: Operational vs. multidimensional data (a) Orders star schema PRODUCT PRODUCT_ID DESCRIPTION PROD_TYPE_ID BRAND COLOUR SIZE PACKAGE 3000 rows VENDOR VENDOR_ID VENDOR_NAME 50 rows (b) Sales star schema LOCATION LOCATION_ID DESCRIPTION REGION_ID STATE CITY 25 rows PERSON PERSON_ID NAME GENDER 125 rows ORDER TIME_ID PRODUCT_ID VENDOR_ID QUANTITY PRICE AMOUNT TIME TIME_ID YEAR QUARTER MONTH WEEK DAY rows 1827 rows (5 years) Daily aggregates by product and vendor SOURCE: adapted from Rob & Coronel, Figure SALE TIME_ID LOCATION_ID PERSON_ID PRODUCT_ID QUANTITY PRICE AMOUNT rows Daily aggregates by store, person and product SOURCE: adapted from Rob & Coronel, Figure TIME TIME_ID YEAR QUARTER MONTH WEEK DAY 1827 rows PRODUCT PRODUCT_ID DESCRIPTION PROD_TYPE_ID BRAND COLOUR SIZE PACKAGE 3000 rows INVOICE_HEADER INV_NUM INV_DATE CUST_NO Customer Dimension Circuit Central Small Bytes Computer House Totals 15-May May May May May (Circuit Central) (Small Bytes) (Circuit Central) (Small Bytes) (Computer House) INVOICE_HEADER INV_NUM PROD_NUM LINE_PRICE LINE_QTY 2034 M34661 $ D99280 $ D44346 $ M34661 $ M34661 $ C74316 $ D44346 $ S64371 $ Time Dimension 15-May May-2007 $1, $ Sales are located at the intersection of a customer row and a time column Operational View of Sales Two-dimensional View of Sales $1, $2, $ $2, $4, (Microsoft Wireless Mouse) (SanDisk USB Flash Drive, 8 GB) (Seagate Hard Drive, 2 TB) (Microsoft Wireless Mouse) (Microsoft Wireless Mouse) (D-Link 4-Port Ethernet Switch) (Seagate Hard Drive, 2 TB) (Creative Speaker System) Totals $2, $3, $ $6, Aggregrations are calculated for both dimensions SOURCE: adapted from Coronel, Morris & Rob, Figure 13.5 $Id: Chapter4figures.tex,v /08/09 05:06:43 nstanger Exp $ INFO 321 Chapter 3: Decision Support Systems F-3 INFO 321 Chapter 3: Decision Support Systems F-4 Chapter 3: Decision Support Systems Figures & Examples ( ) 70