Business Intelligence : a primer

Business Intelligence : a primer Rev April 2012 - Gianmario Motta motta05@unipv.it Introduction & overview The paradigm of BI systems Platforms Appendix Review questions

Introduction & overview

Business Intelligence: the role within Enterprise Systems Management support Management Information Systems [Planning & Management Control + Business Intelligence ] Front-end systems (Support the life cycle of customers and end products) Back-end systems (Support the cycle of production and delivery) Administrative systems (Finance, HR etc.) Operations support

Acronyms ABC: Activity Base Costing ABM: Activity Based Management BI: Business Intelligence BW: Business Warehouse (synonym of DW) BSC: Balanced Score Card CPM: Corporate Performance Management (synonym of SEM) CRM: Customer Relationship Management CSF: Critical Success Factor DBMS: Data Base Management System DSS: Decision Support System DW: Data Warehouse EIS: Executive Information System EPM: Enterprise Performance Management (synonym of SEM) ERP: Enterprise Resource Planning ERM: Enterprise Resource Management ES: Enterprise System KPI: Key Performance Indicator MBO: Management By Objectives MRP: Manufacturing Resource Management ODS: Operational Data Store OLAP: On Line Analytical Processing OLTP: On Line Transaction Processing SCM: Supply Chain Management SEM: Strategic Enterprise Management

Characteristics of Analytic & Management Information Information is Periodical Output of computation or aggregations Reflects objectives or actual data E.g. data of P& L of an imaginative Car Company come from different transaction processing systems Sales Purchasing Accounting Etc. Therefore, the design of BI / MIS : Is top-own Defines first target data i.e. the variables that BI should process Identifies corresponding source data Defines the process to extract and transform source in target data

The 4-layer paradigm of BI /MIS systems Decision support engines (DSS) Presentation / reporting engine (EIS, reporting) Mining & other application engines DATA MART DATA WAREHOUSE Loading Tranformation Extraction DATA ENTRY BASI DATI OPERATIVE BASI DATI OPERATIVE BASI DATI OPERATIVE Transactions Data Bases

The 4-layer paradigm of BI /MIS systems BI/MIS applications are based on 4 layers Layer 1 contains source data, typically stored in Transaction Data Base Layer 2 extracts information, and transforms source data into Multi-key & Time-dependent data Layer 3 stores such transformed information Layer 4 processes transformed information according various purposes Support decisions (DSS) E.g. define the sale budget Prepare reports and dashboard (Report) E.g., sales performance Mine stored data (Mining) E.g. identify customer who may churn

The paradigm of BI systems

Jones case study CONTEXT The Supermarket Chain «Jones» includes 300 shops in 3 regions with 60k items on sale A POS (Point Of Sale) system supports all activities of each shop : item receiving, storing, scrapping, selling Specifically, POS terminals record sales transactions and issue receipts REQUIREMENTS Management want to analyze sales Facts : Sales Measures: amount, quantity, number of tickets Analysis dimensions Date Item Shop Time span : 24 months rolling

Level 1 (source data) «Jones» case study Ticket # 2002a23b11 Store #0021MI Item Des Price Qty Amount #190 Pen 3560 2 7.12 #69 Mat 550 10 5.50 #90 Lib 32000 1 32.00 TOTALE 44.62 Payment Fidelity P. Date 120109 Item Master Data # Item # Store Description Price Qunatity mesuere Stock on hand Stock at the beginning of the day Average forecasted dayly sale Receipt Heading # Store # Ticket Amount Payment Date Receipt detail # Ticket # Item Amount Qty

Level 2 DSS LOADING TRANSFORMATION EXTRACTION Report/ dashboard DATA MART DATA WAREHOUSE DATA ENTRY TRANSACTIONS DATABASES Mining & other Extraction includes Select source data Check and clean source data (data cleaning o data cleansing) Staging of extracted data (as needed) Log of extractions Extraction can be Automatic: a batch procedure that runs periodically (e.g. daily, weekly, monthly) Interactive: integrates and fixes automatic data ETL can use intermediate databases Staging Area : where extracted data are temporarily parked (e.g. Data of each individual shop) Operational Data Store (ODS): where granular data are stored and reconciled for future use (e.g. receipt data)

Level 3 DSS LOADING TRANSFORMATION EXTRACTION Report/ dashboard DATA MART DATA WAREHOUSE DATA ENTRY TRANSACTIONS DATABASES Mining & other Data are stored in Data Warehouse and Data Marts A Data Warehouse is a subject-oriented, integrated, time-variant (temporal), non volatile collection of summary and detailed data, used to support strategic decisionmaking process for the enterprise (Inmon 1996) Data Mart is a smaller warehouse, often a subset or extraction of a warehouse. Warehouse e Mart typically adopt different data schemas

Level 3 : Data Warehouse Key table 1 Key 1 Attribute 1 Attribute 2 Attribute. Key table 2 Key 2 Attribute 1 Attribute 2 Attribute. Fact table Key 1 Key 2 Key Measure 1 Measure 2 Measure. Key table Key Attribute 1 Attribute 2 Attribute. The warehouse is typically implemented by relational database, whose schema reflects the corresponding DFM (Dimensional Fact Model). In relational schemas: Fact tables: Store the value of facts (measures) Are identified by multiple keys (K>= 2) Key tables Describe the attributes of dimensions

Level 3: Data Warehouse: star schema Jones case study Shop Shop# Description Shop-class ZIP-code Time Date# Week-day Flag work/holyday for local calendar Date in muslim calendar Flag work/holyday for muslim calendar Sales Date# Item# Shop# Sales amount Sales qty Number of receipts Item Item# Billing-metric Item description Bar-code# Package qty Package-class Supplier-brand Item-class A simple implementation of the DFM is a STAR schema where key tables are implemented only for immediate keys Further analysis / segmentation is obtained by queries on attributes of key tables

Level 3 : Data Warehouse : Snow flake schema Jones Case study A full implementation of the DFM requirements implies a snow flake schema with a key table for every hierarchy node

Level 3: design steps The process from extraction up to data warehouse creation is supported by warehouse building tools that are incorporated in most BI platforms 2 3 4 Target Data design Mapping of Source Data into Target Data ETL code generation 1 Source Data Base Identification 5 Creation of Data Warehouse 6 Data extraction

Level 3: design steps : detail

Level 3: Data Mart DSS Report/ dashboard Mining & other DATA MART DATA WAREHOUSE LOADING TRANSFORMATION DATA ENTRY EXTRACTION TRANSACTIONS DATABASES Data mart store frequently accessed information From a same warehouse multiple data marts can be created Data marts are typically implemented by hypercube (OLAP technology)

Level 3: Data Mart Customer History Sales Analysis Marketing Accounting Shop Data Warehouse From a same warehouse multiple data marts can be created

Level 3: Data Mart : Hyper-cube : display Pages Columns Facts

Time Fact Sales Dimension Level 3: Data Mart : Hyper-cube : logic Event Shop Quantity = 20 Amount= 100 Item An hypercube is a matrix of tables A Fact (e.g. Sales) is identified in a multidimensional space whose axes are Analysis Dimensions (e.g. Shop, Time, Item) An hypercube enables to instantly retrieve complex information e.g. : Sales in last Year (aggregation of Time) by Region (=aggregation of Shops) by Category (= aggregation of Product)

Level 3: Data Mart : Hyper-cube : logic Shops Shop Item Month BUDGET MB21000 MB31000 MB41000 MB21000 MB21000 MB21000 MB21000 MB31000 MB31000 MB31000 MB31000 MB41000 MB41000 MB41000 MB41000 0601 0601 0602 0602 0601 0601 0602 0602 0601 0601 0602 0602 Jan Feb Jan Feb Jan Feb Jan Feb Jan Feb Jan Feb 50 55 50 60 65 45 55 50 60 70 65 75 Item 0601 0602 Date Jan Feb Mar Apr ITEM SHOP OLAP dimensions = warehouse key MONTH

Level 3: Data Mart : Hyper-cube : logic Dimension Product Svelto. Ajax Dash Palmolive Type Washing powder Soap Hierarchy Category House Cleaning Dimensions are arranged in «aggregation hierarchies» (roll-up) Levels of hierarchies are called «dimensional attributes» Dairy Bread & Biscuit Drinks Food All Products A multidimensional analysis is performed by navigating trough aggregation levels of dimensions Tools Nuts & bolts Hardware

Level 3: Data Mart : Hyper-cube : implementation Shop Time Sales-qty Item TIME Tempo (ch) Tempo attributi (da def.) Shop FACT Date Item Shop Sales-amount Sales-qty Receipt-number Time Receipt-number Item ITEM Prodotto (ch) Prodotto attributi (da def.) Shop PuntoVendita (ch) PuntoVendita attributi (da def.) Shop Time Sales-amount Item A wise approach to implement multidimensional information is to have an hyper-cube for each measure This easies arithmetic operations and keeps hyper-cubes light

Level 4 DSS LOADING TRANSFORMATION EXTRACTION Report/ dashboard DATA MART DATA WAREHOUSE DATA ENTRY TRANSACTIONS DATABASES Mining & other It processes information for management from various perspectives Define / assess decisions and program (DSS) Present information with a friendly navigation that enables roll up and drill down (EIS & dashboard) Produce structured reports (reporting) Identify trends an pattern in stored information (mining and profiling)

Leve 4 : reporting Information distribution and privileges handling Format editing Semantic Layer Data Marts Data Bases Data warehouse

Level 4: reporting : semantic layer Purpose: to map data from heterogeneous sources Generally semantic layer includes a set of types e.g.: Dimensions (= warehouse keys) Dimensions attributes ( = key attributes) Measures and Facts

Level 4: reporting : format editing Includes editing functions by which report pages are defined. He content of the report is obtained by dragging an dropping information item from the catalogue of the semantic layer Further activities manage the layout of pages

Level 4: reporting : information distribution

Level 4 : DSS A DSS is a computer based application designed to support semi-structured management decisions by Searching and analyzing information on a collection of sources Compute and assess results (e.g. sensitivity analysis) Typical application fields are: Planning Budgeting Optimization Funding and Investment Decisions ERP / CRM vendors offer DSS suites for corporate planning as Oracle s EPM and SAP s BO

Level 4 : DSS : an example (budgeting) The control system produces monthly a financial report and a report with physical performance indicators (KPI) Financial report and KPI report are on 5 dimensions: 1. Time 2. Cost centers Ricavi a budget Ricavi Processi di calcolo Conto economico 3. Item 4. Sales channel 5. Activity Sales data come from the Sales systems and are stored in a data mart; the same approach is also for sales budget, actual costs and budget costs Data marts are merged in two hyper-cubes, respectively KPI and Financial. Over hyper-cubes a software processes reports on P&L, A&L, Cashflow, KPI Sistema di vendita Ricavi Sistema amministrativo Voci economiche e patrimoniali Spese e costi a budget Costi Processi di calcolo Dati Processi finanziari di calcolo KPI Processi di calcolo Stato patrimoniale Cashflow KPI Memorizzazione e calcolo Elaborazione report

Level 4 : Analysis Engines Data mining applications for research and marketing are designed for Discover in a data base relations and associations previously unknown ( data mining helps end user extract useful business information from large databases (Berson 1997)). mining software are a key in analytic marketing systems to calculate predictive indicators e.g. churning, fraud etc. Customer Profiling systems (Analytic CRM).

Business Intelligence Platforms

BI platforms All main sw vendors offer products, tools &applications for BI SAS: founder of BI and the largest BI independent vendor, offers a wide range of suite by industry, by business area, and specific solutions Oracle : the largest DB vendor offers products on Warehousing and applications from vendors acquired (Essbase, Hyperion) and also an EPM suite similar to SAP s SEM SAP : the largest ERP vendors offers Strategic Enterprise Management (SEM) supports the entire management and analysis life cycle: Products : Crystal report, Business Object (the founder of BI reporting paradigm) Microsoft : Office products, SQL server family OS : e.g. Pentaho Etc.

Business Intelligence Platforms : SAS By industry Education Financial Services Government.. By solution Analytics Business Analytics Business Intelligence Customer Intelligence Data Management Fraud & Financial Crimes High-Performance Analytics IT & CIO Enablement On Demand Solutions Performance Management Risk Management SAS 9.3 Supply Chain Intelligence Sustainability Management Featured solutions SAS 9.3 SAS Clinical Data Integration SAS Curriculum Pathways SAS Enterprise Guide SAS Enterprise Miner SAS Fraud Framework for Government SAS High-Performance Analytics SAS Inventory Optimization SAS OnDemand for Academics SAS Social Media Analytics SAS Text Analytics SAS Visual Data Discovery

Appendix: Data Warehouse, Data Mart and Database profiles

Data Warehouse and Data Mart vs Database Conceptual modeling (Rich Semantic Layer) Information type (Master, Event, Analysis) Information organization Data base Data Warehouse Data Mart ERA DFM DFM Master + Event Analysis Analysis Normalized (e.g. 3NF) Star or snowflake Hypercube Data schema Relational Relational OLAP or Relational Processing orientation Create + Update Read Read Typical data operations Insert one individual record or modify one or multiple records Access a vector of records Roll-up, Drill down, Dice Transaction example Enter a customer order Segment customer in Italy with a degree of loyalty >70% by age and region Access one ore multiple a vector of records Roll-up, Drill down, Dice Segment customer in Italy with a degree of loyalty >70% by age and region

Review questions

Review questions Illustrate the input, process and output of the four layers of BI systems What is an Hypercube? What is a data mart? What is a data warehouse? Compare data warehouse versus classic database in terms of Conceptual modeling (Rich Semantic Layer) Implementation (DB schema) Information type (Master, Event, Analysis) Processing orientation