Data Vault + Data Virtualization = Double Flexibility



Similar documents
Data Vault and Data Virtualization: Double Agility

So Many Tools, So Much Data, and So Much Meta Data

Big Data: Big IT Party?

What is Data Virtualization? Rick F. van der Lans, R20/Consultancy

Creating an Agile Data Integration Platform using Data Virtualization

What is Data Virtualization?

Data Virtualization for Agile Business Intelligence Systems and Virtual MDM. To View This Presentation as a Video Click Here

Migrating to Virtual Data Marts using Data Virtualization Simplifying Business Intelligence Systems

Data Services: The Marriage of Data Integration and Application Integration

Key Data Replication Criteria for Enabling Operational Reporting and Analytics

Data Warehouse Optimization

Data Virtualization Usage Patterns for Business Intelligence/ Data Warehouse Architectures

Data Virtualization. Paul Moxon Denodo Technologies. Alberta Data Architecture Community January 22 nd, Denodo Technologies

Cost-Effective Business Intelligence with Red Hat and Open Source

Oracle BI 10g: Analytics Overview

JBoss Data Services. Enabling Data as a Service with. Gnanaguru Sattanathan Twitter:@gnanagurus Website: bushorn.com

Data Virtualization for Business Intelligence Agility

What s New with Informatica Data Services & PowerCenter Data Virtualization Edition

MDM and Data Warehousing Complement Each Other

The New Rules for Integration

GAIN BETTER INSIGHT FROM BIG DATA USING JBOSS DATA VIRTUALIZATION

Lost in Space? Methodology for a Guided Drill-Through Analysis Out of the Wormhole

MicroStrategy Course Catalog

Integrating Netezza into your existing IT landscape

Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco

IST722 Data Warehousing

AV-005: Administering and Implementing a Data Warehouse with SQL Server 2014

DATA VIRTUALIZATION Whitepaper. Data Virtualization. and how to leverage a SOA implementation.

SQL Server 2012 Gives You More Advanced Features (Out-Of-The-Box)

Deploy. Friction-free self-service BI solutions for everyone Scalable analytics on a modern architecture

Relational Databases for the Business Analyst

Klarna Tech Talk: Mind the Data! Jeff Pollock InfoSphere Information Integration & Governance

TRANSFORM BIG DATA INTO ACTIONABLE INFORMATION

Oracle Database 10g: Introduction to SQL

Hadoop Data Hubs and BI. Supporting the migration from siloed reporting and BI to centralized services with Hadoop

IBM AND NEXT GENERATION ARCHITECTURE FOR BIG DATA & ANALYTICS!

Harnessing big data with Hortonworks Data Platform and Red Hat JBoss Data Virtualization

Data Virtualization and ETL. Denodo Technologies Architecture Brief

A Perspective on the Benefits of Data Virtualization Technology

Suresh Chandrasekaran, SVP North America and APAC Pablo Alvarez, Sales Engineer Denodo Technologies

Cloud First Does Not Have to Mean Cloud Exclusively. Digital Government Institute s Cloud Computing & Data Center Conference, September 2014

Data Vault at work. Does Data Vault fulfill its promise? GDF SUEZ Energie Nederland

<Insert Picture Here> Extending Hyperion BI with the Oracle BI Server

MS 20467: Designing Business Intelligence Solutions with Microsoft SQL Server 2012

Discovering Business Insights in Big Data Using SQL-MapReduce

ORACLE BUSINESS INTELLIGENCE, ORACLE DATABASE, AND EXADATA INTEGRATION

Cloud Integration and the Big Data Journey - Common Use-Case Patterns

Optimizing the Performance of the Oracle BI Applications using Oracle Datawarehousing Features and Oracle DAC

Oracle Database 11g: SQL Tuning Workshop Release 2

Building a Data Warehouse

JBoss Enterprise Data Services Platform in the Enterprise

Chapter 6 Basics of Data Integration. Fundamentals of Business Analytics RN Prasad and Seema Acharya

Enterprise Enabler and the Microsoft Integration Stack

Cúram Business Intelligence Reporting Developer Guide

About the Tutorial. Audience. Prerequisites. Disclaimer & Copyright. ETL Testing

Foundations of Business Intelligence: Databases and Information Management

Extraction Transformation Loading ETL Get data out of sources and load into the DW

Oracle Warehouse Builder 10g

Cost Savings THINK ORACLE BI. THINK KPI. THINK ORACLE BI. THINK KPI. THINK ORACLE BI. THINK KPI.

Implementing a Data Warehouse with Microsoft SQL Server MOC 20463

COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER

Luncheon Webinar Series May 13, 2013

Oracle Database 11g: SQL Tuning Workshop

Establish and maintain Center of Excellence (CoE) around Data Architecture

Enterprise Data Warehouse (EDW) UC Berkeley Peter Cava Manager Data Warehouse Services October 5, 2006

ETL-EXTRACT, TRANSFORM & LOAD TESTING

Implementing a Data Warehouse with Microsoft SQL Server

The IBM Cognos Platform

Course Outline: Course: Implementing a Data Warehouse with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning

Building Cubes and Analyzing Data using Oracle OLAP 11g

Oracle Database 12c: Introduction to SQL Ed 1.1

CONCEPTUALIZING BUSINESS INTELLIGENCE ARCHITECTURE MOHAMMAD SHARIAT, Florida A&M University ROSCOE HIGHTOWER, JR., Florida A&M University

MDM for the Enterprise: Complementing and extending your Active Data Warehousing strategy. Satish Krishnaswamy VP MDM Solutions - Teradata

Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777

Migrating a Discoverer System to Oracle Business Intelligence Enterprise Edition

Data Warehouse Overview. Srini Rengarajan

Online Courses. Version 9 Comprehensive Series. What's New Series

SQL Server 2012 Business Intelligence Boot Camp

Data Integration and ETL with Oracle Warehouse Builder NEW

A Whole New World. Big Data Technologies Big Discovery Big Insights Endless Possibilities

SAS BI Course Content; Introduction to DWH / BI Concepts

SQL Server 2008 Performance and Scale

Sterling Business Intelligence

Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Course 20467A; 5 Days

Implementing a Data Warehouse with Microsoft SQL Server

Service Oriented Data Management

ENTERPRISE EDITION ORACLE DATA SHEET KEY FEATURES AND BENEFITS ORACLE DATA INTEGRATOR

LEARNING SOLUTIONS website milner.com/learning phone

Data Warehousing Systems: Foundations and Architectures

Data Virtualization A Potential Antidote for Big Data Growing Pains

A roadmap to enterprise data integration.

Five Steps to Integrate SalesForce.com with 3 rd -Party Systems and Avoid Most Common Mistakes

Practical Considerations for Real-Time Business Intelligence. Donovan Schneider Yahoo! September 11, 2006

<Insert Picture Here> Oracle BI Standard Edition One The Right BI Foundation for the Emerging Enterprise

Transcription:

Vault + Virtualization = Double Flexibility Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. by Rick F. van der Lans R20/Consultancy BV Twitter @rick_vanderlans www.r20.nl Rick F. van der Lans Rick F. van der Lans is an independent consultant, lecturer, and author. He specializes in data warehousing, business intelligence, database technology, and data virtualization. He is managing director of R20/Consultancy B.V.. Rick has been involved in various projects in which data warehousing, and integration technology was applied. Rick van der Lans is an internationally acclaimed lecturer. He has lectured professionally for the last twenty five years in many of the European and Middle East countries, the USA, South America, and in Australia. He has been invited by several major software vendors to present keynote speeches. He is the author of several books on computing, including his new Virtualization for Business Intelligence Systems. Some of these books are available in different languages. Books such as the popular Introduction to SQL is available in English, Dutch, Italian, Chinese, and German and is sold world wide. He also authored The SQL Guide to Ingres and SQL for MySQL Developers. As author for TechTarget.com and BeyeNetwork.com, writer of whitepapers, chairman for the annual European Enterprise and Business Intelligence Conference, and as columnist for a few IT magazines, he has close contacts with many vendors. R20/Consultancy B.V. is located in The Hague, The Netherlands, www.r20.nl. You can get in touch with Rick via: Email: rick@r20.nl Twitter: @Rick_vanderlans LinkedIn: http://www.linkedin.com/pub/rick-van-der-lans/9/207/223 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 2 1

Reporting on a Vault DW?? Reporting and Analytics staging area DV EDW production databases Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 3 Flexibility is Gone! store store store production databases staging area DV EDW store store store store store Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 4 2

Physical Marts Define data structures Define ETL logic Install a database instance Create a database Implement the tables Design physical database structure Initial load of the tables Periodic load of the tables Tune and optimize the database (regularly) Tune and optimize ETL logic Monitor database usage Develop and run backup and recovery processes Unload data Change data structure Change ETL logic Tune and optimize physical database design Tune and optimize ETL logic Reload data Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 5 Remarks on Marts and Cubes Gartner in Management Cost- Cutting Tips, March 10, 2008: Consolidate data marts into an application-neutral data warehouse or smaller data marts to reduce the cost and complexity of the data integration processes feeding the data marts. Gartner predicts this could save you 50 percent of what you're spending to support the siloed data marts. Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 6 3

Flexibility Through Virtualization Virtualization Server staging area DV EDW production databases Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 7 Virtualization Overview (1) production application analytics & reporting internal portal mobile App website dashboard Virtualization Server production databases data warehouse & data marts streaming applications databases unstructured data ESB big data stores social media data private data external data Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 8 4

Virtualization Overview (2) production application analytics & reporting internal portal mobile App website dashboard SQL statement ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX JMS message SQL statement SOAP message Virtualization Server CICS JMS SQL SQL+ XSLT SOAP Hive Prop. Excel JSON production databases data warehouse & data marts streaming applications databases unstructured data ESB big data stores social media data private data external data Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 9 Indonesian Rijsttafel Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 10 5

The Service Hatch Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 11 Virtualization as Service Hatch Kitchen Service hatch Food Restaurant sources virtualization server End Users Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 12 6

The Market of Virtualization Servers Cirro Hub Cisco/Composite Information Server Denodo Platform IBM InfoSphere Federation Server Informatica Services Information Builders EII Oracle Services Integrator Progress Easyl Red Hat Teiid and Jboss Virtualization Stone Bond Enterprise Enabler Virtuoso And many more Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 13 Gartner on Integration Tools Source: Gartner 2014: Modernize Your Integration Capabilities for Diverse Use-Cases, Ted Friedman Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 14 7

Developing Virtual Tables consumer Virtual table: May contain row selections, column selections, column concatenations, transformations, column and table name changes, groupings, aggregations, data cleansing, Source table Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 15 Nesting Virtual Tables Nested virtual table Virtual table Source table Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 16 8

Layers of Virtual Tables Virtualization Server base 1 base 2 base 3 base 4 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 17 Caches Mimimize Access to Stores Virtual table without cache Virtual table with cache Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 18 9

Enable caching Table where cache should should be be stored stored Refresh Refresh specification Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 19 Virtualization and Vault production application analytics & reporting internal portal mobile App website dashboard ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX Virtualization Server SQL Vault - EDW Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 20 10

Solution With Virtualization Users and Reports Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Vault Operational Systems PDB PDB PDB PDB Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 21 virtualization storage The Challenge: The Versions Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 22 11

Example: A Vault Model Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Operational Systems Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 23 Example: The SuperNova Model All the satellite data is added to hubs and links A record in a hub table represents a version of a hub object A record in a link table represents a version of a link object The hub/link id + startdate are the primary keys Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 24 12

Why the Name SuperNova? Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 25 Determining Versions of Hubs Satellite 1 records for hub object 1: HUB_ID META_LOAD_DTS META_LOAD_END_DTS 1 2012-06-01 00:00:00 2013-11-14 23:59:59 1 2013-11-15 00:00:00 2014-03-06 23:59:59 1 2014-03-07 00:00:00 9999-12-31 00:00:00 Satellite 1 records for hub object 1: HUB_ID META_LOAD_DTS META_LOAD_END_DTS 1 2013-06-21 00:00:00 2013-07-20 23:59:59 1 2013-07-21 00:00:00 2013-11-12 23:59:59 1 2013-11-13 00:00:00 9999-12-31 00:00:00 Merged result showing all versions of hub 1: HUB_ID STARTDATE ENDDATE 1 2012-06-01 00:00:00 2013-06-21 23:59:59 1 2013-06-22 00:00:00 2013-07-20 23:59:59 1 2013-07-21 00:00:00 2013-11-12 23:59:59 1 2013-11-13 00:00:00 2013-11-14 23:59:59 1 2013-11-15 00:00:00 2014-03-06 23:59:59 1 2014-03-07 00:00:00 9999-12-31 00:00:00 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 26 13

Visualization of Merge Process 1900 01-01 2012 06-01 2013 06-21 2013 07-21 2013 11-13 2013 11-15 2014 03-07 9999 12-31 Versions of hub 1 from satellite1 table Versions of hub 1 from satellite2 table + Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 27 Step 1 of Determining Hub Versions Merge all the satellites (with a union operator) : (SELECT FROM UNION SELECT FROM HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE HUB1_SATELLITE1 HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS HUB1_SATELLITE2) Intermediate result: SATELLITES HUB_ID STARTDATE ENDDATE 1 2012-06-01 00:00:00 2013-11-14 23:59:59 1 2013-06-21 00:00:00 2013-07-20 23:59:59 1 2013-07-21 00:00:00 2013-11-12 23:59:59 1 2013-11-13 00:00:00 9999-12-31 00:00:00 1 2013-11-15 00:00:00 2014-03-06 23:59:59 1 2014-03-07 00:00:00 9999-12-31 00:00:00 2 2011-03-20 00:00:00 2012-02-25 23:59:59 2 2012-02-26 00:00:00 2014-02-25 23:59:59 2 2012-02-26 00:00:00 9999-12-31 00:00:00 2 2014-02-26 00:00:00 9999-12-31 00:00:00 3 2013-09-09 00:00:00 2013-11-11 00:00:00 3 2013-11-12 00:00:00 2013-11-12 00:00:00 Note that this result does not include hub object 4, because it has no satellite data. Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 28 14

Step 2 of Determining Hub Versions Join with the original Hub table and get the business key(s): SELECT FROM HUB1.HUB_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, HUB1.BUSINESS_KEY HUB1 LEFT OUTER JOIN (SELECT HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM HUB1_SATELLITE1 UNION SELECT HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM HUB1_SATELLITE2) AS SATELLITES ON HUB1.HUB_ID = SATELLITES.HUB_ID) Intermediate result: STARTDATES HUB_ID STARTDATE ENDDATE BUSINESS_KEY 1 2012-06-01 00:00:00 2013-11-14 23:59:59 b1 1 2013-06-21 00:00:00 2013-07-20 23:59:59 b1 1 2013-07-21 00:00:00 2013-11-12 23:59:59 b1 1 2013-11-13 00:00:00 9999-12-31 00:00:00 b1 1 2013-11-15 00:00:00 2014-03-06 23:59:59 b1 1 2014-03-07 00:00:00 9999-12-31 00:00:00 b1 2 2011-03-20 00:00:00 2012-02-25 23:59:59 b2 2 2012-02-26 00:00:00 2014-02-25 23:59:59 b2 2 2012-02-26 00:00:00 9999-12-31 00:00:00 b2 2 2014-02-26 00:00:00 9999-12-31 00:00:00 b2 3 2013-09-09 00:00:00 2013-11-11 00:00:00 b3 Table continues on the next p 3 2013-11-12 00:00:00 2013-11-12 00:00:00 b3 4 NULL NULL b4-1 NULL NULL Unknown -2 NULL NULL N.a. Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 29 Step 3 of Determining Hub Versions Find for each hub the correct versions: HUB1_VERSIONS HUB_ID STARTDATE ENDDATE HUB_BUSINESS_KEY 1 2012-06-01 00:00:00 2013-07-20 23:59:59 b1 1 2013-07-21 00:00:00 2013-11-12 23:59:59 b1 1 2013-11-13 00:00:00 2013-11-14 23:59:59 b1 1 2013-11-15 00:00:00 2014-03-06 23:59:59 b1 1 2014-03-07 00:00:00 9999-12-31 00:00:00 b1 2 2011-03-20 00:00:00 2012-02-25 23:59:59 b2 2 2102-02-26 00:00:00 2014-02-25 23:59:59 b2 2 2014-02-26 00:00:00 9999-12-31 00:00:00 b2 3 2013-09-09 00:00:00 2013-11-11 23:59:59 b3 3 2013-11-12 00:00:00 2013-11-30 00:00:00 b3 4 1900-01-01 00:00:00 9999-12-31 00:00:00 b4-1 1900-01-01 00:00:00 9999-12-31 00:00:00 Unknown -2 1900-01-01 00:00:00 9999-12-31 00:00:00 N.a. Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 30 15

The Three Steps Combined CREATE VIEW HUB1_VERSIONS AS WITH STARTDATES (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS ( SELECT HUB1.HUB_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, HUB1.BUSINESS_KEY FROM HUB1 LEFT OUTER JOIN (SELECT HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM HUB1_SATELLITE1 UNION SELECT HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM HUB1_SATELLITE2) AS SATELLITES ON HUB1.HUB_ID = SATELLITES.HUB_ID) SELECT DISTINCT HUB_ID, STARTDATE, CASE WHEN ENDDATE_NEW <= ENDDATE_OLD THEN ENDDATE_NEW ELSE ENDDATE_OLD END AS ENDDATE, BUSINESS_KEY FROM (SELECT S1.HUB_ID, ISNULL(S1.STARTDATE,'1900-01-01 00:00:00') AS STARTDATE, (SELECT ISNULL(MIN(STARTDATE - '1' SECOND),'9999-12-31 00:00:00') FROM STARTDATES AS S2 WHERE S1.HUB_ID = S2.HUB_ID AND S1.STARTDATE < S2.STARTDATE) AS ENDDATE_NEW, ISNULL(S1.ENDDATE,'9999-12-31 00:00:00') AS ENDDATE_OLD, S1.BUSINESS_KEY FROM STARTDATES AS S1) AS S3 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 31 Hubs with Less Than Two Satellites Hubs with no satellites: CREATE SELECT FROM VIEW HUB3_VERSIONS (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS HUB_ID, ISNULL(META_LOAD_DTS, '1900-01-01 00:00:00'), '9999-12-31 00:00:00', BUSINESS_KEY HUB3 Hubs with one satellite: CREATE SELECT FROM VIEW HUB2_VERSIONS (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS HUB2.HUB_ID, ISNULL(HUB2_SATELLITE1.META_LOAD_DTS, '1900-01-01 00:00:00'), ISNULL(HUB2_SATELLITE1.META_LOAD_END_DTS, '9999-12-31 00:00:00'), HUB2.BUSINESS_KEY HUB2 LEFT OUTER JOIN HUB2_SATELLITE1 ON HUB2.HUB_ID = HUB2_SATELLITE1.HUB_ID Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 32 16

Creating the SuperNova Hub Views A hub is joined with all its satellites using the data in the hub_versions views: CREATE SELECT FROM VIEW SUPERNOVA_HUB1 (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY, ATTRIBUTE1, ATTRIBUTE2) AS HUB1_VERSIONS.HUB_ID, HUB1_VERSIONS.STARTDATE, HUB1_VERSIONS.ENDDATE, HUB1_VERSIONS.BUSINESS_KEY, HUB1_SATELLITE1.ATTRIBUTE, HUB1_SATELLITE2.ATTRIBUTE HUB1_VERSIONS LEFT OUTER JOIN HUB1_SATELLITE1 ON HUB1_VERSIONS.HUB_ID = HUB1_SATELLITE1.HUB_ID AND (HUB1_VERSIONS.STARTDATE <= HUB1_SATELLITE1.META_LOAD_END_DTS AND HUB1_VERSIONS.ENDDATE >= HUB1_SATELLITE1.META_LOAD_DTS) LEFT OUTER JOIN HUB1_SATELLITE2 ON HUB1_VERSIONS.HUB_ID = HUB1_SATELLITE2.HUB_ID AND (HUB1_VERSIONS.STARTDATE <= HUB1_SATELLITE2.META_LOAD_END_DTS AND HUB1_VERSIONS.ENDDATE >= HUB1_SATELLITE2.META_LOAD_DTS) Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 33 Virtual Contents of the SuperNova Hub SUPERNOVA_HUB1 HUB_ID STARTDATE ENDDATE HUB_BUSINESS ATTRIBUTE1 ATTRIBUTE2 _KEY 1 2012-06-01 00:00:00 2013-07-20 23:59:59 b1 a1 a7 1 2013-07-21 00:00:00 2013-11-12 23:59:59 b1 a1 a8 1 2013-11-13 00:00:00 2013-11-14 23:59:59 b1 a1 a9 1 2013-11-15 00:00:00 2014-03-06 23:59:59 b1 a2 a9 1 2014-03-07 00:00:00 9999-12-31 00:00:00 b1 a3 a9 2 2011-03-20 00:00:00 2012-02-25 23:59:59 b2 a4 a10 2 2102-02-26 00:00:00 2014-02-25 23:59:59 b2 a5 a11 2 2014-02-26 00:00:00 9999-12-31 00:00:00 b2 a6 a11 3 2013-09-09 00:00:00 2013-11-11 23:59:59 b3 NULL a12 3 2013-11-12 00:00:00 2013-11-30 00:00:00 b3 NULL a13 4 1900-01-01 00:00:00 9999-12-31 00:00:00 b4 NULL NULL -1 1900-01-01 00:00:00 9999-12-31 00:00:00 Unknown NULL NULL -2 1900-01-01 00:00:00 9999-12-31 00:00:00 N.a. NULL NULL Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 34 17

Creating Version Views for Links CREATE VIEW LINK_VERSIONS AS WITH STARTDATES (LINK_ID, STARTDATE, ENDDATE, HUB1_ID, HUB2_ID, EVENTDATE) AS ( SELECT LINK.LINK_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, LINK.HUB1_ID, LINK.HUB2_ID, LINK.EVENTDATE FROM LINK LEFT OUTER JOIN (SELECT LINK_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM LINK_SATELLITE1 UNION SELECT LINK_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM LINK_SATELLITE2) AS SATELLITES ON LINK.LINK_ID = SATELLITES.LINK_ID) SELECT DISTINCT LINK_ID, STARTDATE, CASE WHEN ENDDATE_NEW <= ENDDATE_OLD THEN ENDDATE_NEW ELSE ENDDATE_OLD END AS ENDDATE, HUB1_ID, HUB2_ID, EVENTDATE FROM (SELECT S1.LINK_ID, ISNULL(S1.STARTDATE, '1900-01-01') AS STARTDATE, (SELECT ISNULL(MIN(STARTDATE - INTERVAL '1' SECOND),'9999-12-31 00:00:00') FROM STARTDATES AS S2 WHERE S1.LINK_ID = S2.LINK_ID AND S1.STARTDATE < S2.STARTDATE) AS ENDDATE_NEW, ISNULL(S1.ENDDATE,'9999-12-31') AS ENDDATE_OLD, S1.HUB1_ID, S1.HUB2_ID, S1.EVENTDATE FROM STARTDATES AS S1) AS S3 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 35 Creating the SuperNova Link Views A link is joined with all its satellites using the data in the link_versions views: CREATE SELECT FROM VIEW SUPERNOVA_LINK (LINK_ID, HUB1_ID, HUB2_ID, STARTDATE, ENDDATE, EVENTDATE, ATTRIBUTE1, ATTRIBUTE2) AS LINK_VERSIONS.LINK_ID, LINK_VERSIONS.HUB1_ID, LINK_VERSIONS.HUB2_ID, LINK_VERSIONS.STARTDATE, LINK_VERSIONS.ENDDATE, LINK_VERSIONS.EVENTDATE, LINK_SATELLITE1.ATTRIBUTE, LINK_SATELLITE2.ATTRIBUTE LINK_VERSIONS LEFT OUTER JOIN LINK_SATELLITE1 ON LINK_VERSIONS.LINK_ID = LINK_SATELLITE1.LINK_ID AND (LINK_VERSIONS.STARTDATE <= LINK_SATELLITE1.META_LOAD_END_DTS AND LINK_VERSIONS.ENDDATE >= LINK_SATELLITE1.META_LOAD_DTS) LEFT OUTER JOIN LINK_SATELLITE2 ON LINK_VERSIONS.LINK_ID = LINK_SATELLITE2.LINK_ID AND (LINK_VERSIONS.STARTDATE <= LINK_SATELLITE2.META_LOAD_END_DTS AND LINK_VERSIONS.ENDDATE >= LINK_SATELLITE2.META_LOAD_DTS) Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 36 18

Virtual Contents of the SuperNova Link LINK_VERSIONS LINK_ID STARTDATE ENDDATE HUB1_ID HUB2_ID EVENTDATE 1 2013-12-01 00:00:00 2013-12-24 23:59:59 1 5 2013-12-01 1 2013-12-25 00:00:00 2014-01-23 23:59:59 1 5 2013-12-01 1 2014-01-24 00:00:00 9999-12-31 00:00:00 1 5 2013-12-01 2 2014-03-12 00:00:00 9999-12-31 00:00:00 1 6 2014-01-01 3 2013-12-27 00:00:00 2014-02-01 23:59:59 2 6 2013-12-25 3 2014-02-02 00:00:00 9999-12-31 00:00:00 2 6 2013-12-25 4 2013-12-08 00:00:00 9999-12-31 00:00:00 3-1 2013-06-24 Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 37 Lineage Analysis of All Views Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 38 19

Defining Primary and Foreign Keys Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 39 Caching of SuperNova Views Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 40 20

The Extended SuperNova Model Delivery Layer Extended Supernova Layer Add derived data Transform data Reuse of definitions Always use the XSN layer Supernova Layer Vault EDW Operational Systems Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 41 The Delivery Model Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Operational Systems is shown in a filtered manner is shown in aggregated form is shown in one large, highly denormalized table is shown in a star schema form is shown with a service interface Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 42 21

Virtual Marts Define data structures Define ETL/DV logic Install a database instance Create a database Implement the tables Design physical database structure Initial load of the tables Periodic load of the tables Tune and optimize the database (regularly) Tune and optimize ETL logic Monitor database usage Develop and run backup and recovery processes Unload data Change data structure Change ETL/DV logic Tune and optimize physical database design Tune and optimize ETL logic Reload data Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 43 Why Not base Views? Not database server independent More advanced distributed join features More advanced heterogeneous join features More advanced caching/refreshing features base views offer no lineage/impact analysis base views offer only one API: SQL No versioning of joins No data cleansing features No business glossary Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 44 22

The Whitepaper Download: www.r20.nl or http://www.cisco.com/web/ser vices/enterprise-itservices/datavirtualization/documents/whit epaper-cisco-datavaul.pdf Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 45 Closing Remarks Vault offers data model extensibility and report reproducibility vault is half the solution SuperNova (with data virtualization) is the other half With data virtualization a more flexible reporting and analytical environment can be developed (quickly) Avoid the (physical) data mart explosion! Go virtual! Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 46 23

Copyright 1991-2015 R20/Consultancy B.V., The Hague, The Netherlands 47 24