Data Vault + Data Virtualization = Double Flexibility

Transcription

1 Vault + Virtualization = Double Flexibility Copyright R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. by Rick F. van der Lans R20/Consultancy BV Rick F. van der Lans Rick F. van der Lans is an independent consultant, lecturer, and author. He specializes in data warehousing, business intelligence, database technology, and data virtualization. He is managing director of R20/Consultancy B.V.. Rick has been involved in various projects in which data warehousing, and integration technology was applied. Rick van der Lans is an internationally acclaimed lecturer. He has lectured professionally for the last twenty five years in many of the European and Middle East countries, the USA, South America, and in Australia. He has been invited by several major software vendors to present keynote speeches. He is the author of several books on computing, including his new Virtualization for Business Intelligence Systems. Some of these books are available in different languages. Books such as the popular Introduction to SQL is available in English, Dutch, Italian, Chinese, and German and is sold world wide. He also authored The SQL Guide to Ingres and SQL for MySQL Developers. As author for TechTarget.com and BeyeNetwork.com, writer of whitepapers, chairman for the annual European Enterprise and Business Intelligence Conference, and as columnist for a few IT magazines, he has close contacts with many vendors. R20/Consultancy B.V. is located in The Hague, The Netherlands, You can get in touch with Rick via: [email protected] LinkedIn: Copyright R20/Consultancy B.V., The Hague, The Netherlands 2 1

2 Reporting on a Vault DW?? Reporting and Analytics staging area DV EDW production databases Copyright R20/Consultancy B.V., The Hague, The Netherlands 3 Flexibility is Gone! store store store production databases staging area DV EDW store store store store store Copyright R20/Consultancy B.V., The Hague, The Netherlands 4 2

3 Physical Marts Define data structures Define ETL logic Install a database instance Create a database Implement the tables Design physical database structure Initial load of the tables Periodic load of the tables Tune and optimize the database (regularly) Tune and optimize ETL logic Monitor database usage Develop and run backup and recovery processes Unload data Change data structure Change ETL logic Tune and optimize physical database design Tune and optimize ETL logic Reload data Copyright R20/Consultancy B.V., The Hague, The Netherlands 5 Remarks on Marts and Cubes Gartner in Management Cost- Cutting Tips, March 10, 2008: Consolidate data marts into an application-neutral data warehouse or smaller data marts to reduce the cost and complexity of the data integration processes feeding the data marts. Gartner predicts this could save you 50 percent of what you're spending to support the siloed data marts. Copyright R20/Consultancy B.V., The Hague, The Netherlands 6 3

4 Flexibility Through Virtualization Virtualization Server staging area DV EDW production databases Copyright R20/Consultancy B.V., The Hague, The Netherlands 7 Virtualization Overview (1) production application analytics & reporting internal portal mobile App website dashboard Virtualization Server production databases data warehouse & data marts streaming applications databases unstructured data ESB big data stores social media data private data external data Copyright R20/Consultancy B.V., The Hague, The Netherlands 8 4

5 Virtualization Overview (2) production application analytics & reporting internal portal mobile App website dashboard SQL statement ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX JMS message SQL statement SOAP message Virtualization Server CICS JMS SQL SQL+ XSLT SOAP Hive Prop. Excel JSON production databases data warehouse & data marts streaming applications databases unstructured data ESB big data stores social media data private data external data Copyright R20/Consultancy B.V., The Hague, The Netherlands 9 Indonesian Rijsttafel Copyright R20/Consultancy B.V., The Hague, The Netherlands 10 5

6 The Service Hatch Copyright R20/Consultancy B.V., The Hague, The Netherlands 11 Virtualization as Service Hatch Kitchen Service hatch Food Restaurant sources virtualization server End Users Copyright R20/Consultancy B.V., The Hague, The Netherlands 12 6

7 The Market of Virtualization Servers Cirro Hub Cisco/Composite Information Server Denodo Platform IBM InfoSphere Federation Server Informatica Services Information Builders EII Oracle Services Integrator Progress Easyl Red Hat Teiid and Jboss Virtualization Stone Bond Enterprise Enabler Virtuoso And many more Copyright R20/Consultancy B.V., The Hague, The Netherlands 13 Gartner on Integration Tools Source: Gartner 2014: Modernize Your Integration Capabilities for Diverse Use-Cases, Ted Friedman Copyright R20/Consultancy B.V., The Hague, The Netherlands 14 7

8 Developing Virtual Tables consumer Virtual table: May contain row selections, column selections, column concatenations, transformations, column and table name changes, groupings, aggregations, data cleansing, Source table Copyright R20/Consultancy B.V., The Hague, The Netherlands 15 Nesting Virtual Tables Nested virtual table Virtual table Source table Copyright R20/Consultancy B.V., The Hague, The Netherlands 16 8

9 Layers of Virtual Tables Virtualization Server base 1 base 2 base 3 base 4 Copyright R20/Consultancy B.V., The Hague, The Netherlands 17 Caches Mimimize Access to Stores Virtual table without cache Virtual table with cache Copyright R20/Consultancy B.V., The Hague, The Netherlands 18 9

10 Enable caching Table where cache should should be be stored stored Refresh Refresh specification Copyright R20/Consultancy B.V., The Hague, The Netherlands 19 Virtualization and Vault production application analytics & reporting internal portal mobile App website dashboard ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX Virtualization Server SQL Vault - EDW Copyright R20/Consultancy B.V., The Hague, The Netherlands 20 10

11 Solution With Virtualization Users and Reports Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Vault Operational Systems PDB PDB PDB PDB Copyright R20/Consultancy B.V., The Hague, The Netherlands 21 virtualization storage The Challenge: The Versions Copyright R20/Consultancy B.V., The Hague, The Netherlands 22 11

12 Example: A Vault Model Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Operational Systems Copyright R20/Consultancy B.V., The Hague, The Netherlands 23 Example: The SuperNova Model All the satellite data is added to hubs and links A record in a hub table represents a version of a hub object A record in a link table represents a version of a link object The hub/link id + startdate are the primary keys Copyright R20/Consultancy B.V., The Hague, The Netherlands 24 12

13 Why the Name SuperNova? Copyright R20/Consultancy B.V., The Hague, The Netherlands 25 Determining Versions of Hubs Satellite 1 records for hub object 1: HUB_ID META_LOAD_DTS META_LOAD_END_DTS :00: :59: :00: :59: :00: :00:00 Satellite 1 records for hub object 1: HUB_ID META_LOAD_DTS META_LOAD_END_DTS :00: :59: :00: :59: :00: :00:00 Merged result showing all versions of hub 1: HUB_ID STARTDATE ENDDATE :00: :59: :00: :59: :00: :59: :00: :59: :00: :59: :00: :00:00 Copyright R20/Consultancy B.V., The Hague, The Netherlands 26 13

14 Visualization of Merge Process Versions of hub 1 from satellite1 table Versions of hub 1 from satellite2 table + Copyright R20/Consultancy B.V., The Hague, The Netherlands 27 Step 1 of Determining Hub Versions Merge all the satellites (with a union operator) : (SELECT FROM UNION SELECT FROM HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE HUB1_SATELLITE1 HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS HUB1_SATELLITE2) Intermediate result: SATELLITES HUB_ID STARTDATE ENDDATE :00: :59: :00: :59: :00: :59: :00: :00: :00: :59: :00: :00: :00: :59: :00: :59: :00: :00: :00: :00: :00: :00: :00: :00:00 Note that this result does not include hub object 4, because it has no satellite data. Copyright R20/Consultancy B.V., The Hague, The Netherlands 28 14

15 Step 2 of Determining Hub Versions Join with the original Hub table and get the business key(s): SELECT FROM HUB1.HUB_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, HUB1.BUSINESS_KEY HUB1 LEFT OUTER JOIN (SELECT HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM HUB1_SATELLITE1 UNION SELECT HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM HUB1_SATELLITE2) AS SATELLITES ON HUB1.HUB_ID = SATELLITES.HUB_ID) Intermediate result: STARTDATES HUB_ID STARTDATE ENDDATE BUSINESS_KEY :00: :59:59 b :00: :59:59 b :00: :59:59 b :00: :00:00 b :00: :59:59 b :00: :00:00 b :00: :59:59 b :00: :59:59 b :00: :00:00 b :00: :00:00 b :00: :00:00 b3 Table continues on the next p :00: :00:00 b3 4 NULL NULL b4-1 NULL NULL Unknown -2 NULL NULL N.a. Copyright R20/Consultancy B.V., The Hague, The Netherlands 29 Step 3 of Determining Hub Versions Find for each hub the correct versions: HUB1_VERSIONS HUB_ID STARTDATE ENDDATE HUB_BUSINESS_KEY :00: :59:59 b :00: :59:59 b :00: :59:59 b :00: :59:59 b :00: :00:00 b :00: :59:59 b :00: :59:59 b :00: :00:00 b :00: :59:59 b :00: :00:00 b :00: :00:00 b :00: :00:00 Unknown :00: :00:00 N.a. Copyright R20/Consultancy B.V., The Hague, The Netherlands 30 15

16 The Three Steps Combined CREATE VIEW HUB1_VERSIONS AS WITH STARTDATES (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS ( SELECT HUB1.HUB_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, HUB1.BUSINESS_KEY FROM HUB1 LEFT OUTER JOIN (SELECT HUB_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM HUB1_SATELLITE1 UNION SELECT HUB_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM HUB1_SATELLITE2) AS SATELLITES ON HUB1.HUB_ID = SATELLITES.HUB_ID) SELECT DISTINCT HUB_ID, STARTDATE, CASE WHEN ENDDATE_NEW <= ENDDATE_OLD THEN ENDDATE_NEW ELSE ENDDATE_OLD END AS ENDDATE, BUSINESS_KEY FROM (SELECT S1.HUB_ID, ISNULL(S1.STARTDATE,' :00:00') AS STARTDATE, (SELECT ISNULL(MIN(STARTDATE - '1' SECOND),' :00:00') FROM STARTDATES AS S2 WHERE S1.HUB_ID = S2.HUB_ID AND S1.STARTDATE < S2.STARTDATE) AS ENDDATE_NEW, ISNULL(S1.ENDDATE,' :00:00') AS ENDDATE_OLD, S1.BUSINESS_KEY FROM STARTDATES AS S1) AS S3 Copyright R20/Consultancy B.V., The Hague, The Netherlands 31 Hubs with Less Than Two Satellites Hubs with no satellites: CREATE SELECT FROM VIEW HUB3_VERSIONS (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS HUB_ID, ISNULL(META_LOAD_DTS, ' :00:00'), ' :00:00', BUSINESS_KEY HUB3 Hubs with one satellite: CREATE SELECT FROM VIEW HUB2_VERSIONS (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY) AS HUB2.HUB_ID, ISNULL(HUB2_SATELLITE1.META_LOAD_DTS, ' :00:00'), ISNULL(HUB2_SATELLITE1.META_LOAD_END_DTS, ' :00:00'), HUB2.BUSINESS_KEY HUB2 LEFT OUTER JOIN HUB2_SATELLITE1 ON HUB2.HUB_ID = HUB2_SATELLITE1.HUB_ID Copyright R20/Consultancy B.V., The Hague, The Netherlands 32 16

17 Creating the SuperNova Hub Views A hub is joined with all its satellites using the data in the hub_versions views: CREATE SELECT FROM VIEW SUPERNOVA_HUB1 (HUB_ID, STARTDATE, ENDDATE, BUSINESS_KEY, ATTRIBUTE1, ATTRIBUTE2) AS HUB1_VERSIONS.HUB_ID, HUB1_VERSIONS.STARTDATE, HUB1_VERSIONS.ENDDATE, HUB1_VERSIONS.BUSINESS_KEY, HUB1_SATELLITE1.ATTRIBUTE, HUB1_SATELLITE2.ATTRIBUTE HUB1_VERSIONS LEFT OUTER JOIN HUB1_SATELLITE1 ON HUB1_VERSIONS.HUB_ID = HUB1_SATELLITE1.HUB_ID AND (HUB1_VERSIONS.STARTDATE <= HUB1_SATELLITE1.META_LOAD_END_DTS AND HUB1_VERSIONS.ENDDATE >= HUB1_SATELLITE1.META_LOAD_DTS) LEFT OUTER JOIN HUB1_SATELLITE2 ON HUB1_VERSIONS.HUB_ID = HUB1_SATELLITE2.HUB_ID AND (HUB1_VERSIONS.STARTDATE <= HUB1_SATELLITE2.META_LOAD_END_DTS AND HUB1_VERSIONS.ENDDATE >= HUB1_SATELLITE2.META_LOAD_DTS) Copyright R20/Consultancy B.V., The Hague, The Netherlands 33 Virtual Contents of the SuperNova Hub SUPERNOVA_HUB1 HUB_ID STARTDATE ENDDATE HUB_BUSINESS ATTRIBUTE1 ATTRIBUTE2 _KEY :00: :59:59 b1 a1 a :00: :59:59 b1 a1 a :00: :59:59 b1 a1 a :00: :59:59 b1 a2 a :00: :00:00 b1 a3 a :00: :59:59 b2 a4 a :00: :59:59 b2 a5 a :00: :00:00 b2 a6 a :00: :59:59 b3 NULL a :00: :00:00 b3 NULL a :00: :00:00 b4 NULL NULL :00: :00:00 Unknown NULL NULL :00: :00:00 N.a. NULL NULL Copyright R20/Consultancy B.V., The Hague, The Netherlands 34 17

18 Creating Version Views for Links CREATE VIEW LINK_VERSIONS AS WITH STARTDATES (LINK_ID, STARTDATE, ENDDATE, HUB1_ID, HUB2_ID, EVENTDATE) AS ( SELECT LINK.LINK_ID, SATELLITES.STARTDATE, SATELLITES.ENDDATE, LINK.HUB1_ID, LINK.HUB2_ID, LINK.EVENTDATE FROM LINK LEFT OUTER JOIN (SELECT LINK_ID, META_LOAD_DTS AS STARTDATE, META_LOAD_END_DTS AS ENDDATE FROM LINK_SATELLITE1 UNION SELECT LINK_ID, META_LOAD_DTS, META_LOAD_END_DTS FROM LINK_SATELLITE2) AS SATELLITES ON LINK.LINK_ID = SATELLITES.LINK_ID) SELECT DISTINCT LINK_ID, STARTDATE, CASE WHEN ENDDATE_NEW <= ENDDATE_OLD THEN ENDDATE_NEW ELSE ENDDATE_OLD END AS ENDDATE, HUB1_ID, HUB2_ID, EVENTDATE FROM (SELECT S1.LINK_ID, ISNULL(S1.STARTDATE, ' ') AS STARTDATE, (SELECT ISNULL(MIN(STARTDATE - INTERVAL '1' SECOND),' :00:00') FROM STARTDATES AS S2 WHERE S1.LINK_ID = S2.LINK_ID AND S1.STARTDATE < S2.STARTDATE) AS ENDDATE_NEW, ISNULL(S1.ENDDATE,' ') AS ENDDATE_OLD, S1.HUB1_ID, S1.HUB2_ID, S1.EVENTDATE FROM STARTDATES AS S1) AS S3 Copyright R20/Consultancy B.V., The Hague, The Netherlands 35 Creating the SuperNova Link Views A link is joined with all its satellites using the data in the link_versions views: CREATE SELECT FROM VIEW SUPERNOVA_LINK (LINK_ID, HUB1_ID, HUB2_ID, STARTDATE, ENDDATE, EVENTDATE, ATTRIBUTE1, ATTRIBUTE2) AS LINK_VERSIONS.LINK_ID, LINK_VERSIONS.HUB1_ID, LINK_VERSIONS.HUB2_ID, LINK_VERSIONS.STARTDATE, LINK_VERSIONS.ENDDATE, LINK_VERSIONS.EVENTDATE, LINK_SATELLITE1.ATTRIBUTE, LINK_SATELLITE2.ATTRIBUTE LINK_VERSIONS LEFT OUTER JOIN LINK_SATELLITE1 ON LINK_VERSIONS.LINK_ID = LINK_SATELLITE1.LINK_ID AND (LINK_VERSIONS.STARTDATE <= LINK_SATELLITE1.META_LOAD_END_DTS AND LINK_VERSIONS.ENDDATE >= LINK_SATELLITE1.META_LOAD_DTS) LEFT OUTER JOIN LINK_SATELLITE2 ON LINK_VERSIONS.LINK_ID = LINK_SATELLITE2.LINK_ID AND (LINK_VERSIONS.STARTDATE <= LINK_SATELLITE2.META_LOAD_END_DTS AND LINK_VERSIONS.ENDDATE >= LINK_SATELLITE2.META_LOAD_DTS) Copyright R20/Consultancy B.V., The Hague, The Netherlands 36 18

19 Virtual Contents of the SuperNova Link LINK_VERSIONS LINK_ID STARTDATE ENDDATE HUB1_ID HUB2_ID EVENTDATE :00: :59: :00: :59: :00: :00: :00: :00: :00: :59: :00: :00: :00: :00: Copyright R20/Consultancy B.V., The Hague, The Netherlands 37 Lineage Analysis of All Views Copyright R20/Consultancy B.V., The Hague, The Netherlands 38 19

20 Defining Primary and Foreign Keys Copyright R20/Consultancy B.V., The Hague, The Netherlands 39 Caching of SuperNova Views Copyright R20/Consultancy B.V., The Hague, The Netherlands 40 20

21 The Extended SuperNova Model Delivery Layer Extended Supernova Layer Add derived data Transform data Reuse of definitions Always use the XSN layer Supernova Layer Vault EDW Operational Systems Copyright R20/Consultancy B.V., The Hague, The Netherlands 41 The Delivery Model Delivery Layer Extended Supernova Layer Supernova Layer Vault EDW Operational Systems is shown in a filtered manner is shown in aggregated form is shown in one large, highly denormalized table is shown in a star schema form is shown with a service interface Copyright R20/Consultancy B.V., The Hague, The Netherlands 42 21

22 Virtual Marts Define data structures Define ETL/DV logic Install a database instance Create a database Implement the tables Design physical database structure Initial load of the tables Periodic load of the tables Tune and optimize the database (regularly) Tune and optimize ETL logic Monitor database usage Develop and run backup and recovery processes Unload data Change data structure Change ETL/DV logic Tune and optimize physical database design Tune and optimize ETL logic Reload data Copyright R20/Consultancy B.V., The Hague, The Netherlands 43 Why Not base Views? Not database server independent More advanced distributed join features More advanced heterogeneous join features More advanced caching/refreshing features base views offer no lineage/impact analysis base views offer only one API: SQL No versioning of joins No data cleansing features No business glossary Copyright R20/Consultancy B.V., The Hague, The Netherlands 44 22

23 The Whitepaper Download: or vices/enterprise-itservices/datavirtualization/documents/whit epaper-cisco-datavaul.pdf Copyright R20/Consultancy B.V., The Hague, The Netherlands 45 Closing Remarks Vault offers data model extensibility and report reproducibility vault is half the solution SuperNova (with data virtualization) is the other half With data virtualization a more flexible reporting and analytical environment can be developed (quickly) Avoid the (physical) data mart explosion! Go virtual! Copyright R20/Consultancy B.V., The Hague, The Netherlands 46 23

24 Copyright R20/Consultancy B.V., The Hague, The Netherlands 47 24