The Big Business Intelligence Dilemma

Transcription

1 The Big Business Intelligence Dilemma Copyright R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. Rick F. van der Lans Industry analyst rick@r20.nl Rick F. van der Lans Rick F. van der Lans is an independent consultant, lecturer, and author. He specializes in data warehousing, business intelligence, database technology, and data virtualization. He is managing director of R20/Consultancy B.V.. Rick has been involved in various projects in which data warehousing, and integration technology was applied. Rick van der Lans is an internationally acclaimed lecturer. He has lectured professionally for the last twenty five years in many of the European and Middle East countries, the USA, South America, and in Australia. He has been invited by several major software vendors to present keynote speeches. He is the author of several books on computing, including his new Data Virtualization for Business Intelligence Systems. Some of these books are available in different languages. Books such as the popular Introduction to SQL is available in English, Dutch, Italian, Chinese, and German and is sold world wide. He also authored The SQL Guide to Ingres and SQL for MySQL Developers. As author for TechTarget.com and BeyeNetwork.com, writer of whitepapers, chairman for the annual European Enterprise Data and Business Intelligence Conference, and as columnist for a few IT magazines, he has close contacts with many vendors. R20/Consultancy B.V. is located in The Hague, The Netherlands, You can get in touch with Rick via: rick@r20.nl LinkedIn: Copyright R20/Consultancy B.V., The Hague, The Netherlands 2 1

2 The Big BI Dilemma Copyright R20/Consultancy B.V., The Hague, The Netherlands 3 Part 1: The First Stage of Business Intelligence Copyright R20/Consultancy B.V., The Hague, The Netherlands 4 2

3 Definition of Business Intelligence Definition by Boris Evelson of Forrester Research: Business Intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making. Copyright R20/Consultancy B.V., The Hague, The Netherlands 5 My Interpretation of this Definition All the data All the reporting and analytical environments Business Intelligence All business insights All forms of decision-making Copyright R20/Consultancy B.V., The Hague, The Netherlands 6 3

4 Most Current BI Systems Primarily transactional data IT-driven reporting and analytical environments Business Intelligence Strategic and tactical business insights Strategic and tactical and a little operational decisionmaking Copyright R20/Consultancy B.V., The Hague, The Netherlands 7 The Classic Data Warehouse Architecture Source systems Staging area Data warehouse Data marts Analytics & reporting Copyright R20/Consultancy B.V., The Hague, The Netherlands 8 4

5 Characteristics of Classic Reporting Reporting High data quality Consistent report results Integrated data Report reproducibility Internal production data High data latency High and stable performance Potentially large user group Repetitive usage Minimal analytics Copyright R20/Consultancy B.V., The Hague, The Netherlands 9 Rigid Processes for Development, Operation, and Management Programming logic to verify, transform, cleanse, integrate, interpret, and standardize source data from production systems Backup and recovery mechanisms Security mechanisms and policies to protect against unauthorized access and misuse of the data and reports A priority scheme for developing and maintaining reports Manual procedures initiated by calamities The monitoring of reporting performance, scalability, availability and other nonfunctional aspects of the operational reporting environment Various administrative procedures related to human and computer resources Data governance rules Master data management Copyright R20/Consultancy B.V., The Hague, The Netherlands 10 5

6 Part 2: The Second Stage: Self Service and Big Data Copyright R20/Consultancy B.V., The Hague, The Netherlands 11 Self Service Business Intelligence Self-Service Reporting Self-Service Analytics Self-Service Self-Service Data Preparation Self-Service Copyright R20/Consultancy B.V., The Hague, The Netherlands 12 6

7 Self Service BI: Reporting Chaos? Source systems Staging area Data warehouse Data marts Reporting? Self service BI Copyright R20/Consultancy B.V., The Hague, The Netherlands 13 Analytics Wikipedia: Analytics is the discovery, interpretation, and communication of meaningful patterns in data Analytics is shown what may happen + how what may happen can be influenced Forms: predictive analytics, prescriptive analytics, Techniques: Forecasting: averages, naïve, drift, time series, Application areas: fraud analytics, weather forecasts, customer churning, credit risk analysis, Copyright R20/Consultancy B.V., The Hague, The Netherlands 14 7

8 Result of a Data Science Exercise Result of a data science exercise is business insight in the form of a model (not values) Models Implemented by management in policies and decisions Implemented in reports as KPIs Implemented in operational systems: Risk analysis Customer churn risk Traffic lights Copyright R20/Consultancy B.V., The Hague, The Netherlands 15 The Data Lake or Data Sandbox Source systems Staging area Data warehouse Data marts Reporting? Data science Data lake Copyright R20/Consultancy B.V., The Hague, The Netherlands 16 8

9 Importing External Data Option 1 Social media data Source systems Staging area Data warehouse Data marts Analytics & Data Science Open data Spreadsheets Copyright R20/Consultancy B.V., The Hague, The Netherlands 17 Importing External Data Option 2 Social media data Source systems Staging area Data warehouse Data marts Analytics & Data Science Open data Spreadsheets Copyright R20/Consultancy B.V., The Hague, The Netherlands 18 9

10 Where and How do we Plug In Big Data? Source systems Staging area Data warehouse Data marts Analytics & reporting Big data?? Copyright R20/Consultancy B.V., The Hague, The Netherlands 19 Distributed Big Data Production Copyright R20/Consultancy B.V., The Hague, The Netherlands 20 10

11 Everyone is Doing Business Intelligence Users of self-service BI tools BI specialists Website developers Data scientists Developers of production systems Copyright R20/Consultancy B.V., The Hague, The Netherlands 21 Analytical Islands Everywhere Graph analytics BI in the cloud Self-service BI with cubes Dedicated Analytics for app Excel Standard reports and classic DW Data science Copyright R20/Consultancy B.V., The Hague, The Netherlands 22 11

12 Metadata Specifications Everywhere Self-service BI with cubes Dedicated Analytics for app Excel Standard reports and classic DW Data science Copyright R20/Consultancy B.V., The Hague, The Netherlands 23 The Big BI Dilemma: Integrating Classic BI Forms with New BI Forms Copyright R20/Consultancy B.V., The Hague, The Netherlands 24 12

13 Part 3: The Next Frontier: Fast Data Copyright R20/Consultancy B.V., The Hague, The Netherlands 25 Fast Data = Big Data + Fast Streaming Data + Fast Analytical Decisions Copyright R20/Consultancy B.V., The Hague, The Netherlands 26 13

14 Traffic Lights for Bicyclists in Rotterdam Heat sensor Copyright R20/Consultancy B.V., The Hague, The Netherlands 27 Market Overview Transmitting fast data: Classic message queuing technology, Apache Kafka (developed by LinkedIn) & Flume, Confluent (Kafka), Apache Storm (developed by BackType Twitter), RabbitMQ, Yahoo S4, Storage of fast data: Files, Hadoop HDFS & Hbase, NoSQL, NewSQL, Analyzing fast data: Spark Streaming, SQLStream, StreamBase, Data mining of big data streams MOA (Massive Online Analysis), SOMOA, RapidMiner, Monitoring and managing streaming data Apache NiFi, HortonWorks DataFlow (Apache NiFi++, developed NSA), Copyright R20/Consultancy B.V., The Hague, The Netherlands 28 14

15 Fast Data and the Classic Data Warehouse Architecture Source systems Staging area Data warehouse Data marts Analytics & reporting?? Listener Application Reaction Copyright R20/Consultancy B.V., The Hague, The Netherlands 29 And More Analytical Islands Graph analytics Remote analytics Excel Self-service BI with cubes Dedicated Analytics for app BI in the cloud Fast data analytics Standard reports and classic DW Data science Copyright R20/Consultancy B.V., The Hague, The Netherlands 30 15

16 And More Analytical Islands Remote analytics Self-service BI with cubes Dedicated Analytics for app Excel Fast data analytics Standard reports and classic DW Data science Copyright R20/Consultancy B.V., The Hague, The Netherlands 31 The Even Bigger BI Dilemma: Integrating Classic BI Forms with New BI Forms and with Fast Data Analytics Copyright R20/Consultancy B.V., The Hague, The Netherlands 32 16

17 Part 4: Closing Remarks Copyright R20/Consultancy B.V., The Hague, The Netherlands 33 Every Technology has an Expiration Date Copyright R20/Consultancy B.V., The Hague, The Netherlands 34 17

18 Expired? Source systems Staging area Data warehouse Data marts Analytics & reporting Copyright R20/Consultancy B.V., The Hague, The Netherlands 35 The Logical Data Warehouse Architecture to the Rescue? Copyright R20/Consultancy B.V., The Hague, The Netherlands 36 18

19 The Logical Data Warehouse Architecture Source systems Staging area Data warehouse Analytics & reporting Big data Social media data Open data Spreadsheets Listener Fast Data Logical Data Warehouse Architecture Copyright R20/Consultancy B.V., The Hague, The Netherlands 37 Current Skills of BI Specialist Business Skills Technical Skills Copyright R20/Consultancy B.V., The Hague, The Netherlands 38 19

20 Skills of Fast Data Specialist Business Skills Business Skills Technical Skills Technical Skills Copyright R20/Consultancy B.V., The Hague, The Netherlands 39 Skills of Future BI Specialist Business Skills Business Skills Technical Skills Technical Skills Copyright R20/Consultancy B.V., The Hague, The Netherlands 40 20

21 The Key Challenge is: Solve the Big BI Dilemma! Copyright R20/Consultancy B.V., The Hague, The Netherlands 41 Copyright R20/Consultancy B.V., The Hague, The Netherlands 42 21