April 2016 JPoint Moscow, Russia. How to Apply Big Data Analytics and Machine Learning to Real Time Processing. Kai Wähner.

Transcription

1 April 2016 JPoint Moscow, Russia How to Apply Big Data Analytics and Machine Learning to Real Time Processing Kai LinkedIn / Xing Please connect!

2 Copyright TIBCO Software Inc. Analyse and Act on Critical Business Moments 2

3 Key Take-Aways Insights are hidden in Historical Data on Big Data Platforms Machine Learning and Big Data Analytics find these Insights by building Analytics Models Event Processing uses these Models (without Rebuilding) to take Action in Real Time

4 Copyright TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Analysis of Historical Data 3) Real Time Processing 4) Live Demo 4

5 Copyright TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Analysis of Historical Data 3) Real Time Processing 4) Live Demo 5

6 Copyright TIBCO Software Inc. Machine Learning Machine learning is a method of data analysis that automates analytical model building. Using algorithms that iteratively learn from data, machine learning allows computers to find hidden insights without being explicitly programmed where to look. 6

7 Copyright TIBCO Software Inc. 10 Examples of Machine Learning Spam Detection Credit Card Fraud Detection Digit Recognition Speech Understanding Face Detection Shape Detection Product Recommendation Medical Diagnosis Stock Trading Customer Segmentation 7

8 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 8

9 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Visual Analytics Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 9

10 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 10

11 Copyright TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Analysis of Historical Data 3) Real Time Processing 4) Live Demo 11

12 Copyright TIBCO Software Inc. Analytical Pipeline 12

13 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 13

14 Copyright TIBCO Software Inc. Analytical Pipeline 14

15 Data Acquisition Copyright TIBCO Software Inc.

16 Copyright TIBCO Software Inc. Analytical Pipeline 16

17 Data Munging / Wrangling / Mash-up Copyright TIBCO Software Inc.

18 Copyright TIBCO Software Inc. Data Munging - Transformations cust_id dept sku dollar gift date C FALSE A FALSE C TRUE B FALSE C TRUE B FALSE C FALSE n cust_id A B C total # orders first_date last_date

19 Copyright TIBCO Software Inc. Analytical Pipeline 19

20 Exploratory Data Analysis Copyright TIBCO Software Inc.

21 Copyright TIBCO Software Inc. Exploratory Data Analysis The greatest value of a picture is when it forces us to notice what we never expected to see John W. Tukey, 1977

22 Visual Analytics - Interactive Brush-Linked Copyright TIBCO Software Inc.

23 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Visual Analytics Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 23

24 Copyright TIBCO Software Inc. Analytical Pipeline 24

25 Copyright TIBCO Software Inc. Which picture represents a model? A model is a simplification of the truth that helps you with decision making.

26 Copyright TIBCO Software Inc. Model Building Supervised Models known, labeled responses Regression (for example Linear Regression) Categorical (for example Random Forest) Unsupervised Models no labeled responses Clustering (for example k-means clustering)

27 Model Building Copyright TIBCO Software Inc.

28 Model Building Employees who write longer s earn higher salaries! Copyright TIBCO Software Inc.

29 Model Improvement Copyright TIBCO Software Inc.

30 Model Improvement Managers Staff Copyright TIBCO Software Inc.

31 Copyright TIBCO Software Inc. Analytical Pipeline 31

32 Copyright TIBCO Software Inc. Model Validation How is the IQ of a kid related to the IQ of his / her mum?

33 What tools do Data Scientists use? Copyright TIBCO Software Inc.

34 Copyright TIBCO Software Inc. Alternatives for Data Scientists (no complete list) Tooling Open Source Closed Source R Source Code 34

35 R Language R is well known as the most and increasingly getting more popular programming language used by data scientists for modeling. It is developing very rapidly with a very active community. Copyright TIBCO Software Inc.

36 R with Revolution Analytics (now Microsoft) Open Source GPL License (including its restrictions) Copyright TIBCO Software Inc.

37 TERR - TIBCO s Enterprise Runtime for R TIBCO has rewritten R as a Commercial Compute Engine Latest statistics scripting engine: S a S-PLUS a R a TERR Runs R code including CRAN packages Engine internals rebuilt from scratch at low-level Redesigned data objects, memory management High performance + Big Data TERR is licensed from TIBCO TERR Installs (free) with Spotfire Analyst / Desktop + other TIBCO products Spotfire Server can manage all TERR / R scripts, artifacts for reuse Standalone Developer Edition Supported by TIBCO No GPL license issues Copyright TIBCO Software Inc.

38 Spark MLlib MLlib is Spark s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. It consists of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as lower-level optimization primitives and higher-level pipeline APIs. You can even combine Mllib module with R language Copyright TIBCO Software Inc.

39 Copyright TIBCO Software Inc. H20 An Extensible Open Source Platform for Analytics Best of Breed Open Source Technology Easy-to-use WebUI and Familiar Interfaces Data Agnostic Support for all Common Database and File Types Massively Scalable Big Data Analysis Real-time Data Scoring (Nanofast Scoring Engine)

40 TIBCO Spotfire with R / TERR Integration Let the business user leverage Analytic Models (created by the Data Scientist) to find insights! Example: Customer Churn with Random Forest Algorithm refresh model button lives a random forest algorithm requires no a priori assumptions at all, it just always works The business user doesn t need to know what random forest is to be empowered by it Select variables for the model Copyright TIBCO Software Inc.

41 TIBCO Spotfire with H2O Integration Example: Predictive Analytics for Manufacturing ( scrap parts as early as possible ) Copyright TIBCO Software Inc.

42 TIBCO Spotfire with H2O Integration Example: Predictive Analytics for Manufacturing ( scrap parts as early as possible ) Copyright TIBCO Software Inc.

43 SaaS Machine Learning Managed SaaS service for building ML models and generating predictions Integrated into the corresponding cloud ecosystem Easy to use, but limited feature set and potential latency issues if combined with external data or applications Copyright TIBCO Software Inc.

44 Copyright TIBCO Software Inc. PMML (Predictive Model Markup Language ) XML-based de facto standard to represent predictive analytic models Developed by the Data Mining Group (DMG) Easily share models between PMML compliant applications (e.g. between model creation and deployment for operations)

45 Copyright TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Analysis of Historical Data 3) Real Time Processing 4) Live Demo 45

46 Copyright TIBCO Software Inc. Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases 46

47 Copyright TIBCO Software Inc. Streaming Analytics Event Streams time Continuous Queries Sliding Windows Filter Aggregation Correlation

48 Copyright TIBCO Software Inc. Operational Intelligence in Action Machine-to-Machine Automation Automated action based on models of history combined with live context and business rules Actions by Operations Human decisions in real time informed by up to date information The Challenge: Create, understand, and deploy algorithms & rules that automate key business reactions 48 The Challenge: Empower operations staff to see and seize key business moments

49 Copyright TIBCO Software Inc. Alternatives for Stream Processing (no complete list!) PRODUCT OPEN SOURCE CLOSED SOURCE Azure Microsoft Stream Analytics FRAMEWORK 49

50 Copyright TIBCO Software Inc. Comparison of Stream Processing Frameworks and Products Slide Deck from JavaOne 2016: 50

51 Copyright TIBCO Software Inc. Visual Development of Streaming Analytics Streaming Operators Connectivity Visual Development Testing & Simulation Mature Tooling / Support Middleware Integration

52 Live Datamart Ad-hoc continuous query Alerts Dynamic aggregation Action Live visualization 52

53 How to apply analytic models to real time processing without rebuilding them? Copyright TIBCO Software Inc.

54 Real Time Close Loop: Understand Anticipate Act Streaming Analytics to operationalize insights and patterns in real time without rebuilding the models TERR Spark MLlib MATLAB Open Source R SAS H20 Stream Processing PMML

55 TIBCO StreamBase + R / TERR

56 TIBCO StreamBase + H20

57 TIBCO StreamBase + PMML

58 Closed Loop: Automatically Recompute (and Improve) the Analytic Model Compute your performance metric Spot not good enough performance Recompute model

59 Copyright TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Analysis of Historical Data 3) Real Time Processing 4) Live Demo 59

60 An outage on one well can cost $10M per hour. We have outages per year. - Drilling operations VP, major oil company Copyright TIBCO Software Inc.

61 Predictive Analytics (Fault Management) Temporal analytic: If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert. Voltage Temperature Vibration Device history

62 Complete Big Data Architecture SENSOR DATA TRANSACTIONS Action Streaming Event Server Analytics Aggregate Correlate Analytics Live Monitoring Continuous query processing Alerts Operational Analytics MESSAGE BUS MACHINE DATA Integration Bus Rules Stream Processing Manual action, escalation Live UI Operations SOCIAL DATA Internal Data Cleansed Data History Data Storage Big Data HISTORICAL ANALYSIS Data Discovery Data Sheets BI Machine Learning Data Scientists API Enterprise Service Bus SOA ERP MDM DB WMS

63 Demo Environment CSV Batch JSON Real Time Action Streaming StreamBase Analytics Aggregate Correlate Analytics Live Datamart Continuous query processing Alerts Operational Analytics XML Real Time Rules Manual action, escalation Live UI Operations Internal Data Flume HDFS HDFS HISTORICAL ANALYSIS R / TERR Data Scientists H2O Oracle RDBMS Avro Parquet PMML Hadoop (Cloudera) Spotfire TIBCO Fast Data Platform

64 Live Demo TIBCO Spotfire + StreamBase + TERR + Live Datamart

65 Key Take-Aways Insights are hidden in Historical Data on Big Data Platforms Machine Learning and Big Data Analytics find these Insights by building Analytics Models Event Processing uses these Models (without Rebuilding) to take Action in Real Time

66 Questions? Please contact me! Kai LinkedIn / Xing Please connect!