An Open-Source Streaming Machine Learning and Real-Time Analytics Architecture

Transcription

1 An Open-Source Streaming Machine Learning and Real-Time Analytics Architecture Using an IoT example (incubating) (incubating) Fred 1 William

2 Traditional Data Analytics - Limitations Store Data Lake Analyti cs HDFS No real-time information ETL based Data-source specific Hard to change Labor intensive Inefficient 2

3 Stream-based, Real-Time Closed-Loop Analytics Data Stream Pipeline In-Memory Real- Time Data Data Lake HDFS Expert System / Machine Learning Multiple Data Sources Real-Time Processing Store Everything Continuous Learning Continuous Improvement Continuous Adapting 3

4 A Streaming Machine Learning for IoT Example Predictive Maintenance Scenario Sensor Data Real-Time Evaluates LIVE DATA According to historical trends, there s an 80% chance this equipment would fail in the next 12 hours" Live data becomes historical over time Smart System Historical Learns with HISTORICAL TRENDS "How were the temperature and vibration sensors reading when the latest failures happened? "

5 Streaming Machine Learning Info Machine Learning Analysis Look at past trends (for similar input) Evaluate current input Score / Predict 5

6 Streaming Machine Learning Info Filter Analysis 6 [ json ] Machine Learning

7 Streaming Machine Learning Info Filter Analysis 7 Enrich Machine Learning

8 Streaming Machine Learning Info Filter Analysis 8 Enrich Transform Machine Learning

9 Streaming Machine Learning Info Filter Enrich Transform ML Model Analysis 9

10 Streaming Machine Learning Info Filter Enrich Transform ML Model Analysis Transform 10

11 Streaming Machine Learning ML Model Update In-Memory Data Grid Push Front-end 11

12 Streaming Machine Learning Supervised Learning Example Neural Network Real-time scoring In-Memory Data Grid Train 12

13 A Streaming Machine Learning Reference Architecture Other Sources and Destinations Distributed Computing JMS Fast Data Ingest Transform Sink SpringXD Store / Analyze 13 Predict / Machine Learning

14 Indoors Localization - Applied Example 14

15 Trilateration and its limitations Noisy Data Physical Barriers Large Overlap Areas Moving Targets Innacuracy Large Overlap Areas 15

16 16 Particle Filters - Calculating the optimum solution

17 17 Particle Filters - Calculating the optimum solution

18 The Solution 1. Capture signal strength 2. Calculate distance from antenna 3. Trilaterate different sensors to predict location in real-time 4. Show on a map with live updates 18

19 Architecture Overview JSON HTTP Calculate Device Distance Groovy + Distance Predict Location Spring Boot Ingest Transform Sink SpringXD GUI Application Platform 19

20 Geode Basic Concepts Cache Configurable through XML,,Java Region Distributed j.u.map on steroids Highly available, redundant Member Locator, Server, Client Callbacks Listener, Writer, AsyncEventListener, Parallel/Serial 20

21 Introduction to SpringXD Runs as a distributed application or as a single node 21

22 Spring XD A stream is composed from modules. Each module is deployed to a container and its channels are bound to the transport. 22

23 Demo

24 Why have we selected those projects Iterative & Exploratory model Web based REPL Multiple Interpreters Apache Geode Apache Spark Markdown Flink Python Productivity Built-in connectors Cloud Agnostic Highly Scalable Easy to setup Streams without coding In-memory & Persistent Highly Consistent Extreme transaction processing Thousands of concurrent clients Reliable event model 24

25 Source code and detailed instructions available at: Follow us on GitHub! Fred 25 William 25

26 Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode (incubating), R and Spring XD Room: Tohotom - 14:30, Sep 30 Fred Melo, Pivotal, William Markito, Pivotal Fred 26 William 26