Big Data for Satellite Business Intelligence

Transcription

1 Big Data for Satellite Business Intelligence GSAW 2015 Loic COULET, Kratos ISE 2015 by Kratos ISE. Published by The Aerospace Corporation with permission.

2 Who s talking? Computer Science Passionate Kratos ISE 10 years Learning Big Data Analytics NoSQL Now Learning Loic COULET Software Engineer Systems Intregration M&C CSM Satellite C2 Big Data Business Intelligence Presenting today lcoulet@integ.com Software Java Databases Web 2

3 Kratos Integral Systems Europe (KISE) Subsidiary of Kratos / Kratos ISI, Toulouse, France Toulouse 3

4 KISE provides Ground stations solutions 4

5 Why am I talking today? SBI Press Release caught attention 5

6 Es hailsat The Qatar Satellite Company The Qatar Satellite Company 6

7 Why am I talking today? Kratos Brings Big Data Analysis to Satellite Operations for Es'hailSat New Satellite Business Intelligence (SBI) Dashboard Correlates Data Across Key Ground Segment Solutions to Enhance Service Level Management 7

8 Using smart keywords For which 8

9 1 How Big? 3 What kind? I have minutes to speak about 5 What kind of correlation? 2 What does SBI mean? 4 How is it presented? 6 How does it work? 9

10 How big is Big? 10

11 Big Data = 3V s 3 V s 11

12 Big Data = 3V s STORAGE REQUIREMENTS THROUGHPUT NUMBER OF METRICS 12

13 Data Source Systems Monics Compass Epoch IPS Neuralstar CSM M&C Satellite C2 Network Mgmt 13

14 Number of metrics Monics Compass Epoch Neuralstar CSM M&C Satellite C2 Network Mgmt Low High High High 14

15 Data Troughput Monics Compass Epoch Neuralstar CSM M&C Satellite C2 Network Mgmt High Low High High 15

16 Storage requirements Monics Compass Epoch Neuralstar CSM M&C Satellite C2 Network Mgmt High Medium High High 16

17 Big Data Problem? Systems generate data amount Real-Time processing Store all data for any length of time? Correlation between data sources? Legacy Storage is everything archived? AND THEN Further analysis to detect unknown information? Learning model to anticipate failures? How efficiently is data stored and used? 17

18 How Big? What kind? What kind of correlation? What does SBI mean? How is it presented? How does it work? 18

19 What does Satellite Business Intelligence mean? 19

20 Satellite Business Intelligence =? Business 20

21 + Intelligence Satellite Business Intelligence =? Evaluate Analyze Business Evaluation KPI Implement 21

22 KPI? 22

23 For your satellite services 23

24 How Big? What kind? What kind of correlation? What does SBI mean? How is it presented? How does it work? 24

25 What kind of analysis? 25

26 Correlations for services monitoring dashboard Data Source System Metrics KPI Limit checking Rules SLA Check Rules Data Source System + Business rules Services Report Correlations 26

27 Time Series Prediction analysis 1. Generic predictive analysis (in the query engine) is being implemented Actual Data DLM Prediction 2. Several predictors : linear (exponential Smoothing, holt, least squares), or dymanic with Dynamic Linear Model (DLM) 27

28 How Big? What kind? What kind of correlation? What does SBI mean? How is it presented? How does it work? 28

29 How is it presented? 29

30 How is it presented? As you like! 30

31 A Dashboard Grafana Time Series Dashboard 31

32 THE Dashboard Es hailsat Monitoring Dashboard displays on a Web Browser with real-time information 32

33 Configuration using a CSV file The CMC Monitoring Dashboard configuration file (CSV file) is edited manually by the CMC operator. Configures: KPI Thresholds Monitoring Plans Monitored Services 33

34 CMC Monitoring Dashboard (Views) The CMC Monitoring Dashboard User can see 2 different tables (one with the KPIs and one with the MARGINs) Click on the KPI button to view the latest Keys Performance Indicators System A System B System C 34

35 CMC Monitoring Dashboard (Margin View) Click on the MARGIN button to view the margins 35

36 CMC Monitoring Dashboard (KPI view Plots) Double Click on a KPI to see the real time plot 36

37 CMC Monitoring Dashboard (KPI view Trend) Click on the Trend View Button to view the historical values of the KPIs 37

38 How Big? What kind? What kind of correlation? What does SBI mean? How is it presented? How does it work? 38

39 What kind of correlation What kind of Correlations? 39

40 Correlations for services monitoring dashboard Data Source System Metrics KPI Limit checking Rules SLA Check Rules Data Source System + Business rules Services Report Correlations 40

41 Search correlations Correlate one reference series to many others Interactive histogram represents most correlated series The higher the bar, the most correlated the series (first is the reference to itself) Green to red colors indicate reliability of the correlation score (based on relative number of samples) eirp carrier=carrier_1_ref A Single chart is displayed by default presenting the reference data 41

42 Time Series Correlations Analysis - Search Visualizing Correlation results This shows real data samples Click on a bar to visualize a correlated series on the bottom of the page. eirp carrier=carrier_1_ref The first plot always is the reference data eirp carrier=carrier_3_tst This plot is the second most correlated series eirp carrier=carrier_2 This plot is the fourth most correlated series 42

43 Time Series Correlations Analysis - Discover Same query model than in correlations search Interactive correlation matrix Mouse over a square in the matrix provides information about correlated metrics. Click shows the two plots. Change reliability threshold Do clustering using metric tags 43

44 How Big? What kind? What kind of correlation? What does SBI mean? How is it presented? How does it work? 44

45 How does it work? How does it work? Time Series Database (KairosDB) NoSQL Database as storage backend (Apache Cassandra) Domain expertise and deep integration 45

46 How does it work? Time Series Database (KairosDB) NoSQL Database as storage backend (Apache Cassandra) Domain expertise and integration skills 46

47 KairosDB is modular Real-time KPI & Dashboard Module core Skyminer Analytics Module 47

48 Solution Carrier Monitoring M&C Satellite C2 NMS Monics Compass Epoch IPS Neuralstar Data Collector agent Data Collector agent Data Collector agent Data Collector agent Other Data Sources External Analytics systems Data Integration Frontend Storage Skyminer Web UI Reporting & analytics Frontend 48

49 Primary Goals 1. Time series monitoring & analytics 2. Fast 3. Flexible & Scalable 4. Fault tolerant 5. Incorporates useful analysis features 6. Open to other systems 7. Cost-controlled 49

50 Typical System(s) Fault-Tolerant Small Cluster Using Apache Cassandra DB Low cost Quick start Fault management Data replication Easy administration Scale to any size Best performances 50

51 Interoperability Features 1. All features are provided as web services (HTTP / REST) 2. Open APIs 3. Interoperable data format based on JSON 4. Intuitive Web UI for starting using the system 5. APIs include: Data acquisition Data querying Analysis features (prediction, correlations) 51

52 Query Engine & aggregations Ad-hoc queries and statistics calculation Business Intelligence features already implemented (aggregate, drill & pivot) Data aggregates: Min, Max, Sum, Average, Count, Rate, Std Deviation etc Multi-level Group-by feature using tags, value, or time Filter by tags values 52

53 Reporting Using BIRT reporting tool 53

54 How Big? What kind? What kind of correlation? What does it mean? How is it presented? How does it work? 54

55 Conclusion Systems generate data amount Usual Real-Time Processing Virtually keep all data forever Scale to any size! Big Data Storage AND THEN Correlation between data sources! Automated analysis for new information? Learning model to anticipate failures? 55

56 Thank You!

57 Any Questions?