Big Data Anwendungen in Industrie und Forschung

Transcription

1 Big Data Anwendungen in Industrie und Forschung Dr. Reinhard Stumptner Das SCCH ist eine Initiative der Das SCCH befindet sich im

2 SCCH Key Facts Application-oriented research organization Initiated by institutes of the Johannes Kepler University Linz Cooperation science - industry Non-profit organization constituted as Ltd Owners Johannes Kepler University Linz Upper Austrian Research GmbH Association of Company Partners of SCCH ~ 65 employees (~ 100 incl. partners) 5,8 mio euros income incl. subsidies in business year 2013 Founded in July 1999 in the realm of the K plus Program Since 2008 COMET competence center 2

3 Research Topics Rigorous Methods in Software Engineering software specification, verification, validation formal methods (ASM, Event-B, etc.) process moing, workflows s, Architectures and Tools software architecture mo-based development integration of architecture in development Knowledge-Based Vision Systems machine vision object recognition object tracking Data Analysis Systems automated and intelligent data analysis prediction and optimization knowledge discovery 3

4 Application Domains Data Analysis Systems Topics Topics Computational s Semantic Knowledge s Knowledge Discovery Machine Learning Stream Data Analysis Data Warehousing Data Management 4

5 Overview Temporal Analytics on Big Data Applications Failure detection Proposed Architecture Integrated NoSQL and relational systems for data warehousing tivation Use case INDYCO 5

6 Internet of Things in Industry Coined as Industrial Internet by Evans & Annunziata, 2012 (General Electrics) 6

7 Domain: Industrial Production system 1 system 2 system i system n PIMS Subsystems generate streams of sensor data Stored in Production Information Management System Analysis Tasks Quality Assurance Process Optimization Fault Detection Fault Diagnosis... 7

8 Domain: Machine Manufacturing Data Center Machines at different locations generate streams of sensor data Stored in data center Analysis Tasks Usage nitoring Profile Analysis Condition nitoring Fault Detection Fault Diagnosis... 8

9 Domain: Decentralized Renewable Energy, Home Automation Data Center Sensors of different kind at each building generate streams of sensor data Temperature Solar radiation Energy production... Analysis Tasks Usage nitoring Profile Analysis Condition nitoring Fault Detection Fault Diagnosis 9

10 Fault Detection Algorithms Compare measured channels to a mo Deviation indicate fault and its type A good mo needs to be identified (learned) Typically using historical good data 10

11 A Pattern for Processing and Analysing Industrial Big Data Combining Big Data Storages (BDS) and Stream Processing Engines (SPE) BDS: for offline data processing and analysis SPE: for online, near real-time data processing and analysis unit 1 SPE unit 2 unit i MUX REPLAY unit n BDS MapReduce Implemented with current technology Replay for testing 11

12 Fault Detection Summary Big data in industrial applications Many machines or devices Complex facilities Temporal context (sensor data) Complex feature set Industrial big data processing: Usually achievable with current big data technologies. Industrial big data analysis: Depends strongly on the application. 12

13 Overview Temporal Analytics on Big Data Applications Failure detection Proposed Architecture Integrated NoSQL and relational systems for data warehousing tivation Use case INDYCO 13

14 NoSQL Storages Document Table Big Data Storages Key-Value Graph 14

15 NoSQL Storages Transactions Joins Normalization Query Optimizer + Commodity Hardware + Scalability + Availability + Performance + Flexible Schema Spares tables Semi-structured data Unstructured data 15

16 ngodb Humongous Document-Oriented storege Full Index Support Replication & High Availability Auto-Sharding Querying Fast In-Place Updates MapReduce Agile Scalable 16

17 ngodb Sharding What is sharding? Purpose of sharding Horizontal scaling out Query Routers mongos Shard keys Range based sharding Avoid hotspotting

18 Project INDYCO Integrated Dynamic Decision Support System for Disaster Management Systems EraSME 10th Call for Transnational Cooperative Research Project Proposals (Austria-FFG + Germany-AiF; near-tomarket programme) Project duration: partners from Germany and Austria 3 scientific partners 7 SMEs 1 potential end-user ISSASiM Workshop yearly organized by SCCH since 2011 (DEXA Workshop on Information Systems for Situation Awareness and Situation Management) 18

19 Project INDYCO s1 s2 si sn Sensor DB Sensor data stream Environmental, mobile, social, and other sensors Stored in sensor database (ngodb) Integration of expert knowledge Analysis Tasks Machine Learning (learn mo) Continuous Situation Awareness (apply mo) 19

20 System Overview Online monitoring (interpret sensor data) Long-term learning (mo updates) React on warnings (start dynamic workflows) Requirements to data store Flexibility Performance Connectivity / compatibility 20

21 NoSQL DWH Integration NoSQL Database (ngodb) to store live sensor data Periodic MapReduce job to build aggregates Aggregated sensor data, situations, workflows transferred in DWH (SQL Server Integration Services) Basis for traditional BI (Analysis/Reporting Services) and advanced analytics (R, KNIME, Rapidminer) 21

22 NoSQL DWH Integration Sensor data distributed over n ngodb shards Continuously aggregated ( Aggregate Pool ) and provided to ETL processes for being integrated into a data warehouse (DWH) OLAP processing and further analytics on DWH level Conceptual de-aggregation for Drill- Through operations on original data (via http links) 22

23 23

24 Kontakt Dr. Reinhard Stumptner