MaDaM the web-based Measurement Data Management for Big Data. by Dr. Bernhard Sünder, Managing Director, AMS GmbH

Transcription

1 MaDaM the web-based Measurement Data Management for Big Data by Dr. Bernhard Sünder, Managing Director, AMS GmbH

2 AMS GmbH located in Chemnitz (Saxonia) founded 1993 by Dr. B. Sünder since 1998 our vision is: Using Internet Technologies for distributed Work-Flows in Measurement-Data Postprocessing 17. June 2015 MaDaM Measurement Data Management for Big Data Page 2

3 17. June 2015 MaDaM Measurement Data Management for Big Data Page 3

4 Architecture MDF4 xxx Importer MaDaM - Server Datenbank Web Service by pure HTML-5 MDF4 Desktop ipad Client Optimized traffic with Windows, Linux, Mac, ios, Android 17. June 2015 MaDaM Measurement Data Management for Big Data Page 4

5 2014: introduced MaDaM Server-based Measurement Data Management (MDM-) System with web interface Flexible and fast search system PDF-report creation and interactive analysis Access from desktop and smart devices Using the best technologies for distributed working-flow scenarios 17. June 2015 MaDaM Measurement Data Management for Big Data Page 5

6 MaDaM DIAdem MDF3 MDF4 ATFX Lucene Lucene: Indexing Technology No relational database necessary Just a file system storage NetCDF MATLAB many more MaDaM -Web Server Service MaDaM General and business logic for organizing test data by pure HTML-5 HTML5: User interface, available in nearly any browser PDF: Reports as PDF-files readable in any PDF-reader : Client post processing software for interactive analysis, connected via EnCom Optimized traffic with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 6

7 Importing of Data Dedicated importer components Specialized for certain data formats, e.g. MDF v3 & v4, ISO13499 MME, Diadem, Excel, Running on client side Based on JavaFX With modern user guidance (wizard) Guides the editing process and force consistency 17. June 2015 MaDaM Measurement Data Management for Big Data Page 7

8 Sample-Importer: Passive Safety (Crash) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 8

9 File System DIAdem MDF3 MDF4 ATFX Lucene MaDaM Indexing Technology No relational database necessary NetCDF MATLAB many others - Web Server Service by pure HTML-5 Just a file system storage ipad Client PC Windows, Linux, Mac, ios, Android 17. June 2015 MaDaM Measurement Data Management for Big Data Page 9

10 Apache Lucene Objective: Powerful search functionality with high performance Data-based index Project of the Apache Foundation Used by: Google, Apple, IBM, LinkedIn, Twitter, Eclipse Foundation, OpenSource 17. June 2015 MaDaM Measurement Data Management for Big Data Page 10

11 Relational Database Problem: Model unnecessarily complex Problem: Adaptation to customer needs and new formats difficult Problem: Queries are complex & complicated Problem: Additional license costs for high-performance database solutions (Oracle, Microsoft SQL or others) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 11

12 Search for Tests: Web-Frontend Web-Frontend based on modern web technologies: HTML5 JavaScript AJAX Requires only a browser on client side Device independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 12

13 Desktop Windows Linux Mac OS X Solaris Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 13

14 Desktop Windows Linux Mac OS X Solaris Smart device ios Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 14

15 Desktop Windows Linux Mac OS X Solaris Smart device ios Android Windows 8 Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 15

16 Search for Measurements Full text search Full text search: Required time: 118 ms, show 100 of 1000 result(s) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 16

17 Search for Measurements Combination of search items Full text search: Required time: 21 ms, show 100 of 1000 result(s) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 17

18 Search for Measurements Facet search Show all China NCAP tests which were performed by test lab LaboTron 17. June 2015 MaDaM Measurement Data Management for Big Data Page 18

19 Search for Measurements Any combination of facet and full text search Search for: Date of measurement Test engineer, Laboratory Many more Search in measurements Statistical data of channels are extracted during import 17. June 2015 MaDaM Measurement Data Management for Big Data Page 19

20 Search for Measurements Search for statistical quantities: Min, Max, 17. June 2015 MaDaM Measurement Data Management for Big Data Page 20

21 Search for Measurements High-performance 1000 measurements in < 20ms Scales independent from measurement size Flexible and powerful search syntax Full text search Search in channels Data found, what now? 17. June 2015 MaDaM Measurement Data Management for Big Data Page 21

22 Reports PDF Reports in 3 simple steps 1. Search / Choose 2. Choose template 3. Click button 17. June 2015 MaDaM Measurement Data Management for Big Data Page 22

23 17. June 2015 MaDaM Measurement Data Management for Big Data Page 23

24 Reports (on demand) Reports are generated on demand on the server using as a web service Only a PDF-viewer is required on the client PCs or smart devices MaDaM - Web Server Service File System Lucene by pure HTML June 2015 MaDaM Measurement Data Management for Big Data Page 24

25 Interactive Analysis also works as an app on client PC No pre-installation on the client: Using Java WebStart technology, is deployed over the web on-the-fly PC - Web Server Service Client Lucene MaDaM by pure HTML-5 optimized datatransfer with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 25

26 Acceleration Date: Channels Results Example: Crash Evaluation Based on the MME Standard page 1 of 37 X Y Z Res 3 ms continuous: 58,29 grav Interactive Analysis: NetCDF ASCII ATFX DIAdem Excel MDF Crash Head Accelerations Dummy front left, Hybrid III mid-sized adult male 50% Test Test Object Laboratory No.: 98/7707 Name: Customer No.: M5533 Velocity: Type: Vehicle into Vehicle frontal 0 Mass: 70 grav FIESTA (HTE) 14,5 m/s kg ,00 0,05 0,10 0,15 0,20 s 0,25 Time AXL TETE HVG CFC1000 Calculation range: ms AXT TETE HVG AXV TETE HVG CFC1000 CFC1000 Max.res.ac: 60,01 grav (76,3 ms) HIC: 534,4 (57,6 ms - 96,8 ms) Resulting acceleration CFC1000 HIC 36: 529,0 (60,7 ms - 96,6 ms) HIC 15: 348,8 (70,6 ms - 85,6 ms) 3 ms cumulative: 58,09 grav 17. June 2015 MaDaM Measurement Data Management for Big Data Page 26

27 Lucene DIAdem NetCDF MDF3 MDF4 MATLAB MME many more MaDaM - Web Server Service by pure HTML-5 Optimized traffic using Client Client Client Client ipad PC 17. June 2015 MaDaM Measurement Data Management for Big Data Page 27

28 Enterprise Communication Data Server Client 17. June 2015 MaDaM Measurement Data Management for Big Data Page 28

29 Enterprise Communication Only channels actually used are transmitted Only value ranges actually needed are transmitted What was requested? 2 MB Data Server What do we have? 1 GB Client 17. June 2015 MaDaM Measurement Data Management for Big Data Page 29

30 Enterprise Communication Only required channels are transmitted The channel data transmission is optimized In less than ½ second, the user sees the complete graphic of a channel with more than 10 Mio. values in on client side the user can gradually zoom in even to individual values of the signal In summary: minimizes network traffic by intelligent and selective data transfer. 17. June 2015 MaDaM Measurement Data Management for Big Data Page 30

31 : an open ASAM-CEA Framework Online Measurement & Control FFT Formula User CEA-Calc >100 Calculations Automation System ATFX Test Bench SCADA Operator GUI Data Base Input Component & Flow Management Graphic Layout Report DIAdem MDF UFF NetCDF Offline File Import Unit Services 17. June 2015 MaDaM Measurement Data Management for Big Data Page 31 Data Item EnCom 3rd party MDM other

32 : > 50 Data formats are supported ASAM-ODS: ATF and ATFX Diadem: DAT, TDM & TDMS MDF: version 3, incl. resampling version 4, ASAM compliant NetCDF ASCII: comma, tab, blanks Excel: binary and CVS CAN log Famos EDAS ETAS: DCM LabVIEW ISO13499: MME, ISO DBase 3 Dasylab Google: KML GPS-exchange: GPX Garmin Database AVL: ifile Q-DAS Racelogic-Vbox RPC-III UFF: 15, 58 and 151 TEAC TecPlot 17. June 2015 MaDaM Measurement Data Management for Big Data Page 32

33 : > Curves with >20,000,000 values All > 20,000,000 values in one diagram without delay Tracking window, Values at cursor, reference/delta Axis, others 17. June 2015 MaDaM Measurement Data Management for Big Data Page 33

34 : > Georeferenced Data 17. June 2015 MaDaM Measurement Data Management for Big Data Page 34

35 : Geo & Video & Num. Data Video of Test Graph content: 1. Google Maps as background 2. GPS-Track with speed as color 3. Sprite Car Linegraph: Speed vs. Time 17. June 2015 MaDaM Measurement Data Management for Big Data Page 35

36 : Orderanalysis 17. June 2015 MaDaM Measurement Data Management for Big Data Page 36

37 : Sound FFT, Ampl., Player Sonogram RMS-signal display Audio-player Synchronized 17. June 2015 MaDaM Measurement Data Management for Big Data Page 37

38 : Testdata in 3D-Space 17. June 2015 MaDaM Measurement Data Management for Big Data Page 38

39 Summary Measurement Data Management with MaDaM : Modern web technologies (HTML 5) for simple workflows under PC and smart devices Fast and powerful search function No license costs for third party database products Server based NCAP-report generation using Client based interactive analysis with Optimized data transmission with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 39

40 Gesellschaft für angewandte Mess- und Systemtechnik mbh Bahnhofstraße Opdyke Court German Centre, Unit 719A Chemnitz Auburn Hills, MI Keyuan Road, Pudong Germany USA Shanghai / PR China Tel.: +49 (371) Tel.: +1 (248) Tel.: +86 (21) Fax.: +49 (371) Fax: +1 (248) Fax: +86 (21) info@amsonline.de info@amsonline.eu info@amsonline.cn Web: Web: Web: