MaDaM the web-based Measurement Data Management for Big Data by Dr. Bernhard Sünder, Managing Director, AMS GmbH
AMS GmbH located in Chemnitz (Saxonia) founded 1993 by Dr. B. Sünder since 1998 our vision is: Using Internet Technologies for distributed Work-Flows in Measurement-Data Postprocessing 17. June 2015 MaDaM Measurement Data Management for Big Data Page 2
17. June 2015 MaDaM Measurement Data Management for Big Data Page 3
Architecture MDF4 xxx Importer MaDaM - Server Datenbank Web Service by pure HTML-5 MDF4 Desktop ipad Client Optimized traffic with Windows, Linux, Mac, ios, Android 17. June 2015 MaDaM Measurement Data Management for Big Data Page 4
2014: introduced MaDaM Server-based Measurement Data Management (MDM-) System with web interface Flexible and fast search system PDF-report creation and interactive analysis Access from desktop and smart devices Using the best technologies for distributed working-flow scenarios 17. June 2015 MaDaM Measurement Data Management for Big Data Page 5
MaDaM DIAdem MDF3 MDF4 ATFX Lucene Lucene: Indexing Technology No relational database necessary Just a file system storage NetCDF MATLAB many more MaDaM -Web Server Service MaDaM General and business logic for organizing test data by pure HTML-5 HTML5: User interface, available in nearly any browser PDF: Reports as PDF-files readable in any PDF-reader : Client post processing software for interactive analysis, connected via EnCom Optimized traffic with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 6
Importing of Data Dedicated importer components Specialized for certain data formats, e.g. MDF v3 & v4, ISO13499 MME, Diadem, Excel, Running on client side Based on JavaFX With modern user guidance (wizard) Guides the editing process and force consistency 17. June 2015 MaDaM Measurement Data Management for Big Data Page 7
Sample-Importer: Passive Safety (Crash) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 8
File System DIAdem MDF3 MDF4 ATFX Lucene MaDaM Indexing Technology No relational database necessary NetCDF MATLAB many others - Web Server Service by pure HTML-5 Just a file system storage ipad Client PC Windows, Linux, Mac, ios, Android 17. June 2015 MaDaM Measurement Data Management for Big Data Page 9
Apache Lucene Objective: Powerful search functionality with high performance Data-based index Project of the Apache Foundation Used by: Google, Apple, IBM, LinkedIn, Twitter, Eclipse Foundation, OpenSource 17. June 2015 MaDaM Measurement Data Management for Big Data Page 10
Relational Database Problem: Model unnecessarily complex Problem: Adaptation to customer needs and new formats difficult Problem: Queries are complex & complicated Problem: Additional license costs for high-performance database solutions (Oracle, Microsoft SQL or others) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 11
Search for Tests: Web-Frontend Web-Frontend based on modern web technologies: HTML5 JavaScript AJAX Requires only a browser on client side Device independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 12
Desktop Windows Linux Mac OS X Solaris Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 13
Desktop Windows Linux Mac OS X Solaris Smart device ios Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 14
Desktop Windows Linux Mac OS X Solaris Smart device ios Android Windows 8 Device Independent 17. June 2015 MaDaM Measurement Data Management for Big Data Page 15
Search for Measurements Full text search Full text search: Required time: 118 ms, show 100 of 1000 result(s) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 16
Search for Measurements Combination of search items Full text search: Required time: 21 ms, show 100 of 1000 result(s) 17. June 2015 MaDaM Measurement Data Management for Big Data Page 17
Search for Measurements Facet search Show all China NCAP tests which were performed by test lab LaboTron 17. June 2015 MaDaM Measurement Data Management for Big Data Page 18
Search for Measurements Any combination of facet and full text search Search for: Date of measurement Test engineer, Laboratory Many more Search in measurements Statistical data of channels are extracted during import 17. June 2015 MaDaM Measurement Data Management for Big Data Page 19
Search for Measurements Search for statistical quantities: Min, Max, 17. June 2015 MaDaM Measurement Data Management for Big Data Page 20
Search for Measurements High-performance 1000 measurements in < 20ms Scales independent from measurement size Flexible and powerful search syntax Full text search Search in channels Data found, what now? 17. June 2015 MaDaM Measurement Data Management for Big Data Page 21
Reports PDF Reports in 3 simple steps 1. Search / Choose 2. Choose template 3. Click button 17. June 2015 MaDaM Measurement Data Management for Big Data Page 22
17. June 2015 MaDaM Measurement Data Management for Big Data Page 23
Reports (on demand) Reports are generated on demand on the server using as a web service Only a PDF-viewer is required on the client PCs or smart devices MaDaM - Web Server Service File System Lucene by pure HTML-5 17. June 2015 MaDaM Measurement Data Management for Big Data Page 24
Interactive Analysis also works as an app on client PC No pre-installation on the client: Using Java WebStart technology, is deployed over the web on-the-fly PC - Web Server Service Client Lucene MaDaM by pure HTML-5 optimized datatransfer with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 25
Acceleration Date: 1998-11-06 Channels Results Example: Crash Evaluation Based on the MME Standard page 1 of 37 X Y Z Res 3 ms continuous: 58,29 grav Interactive Analysis: NetCDF ASCII ATFX DIAdem Excel MDF Crash Head Accelerations Dummy front left, Hybrid III mid-sized adult male 50% Test Test Object Laboratory No.: 98/7707 Name: Customer No.: M5533 Velocity: Type: Vehicle into Vehicle frontal 0 Mass: 70 grav 60 50 40 30 FIESTA (HTE) 14,5 m/s 1.151 kg 20 10 0-10 0,00 0,05 0,10 0,15 0,20 s 0,25 Time AXL TETE HVG CFC1000 Calculation range: 0-250 ms AXT TETE HVG AXV TETE HVG CFC1000 CFC1000 Max.res.ac: 60,01 grav (76,3 ms) HIC: 534,4 (57,6 ms - 96,8 ms) Resulting acceleration CFC1000 HIC 36: 529,0 (60,7 ms - 96,6 ms) HIC 15: 348,8 (70,6 ms - 85,6 ms) 3 ms cumulative: 58,09 grav 17. June 2015 MaDaM Measurement Data Management for Big Data Page 26
Lucene DIAdem NetCDF MDF3 MDF4 MATLAB MME many more MaDaM - Web Server Service by pure HTML-5 Optimized traffic using Client Client Client Client ipad PC 17. June 2015 MaDaM Measurement Data Management for Big Data Page 27
Enterprise Communication Data Server Client 17. June 2015 MaDaM Measurement Data Management for Big Data Page 28
Enterprise Communication Only channels actually used are transmitted Only value ranges actually needed are transmitted What was requested? 2 MB Data Server What do we have? 1 GB Client 17. June 2015 MaDaM Measurement Data Management for Big Data Page 29
Enterprise Communication Only required channels are transmitted The channel data transmission is optimized In less than ½ second, the user sees the complete graphic of a channel with more than 10 Mio. values in on client side the user can gradually zoom in even to individual values of the signal In summary: minimizes network traffic by intelligent and selective data transfer. 17. June 2015 MaDaM Measurement Data Management for Big Data Page 30
: an open ASAM-CEA Framework Online Measurement & Control FFT Formula User CEA-Calc >100 Calculations Automation System ATFX Test Bench SCADA Operator GUI Data Base Input Component & Flow Management Graphic Layout Report DIAdem MDF UFF NetCDF Offline File Import Unit Services 17. June 2015 MaDaM Measurement Data Management for Big Data Page 31 Data Item EnCom 3rd party MDM other
: > 50 Data formats are supported ASAM-ODS: ATF and ATFX Diadem: DAT, TDM & TDMS MDF: version 3, incl. resampling version 4, ASAM compliant NetCDF ASCII: comma, tab, blanks Excel: binary and CVS CAN log Famos EDAS ETAS: DCM LabVIEW ISO13499: MME, ISO DBase 3 Dasylab Google: KML GPS-exchange: GPX Garmin Database AVL: ifile Q-DAS Racelogic-Vbox RPC-III UFF: 15, 58 and 151 TEAC TecPlot 17. June 2015 MaDaM Measurement Data Management for Big Data Page 32
: > Curves with >20,000,000 values All > 20,000,000 values in one diagram without delay Tracking window, Values at cursor, reference/delta Axis, others 17. June 2015 MaDaM Measurement Data Management for Big Data Page 33
: > Georeferenced Data 17. June 2015 MaDaM Measurement Data Management for Big Data Page 34
: Geo & Video & Num. Data Video of Test Graph content: 1. Google Maps as background 2. GPS-Track with speed as color 3. Sprite Car Linegraph: Speed vs. Time 17. June 2015 MaDaM Measurement Data Management for Big Data Page 35
: Orderanalysis 17. June 2015 MaDaM Measurement Data Management for Big Data Page 36
: Sound FFT, Ampl., Player Sonogram RMS-signal display Audio-player Synchronized 17. June 2015 MaDaM Measurement Data Management for Big Data Page 37
: Testdata in 3D-Space 17. June 2015 MaDaM Measurement Data Management for Big Data Page 38
Summary Measurement Data Management with MaDaM : Modern web technologies (HTML 5) for simple workflows under PC and smart devices Fast and powerful search function No license costs for third party database products Server based NCAP-report generation using Client based interactive analysis with Optimized data transmission with 17. June 2015 MaDaM Measurement Data Management for Big Data Page 39
Gesellschaft für angewandte Mess- und Systemtechnik mbh Bahnhofstraße 6 1760 Opdyke Court German Centre, Unit 719A 09111 Chemnitz Auburn Hills, MI 48326 88 Keyuan Road, Pudong Germany USA Shanghai 201203 / PR China Tel.: +49 (371) 918 668-0 Tel.: +1 (248) 270-7779 Tel.: +86 (21) 289 866 19 Fax.: +49 (371) 918 668-99 Fax: +1 (248) 393-0340 Fax: +86 (21) 289 865 11 E-Mail: info@amsonline.de E-Mail: info@amsonline.eu E-Mail: info@amsonline.cn Web: www.amsonline.de Web: www.amsonline.eu Web: www.amsonline.cn