Analysis of Large Scale Data Volumes

Transcription

1 Analysis of Large Scale Data Volumes HighQSoft s Analysis Server Merlin 2G ASAM US Workshop, Novi Dr. Ralf Nörenberg ralf.noerenberg@highqsoft.de

2 HighQSoft s Analysis Server Merlin 2G Content The basic ideas of Merlin The basic ideas of an analysis Layout of an analysis Merlin as an infrastructure Setup of Big Data Use Case II setup

3 The basic ideas of Merlin Any test/measurement is subject to analysis Domain-Experts lose time and resources developing duplicate analysis programs which also run next door. The analysis programs contain a high level on domain specific know-how will only run once (locally) and can not be contributed to a automated process

4 The basic ideas of Merlin Basic Setup WEB Server Importer ODS Server Notification Oracle and Files Indexer

5 The basic ideas of Merlin Automated Analysis WEB Server Merlin Analysis Server Importer ODS Server Notification Indexer

6 The basic ideas of Merlin Automated Report / Results WEB Server Merlin Analysis Server Importer ODS Server Notification Indexer

7 HighQSoft s Analysis Server Merlin 2G Content The basic ideas of Merlin The basic ideas of an analysis Layout of an analysis Merlin as an infrastructure Setup of Big Data Use Case II setup

8 Basic ideas of an analysis Analyses shall be small and manageable and include domain know-how, no IT know-how

9 Basic ideas of an analysis Analyses have read- and write access on ODS data Read and write of meta- and mass-data Calculation of missing attributes, consistency checks of data

10 Basic ideas of an analysis Analyses may be affected by defined parameters Example: Channel names, definition of scopes, arguments for calculations, internationalization End-Users only have limited and defined possibilities of interaction

11 Basic ideas of an analysis Analyses will deliver results based on data file MIME-types Example: generated images, movies, sounds, PDF reports, ATF/XML

12 Basic ideas of an analysis Analyses may use other analyses and are interchangeable Example: Statistics uses Min and Max

13 Basic ideas of an analysis Analyses may be integrated in different languages Examples: JAVA, MATLAB, DIAdem, C/C++, System Scripts,..

14 Basic ideas of an analysis Analyses are to be understood as standard analyses Example: Every measurement of certain test stands

15 Analyses: NOT intended Analyses having access on local resources of the end-users Examples: local files, system settings, CPU or memory.

16 Analyses: NOT intended End-users having access to the source-code or the scripts of analyses Example: Editing of DIAdem scripts or MATLAB Code

17 HighQSoft s Analysis Server Merlin 2G Content The basic ideas of Merlin The basic ideas of an analysis Layout of an analysis Merlin as an infrastructure Setup of Data Use Case II

18 The layout of an analysis 3-Level architecture Structure Call 1:1 1:n n 1 1:n The structure defines a program procedure and the utilization of a return value to the client application to the subsequent analysis Analysis Program Parameter

19 The layout of an analysis 3-Level architecture Structure 1:n Relations ensure the reusability of every entity. n 1 Call 1:1 1:n Analysis Program Parameter

20 The layout of an analysis 3-Level architecture Structure Call 1:1 1:n Analysis Program Parameter n 1 1:n The call is a bracket around the analysis program and its configuration. This ensures multiple combinations of both. Parameters are settings for the controller, e.g. a timeout can be defined.

21 The layout of an analysis 3-Level architecture Structure n 1 Call 1:n Analysis programs may be combined with multiple parameters. Parameters will be passed on from the Merlin Engine. 1:1 1:n Analysis Program Parameter Parameters may contain complex information, e.g. be a XML file.

22 The layout of an analysis Example

23 HighQSoft s Analysis Server Merlin 2G Content The basic ideas of Merlin The basic ideas of an analysis Layout of an analysis Merlin as an infrastructure Setup Big Data Use Case II

24 Merlin as an infrastructure Web Interface End User Administration Starting analyses and retrieving results Management of analyses and parameters JAVA Loader Load and Start Matlab Loader XY Loader Merlin Engine Structures & Parameters Pool of analysis programs and code

25 Merlin as an infrastructure Web Interface End User Administration Starting analyses and retrieving results Management of analyses and parameters Load and Start JAVA Loader Matlab Loader XY Loader Merlin Engine Pool of analysis programs and code Structures & Parameters Queuing Contexts Parallelism

26 Merlin as an infrastructure Integration of the whole process Analysis is a task that is standardized and strictly repetitive (>80%) Analysis can be done on-site, only results need to be forwarded Measurement Data Avalon ODS Server Notification Service Merlin Server Automated flow based on measurement MIME types. Access Result Report / Data JAVA / Matlab / XY (Individual analysis by specialists will always require a download)

27 Merlin as an infrastructure Integration of Merlin in a state-of-the-art setup Data Generator (Supplier/OEM) CUSTOMER 3 rd party tools Web Browser compare /analyze / verify a specific behavior / functionality on one (or multiple measurements) ATFx Files MDF Files Any Files Tool-Level Data Management & Individual Analysis Descriptive data Measurement data / mass data Importer (Server) ODS API ODS Server Web Server Analysis Server Server-Level (HQS) Data Pre-Processing Data Organization Data Post-Processing ASAM ODS Specification for ODS API and Application Model Data Base (Oracle) Data-Level Physical Storage File Server (hard drive) HighQSoft GmbH

28 HighQSoft s Analysis Server Merlin 2G Content The basic ideas of Merlin The basic ideas of an analysis Layout of an analysis Merlin as an infrastructure Setup of Big Data Use Case II setup

29 Big Data Use Case II setup Vehicle Fleet Testing (planned tests) This road-load data project is set up in an environment of a car manufacture who has all vehicle fleet testing data (and most activities) centralized within one project. The scope of this projects contains many customers (all internal development departments) all passenger vehicle types and almost all ECUs available all domains and data types over 500 testing vehicles to be managed Within ODS, this is the upmost state-of-the-art system to manage the test execution flow and standardized measurement data.

30 Big Data Use Case II setup Overall System Architecture (detailed) Customer / Employee Web Server Third Party Tools (e.g. DIAdem) Test Request Form Analysis Server ODS Server & Oracle Database Data Importer Data Pre- Processor Notification Services Dropper Engine Measurement Points MGMT System Vehicles & Parts Management System FTP Storage System Data Archive Measurement Equipment Vehicles Test Run company network vehicle Vehicle File Storage System Standardized Data Interface OEM / Supplier HighQSoft GmbH Vehicle External System

31 Big Data Use Case II setup In Numbers Data Base Size (Oracle) Files on File Server(s) Files Volume Absolute growth Files in tape archive Files Volume Analysis Servers No. of Servers No. of Analyses ODS Servers Productive ODS Servers GB Files GB GB per day Files GB in total per month in total Project is state-of-the-art and meets performance requirements

32 Thank you! Any questions, suggestions and ideas? HighQSoft GmbH