SQMB '11 Automated Model Quality Rating of Embedded Systems

Transcription

1 SQMB '11 Automated Model Quality Rating of Embedded Systems Jan Scheible Daimler AG - Group Research and Advanced Engineering Hartmut Pohlheim (pohlheim@model-engineers.com) Model Engineering Solutions GmbH GR/PSS / Model-based Development /

2 Automated Model Quality Rating of Embedded Systems / GR/PSS / Agenda Challenge Approach Further Evaluation Summary and Outlook

3 Automated Model Quality Rating of Embedded Systems / GR/PSS / Challenge (1/2) Current Situation - Model-based software development increasingly becoming standard in automotive industry. - Size and complexity of models becoming ever larger: - Example of a large Matlab Simulink model from passenger vehicle domain: - approx. 15,000 blocks subsystems - Subsystem hierarchy with 16 levels - Considerable time pressure in development. - Despite high abstraction level, developers have much freedom. Lots of possibilities for potential errors.

4 Automated Model Quality Rating of Embedded Systems / GR/PSS / Challenge (2/2) Goals - Rating of model quality in an automatic and comprehensible manner. - Reduce costs and effort. - Use of existing information to as high a degree as possible. - Provide a compact overview of model quality. - Visualization of current status and progress of model quality.

5 Automated Model Quality Rating of Embedded Systems / GR/PSS / Agenda Challenge Approach Model Quality Quality Model in the Development Process Evaluation of Measured Values Aggregation of Evaluations Further Evaluation Summary and Outlook

6 Automated Model Quality Rating of Embedded Systems / GR/PSS / Approach Overview Framework zur for automatisierten Automatic Quality Ermittlung Rating of der Modellqualität Quality analytisch analytic Quality Model Aspects Factors Metrics Criteria Quality Rating constructive Patterns Guidance Architecture Patterns Guidelines

7 Automated Model Quality Rating of Embedded Systems / GR/PSS / Approach Overview Framework zur for automatisierten Automatic Quality Ermittlung Rating of der Modellqualität Quality analytisch analytic Quality Model Aspects Factors Metrics Criteria Quality Rating constructive Patterns Guidance Architecture Patterns Guidelines

8 Automated Model Quality Rating of Embedded Systems / GR/PSS / Model Quality (1/2) High-Quality Simulink Models - A model may fulfill criteria with a positive influence on desired factors (e.g. maintainability Maintainability, comprehensibility Comprehensibility or reliability Reliability ). - The more criteria a Simulink model fulfills, the higher its quality. - All criteria that allow inference about desired factors are of interest.

9 Automated Model Quality Rating of Embedded Systems / GR/PSS / Model Quality (2/2) Objective Rating - An objective rating is possible with the help of a quality model. Model Quality Factors are desired properties for high-quality models Factor 1 Factor 2 Criteria are indicators of desired properties Criterion 1 Criterion 2 Evaluation Metrics check the fulfillment of the criteria Metric 1 Metric 2 Metric 3 - The quality model currently consists of 6 factors, 18 criteria and 46 metrics.

10 Automated Model Quality Rating of Embedded Systems / GR/PSS / Quality Model in the Development Process (1/2) Software requirements Integration test Requirements coverage of model Quality criteria of model Guideline conformity of model Requirements Software design Models Software integration Module test Test implementation Test specification Requirements coverage of test specification Complexity of model Implementation Code Result of model test Result of static code analysis Model coverage of test implementation Quality Model Framework for rating Rating Model model Quality quality

11 Automated Model Quality Rating of Embedded Systems / GR/PSS / Quality Model in the Development Process (2/2) Classification Model Quality Correctness Comprehensibility Testability Code generation ability Reliability Maintainability Functionality Scale Test coverage Runtime errors Standard conformity Requirements coverage of model Complexity of model Requirements coverage of test specification Result of model test Result of static code analysis Guideline conformity of model Model coverage of test implementation

12 Automated Model Quality Rating of Embedded Systems / GR/PSS / Quality Model in the Development Process (2/2) Classification Model Quality Correctness Reliability Correctness Reliability Comprehensibility Maintainability Testability Functionality Code generation ability Functionality Scale Test coverage Runtime errors Requirements coverage of model Runtime errors Complexity of model Standard conformity Requirements coverage of model Requirements coverage of test specification Result of model test Result of static code analysis Guideline conformity of model Model coverage of test implementation Result of model test Result of static code analysis

13 Automated Model Quality Rating of Embedded Systems / GR/PSS / Evaluation of Measured Values (1/5) Measurements Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828 v 2 = 87 v 3 = 18.1 v 4 = 54 v 5 = 0.83 v 6 = 1.0

14 Automated Model Quality Rating of Embedded Systems / GR/PSS / Evaluation of Measured Values (2/5) Determination of Permissible Values Model quality - Define limits for each metric in order to check their permissible values. Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828? v 2 = 87? v 3 = 18.1? v 4 = 54? v 5 = 0.83 e 5 = 83% v 6 = 1.0 e 6 = 100% min max min max min max min max

15 Automated Model Quality Rating of Embedded Systems / GR/PSS / Evaluation of Measured Values (3/5) Reference Model Comprehensibility Model quality Maintainability Correctness - A reference model describes how an average Simulink model should look. Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828 e 1 = 100% v 2 = 87 e 2 = 20% v 3 = 18.1 e 3 = 98% v 4 = 54? v 5 = 0.83 e 5 = 83% v 6 = 1.0 e 6 = 100% 828 2, min max 3,735 blocks => Minimum Maximum # Crossed lines # Subsystems - 0 2, Ø Signals per Bus

16 Automated Model Quality Rating of Embedded Systems / GR/PSS / Model quality Evaluation of Measured Values (4/5) Rules Comprehensibility Maintainability - Rules specifying relations between measured values. Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines v 1 = 828 e 1 = 100% #Subsystems v 2 = 87 e 2 = 20% ØSignals per Bus v 3 = 18.1 e 3 = 98% #DataStore Read v 4 = 54 e 4 = 50% Passed tests v 5 = 0.83 e 5 = 83% Requirements coverage v 6 = 1.0 e 6 = 100% #DataStore Memory = 57 #DataStore Read #DataStore Memory

17 Automated Model Quality Rating of Embedded Systems / GR/PSS / Evaluation of Measured Values (5/5) Visualization ø Signals per Bus # Internal states 1,335 11,541 12,601 # Lines # Clones 725 1,065 5,040 6,683 # Crossed lines ,099 ø Outport links 1, # Subsystems 2, ,908 1, Depth of subsystem hierarchy ø Height of subsystem representation ø Width of subsystem representation ø Width of output interface 15.1 ø Width of input interface

18 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (1/5) Evaluated Metrics Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%

19 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (2/5) Algorithm - Aggregated values provide a quick overview of a model s quality. - An arithmetic mean is not suitable: If one or more metrics are not fulfilled to a sufficient extent, this must lead to a devaluation. 60 Evaluations: e 1 = [25%, 30%, 35%] e 2 = [10%, 80%, 90%] Aggregated evaluation (in percent) Number of evaluations under threshold

20 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%

21 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%

22 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality e 11 = 32,4% e 12 = 50% e 13 = 91,5% Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%

23 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (3/5) Aggregation in the Quality Model e 14 = 58% Threshold = 30% Model quality e 11 = 32,4% e 12 = 50% e 13 = 91,5% Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%

24 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (4/5) Visualization 13% 36% 34% 34% 40% 100% 50% 90% 100% 90% 31% 31% 33% 33% 42% 31% 0% 0% 36% 36% 36%

25 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (5/5) Retracing Quality Rating over Time - Model quality rating provides a snapshot of a model s evaluations at a specific time. - Trend analysis is used to retrace the evolution and evaluation of measured values over time. - Shows how the metrics measured values lie in relation to their limits. 100% 100% 100% 100% Metric 1 50% Metric 2 25% 0% 0% 1st iteration 2nd iteration 3rd iteration 1,350 Maximum 1,300 1,200 Metric 2 1,250 1,150 Minimum Maximum Metric st iteration 2nd iteration 3rd iteration

27 Automated Model Quality Rating of Embedded Systems / GR/PSS / Further Evaluation 1. Comparison of a model s quality rating with perceived model quality by the developers. 2. Verify whether the number of findings in model reviews correlates with the quality rating. 3. Verify whether the appearance of actual bugs correlates with the quality rating. Rate as many models as possible.

29 Automated Model Quality Rating of Embedded Systems / GR/PSS / Summary - A quality model is used for defining our notion of model quality. - Most important artifacts of the model-based development process are taken into account. - Metrics check the fulfillment of desired quality criteria. - The measured values of the metrics are evaluated. - Evaluations are aggregated in the quality model to provide a quick overview. - Retracing quality rating over time.

30 Automated Model Quality Rating of Embedded Systems / GR/PSS / Outlook Evaluation - Verify normalization by block count in the reference model [in progress]. - Check if the approach is only applicable to models in the same project [in progress]. Quality model - Potentially the expressive power of the quality model must be enhanced (e.g. handling of conflicting measurements). Metrics - Integration of more Simulink-specific metrics [in progress]. - Identification of the metrics with the highest relevance [in progress]. - Add more interfaces to tools for automatic rating [in progress]. - Compare the frequency distribution of the per subsystem measurements.

31 Automated Model Quality Rating of Embedded Systems / GR/PSS / Thank you for your attention!

32 Automated Model Quality Rating of Embedded Systems / GR/PSS / Backup

33 Reference Model - Average values of arbitrary models normed against the model s reference (#blocks) Measurements database Modell 1 Modell 2 Modell 3 Modell 4 Modell 5 Average Variance #Subsystemanmerkungen Gotos # Subsystems , # Crossed lines , , # Blocks 1,120 2,924 9,017 2,526 4, Instantiated reference model # Subsystems # Lines # Crossed lines 200 blocks 1,000 blocks 5,000 blocks Min. Max. Min. Max. Min. Max , ,120 1,280 5,600 6, ,400 - Average values multiplied by reference (#blocks) - Min and max result from variance of averages. Automated Model Quality Rating of Embedded Systems / GR/PSS /

34 Prototype Infrastructure Qualitätsmodell Metamodell Faktoren Metriken Kriterien Simulink-Modell Data-Provider MXAM MXRAY manuelle Eingaben sldiagnostics Metrik 1 Metrik 2 Metrik 3... Metrik n Qualitätsbewertung Messergebnisse Messwert 1 Messwert 2 Messwert 3... Messwert n Präsentation Messobjekt

35 Automated Model Quality Rating of Embedded Systems / GR/PSS / Aggregation of Evaluations (?/?) Algorithm and Formula e 1 = [25%, 30%, 35%] e 2 = [10%, 80%, 90%]

36 Evaluation of Measured Values (?/?) Limits Tolerance Tolerance 100% 0% 0 Minimum Maximum Tolerance Tolerance 100% 0% 0 Minimum 0% 0 Maximum