SQMB '11 Automated Model Quality Rating of Embedded Systems Jan Scheible (jan.scheible@daimler.com) Daimler AG - Group Research and Advanced Engineering Hartmut Pohlheim (pohlheim@model-engineers.com) Model Engineering Solutions GmbH GR/PSS / Model-based Development / 21.02.2011
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 2 Agenda Challenge Approach Further Evaluation Summary and Outlook
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 3 Challenge (1/2) Current Situation - Model-based software development increasingly becoming standard in automotive industry. - Size and complexity of models becoming ever larger: - Example of a large Matlab Simulink model from passenger vehicle domain: - approx. 15,000 blocks - 700 subsystems - Subsystem hierarchy with 16 levels - Considerable time pressure in development. - Despite high abstraction level, developers have much freedom. Lots of possibilities for potential errors.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 4 Challenge (2/2) Goals - Rating of model quality in an automatic and comprehensible manner. - Reduce costs and effort. - Use of existing information to as high a degree as possible. - Provide a compact overview of model quality. - Visualization of current status and progress of model quality.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 5 Agenda Challenge Approach Model Quality Quality Model in the Development Process Evaluation of Measured Values Aggregation of Evaluations Further Evaluation Summary and Outlook
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 6 Approach Overview Framework zur for automatisierten Automatic Quality Ermittlung Rating of der Modellqualität Quality analytisch analytic Quality Model Aspects Factors Metrics Criteria Quality Rating constructive Patterns Guidance Architecture Patterns Guidelines
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 7 Approach Overview Framework zur for automatisierten Automatic Quality Ermittlung Rating of der Modellqualität Quality analytisch analytic Quality Model Aspects Factors Metrics Criteria Quality Rating constructive Patterns Guidance Architecture Patterns Guidelines
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 8 Model Quality (1/2) High-Quality Simulink Models - A model may fulfill criteria with a positive influence on desired factors (e.g. maintainability Maintainability, comprehensibility Comprehensibility or reliability Reliability ). - The more criteria a Simulink model fulfills, the higher its quality. - All criteria that allow inference about desired factors are of interest.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 9 Model Quality (2/2) Objective Rating - An objective rating is possible with the help of a quality model. Model Quality Factors are desired properties for high-quality models Factor 1 Factor 2 Criteria are indicators of desired properties Criterion 1 Criterion 2 Evaluation Metrics check the fulfillment of the criteria Metric 1 Metric 2 Metric 3 - The quality model currently consists of 6 factors, 18 criteria and 46 metrics.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 10 Quality Model in the Development Process (1/2) Software requirements Integration test Requirements coverage of model Quality criteria of model Guideline conformity of model Requirements Software design Models Software integration Module test Test implementation Test specification Requirements coverage of test specification Complexity of model Implementation Code Result of model test Result of static code analysis Model coverage of test implementation Quality Model Framework for rating Rating Model model Quality quality
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 11 Quality Model in the Development Process (2/2) Classification Model Quality Correctness Comprehensibility Testability Code generation ability Reliability Maintainability Functionality Scale Test coverage Runtime errors Standard conformity Requirements coverage of model Complexity of model Requirements coverage of test specification Result of model test Result of static code analysis Guideline conformity of model Model coverage of test implementation
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 12 Quality Model in the Development Process (2/2) Classification Model Quality Correctness Reliability Correctness Reliability Comprehensibility Maintainability Testability Functionality Code generation ability Functionality Scale Test coverage Runtime errors Requirements coverage of model Runtime errors Complexity of model Standard conformity Requirements coverage of model Requirements coverage of test specification Result of model test Result of static code analysis Guideline conformity of model Model coverage of test implementation Result of model test Result of static code analysis
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 13 Evaluation of Measured Values (1/5) Measurements Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828 v 2 = 87 v 3 = 18.1 v 4 = 54 v 5 = 0.83 v 6 = 1.0
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 14 Evaluation of Measured Values (2/5) Determination of Permissible Values Model quality - Define limits for each metric in order to check their permissible values. Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828? v 2 = 87? v 3 = 18.1? v 4 = 54? v 5 = 0.83 e 5 = 83% v 6 = 1.0 e 6 = 100% min max min max min max min max
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 15 Evaluation of Measured Values (3/5) Reference Model Comprehensibility Model quality Maintainability Correctness - A reference model describes how an average Simulink model should look. Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage v 1 = 828 e 1 = 100% v 2 = 87 e 2 = 20% v 3 = 18.1 e 3 = 98% v 4 = 54? v 5 = 0.83 e 5 = 83% v 6 = 1.0 e 6 = 100% 828 2,146 0 87 869 0.0 18.1 36.4 min max 3,735 blocks => Minimum Maximum # Crossed lines # Subsystems - 0 2,146 869 Ø Signals per Bus 0.0 36.4
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 16 Model quality Evaluation of Measured Values (4/5) Rules Comprehensibility Maintainability - Rules specifying relations between measured values. Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines v 1 = 828 e 1 = 100% #Subsystems v 2 = 87 e 2 = 20% ØSignals per Bus v 3 = 18.1 e 3 = 98% #DataStore Read v 4 = 54 e 4 = 50% Passed tests v 5 = 0.83 e 5 = 83% Requirements coverage v 6 = 1.0 e 6 = 100% 54 57 60 #DataStore Memory = 57 #DataStore Read #DataStore Memory
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 17 Evaluation of Measured Values (5/5) Visualization ø Signals per Bus 32.8 25.3 # Internal states 1,335 11,541 12,601 # Lines # Clones 725 1,065 5,040 6,683 # Crossed lines 499 7.7 10,099 ø Outport links 1,255 1.25 # Subsystems 2,437 1.21 1,908 1,840 48 44 48 40 0.7 6 2.2 9.0 8 9 Depth of subsystem hierarchy ø Height of subsystem representation 110 68 ø Width of subsystem representation 1.3 1.5 ø Width of output interface 15.1 ø Width of input interface
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 18 Aggregation of Evaluations (1/5) Evaluated Metrics Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 19 Aggregation of Evaluations (2/5) Algorithm - Aggregated values provide a quick overview of a model s quality. - An arithmetic mean is not suitable: If one or more metrics are not fulfilled to a sufficient extent, this must lead to a devaluation. 60 Evaluations: e 1 = [25%, 30%, 35%] e 2 = [10%, 80%, 90%] Aggregated evaluation (in percent) 50 40 30 20 10 0 0 1 2 3 Number of evaluations under threshold
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 20 Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality Comprehensibility Maintainability Correctness Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 21 Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 22 Aggregation of Evaluations (3/5) Aggregation in the Quality Model Threshold = 30% Model quality e 11 = 32,4% e 12 = 50% e 13 = 91,5% Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 23 Aggregation of Evaluations (3/5) Aggregation in the Quality Model e 14 = 58% Threshold = 30% Model quality e 11 = 32,4% e 12 = 50% e 13 = 91,5% Comprehensibility Maintainability Correctness e 7 = 100% e 8 = 29,5% e 9 = 50% e 10 = 91,5% Readability Scale Avoidance of implicit constructs Functional correctness #Crossed lines #Subsystems ØSignals per Bus #DataStore Read Passed tests Requirements coverage e 1 = 100% e 2 = 20% e 3 = 98% e 4 = 50% e 5 = 83% e 6 = 100%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 24 Aggregation of Evaluations (4/5) Visualization 13% 36% 34% 34% 40% 100% 50% 90% 100% 90% 31% 31% 33% 33% 42% 31% 0% 0% 36% 36% 36%
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 25 Aggregation of Evaluations (5/5) Retracing Quality Rating over Time - Model quality rating provides a snapshot of a model s evaluations at a specific time. - Trend analysis is used to retrace the evolution and evaluation of measured values over time. - Shows how the metrics measured values lie in relation to their limits. 100% 100% 100% 100% Metric 1 50% Metric 2 25% 0% 0% 1st iteration 2nd iteration 3rd iteration 1,350 Maximum 1,300 1,200 Metric 2 1,250 1,150 Minimum 80 60 Maximum Metric 1 45 52 1st iteration 2nd iteration 3rd iteration
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 26 Agenda Challenge Approach Further Evaluation Summary and Outlook
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 27 Further Evaluation 1. Comparison of a model s quality rating with perceived model quality by the developers. 2. Verify whether the number of findings in model reviews correlates with the quality rating. 3. Verify whether the appearance of actual bugs correlates with the quality rating. Rate as many models as possible.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 28 Agenda Challenge Approach Further Evaluation Summary and Outlook
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 29 Summary - A quality model is used for defining our notion of model quality. - Most important artifacts of the model-based development process are taken into account. - Metrics check the fulfillment of desired quality criteria. - The measured values of the metrics are evaluated. - Evaluations are aggregated in the quality model to provide a quick overview. - Retracing quality rating over time.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 30 Outlook Evaluation - Verify normalization by block count in the reference model [in progress]. - Check if the approach is only applicable to models in the same project [in progress]. Quality model - Potentially the expressive power of the quality model must be enhanced (e.g. handling of conflicting measurements). Metrics - Integration of more Simulink-specific metrics [in progress]. - Identification of the metrics with the highest relevance [in progress]. - Add more interfaces to tools for automatic rating [in progress]. - Compare the frequency distribution of the per subsystem measurements.
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 31 Thank you for your attention!
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 32 Backup
Reference Model - Average values of arbitrary models normed against the model s reference (#blocks) Measurements database Modell 1 Modell 2 Modell 3 Modell 4 Modell 5 Average Variance #Subsystemanmerkungen Gotos 44 9 0 32 47 0.01318 0.01551 # Subsystems 340 677 231 150 1,083 0.17428 0.12372 # Crossed lines 299 931 6,683 828 2,581 0.45065 0.20779 # Blocks 1,120 2,924 9,017 2,526 4,309 1.0 0.0 Instantiated reference model # Subsystems # Lines # Crossed lines 200 blocks 1,000 blocks 5,000 blocks Min. Max. Min. Max. Min. Max. 14 46 70 230 348 1,149 224 256 1,120 1,280 5,600 6,400-96 - 480-2,400 - Average values multiplied by reference (#blocks) - Min and max result from variance of averages. Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 33
Prototype Infrastructure Qualitätsmodell Metamodell Faktoren Metriken Kriterien Simulink-Modell Data-Provider MXAM MXRAY manuelle Eingaben sldiagnostics Metrik 1 Metrik 2 Metrik 3... Metrik n Qualitätsbewertung Messergebnisse Messwert 1 Messwert 2 Messwert 3... Messwert n Präsentation Messobjekt
Automated Model Quality Rating of Embedded Systems / GR/PSS / 21.02.2011 35 Aggregation of Evaluations (?/?) Algorithm and Formula e 1 = [25%, 30%, 35%] e 2 = [10%, 80%, 90%]
Evaluation of Measured Values (?/?) Limits Tolerance Tolerance 100% 0% 0 Minimum Maximum Tolerance Tolerance 100% 0% 0 Minimum 0% 0 Maximum