OPM Example- Improving Software Code Quality by reducing Code Complexity using Klocwork Sarit Assaraf sassaraf@iai.co.il Yossi Cohen Yscohen@iai.co.il SEPG NORTH AMERICA The CMMI CONFERENCE 6-7 May 2014 Washington DC.
( IAI ) Israel Aerospace Industries Largest industrial company in Israel Missiles, Satellites, UAs, Avionics, Upgrades, Radars, etc. Activities encompassing: Development, Production, Maintenance and Service of Aerospace Systems IAI divisions are ISO000 and AS0-C certified CMMI ML5- Systems Missiles & Space Group(5 Div.), ELTA Systems Group Ltd. (4 Div.), MALAT Div. CMMI ML3- LAHA Div., ENG Div. 2
SYSTEMS MISSILES & SPACE GROUP Agenda enus Observation satellites Introduction improvement and and validation TECSAR Synthetic Aperture Radar EROS C OPTSAT 3000 OFEQ SHAIT launch vehicles Ground control stations Communication satellites AMOS 3 AMOS 1,2 AMOS 4 Advanced comm. Sat. 3
Introduction and improvement Challenges Testing Phase s Integration and testing of large scale systems can be a long and tedious phase in the systems lifecycle. Typical characteristics of this phase include many defects and failures, not enough resources and significantly tight schedule. In high maturity organizations we expect to find relatively few defects in these late stages and expect even to prevent their injection in the early lifecycle stages. Software code complexity is an important indicator of the amount of defects expected at integration, by lowering the complexity, the defect rate will drop and so will costs and schedule deviations. Controlling complexity means controlling injection, and allows planning realistic schedules, thus supporting the achievement of quality and operational goals 4
improvement Software Code Quality Studies and publications show relationships between external SW characteristics (quality) and internal product (static code) attributes: Code complexity can predict defective SW components Static code complexity measures can be useful indicators of software quality, i.e. the amount of defects Since defects cost increases through lifecycle, project costs may increase dramatically The "ideal" design process implies lower defect injection and early detection at early project stages, thus preventing bugs and lowering design costs and meeting projects' schedule. This leads to a change in the SW design process and provides a new tool for project s risk management 5
improvement Cyclomatic Complexity vs. Defects 1 A positive correlation between code complexity and the amount of defects (faults) was suggested Code complexity therefore, would be an important SW attribute we will need to measure and control The software engineers evaluate complexity, and by simplifying the software code, adjust it until the defined goal. (1) Fenton & Ohlsson (2007) 6
improvement Defect detection- Caper Jones' research (1) Research results Injected: 85% of the defects during Software coding Discovered: 30% during Unit Testing 50% during later testing stages Conclusions: Combining inspections, static analysis, and testing is cheaper than testing by itself and leads to much better defect removal efficiency levels. In concert, these approaches also shorten development schedules by more than 45% because, when testing starts after inspections, almost 85% of the defects already will have been addressed. (1) (Dr. Dobb's,June 20) 7
improvement The tools Process The research directed us to control code complexity By analyzing code complexity immediately after compilation using the static tool metrics, and prior to Unit Testing and Build release, the Software designer can proactively control the actual amount of bugs expected during testing stage A benchmark research and analysis to evaluate and compare static code tools was performed, collecting knowledge and experience from other divisions and groups The tools evaluation process roadmap: Mapping of current SW Dev. environments in our projects Identification of the alternatives (COTS available tools) Definition of the criteria to be used for the alternatives evaluation Alternatives evaluation according to criteria and decision 8
SYSTEMS MISSILES & SPACE GROUP improvement Tools identification and mapping (projects) Aircraft name Target Environment Project Name PcLint DC Parasoft Multi 5 S MCPA 486 C & RT.., R MCPA 486 C & RT H MCPA PPc C & GHS. MCPA 486 C & RT.,, M1 MCPA PPc C & GHS... UMAM PPc C & GHS.. AMAC PPc C++, Rhapsody & GHS p M2 AMAC PPc C++, Rhapsody & GHS.. EIOC PPc C++, Rhapsody & GHS.. p DSim PC C Borland. A UNI C Solaris Workshop, Motif (Lint for Unix) I UNI + PC C# p DC Double Check: Currently, p In the past: Project: Currently in process, Was in process, Currently not in process.
improvement Alternatives Compatibility Multi+DC Parasoft Klocwork LDRA Compatibility with dev. languages (DC) -(2) Compatibility with Operation systems? RT (make file) initial GHS -Multi - + adjusted GHS -Multi? Borland Solaris Motif PC UNI Check rules Misra2004 Misra2008
improvement Alternatives evaluation according to criteria (1) 1. Contact Initiation & general issues Subject Multi+DC Parasoft Klocwork LDRA Response to RFI 4 Responsiveness (to questions) 4 Response time 2 Appointments 4 Detailed explanations upon request 1 Installation easiness 3 Installation support 2 Compatibility to existing environment 5 5 8 Work environment Sub Total 7.00 8.50.14 4.22 Criteria Definition: 1 to (=full compliance) no response 11
improvement Alternatives evaluation according to criteria (2) 2. Tests Run process Subject Multi+DC Parasoft Klocwork LDRA Project selection 7 Environment and compiler 2 5 Files addition and removal 3 Tailoring to project needs 4 8 Specific tests 8 8 Specific testing rules 7 Tests run Tests run lead time for project M1 2 Sub Total 7.75 8.38.50 7.63 Criteria Definition: 1 to (=full compliance) no response 12
improvement Alternatives evaluation summary Multi+DC Parasoft Klocwork LDRA 1. Contact Initiation & general issues 7.00 8.50.14 4.22 2. Tests Run process 7.75 8.38.50 7.63 3. Report generation 2.25 7.88 7.88 5.75 Total 5.22 8.20 8.83 5.80 8 7 6 5 4 3 2 1 0 סיכום Total כללי Report הפקת דוחות generation Tests הרצת run בדיקות קדם General issues וכללי At the end of the decision LDRA process Klocwork that involved senior SW Parasoft managers, Klocwork was Multi+DC selected 13
improvement improvemen t and validation Improvement Analysis and alidation One of the requirements of the OPM process is to analyze and validate the impact of the suggested improvement The validation can be performed either by conducting a pilot, or by simulation The research implied that complexity influences the amount of defects, we needed to validate it on our processes We would improve the SW code development process, and particularly the sub-process of code writing to meet the quality goal of the project By selecting projects at a late stage of their lifecycle, that did not used Klocwork nor measured and controlled complexity, we could measure: The Quality Goal during later testing stages (i.e. No. of defects) The Process Performance prior to the improvement (i.e. Complexity) The data was analyzed and the complexity baseline and the defects amounts baselines were established 14
improvement improvemen t and validation Software Code Complexity Baseline The complexity of routines/functions from projects was analyzed using Klocwork: 70% of the items had less than Complexity=60 Scree plot of Complexity vs. Routine# Pareto dist. of Complexity 60 #45 #70 15
Number of Defects Baseline The number of defects per 00 work-hours from projects was analyzed Analysis showed that project's lifecycle, Reuse or New (i.e. full scale development) was also a factor, and resulted in establishing two different baselines: Total Defects vs Actual K work hours Two project's Types: New & Reuse improvement improvemen t and validation 30 NEW UCL=27.17 REUSE 20 USL=21.74 1.71 Individual alue 0 LCL=12.24 USL=.5 UCL=12.44 _ =3.20 LCL=-6.04 - H LR-WCS H-super A3 GS Tir proj A4 GS A4 FS LR-SKR EN Descriptive Statistics: ariable Type Mean StDev Minimum Maximum defects vs. K work hours NEW 1.71 2.83 15.40 22.72 REUSE 3.203 1.81 0.63 5.000 16
Process improvement goal We set the target of lowering the amount of defects found during integration and testing stage, by 20% relative to the current baseline. By analyzing code complexity immediately after compilation and prior to Unit Testing and Build Release, the Software designer can predict and proactively control the actual amount of bugs expected during the later testing stages. No. defects per 00 WorkHs.(y): improvement improvemen t and validation Project type NEW Current Baseline 12.24 < y New < 27.17 (y avg. =20) Goal baseline.8<y New <21.74 (y avg. =16) REUSE y Reuse < 12.44 (y avg. =3) y Reuse <.5 (y avg. =2.5) 17
improvement and validation Plan alidation on two projects of each type was performed, and the deployment plan was established, including: Success criteria of the deployment stage, and evaluation milestones List of the projects that will use Klocwork Resources budget, number of licenses Installation, training and coaching activities Monitoring and meetings milestones and activities Measurements management activities Data analysis (from Klocwork) Complexity baseline update (every 6 months) Number of defects prediction and confidence interval analysis Klocwork will be used during compilation, and according to the complexity results the code will be corrected (<60), thus allowing meeting the number of defects goal. 18
improvement and validation The IAI conclusions Meeting project milestones strongly depends on a system readiness to be integrated, tested, delivered, and most importantly whether it will adequately perform at the deployment stage. Software code complexity is an important indicator of the amount of defects expected at integration, by lowering the complexity, the defect rate will drop and so will costs and schedule deviations. Klocwork tool provides complexity measurement immediately after compilation and prior to Unit Testing and Build release, by simplifying the software code the Software designer adjusts it until the defined target goal. We distinguished two types of projects ( Reuse and New ), thus established two improvement goals values, and planned the deployment accordingly. The OPM implementation triggered changes in SW design processes (specifically of SW code writing ), leading to a different approach of project s risk mitigation based on the amount of defects prediction continues and final conclusions will be analyzed during this year 1
20
improvement and validation References Jones, C., (20), Get SW Quality Right, Dr. Dobb s June 20 Zhang, H., Zhang,., Gu, M., (2007) Predicting Defective Software Components from Code Complexity Measures, 13 th IEEE International Symposium on Pacific dependable,, Khaled, E., Hailey,., (2003) The Costumer Costs of SW Quality, Technical report, number 02-03, K Sharp Technology Inc., pp. 3-6 July Fenton N., Ohlsson N. (2007) Quantitative Analysis of Faults and Failures in a Complex Software System, IEEE Transactions on SW Engineering. ol.26 no.7 21