Tests For EDA Testing Strategy

Transcription

1 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Test Strategy fr High Quality EDA Sftware Richard Léveillé Synpsys Inc. Muntain View, Califrnia, USA Abstract The cmplexity f EDA (Electrnic Design Autmatin) sftware is rapidly increasing t keep pace with Mre's law, technlgy advances, and time-t-market pressure impsed by the fast-pace cnsumer electrnic marketplace. T be relevant in the value-chain f the electrnic industry, nt nly EDA sftware needs t prvide the right set f features, but it must als deliver them in a timely manner t integrate in the design flw f cnsumer electrnics. This paper describes a prven test strategy in use at Synpsys t achieve predictable quality and release by skillfully using cde develpment tls and verificatin methds thrughut the develpment cycle. The descriptin ffers insights as hw turnarund time, cst-f-testing, and high standard thrughut sftware develpment help shape the strategy fr high quality releases. 1. Intrductin The EDA (Electrnic Design Autmatin) sftware industry is an integral part f the value-chain enabling the electrnic cnsumer market, prviding the design engineers the tls they need fr the develpment f Integrated Circuit (IC). The cnsumer market is cyclical and meeting these cycles ften means success r failure fr a prduct r a cmpany. With an increasingly shrt prduct develpment cycle, there is little rm fr iteratin t ensure success in a cmpetitive envirnment. In this cntext, the EDA sftware must meet custmer expectatins fr functinality, perfrmance and quality, as well as satisfy predictability f the release t meet needs f the prduct launch. In additin, the EDA sftware must: supprt latest technlgical breakthrughs, harness the increased cmplexity f the chip design, cntrl the very high cst f prductin, and meet the time-t-market bjectives. This requires executing quickly n the latest requirements while keeping stability high and predictable delivery dates. This article describes an effective strategy, in use by Synpsys, fr testing f EDA sftware. The dcument details the apprach, the criteria used t define the strategy, the principles guiding its implementatin and executin, and finally ffers cncrete actins taken and the results achieved. 2. Defect Preventin and Remval are the Gals The Test Strategy is nly ne part f the verall Quality Prgram cntributing in meeting the bjectives f functinality, perfrmance, quality, and timely availability. Synpsys s Quality prgram is anchred by three vectrs: Quality by Design: Develping high-quality sftware. Fcus n design-in quality methd and cntinuus prcess imprvement based n Best Practices fr sftware develpment. Quality by Validatin: Detect and fix defects befre release. Anchred by test methdlgies and cde analysis tls. Quality by Defect Management: Timely reslutin f custmer reprted defects.

2 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 This article fcuses n Quality by Validatin, mre specifically fr the prtin f the develpment life cycle starting nce the cde is written. We, first, explain the business reasns supprting effrt in ptimizing testing, secnd, we describe the Test Strategy Matrix used t select the mst efficient test methds, and finally, share Synpsys results achieved in the past few years based n this apprach. 2.1 Managing Quality by Validatin Even tday, despite the industry s best effrt t design-in quality in early stage f develpment with better requirement definitin, design reviews, etc.; the number f defects present in the cde is still significant. Industry studies have shwn that, even after cmpletin f unit testing, 10 t 20 defects/kloc (Lines f Cde in thusands) are still present in the cde as we enter the integratin and system test phases [1]. Findings these defects is a cnstant challenge especially with applicatins reaching 1 millin LOC; this means 10,000 defects. There is a cnsiderable amunt f resurces, capital and human resurces, dedicated t the verificatin and validatin f sftware prducts. Any gain in test prductivity will directly benefit the whle rganizatin by reducing cst f testing. represents a significant prtin f the develpment effrt: Abut 40-50% f the schedule is dedicated t ne frm r anther f testing. Althugh, a sund test apprach and autmatin will ptimize the effrt dedicated t the activity, the scpe f testing is challenged by a cnstant increase in cde cmplexity and the expanding number f cnfiguratins and platfrms t supprt. ffers best measure f frward-lking Quality: By taking a clse lk at the prduct develpment life cycle, we quickly discver that the testing activities prvide us with mst f the infrmatin n the quality f the prduct release. In fact, very little infrmatin prir t the first cycle f testing can help quantify the quality f the release. Althugh best practices prevent defect insertins, it des nt quantify prduct quality. Predictive mdels can be used but they are estimatin f quality, nt an actual measurement f it. cntributes t cst reductin: Well-established industry studies have shwn that the cst f fixing defects fund during pst-release is 5 times higher than when fund during design [2]. Anther fact is that, typically, it takes n average 10 t 20 hurs t find ne defect and t fix it [3]. A quick multiplicatin by a factr f 5 fr defect fund pst-release makes a cnvincing argument fr early defect detectin. is Time-Bund: As highlighted earlier, the quality challenge is time-bund as features must be available n time t satisfy the time-t-market requirement frm custmers. Withut the time cnstraint, the verall apprach t the quality wuld be different and the effrt wuld be shaped t ptimize against a different set f parameters. supprts Develpers: is als crucial as a mechanism t prvide quick feedback t the sftware develpers n his/her wn prgress during develpment. 2.2 Test Strategy Objectives The previus bservatins and cnsideratins fr testing can be summarized in Figure 1 shwing the typical Defect Finding Rate pattern bserved in the sftware industry. It illustrates the fact that a) the majrity f defects are fund in the last phase f the develpment cycle during testing, b) late fund defects increase the risk t the release, and c) testing represents a majr prtin f the develpment effrt. Test Strategy fr High Quality EDA Sftware Page 2 f 8

3 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Figure 1 - Defect Finding Rate Pattern Based n these facts, the bjectives f ur testing strategy are t: Detect defects early Use efficient detectin methds Minimize Cst Minimize Risk By having a disciplined apprach, selecting test methds with best return-n-investment, and ptimizing the test cycle; quality releases are achievable. The fllwing sectin describes the matrix fr selecting the test methdlgies. 3. Test Strategy Matrix Based n the defined bjectives fr ur test strategy, let s examine the varius testing methds used at Synpsys. Because mre than ne test methd will be necessary t cver the test dmain, and because each test methd has its respective advantages and limitatins, we characterize each testing methds by: Cnditins fr test initiatin, Cverage, Turnarund time, Cst-f-, Fix Cmplexity, and Risk f the Fix. The Test Matrix, in Table 1, prvides classificatin f the test methds fr the six characteristics. Test Strategy fr High Quality EDA Sftware Page 3 f 8

4 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Table 1 - Test Strategy Matrix Test Matrix Defect Reductin per Test Methd Early Detectin Efficient Detectin Minimize cst Minimize risk Cverage Methd Min. Cnditins fr testing Prbability f Detectin Turnarund Time Cst-f- Fix Cmplexity Risk fixing t the release Static Cde Analysis Cde Review Cde written 100% Few hurs Lw Lw Lw Cde written Partial Few hurs Lw Lw/Med Lw Regressin Cde integrated Partial Daily (typical) Lw/Med Lw/Med Med Dynamic Cde Analysis Cde integrated Partial = F x (regressin) Weekly Med Lw/Med Med Functinal Benchmark User Scenari Flw testing Cde integrated and regressed Cde integrated, regressed and stable High cde stability High cde stability with multiple prducts New features > 1 week Med Med/High Med/High Partial > 1 week High High High Limited > 2 weeks High High High Limited > 2 weeks High High High 3.1 Early Detectin shuld take place as sn as the cnditins are sufficient t prvide valuable data; nt when ne expects the results t be perfect. Fr example, fr Static Cde analysis, testing can start as sn as the cde is written and cmpiled. In the case f the User Scenari testing, the cde must be fully integrated, Test Strategy fr High Quality EDA Sftware Page 4 f 8

5 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 regressin tests passing, and feature testing cmplete befre testing can start. By initiating testing early n, we btain early feedback n the quality f the sftware and eliminate defects that might take hurs t find later. 3.2 Efficient Detectin Defect detectin is the ultimate bjective but it must be achieved efficiently t supprt iterative develpment wrk and predictable results. We lk at tw aspects f efficient defect detectin: High Cverage and Fast Turnarund Time. High Cverage All test methds serve a different purpse but they are nt all equal in cvering the test dmain. The cverage and the prbability f detectin ne test methd ffers cmpared t the ther ne drive the selectin f the test methd r tl. The best example f high cverage is the Static Cde Analysis where every line f cde is cvered. On the ther end, Dynamic Cde Analysis is a functin f the regressin testing cverage. Our test strategy favrs the test methd with high cverage which can be run early in develpment. Fast Turnarund Time (TAT) The bjective that we pursue here is t create a quick reslutin lp fr the develper. The Turnarund Time is crucial t ptimize the develper s prductivity in prviding a shrt find/fix/validate cycle t reslve defects. Anther factr favring fast TAT test methd is hw frequent measurements can be btained t track prgress against set targets. Test autmatin is instrumental here as it prvides efficiency, and timely metrics in supprt f decisin-making and crrective actins. 3.3 Minimize Cst As mentined previusly, the average time t find and t fix a defect is hurs [3]. Fixing defects is a cnsiderable cst just by itself but we als have t accunt fr the initial investment t create the tests and infrastructure t execute thse tests. The Cst-f- encmpasses the necessary effrt and resurces t define test cases, instrument the tests, create/maintain test harnesses, perfrm the tests, and validate the fixes. As applicatins becme mre cmplex, with a cnstantly grwing size f cde, the design f thse tests requires a substantial engineering effrt. The fllwing Figure 2 illustrates the Cst-f- fr the different test methds relative t the Turnarund Time required t perfrm these tests and clse the find/fix/validate lp fr a defect. Test Strategy fr High Quality EDA Sftware Page 5 f 8

6 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Figure 2 - Cst-f- Vs. Turnarund Time 3.4 Minimize Risk Fixing defects, even fr a small change in the cde, can be very risky. Cnsidering the Fix Cmplexity is nt sufficient; the risk t the release is als accunted in ur develpment practice. Fix cmplexity is linked t risk Sme f the tests will reveal flaw in the design, requiring fixes that can cnsiderably increase the risk t the release. Benchmark testing, fr instance, is mre likely t require mre fundamental changes t the cde than the nes revealed with Static Cde Analysis. The cmplexity f sme f the changes becmes a majr factr. Last minute cde re-engineering may pse serius delay t a release and may cmprmise its quality. Risk f applying the fix late in the release cycle The ther cnsideratin is the risk in making changes t the cde twards the tail-end f the release cycle. Gd practice in cde change management will gradually impse mre restrictive cnditins fr cde changes; as we get clser t the release date, the nature and number f changes being intrduced are carefully cntrlled, and limited. In reality thugh, the changes might still be required in rder t meet yur release bjective. This is why a disciplined apprach and early testing is crucial. 4. Actins lead t Results The strategy is nly as gd as the results it prduces. Over the past years, Synpsys tk specific actins t imprve sftware quality and the develpment infrastructure t meet the set bjectives: detect early, detect efficiently, minimize cst, and minimize risk. The Test Matrix allwed the rganizatin t identify any gaps in the test apprach. Practive actins were taken t cmplement and imprve ur test methdlgy, imprve the develpment infrastructure, and imprve test prcesses and executin. Test Strategy fr High Quality EDA Sftware Page 6 f 8

7 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Cmplement testing methdlgy t increase cverage and t test earlier in develpment Strengthen early detectin and imprve cverage by intrducing new tl fr Static Cde Analysis Prmte best practices fr Cde Review and increase review prductivity with cllabrative tl Imprve develpment infrastructure by prviding better set f tls and increase test autmatin Add testing cmpute resurces t imprve Turnarund Time fr test results Increase frequency f measurement fr Static and Dynamic Cde Analysis fr better prgress tracking Autmate benchmark testing Imprve test prcesses fr predictable release and quality Release management: Setting mre stringent gals, intermediary milestnes fr each metrics Plan and set measurement pints t gain better visibility in the state f the release Autmated measurements/cllectin/reprting f key metrics in supprt f release tracking Maintain high standard f cde integratin and regressin right frm the start f the release in supprt f early benchmark and flw testing All these effrts have made a difference as shwn bth, internally, and externally, directly benefiting ur custmers. Over the curse f the past few years the changes have cntributed in many ways: Internally: Reduced number f last minutes issues putting releases at risk Increased visibility f intermediate results thrughut the develpment cycle t better manage and react t changes Increased prductivity fr develpers: An internal survey revealed that mre than 70% recgnize that the effrt had a psitive impact n their prductivity Imprved Turnarund Time n daily build and regressin testing by a factr f 3 in the last 3 years Imprved Turnarund Time n lcal (develper sandbx) build and regressin testing by a factr f 2 in the last 3 years Frm a custmer pint-f-view: Reduced Incming Defect Rate frm release t release Reduced Defect Backlg by 35% ver the past 4 years Reduced Median Age f custmer defects by 34% Test Strategy fr High Quality EDA Sftware Page 7 f 8

8 5th Wrld Cngress fr Sftware Quality Shanghai, China Nvember 2011 Allwed Synpsys predictable releases t scale with increasing number f prducts Custmer satisfactin surveys f the past 2 years shw that 75% f Synpsys custmers indicate quality is as gd r better than previus year 5. Summary Custmer expectatins dictate ur apprach t sftware develpment. The methds and the sftware develpment tls used are carefully chsen and tailred t meet thse expectatins. In understanding the advantages and limitatins f each the Methds, we have prven that predictable quality is attainable. The tls and methdlgies are effective because they are framed by a structured apprach t develpment. Setting gals, frequent measuring pints, and planning fr test activities are essential in achieving predictable quality. The reward cmes frm the fact that specific actins have led t significant results fr quality imprvement and n-time delivery. 6. References [1] Watts S. Humphrey, A Discipline fr Sftware Engineering, Addisn Wesley, 1996 [2] Sni Mukesh, Defect Preventin: Reducing Csts and Enhancing Quality, citing IBM Systems Sciences Institute [3] Watts Humphrey, Learning frm Hardware: Design and Quality, Acknwledgments Sincere thanks t Pramd Suratkar, Synpsys Quality Analyst, fr his cnstructive feedback. Test Strategy fr High Quality EDA Sftware Page 8 f 8