Pareon Verify. Test suite for dynamic software verification. Detect the hard-to-find errors other tools can t

Transcription

1 Pareon Verify Test suite for dynamic software verification Detect the hard-to-find errors other tools can t We are going to definitely incorporate the Vector Fabrics tools as an important part of our quality offering Copyright 2016 Vector Fabrics B.V. Jan Asselman Quality team leader / Altran Intelligent Systems

2 Avoid software errors and high repair costs with Pareon Verify Test suite for dynamic software verification finds difficult-to-detect software errors and makes lengthy debugging sessions a thing of the past. Software in embedded systems has become fundamental to the daily infrastructure of our economy, infrastructure, and daily life. And, the amount of embedded software whether in cell phones, automobiles, airplanes or medical devices has exploded in size and complexity. The result: an exponential growth in the potential and number of software errors or bugs. Cost of an error A NASA study 1 shows the cost of fixing a software error found in the field as 100x more expensive than finding the same error during development. The common use of dynamic memory allocation and multithreading in C++ triggers a new category of errors that are hard to detect using the traditional testing tools and methods. Errors are expensive Repairing software errors once the product has released or is in the field involves significantly higher costs, even without accounting for damage to the brand and corporate image. Development Testing In the field The cost of a software defect error mounts exponentially the longer it takes to identify and fix. To contain cost and financial risk, it is crucial to detect and correct errors during development - before the product enters the market. 1: It is more important than ever to effectively and thoroughly test software before it is shipped. 2 Copyright 2016 Vector Fabrics B.V.

3 Car Sofware Average high-end car F-35 Fighter Jet 2013 Boeing 787 Total flight software *million lines of code 19,4 21,4 8,5 30,1 20,6 % Single processor % Multi processor % Multi core % Multi core & Multi processor % Don t know Today s code includes millions of lines of code and increased use of multicore and multiprocessor systems has magnified complexity and introduced timing challenges. The result: more software errors that can only be found through dynamic analysis. 100 million lines of complex code Today s automobiles and airplanes commonly depend on a plethora of processors some of them multicore to run over 100 million lines of code. And, with the adoption of multicore devices, code complexity has reached new heights. As the amount and complexity of code increases, so does the number of coding errors. Worse, finding those errors is infinitely more challenging in a multithreaded, multicore environment, which brings new challenges to software correctness and verification. Increasingly dynamic behavior Modern software testers must cope with the everincreasing dynamic behavior of the software. Dynamic memory use and multithreading, for example, is far harder to test as the bugs are hard to reproduce, but their consequences buffer overflows, memory leaks, and invalid memory accesses lead to significant system failures. Heisenbugs The criticality of system timing has bred a new category of software error called Heisenbug so named after the uncertainty principle discovered by Theoretical Physicist Werner Heisenberg. Heisenbugs are hard to diagnose because they tend to disappear because the debug process affects run-time performance enough that the bugs don t show up. Detecting and resolving Heisenbugs is therefore extremely complex and costly. Validating software without dynamic analysis is no longer effective. Software projects are growing in size and the hardware running the software is becoming increasingly sophisticated. Without dynamic analysis, the complex software errors remain undetected and cause problems later. Pareon Verify automates dynamic analysis Because of the new debug challenges, Vector Fabrics developed Pareon Verify a tool for dynamic software verification. Pareon Verify automates dynamic analysis of large volumes of source code in order to detect software errors that cannot be identified using other tools or methods. Increasing size and complexity of applications cause more errors to occur in software. Copyright 2016 Vector Fabrics B.V. 3

4 The advantages of dynamic code analysis Dynamic analysis or runtime analysis, as it s sometimes called can be seen as an MRI scanner for software. By looking inside the application while it executes, Pareon Verify detects errors in the internal behavior. By observing the application s internal structures and behavior, it identifies where errors originate and, in doing so, pinpoints the primary cause of the error. This way, dynamic analysis identifies errors that remain undetected during other tests. Limits of existing methods Errors in dynamic behavior are notoriously difficult to detect with traditional testing tools and methods. Such errors typically aren t triggered when the application is run in its normal testing environment, but cause problems when environment, timing, or circumstances change. Dynamic analysis looks at the entire application and is not limited to a local unit scope in its error detection. Additionally, as it looks inside the application to see errors in individual operations in a white-box fashion, it reports application errors even if that error does not cause the application to fail its black-box test on a particular test case. For these reasons, dynamic code analysis plays a critical role in comprehensive software testing. Dynamic analysis complements static code analysis and unit testing while offering the key advantage of revealing the most expensive errors that would otherwise remain undetected. Dynamic analysis Heisenbugs Static analysis is limited in its capability to predict the dynamic behavior of code and cannot foresee all possible combinations of inputs, code paths, dynamic memory, and thread timing of the full application. Cost of an error Unit & system tests Static analysis Coding style checks Functional errors Coding errors Coding style violations Similarly, unit testing is limited in its scope to a smaller part of the application. It cannot detect errors caused by the application s global behavior or environment. The dynamic analysis edge Dynamic analysis is not hindered by these limitations. It observes the application s actual operation during execution. It can see all the combinations of dynamic and global behavior triggered during execution, regardless of complexity. Compilation Syntax errors Dynamic analysis reveals the most expensive and allusive bugs in the software verification process. 4 Copyright 2016 Vector Fabrics B.V.

5 How it works? Dynamic analysis begins by instrumenting the target application to trace the behavior taking place inside the application. The instrumentation in the application binary creates a stream of events also called a trace that is captured by the analyzer and studied for behavioral errors. The analyzer then reports the errors in a clear and concise way with detailed references to source code for easy correction of the error. Pareon Verify detects a range of hard-to-detect software errors. When integrated into the development process, Pareon Verify can continuously screen for errors as software changes take place. Pareon Verify verifies software using the following steps: 1. Compile Pareon Verify s instrumenting C and C++ compiler becomes a drop-in replacement for the (cross) compiler and a build can be setup within minutes. 2. Execute Pareon Verify executes the instrumented code on a host PC or embedded target using representative input data or test suite. 3. Analyze The resulting execution trace is conveniently analyzed on the host PC to minimize the impact of analysis on target memory and performance. 4. Resolve bugs Detailed error messages quickly identify the problem and show developers how to fix the error, including variable names and full call stacks. No false positives! Invalid memory access Array out-of-bounds Uninitialized memory reads Use-after-free Race conditions Memory leaks Invalid pointer usage and buffer overflows due to bad pointer arithmetic, etc. Access outside a buffer or by using wrong indexes into a multidimensional array Use of variables that have not been written yet, generally leading to data corruption or crashes Use of dangling pointers that point to memory that no longer exists Access to the same data from multiple threads without proper synchronization, which generally leads to data corruption Improper memory management causing the system slow down or loss of memory Pareon Verify uncovers hard-to-detect software errors, saving developers tremendous time and cost. Copyright 2016 Vector Fabrics B.V. 5

6 Write 10? Breaking the multicore challenge 10 Read Even a cell phone includes up to eight cores these days, creating a tremendous software challenge for developers. So-called data races already possible in single core applications with interrupts have become much more widespread as multithreaded programs run parallel threads. Time Thread A Shared memory Normal operation: read value 10 A read and write from different threads to shared memory.? Thread B Data races, caused by two threads accessing the same variable without proper synchronization, result in threads reading stale or incorrect data when data corruption has occurred. Since multi-threaded programs have more dynamic behavior, they are more susceptible to dynamic software errors. With Pareon Verify, we are now able to detect and fix some really difficult bugs much earlier in development. It took us a few hours to fix the problem instead of a week or longer. Daniele Lacamera Lead developer / PicoTCP Read Write Time Thread A Shared memory Thread B Data race: read of uninitialized data A data race occurs when the timing changes. The read now takes place before the write is done, resulting in a read of uninitialized data. 6 Copyright 2016 Vector Fabrics B.V.

7 Solving errors early through continuous integration Many software development companies deploy a development process called continuous integration. When a developer checks in a new piece of code, a central server builds and runs a set of tests on all source code. If any of the tests fail, the newly submitted code is rejected and the developer has to fix it. This allows developers to find and fix errors earlier in the process while they re still working on the code and it is still fresh on their mind. This ensures that fixes are done faster. Pareon Verify delivers: Fewer errors in the field By capturing the errors before the product is deployed, companies can reduce maintenance costs of software and maintain a high level of customer satisfaction and brand reputation. An error found during testing is six times more expensive than one found during development and up to 100 times more expensive when found in the field. Using Pareon Verify, dynamic analysis can become part of the continuous integration process. Instead of just receiving a pass or fail from the automated tests, Pareon Verify also reports: if the code flow leading to the verdict is correct; if the test result is only correct by accident; if there are unwanted side effects. The benefits: the developer can correct the error immediately even before it triggers a test case to fail and a crash to happen, saving a lot of time and costs in the system development. To reduce maintenance costs, it is essential that errors be removed from a software design as early as possible. A well-known Japanese car manufacturer notably had to issue recalls for problems that were linked to software errors in a number of its models. These are estimated to have costed the company two billion dollars, not to mention the damage to its corporate image. Improved time to market Often, during the integration and system testing phase, the number of bugs found by the test team only grows. Developers cannot tell when they revisions are complete as issue might be hiding the next. As a result, project and release schedules tend to slip, delaying the release or the end of the project. With Pareon Verify, companies can dramatically slash the time needed to debug and correct errors. This in turn reduces the risk of project delays. In practice, this can result in weeks worth of time saved and a more predictable test and integration phase. Summary Pareon Verify helps companies build software that can be brought to market more quickly, with less cost, and with fewer errors By using Pareon Verify, software companies improve brand reputation and customer satisfaction. Pareon Verify enables its customers product success. For more information about Pareon Verify, please visit: Copyright 2016 Vector Fabrics B.V. 7

8 About Vector Fabrics Vector Fabrics specializes in development and testing tools for the complex software used in embedded systems, such as automotive, IOT devices, network equipment, consumer electronics, aerospace, medical and industrial systems. The Pareon family of dynamic analysis tools provides customers with deep insight into how their software will behave when executed. This insight ensures a higher level of reliability, safety, and performance. Using these tools, businesses can identify and fix costly software bugs and improve product time-to-market. Hogeweg 39A 5301 LJ Zaltbommel The Netherlands Phone info@vectorfabrics.com Copyright 2016 Vector Fabrics B.V. All Rights Reserved.