Testing the Key to an Efficient and Predictable Software Engineering Process

Transcription

1 Testing the Key to an Efficient and Predictable Software Engineering Process Dr. Tom Sprenger CIO AdNovum Informatik Dipl. Informatik-Ing. ETH May, 28 th ETH Zürich AdNovum = Engineering Quality Core Competence High-End Software & Security Engineering Customers Financial services, Banks, Private banks, Swiss Post, Government, and Insurance Key Data Est employees, 70% graduate engineers Zurich (HQ), Bern, Budapest (Hungary) Strategic Partners Sun, BEA, Swiss Post, HP, IBM, Oracle, SafeNet, Siemens Switzerland, Vasco 2 (c) AdNovum Informatik AG 1

2 Agenda Starting Position Motivation Challenges in Practice Quality Assurance System Constructive Measures SW-Engineering Process Technology Management Risk Analysis Testing Toolbox Analytical Measures Automatic Testing Continuous Testing (NightlyBuild) Q&A 3 Starting Position 4 (c) AdNovum Informatik AG 2

3 Motivation Improving quality of the results while reducing risk Improving efficiency while keeping high quality Manage distributed development Near-/Off-shoring Provide periodical information points for project management Controlling Provide measurements for assessing improvement efforts Well established Testing/QAE is a differentiation in the market 5 Challenges in Practice Time/Budget vs. Quality/Testing Functionality vs. Quality/Testing Testing/QA must be part of the offer and the development process! Problem of testing today: 80% of the time is used to build up a suitable test environment; only 20% of the time remains to actually write and execute tests Establish an efficient testing infrastructure and make use of templates and blueprints Tests must be 100% repeatable and reproducible Prerequisite for efficient testing is a well defined software engineering process 6 (c) AdNovum Informatik AG 3

4 Quality Assurance System 7 Constructive Measures: SW-Eng Process The core process 1. Analysis & design 2. Development / implementation 3. Test & integration 4. Manage & optimize Each phase defines Roles & tasks Deliverables & checkpoints 8 (c) AdNovum Informatik AG 4

5 Constructive Measures: SW-Eng Process What about a Quality Assurance Process? There is NO such thing as a QA process Avoiding the risk of working for the QA process (instead of SW-Eng Process) Only process is the Software Engineering Process We follow an Aspect-oriented QA-Approach" define Q-Aspects along the Software Engineering Process keep the focus on producing software enable the developers for QA QA is an integral part of engineering software Each Q-Aspect has one or more Q-Metrics attached Q-Metrics can be verified automatically 9 Aspect-Oriented QA-Engineering Q-Aspekt: A&S Sign-off Q-Aspekt: Risk Assessment Q-Aspekt: Code Metrics Q-Aspekt: Test Coverage Q-Aspekt: A&S Sign-off 10 (c) AdNovum Informatik AG 5

6 Constructive Measures: Tech. Management Software tool set 3 levels Strategic Tactical Operative Central information system Explicitly show dependencies Development from multiple locations Expert know-how Ability to anticipate Cost control Condition: automation 11 Constructive Measures: Risk Analysis Matrix 12 (c) AdNovum Informatik AG 6

7 Constructive Measures: Testing Toolbox I Test Type Tool Description Source Code Metrics PMD Checks Java code against a defined set of rules to detect common errors. Functional Testing JDepend JUnit Cactus JTestCase Generates design quality metrics for each Java package to measure the quality of a design in terms of its extensibility, reusability, and maintainability. Tests units of Java code. Simple test framework for unit testing server-side Java code (Servlets, EJBs,...). Enables a systematic separation of test code and test data. 13 Constructive Measures: Testing Toolbox II Integration Testing HttpUnit Emulates basic browser behavior to test form submission, JavaScript, basic HTTP auth., etc. Selenium Used for Ajax web sites or extensive JavaScript functionality. Code Coverage Emma Discovers which parts of the code have not been executed during the tests. Load Testing Sams / Grinder Proxy Sniffer Tests your system with heavy load, re-using your integration tests. Load generator that supports customized load and stress testing of web applications. Profiling JProbe Detects performance problems and memory leaks in Java code. 14 (c) AdNovum Informatik AG 7

8 Analytical Measures: Automatic Testing I Automatic Testing is your friend! Prerequisite Homogenous environment over all projects in your company Environment supports script-based execution (no user interaction required) Make consistent use of versioning (source and 3rd-party components) These are the minimum requirements for reliable, reproducible builds Defined project structure Defined set of mandatory build targets (e.g. init, build, test, etc.) with Well-known semantics Well-known result/output 15 Analytical Measures: Automatic Testing II (Automatic) testing starts with the specification Risk Analysis (minimize effort, maximize benefits) Use Cases Requirements (services, test data, etc.) Make use of Testing Toolbox (JUnit, HttpUnit, Selenium, Grinder,...) Distinguish different aspects: functional, performance, load testing Use clever approach to enable reuse of functional tests for load testing Distinguish between test scenario and test cases test scenario test code test case #test data Separate test data from test code and enable reuse of test code Define clear and reasonable quality and testing goals! 16 (c) AdNovum Informatik AG 8

9 Analytical Measures: NightlyBuild I Continuous Integration Testing Framework Builds all our active projects every 24 hours Repository checkout of the project head (latest version) Verification of source code quality (code-style metrics, pattern detection, etc.) Complete build of the entire project Start-up of test environment Project deployment Execution of tests (e.g. source quality, unit, integration, and load tests) Pull-down of test environment Reporting Statistics: Number of Builds: 292 Number of Jobs: > 5000 Number of Platforms: 15 Number of Hosts: Analytical Measures: NightlyBuild II 18 (c) AdNovum Informatik AG 9

10 Analytical Measures: NightlyBuild III : : 19 Analytical Measures: NightlyBuild IV : : 20 (c) AdNovum Informatik AG 10

11 Analytical Measures: Application Web Display project dependencies as a graph Investigate critical project dependencies Get information about Software versions in use Unused versions Toolbox information 21 Quality Assurance Engineering Team Project Support Testing opportunities consulting Setup of testing infrastructure OpenSource Tool Productizing JDepend integration Emma extensions HttpUnit & Selenium extensions Meta-Software-Engineering Autom. architecture validation Autom. risk-based code coverage AQUARIUS - Quality and Risk Unit Sensing Customer Projects 22 (c) AdNovum Informatik AG 11

12 Questions? 23 (c) AdNovum Informatik AG 12