Quality Assurance. Pragmatic Programmer Tip Think about Your Work

Size: px

Start display at page:

Download "Quality Assurance. Pragmatic Programmer Tip Think about Your Work"

Janel Marshall
7 years ago
Views:

1 Quality Assurance Pragmatic Programmer Tip Think about Your Work Turn off the autopilot and take control. Constantly critique and appraise your work.

2 Readings Required: Rapid Development, McConnell, Chapter 4, Section 4.3 QA Fundamentals Pragmatic Programmer, Hunt and Thomas, Chapter 8, p Ruthless Testing Other good stuff: If you didn t test it, it doesn t work, Coldwell (link on class Resource list)

3 Outline QA basics getting off to a good start Test o What makes a good tester? o What types of testing can we do? o Can tools help? Bugs! Test material adapted from several talks by Ian King, tester at Microsoft for many years.

4 What does QA mean to you? Your thoughts? Things we can do to ensure we produce a high quality (low defect) product Two main approaches to QA: o Process: Build in quality from the start o Test: Add quality through removing bugs

5 What are ways to foster quality from the get go? Over to you again 3 ideas Some practices: Good design and planning Coding style guides Code reviews/walkthroughs Pair programming

6 Tools can help! Configuration Management o Document changes, enforce good practices CVS Memory behavior o Detects corruption and leaks Performance o Time of routines Gnu gprof

7 Interesting fact Up to 4x the normal number of defects are reported for released products that were developed under excessive schedule pressure (Jones, 94) Why?

8 Test the most common QA practice Verify: Did we build the system right? Validate: Did we build the right system?

9 What makes a good tester? Analytical o Ask the right questions o Develop experiments to get answers Methodical o Follow experimental procedures precisely o Document observed behaviors, their precursors and environment Brutally honest o You can t argue with the data

10 How do test engineers fail? Desire to make it work o Impartial judge, not handyman Trust in opinion or expertise o Trust no one the truth (data) is in there Failure to follow defined test procedure o How did we get here? Failure to document the data Failure to believe the data

11 What types of testing can we do? Back to you to generate some ideas! Functional (include boundaries) Performance Security Stress Resource exhaustion, errors, and recovery Reliability/availability Usability (is it obvious)

12 Some testing jargon Black box testing Treats the system as atomic Best simulates the customer experience White box testing Examines the system internals Trace data flow directly (ie, in the debugger Bug report contains more detail on source of defect May obscure timing problems (race conditions)

13 It s black and white, right?! In black box, the tests are usually intended to cover the space of behavior In white box, the tests are usually intended to cover the space of parts of the program Often tester driven Often developer driven

14 How do we design good tests? Well-defined inputs and outputs o Consider environment as inputs o Consider side effects as outputs Clearly defined initial conditions Clearly described expected behavior Specific small granularity provides greater precision in analysis Test must be at least as verifiable as feature

15 and good test cases? Valid cases - What should work? Invalid cases What shouldn t? Boundary conditions Error situations ie, resource exhaustion

16 What s the trick to running tests? Manual Runs Tests that require direct human intervention with the system Necessary when: o GUI is present o Behavior is dependent on physical activity Advisable when: o Automation is more complex system being tested! o System is changing rapidly (early development)

17 Automated Testing Tests that can be executed independent of human interaction Good: replaces manual testing Better: performs tests difficult for manual testing (e.g. timing related issues) Best: enables other types of testing (regression, perf, stress, lifetime) Cost: Time investment to link tests into harness

18 Tools rock! Example Test Harness Tests are programmed as modules, then run by harness Harness provides control and reporting Example Code Coverage Freescale CodeTEST Software Analysis Tools

19 Pragmatic Programmer Tips Test early, test often, test automatically Coding ain t done til all the tests run Find bugs once

20 BUGS! Managing Bugs

21 What is a bug? Formally, a software defect System fails to perform to spec System causes something else to fail System functions, but does not satisfy usability criteria If the system works to spec and someone wants it changed, that s a feature request

22 What makes a good bug report? Include: Reproducible steps how did you cause the failure? Observed result what did it do? Expected result what should it have done? Any collateral information: return values/output, Environment o OS version, env variables, compiler flags, o Test platforms must be reproducible o It doesn t do it on my machine

23 Generally have a form to help

24 And a database to file it in Why is a bug db useful? Don t lose bugs Can pass them around easily Justify resources Justify your work Help indicate when the product ready for release Based on the number and type of bugs Based on a graph of the rate of bugs occurring Enables all sorts of cool metrics that management (and customers!) likes!

25 SPR rate 30 Open Closed Backlog TIME

26 Classifying bugs Cray links severity with priority. Some keep it separate.

27 Another classification Severity o Sev 1: crash, hang, data o o o loss Sev 2: blocks feature, no workaround Sev 3: blocks feature, workaround available Sev 4: trivial (e.g. cosmetic) Priority o o o o Pri 1: Fix immediately Pri 2: Fix before next release outside team Pri 3: Fix before ship Pri 4: Fix if nothing better to do

28 A Bug s Life (idealistic)

29 Regression Testing 1. Ensure that the bug you just fixed, doesn t reappear later though other mods o Add a test case to your suite for it! 2. Ensure that the fix you just added doesn t break things working previously o Rerun the test suite before checkin

30 When can I ship? Tune in on Monday!

FSW QA Testing Levels Definitions

FSW QA Testing Levels Definitions 1. Overview This document is used to help determine the amount and quality of testing (or its scope) that is planned for or has been performed on a project. This analysis