Everything I know about Testing I learned from the Scientific Method Paul

Transcription

1 Everything I know about Testing I learned from the Scientific Method Paul Clydesdale Bank plc

2 Me! Paul Coyne Test Architect Working in testing since 1989 Mainly in finance & utilities Currently at Clydesdale Bank Failed zoologist!

3 Life after zoology Postgrad in Systems Analysis & Design SSADM Programming Business Analysis and, of course, Testing Sorry, did I say testing? That s obviously a mistake

4 How I learned to Test The situation was I was the new wildly enthusiastic programmer on the block I was going to revolutionise COBOL programming (oh yeah!) My first assignment was The Test Team? You re kidding, right? What did I do wrong???

5 How I learned how to Test

6 So what is the Scientific Method? Science is not. A body of knowledge Expert opinion or Textbooks Tools and lab techniques Revealed wisdom Unchangeable

7 The Scientific Method is A way of thinking A way of investigating A means to ask (the right) questions A way to learn from the answers to those questions Even if the answer is not what you expected! In all cases the simplest explanations are preferred Until they are discounted

8 Observe, Imagine, Query Start with an observation leads to an hypothesis It should be consistent enough with what is known to be considered reasonable But propose something new that is worth finding out about

9 A Good Hypothesis Hypotheses must be able to be proven to be false and should make specific, testable predictions Otherwise they re just speculation! There s an invisible dragon in my garage which only I can see One Black Swan is all it takes, despite a virtual infinity of white ones.

10 Science is Active Observation Science is based on real world observations It is empirical It relies on those observations being repeatable Transparency of methods is paramount Depends on accurate and timely recording of results and publication of ALL outcomes No hiding unfortunate results

11 So it s a method, then? More like a set of methods A way of thinking and doing A systematic and logical approach to discovering how things work Rooted in rigorous observation under controlled circumstances Is any of this sounding familiar?

12 And what about experiments? Well, we have a hypothesis which makes testable predictions which we know we can (in theory) prove wrong So that s what we set out to do

13 Good Experiments are Controlled changing only one thing at a time (in so far as is possible) Repeatable Clear methods, well stated, outcomes recorded Based on a model of the experimental subject Of necessity less detailed than reality Blinded (single, double)

14 Correlation is not Causation

15 Experimental failures Failure in an experiment is not the failure to get the expected result It s a methodological failure.. To control for bias or variables To report accurately and dispassionately With conclusions not supported by the evidence The press rarely get it right either!

16 Hold on what s this got to do with me? Good software testing has a lot in common with the Scientific Method Testable predictions Control of variables Repeatability / Reporting of Methods Discovery of new information

17 Good Testers are into SM Rigour in thought processes Scepticism & critical thinking Seeking the negative outcome Clear methods, stated in advance Concise fact based reporting Ability to repeat All results are valid* * provided you have been rigorous in your methods and honest in your reporting

18 Classic Mistakes in Science Testing too.. Owning quality Trying to prove it works Poor planning non methodical Inadequate controls Trusting in documentation Mistaking quantity for quality Ambiguous reporting

19 Lessons for Testing 1 1. Do not seek to confirm, but to disprove. You re looking for bugs 2. Passes and Fails are not symmetrical. A test that has passed is weak evidence. A test that has failed is much stronger evidence. 3. Design your tests to eliminate uncontrolled variables or to limit their effects. 4. Observe carefully, record dispassionately and be aware of your own biases and preconceptions. Keep good records. 5. Understand that correlation and causation are not the same consider what other plausible mechanisms might explain what you observed.

20 Lessons for Testing 2 6. Have clear goals. Understand what is valuable in your context and pursue that. 7. Follow your nose, but don t lose your way. Someone has to pay for your time! 8. All prior knowledge is provisional until disproven. However all observed facts are true, provided your methods are robust. 9. Be audacious in what you do. But be humble in your own knowledge and interpretation. Let the facts speak. 10. Challenge authority (diplomatically). Use other people s skills wisely, but don t rely on their truths.

21 Testers are Scientists Software Development is NOT Engineering Otherwise it would work better! Software Testing is Experimentation Testers are methodical and systematic Testers control for variability in the system Testers record and report all results without fear or favour We re really Scientists at heart

22 Just don t end up like these zoologists.

23 Thanks