Translating Business Risks into a Risk-Based Test Plan

Transcription

1 Translating Risks into a Risk-Based Test Plan Ruud Teunissen Polteq IT Services BV The Netherlands Vught April 21 st 2006 slide nr. 1 slide nr. 2

2 Risks Regulatory Customer Satisfaction Financial Operational slide nr. 3 Risks Consequences of a failing product / system - Loose business (opportunities) - Too little, too late! - Loose customers - Break the law - Cost of repair - Image loss, bad publicity! - Ask the business Organise a workshop slide nr. 4

3 Prioritize Risk Risk 1. Users are able to update all information 2. Incorrect inventory information leads to no sale 3. Unable to print invoice overview 4. Invoices are incorrect 5. Idle time due to low performance 6. Unable to add new orders 7. Interfacing with other systems is impossible 8. Unable to update client information 9. New user interface is difficult to understand 10. slide nr. 5 Divide the system into testable parts and define priorities Sub-Systems, (User) Functions, Transactions, Processes, 1 2 Complete system Product slide nr. 6

4 s slide nr. 7 versus s Risk 2. Incorrect inventory information leads to no sale 4. Invoices are incorrect 6. Unable to add new orders 8. Unable to update client information 3. Unable to print invoice overview 7. Interfacing with other systems is impossible 1. Users are able to update all information 5. Idle time due to low performance 9. New user interface is difficult to understand slide nr. 8

5 versus Product Risk Product System slide nr. 9 versus Product Product System slide nr. 10

6 versus Product Product slide nr. 11 versus Product Product System slide nr. 12

7 Risk Analysis Product Risks Probability (In)direct loss Intensity Chance of error slide nr. 13 Probability Chance of Error - Did they perform a Unit and Integration Test? - Time pressure! Deadlines - Fresh and ambitious, yet inexperienced developers - New tools - No or little Requirements - Performance was not a design issue! - Intensity - The more usage / exposure, the higher the probability that it will go wrong! the higher the impact slide nr. 14

8 versus Probability Test Probability slide nr. 15 Test Probability Test System slide nr. 16

9 F u n c t i o n a l i t e i t C o n t r o l e e r b a a r h e i d F u n c t i o n a l i t e i t C o n t r o l e e r b a a r h e i d B e v e i l i g i n g C o n t r o l e e r b a a r h e i d F u n c t i o n a l i t e i t G e b r u i k e r s v r. h e i d, I n p a s b a a r h e i d, P e r f o r m a n c e, Z u i n i g h e i d B e v e i l i g i n g B r u i k b a a r h e i d G e b r u i k e r s v r. h e i d I n p a s b a a r h e i d Translating Risks into a Risk-Based Test Plan Test Design Techniques coverage - formal or informal - the principle by which the test cases are derived type of functionality to be tested test level (white-box or black-box) quality characteristic(s) to be tested required test basis T M a p - t e c h n i e k W h i t e - b o x ( W B ) o f T e s t b a s i s P r i n c i p e v a n a f l e i d e n ( m e t B l a c k - b o x ( B B ), d i e p g a n g ) : F o r m e e l o f V e r w e r k i n g s l o g i c a, I n f o r m e e l E q u i v a l e n t i e k l a s s e n, O p e r a t i o n e e l g e b r u i k, C R U D, required skills O v e r i g A l g o r i t m e t e s t W B, f o r m e e l I n t e r n e s t r u c t u u r, V e r w e r k i n g s l o g i c a : d e c i s i o n slide nr. 17 B e s l i s s i n g s - t a b e l l e n t e s t b i j v o o r b e e l d p r o g r a m m a - c o d e o f t e c h n i s c h o n t w e r p W B é n B B, f o r m e e l B e s l i s s i n g s - t a b e l l e n, z o w e l i n t e r n e s t r u c t u u r a l s ( f u n c t i o n e l e ) s p e c i f i c a t i e s D a t a f l o w t e s t B B, i n f o r m e e l F u n c t i o n e l e s p e c i f i c a t i e s E l e m e n t a i r e W B é n B B, f o r m e e l I n t e r n e s t r u c t u u r v e r g e l i j k i n g e n t e s t ( W B ) o f f o r m e l e f u n c t i o n e l e s p e c i f i c a t i e s ( B B ), b i j v o o r b e e l d p s e u d o - c o d e o f g e s t r u c t u r e e r d N e d e r l a n d s E r r o r g u e s s i n g B B, i n f o r m e e l A l l e s o o r t e n t e s t b a s i s G e g e v e n s c y c l u s - B B, i n f o r m e e l F u n c t i o n e l e t e s t s p e c i f i c a t i e s P r o c e s c y c l u s t e s t B B, f o r m e e l A d m i n i s t r a t i e v e O r g a n i s a t i e ( A O ) p r o c e d u r e s c o v e r a g e i n c o m b i n a t i e m e t p a t h c o v e r a g e, n a a r k e u z e t e v e r z w a r e n m e t t e s t m a a t 2 o f h o g e r V e r w e r k i n g s l o g i c a : d e c i s i o n c o v e r a g e i n c o m b i n a t i e m e t p a t h c o v e r a g e, n a a r k e u z e t e v e r z w a r e n m e t d e c i s i o n / c o n d i t i o n c o v e r a g e e n / o f m e t e e n h o g e r e t e s t m a a t E q u i v a l e n t i e k l a s s e n V e r w e r k i n g s l o g i c a : m o d i f i e d d e c i s i o n / c o n d i t i o n c o v e r a g e O v e r i g : o p b a s i s v a n v e r m o e d e n s w a a r d e f o u t e n z i t t e n C R U D : o p b a s i s v a n d e l e v e n s c y c l u s v a n g e g e v e n s V e r w e r k i n g s l o g i c a : d e c i s i o n c o v e r a g e, s t a n d a a r d m e t t e s t m a a t 2, m a a r n a a r k e u z e i s d e z e t e s t m a a t t e v e r l i c h t e n o f t e v e r z w a r e n K w a l. a t t r i b u t e n T o e p a s s i n g s g e b i e d e n F u n c t i o n a l i t e i t V e r w e r k i n g F u n c t i o n a l i t e i t C o m p l e x e v e r w e r k i n g V e r w e r k i n g, i n t e g r a t i e t u s s e n f u n c t i e s e n g e g e v e n s C o m p l e x e v e r w e r k i n g A l l e F u n c t i o n a l i t e i t I n t e g r a t i e t u s s e n f u n c t i e s e n g e g e v e n s I n t e g r a t i e t u s s e n d e a d m i n i s t r a t i e v e o r g a n i s a t i e ( A O ) e n h e t s y s t e e m Test Design Techniques (extract) Functionality Processing Screen checks DCoT Error Guessing Error guessing Error Guessing Error Guessing and SUMI DCoT Sample SEM Error guessing The thoroughness of the RLT is variable and will thus be determined by the rating available as a consequence. and the amount of hours that comes SEM sample user profiles Use Cases or PCT and SUMI ECT DCoT Sample both SEM and SYN SEM user profiles Use Cases or PCT and SUMI ECT SEM and sample SYN SEM user profiles and overall system PCT and Laboratory Test DCoT Data Combination Test SYN Syntax Test ECT Elementary Comparison Test RLT Real Life Test SEM Semantic Test PCT Process Cycle Test SUMI Software Measurement slide nr. 18

10 Test Strategy Test Test Design Technique(s) ECT SEM and sample SYN PCT and SUMI System RLT ECT, DCoT Sample SEM Error guessing DCoT Sample SEM Error guessing PCT SEM, user profiles slide nr. 19 Test Effort Estimation Size Strategy Productivity Estimation Method Estimated test effort slide nr. 20

11 Test Effort Estimation Size Test Productivity Effort Estimate S1 S2 System Estimated Effort S3 S 400 hrs slide nr. 21 An Iterative Process thus far Regulatory Customer Satisfaction Financial Operational slide nr. 22

12 Example : Test Progress Test Design 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% System slide nr. 23 Prioritize test activities in such a way that whenever you have to stop, you have done the best testing in the time available slide nr. 24

13 Example: Defects - Status New Open Rejected Retest Solved Total slide nr. 25 Acceptance Criteria P1 P2 P3 QC 1 QC 2 QC 3 QC 4 Perc. Exec. 100% 100% 75% 50% Total Pending Defects Total slide nr. 26

14 Pending Defects versus System Total Perc. Exec. 60% 0% 20% 20% 80% 100% 0% 32% Total Pending Defects slide nr. 27 Example : Test Progress Test Execution 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% P1 P2 P3 P4 P5 slide nr. 28

15 Ruud Teunissen Polteq IT Services B.V. The Netherlands Orlando - May 19 th 2005 slide nr. 29 Ruud Teunissen Polteq IT Services B.V. The Netherlands slide nr. 30