Insiue for Sofware Technology Qualiy Assurance in Sofware Developmen Qualiässicherung in der Sofwareenwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Insiu für Sofwareechnologie (IST) TU Graz Summer Term 2015 1 / 19
Insiue for Sofware Technology Agenda Muaion Tesing Model-based Muaion Tesing Tes-Case Generaion wih ioco checking 2 / 19
Insiue for Sofware Technology Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) ou of boundary access of large arrays due o ineger overflow 9 years undeeced 1 public saic 2 in binarysearch ( in [] a, in key ) 3 { 4 in low = 0; 5 in high = a. lengh - 1; 6 7 while ( low <= high ) { 8 in mid = ( low + high ) / 2; 9 in midval = a[ mid ]; 10 11 if ( midval < key ) 12 low = mid + 1; 13 else if ( midval > key ) 14 high = mid - 1; 15 else 16 reurn mid ; // key found 17 } 18 reurn -( low + 1); // key no found 19 } Beware of bugs in he above code; I have only proved i correc, no ried i. [Knuh77] 3 / 19
Insiue for Sofware Technology Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) ou of boundary access of large arrays due o ineger overflow 9 years undeeced Algorihm was proven correc! Programming Pearls [Benley86, Benley00] assuming infinie inegers :( 1 public saic 2 in binarysearch ( in [] a, in key ) 3 { 4 in low = 0; 5 in high = a. lengh - 1; 6 7 while ( low <= high ) { 8 in mid = ( low + high ) / 2; 9 in midval = a[ mid ]; 10 11 if ( midval < key ) 12 low = mid + 1; 13 else if ( midval > key ) 14 high = mid - 1; 15 else 16 reurn mid ; // key found 17 } 18 reurn -( low + 1); // key no found 19 } Beware of bugs in he above code; I have only proved i correc, no ried i. [Knuh77] 3 / 19
Insiue for Sofware Technology Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) ou of boundary access of large arrays due o ineger overflow 9 years undeeced Algorihm was proven correc! Programming Pearls [Benley86, Benley00] assuming infinie inegers :( 1 public saic 2 in binarysearch ( in [] a, in key ) 3 { 4 in low = 0; 5 in high = a. lengh - 1; 6 7 while ( low <= high ) { 8 in mid = ( low + high ) / 2; 9 in midval = a[ mid ]; 10 11 if ( midval < key ) 12 low = mid + 1; 13 else if ( midval > key ) 14 high = mid - 1; 15 else 16 reurn mid ; // key found 17 } 18 reurn -( low + 1); // key no found 19 } Beware of bugs in he above code; I have only proved i correc, no ried i. [Knuh77] 3 / 19
Insiue for Sofware Technology Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) ou of boundary access of large arrays due o ineger overflow 9 years undeeced Algorihm was proven correc! Programming Pearls [Benley86, Benley00] assuming infinie inegers :( 1 public saic 2 in binarysearch ( in [] a, in key ) 3 { 4 in low = 0; 5 in high = a. lengh - 1; 6 7 while ( low <= high ) { 8 in mid = ( low + high ) >>> 1; 9 in midval = a[ mid ]; 10 11 if ( midval < key ) 12 low = mid + 1; 13 else if ( midval > key ) 14 high = mid - 1; 15 else 16 reurn mid ; // key found 17 } 18 reurn -( low + 1); // key no found 19 } Beware of bugs in he above code; I have only proved i correc, no ried i. [Knuh77] 3 / 19
Insiue for Sofware Technology Observaions Verificaion failed (wrong assumpion) Esablished esing sraegies failed: saemen coverage branch coverage fails muliple condiion coverage MC/DC: sandard in avionics [DO-178B/ED109] Long array needed: in[] a = new in[ineger.max_value/2+2] Lesson Concenrae on possible fauls, no on srucure. Generae es cases covering hese fauls Muaion Tesing [Lipon71, Hamle77, DeMillo e al.78] 4 / 19
Insiue for Sofware Technology Observaions Verificaion failed (wrong assumpion) Esablished esing sraegies failed: saemen coverage branch coverage fails muliple condiion coverage MC/DC: sandard in avionics [DO-178B/ED109] Long array needed: in[] a = new in[ineger.max_value/2+2] Lesson Concenrae on possible fauls, no on srucure. Generae es cases covering hese fauls Muaion Tesing [Lipon71, Hamle77, DeMillo e al.78] 4 / 19
Insiue for Sofware Technology Wha Is Muaion Tesing? Originally: Technique o verify he qualiy of es cases There is a pressing need o address he, currenly unresolved, problem of es case generaion. [Jia&Harman11] 5 / 19
Insiue for Sofware Technology Wha Is Muaion Tesing? Originally: Technique o verify he qualiy of es cases There is a pressing need o address he, currenly unresolved, problem of es case generaion. [Jia&Harman11] 5 / 19
Insiue for Sofware Technology How Does I Work? Sep 1: Creae muans Muaion Process Source Code Muan Muaion Operaor 6 / 19
Insiue for Sofware Technology Example: Scala Program Kind of riangles: equilaeral isosceles scalene Creae muans muaion operaor == >= creaes 5 muans 1 objec riangle { 2 3 def riype (a : In, b : In, c: In ) = 4 (a,b,c) mach { 5 case _ if (a <= c-b) => " no riangle " 6 case _ if (a <= b-c) => " no riangle " 7 case _ if (b <= a-c) => " no riangle " 8 case _ if (a == b && b == c) => 9 " equilaeral " 10 case _ if (a == b) => " isosceles " 11 case _ if (b == c) => " isosceles " 12 case _ if (a == c) => " isosceles " 13 case _ => " scalene " 14 } 15 } Source code in Scala 7 / 19
Insiue for Sofware Technology Example: Scala Program Kind of riangles: equilaeral isosceles scalene Creae muans muaion operaor == >= creaes 5 muans 1 objec riangle { 2 3 def riype (a : In, b : In, c: In ) = 4 (a,b,c) mach { 5 case _ if (a <= c-b) => " no riangle " 6 case _ if (a <= b-c) => " no riangle " 7 case _ if (b <= a-c) => " no riangle " 8 case _ if (a >= b && b == c) => 9 " equilaeral " 10 case _ if (a == b) => " isosceles " 11 case _ if (b == c) => " isosceles " 12 case _ if (a == c) => " isosceles " 13 case _ => " scalene " 14 } 15 } Muan 7 / 19
Insiue for Sofware Technology Example: UML Model Car Alarm Sysem even-based conrollable evens observable evens Muae he model muaion operaor ClosedAndUnlocked 20 Open Unlock OpenAndUnlocked Close Lock AlarmSysem_SaeMachine Lock ClosedAndLocked OpenAndLocked Close Unlock Open Unlock Alarm Acivae Alarms /enry Deacivae Alarms /exi FlashAndSound 30 / Deacivae Sound Flash 17 muans Unlock Armed Show Armed /enry Show Unarmed /exi Close SilenAndOpen Open 300 Sae machine model in UML 8 / 19
Insiue for Sofware Technology Example: UML Model Car Alarm Sysem even-based conrollable evens observable evens Muae he model muaion operaor 17 muans Muaed UML model 8 / 19
Insiue for Sofware Technology How Does I Work? Sep 2: Try o kill muans A es case kills a muan if is run shows differen behaviour. 9 / 19
Insiue for Sofware Technology Example: Scala Program Muan survives pah coverage (MC/DC): riype(0,1,1) riype(1,0,1) riype(1,1,0) riype(1,1,1) riype(2,3,3) riype(3,2,3) riype(3,3,2) riype(2,3,4) Muan killed by riype(3,2,2) 1 objec riangle { 2 3 def riype (a : In, b : In, c: In ) = 4 (a,b,c) mach { 5 case _ if (a <= c-b) => " no riangle " 6 case _ if (a <= b-c) => " no riangle " 7 case _ if (b <= a-c) => " no riangle " 8 case _ if (a >= b && b == c) => 9 " equilaeral " 10 case _ if (a == b) => " isosceles " 11 case _ if (b == c) => " isosceles " 12 case _ if (a == c) => " isosceles " 13 case _ => " scalene " 14 } 15 } Muan 10 / 19
Insiue for Sofware Technology Example: Scala Program Muan survives pah coverage (MC/DC): riype(0,1,1) riype(1,0,1) riype(1,1,0) riype(1,1,1) riype(2,3,3) riype(3,2,3) riype(3,3,2) riype(2,3,4) Muan killed by riype(3,2,2) 1 objec riangle { 2 3 def riype (a : In, b : In, c: In ) = 4 (a,b,c) mach { 5 case _ if (a <= c-b) => " no riangle " 6 case _ if (a <= b-c) => " no riangle " 7 case _ if (b <= a-c) => " no riangle " 8 case _ if (a >= b && b == c) => 9 " equilaeral " 10 case _ if (a == b) => " isosceles " 11 case _ if (b == c) => " isosceles " 12 case _ if (a == c) => " isosceles " 13 case _ => " scalene " 14 } 15 } Muan 10 / 19
Insiue for Sofware Technology Example: UML Model Muan survives Killed by Lock(); Close(); Afer(20); funcion coverage sae coverage ransiion coverage Muaed UML model 11 / 19
Insiue for Sofware Technology Example: UML Model Muan survives Killed by Lock(); Close(); Afer(20); funcion coverage sae coverage ransiion coverage Muaed UML model 11 / 19
Insiue for Sofware Technology From Analysis o Synhesis Sae of ar: Analysis of es cases How many muans killed by es cases? muaion score = #killed muans #muans Problem: equivalen muans Soluion: review of surviving muans Research: Synhesis of es cases Find es cases ha maximise muaion score. Idea: Check equivalence beween original and muan Use couner-example as es case. Problem: equivalence checking is hard (undecidable in general) Soluion: generae from models (absracion) model-based muaion esing 12 / 19
Insiue for Sofware Technology From Analysis o Synhesis Sae of ar: Analysis of es cases How many muans killed by es cases? muaion score = #killed muans #muans Problem: equivalen muans Soluion: review of surviving muans Research: Synhesis of es cases Find es cases ha maximise muaion score. Idea: Check equivalence beween original and muan Use couner-example as es case. Problem: equivalence checking is hard (undecidable in general) Soluion: generae from models (absracion) model-based muaion esing 12 / 19
Insiue for Sofware Technology From Analysis o Synhesis Sae of ar: Analysis of es cases How many muans killed by es cases? muaion score = #killed muans #muans Problem: equivalen muans Soluion: review of surviving muans Research: Synhesis of es cases Find es cases ha maximise muaion score. Idea: Check equivalence beween original and muan Use couner-example as es case. Problem: equivalence checking is hard (undecidable in general) Soluion: generae from models (absracion) model-based muaion esing 12 / 19
Insiue for Sofware Technology From Analysis o Synhesis Sae of ar: Analysis of es cases How many muans killed by es cases? muaion score = #killed muans #muans Problem: equivalen muans Soluion: review of surviving muans Research: Synhesis of es cases Find es cases ha maximise muaion score. Idea: Check equivalence beween original and muan Use couner-example as es case. Problem: equivalence checking is hard (undecidable in general) Soluion: generae from models (absracion) model-based muaion esing 12 / 19
Insiue for Sofware Technology Model-Based Tesing Tes Case Generaor SUT Tes Driver 13 / 19
Insiue for Sofware Technology Model-Based Tesing Model Tes Case Generaor SUT Tes Driver 13 / 19
Insiue for Sofware Technology Model-Based Tesing Model Tes Case Generaor Absrac Tes Case SUT Tes Driver 13 / 19
Insiue for Sofware Technology Model-Based Tesing Model Tes Case Generaor Absrac Tes Case SUT Tes Driver pass / fail 13 / 19
Insiue for Sofware Technology Model-Based Tesing Model if conforms Tes Case Generaor Absrac Tes Case SUT Tes Driver hen pass 13 / 19
Insiue for Sofware Technology Model-Based Tesing Model if conforms Tes Case Generaor Absrac Tes Case SUT Tes Driver hen pass/fail 13 / 19
Insiue for Sofware Technology Model-Based Muaion Tesing Model Muaion Tool Tes Case Generaor Absrac Tes Case SUT Tes Driver 13 / 19
Insiue for Sofware Technology Model-Based Muaion Tesing Model Muaion Tool Model Muan Tes Case Generaor Absrac Tes Case SUT Tes Driver 13 / 19
Insiue for Sofware Technology Model-Based Muaion Tesing Model Muaion Tool Model Muan if conforms Tes Case Generaor Absrac Tes Case SUT Tes Driver hen pass/fail 13 / 19
Insiue for Sofware Technology Model-Based Muaion Tesing Model Muaion Tool Model Muan if conforms Tes Case Generaor Absrac Tes Case if conforms SUT Tes Driver hen fail 13 / 19
Insiue for Sofware Technology Model-Based Muaion Tesing hen conforms Model Muaion Tool Model Muan if conforms Tes Case Generaor Absrac Tes Case if conforms SUT Tes Driver hen fail 13 / 19
Insiue for Sofware Technology Reacive Sysems Reac o he environmen Do no erminae Servers and Conrollers Evens: conrollable and observable communicaion evens Tes cases: sequences of evens Unlock AlarmSysem_SaeMachine Unlock OpenAndUnlocked Open Close Lock Unlock ClosedAndUnlocked OpenAndLocked Unlock Lock Close Open ClosedAndLocked 20 Close Armed SilenAndOpen Show Armed /enry Show Unarmed /exi Open Alarm Acivae Alarms /enry Deacivae Alarms /exi FlashAndSound 30 / Deacivae Sound Flash 300 17 obs pass obs AlarmArmed_SeOff 11 cr Unlock 16 obs AlarmArmed_SeOn 15 cr Close 14 obs OpicalAlarm_SeOff 13 obs AcousicAlarm_SeOff 12 obs afer(270) 10 obs AcousicAlarm_SeOff 9 obs afer(30) 8 obs AcousicAlarm_SeOn 7 obs OpicalAlarm_SeOn 6 obs AlarmArmed_SeOff 5 cr Open 4 obs AlarmArmed_SeOn 3 obs afer(20) 2 cr Lock 1 Adapive es cases: rees branching a non-deerminisic observaions cr Close 0 14 / 19
Insiue for Sofware Technology Semanics Operaional semanics e.g. Labelled Transiion Sysems Inpu-oupu conformance (ioco) [Tremans96] 10 cr Unlock 11 obs AcousicAlarm_SeOff cr Unlock 2 obs afer (c_waiime: 30 ) 8 obs afer (c_waiime: 270 ) obs AcousicAlarm_SeOn 7 SUT ioco Model = df 15 obs OpicalAlarm_SeOn 4 σ races(model) : ou(sut afer σ) ou(model afer σ) obs AcousicAlarm_SeOff obs AlarmArmed_SeOff 14 cr Open 13 obs AcousicAlarm_SeOff 1 ou... oupus + quiescence afer... reachable saes afer race obs AlarmArmed_SeOn cr Unlock 12 5 obs afer (c_waiime: 20 ) obs AlarmArmed_SeOff 16 cr Lock cr Close cr Open cr Unlock obs OpicalAlarm_SeOff cr Close 6 17 9 cr Close cr Open obs OpicalAlarm_SeOff 0 cr Lock cr Unlock cr Unlock 3 15 / 19
Insiue for Sofware Technology Semanics Operaional semanics e.g. Labelled Transiion Sysems Inpu-oupu conformance (ioco) [Tremans96] 10 cr Unlock 11 obs AcousicAlarm_SeOff cr Unlock 2 obs afer (c_waiime: 30 ) 8 obs afer (c_waiime: 270 ) obs AcousicAlarm_SeOn 7 SUT ioco Model = df 15 obs OpicalAlarm_SeOn 4 σ races(model) : ou(sut afer σ) ou(model afer σ) obs AcousicAlarm_SeOff obs AlarmArmed_SeOff 14 cr Open 13 obs AcousicAlarm_SeOff 1 ou... oupus + quiescence afer... reachable saes afer race obs AlarmArmed_SeOn cr Unlock 12 5 obs afer (c_waiime: 20 ) obs AlarmArmed_SeOff 16 cr Lock cr Close cr Open cr Unlock obs OpicalAlarm_SeOff cr Close 6 17 9 cr Close cr Open obs OpicalAlarm_SeOff 0 cr Lock cr Unlock cr Unlock 3 15 / 19
Insiue for Sofware Technology Semanics Operaional semanics e.g. Labelled Transiion Sysems Inpu-oupu conformance (ioco) [Tremans96] Model:!soundOn!flashOn!soundOn!flashOn SUT ioco Model = df σ races(model) : ou(sut afer σ) ou(model afer σ) SUT:!flashOn!soundOn ou... oupus + quiescence afer... reachable saes afer race?unlock SUT ioco Model 15 / 19
Insiue for Sofware Technology Explici Conformance Checking Model and Muan LTS Deerminisaion Model:!soundOn!flashOn!soundOn!flashOn Muan:!soundOff!flashOn?unlock Build synchronous produc modulo ioco If muan has addiional!oupu: fail sink sae?inpu: pass sink sae Model ioco Muan:!soundOn!flashOn?unlock!soundOn pass!soundoff pass pass fail Exrac es case covering fail sae 16 / 19
Insiue for Sofware Technology Explici Conformance Checking Model and Muan LTS Deerminisaion Model:!soundOn!flashOn!soundOn!flashOn Muan:!soundOff!flashOn?unlock Build synchronous produc modulo ioco If muan has addiional!oupu: fail sink sae?inpu: pass sink sae Model ioco Muan:!soundOn!flashOn?unlock!soundOn pass!soundoff pass pass fail Exrac es case covering fail sae 16 / 19
Insiue for Sofware Technology Explici Conformance Checking Model and Muan LTS Deerminisaion Model:!soundOn!flashOn!soundOn!flashOn Muan:!soundOff!flashOn?unlock Build synchronous produc modulo ioco If muan has addiional!oupu: fail sink sae?inpu: pass sink sae Model ioco Muan:!soundOn!flashOn?unlock!soundOn pass!soundoff pass pass fail Exrac es case covering fail sae 16 / 19
Insiue for Sofware Technology Applicaions of Explici Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Conrollers (UML) Hybrid Sysems (Acion Sysem) Scalabiliy: absracions for daa-inensive models Bernhard K. Aichernig and Corrales Delgado. From Fauls via Tes Purposes o Tes Cases: On he Faul-Based Tesing of Concurren Sysems, FASE 2006. Marin Weiglhofer, Bernhard K. Aichernig, and Franz Woawa. Faul-based conformance esing in pracice. Inernaional Journal of Sofware and Informaics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeh Jöbsl, and Willibald Krenn. Efficien muaion killers in acion, ICST 2011. Harald Brandl, Marin Weiglhofer, and Bernhard K. Aichernig. Auomaed conformance verificaion of hybrid sysems, QSIC 2010. 17 / 19
Insiue for Sofware Technology Applicaions of Explici Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Conrollers (UML) Hybrid Sysems (Acion Sysem) Scalabiliy: absracions for daa-inensive models Bernhard K. Aichernig and Corrales Delgado. From Fauls via Tes Purposes o Tes Cases: On he Faul-Based Tesing of Concurren Sysems, FASE 2006. Marin Weiglhofer, Bernhard K. Aichernig, and Franz Woawa. Faul-based conformance esing in pracice. Inernaional Journal of Sofware and Informaics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeh Jöbsl, and Willibald Krenn. Efficien muaion killers in acion, ICST 2011. Harald Brandl, Marin Weiglhofer, and Bernhard K. Aichernig. Auomaed conformance verificaion of hybrid sysems, QSIC 2010. 17 / 19
Insiue for Sofware Technology Applicaions of Explici Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Conrollers (UML) Hybrid Sysems (Acion Sysem) Scalabiliy: absracions for daa-inensive models Bernhard K. Aichernig and Corrales Delgado. From Fauls via Tes Purposes o Tes Cases: On he Faul-Based Tesing of Concurren Sysems, FASE 2006. Marin Weiglhofer, Bernhard K. Aichernig, and Franz Woawa. Faul-based conformance esing in pracice. Inernaional Journal of Sofware and Informaics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeh Jöbsl, and Willibald Krenn. Efficien muaion killers in acion, ICST 2011. Harald Brandl, Marin Weiglhofer, and Bernhard K. Aichernig. Auomaed conformance verificaion of hybrid sysems, QSIC 2010. 17 / 19
Insiue for Sofware Technology Applicaions of Explici Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Conrollers (UML) Hybrid Sysems (Acion Sysem) Scalabiliy: absracions for daa-inensive models Bernhard K. Aichernig and Corrales Delgado. From Fauls via Tes Purposes o Tes Cases: On he Faul-Based Tesing of Concurren Sysems, FASE 2006. Marin Weiglhofer, Bernhard K. Aichernig, and Franz Woawa. Faul-based conformance esing in pracice. Inernaional Journal of Sofware and Informaics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeh Jöbsl, and Willibald Krenn. Efficien muaion killers in acion, ICST 2011. Harald Brandl, Marin Weiglhofer, and Bernhard K. Aichernig. Auomaed conformance verificaion of hybrid sysems, QSIC 2010. 17 / 19
Insiue for Sofware Technology Applicaions of Explici Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Conrollers (UML) Hybrid Sysems (Acion Sysem) Scalabiliy: absracions for daa-inensive models Bernhard K. Aichernig and Corrales Delgado. From Fauls via Tes Purposes o Tes Cases: On he Faul-Based Tesing of Concurren Sysems, FASE 2006. Marin Weiglhofer, Bernhard K. Aichernig, and Franz Woawa. Faul-based conformance esing in pracice. Inernaional Journal of Sofware and Informaics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeh Jöbsl, and Willibald Krenn. Efficien muaion killers in acion, ICST 2011. Harald Brandl, Marin Weiglhofer, and Bernhard K. Aichernig. Auomaed conformance verificaion of hybrid sysems, QSIC 2010. 17 / 19
Insiue for Sofware Technology Agile Developmen 6$1*7+")& 345%8&!"#$%& 345%$4$(+&,$-+&.*-$-& '$($)*+$&&,$-+&.*-$-& /$)012&,$-+&.*-$-& Model-driven developmen Model-based es case generaion Formal verificaion Tes-driven developmen 18 / 19
Insiue for Sofware Technology Summary Model-based Tesing + Muaion Tesing Tes case generaion via ioco check Indusrial applicaions in EU projecs MOGENTES, MBAT, CRYSTAL Tesing canno show he absence of bugs [Dijksra72]. Tesing can show he absence of specific bugs [Aichernig12]. 19 / 19
Insiue for Sofware Technology Summary Model-based Tesing + Muaion Tesing Tes case generaion via ioco check Indusrial applicaions in EU projecs MOGENTES, MBAT, CRYSTAL Tesing canno show he absence of bugs [Dijksra72]. Tesing can show he absence of specific bugs [Aichernig12]. 19 / 19
Insiue for Sofware Technology Summary Model-based Tesing + Muaion Tesing Tes case generaion via ioco check Indusrial applicaions in EU projecs MOGENTES, MBAT, CRYSTAL Tesing canno show he absence of bugs [Dijksra72]. Tesing can show he absence of specific bugs [Aichernig12]. 19 / 19