Software Quality Requirements and Evaluation, the ISO 25000 Series

Pittsburgh, PA 15213-3890 Software Quality Requirements and Evaluation, the ISO 25000 Series PSM Technical Working Group February 2004 Dave Zubrow Sponsored by the U.S. Department of Defense

Background This presentation reviews the latest developments with the ISO 25000 SQuaRE series of standards. This series on Software Quality Requirements and Evaluation (SQuaRE) is an effort to harmonize ISO 9126 and ISO 14598. The information presented here is based on the current state of the standards/ Note: Many of the slides in this presentation are taken from WG 6 documents and materials, especially slides from Professor Azuma, WG convener. page 2

Outline Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation page 3

Relationship between ISO/IEC 9126 and ISO/IEC 14598 R esources and environm ent Evaluation process S o ftw a re product Effect of the software product Evaluation support Evaluation process Internal m etrics External m etrics Q uality in use m etrics 14598-1 14598-2 14598-6 14598-3 14598-4 14598-5 9126-3 9126-1 9126-2 9126-4 page 4

Relationship and transition process between ISO/IEC 9126, ISO/IEC 14598 and SQuaRE series of standards C U R R E N T 9126: Product quality -1 : Q u ality m od el -2 : E x tern al m etrics -3 : In tern al m etrics -4 : Q u ality in u se m etrics New Proposal G uides to use 9126 & 14598 B ase m etrics Q u ality req u irem ents 14598: Product evaluation -1: G eneral overview -2: Planning and m anagem ent -3: Proc for developers -4: P roc for acquirers -5: Proc for evaluators -6: D oc of evaluation m odules SQ uare 25000: Q uality M anagem ent Division 25000: G uide to SQ uare (N P) 25001: Planning and m anagem ent 25010: Q uality M odel Division 25010: Q uality m od el an d gu id e (R ev) 25020: Q uality M easurem ent Division 25020: M easurem ent reference m odel and guide (N P) 25021: M easurem ent prim itives (N P) 25022: M easurem ent of internal quality 25023: M easurem ent of external quality 25024: M easurem ent of quality in u se 25030: Q uality Requirem ents Division 25030: Q uality req u irem en ts an d gu id e (N P ) 25040: Q uality Evaluation Division 25040: Q uality evalu ation overview an d gu id e 25041: Evaluation m odules 25042: Process for developers 25043: Process for acquirers 25044: Process for evaluators page 5

State of the Standards 25000, Guide to the SQuaRE series Ballot out for FCD 25020, Software Product Quality Measurement Reference Model Ballot out for 2 nd CD 25021, Measurement Primitives Ballot out for 1 st CD 25030, Quality Requirements Ballot out for 2 nd CD All Ballots closing Mid-April to Early May Editors assigned but no drafts out 25010, Quality Model 25023, External Quality Measures page 6

SQuaRE: Architecture ISO/IEC 2503n Quality Requirement Division ISO/IEC 2501n Quality Model Division ISO/IEC 2500n Product Quality General Division General Overview and Guide to the SQuaRE Planning and Management ISO/IEC 2504n Quality Evaluation Division ISO/IEC 2502n Quality Metrics Division page 7

SQuaRE general reference model Target of process Software Product User Task (Effect) Process Requirements Specification Planning Measurement Evaluatioin Requirements Specification Planning Measurement Evaluation Execution 25030 (QR) 25001 (PM) 25021( MP) 25022 (MfIQ) 25023 (MfEQ) 25041(EM) 25042 (PD) 25030 (QR) 25001 (PM) 25043 (PA) 25044 (PE) 25021( MP) 25022 (MfIQ) 25024 (MfQU) 25025(DEM) 25041(EM) 25042 (PD) 25043 (PA) 25044 (PE) 25020 (Measurement Reference Model and Guide) 25030 (Quality Requirements and Guide) 25040 (Evaluation Process Overview and Guide) Particular Guidance Legend : applied to used by MP Measurement Primitives MfIQ Metrics for Internal Quality MfEQ Metrics for External Quality MfQU Metrics for Quality in Use EM Evaluation Modules PM Planning and Management PA Process for Acquirers PE Process for Evaluators PD Process for Developers QR Quality Requirements 25010 (Quality Model and Guide) 25000 (Guide to SQuaRE) General Guidance page 8

Outline Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation page 9

The Product Quality Measurement Reference Model page 10

Software Product Quality Life-Cycle and Quality Measures Quality In Use Measures Requirements Users Users Quality Quality Needs Needs Needs Validation Products Quality Quality In In Use Use External Quality Measures Internal Quality Measures External Quality Quality External Requirement Quality Quality Verification And Internal Quality Validation Quality Internal Requirement Quality Quality Verification Implementation page 11

Quality Model Quality model Characteristic 1 Characteristic 2 Characteristic n Subcharacteristic 1 Subcharacteristic 2 Subcharacteristic m Quality Attributes page 12

ISO/IEC 9126-1 Quality In Use Quality In In Use Use Effectiveness Productivity Safety Satisfaction page 13

ISO/IEC 9126-1 - Quality Model Quality Characteristics Subcharacteristics Functionality Suitability Accuracy Interoperability Security Compliance Reliability Maturity Fault tolerance Recoverability Compliance Usability Understandability Learnability Operability Comp Attractiveness Efficiency Time behavior Maintainability Resource utilization Compliance Analyzability Changeability Stability Testability Compliance Portability Adaptability Installability Co-existence Replaceability Comp page 14

Process and Product Measurement External Quality Measure Internal Quality Measure Quality in Use Measure Software Computer System Information System Business System Computer Centered System page 15

Software product quality measurement reference model (SPQM-RM) page 16

Measurement concept of the SQuaRE model page 17

Relationship between the SQuaRE SPQM-RM and the ISO/IEC 15939 Information Model Q uality Needs / Q uality Evaluation R e p o r t In fo rm a tio n Product Assessm ent A n a ly s is R a t in g Q u a lity M e a s u re s In te rp re ta tio n In d ic a to r A n a ly s is M o d e l Function (F o rm u la ) M easurem ent P rim itiv e s M easurem ent M e t h o d A ttrib u te s fo r Q u a lity M e a s u r e Derived M easure M easurem ent Function B ase M easure M easurem ent M e t h o d A t t r ib u t e s S Q u a R E S P Q M - R M IS O /IE C 1 5 9 3 9 page 18

Outline Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation page 19

Titles of the Measurement Standards: 2502n Common: Software Engineering - Software product Quality Requirements and Evaluation (SQuaRE) - Division: Software Quality Measurement (For Explanation Only) 25020 Measurement Reference Model and Guide 25021 Measurement Primitives 25022 Measurement of Internal Quality 25023 Measurement of External Quality 25024 Measurement of Quality In Use page 20

Quality Measure and Measurement Method (Concept) Quality Measures: A variable, which shows satisfactory levels of a quality characteristic, subcharacteristic or sub-sub-characteristic, to which a value is assigned as the result of measurement Quality Measure Set: A set of a Quality Measure, Measurement Primitives that are used for deriving the quality measure, associated Scales and Measurement Methods, a Formula to combine them to generate the value of Quality Measure, and Guide to use them and analyze the results. for a Quality Characteristic or Subcharacteristic. (A line of Quality Measure Table) Quality Measurement Table: A set of Quality Measure Set for each Quality Subcharacteristic page 21

Application of SQuaRE Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8 Phase Requirement analysis (Software and systems) Architectural design (Software and systems) Software detailed design Software coding and testing Software integration and software qualification testing System integration and system qualification testing Software installation Software acceptance support 25000 (SQuaRE) series model reference Required quality in use, Required internal quality, Required external quality Predicted quality in use, Predicted external quality, Measured internal quality Predicted quality in use, Predicted external quality, Measured internal quality Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality Predicted quality in use, Measured external quality, Measured internal quality Predicted quality in use, Measured external quality, Measured internal quality Measured quality in use, Measured external quality, Measured internal quality Key deliverables of activity Quality in use requirements (specified), External quality requirements (specified), Internal quality requirements (specified) Architecture design of Software / system Software detailed design Software code, Test results Software product, Test results Integrated system, Test results Installed system Delivered software product Applicable measures Internal measures (External measures may be applied to validate specifications) Internal measures Internal measures Internal measures External measures Internal measures External measures Internal measures External measures Internal measures External measures Quality in use measures Internal measures External measures page 22

User needs, characteristics & weights Q u a lity in u s e CHARACTERISTIC W E IG H T Effectiveness P r o d u c tiv ity S a fe ty S a tis fa c tio n H H L M E x te rn a l & in te rn a l q u a lity C H A R A C T E R IS T IC Functionality S U B C H A R A C T E R IS T IC S u ita b ility W E IG H T (High/M edium /Low) H Accuracy H Interoperability L Security L C o m p lia n c e M R e lia b ility M a tu rity L Fault tolerance L R ecoverability H C o m p lia n c e H U s a b ility U nderstandability M Learnability L O perability H A ttra c tiv e n e s s M C o m p lia n c e H Efficiency Tim e behaviour H R esource utilization H C o m p lia n c e H M a in ta in a b ility Analyzability H C hangeability M Stability L T e s ta b ility M C o m p lia n c e H P o rta b ility Adaptability H Installability L C o-existence H Replaceability M C o m p lia n c e H page 23

Quality measurement tables Q uality in use m easurem ent category C H A R A C T E R IS T IC M E A S U R E S R E Q U IR E D L E V E L A S S E S S M E N T A C T U A L R E S U L T Effectiveness P r o d u c tiv ity S a fe ty S a tis fa c tio n External quality m easurem ent category C H A R A C T E R IS T IC S U B C H A R A C T E R IS T IC M E A S U R E S R E Q U IR E D L E V E L A S S E S S M E N T ACTUAL R E S U L T Internal quality m easurem ent category C H A R A C T E R IS T IC S U B C H A R A C T E R IS T IC M E A S U R E S Functionality S u ita b ility R E Q U IR E D L E V E L A S S E S S M E N T A C T U A L R E S U L T Functionality S u ita b ility Accuracy R e lia b ility Accuracy In te r o p e r a b ility Security C om pliance M a t u r ity R e lia b ility Interoperability S e c u r it y C o m p lia n c e M a tu rity Fault tolerance R ecoverability (data, process, Fault tolerance R e c o v e ra b ility ( d a ta, p r o c e s s, U s a b ility C o m p lia n c e U n d e rs ta n d a b ility C om pliance Learnability U s a b il it y U n d e rs ta n d a b ility O p e r a b ility L e a r n a b ility A ttra c tiv e n e s s O p e r a b ility C o m p lia n c e A ttra c tiv e n e s s Efficiency Tim e behaviour Efficiency C om pliance Tim e behaviour Resource utilisation C o m p lia n c e R esource utilisation M a in ta in a b ility A n a ly z a b ility C om pliance C h a n g e a b ility S ta b ility M a in ta in a b ility Analyzability T e s ta b ility C h a n g e a b ility C o m p lia n c e S ta b ility P o rta b ility A d a p ta b ility T e s ta b ility In s ta b ility C om pliance C o-existence P o r t a b il it y A d a p ta b ility Replaceability In s ta b ility C o m p lia n c e C o-existence R e p la c e a b ility C om pliance page 24

Measurement plan implications S U B C H A R A C T E R I DELIVERABLES IN T E R N A L EXTERNAL Q U A L IT Y IN U S E STIC TO BE M EASURES TO BE M E A S U R E S T O B E M EASURES TO BE E V A L U A T E D A P P L IE D A P P L IE D A P P L IE D 1. S u ita b ility 1. 1. 1. (N ot Applicable) 2. 2. 2. 3. 3. 3. 2. Satisfaction 1. 2. 3. (N ot Applicable) (N ot Applicable) 1. 2. 3. 3. 4. 5. 6. page 25

Measurement Primitive A measure, either a base measure or a derived measure, that is commonly used for deriving internal quality measures, external quality measures and quality in use measures. No definition (25000) measure collected during Software Product Lifecycle from which Internal, External and Quality in Use Measures are derived. (25020 and 25021) Single value of measurement primitive generally does not indicate the quality of the measured entity. NOTE The Quality is measured afterwards by calculating the Quality Measures. (25020 and 25021) page 26

Measurement Primitives in SPQM-RM Measurement primitives concept page 27

Consideration views Considerations Measurement Concept Measurement Scale Type Measurement Focus Measurement Method Type Base Nominal Ordinal Internal External Interval Ratio Derived Absolute Q in use Objec tive Subjec tive Measurement primitive page 28

Measurement Primitives Hyperlink Format example Consideration View s M easurem ent concept B a s e D e r iv e d M easurem ent scale N o m in a l : R a tio A b s o lu te M easurem ent focus In te rn a l E x te r n a l Q u a lity in U s e M easurem ent m ethod O b je c tiv e Subjective M easurem ent Prim itives Q u a lit y M e a s u r e s : : Num ber of functions Physical accessibility : : : : Task tim e W a itin g tim e : : : : T im e o r n u m b e r o f ta s k s Task efficiency : : page 29

Outline Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation page 30

Relationship between Needs and Requirements Stakeholders Needs in their Minds Stated, Implied or Unaware Needs Solicit & Identify Collected and Identified Stakeholders (Business) Needs Select & Specify Selected and Specified Needs & QIU Requirements Functional Requirements External Quality Requirements Functional Design & Internal Quality Requirements Non Functional Design & Internal Quality Requirements page 31

ISO/IEC 15288 System Life Cycle Processes to appear in 25030 ISO/IEC 25030 Quality requirements SQuaRE Elicit and define quality requirements Formalise identified quality requirements Identify & formalise internal quality requirements Stakeholder requirements definition process Requirements analysis process Architectural design process Implementation process Integration process Verification process Transition process Validation process Operation process Maintenance process ISO/IEC 15288 (Technical Processes) page 32

Evaluation process view according to ISO/IEC 14598-1 Establish ev a lu a tion req u irem en ts S p ecify th e ev a lu a tion D e s i g n t h e ev a lu ation E s ta b lis h p urp ose of evaluation Iden tify ty p es of p ro d uct(s) S p ecify quality m odel Select metrics E s ta b lis h ra tin g le ve ls fo r m e trics E s ta b lis h c riteria fo r a s s e s s m e n t P ro d u c e evalu a tio n p lan 9126-1 Q uality C h a ra cte ris tic s 9126-2 14596-6 External Evaluation - M etric o r d u sles r i 19126-3 8-6Internal E va lu a tio M etric o d u le s s 9126-4 Q uality in use M etric s 14598-6 Evaluation M o d u les Execute the ev a lu ation M easure characteristics C o m p a re w ith c riteria Assess results page 33

Summary - 1 SQuaRE series addresses requirements and evaluation of software product quality Internal, External, and Quality in Use Measures are the link between requirements and evaluation Software Product Quality Measurement Reference Model is a specific instance of 15939 information model Additional standards to create catalogues of measures for quality attributes and measurement primitives Need for validated measures to populate catalogues page 34

Summary - 2 Working to reconcile and harmonize SQuaRE series with other standards Concern over number of standards and fragmentation of content Reviewers sought page 35