Unit 7: Metric for Process and Product 7.1 Software Measurement Measurement is the process by which numbers or symbols are assigned to the attributes of entities in the real world in such a way as to define them according to clearly defined rules. The ability to measure attributes of software and the software development process are essential to improvements in the practice of software engineering. In software development, measurements are performed by using metrics, which are experimental designations of a value to an object aiming to characterize a definite quality of this object. Measurement is the act of determining a measure. The IEEE Standard Glossary of Software Engineering Terms [IEE93] defines metric as a quantitative measure of the degree to which a system, component, or process possesses a given attribute. Why do we do software measure? o Measurement is fundamental to any engineering discipline o To get an insight into the software process o To do process improvement o To make the quality of the product better o To express the qualitative factor by a number o To make a better estimate of the cost, schedule, complexity, effort requirements etc of the software development process o Measurement is important for prediction. o Measurement is necessary for quality control and assurance Measure is a quantitative indication of the extent, size, amount, and dimension, capacity of some attributes of a product or a process. Measurement is the act of determining a measure. Types of Software Measurements 1. Direct measures Easy to collect E.g. Cost, Effort, Lines of codes (LOC), Execution Speed, Memory size, Defects etc. 2. Indirect measures More difficult to assess & can be measured indirectly only. Quality, Functionality, Complexity, Reliability, Efficiency, Maintainability etc. An example Software Engineering Compiled By: Roshani Ghimire Page 1
o 2 different project teams are working to record errors in a software process o Team A Finds 342 errors during software process before release o Team B- Finds 184 errors o Which team do you think is more effective in finding errors? An example of Measures, Measurements, Metrics, Indicators All test engineers of a certain project started collecting the following measure The errors found in the test cycle. But this measure is not enough to compute a suitable metric. You need two more measurements, which are number of test engineer and the number of months spent on the testing process. Suppose there were 5 test engineers and they spent about 6 months to do the necessary testing of the product. So, the total engineer-month is 30. And let s assume, all together they found about 60 errors. So, using all those measures mentioned above we can compute a Metric, which could be Errors found per Engineer-Month ; and the value of that metric here is 2. This metric can be used here as an indicator regarding how efficient the test teams are or how good the product is. For example, after introducing an automated testing tool, the above metric may reach a value of 5 ; which indicates the effectiveness of the automated testing tool used by the test team. So, this Metric can be used as an indicator of the efficiency of a project team 7.2 Metric for software quality Metric is a quantitative measure of the degree to which a system, component, or a process possesses a given attribute. An indicator is a metric or a combination of metrics that provide insight into the s/w process, a s/w project or the s/w product itself. Types of Metrics o Process Metrics Software Engineering Compiled By: Roshani Ghimire Page 2
Process Metrics: Project Metrics: Product Metrics: o Product Metrics o Project Metrics o Those metrics that are used to improve the process efficiency of the Software Development Life Cycle (SDLC) are called process metrics. For example, the process metric Defect Rate the amount of defects reaching the customer can be used to improve the efficiency of the Development and the Testing team. If Defect rate is less, we know that the efficiency of the Development and the Testing team is improving. o Process metrics concentrate on the process of software development and measure process structures with the aim of either distinguishing problems or pushing forward effective practices. o Process metrics are measures of the software development process, such as Overall development time Type of methodology used o Process metrics are collected across all projects and over long periods of time. o Their intent is to provide indicators that lead to long-term software process improvement. o The project metrics are those metrics which are more relevant to a project team. They can be used to measure the efficiency of a project team or any other tools being used by the team members. One example of a project metric is Errors found per engineer- month. This is a relevant project metric for a test team. o Project Metrics are the measures of Software Project and are used to monitor and control the project. They enable a software project manager to: Minimize the development time by making the adjustments necessary to avoid delays and potential problems and risks. Assess product quality on an ongoing basis & modify the technical approach to improve quality. o The product metrics are those metrics which has more meaning in the perspective of the software product being developed. One of the examples is quality of the developed product. Considering the fact that quality is a subjective attribute, one needs a very good understanding of the measures based on which the quality metrics are computed. Software Engineering Compiled By: Roshani Ghimire Page 3
Software Metrics: o Product metrics are also referred to as software metrics. They are directly associated with the product itself and attempt to measure product quality or characteristics of the product that can be connected with product quality. o Product metrics are measures of the software product at any stage of its development, from requirements to installed system. o Product metrics may measure: The complexity of the software design The size of the final program The number of pages of documentation produced o Software metric is a measure of some property of a piece of software or its specifications. Since quantitative measurements are essential in all sciences, there is a continuous effort by computer science practitioners and theoreticians to bring similar approaches to software development. The goal is obtaining objective, reproducible and quantifiable measurements, which may have numerous valuable applications in schedule and budget planning, cost estimation, quality assurance testing, software debugging, software performance optimization, and optimal personnel task assignments. o Software metrics can be private and public metrics o The Private metrics are those metrics which are collected by individual software professionals. They are mostly used by any software professional to get an insight regarding how his productivity or any other parameter of interests to him is. For example, a test engineer may keep errors found in a week as a private metric. Similarly, for a development engineer, lines of code written in a week could be a private metric of interests to him. Also, an IT professional may keep Number of new technology studied in a month as a private metric. o The public metrics has more meaning on an overall team basis. The public metrics can be computed depending upon the private metrics made public by the individual software professional. They are more concerned with the project team rather than any individual software professional. The examples are, Errors found per engineer-month, Lines of code written per engineermonth, etc It may happen that some of the product metrics are private metrics maintained by the individual engineer. When these metrics are shared by the entire engineer on the team basis, those metrics can be combined to develop a set of project metrics on a project level. In addition to that, these project metrics can be consolidated to create a set of process metrics. So, at the end these process metrics become the public metrics for the whole organization. Size Oriented Metrics: Software Engineering Compiled By: Roshani Ghimire Page 4
o The size oriented metrics are those metrics, which are computed keeping size of the software as main consideration. The sizes of the software are usually expressed in terms of KLOC (Kilo Line of Code). General Uses of Metrics Software metrics are used to obtain objective reproducible measurements that can be useful for quality assurance, performance, debugging, management, and estimating costs. Finding defects in code (post release and prior to release), predicting defective code, predicting project success, and predicting project risk. There is still some debate around which metrics matter and what they mean, the utility of metrics is limited to quantifying one of the following goals: Schedule of a software project, Size/complexity of development involved, cost of project, quality of software Types of metrics 1. Requirements metrics Size of requirements Traceability Completeness Volatility 2. Product Metrics Code metrics Lines of code LOC 3. Design metrics computed from requirements or design documents before the system has been implemented 4. Object oriented metrics- help identify faults, and allow developers to see directly how to make their classes and objects simpler. 5. Test metrics 6. Communication metrics looking at artifacts i.e. email, and meetings. 7. Process metrics Measure the process of software development Commonly used by management to check the budget and office procedures for efficiency Evaluate and track aspects of the software design process like: o Human resources o Time o Schedule o Methodology Software Engineering Compiled By: Roshani Ghimire Page 5
The standard identifies six key quality attributes: 1. Functionality: The degree to which the software satisfies stated needs as indicated by the following sub attributes: suitability, accuracy, interoperability, compliance, and security. 2. Reliability: The amount of time that the software is available for use as indicated by the following sub attributes: maturity, fault tolerance, recoverability. 3. Usability: The degree to which the software is easy to use as indicated by the following sub attributes: understandability, learn ability, operability. 4. Efficiency: The degree to which the software makes optimal use of system resources as indicated by the following sub attributes: time behavior, resource behavior. 5. Maintainability: The ease with which repair may be made to the software as indicated by the following sub attributes: analyzability, changeability, stability, testability. 6. Portability: The ease with which the software can be transposed from one environment to another as indicated by the following sub attributes: adaptability, install ability, conformance, and replace ability. Why Software Metrics To characterize. To evaluate To predict. To improve. Attributes of Software Metrics o Practical support is expected. o Measurement demands should be simple. o Less esoteric and easy to understand for real world professionals. o Should specify the basic intuitive notion of high-quality software. o Derived metrics and measures should lead to simple and computable conclusions o Easy to derive the metrics and should not demand inordinate effort or time 7.3 Software quality assurance Software quality is the conformance to explicitly stated functional and performance requirements, explicitly documented development standards and implicit characteristics that are expected of all professionally developed software Software requirements are reference from which quality is measured; lack of conformance is lack of quality. Specified standards define a set of development criteria that guide the manner in which software is engineered There are some implied requirements which are must with specified criteria The six standard attributes that are involved are: Software Engineering Compiled By: Roshani Ghimire Page 6
o Functionality o Usability o Maintainability o Portability o Reliability o Efficiency Software quality assurance (SQA) is the concern of every software engineer to reduce costs and improve product time-to-market. A Software Quality Assurance Plan is not merely another name for a test plan, though test plans are included in an SQA plan. SQA activities are performed on every software project. Use of metrics is an important part of developing a strategy to improve the quality of both software processes and work products. Software Quality Assurance is o An Umbrella activity applied throughout the software process o Planned and systematic pattern of actions required to ensure high quality in software o Responsibility of many stakeholders (software engineers, project managers, customers, salespeople, SQA group) SQA Questions are: o Does the software adequately meet its quality factors? o Has software development been conducted according to pre-established standards? o Have technical disciplines performed their SQA roles properly? Quality Assurance Elements are: o Standards ensure that standards are adopted and followed o Reviews and audits audits are reviews performed by SQA personnel to ensure that quality guidelines are followed for all software engineering work o Testing ensure that testing id properly planned and conducted o Error/defect collection and analysis collects and analyses error and defect data to better understand how errors are introduced and can be eliminated o Changes management ensures that adequate change management practices have been instituted o Education takes lead in software process improvement and educational program o Vendor management suggests specific quality practices vendor should follow and incorporates quality mandates in vendor contracts o Security management ensures use of appropriate process and technology to achieve desired security level o Safety responsible for assessing impact of software failure and initiating steps to reduce risk Software Engineering Compiled By: Roshani Ghimire Page 7
o Risk management ensures risk management activities are properly conducted and that contingency plans have been established SQA Tasks are: o Prepare SQA plan for the project. o Participate in the development of the project's software process description. o Review software engineering activities to verify compliance with the defined software process. o Audit designated software work products to verify compliance with those defined as part of the software process. o Ensure that any deviations in software or work products are documented and handled according to a documented procedure. o Record any evidence of noncompliance and reports them to management. SQA Goals are: Requirements quality o Ambiguity o Completeness o Volatility o Traceability o Model clarity Design quality o Architectural integrity o Component completeness o Interface complexity o Patterns Code quality o Complexity o Maintainability o Understandability o Reusability o Documentation Quality control effectiveness o Resource allocation o Completion rate o Review effectiveness o Testing effectiveness Six Sigma Software Engineering Software Engineering Compiled By: Roshani Ghimire Page 8
o Define customer requirements, deliverables, and project goals via well-defined methods of customer communication. o Measure each existing process and its output to determine current quality performance (e.g. compute defect metrics) o Analyze defect metrics and determine viral few causes. o For an existing process that needs improvement Improve process by eliminating the root causes for defects Control future work to ensure that future work does not reintroduce causes of defects o If new processes are being developed Design each new process to avoid root causes of defects and to meet customer requirements Verify that the process model will avoid defects and meet customer requirements What is a Software Quality Assurance Engineer? Software QA (quality assurance) engineers are individuals who monitor every phase of the software development process so as to ensure design quality, making sure that the software adheres to the standards set by the development company. Sometimes software QA engineers are confused with software testers, which is a mistake. Software testers test parts of the software at different stages of development, whereas a software QA engineer oversees the entire development process, which includes software testing, from start to finish. The monetary success of the software product is largely due in part to the quality of the product as well as the product s ability to hit the market on time. Both are the responsibility of the software QA engineer. Software Quality Attribute 1. Functionality: o Functionality represents all the system-wide functional. o These functional requirements represent the main product features and answer the question: o What the product does for us rather than how does it do it. o The easiest way to think of functional requirements is to ask the question why this piece of software exists. o This question of reason for being is distinct from security, look and feel and reliability concerns which are important but are not the concerned with the main function (or value add) of the software. 2. Usability Software Engineering Compiled By: Roshani Ghimire Page 9
3. Reliability o Usability includes looking at, capturing, and stating requirements based around user interface issues, e.g. issues such as accessibility, interface aesthetics, and consistency within the user interface. o Reliability includes aspects such as availability, accuracy, and recoverability, for example recoverability of the system from shut-down failure. 4. Performance o Performance involves issues such as throughput of information, system response time (which also relates to usability), recovery time, and startup time. 5. Supportability o This is a general bucket of requirements that address supporting the software: for example testability, adaptability, maintainability, compatibility, configurability, installs ability, scalability, localizability, and so on. 7.4 Software reliability Defined as the probability of failure free operation of a computer program in a specified environment for a specified time period Can be measured directly and estimated using historical and developmental data (unlike many other software quality factors) Software reliability problems can usually be traced back to errors in design or implementation. Measures of Reliability Mean time between failure (MTBF) = MTTF + MTTR MTTF = mean time to failure MTTR = mean time to repair Availability = [MTTF / (MTTF + MTTR)] x 100% Software Reliability is the probability of failure-free software operation for a specified period of time in a specified environment. Software Reliability is also an important factor affecting system reliability. It differs from hardware reliability in that it reflects the design perfection, rather than manufacturing perfection. The high complexity of software is the major contributing factor of Software Reliability problems. Software Reliability is not a function of time - although researchers have come up with models relating the two. The modeling technique for Software Reliability is reaching its prosperity, but before using the technique, we must carefully select the appropriate model that can best suit our case. Measurement in software is still in its infancy. No good quantitative methods have been developed to represent Software Reliability without Software Engineering Compiled By: Roshani Ghimire Page 10
excessive limitations. Various approaches can be used to improve the reliability of software, however, it is hard to balance development time and budget with software reliability. 7.5 The ISO 9000 quality standard Quality assurance systems are defined as the organizational structure, responsibilities, procedures, processes, and resources for implementing quality management. ISO 9000 describes the quality elements that must be present for a quality assurance system to be compliant with the standard, but it does not describe how an organization should implement these elements. ISO 9001:2000 is the quality standard that contains 20 requirements that must be present in an effective software quality assurance system. ISO 9000 certification ISO (International Standards Organization) is a consortium of 63 countries established to formulate and foster standardization. ISO published its 9000 series of standards in 1987. ISO certification serves as a reference for contract between independent parties. The ISO 9000 standard specifies the guidelines for maintaining a quality system. We have already seen that the quality system of an organization applies to all activities related to its product or service. The ISO standard mainly addresses operational aspects and organizational aspects such as responsibilities, reporting, etc. In a nutshell, ISO 9000 specifies a set of guidelines for repeatable and high quality product development. It is important to realize that ISO 9000 standard is a set of guidelines for the production process and is not directly concerned about the product itself. Types of ISO 9000 quality standards ISO 9000 is a series of three standards: o ISO 9001, ISO 9002, and ISO 9003. The ISO 9000 series of standards is based on the premise that if a proper process is followed for production, then good quality products are bound to follow automatically. The types of industries to which the different ISO standards apply are as follows. ISO 9001 applies to the organizations engaged in design, development, production, and servicing of goods. This is the standard that is applicable to most software development organizations. ISO 9002 applies to those organizations which do not design products but are only involved in production. Examples of these category industries include steel and car Software Engineering Compiled By: Roshani Ghimire Page 11
manufacturing industries that buy the product and plant designs from external sources and are involved in only manufacturing those products. Therefore, ISO 9002 is not applicable to software development organizations. ISO 9003 applies to organizations that are involved only in installation and testing of the products. Software products vs. other products There are mainly two differences between software products and any other type of products. Software is intangible in nature and therefore difficult to control. It is very difficult to control and manage anything that is not seen. In contrast, any other industries such as car manufacturing industries where one can see a product being developed through various stages such as fitting engine, fitting doors, etc. Therefore, it is easy to accurately determine how much work has been completed and to estimate how much more time will it take. During software development, the only raw material consumed is data. In contrast, large quantities of raw materials are consumed during the development of any other product. Need for obtaining ISO 9000 certification There is a mad scramble among software development organizations for obtaining ISO certification due to the benefits it offers. Some benefits that can be acquired to organizations by obtaining ISO certification are as follows: o Confidence of customers in an organization increases when organization qualifies for ISO certification. This is especially true in the international market. In fact, many organizations awarding international software development contracts insist that the development organization have ISO 9000 certification. For this reason, it is vital for software organizations involved in software export to obtain ISO 9000 certification. o ISO 9000 requires a well-documented software production process to be in place. o A well-documented software production process contributes to repeatable and higher quality of the developed software. o ISO 9000 makes the development process focused, efficient, and cost-effective. o ISO 9000 certification points out the weak points of an organization and recommends remedial action. o ISO 9000 sets the basic framework for the development of an optimal process and Total Quality Management (TQM). Software Engineering Compiled By: Roshani Ghimire Page 12