Trillium: A Customer-Oriented Assessment Method for Software System Development Capability

Transcription

1 Trillium: A Customer-Oriented Assessment Method for Software System Development Capability Alain April, François Coallier Bell Canada 2265 Roland Therrien Longueuil, Québec Canada J4N 1C5 aapril@qc.bell.ca Abstract Since the early 70's Bell Canada has carried out vendor assessments covering commercial and quality issues. The quality assessment has been traditionally oriented toward manufacturing and customer support functions. In the early 80's, with the growing importance of the software component in all products, the development part of the lifecycle saw its significance increase. The increasing complexity and dimension of telecommunication software s has led to numerous research activities and the development of quality tools in telecommunication product and services development. Since 1982 Bell Canada has invested toward developing a model to assess the software development process of existing and prospective suppliers as a mean to minimize the risks involved and ensure both the performance and timely delivery of software systems purchased. This paper presents the revised 1995 Trillium assessment model (version 3.0). 1. Introduction An assessment method is in itself a product designed from specifications. The specification criteria should be driven by the expectations that the assessing organization has from the supplier organizations that will be assessed. In software development it is essential that practices and methods used be measurable according to a set of well defined standards that are used industry wide. Having a good development process (in the classical sense) will contribute very significantly to an increase in the capability of a software product development organization. But other factors are important and must be considered: personnel, technology, management, development environment & tools. Traditional assessment approaches were typically process oriented a series of actions or operations conducting to an end. The assessment would focus on actions and operations and how they contribute to a desired purpose. When adding environmental factors in the assessment method we align with a "TQM" definition of process which is taking in account the mechanisms and resources as a whole. Bell Canada telecommunication network operations depends on more than 50 million lines of code. This number increases yearly as the network becomes more digital and the operational support systems are introduced to mechanize the administration. The median size of these software systems lies between 200 To 400 Kloc (thousands of lines of code), with the largest having over 20 million lines of code. Most of this software is real-time, with a majority of the systems of a hybrid nature (combination of real-time, database and graphical). In this environment some software s control safety/critical systems and must be reliable in order for Bell Canada to offer a level of service required by its customers. In April 1991 the first version of the Trillium model was elaborated by Bell Canada, Nortel and Research Bell Northern. This model and its application method have been improved based on experience and feedback from suppliers. From 1992 to 1993 versions 2 and 2.3d were used extensively and data about assessment was collected from assessment teams and management. In January 1995 version 3.0 was issued to Bell Canada suppliers and was placed in the public domain.

2 2. Model 2.1. Purpose The Trillium model has been developed from a customer perspective, as perceived in a competitive, commercial environment. In this context, Capability refers to: The ability of a development organization to consistently deliver a product or an enhancement to an existing product: that meets customer expectations, with minimal defects 1, for the lowest life-cycle cost, and in the shortest calendar time. Product in the Trillium context refers to what the customer receive, use and perceive. For an embedded telecommunication product, this would include: hardware, software, documentation, training and support services. Trillium can be used as a customer focused benchmark for either: auditing the product development and support capability of a supplier of telecommunication software product. an internal product development and support capability continuous improvement program for organizations developing and supporting telecommunication products. The ultimate purpose of improvement programs initiated as a result of a Trillium assessment is increased customer (and shareholder) satisfaction, rather than rigid conformance to the standards referenced by this document. A higher capability, in the Trillium context, means for customer organizations that: the development organization is more responsive to customer and market demands, the life-cycle cost of the product(s) is minimized, and end-user satisfaction is maximized. 1 In telecommunications and other industries defects resulting in system failure or service outages are unacceptable. However, certain defects may be tolerable provided they have no impact on service levels and operations. For the development organization, achieving a higher capability can result in: lower development and maintenance costs, shorter cycle time and development intervals, an increased ability to achieve content and schedule commitments due to effective project risk analysis and effort estimation, and, an increasing ability to meet quantifiable design and quality objectives at all stages of the development life-cycle. 2.2 Scope Trillium was designed initially to be applied to embedded software systems such as telecommunications systems. Although Trillium is aimed at embedded software systems, much of the model can be applied to other segments of the software industry such as Management Information Systems (MIS). Furthermore, a significant percentage of the practices described in the model can be applied directly to hardware development. The Trillium Model covers all aspects of the software development life-cycle, most system and product development and support activities, and a significant number of related marketing activities 2.3 Foundation The basis of the process assessment models for software relies on benchmarking, e.g.. Comparing your practices with the best and successful organizations. It is also a basic tool that you will find in the TQM literature. For software assessment we have used initially two level of benchmarks to develop the model: Professional, national and international standards; Comparisons to other organizations in the same market segment. The software assessment model should therefore map to existing engineering standards as well as quality standards. It should also provide an output that can be used easily to benchmark against "best-in-class" organizations. The Trillium model is a set of practices derived from a benchmarking exercise. The scope of these benchmarks aim at all practices

3 that would contribute to an organization product development and support capability as defined previously. The main sources of benchmarking information are, in decreasing order of importance, are: Version 1.1 of the Software Engineering Institute's Capability Maturity Model; ISO 9001:1994 International Standard and : 1991 Guideline; Bellcore's TR- NWT : Issue 2, June 1993 and TA-NWT : Issue 1, December 1993; Malcom Baldrige National Quality Award, 1995 criteria; IEEE Software Engineering standards: 1993 Edition and IEC 300: The Trillium model is based on the Software Engineering Institute (SEI) Capability Maturity Model (CMM) version 1.1. In the Trillium context, the product development and customer support processes are viewed as an integral part of the organization's business processes. The architecture of the Trillium model (described in Section 2.4) differs significantly from the one of the CMM version 1.1. The most significant differences are: a roadmap based concept [instead of key product areas], a product perspective [instead of software], a wider coverage of capability impacting issues, a strong customer focus. The Trillium model incorporates additional practices from the following topics: Organizational Processes Quality, Business Process Engineering, Technological Maturity, Development Environment, Systems Engineering, Life-Cycle Cost Modeling, Concurrent Engineering, Reliability Management, Customer Support/Partnership, Usability Engineering. To understand the Trillium model, it is desirable to have a background in product engineering and quality management, and a solid understanding of the source documents listed above. 2.4 Roadmap concept The Trillium model is based on the concept of a roadmap. A roadmap is a set of related practices that focus on an organizational area or need, or a specific element within the product development process. Each roadmap represents a significant capability for a software development organization. Within a given roadmap, the level of the practices is based on their respective degree of maturity. The most fundamental practices are at a lower level whereas the most advanced ones are located at the higher level. Each of the 5 levels can be characterized, in a commercial environment, as described in Table 1. An organization will mature through the roadmap levels. Lower level practices must be implemented and sustained for higher level practices to achieve maximum effectiveness. Roadmaps are organized into Capability Areas. Each represents a significant capability for a software development organization. There are 8 Capability Areas (see model for details) that can span the 5 Trillium Capability levels. A representation of the span of each Capability Area is shown in Table 2. For the purpose of Trillium, note that the IEEE Software Engineering Standards, which are mostly work products (e.g. Software design description, project management plan, etc.) oriented, are to be used as guidelines only. Trillium is based on Capability Areas, Roadmaps and Practices. Each Capability Area incorporates one or more Roadmaps (see Table 2). Each Roadmap comprises one or more Practices that span several Trillium levels. The complete version 3.0 has 8 capability areas, 28 roadmaps and 508 practices.

4 Lvl Level Name Risk Interpretation Unstructured High Ad-hoc development process 2 Repeatable and Project Oriented Medium Project based process 3 Defined and Process Oriented Low State of the art development process 4 Managed and Integrated Low Generally difficult to achieve now 5 Fully Integrated Low Technologically challenging Table 1: Trillium Capability Levels. Capability Area Roadmap Level Span Organizational Processes Quality Quality Management 2-4 Business Process Engineering 2-3 Human Resource Development Human Resource Development and Management and Management 2-4 Process Process Definition 2-4 Technology Management 2-4 Process Improvement & Engineering 2-5 Measurements 2-4 Management Project Management 2-4 Subcontractor Management 2-3 Customer-Supplier Relationship 2-3 Requirements Management 2-4 Estimation 2-3 Quality System Quality System 2-5 Development Practices Development Process 2-5 Development Techniques 2-5 Internal Documentation 2-4 Verification & Validation 2-4 Configuration Management 2-5 Re-Use 2-5 Reliability Management 2-4 Development Environment Development Environment 2-5 Customer Support Problem Response System 2-3 Usability Engineering 2-4 Life-Cycle Cost Modeling 2-3 User Documentation 2-3 Customer Engineering 2-3 User Training 2-3 Table 2: Capability Area/Roadmap Relationship

5 2.5 Integration Steps The Trillium model was built by integrating practices as per the following algorithm: 1- Practices are taken from the SEI CMM Version A mapping is performed between the CMM practices and ISO 9001 and ISO clauses. CMM practices may be modified to accommodate mapping. 3- From ISO 9001 and ISO clauses that cannot be mapped to CMM practices are extracted, added and integrated. 4- Bellcore standards clauses are mapped to the practices generated by steps 1, 2 & 3. Some practices may be adjusted to accommodate mapping. 5- From Bellcore standards clauses that cannot be mapped practices are extracted, added and integrated. 6- The same process is repeated with relevant portions of the Malcom Baldrige examination criteria. 7- Practices from IEC 300 are added. 8- Pointers to relevant IEEE standards are added. 9- Trillium specific practices are added to provide coverage of additional areas important to the telecommunications industry. These are based on professional benchmarks generated through the consensus of subject matter experts and validated in a peer review process. When practices are extracted from the CMM, they go through the following transformation if applicable: 1- The practice is generalized by either removing references to "software", or replacing them by "product and services" or "systems". 2- The practice is generalized by either removing references to "development", or replacing them by "development and support". 3- References to "group" or other specific organizational units are replaced by "function". 4- Allusions to specific documents are replaced by allusion to a process (e.g., "quality plan" by "quality planning") or to "documentation" or "information". The same type of transformations were applied when extracting practices from other standards. Assignation to a given level is based on the general guidelines of Table 1. - Practices that are considered fundamental for the successful conclusion of a development project are assigned to Level 2. - Practices that are considered to be organization-wide in scope or fundamental to the continuous improvement of the development process are assigned to a level 3. - Practices that deal wit CASE technology or characterize advanced process maturity (e.g. change management, integration of defect prevention, statistical process control and advanced metrics) are generally assigned to level 4. - Level 5 practices typically deals with advancing technology as it applies to process automation, formal methodologies and strategic utilization of organization repositories. 3. Trillium activities The following Trillium based activities can be performed: supplier capability evaluation, participation in supplier capability joint-assessment, participation in supplier capability selfassessment, participation in supplier capability continuous improvement program. While the first activity does not require that a special relationship exists between a customer and its supplier, the next three require an increasing level of partnering to be really effective. Succinct description of these activities follows. Readers requiring further information should consult the references. 3.1 Supplier Capability Evaluation The objective of this activity is to assess, by way of an auditing exercise using the Trillium model as the reference benchmark, the product development and support capability of a supplier. This activity is performed as per recognized auditing practices such as ISO , but as per the application principles described in section 2.6.

6 Trillium supplier capability evaluation are performed for assessing the risks associated with the procurement of a given product, and incidentally doing business with its supplier, monitoring a quality/capability improvement program as part of its acquisition supplier quality management program. To be successful, such an evaluation needs: an assessor or team of assessor with adequate qualifications, experience and skills, a minimum of cooperation from the supplier, adequate preparation. 3.2 Supplier Capability Joint-Assessment A joint-assessment in the Trillium context is an assessment with the difference that it is performed by both customer and supplier as a team exercise. This implies that the conclusions and recommendations of a Trillium jointassessment represent a consensus of the joint team. To be successful, a joint-assessment needs: a team of assessors with adequate qualifications, experience and skills, adequate partnering maturity between both organization, clear and unequivocal buy-in from senior management in both organizations. A Trillium joint-assessment that meets the above criteria will be far more efficient and reliable in getting a comprehensive appraisal of the supplier organization, but will require more resources that the normal assessment. With skilled personnel, a joint-assessment will be very efficient in appraising a low maturity organization (20/80 or Pareto rule). 3.3 Supplier Capability Self-Assessment A Trillium capability assessment is essentially an SEI self-assessment (as described in SEI-89-TR-7) with the following differences: at least one customer representative is on the self-assessment team. This representative is adequately qualified and has an improvement focused attitude, customer issues are systematically considered by the self-assessment team thought all phases of the self-assessment, the self-assessment is implemented as per the application principles described in section 2.6, the Trillium model is used as the reference benchmark. Like a SEI self-assessment, a Trillium capability assessment can become very demanding resource-wise, but is an efficient initiator of a capability improvement program because of the buy-in it is generating in the organization for its findings and recommendations. 3.4 Supplier Continuous Improvement Program A capability continuous improvement (CI) program is essentially a Deming quality cycle. A Trillium capability continuous improvement program is a Deming PDCA cycle where: at least one customer representative is on the CI program steering and management teams. This representative is adequately qualified and has an improvement-focused attitude, customer issues are systematically considered by the CI teams thought all phases of the CI cycle. the Trillium model is used as the reference benchmark, the "Check" step of the Deming quality cycle is always associated with the contribution to customer (and shareholder) satisfaction. In order to achieve the highest possible capability, development and support organizations must strive to: make the continuous improvement philosophy part of the organizational culture and ensure that proper mechanisms and processes are in place to support and encourage this culture, engineer and optimize their processes to meet customer or market requirements and objectives for the specific product and/or service, continuously optimize the resources allocated to each development project. The presence of an efficient capability continuous improvement program in a supplier is, from a Bell Canada perspective, a reassuring sign of organizational maturity. Being invited to participate in such a program

7 is a sign, from a supplier quality management perspective, of high maturity in the customersupplier partnership. The efficiency of a capability CI program should be measured in term of: contributions to customer (and shareholder) satisfaction improvement, improvements in capability maturity. 4. Applicability context The Trillium model should be always used in a pragmatic fashion keeping in mind that the bottom-line in all Trillium-based activities or program is improving customer (and shareholder) satisfaction. All Trillium based activities should thus be context driven. This means taking into account factors such as: -the nature of the product, -the context in which the product will be used, -the perception that the customer(s) has of the product (or products line) and its evolution, -only the organizations that are relevant to the development and support of the product. This implies that applicability of some practices and their implementation strategy will changes in function of the above factors. The professionalism of the staff and the organization involved in Trillium-based activities is thus critical. All Trillium-based activities should be lead by experienced software engineering personnel having, for example, tickit lead auditors training, practical knowledge of ISO auditing guidelines and SEI CMM selfassessment formal qualifications or equivalent. Malcom Baldridge assessor and QAI: CQA, or equivalent, would also be good assets. Customers perceive a supplier organization as a black box delivering products and services (e.g., Marketing, Sales, Customer Support, Research & Development [hardware, software, system, silicon], Quality Assurance, etc.). Trillium-based activities should thus, ultimately, make abstraction of internal organizational boundaries. In a Trillium context, internal organizations such as marketing, engineering, customer support, etc... are all contributing to customer (and shareholder) satisfaction. The customer will always perceive more acutely the impact of the weakest contributor. All these organizations should thus, for instance, adhere to a systematic improvement program with common and compatible goals. 5. Future versions The next experimental release of Trillium will be version 3.1 draft. This version will essentially be version 3.0 with added capability areas to address the MIS, Data Center Operations and Internal Communications Networks (LAN/WAN) practices. Version 3.1 draft will be published in a Bell Canada edition. Also a Québec-France cooperative project, titled Camélia is underway. This project funded by both the Québec and French government is aiming at translating the Trillium model to French as well as optimizing Trillium to the Management Information System focus. The Camélia v1.0 model will be tested during autumn 1995 with a full model training and experiments in Québec and France. The following additional roadmaps are covered in the Camelia model in addition to the coverage of Trillium : Re-Engineering, Business Process Engineering, Software Systems Architecture, Data Management, Data Center Management. Acknowledgments Trillium is the result of the cooperative work and contribution of many individuals. The authors would like to specially thank Neil Gammage, Allan Graydon, Jean-Normand Drouin, Jean Mayrand, John Wilson, Joe Hatz and Richard McKenzie for their work on Trillium v3.0. Denis Bistodeau, Richard Basque, Armelle LeGuevel (as well as the French Camélia team) and Michel Kwas for their work on Camélia. To obtain a version of the Trillium v3.0 model, use the following internet addresses: -ftp://ftp.quality.org/pub/qc/trillium -ftp://matrix.casti.com/pub/qc/trillium -WWW:

8 References Anonymus, 1991, IEEE Software Engineering Standards Collection, 1991, IEEE Inc. Anonymus, 1991, ISO Part 3: Guidelines for the application of ISO to the development and maintenance of software. Anonymus, 1994, Malcolm Baldrige National Quality Award 1990 Application Guidelines, US Department of Commerce. Anonymus, 1990, A Guide to Software Quality Management System Construction and Certification using EN29001 [ISO 9001], Issue 1, UK Department of Trade and Industry. Anonymus, 1989, Software Quality Program Generic Requirements (SQPR), Bellcore TR-TSY First edition, july Anonymus, 1987, Quality systems - Model for quality assurance in design/development, production, installation and servicing, ISO. Christiansen, D. [ed], 1987, Engineering Excellence - Cultural and Organizational Factors, IEEE Press. Coallier F., 1991, A Method for the Assessment of Telecom Software System Development Capability, Proceedings of the ASQC 1 st st International Conference on Software Quality. CSA Q396: 1989, Crosby P.B., 1979, Quality is Free, McGraw-Hill. Cusumao M.A., 1991, Japan s Software Factorie - A Challenge to US Management, Oxford University Press. DeMarco T. & Lister T., 1987, Peopleware - Productive Projects and Teams, Dorset House Publishing Co. Dorling A. & Simms P., 1992, Study Report: The Need and Requirements for a Software Process Assessment Standard, ISO/IEC JTC1/SC7/WG7/SG1 N944R Issue 2.0, june 11, 1992, Admiral plc, Cranfield IT Institute and UK Ministry of Defence. Grady R.B. & Caswell D.L., 1987, Software Metrics: Establishing a Company-Wide Program, Prentice-Hall. Humphrey W.S., 1989, Managing the Software Process, SEI series in Software Engineering, Addison Wesley. Humphrey W.S. & Sweet W.L., 1987, A Method for Assessing the Software Engineering Capability of Contractors, U.S. Department of Commerce NTIS Technical Report CMU/SEI-87-TR-23, sept Imai M., 1986, Kaizen - The Key to Japan s Competitive Success, McGraw-Hill Publishing Company. ISO :1991 ISO 8402:1991, Quality management and quality assurance - Vocabulary. ISO/IEC JTC1/SC7, The Need and Requirements for a Software Process Assessment Standard, Study Report N944R, Publication 2.0, june 1992 ISO/IEC JTC1/SC7/WG7, Information Technology Software Life Cycle Process, Work Item 7.21, Madhavji N.H. & Shäfer W. [eds], 1991, Software process and its support, special issue of the IEEE Sotfware Engineering Journal, september Musa J.D. et al., 1987, Sotfware Reliability - Measurement, Prediction, Application, McGraw- Hill, p178. Paulk M.C., Humphrey W.S. et al., 1991, Software Process Assessments: Issues and Lessons Learned, SEI. Paulk M.C. et al., 1991, Capability Maturity Model for Software, CMU/SEI-91-TR-00 & CMU/SEI-91- TR-24. Pressman R.S., 1987, Software Engineering - A Practitioner s Approach, McGraw-Hill Series in- Software Engineering and Technology. Rosenblatt A. & Watson G.F., 1991, Concurrent Engineering, IEEE Spectrum 28(7) july 1991, p

9 Shaw M., 1990, Prospect for an Engineering Discipline of Software, IEEE Software, november Silver W., 1992, TQM vs the SEI Capability Maturity Model, Software Quality World 4(2), p Nielsen J., The Usability Engineering Life Cycle, Computer 25(3), p , march Weber C.V. et al., 1991, Key Practices of the Capability Maturity Model, CMU/SEI-91-TR-25.