IP Billing and the Batelco Product Assessment Process

Transcription

1 SOFTWARE PRODUCT MEASUREMENT FOR SUPPLIER EVALUATION Alain A. April, IS Process Improvement Bahrain Telecommunications Company P.O. Box 14 Manama, Bahrain Dhiya Al-Shurougi, IS Planning Bahrain Telecommunications Company P.O. Box 14 Manama Bahrain Abstract The quality of the services Batelco offers to its customers is more and more related to the quality of the software products and systems acquired. Real time services like IP and ecommerce will now be totally software dependant, real-time with a 7days 24hours-availability objective. We realize that software acquisitions in this area have to be of the highest quality. IP or ecommerce services that fail in operations and in the hands of customers will mean dissatisfaction, revenue loss and potentially lost market share. To minimize the acquisition risks, the IP billing project is considering the introduction of a software product assessment process and associated tools to obtain maximum insights on the software quality characteristics during the selection process. This new direction requires the usage of sophisticated source code analysis tools where key software measures are extracted from the candidate software and are readily available for interpretation. As a first step the product assessment will aim principally in assessing the as much as possible of the software product quality characteristics as defined by the ISO-9126 standard. This paper first presents the Software Acquisition Life Cycle where intended software product assessment activities are to take place. Second we describe the software assessment process then provide a mapping to ISO9126 of the pilot measures and describe how they can be used to assess a supplier software. 1. INTRODUCTION Batelco s telecommunication network operations depends yearly more and more on software products. The planned additions of a new layer of IP products and systems with the introduction of ecommerce realtime software s will require careful planning and product selection to ensure stability and continuity of service for our customers. In this quickly evolving environment the IP and ecommerce software control critical components must be reliable where the availability criteria is typically set to be a few minutes downtime over a period of years. Batelco is investigating the inclusion of an additional step in the supplier selection process to assess the software product quality before it is acquired. This is a challenging task as we do not want to extend the selection cycle time more than two additional weeks. As a trial, a rapid and in depth quality assessment of an IP billing software product built in C, C++ and Java is planned. To address this issue Batelco has reviewed approaches and tools for the software product evaluation and assessment [1,6]. Four commercial tools to evaluate C++ and Java source code quality have been selected to be candidates for the pilot: Logiscope, CodeCheck, Insure++ and Datrix. The objective of the software product assessment is to get the maximum information, for contract adaptation recommendations, within two weeks. This schedule pressure imposes mechanization of some key steps of the software product assessment. As presented by Boloix [6] «the metrics [interpretation] is difficult because there are no mechanisms for synthesizing the information.» 2. SOFTWARE ACQUISITION LIFE-CYCLE The Batelco Software Acquisition Life Cycle (SALC) includes all the activities from the definition of the product requirements up to its implementation and support. The following paragraphs details the sequence of activities involved during the Feasibility stage of a software acquisition project where most of the software product assessment activities are conducted. The acquisition process contains six steps and is in line with recognized practices as described in the ISO/IEC standard. [9].

2 2.1 Product & Services Plans The first activity is the expression by the Product manager of a need for new product or services and the associated requirements from a functionality, pricing structure and time to market. 2.2 Product Requirement Specifications (PRS) The second activity is the definition of the Product Requirements Specifications for the product. For the software components of the product specific requirements are further categorized in 7 areas: 1) Functional Requirements 2) Performance Requirements 3) Interface Requirements 4) Operational Requirements 5) Resource Requirements 6) Documentation Requirements 7) Security Requirements 2.3 Invitation to Tender (ITT) In addition to the requirements defined by the Product manager, general IS requirements are added and an ITT is issued to the supplier community. The IS based requirements are further categorized in an additional 7 areas: 1) Quality Requirements 2) Reliability Requirements 3) Maintainability Requirements 4) Safety Requirements 5) Portability Requirements 6) Verification Requirements 7) Acceptance Testing Requirements 2.4 Supplier Evaluation Using the ITT answers and the information sessions presentations from the suppliers, the procurement agent and the supplier evaluation working group makes a pre-selection based on cost and functionality. This pre-selection usually narrows down the list to a few candidates. 2.5 Supplier Selection The selection of a supplier is a business decision that requires gathering of all pertinent information. All criteria are reviewed and a comparison table is drawn where stakeholders rate the resulting candidates in detail. At this time a final supplier is identified and a letter of intent is generally issued to initiate the mobilization of the supplier resources for initial stages of studies. 2.6 Contractual Agreement The agreement with the Integrator details the general Terms and Conditions and the Controlling Specifications, where system performance and quality goals are expressed. A typical contract outline is presented here : A- General Terms and Conditions B- controlling specification 1) Business process definition 2) User Requirement Specification 3) Specifications for H/W, S/W 4) Documentation including media 5) Functional requirements matrix 6) Software parametrization 7) Functional Customization 8) System performance criteria 9) Sites schedule 10) Network bandwidth assumptions 11) Interface specifications 12) Disaster Recovery 13) Quality Plan C- Master Project Plan 1) Project plan 2) Work breakdown structure D- Cost schedule E- Software license agreement 1) Escrow agreement F- Maintenance Support Contract G- Training Contract H- Non-disclosure Agreement 2.7 Independent Quality Assurance The Independent Quality Assurance activities are mainly audits based on the Quality Plan of the project. It includes conformance audits [3] on the process quality and covers the project roles and responsibilities, deliverables presence and quality, change control on deliverables and contract, identification of new functionality, system performance, availability and support issues. The scope and efforts required by each of these activities greatly depend on the criticality of the project. For each project Batelco customizes the activities derived from its Software Acquisition Life Cycle. 3. SOFTWARE ASSESSMENT PROCESS The Current Batelco software supplier selection process does not assess the Product or Process Quality [1] associated with the final candidate. The intention of this pilot activity is to test the benefit of a software product quality assessment and ease of use of tools. The pilot product assessment exercise aims principally in assessing as much as possible of the software product as defined by the ISO-9126 standard [10]. The software is thus assessed from an architectural, design, and an implementation point of view. The motivation for assessing the software beforehand is to get a clear

3 understanding of its quality and have the opportunity to include some additional safeguards in our contractual agreements when weaknesses are found. Few Companies conduct such product analysis that involves in-depth information sharing with a potential supplier. We believe that because of the growing importance of software acquisitions a proactive approach to quality is an important and necessary activity. 3.1 Overview of the process As presented earlier, the goal of the software product assessment is to obtain an objective evaluation of the internal characteristics of software before it is purchased. The time requirement for producing such assessments has been set to a period not to exceed 2 weeks. This constraint is imposed to ensure the Contract Finalization Working Group cycle time not be affected by this parallel technical activity. The process used by the pilot project, described in detail by Mayrand [13], will follow 5 steps: Step 1 - Supplier agreement: Where a letter of intent is signed with the supplier it becomes possible to ask to inspect further their software product documentation and source code. If agreed, a specific non-disclosure agreement may be required. Step 2 - Programming and design guidelines: At this step the team will review the software design and coding guidelines of the supplier, if any, to ensure that the review activities take into considerations specific software implementation details and the rationale for their presence. Step 3 - Product source code sample analysis and tool adjustment: Every tool will react differently with a specific software implementation. At this step the team will adjust the evaluation tools to work with sample given (average 1/10 of the actual system) and the specific C and C++ implementation. As a second adjustment each measurement output of the sample is reviewed to see if the default upper and lower bounds are set at an appropriate level. If the resulting output graphs distribution is inadequate, thresholds (also called upper and lower bounds) are adjusted. Also based on the previous phase activity thresholds can also be adjusted to align with supplier design and coding guidelines. Step 4 - Complete source code analysis: A full software run is done with the calibrated tools. Step 5 - Visual inspection: Using the measures, focussed inspections that examines a potential problem area are done to confirm the tool observation. The software product assessment appraises to a certain extent the effectiveness of the software development process used by the supplier. A good development process [2] should be reflected in the software product attributes. This exercise can also attend to mechanize the assessment of supplier software against published programming guidelines [15] or measure the risk against known traps and pitfalls of a specific compiler or language [11]. Finally it can also measure the conformance of the software against ANSI standards recommendations for coding [4]. 3.2 Assessment techniques and tools Batelco s IS pilot assessment of software product covers only the assessments of the source code. Source code assessment can be partially mechanized, which makes it practical given the time and resource constraints imposed on the Contract Evaluation Working Group schedule. After a survey of the C++ and Java source code quality tools available on the market, we are investigating four finalists that would be used for specific aspects of the ISO9126 characteristics. Four industrial tools to evaluate C++ and Java source code are Logiscope [16], CodeCheck, Insure++, and Datrix. Logiscope, CodeCheck, and Datrix are to be used for the static analysis and rules checking. The Insure++, visualizes the execution of the program and operates in place of the compiler [14]. It is not classified as a static analyzer instead aims at profiling memory corruption, leaks and allocation errors. C, C++ and Java based systems are being developed using object-oriented paradigms. This paradigm of building software has reintroduced the use of recursive programming. Where in procedural languages it was considered not to be a good practice, it is now considered a good practice since objects are viewed as service providers regardless of how the service is being provided. Also compilers and equipments have progressively more resources available and are better protected against lockup. The Datrix object oriented measures assessment is focused on the control of cohesion and coupling between classes [12]. Datrix object oriented class metrics is related to size and dependency of classes. [8] [12]. 4. MAPPING PILOT MEASURES TO ISO9126 ISO/IEC-9126 defines 6 software product quality ISO 9126 Functionality Reliability Usability Efficiency Maintainability Portability

4 Figure 1: ISO9126 Quality Characteristics characteristics, e.g. fig. 1, and is the target for software product evaluation. These characteristics are then decomposed into sub-characteristics and metrics. The Work Group 6 of the ISO Joint Technical Committee is currently reviewing this level of detail and should be available this year. The set of measures that are available from the tools selected for the pilot does not cover the full characteristics of the ISO9126 Quality Model. We have not found any measurement for the Usability characteristics [5]. Figure 2 demonstrates where we intend to use each tool which is mainly the maintainability and portability aspects software. The efficiency mapping was done with Insure ++ where measures oriented on memory usage and allocation have been chosen. Insure ++ Codecheck Efficiency Portability Doc. and Rules Checking Figure 2: Pilot measures mapping to ISO9126 It is clear that response-time and resource utilization currently addressed by another set of supplier performance related tools as opposed to source code quality tools. The principal elements of good programming style are the requirements of æsthetics, maintenability and portability. 4.1 Maintainability & Portability Logiscope Datrix Maintainability The maintainability of a system is highly dependent on the source code characteristics. In systems made up of independent components with well designed boundaries, linked through normalized interfaces. A maintainer knows easily how to correct a problem since determining which component is responsible for a desired functionality is built in. The tools we have chosen for measurement of maintainability cover many aspects that are necessary for uncovering those potential problem areas. The first aspect of maintainability to be evaluated is the internal documentation of source code. We assess if documentation is done and how sufficient. Internal code documentation helps the maintenance personnel, at a detailed level, particularly to validate assumptions about what the code is actually doing. The tools report on various aspects of internal documentation into. The second aspect to be assessed is if structured programming was enforced. Structured programming is based on the premise that a problem is broken down into a finite number of components and that each component of source code has one point of entry and one point of exit. The source code analysis tools already have a good number of measures that can be used readily for both documentation and structure. These are not covered by this paper as they have been available for some years and now more commonly used [7][12][16]. The next important aspect of maintainability to be assessed are the coupling measures which determine to what extent a given routine, object and class is coupled to the rest of the system, regarding the number of global identifiers it uses (global variables references, and calls to other routines). High level of coupling, particularly between routines and global variables, indicate that the components are not independent. The OO paradigm measures provides similar information about the nature of the methods found in each class. Measures are available to assess Inheritance coupling. These measures show whether or not the object oriented concept of inheritance is used, and how well. Coupling measures from Datrix and Logiscope have been selected, e.g. fig. 3. Datrix and Logiscope provides coupling measures for routines, classes and files. The next assessment aspect of maintainability is size and complexity. Measures of the size and complexity also contribute information to form an opinion on the system maintainability and testability. Various measurements are available in the basic offering of tools on the size of the programs, as well as the complexity of the statements and of the control flow. Measures that influence the number of test cases required for unit testing, as well as the effort required to build each test case are classified as testability within the maintainability characteristic. Similarly, the use of programming guidelines or standards, by reducing the variability in the code, can help a designer to understand the different parts of the code in the system. Hence, the programming measures can be used to assess its maintainability. The conclusions that can be drawn from the measures about the maintainability of the source code are limited but can be automated easily by tools. The first assessment concerns the attributes, and includes characteristics of a class, like simple counts on the number of private, protected and public attributes. Maintainability of software is also interested in the effort required to change the code and the extent to which a given functionality is localized in a given component in the overall system.

5 TOOL MEASURE ISO 9126 CHARACTERISTICS RELIABILITY EFFICIENCY MAINTAINABILITY PORTABILITY Codecheck Array, Structure and Union Initializers Logiscope Average Coupling between object Logiscope Coupling Between Classes Logiscope Coupling between objects Codecheck Bitfields Standards Datrix Class Attributes Datrix Class Methods Datrix Class Inheritance Coupling Datrix Class Documentation Codecheck Comments in Macro definitions Logiscope Direct Recursive Datrix File Coupling Logiscope Indirect Recursive Insure++ I/O Errors Codecheck Leading Whitespace within included File Names Codecheck Lex_nonstandard Codecheck Lex_trigraph Insure++ Memory Corruption Insure++ Memory Leak Insure++ Memory Allocation Insure++ Program Optimisation Codecheck Nested Name Tags Datrix Routines Coupling Datrix Routines Dimension and Complexity Datrix Routines Programming Rule Datrix Routines Testability Codecheck System Variables Codecheck Whitespace within Preprocessor directives Figure 3 Chosen pilot measures mapped to ISO 9126 Characteristics

6 Most of the time, the changes should be localized in a few, tightly related functions and classes. Thus, again, Datrix and Logiscope coupling metrics provide useful insight in this assessment. When doing a change to existing code there is always a risk of introducing undesirable impacts else where in the system. This side effect has been found to be dependent on the coupling between source code components. The number of public attributes and methods of classes is a measurement of the potential communication channels between classes. Measures are used to assess the actual usage of these channels. Another aspect that should be considered when changing the code, is whether or not other parts of the system would require the same change. Again, the automatic identification of clones is a usefull measure and can be obtained both from Logiscope and Datrix. Finally the direct and indirect recursive identification from Logiscope is also an interesting measure since it is a potential resource problem area and is not easily maintainable. Fortunately there are significant overlaps between the topics presented so far. C, C++ and Java programmers should strive to achieve a style that is portable, easily maintainable, and elegant. Code that is technically "portable" but not easily maintained is not considered to be truly portable. A nontrivial C program that is universally portable is very rare. There are so many flexibilities in the syntax and semantics of C, C++ and Java that universal portability is practically unachievable. Therefore, designers must make their code simple and straighforward as this is the essence of portability and maintainability. Simplicity lies at the heart of maintainability and programming standards. For reasons difficult to understand, many C, C++ and Java programmers intentionally do not conform to guidelines and standards as part of their culture. This culture, which is originates back from the early 60 s, seems to value complexity, hidden logic and the use of less known operators over all other considerations. The real programmer creates code that others find impossible to understand. Whatever the reason, these values militate against simple and standard coding practices. The reasons for this fact of life are many, as presented by Abraxas Software [ 7 ], but there are three principal forces at work which underlie almost all portability issues. Force #1: Language Parochialism : which is the use of the non-standard features at the lowest levels of operations: the preprocessor and the lexical analyzer of the compiler, the file management routines of the operating system, and the bit-manipulation instructions of the machine; Force #2: Programmer Machismo: Getting all the power on one single completely undocumented line to show they have achieved something special. What they have actually created is a maintenance nightmare for someone else; Force #3: Compiler Drift: extension (nonstandard, naturally) that will ease the life of programmers and sell lots more compilers. The temptation to include these features is certainly not all bad, as it does generate an endless stream of new ideas for compilers, but it feeds directly into the other two forces. Ten measures from Codecheck [7] have been selected to evaluate the portability of the software. Although much more than portability can be achieved by Codecheck, we have limited our scope, at this time, to the portability measures. The first measure addresses the compiler automatic initialization of arrays, structure and Union. Since not all compilers permit this usage, it is not considered portable and is interesting to detect in existing software. The second measure reports on the usage of Bitfields. A first issue with bitfields is that compilers differ with respect to the maximum size of the bitfields that they allow. A second issue arise when they are not explicitly declared unsigned. The third measure concerns the inclusion of comment in a macro definition: The practice of placing a comment in a macro definition: ex: #define PASTE(c,d) c/**/d could cause the preprocessor to "paste" the actual arguments for c and d together, so that the compiler would see only the single token cd. From this practice there sprang up a whole industry of token pasting by programmers eager to exploit the loophole. Since not all compilers will be so accommodating as to ignore the embedded comment ANSI has come to the rescue with the following provisions: (a) comments within macro definitions must be treated as whitespace, and (b) programmers who feel the need to paste tokens may use the new "paste" operator (##) for the preprocessor. The fourth measure is concerned with the end of file. The last character before the end-of-file marker in a non empty source file must be a newline character. Some older compilers allow a file to terminate with any character. Consequently, the ends of files must be checked for portability to ANSI compliant compilers. The fifth portability measure is with the leading whitespace within an included File Name. Some C preprocessors do not automatically delete any whitespace characters (spaces or tabs) that precede the

7 filename within #include directives. For example: #include < ipbills.h> and #include " ipinvoice.h". Codecheck provides variables to detect this practice. The sixth measure is measuring the conformity to the standard C character set. Many C compilers allow the use of characters that are not in the standard C character set. Nonstandard characters are, by definition, not portable. Seventh measure identifies the ANSI definition of Trigraph. Trigraphs are special 3- character sequences introduced in ANSI C. Trigraphs portability issue only to the extent that older programs where trigraph will no longer compile correctly. The eight measure addresses older C code where Some older compilers permit macro expansion within string literals. This practice is permitted neither in ANSI C nor in the majority of non-ansi compilers. The ninth measure identifies the use of the leading underscore. This convention has evolved within the C community in which identifiers that are defined and used by the system (i.e. variables used by the compiler, the linker, or standard system header files) are spelled with a leading underscore character. If programmers conform to this convention by never spelling an identifier in this way, then name conflicts are prevented. Unfortunately, some programmers are unaware of this convention, and may inadvertently spell an identifier with a leading underscore. Even if the program compiles without error, it will break as soon as it is compiled on another system that happens to use one of these names. The last portability measure we have chosen from Codecheck is where some C preprocessors allow whitespace to precede the # symbol. Such use of whitespace is not portable to older compilers, although the trend is toward permitting it. Together those 10 measures give us an indication of portability of the software. 5. CONCLUSIONS AND FUTURE WORK Source code quality analysis tools can provide insight about the code being assessed, which could give Batelco a better understanding of the quality of a system before it is acquired. Portability measures have been found to be easily implementable and available. Tools could also help with measures for assessing the maintainability, efficiency characteristics of the ISO 9126 standard. In our opinion, such information can quickly be obtained by automation for an objective view of the design and standards used in a software system. We have not been able to fully assess the all the ISO9126 characteristics, especially in the usability and functionality areas. Documentation, size and complexity measures are readily available for use in commercial tools. Coupling metrics are usefull to provide insight in the assessment of design. The pilot measures we have chosen are promising to assess good object oriented design and coding of information systems. Measures can also be usefull when used to assess to what extent a system is made of components, each of them being as self-contained and independent as possible. The industrial trial of the tools and measures will be developed as a second step of our pilot. As well assessment modules [10] will be developed for documenting the evaluations and visual inspections to specific areas of the software. Finally identifying testability measures could reveal interesting for probing the completeness of the coverage of the supplier unit and system tests scenarios before acceptance. 6. REFERENCES [1] April A., Coallier F., «A model for the Assessment of Telecom Software System Development Capability», Proceedings International SPICE conference, Australia, [2] April A., Coallier F., «Trillium: A Customer-Oriented Assessment Method for Software System Development Capability», Proceedings Quebec-German workshop on software measurement, Bonn, October [3] April A., Abran A., Merlo E. «Process Assurance Audits», Proceedings 20 th International Conference on Software Engineering, ICSE 98, Kyoto, JAPAN, April 19-25, [4] ANSI, «Working Draft Proposed International Standard for Information Systems Programming Language C++», 1995, ANSI. [5] Bevans, «Measuring usability as quality of use», Jounal of Software Quality, 4, p , [6] Boloix G., Robillard P., «A Software System Evaluation Framework», Computer, Magazine of the IEEE Computer Society, December 1995, pp [7] Codecheck, «C and C++ analysis using Codecheck», [8] Datrix, «Software evolvability analysis», Bell Canada, [9] ISO 12207, «Information technology Software life-cycle processes», [10] ISO 9126, «Information technology Software product evaluation Quality characteristics and guidelines for their use», [11] Koenig,A., «C Traps and Pitfalls», Addison-Wesley, [12] Laguë B., April A., «Mapping of the ISO 9126 Maintainability Internal Metrics to an industrial research tool», SESS conference proceedings, Montréal, October 21-25, [13] Mayrand J., Coallier F., «System Acquisition based on Software Product Assessment», Proceedings of the 18th International Conference on Software Engineering, Berlin, march, [14] Parasoft, Insure++, [15] Plum T., «C Programming Guidelines», 2 nd Edition, Prentice Hall, [16] Telelogic, «Logiscope», Codecheck is a product of Abraxas Software Inc. Datrix is a Trademark of Bell Canada. Logiscope Tau is a product of Telelogic.