RAPID SOFTWARE PROCESS ASSESSMENT TO PROMOTE INNOVATION IN SMES Giovanni A. Cignoni 1 Abstract Products and processes innovation is a key-topic for the competitiveness of small and medium enterprises operating in the Information Technology field. The ESPINODE network links 18 European projects with the common goals of diffusing Software Process Improvement concepts and supporting the enterprises already involved in improvement experiments. This paper describes the methodology developed by the TOPS project, the ESPINODE for Central Italy, to use rapid software process assessment as a way to promote innovation. Rapid assessments are offered as a free service to small and medium enterprises that join TOPS as target enterprises. The offer of an assessment is a powerful way to both diffuse software best practices and propose actual improvement paths to the enterprises. Assessment data are collected by TOPS for a survey about software process maturity in Italy: the paper reports the first results of this survey. 1. Introduction Information Technology will be one of the key factors driving progress in the 21st century it will transform the way we live, learn, work, and play. Advances in computing and communications technology will create a new infrastructure for business, scientific research, and social interaction. As indicated in the findings of the President s Information Technology Advisory Committee (PI- TAC) [7], research in the Information Technology (IT) field is considered of fundamental importance for the progress of an industrial economy. At the end of its work, the committee suggested the US President to strongly increase the funding for research in the IT field for the next five years and the increase is measured in billion of dollars. 1 CESVIT SpA TOPS, ESPRIT Project no. 27977; e-mail: cignoni@cesvit.it
But to actually foster such progress, research should be coupled with industrial innovation. Excellent research results, to become widely spread technologies and provide all the benefits cited in the PITAC report, have to be transferred to competitive enterprises. In this paper we present how, within the context of an initiative funded by the European Commission, we use a rapid software process assessment methodology to promote innovation in IT Small and Medium Enterprises (SMEs). Rapid assessment meetings are offered as a free service to enterprises. The offer of a free assessment is a way to both diffuse process quality concepts and propose actual improvement paths to the enterprises. Moreover, assessment data are collected and used for a survey about software process maturity in Italy: here we report the first results of this survey. This work is part of the ESPINODE initiative [4] funded by the European Commission within the ESPRIT-ESSI programme. The ESPINODE network links a set of 18 projects (17 in Europe plus one in Israel). All projects aim to diffuse Software Best Practices (SBP) and Software Process Improvement (SPI) concepts and methodologies, and to support enterprises already involved in Process Improvement Experiments (PIEs) funded by the ESPRIT-ESSI programme. The ESPINODE for Central Italy is TOPS (TOPS stands for Toward Organised Processes in SMEs) [11]. Section 2 describes specific goals and characteristics of TOPS. Section 3 presents the methodology that TOPS developed for rapid software process assessment. Section 4 collects the results of the first assessments we made; they depict the status of software process maturity in Central Italy. Section 5 draws some conclusions about this experience, both regarding the assessed average maturity level and the results in terms of effective promotion of innovation initiatives on the part of the enterprises. 2. TOPS, the ESPINODE for Central Italy The TOPS ESPINODE derives from a proposal made by CESVIT in response to a call of the European Commission. CESVIT SpA is the Agency for High Technology in Tuscany: its mission is to promote technological innovation in order to improve the competitiveness of the Region. Within CESVIT, the Centre for Software Quality has specific competencies in the fields of software engineering and quality control applied to IT industries. CESVIT provides services concerning technical advising, technology transfer, training and assistance in preparing proposals for European and national calls for project funding.
TOPS is the ESPINODE for Central Italy, providing services to the enterprises located in Tuscany, Liguria, Latium, Umbria and Marche. CESVIT is the main contractor of the project and Innova SRL is the partner for the zone of Rome. The target audience to which TOPS offers its services includes the following entities [14]: attached PIEs, enterprises that have on-going SPI projects funded within the ESPRIT-ESSI initiative; such enterprises have in TOPS the reference ESPINODE; other PIEs, enterprises that have performed or completed SPI experiments and are willing to verify and disseminate the results of the experiments or to give continuity to their SPI activities; regional industry, enterprises that are interested in improving their software process and that are looking for financial support from the European Commission or from national or local programmes or funds; ESPINODE network, the European network made by all the ESPINODEs, which acts as a mean to share information so as to identify and propose European initiatives that provide support to IT industry. TOPS activities targeted to PIEs generally comprehend technical advising and training, support to the dissemination, and assistance in the relationship with the European Commission. As for all the ESPINODEs, the generic goal of TOPS towards regional industry not already involved in PIEs is to promote awareness regarding the benefits of SPI. More specifically, TOPS activities regarding regional industry comprehends: public events, TOPS organises awareness events about SPI and SBP benefits, training events aiming to presents methodologies for process assessment and improvement as well as techniques and tools for software development, and information events to present European initiatives to support innovation; direct contacts, TOPS asks the regional industries to join the project as target enterprises to receive a visit for a rapid assessment of their software process; the assessment serves both as an awareness action and as a practical way to identify and plan specific actions to improve software processes. Other activities comprehend demonstrations of software tools, voice and telematic help desk, web services and traditional publications. However, with respect to regional industry, the specific char-
acteristic and the main goal of TOPS is contacting the enterprises, first inviting them at public events and then assessing their software processes. Assessment visits are an interesting chance for the enterprises that can try what is a software process assessment. Moreover the assessments will result in a collection of data about the maturity of software processes in Central Italy. From these data and from direct suggestions, which are possible to obtain only in close contacts with the companies, it is possible to identify, with some reasonable confidence, the actual needs of regional industry and use them to direct the future initiatives of the European Commission. Public events and assessment visits are free for the enterprises, i.e. funded by the project. Other services, as specific training courses or advising requests, are out of the TOPS scope and they will follow the market prices. CESVIT guarantees the maximum commitment to meet the enterprise demand. Maximum commitment is also guaranteed in the finding of financing opportunities (at local, national or European level) to support the enterprise in the costs of the requested services. In this perspective, we naturally regard to TOPS as a marketing initiative for CESVIT services. But we also think that services provided out of TOPS scope demonstrate a success of TOPS itself: an enterprise that decides to invest in a SPI activity is the most meaningful result of the project promotion action. 3. Rapid Assessment Methodology The policy of TOPS towards regional industry is contacting the enterprises. To implement such policy, we decided to use awareness and training events as a way to establish the first contact with the enterprises. During the events, we present the TOPS initiative and the opportunity of a rapid software process assessment is offered to the enterprises. The subsequent assessment programme has the following specific results: to stimulate interest in software process assessment and improvement; to contribute to the definition of specific improvement plans; to collect data and statistics about software process maturity in regional industry; to create a database of companies really interested in SPI and SBP.
Moreover, in the search for a methodology to use for the rapid assessments, we had to cope with two other requirements: to be compliant with the Regional Industry Survey (RIS); this survey, which is based on a common questionnaire, is an initiative regarding SPI awareness promoted by the ESPINODE network; to stay within project budget; we wanted to perform a considerable number of assessment, both to obtain a wide awareness action and to have a good statistical sample, therefore single assessments had not to be too expensive. In search for a good compromise between accurate results and low costs, we decided to develop our own methodology. Pre-existent tools, such as, for instance, Bootcheck [3], or Process Advisor [8], need a full day or more to perform an assessment, while we want to stay in half a day and have time for explaining and discussing SBP and SPI concepts with the enterprise people. Our methodology is based on a two parts questionnaire to be used for the assessment. Questionnaire is compiled by TOPS experts that interview one or more enterprise representatives, but Part 1 (QP1) is made by phone while Part 2 (QP2) is compiled within a direct audit meeting. The Questionnaire is public and it is available on the TOPS Web site [13]. 3.1. Assessment questionnaires and tools QP1 is to be made by phone: we contact the enterprise asking to answer some preliminary questions. QP1 is organised in 5 sections. Sections 1 and 2 collect general data about the enterprise and the person responsible for the questionnaire. Section 3 defines the enterprise characteristics such as company type, dimensions, turnover, and so on. Section 4 and 5 are specific of the RIS. Section 3 assesses enterprise goals for the near future and awareness about SPI methodologies and benefits. Section 5 evaluates the knowledge of European initiatives to support enterprises in their SPI experiments. For QP1, answers are predefined and the enterprise has to pick the most applicable one. QP2 is the base for the assessment and is compiled during the meeting with the enterprise. It is organised in three sections. Section 1 is aimed to collect some general data about the software development unit such as dimensions, type of life-cycle, type of software products, and so on. Section 2 of QP2 assesses the organisational and technological characteristics of the software development unit. While in some cases questions regard issues that are considered SPB (as, for instance, project planning or risk evaluation), in fact the goal of this section is to evaluate how much
the enterprise is well organised and technologically mature. In the analysis of the assessment results, we do not use these data to define the software process maturity; instead we use them to give an evaluation of the improvement capability of the enterprise. A well defined organisation is a strong advantage in the enactment of a software process improvement action. From another point of view, technology is often a factor of change: for instance when the introduction of a development methodology pushes forward the formalisation of the development process. Section 3 of QP2 finally is the true software process assessment. Assessment is made with respect to three specific processes: requirement analysis, verification and test, and joint review. With respect to the ISO 15504 TR standard [6] they correspond to the processes ENG.2 (develop software requirements), ENG.5 (integrate and test software), and CUS.4 (perform joint audits and reviews). We restricted the assessment to these processes mainly for time constraints: we want to have time for discussion rather than to submit the enterprise to a long and rapid succession of questions. We also want to avoid the risk of difficult topics, as for instance configuration management, that may need explanation diverting the focus of the meeting. Moreover, the chosen processes, because they directly impact product quality and customer satisfaction, are often naturally seen as the ones in which improvement is more important and then they are the most suitable for small improvement projects. In this sense, we privileged the use of the assessment meeting as a mean to pick out improvement opportunities and suggest a consequent experiment. For each specific process there are five questions, each one corresponds to a maturity level and summarises the best practices of that level i.e. the generic practices of the ISO standard. Answers to questions of Sections 2 and 3 of QP2 are of yes/no kind, where to answer yes there must exists documentation that proves that processes actually follow the best practice presupposed by the question. Moreover, to the yes/no answer it is added a percentage value. In the yes case, the percentage represents how much the activity is performed as stated; in the no case, percentage represents how much the practice is not stated but de facto applied, for instance by customer request or personal initiative. While yes 100% is the best situation and no 0% the worst, yes 0% (a practice stated but completely disregarded) and no 100% (a praxis established but not formalised) are still possible answers. We use the percentage value to have a measure of the distance between the actual situation and the ideal one. The measure can be interpreted in terms of needs of internal diffusion (yes with low percentages) or needs of procedure formalisation (no with high percentages). Final results of the assessment are expressed in terms of actual maturity level, potential maturity level, and improvement capability. Actual maturity level is defined as in the ISO 15504 TR standard
and requires full application of stated practices. Potential maturity instead captures the situations in which maturity level is not reached because of lack in procedure application or because it exists as a praxis that demonstrates the awareness of the best practice. Finally, improvement capability, also expressed as a score in the range 0-5, depends by percentages (how much close to the full application) and by the organisational and technological levels of the enterprise interpreted as ease factors for improvement. Organisational and technological level are also calculated as assessment results and expressed as a score in the range 0-5, but they are functional to improvement capability and they are not related to the software process maturity level. To support the analysis and the presentation of the survey results, we set-up a database in which data from questionnaires are collected. The database is used to perform queries and to support the data presentation via the TOPS Web site. To aid the assessment enactment we developed a tool based on an MS Excel spreadsheet. The spreadsheet is used to collect data and to calculate results directly during the assessment meeting. The tool is free and it is available for download from the TOPS Web site. 3.2. Contact and meet the enterprise We think that process assessment based on a self-compiled questionnaire may lead to wrong results. When the questionnaire regards process quality, i.e. the quality of the way in which people are used to work, the self-compilation often results in a lack of realism reflecting a good or bad vision closely related to the feeling of the compiler. Using self-compilation at the end of a workshop about SPI and SBP, while it is a way to perform many assessments in one time, it is not a solution to improve accuracy. The event, instead of being a guide to questionnaire compilation, in many cases works as an amplifier of feelings, resulting in very bad or very good visions. Last, using selfcompilation we lose the opportunity to establish a true contact with the enterprise. Then, we decided that TOPS skilled professionals should compile the questionnaire by interviewing the enterprises, explaining the questions and discussing the answers. In spite of its higher cost, this technique is more accurate. Moreover it is less tedious and more useful for the enterprise, which has the opportunity to meet expert people without any direct cost. We use awareness and training events organised within TOPS as a mean to contact regional industries potentially interested in the target enterprise role. Even if event participation is for free, an enterprise invest some of their person/time to participate. This is an indicator that the enterprise is potentially interested in a future SPI experiment.
Then we contact by phone the enterprise for QP1 compilation. QP1 is an occasion to talk about SPI and SBP and serves also to enterprise selection. TOPS funds are limited and then we want to accurately aim our effort towards enterprises. We do not want to bother enterprises with initiatives they are not interested in: the assessment has to be an effective opportunity for their business. The acceptance of the target enterprise role must be formally signed. Formally joining the project, the enterprise accepts that assessment results will be published in aggregate form. While this is in the interest of the enterprise because it is a way to make some benchmarking, it is a privacy issue that needs a clear agreement. To prepare the meeting with the enterprise we have to select the TOPS professional that is more skilled to perform that assessment. In this perspective data previously collected by QP1 are a useful guide to address specific needs of the enterprise and match them with the right skilled professional. More generally the TOPS professionals that meet the enterprises should comply with the following requirements: expertise in software engineering and software process; non trivial experience in software process assessment; good knowledge of ISO 9000 & C (an inevitable hot topic). On the part of the enterprise, we require that the software development manager or quality system manager participates to the assessment. We also suggest the attendance of people involved in software development as well as people involved in customer/supplier relationships, as, for instance, project managers and salesmen. 4. First Results By the end of the project in May 2000, TOPS plans to collect at least 100 QP1 to be used for RIS and 50 rapid assessment based on QP2. Here we present some general considerations about the results of the first half of our task: 50 QP1 and 25 QP2. Full aggregate data and analysis are available as project technical reports and regularly updated on the TOPS Web site [10, 12].
4.1. QP1: Regional Industry Survey The large majority of enterprises are very small both in terms of employed people and turnover. This reflects the typical situation of central Italy enterprises and, in fact, TOPS has SMEs as a specific target audience. The geographical distribution of the collected questionnaires reflects the localisation of CESVIT (in Florence) and Innova (in Rome), with the majority of the interviewed enterprises located in Tuscany, Latium and Liguria. For the end of the survey we plan to obtain a more uniform distribution over the area assigned to TOPS. It is remarkable that the large majority of the interviewed enterprises has expansion (60%) or innovation (20%) as their goals for the near future. On the other part, very few of them want to make their processes more efficient (10%) or consolidate their market position (10%). This depicts a situation in which SPI can rely on very few resources: all enterprise efforts are directed to new targets and not towards consolidation and improvement of the already reached ones. As a natural consequence major barriers to SPI are costs and resources (78%); noticeably very few enterprises do not invest in SPI because of distrust in its benefits (7%). When and where some SPI initiative is performed (46%), the most pursued goal is efficiency (61%). consolidation 10% efficiency 10% innovation 20% expansion 60% competences 15% resources 37% distrust 7% costs 41% Enterprise goals for the near future Barriers to investments in SPI activities Figure 1 In conclusion, it seems that enterprises, probably as a consequence of Euro and Y2K, have lot of business opportunities and logically want to exploit them as much as possible. Quality is put in background and SPI is principally viewed as a way to be more profitable. Figure 1 shows data about enterprise goals for the near future and general barriers to investments in SPI activities.
A negative remark regards the poor awareness of the ESPINODE initiative. Enterprises are interested in ESPINODE services but do not know the existence of the network nor of the ESSI initiative (76%). In other words, ESPINODEs have to introduce themselves and the whole programme. 4.2. QP2: Rapid Assessment As for QP1, the large majority of assessed enterprises are very small both in terms of employed people (88% has less than 100 employees) and turnover (58% does not reach a turnover of 1 million Euro). This partially explains the poor score achieved as average actual maturity level (0.44): small enterprises with few people do not feel the need of internal organisation and process control. Moreover, many of them rely on internal or local customers (40%), a situation that presuppose an elevate grade of confidence often based on personal relations. Enterprises are not pushed to invest in solutions to improve customer confidence by enterprise organisation and quality control: 50% of assessed enterprises does not have at all a quality system and another 20% had only approached its definition. Generally enterprises obtain better evaluations in organisational than in technological aspects (average organisational level is 2.62 where average technological level is 1.82). Going to the details, poor technology evaluations mainly depend by the scarce adoption of tools for project management, version and configuration control, test automation and software measurement. In many cases, regarding pure product technology, enterprises are strongly committed in innovation, for instance about RAD and CASE tools (76%). So we have to read the result not as poor technology at all, but as poor technology in support of process and quality control. As previously anticipated, results about actual maturity are very poor. The very large majority (80%) of enterprises scores a 0 level: they have no documentation evidence that activities are performed. Average value is 0.44 and the maximum scored level is 3. Potential maturity scores, while not exciting, are better, with an average value of 1.67 and a maximum score of 3.67. This reflects a usual situation: there is a general knowledge of SBP but, instead as being received as enterprise standards, they are applied by demand, for instance in the most critical projects. To tune our assessment tool, 5 of the 25 enterprises we assessed were selected between enterprises that had a certified quality system and already had some rigorous SPI experience, for instance in an ESSI-PIE or in a SPIRE experiment [9]. Some of them already received a formal assessment using
CMM or Bootstrap. In these cases our assessment showed an average error of ± 0.18 in terms of potential maturity level. 5 4 All enterprises Average actual maturity levels Selected enterprises 3 2 1 0 ENG.2 ENG.5 CUS.4 Overall Figure 2 Because the maturity scores of the selected enterprises are far better that the average, we use these results to stimulate the regional industry: while the average is low there are some excellent performers that demonstrate that SPI is a valuable investment. Figure 2 and 3 show respectively the assessed actual and potential average maturity levels. 5 4 Average potential maturity levels All enterprises Selected enterprises 3 2 1 0 ENG.2 ENG.5 CUS.4 Overall Figure 3
5. Conclusions The action that TOPS is performing with regional industry is very interesting because it allow us to really meet the enterprises. The choice of three levels of contact with enterprises (events, regional industry survey and rapid assessment) is very promising both for the aim of providing useful services to enterprises and for the target of surveying SPI application in regional industry. In the development of questionnaire we followed the ISO 15504 standard: results are expressed in maturity levels, the audit is performed with respect to selected base practices and assessment is based on the application of generic practices. As a rapid assessment, our methodology is necessarily approximate. Due to time constraints we have to limit the scope and the accuracy of the assessment. We want to perform the assessment meeting in half a day and we need time for discussion. More in detail, declared limitations of our assessment are: scope of the assessment is limited to three processes belonging to two of the five ISO 15504 process categories; accuracy of the assessment is limited to the answers given by the enterprises; in other words, we do not look for sources of evidence or probe for process management indicators. Moreover, it is possible to map QP2 answers in the ones of the SBP Questionnaire [2] realised within the ESPRIT-ESSI programme. In this perspective results of our assessments can be directly used in the survey promoted by the European Commission. From the survey data and from experience gathered in the many meetings with the enterprises, we can identify the following issues that summarise the situation about SPI in the TOPS interest area: very small enterprises with local customers do not feel a strong need for process quality, however many enterprises declare expansion as their next future goal; expansion can become a motivation for more organised processes; maturity level is very low, however it is possible to argue that SBP are in general known, even if not applied; in this perspective, awareness and training events may be targeted to the benefit of adoption of SBP as enterprise standards; technologies to support process control are not applied; this can be viewed as another target for training events.
The rapid assessment initiative is a clear success: we already have booked assessments nearly to the end of the project. For the CESVIT mission (promote innovation by providing services), TOPS is also a valuable marketing tool. About 35% of the target enterprises asked specific advising services to enact improvement activities identified during the assessment meeting. More than half of them accepted our services, in some cases partially funded by local initiatives to promote innovation. We regard to this as one of the most important achievements of TOPS. After the last project review, encouraged by the European Commission, we are detailing an agreement with the other Italian ESPINODEs, namely AINODE [1] and INSIEME [5], to use our approach for an Italy wide survey. The approach was also suggested to WG4, the working group about marketing solutions inside the ESPINODE network. 6. Acknowledgements First of all, it is mandatory to thank our target enterprises: they made the survey possible. Many people contributed to the development of the rapid assessment methodology, in particular we want to thank Francesco Capanni that also performed many of the assessments as did also Giulio Ravizza, while Maurizio Campanai and Paolo Martinez contributed to the paper with useful suggestions. We must also thank the projects and organisations that co-operated in the realisation of awareness and training events and therefore helped TOPS in the establishment of first contacts with enterprises: we cite here the ISA-EUNET project and its partner ENEA, the LSD project and its main contractor Sogea SCpA, the SPIRE project and its partner Etnoteam SpA, the Mcube project, the MA.IN. project, Sitech SCRL, and Tecnomarche SRL. 7. References [1] AINODE, ESPINODE for Northern Italy, Project Web site at http://www.assintel.it/ainode.nsf. [2] EUROPEAN SOFTWARE INSTITUTE, 1997 Software Best Practice Questionnaire Analysis of Results, ESI Technical Report, available on the Internet at http://www.esi.es/publications/reports/tr-sbpqaor3.html, December 1997. [3] EUROPEAN SOFTWARE INSTITUTE, BootCheck Overview, Internet document available at http://www.esi. es/bootcheck/, October 1998. [4] EUROPEAN SYSTEMS AND SOFTWARE INITIATIVE, ESPINODE: A new scheme to implement take-up measures, Internet document available at http://www.cordis.lu/esprit/src/enodegen.htm, September 1998. [5] INSIEME, ESPINODE for Southern Italy, Project Web site at http://www.insieme.tno.it. [6] INTERNATIONAL ORGANISATION FOR STANDARDISATION, ISO/IEC TR 15504:1998 Information technology Software process assessment Parts 1-9, 1998.
[7] PRESIDENT S INFORMATION TECHNOLOGY ADVISORY COMMITTEE, Information Technology Research: Investing in Our Future, Report to the President, available on the Internet at http://www.ccic.gov/ac/report/, February 1999. [8] PRESSMAN, R.S., Process Advisor, RSP&A Inc., 1992. [9] THE SPIRE PROJECT TEAM, The SPIRE Handbook: Better, Faster, Cheaper Software Development in Small Organisations, The European Commission, December 1998. [10] TOPS PROJECT, Analysis of Requests from Local Industry, TOPS Technical Report TR12, available on the Internet at http://www.cesvit.it/tops/processosw/osservatorio.htm, May 1999. [11] TOPS PROJECT, ESPINODE for Central Italy, Project Web site at http://www.cesvit.it/tops/. [12] TOPS PROJECT, Report on the Results of Preliminary Assessment of Needs of Local Industry, TOPS Technical Report TR5, available on the Internet at http://www.cesvit.it/tops/processosw/osservatorio.htm, May 1999. [13] TOPS PROJECT, SME&SPI Questionnaire, Annex of TOPS Technical Report TR5, available on the Internet at http://www.cesvit.it/tops/processosw/tr5-anx1.htm, February 1999. [14] TOPS PROJECT, TOPS Terms of Reference, TOPS Technical Report TOR, available on the Internet at http://www.cesvit.it/tops/docs/tops-tor.pdf, June 1998.