An ISO/IEC based Software Process Assessment in Small Software Companies

Transcription

1 , pp An ISO/IEC based Software Process Assessment in Small Software Companies Yirsaw Ayalew 1 and Kris Motlhala 2 Department of Computer Science, University of Botswana 1 ayalew@mopipi.ub.bw, 2 nmotlhala4@gmail.com Abstract Software process assessment and improvement is widely acknowledged as one of the most important means for achieving competitive and effective software industry. To ensure the production of quality software, companies usually adapt software process standards and go through certifications based on their process capability. The goal of our study is to assess the current state of software process practice in the country in order to determine their process performance and process capability. To achieve this, we conducted a process assessment study of 7 software companies following the METvalCOMPETISOFT assessment methodology. Our findings reveal process areas such as configuration management, software quality assurance, and software maintenance which require more attention for process improvement. Furthermore, the findings help in understanding the organizations current software engineering practices and identification of strengths, major weaknesses and key areas for software process improvement. Key words process assessment; process improvement; process assessment in small companies; software development in Botswana; ISO/IEC Introduction The contribution of small software companies to national economies has been widely documented. Small software companies are usually engaged in the development of individual information systems and/or develop and maintain software that is used in larger systems. Literature shows that among the software development companies in the world, more than 90% are small companies and employ much of the workforce in the software industry [1]. The percentage of such companies is even higher in developing countries and they play significant role in their respective local software industry. According to Fayad et al., [1], small companies are companies with fewer than 50 employees. Very small (or micro) companies are companies with 1 10 employees [2]. In the context of Botswana, many of the software development companies can be classified as small (or very small) as most have less than 50 employees. As software is becoming ubiquitous and many systems and operations depend on software, the demand for quality of software is increasing. To ensure the production of quality software, companies usually adapt software process standards and go through certifications based on their process capability. However, it has been found that small companies have difficulties to relate International Standards to their business needs and to justify the application of the standards to their business practices [3].This is attributed to the fact that small companies do not afford the necessary resources (e.g., in terms of number of employees budget and time), required by the standards. In addition, they do not see a net benefit in establishing software lifecycle processes. One of the reasons behind this is that the existing software process ISSN: IJSEIA Copyright c 2014 SERSC

2 standards are resource intensive and mainly applicable for big software companies. To address this issue, recently, a number of software process assessment and improvement frameworks have been proposed that targeted small (or very small) software development companies. These frameworks provide software process assessment and improvement models that are applicable to small software companies. In addition, in a recent study by Clarke and O Connor [4], it was found that there is a positive correlation between increase in software process improvement (SPI) and increase in business success in software SMEs which provides a motivation for conducting SPI. Therefore, it was suggested that the continuous application of SPI may have a role to play in supporting the achievement of business objectives. In order to determine the process areas where improvement is needed, process assessment is carried out regularly. Process assessment can be carried out by external assessors or internal assessors (i.e., self-assessment). In recognizing the potential that software can help in the socio-economic development of the country, the government of Botswana has identified software industry as one of the strategic priority areas in ICT as documented in its Research, Science and Technology plan [5]. The specific research areas that were identified are software engineering practices; software process improvement; open source methodologies; software architectures; and, human computer interface methodologies, including human language technologies. It has also identified a number of application domains such as health, education, tourism and e- government where software development is required. However, to achieve the objectives set in the plan, it is crucial to strengthen the competitiveness of software companies so that they will be able to develop quality software on time and within budget. If they fail to deliver quality products, their business and hence existence will be at risk and may lose the few customers they have. As indicated in [3], one way to mitigate these risks is to put in place proven software engineering practices. Among the factors that contribute to the development of quality software product, software process plays a significant role as documented in a number of research works (e.g., [6-10]). Furthermore, the software process assessment and improvement practices must be tailored to the context of software companies. However, to what extent the local companies have employed proven software engineering practices has never been investigated. Actually, studies that focus on software process assessment and improvement in African countries are meager. One study that we came across is a study regarding the adoption of CMMI in South Africa [11]. Therefore, this paper discusses the assessment of the current state of software development process practice in the country following the METvalCOMPETISOFT software process assessment framework. The result is expected to provide an understanding of the strengths and weaknesses of software processes implemented and an opportunity for companies to improve on their processes Selection of Process Assessment Method As indicated by different researchers (e.g., [12-15]), software process assessment methods are context sensitive. Therefore, an ideal choice could be an assessment method that has been developed with different contexts in mind and that has been evaluated and found effective/applicable in different contexts. In terms of the evaluation of assessment methods, the literature is weak in that there are not many evaluations carried out on assessment methods and that there is no widely used comparison framework that could help in choosing a particular assessment method over the others [16]. Comparison frameworks are used to compare existing methods so as to provide an insight into their similarities and differences. 122 Copyright c 2014 SERSC

3 As an aid to the selection of appropriate assessment method, we set out requirements that characterize our assessment project. The following are the requirements we used for the selection of an assessment method for our project: The assessment method should be integrated into a software process improvement framework. Even though our focus in this paper is on process assessment, we are of the view that assessment should be carried out with the objective to improve process areas where weakness is observed. Some assessment methods are designed with such objectives in mind. The assessment method should provide tool support. This is very important as the outputs arising from the assessment can be voluminous and complex, requiring tool support to comprehend and analyze them. In addition, the tool should be publicly available as most small software companies in developing countries such as in Africa are not expected to make investment on commercial tools. Only one of the existing methods claims to provide publicly available automated tool support. The assessment method should provide a detailed definition of the assessment process. As this is a research project where there are no certified assessors, it is vital to have a detailed guideline of how to carry out the assessment exercise. In addition to the requirements we set out above, we also take input from the existing few assessment method comparisons that have been made. Zarour et al., [8] compared MARES against TOPS, FAME, RAPID, SPM, EAP, and Micro-Evaluation and found MARES method as the most appropriate as it met most of the criteria they defined. The MARES method allows the selection of process areas to assess. The selection is based on the needs of the company to be assessed as to how the process can help drive the company s business to the level desired. The MARES method also supports identification of risk and improvement suggestions, which is not the case with most other methods. In another comparison, Pino et al., [13] compared METvalCOMPETISOFT with SPINI, RAPID, MARES, EvalProSoft, ADEPT, and MA-MPS and found METvalCOMPETISOFT as more appropriate than the others as provided in Table 1. Table 1. Comparison of Software Process Assessment Methods for Small Software Companies (Pino et al. [13]) Copyright c 2014SERSC 123

4 Based on the requirements we specified and the comparison provided by Pinto et al., [13], we adopt the METvalCOMPETISOFT assessment methodology. The rest of the paper is organized as follows: Section 2 presents the methodology that we followed in order to carry out process assessment in selected small software companies. Section 3 presents the results and analysis of our assessment. Section 4 provides a discussion of the main issues emanating from our assessment study. Finally, Section 5 presents the main results of our study and future work. 2. Process Assessment Approach The overall methodology of our project is guided by the assessment methodology adopted. The assessment methodology is based on the ISO/IEC standard and provides a mechanism to assess the performance and capability of software processes. The relationship between process performance and process capability dimensions, as provided in [17], is shown in Figure The METvalCOMPETISOFT Method METvalCOMPETISOFT is an assessment and improvement methodology which consists of a process for software processes assessment that aids with diagnosing software processes step by step and an assessment model for determining the capability of software processes of small organizations. The method also provides an automated tool support to carryout process assessment. It takes into consideration the characteristics and limitations of small software organizations and the fact that most small organizations do not reach level 2 of capability when they start their first improvement projects [18]. The method is based on standards ISO/IEC :2004 and ISO/IEC :2006 but considers only up to level 2 of the capability model. In addition, it attempted to make assessment easier so that it can be applied by small software organizations economically. Pino et al., [18] provided a measurement strategy that helps in assessing the performance and capability of software processes following international standard ISO/IEC The assessment focuses on process capability dimension and process performance dimension. The process performance dimension focuses on the characteristics and purposes of a specific process and is governed by a process reference model. The goal of this dimension is to help small organizations to be able to assess each one of their processes in order to identify their strong and weak points thereby providing the basis for improvement of processes. 124 Copyright c 2014 SERSC

5 Figure 1. ISO/IEC dimensional Architecture The process dimension consists of the 5 process categories to be assessed (see Figure 1). Each process category has a number of processes even though individual organization s process assessments can be limited to some processes which are relevant for the business objectives of an organization. For example, the process category Organization consists of 5 processes (e.g., Improvement, Infrastructure, etc.) each of which contains a number of base practices. The capability dimension consists of the capability scale which is used to evaluate process capability from level 0 to level 5. The higher the level, the higher the process capability will be. For the rating of levels 2-5, there are 2 process attributes as indicated in Figure 1. Level 1 has only one process attribute (i.e., Process Performance). The rating scheme of processes with respect to process attributes consists of a 4-point achievement scale: Fully achieved (F), Largely achieved (L), Partially achieved (P), and Not achieved (N). A description of the capability levels and their corresponding attributes is provided in Table 2. As our evaluation is up to capability level 2, we show only the process attributes and the corresponding ratings for capability levels 1 and 2 (see Table 3). Level Level 0 Level 1 Level 2 Table 2. Capability Levels and their Descriptions Description Incomplete process: There is general failure to attain the purpose of the process. There are no easily identifiable work products or outputs of the process Performed process: The purpose of the process is generally achieved. The achievement may not be rigorously planned and tracked. Individuals within the organizations recognize that an action should be performed, and there is general agreement that this action is performed as and when required. There are identifiable work products for the process, and these testify to the achievement of the purpose. Attribute: Process Performance Managed process: The process delivers work products of acceptable quality within defined timescales. Performance according to defined procedures is planned and tracked. Work products conform to specified standards and Copyright c 2014SERSC 125

6 requirements. The primary distinction from the performed level is that the performance of the process is planned and managed and progressing towards a defined process. Attributes: Performance Management and Work Product Management Table 3. Ratings for Capability Levels 1 and 2 Capability Level Process Attribute Rating Level 1 Process performance Largely or Fully Level 2 Process performance Performance management Work product management Fully Largely or Fully Largely or Fully As per the measurement strategy provided in [13], the scale of measurement can be represented as follows: % Fully Implemented (F) 51 85% Largely Implemented (L) 16 50% Partially Implemented (P) 0 15% Not Implemented (N) The process performance value of a process is obtained from the percentage average of the values of the base practices based on the verification of the associated work products. As per the assessment methodology (i.e., METvalCOMPETISOFT), each base practice has the same weight in a given process Subjects As the main focus of our research is to address the question of the current state of software process practice in small software organizations in Botswana, the organizations taking part in this study were chosen from the list of software development companies registered with PPADB (Public Procurement and Disposal Board). PPADB was used as the main source because all companies which would tender to carry out software development and maintenance projects for government departments and public enterprises are registered with PPADB. As the government is the main client for most software development companies in the country, the list of companies in the PPADB list is comprehensive enough. From the initial list, purposive sampling was used as only few companies are engaged in major software development projects. Twenty two (22) companies were identified and they were contacted to determine their willingness to take part in the process assessment study. All these companies operate in the capital city- Gaborone. From the 22 companies, 3 indicated that their main focus has shifted to hardware maintenance and server support. The other 9 indicated that they were not interested in taking part in the study. Ten companies accepted our request and took part by filling the process assessment questionnaire. Before the final questionnaire could be administered to all the participating companies, a pilot study was conducted on three companies from those which agreed to take part. After all the corrections and adjustments were made, the questionnaire was distributed to all the participating companies. 126 Copyright c 2014 SERSC

7 2.3. Data Collection For our data collection, we distributed questionnaire based on the assessment reference model (i.e., ISO standard) to the ten organizations. The questions used for the assessment of processes were derived from the predefined questions of the standard. These questions are based on the base practices defined for each process. The process areas used for assessment were Requirement Management, Project Planning, Project Assessment and Control, Software Implementation, Software Maintenance, Software Quality Assurance, and Configuration Management. These are process areas that have been used in many studies for process assessment in small software organizations and relevant in our context. After the questionnaire data was collected and analyzed, each participating organization was requested to arrange for site visits by the assessors to conduct interview and verification of work products related to each base practice. Seven of the companies agreed while the remaining three declined to take part. The interview and verification of work products was carried out based on the responses provided by the person responsible for process assessment for each organization. The person assigned by each organization for this assessment was aware that this project s aim was to form an overall view of the current state of each organization s processes. The data collected from the questionnaire, interview and work products related to each process provides an overall view of the state of the organization s processes. Moreover, the questionnaire data was analyzed using SPSS for correlation analysis. 3. Assessment Findings 3.1. Company Profile The table below provides information regarding the profile of those organizations participated in the study. Note that to preserve confidentiality; companies are referred to as company A, B, C etc., as shown in Table 4. Table 4. Profiles of Software Organizations Number of Employees Size of developme nt Team Years of Operation Any Previous Assessmen t? Assesseme nt Standard Organization A B C D E F G H I J > > >10 >10 YES CM MI Don NO t Kno w - Non e Don t Kno w - Non e NO YES NO Don t Kno w Don t Kno Non e YES - CM MI YES Microsoft Developm ent Copyright c 2014SERSC 127

8 The data clearly shows that most of the organizations which participated in the study are small organizations (with less than 50 employees). As the organizations which participated in this study are understood to be the main actors in the software industry in Botswana, we believe most software organizations in the country can be categorized as small. Hence, any software engineering solution towards the improvement of the software industry in the country should mainly focus on solutions tailored for small organizations. Regarding previous software process assessment experience, organizations were asked whether they have been assessed for the capability of their software processes and if so, which assessment models have been used for their assessment. Six organizations indicated that they did not have any process assessment experience. Four indicated they had been assessed for the capability of their software processes. However, one of them indicated that it was assessed against Microsoft Development, which is not a software process assessment standard. It means the person representing the organization in this process assessment study did not have proper understanding of process assessment standards. Therefore, from those 3 companies which had previous assessment, 2 indicated that they were assessed against CMMI Level 3 and the remaining 1 did not indicate the standard it was assessed against. For those 2 organizations which had an assessment experience against CMMI level 3, they did not want to disclose the outcome of the assessment. Later on (in Section 4.4), we will see to what extent their software processes have improved for those organizations which had previous process assessment experience. We expect those which had prior process assessment experience to have a better process performance and process capability compared to those which did not have. Overall, we can say that software process assessment is a new practice to most software organizations in Botswana. Moreover, none of the companies have any SPI or related certifications Process Performance The questions used for assessment in every process area were derived from the base practices of the ISO/IEC standard. The process areas used for assessment were Requirement Management, Project Planning, Project Assessment and Control, Software Implementation, Software Maintenance, Software Quality Assurance, and Configuration Management. The following table provides the processes and their descriptions. Process Requirements Management (RM) Project Planning (PP) Project Assessment and Control (PAC) Table 5. Processes and their Descriptions w Purpose The purpose of Requirements Management is to establish a common understanding between the customer and the software project of the customer's requirements that will be addressed by the software project. The purpose of Project Planning is to produce and communicate effective and workable project plans. The purpose of Project Assessment and Control is to determine the status of the project and ensure that the project performs 128 Copyright c 2014 SERSC

9 Software Implementation (SI) Software Maintenance (SM) according to plans and schedules, and within projected budgets, and that it satisfies technical objectives. The purpose of Software Implementation is to produce a specified system implemented as a software product or service. The purpose of Software Maintenance is to provide cost-effective support to a delivered software product. Software Quality Assurance (SQA) The purpose of Software Quality Assurance is to provide assurance that work products and processes comply with predefined provisions and plans. Configuration Management (CM) The purpose of the Configuration Management is to establish and maintain the integrity of all the work products of a process or project and make them available to concerned parties. After we collected and analyzed the questionnaire, we went back to the organizations (halfa day in each organization) and interviewed them about their processes and verified the implementation of their processes using the standard work products required for each base practice. From the 10 organizations which participated in the questionnaire, only 7 were willing to take part in the site visits which incorporated interviews and work product verifications. The interview questions were based on the questionnaire filled by each representative earlier in the study and followed by review of work products to verify that the organizations indeed implemented the practices. During the review of work products their quality was assessed based on the ISO/IEC standard. The degree of implementation of the base practices was evaluated by means of three types of evidence: Direct: Products resulting from an activity. Indirect: In general, documents showing that an activity has been carried out. Comments: Opinions of those involved in the process being evaluated. For the assessment of process performance, 71 questions (derived from the base practices of the selected processes) were used to establish whether a certain practice is being carried out by the organization in their software development processes. For each of the 71 questions, rating was assigned based on whether the practice is Always performed, Sometimes performed, or Never performed. That is, the options were Always, Sometimes, Never. As per the METvalCOMPETISOFT assessment methodology [13], the weights for the options were 1, 0.5, and 0 respectively. To calculate the implementation level of a process, we computed percentage average implementation of all the base practices under a particular process. For example, based on the ratings of the 10 base practices of the Requirements Management process, we compute its percentage average which can easily be converted into the measurement scale values of F, L, P, and N as provided in Section 3. The percentage average of the Requirements Management process for organization B is 43.8% which indicates a partial (P) achievement. Copyright c 2014SERSC 129

10 Table 6 shows the percentage average performance of each process area for all the organizations and table 7 shows percentage averages converted into the process rating scheme - F, L, P, and N. Table 6. Process Performance Attribute Rating (Percentage Average) of the Organizations Process Area Companies A B D F G H I RM 62.5% 43.7% 42.5% 50.0% 57.5% 57.5% 59.5% PP 91.1% 65.5% 48.8% 59.9% 88.9% 75.3% 64.6% PAC 75.0% 60.0% 35.4% 39.6% 50.0% 70.8% 52.1% SI 64.7% 56.8% 31.3% 33.9% 51.9% 76.3% 62.9% SM 81.9% 45.3% 24.9% 39.9% 43.5% 70.8% 51.8% SQA 69.1% 42.0% 25.7% 29.9% 52.4% 49.0% 64.7% CM 40.8% 15.2% 32.5% 20.8% 11.7% 43.3% 38.3% 130 Copyright c 2014 SERSC

11 Table 7. Ratings of the Organizations Process Instances for the Process Performance Attribute Process Organization A B D F G H I RM L P P P L L L PP F L P L F L L PAC L L P P P L L SI L L P P L L L SM L P P P P L L SQA L P P P L P L CM P N P P N P P The above tables show the state of the processes of the 7 organizations that participated in the study based on the Process performance attribute. In summary, it is possible to observe that there are no adequate practices related to configuration management, software quality assurance and software maintenance. The lowest performance level is in configuration management at 28.9%. These could be the areas where primary improvement is needed. With the average implementation level of 70% in Project Planning, the results showed that companies are doing well in Project Planning as compared to other process areas. A ranked list of software processes as per the performance of the organizations is provided below in Table 8. Table 8. Ranking of Processes Rank Process area 1 Project Planning 2 Software Implementation 3 Project Assessment and Control 4 Requirements Management 5 Software Maintenance 6 Software Quality Assurance 7 Configuration Management From the interviews most companies generally indicated that they have a challenge in formalizing their processes as it requires time and resources. Most organizations indicated that performance of certain practices is dependent on the customer or the type and size of projects being carried out. For example, with regard to Project Planning process, representative of company D indicated that depending on the nature of the project, sometimes some steps are ignored or skipped. Comparing the Configuration Management performance of software organizations in Botswana with small software organizations in other countries (e.g., Finland [19], Australia [20]), we find significant differences. A RAPID assessment carried out in Queensland, Australia shows that configuration management is one of the highly performed processes. From the assessment of the degree of importance of different processes to the organization in Botswana, we found that Configuration Management is less important compared to other Copyright c 2014SERSC 131

12 processes assessed and all companies clearly indicated that they rarely carry out configuration management. Comparing our requirement management process result with the results of other studies such as Micro-Evaluation [21] assessment and an assessment of companies in Jordan [22], companies in Botswana have a lower level of requirement management practice. In the assessment of Jordanian companies, the results showed a greater performance in requirement management. With regard to Software Quality Assurance process, all companies except one failed to produce quality plan, quality standards, test plans, test results, audit records, review records, and coding standards which are necessary to ascertain implementation of software quality assurance process practices. The results show that reviews, project measures, assessments and/or audits were not carried out systematically. It seems that this process is not properly practiced by small software organizations in general as reported in other studies (e.g., [20], [23]. These studies reported that there was a general weakness pertaining to activities related to quality management process. With regard to software implementation process, the performance of all organizations in this process was not adequate. One of the areas where poor performance was observed is regarding testing. Companies seem to have prepared test plans to a certain extent but they did not implement the plan during testing. There were no documented test cases, test scripts, and test results. Only 2 companies were able to produce evidence of their test cases and test results. In another study [23], it was found that small software organizations performed well in software development practices including testing Process Capability For determining process capability of each organization, we used El Emam and Jung [17] aggregation scheme which provides a scheme for converting ratings on the individual attributes into a capability level. As we are interested in process capability up to level 2, in addition to the process performance attribute, we need to assess performance management and work product management attributes. However, as can be seen from Table 3, for the process of an organization to achieve level 2 of process capability, the process performance attribute has to be fully achieved which is not the case in most of our organizations. Therefore, in terms of process capability, these organizations processes are either at level 0 or level 1. Actually, there is only one organization which achieved a process capability of 2 for their Project Planning process (organizations A) (see Table 9). Table 9. Process Capability of Organizations Process Organization A B D F G H I RM PP PAC SI SM SQA CM Copyright c 2014 SERSC

13 During the interview, all representatives indicated that they implemented Project Planning practices even though some of the work products were not adequate. One company showed evidence of good management and planning as work products for different processes were readily available. All the companies indicated that they rarely perform Configuration Management practices. One representative indicated that they use a software application called Microsoft Studio TFS for configuration management. TFS provides source code management, reporting, requirements management, project management, etc Process Performance Vs Previous Process Assessment From purely a statistical perspective, our sample of 7 organizations may not appear sufficient for correlation analysis. However, taking into consideration our context (where there are few software organizations) and employing appropriate statistical measure for small size, we wanted to assess the relationship between previous assessment experience and current process performance. To test for the existence of correlation between the two variables (i.e., organization current process performance and previous assessment experience, we used Student T-Test (also known as independent T-Test)). The mean process performance of organizations which did assessment before is 55.87% while the mean process performance of organizations which never did assessment is 48.13%. That is a difference of 7.74% between the two groups. The p-value of which is greater than the cutoff of 0.05 indicates that there is no statistically significant difference between the mean adoption level of companies which did assessment before (mean = 93.48, Standard Deviation = 9.735) and those which did not (mean = 76.50, Standard Deviation = 9.439). In other words, the p-value leads to the rejection of the null hypothesis which states that there is no process performance difference among the organizations which had previous assessment experience and those which did not have previous assessment experience. The result suggests that organizations previous assessment has a positive influence in improving process performance. The result specifically suggests that when companies do assessment, they are likely to experience improved process performance on the next assessment. It is for this reason that process assessment needs to be carried out regularly. 4. Discussion First, we want to see the process performance of Botswana software companies in relation to Clarke et al., [24] pyramid of processes for SPI in SMEs. The pyramid shows processes used in SMEs arranged from the least common (bottom of pyramid) to most common (top of pyramid). In the pyramid, the least common process areas for SPI in software SMEs include risk management, software verification, software architectural design, etc. The most common process areas include software construction, project planning, configuration management, software requirements analysis, etc. They indicated that processes at the top of the pyramid are common processes from which SMEs derived significant benefits. In our case, configuration management is the least practiced process area which is not in agreement with the pyramid. This mismatch could be due to lack of awareness of the best practices in the software companies we assessed. On the other hand, we found that project planning and software implementation are relatively properly practiced which are in accordance with the pyramid. The other issue is related to the appropriateness of the 7 chosen processes for assessment. All the 7 processes have been used by different assessment methodologies targeted for small software companies. For example, Project Planning, Project Assessment & Control, Quality Copyright c 2014SERSC 133

14 Assurance, Configuration Management and Requirement Management processes have been used in ADEPT and OWPL assessment methodologies. Quality assurance, software implementation, configuration management and requirement management processes seem to be the most commonly used in almost every assessment methodology proposed for small companies. In addition, looking at Clarke et al. pyramid, we find that software implementation, project planning, and configuration management are the most common areas for SPI in software SMEs appearing at the top of the pyramid. Software maintenance and project planning & control appear at tier 2 of the pyramid. Generally, it appears that software implementation and project management are the critical processes for small software companies. As indicated in [25], a new standard for very small software companies (VSE) focuses on software implementation and project management elements to define the basic profile appropriate for a set of common VSE characteristics. They emphasized the importance of project management stating that The main reason to include project management is that the core business of VSEs is software development, and their financial success depends on successful project completion within schedule and on budget, as well as on making a profit. Another observation regarding the process performance of the organizations in our study is that they found it difficult to deal with the documentation part of the processes. Being small companies, they have limited resources and they indicated that documentation is time consuming and hence costly. Therefore, the question is how can these small companies be supported by process assessment and improvement standard that requires minimal documentation without sacrificing the quality of their products? The answer to such questions may lead to exploring different options. One potential area could be integration of agile methods into process standards designed for small companies such as ISO/IEC For example, the work by Irrazabal et al., [26] advocates for such integration of process assessment and improvement in small companies. Another option could be an automatic acquisition/extraction of evidences of process implementation instead of requiring organizations to avail work products for assessors which is documentation intensive. Such an approach could relieve people responsible for process improvement so that they can focus on the process assessment activities and the assessment could become less expensive. Along this direction, there are some preliminary works (e.g., [27, 28]). Limitations There are some limitations to our study worth discussing here. The first limitation relates to the number of organizations that participated in our study which stands at 7. Even though every effort was made to increase the sample size, it was constrained by two factors. One is that there are not many software development companies in the country. Second, from the existing few companies, some were not interested in to take part in this process assessment exercise. However, comparing our sample size to the sample size used in similar studies in the process assessment domain (i.e., assessment in small companies), it is not that bad. For example, a study by [19] used 8 samples, a study by [13] used 8 samples and still another study by [29] used 4 samples. Nevertheless, we need to be cautious in generalizing the findings of our study. The second limitation relates to the experience of the process assessors. The two assessors involved in the study are not certified assessors as required in some process assessment and improvement approaches. However, the assessors have adequate background in software engineering to carry out process assessment. In addition, the methodology that was adopted provides a detailed guideline for process assessment in small companies which was one of the criterions for the adoption of the methodology. Moreover, it is becoming increasingly important for process assessment and improvement approaches to be flexible enough so that 134 Copyright c 2014 SERSC

15 they can be used for self assessment by small companies. For example, in the ISO/IEC standard by Laporte and Vargas [25], they indicated the need for process improvement guidelines to be flexible and easy to use for beginners on the adoption of practices of international standards focused on processes to support their software development projects needs. 5. Conclusion In this paper we have adopted METValCOMPETISOFT software process assessment method to assess software development processes of 7 small software companies in Botswana. We conducted software process assessments to examine the state of processes in organizations based on practices as defined by the ISO/IEC standard. In those organizations which participated in our study, most of the processes and their corresponding practices were not adequately implemented. It appears that process implementations are mostly ad hoc which may suggest that there is lack of software assessment and process improvement expertise. It was observed that process areas such as software maintenance, configuration management, and quality assurance were poorly implemented and hence process improvement plans need to address such areas. On the positive side, project planning and software implementation were the process areas where systematic implementation was observed and these could be considered as the processes where local organizations have shown strength. We have also observed that those organizations which had performed process assessment in the past performed better than those which did not have any process assessment experience. This indicates the need for regular process assessment which can serve as a basis for process improvement as those areas identified as weaknesses will usually be the targets for improvement. We believe that the results of our assessment could be the first step towards creating a better understanding of software process assessment and improvement in the country thereby allowing software organizations to work towards improving quality of their products. As a continuation of this, process improvement activities need to be initiated in those organizations which participated in this study. As indicated in [19], software process assessment and improvement is widely acknowledged as one of the most important means for achieving competitive and effective software industry. As the government of Botswana has already identified software industry as one strategic area for socio-economic development, further research and development is required to come up with appropriate methods and techniques that can assist in strengthening the productivity and competitiveness of the local software industry. Suitable process assessment and improvement methods will play a significant part in this direction. In addition, the establishment of the Botswana Innovation Hub (BIH) where software development is one focal area is expected to provide more incentive for the advancement of software development in the country. In this regard, this paper can serve as a basis in trying to find appropriate software engineering solutions to improve the development of quality software which can increase the competitiveness of local software development companies. As a future work we plan to explore alternative process standards such as ISO/IEC which has been designed to assist and encourage small software development companies in assessing and improving their software processes. In addition, we plan to extend our work to other African countries in order to gain more insight on the needs of small companies with regard to software process standards. It is worth mentioning that literature on software process assessment and improvement in small software development companies in Africa is Copyright c 2014SERSC 135

16 non-existent. In this direction, we plan to develop an adaptable process assessment and improvement approach with an automated tool support. References [1] M. E. Fayad, M. Laitinen and R. P. Ward, Software Engineering in the Small, Communications of the ACM, vol. 43, (2000), pp [2] ESI, ICT-European Software Institute, (2013). [3] ISO/IEC-TR , Software Engineering - Lifecycle profiles for Very Small Entities (VSEs), (2011). [4] P. Clarke and R. V. O Connor, The influence of SPI on business success in software SMEs: An empirical study, The Journal of Systems and Software, vol. 85, (2012), pp [5] CSIR, Botswana National Research, Science and Technology Plan, Ministry of Communications, Science and Technology, (2005). [6] D. Chevers and E. W. Duggan, A Modified Capability Framework for Improving Software Production Processes in Jamaican Organizations, The Electronic Journal on Information Systems in Developing Countries, vol. 30, (2007), pp [7] T. Dyba and O. Skogstad, Measurement-based Software Process Improvement, Telektronik, vol. 1, (1997), pp [8] M. Zarour, J. Desharnais and A. Abran, A Framework to Compare Software Process Assessment Methods Dedicated to Small and Very Small Organizations, International Conference on the Software Quality - ICSQ'07, (2007). [9] J. Jarvinen, Measurement Based Continuous Assessment of Software Engineering Processes, University of Oulu, (2000). [10] S. B. Basri and R. V. O'Connor, Organizational Commitment towards Software Process Improvement: An Irish software VSEs case study, International Symposium in Information Technology, Kuala Lumpur, Malaysia, (2010) June 15-17, pp [11] D. J. Cohen, Assessing the Business Value of Software Process Improvement using CMMI in South Africa, University of Pretoria, (2008). [12] R. V. O'Connor and C. Y. Laporte, Using ISO/IEC to Harness Process Improvement in Very Small Entities, Systems, Software and Service Process Improvement (Communications in Computer and Information Science), vol. 172, (2011), pp [13] F. J. Pino, C. Pardo, F. García and M. Piattini, Assessment Methodology for Software Process Improvement in Small Organizations, Information and Software Technology, vol. 52, (2010), pp [14] C. G. V. Wangenheim, S. Weber, J. C. R. Hauck and G. Trentin, Experiences on establishing software processes in small companies, Information and Software Technology, vol. 48, (2006), pp [15] H. Oktaba, F. García, M. Piattini, F. Ruiz, F. J. Pino and C. Alquicira, Software Process Improvement: The Competisoft Project, Computer, vol. 40, (2007), pp [16] A. Sampaio, E. Gray, M. Martins and I. B. Sampaio, Towards an Internal Numerical Taxonomy of Software Process Assessment Methods, International Conference on Software Process and Product Measurement (MENSURA'06), Cadiz, Spain, (2006), pp [17] K. El-Emam and H.-W. Jung, An Empirical Evaluation of the ISO/IEC Assessment Model, Journal of Systems and Software, vol. 59, (2000), pp [18] F. J. Pino, F. l. Garcı a and M. Piattini, Software Process Improvement in Small and Medium Software Enterprise: A Systematic Review, Software Quality Journal, vol. 16, (2008), pp [19] I. Saastamoinen and M. Tukiainen, Software Process Improvement in Small and Medium Sized Software Enterprises in Eastern Finland: A State-of-the-Practice Study, European Conference on Software Process Improvement, Trondheim, Norway, (2004), pp [20] A. Cater-Steel, Low-rigor Rapid Software Process Assessments for Small Software Development Firms, Australian Software Engineering Conference (ASWEC 04), Queensland., Australia, (2004), pp [21] C. Y. Laporte, A. Renault, J. Desharnais, N. Habra, M. Abou El Fattah and J. Bamba, Initiating software process improvement in small enterprises: Experiment with micro-evaluation framework, International Conference on Software Development, Reykjavik, Iceland, (2005), pp [22] A. B. M. Isawi, Software Development Process Improvement for Small Palestinian Software Development Companies, Engineering Management, (2011). [23] N. Habra, S. Alexandre, J.-M. Desharnais, C. Y. Laporte and A. Renault, Initiating Software Process Improvement in Very Small Enterprises: Experience with a Light Assessment Tool, Information and Software Technology, vol. 50, (2008), pp Copyright c 2014 SERSC

17 [24] P. Clarke, R. V. O Connor and M. Yilmaz, A Hierarchy of SPI Activities for Software SMEs: Results from ISO/IEC based SPI Assessments, 12th International Conference on Software Process Improvement and Capability determination, Palma, Spain, (2012), pp [25] C. Y. Laporte and E. P. Vargas, The Development of International Standards to Facilitate Process Improvements for Very Small Entities, Software Process Improvement and Management:Approaches and Tools for Practical Development, Edited S. S. M. Fauzi, M. H. N. Nasir, N. Ramli, and S. Sahibuddin, IGI (USA), (2012), pp [26] E. Irrazabal, F. Vásquez, R. Díaz and J. Garzás, Applying ISO/IEC 12207:2008 with SCRUM and Agile Methods, SPICE 2011, Dublin, Ireland, (2011), pp [27] G. Grambow, R. Oberhauser and M. Reichert, Automated Software Engineering Process Assessment: Supporting Diverse Models using an Ontology", International Journal on Advances in Software, vol. 6, (2013), pp [28] I. Garcia and C. Pacheco, Toward Automated Support for Software Process Improvement Initiatives in Small and Medium Size Enterprises, Software Engineering Research, Management & Applications, (2009), pp [29] H. Oktaba, F. Garcia, M. Piattini, F. Ruiz, F. J. Pino and C. Alquicira, Software Process Improvement: The Competisoft Project, IEEE Computer, vol. 40, (2007), pp Authors Yirsaw Ayalew, is a Senior Lecturer at the Department of Computer Science, University of Botswana. He received PhD degree in Computer Science from the University of Klagenfurt, Austria in Prior to joining the University of Botswana, he worked as Assistant Professor at the Addis Ababa University, Ethiopia. He teaches both undergraduate and graduate courses mainly in the area of Software Engineering. His research interests include software testing & debugging, requirements engineering, software maintenance, software process and mobile applications in health care solutions. Mokeresete Kris Motlhala, is an Information Communication and Technology Lecturer and System Administrator/Analyst at the Institute of Health Sciences, Molepolole, Botswana. He received his B.Sc. Degree in Computer Science from the University of Botswana in 2004 and a B.A. Degree in Socio-Informatics (Information Science) from the University of Stellenbosch, South Africa in He recently completed his M.Sc. in Computer Science, University of Botswana. His interests include open source software development, agile software development, software process improvement, requirements engineering, health informatics, embedded systems, wireless sensor networks and agent based systems. Copyright c 2014SERSC 137

18 138 Copyright c 2014 SERSC