Software SMEs unofficial readiness for CMMI Ò -based software process improvement

Transcription

1 DOI /s Software SMEs unofficial readiness for CMMI Ò -based software process improvement Javed Iqbal 1 Rodina Binti Ahmad 1 Mohd Hairul Nizam Md Nasir 1 Mahmood Niazi 2 Shahaboddin Shamshirband 3 Muhammad Asim Noor 4 Ó Springer Science+Business Media New York 2015 Abstract The goal of software process improvement (SPI) is to improve software processes and produce high-quality software, but the results of SPI efforts in small- and medium-sized enterprises (SMEs) that develop software have been unsatisfactory. The objective of this study is to support the prolific and successful CMMI-based implementation of SPI in SMEs by presenting the facts related to the unofficial adoption of CMMI level 2 process area-specific practices by software SMEs. Two questionnaire surveys were performed, and 42 questionnaires were selected for data analysis. The questionnaires were filled out by experts from 42 non-cmmi-certified software SMEs based in Malaysia and Pakistan. In the case of each process area of CMMI level 2, the respondents were asked to & Javed Iqbal javediqbal6000@siswa.um.edu.my Rodina Binti Ahmad rodina@um.edu.my Mohd Hairul Nizam Md Nasir hairulnizam@um.edu.my Mahmood Niazi mkniazi@kfupm.edu.sa Shahaboddin Shamshirband shamshirband@um.edu.my Muhammad Asim Noor asim_noor@comsats.edu.pk Software Engineering Department, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia Information and Computer Science Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia Computer System and Technology Department, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia Computer Science Department, COMSATS Institute of Information Technology, Islamabad, Pakistan

2 choose from three categories, namely below 50 %, %, and above 75 %. The percentages indicated the extent to which process area-specific practices are routinely followed in the respondents respective organizations. To deal with differing standards for defining SMEs, the notion of the common range standard has been introduced. The results of the study show that a large segment of software development SMEs informally follows the specific practices of CMMI level 2 process areas and thus has true potential for rapid and effective CMMI-based SPI. The results further indicate that, in the case of four process areas of CMMI level 2, there are statistically significant differences between the readiness of small and medium software enterprises to adopt the specific practices of those process areas, and between trends on their part to do so unofficially. The findings, manifesting various degrees of unofficial readiness for CMMI-based SPI among SMEs, can be used to define criteria for the selection of SMEs that would be included in SPI initiatives funded by relevant authorities. In the interests of developing fruitful CMMI-based SPI and to enhance the success rate of CMMI-based SPI initiatives, the study suggests that ready or potential SMEs should be given priority for SPI initiatives. Keywords Software process improvement Small- and medium-sized enterprises CMMI adoption trends Specific practices Process areas CMMI level 2 1 Introduction Software is becoming a more significant part of our day-to-day lives. This fact underlines the need to encourage software development organizations to produce high-quality software. The quality of software products relies on the quality of the software process; therefore, software development organizations strive to improve their software processes (Hwang 2009). Process improvement recommends taking a number of measures to pave the way toward enhanced software processes that result in high-quality software delivered on time. Process performance, stability, compliance, capability, and improvement are critical aspects on the way to SPI (Florac and Carleton 1999). The sustainable success of a software development organization is not possible without continuous process improvement. To reach software process maturity, organizations pursue a number of SPI models, such as SPICE, BOOTSTRAP, ISO 9000, Six Sigma, CMM, and CMMI; they have achieved significant improvements by adhering to CMM and CMMI, as well as PSP (Nasir et al. 2008). For instance, IBM Global Australia Application Management Services has effectively reduced costs by using CMMI (Nasir et al. 2008). However, attaining CMMI maturity levels is time-consuming; as according to a software engineering institute report, the average time required to reach CMMI level 5 (SCAMPI) is months (SEI 2006). In view of the time required to reach higher maturity levels, SPI is often deemed costly. Organizations must dedicate considerable resources to it over long period of time. At times, even firms equipped with the resources necessary for process improvement are unable to achieve the expected results (Niazi et al. 2008). Efforts are being made worldwide for facilitating SPI to accelerate the progress of the software industry. Globally, SMEs constitute the principal segment of the software development industry (Pino et al. 2008). Therefore, this study focuses only on SMEs. With respect to the role of SMEs in transforming a developing country into a developed country, the Malaysian government

3 has recently announced a new definition of SMEs and approved over seven billion USD in funding to boost its growth (Newspaper 2013). According to a report on Pakistan s software industry, the industrial advancement necessary for implementing a CMMI framework and achieving the required maturity levels in that country has been slow (Noor et al. 2007), even though the Pakistan Software Export Board has offered several incentives and provided financial support to IT companies to adopt the CMMI framework. One reason for this slow rate of evolution is that IT companies with real potential for successful and rapid CMMI-based SPI implementation have not been targeted, since the relevant authorities are unaware of the trends to follow CMMI-based SPI practices unofficially. This study explores such adoption trends and helps to identify potential software SMEs for CMMI-based SPI. In this context, the following research questions (RQs) are important: RQ1: Are there software SMEs that have not achieved any formal CMMI level but unofficially or informally follow CMMI-based SPI practices? RQ2: What are the trends for the unofficial adoption of CMMI-based SPI practices in SMEs? RQ3: What is the impact of the process area-wise unofficial adoption trends on SMEs? RQ4: Are unofficial adoption trends useful for software SMEs with respect to CMMIbased SPI implementation? RQ5: With respect to various process areas, are there any significant differences between small and medium software enterprises in their unofficial trends to adopt CMMIbased SPI practices? Practices refers to specific practices of CMMI process areas, and trends means the percentage or extent to which specific practices in process areas are unofficially adopted. If an SME has not attained any CMMI level but informally complies with the specific practices of CMMI process areas according to the defined criterion for a potential SME, then we can say that the SME is unofficially ready for a CMMI-based SPI. To answer these RQs, we have studied the trends in the adoption of CMMI level 2 process area-specific practices. We have used the questionnaire method. A number of earlier studies regarding CMMI level 2 have also been based on questionnaires Garcia and Gonzalo (2007), Yucalar and Erdogan (2009) and Garcia et al. (2008). 2 Related work This section is divided into five subsections to differentiate the focus of past studies. Section 2.1 presents studies regarding SPI in small and medium software enterprises; Sect. 2.2 sheds light on issues of SPI implementation through CMMI; Sect. 2.3 is about SPI in SMEs using CMMI; Sect. 2.4 looks at CMMI level 2; and Sect. 2.5 discusses CMMI level 2 implementation in SMEs. 2.1 SPI in software SMEs Small organizations implement SPI as successfully as large organizations, and they can also attain the same level of organizational performance. For effective SPI, small organizations should focus on employee involvement and the exploration of new knowledge (Dybå 2003). The process model PRISMS has been suggested as a guide for SPI in SMEs (Allen et al. 2003), and a Web-based tool is available for managing SPI in small enterprises

4 (García et al. 2012). Pino et al. (2008) performed a systematic literature review regarding SPI implementation efforts in SMEs and related issues. Process improvement is critical for enhancing SMEs, but a lack of the required resources, high implementation costs, and challenging project completion deadlines create obstacles for SPI implementation. To handle SPI implementation issues faced by SMEs, a number of noteworthy methodologies have been discussed and compared (Mishra and Mishra 2008). An approach has been proposed to facilitate rapid SPI implementation in SMEs with limited budgets (Alexandre et al. 2006). The critical factors for SPI in small software firms have also been reviewed (Khokhar et al. 2010). A methodical literature review has pointed out the SPI models and relevant techniques employed by Web development SMEs (Sulayman and Mendes 2009); the review found that SMEs face budget constraints and tight project completion deadlines in SPI and that they lack long-term strategies. An initial framework has been proposed to explore the factors important to SPI success in Web SMEs (Sulayman et al. 2012). To help SMEs take the initiative for SPI, work has been done on automated SPI support (Garcia and Pacheco 2009). Due to certain limitations, small enterprises are unable to assess SPI. A methodology has been introduced to deal with this situation (Pino et al. 2010). In order to improve the process of software development in small companies, different stages of a framework have been discussed using CMMI as the SPI model and extreme programming as a development method (Al-Tarawneh et al. 2011). 2.2 Issues in SPI implementation through CMMI A number of studies concentrate on the reasons why organizations do not adopt CMMI. Typically, the reasons include small company size, cost, lack of time, and the use of another SPI approach (Staples et al. 2007). For software SMEs, the main hindrance is cost, and contributing factors include bureaucratic attitudes, the documentation required to record information, verbal communication, and the creative and flexible nature of SMEs (O Connor and Coleman 2009). However, according to another study (Khurshid et al. 2009), the reasons are cost, uncertainty about the benefits of SPI, small company size, and other organizational priorities. CMMI issues have been identified, and remedies have been proposed for enhancement (Liou 2011). The lessons learned from CMMI failures have also been discussed (Gefen et al. 2006). This study reports that CMMI methodology affects the quality of both off-the-shelf and customized products. Continuous representation of CMMI is required to determine the priorities of process areas for the implementation of process improvements. A model has been presented and validated to support this procedure (Huang and Han 2006). 2.3 SPI in SMEs through CMMI Huang and Zhang (2010) have analyzed problems related to CMMI implementation in SMEs, and they present key points for finding solutions to these problems. CMMI is useful for small companies when the company s environment is taken into account (Cepeda et al. 2008). Appraisal, infrastructure support, and deployment are three areas that challenge the application of CMMI in small companies (Garcia 2005). A study by Staples and Niazi (2010) discusses the motivation and readiness of two small companies to apply CMMI; the study considers why SMEs adopt SPI, what the motivators and de-motivators are in SPI for SME employees, and how ready SMEs are for SPI implementation. Based on CMMI, a framework has been introduced to facilitate SPI in SMEs (Zhang and Shao 2011). The feasibility of merging the scrum method and CMMI has been analyzed to resolve issues

5 with CMMI-based SPI in software development SMEs (Lina and Dan 2012). To facilitate SMEs for the rapid attainment of CMMI levels 2 and 3, a framework combining Six Sigma and a tailored version of CMMI has been suggested (Habib et al. 2008). 2.4 CMMI level 2 inspection CMMI level 2 has been scrutinized from many perspectives. Some authors have focused on the dependencies between CMMI level 2 process areas and also between related specific practices (Chen et al. 2008). Similarly, others have analyzed dependencies that exist among all process areas of a certain CMMI level and among all CMMI process areas (Monteiro et al. 2010). They have also identified the effect on level 2 dependencies when some level 3 process areas are introduced at the time as level 2 process areas are implemented. Another study (Niazi and Babar 2009) has acknowledged that three specific practices related to the Requirements Management process area (CMMI level 2) are high perceived value practices. An approach has been recommended to reconcile validation and verification practices (CMMI level 3) with CMMI level 2 (Monteiro et al. 2009). Based on a level 2-specific practice about managing the requirements changes, a model has been proposed to facilitate requirements change management (Niazi et al. 2008). 2.5 CMMI level 2 implementation in SMEs Several studies focus on CMMI level 2 from the standpoint of CMMI-based SPI implementation in SMEs. CMMI level 2 -specific practices can be mapped with the scrum method (Diaz et al. 2009). Due to the link between agile software development and CMMI- DEV, software development SMEs can utilize lightweight methods in order to achieve CMMI level 2. To support CMMI-based SPI in SMEs, a questionnaire-based technique has been proposed to enhance the process of software acquisition related to the Supplier Agreement Management process area of CMMI level2 (Garcia et al. 2008). Based on the outcomes areas, one study (Chen and Staples 2007) has categorized CMMI level 2 process area-specific practices to realize the perceived values of these specific practices for SMEs. Some authors have appraised six process areas of CMMI level 2 within six software SMEs and have associated low, medium, or high perceived values with the specific practices by using the Express Process Appraisal (EPA) method (Wilkie et al. 2005). Once they grow from small-to-medium size, companies naturally follow a greater number of CMMI practices (Lester et al. 2010). The EPA method was developed by staff at the Centre for Software Process Technologies (CSPT). To apply EPA, a set of base questions is prepared using CMMI in the case of each process area. The questions are posed during appraisal interviews conducted to gather information about the use of relevant Specific Practices. The set of process areas may vary from company to company. For another set of process areas, a different set of base questions must be used. Depending on the answers given to the base questions, additional questions are asked as the interview proceeds. Written notes are made to record the responses, and certain documents are additionally reviewed (Lester et al. 2010). The entire process depends on personal expertise and experience. The present study uses questionnaire-based approach, which relies on wellestablished and defined CMMI-specific practices. To explain the specific practices, the details of respective subpractices were also provided. We believe this ensured that the evaluated SMEs were treated in a consistent manner to obtain results that were unbiased and comparable with one another. Furthermore, we incorporated various SME standards.

6 This study presents a simple and effective way to identify and select potential SMEs to include in sponsored SPI initiatives. As a result, wastage of resources will be avoided and the success rate of CMMI-based SPI efforts will be increased, which in turn will quickly improve the overall situation of SPI in software SMEs. 3 Research methodology Keeping in view the purpose of research, the survey research method was used to gather data on the unofficial adoption of CMMI-based SPI practices in SMEs. This survey research method is used to accumulate both qualitative and quantitative types of data (Niazi et al. 2010). A combination of data-gathering techniques such as interviews and questionnaires or any one of these techniques may be employed in the survey research method (Lethbridge et al. 2005). For the present study, we collected data from Pakistani and Malaysian SMEs, but for broader acceptability and applicability of the results, we wanted to incorporate other SME standards. Therefore, we accommodated the SME size defined by the Australian Bureau of Statistics and the European Union (EU) SMEs standard. By the definition of SME Corporation Malaysia, the number of employees in a small enterprise is 5 29, and in a medium enterprise, (Malaysia SC 2013). According to the EU s definition of SMEs, a small company has employees and a medium company has employees (Lester et al. 2010); while according to the Australian Bureau of Statistics, the number of employees in a small company is 5 19, and in a medium company, (Niazi and Babar 2009). The SMEs development authority in Pakistan states that a small enterprise has employees, and a medium enterprise has employees (Dasanayaka 2008). In the case of all four SMEs standards, we have considered only small- and medium-sized companies, but not micro-companies. 3.1 Common range standard for SMEs A company with 34 employees is small according to the SMEs standards of Pakistan and the EU, but is defined as a medium company by Malaysian and Australian standards. Likewise, a company with 85 employees is a medium company by Pakistani, Australian, and EU standards for SMEs, but not according to the Malaysian standard. To mitigate such conflicts, we have introduced the notion of common range standard as described below: Consider that n is the number of SMEs standards under discussion (4). SL 1,SL 2 SL n are the lower limits, and SU 1,SU 2 SU n are the upper limits of small company size ranges according to various standards for SMEs. Similarly, ML 1,ML 2 ML n are the lower limits, and MU 1,MU 2 MU n are the upper limits of medium company size ranges by different standards for SMEs. Now SL ¼fSL 1 ; SL 2...SL n g; SU ¼ fsu 1 ; SU 2...SU n g; ML ¼fML 1 ; ML 2...ML n g; MU ¼ fmu 1 ; MU 2...MU n g: Algorithm 1 determines the Common Range Standard for SMEs.

7 Algorithm 1: Finding the Common Range Standard. Input: SL represents the lower limits of small company size ranges. SU represents the upper limits of small company size ranges. ML represents the lower limits of medium company size ranges. MU represents the upper limits of medium company size ranges. Data: LLS, ULS, LLM, ULM, SCS, MCS. SET LLS:= FindLargestElement (SL); SET ULS:= FindSmallestElement (SU); SET LLM:= FindLargestElement (ML); SET ULM:= FindSmallestElement (MU); /* LLS: Maximum value in SL. ULS: Minimum value in SU. LLM: Maximum value in ML. ULM: Minimum value in MU. */ SET SCS: = {LLS, ULS}; SET MCS:= {LLM, ULM}; /* SCS: Defines the size of small companies according to the Common Range SMEs Standard. MCS: Defines the size of medium companies according to the Common Range SMEs Standard. */ By applying Algorithm 1, SCS is 10 19, which helps to determine small companies size acceptable according to all four SMEs standards. Similarly, MCS is From this, we define medium companies size in line with four distinct SMEs standards. Thus, according to the common range standard, the number of employees in a small company is 10 19, and in a medium company, the number of employees ranges from 50 to 75. By following this standard, a small or medium software company according to any one of the four given standards is compatible with the remaining three different standards. 3.2 SMEs size groups We divided the SMEs into groups of various sizes based on the common range standard and the four given standards. The small company groups were 1 9, 10 19, 20 35, and 36 49, while the medium company groups were 20 35, 36 49, 50 75, 76 99, , and The small companies with employees and medium companies with employees who perform IT-related activities constituted our required SMEs. They were approached randomly. 3.3 Questionnaire surveys We performed two surveys, from June 2, 2013 to August 30, 2013, using questionnaires containing mainly closed-ended questions. Our respondents, who belonged to Malaysian and Pakistani SMEs, were project managers, software engineers, team leaders, quality

8 assurance managers, system analysts, technical managers, and senior managers with 5 15 years working experience. Two rounds of a pilot study were conducted. The first round was carried out to enhance the layout of the questionnaire, assess language comprehension, and estimate the time needed to fill out the questionnaires. The second round was performed to ensure that the alterations were in accordance with the recommendations given in the first round. Before deploying the questionnaires, we searched for companies that had software development and other IT-related activities such as maintenance and provision of IT infrastructure as their main business; that were non-cmmi-certified; and that met our common range standard for SME size. We identified 110 companies through telephonic contacts, s, and by searching websites. In the first questionnaire survey, representatives of the 110 identified companies were formally asked about their organizations size and CMMI certification. Keeping in view our criteria of authentic information about company size and CMMI certification, selecting an equal number of small and medium companies, and the responses to the questionnaire, 52 SMEs were selected for the second survey. In the second survey, we sent questionnaires to 52 professionals from 52 Malaysian and Pakistani SMEs with no CMMI certification. Out of 52 SMEs, 26 were small- and 26 were medium-sized, with and employees, respectively. The questionnaire was divided into two parts. The first section aimed at obtaining information about each respondent s job, experience and company. The next section aimed to gather data about the unofficial adoption of the specific practices of CMMI level 2 process areas. The questionnaires filled out by the professionals were cross-verified with the respective company representatives. For the sake of confidentiality, the names of participating companies are not mentioned. Furthermore, the data about companies and respondents are presented in aggregate form, so that individual organizations and respondents cannot be identified. All the participants were well aware of the fact that they were taking part in the survey for scientific research. Out of 52 questionnaires, 28 were distributed via and the remaining 24 were circulated and filled out during face-to-face meetings. The study was facilitated through the use of a semi-supervised approach (Pfleeger and Kitchenham 2001). In this approach, the survey objectives, questionnaire format, and pertinent queries are explained during face-toface meetings or using various other techniques such as computer-assisted telephone interviewing (CATI). After the respondents are satisfied, they are given ample time to complete the questionnaire. Out of 28 questionnaires, 26 were successfully returned through owing to continuous contact with the respondents and having obtained their prior consent. Thus, 50 (24? 26) questionnaires were answered in total. From the 50 questionnaires, 42 were selected (one from each company) for purpose of analysis, keeping in view the respondent s job responsibilities and experience, as well as the parameter of selecting an equal number of small and medium companies (21, 21). The second questionnaire survey mentioned 54 specific practices that were related to the seven process areas of CMMI-DEV (V1.3) level 2. For each process area, the practitioners were requested to select what percentage of a process area-specific practices were routinely followed in their respective organizations. The three percentage categories were: below 50 %, %, and above 75 %. Table 1, provided with the second questionnaire survey, helped the respondents choose relevant categories. For a proper understanding of the specific practices, the details of corresponding subpractices were provided as well.

9 3.4 Criteria for determination of trends In a previous study, CMMI level 2 process area-specific practices were categorized as having high, medium and low perceived values based on more than 70 %, between 70 and 40 %, and \40 %, respectively (Wilkie et al. 2005). In the present study, analogous percentages have been used, with slight variations (75 % instead of 70 %, and 50 % instead of 40 %) to reflect the number of specific practices in level 2 process areas. The details in Table 1 helped practitioners to select an appropriate percentage category for each process area, depending on the numbers of specific practices followed in their respective organizations. For example, Table 1 shows that if there are ten specific practices in a process area and an SME follows four or less specific practices, the respondent should choose the \50 % category. If between five and seven specific practices are followed, then the appropriate category is %, and for 8 10 specific practices, the category to choose would be more than 75 %. If an SME unofficially or informally follows more than 75 % of specific practices in the case of at least 50 % of the process areas (at least four out of seven) of CMMI level 2, then that SME is called Fully Potential for CMMI-based SPI. If an SME informally follows % of specific practices in the case of at least 50 % process areas of CMMI level 2, then that SME is called Semi-Potential for CMMI-based SPI. Table 1 Possible number of specific practices that can be followed in a process area and relevant percentage categories No. of specific practices in the various process areas of CMMI level 2 Possible No. of specific practices that can be followed unofficially Relevant percentage categories 4 0, 1 \50 % 2, % 4 More than 75 % 5 0,1,2 \50 % % 4, 5 More than 75 % 6 0,1,2 \50 % 3, % 5, 6 More than 75 % 7 0,1,2,3 \50 % 4, % 6, 7 More than 75 % 8 0,1,2,3 \50 % 4, 5, % 7, 8 More than 75 % 10 0, 1, 2, 3, 4 \50 % 5, 6, % 8, 9, 10 More than 75 % 14 0, 1, 2, 3, 4, 5, 6 \50 % 7, 8, 9, % 11, 12, 13, 14 More than 75 %

10 3.5 Criteria for selection of SMEs with potential for CMMI-based SPI SMEs that have not achieved a CMMI level but unofficially or informally follow at least 50 % of the specific practices in the case of at least 50 % of the process areas of CMMI level 2 are considered potential SMEs for CMMI-based SPI. The 50 % criterion has been used previously (Cox et al. 2009). There are seven process areas of CMMI level 2. To analyze the results, four has been considered to represent 50 % of the seven total process areas. If an SME does not fulfill the basic criterion for becoming a potential SME, then that SME is called Non-potential for CMMI-based SPI. 4 Results and discussion In this section, the results are discussed in order to answer the research questions one by one. 4.1 CMMI-based unofficial SPI implementation in SMEs Table 5 provides detailed results. Small companies were given identification numbers (IDs) ranging from S 1,S 2 to S 21, and medium-size companies were given IDs ranging from M 1,M 2 to M 21. Table 2 summarizes the results. From the data presented in Table 2, it is evident that there are SMEs that have not attained CMMI certification at any level but nonetheless informally follow CMMI-based SPI practices. This provides the answer to RQ1. Table 2 SMEs that informally implement CMMI level 2 process areas specific practices S. No. CMMI level 2 process areas No. of specific practices in process areas No. of SMEs following \50 % of specific practices in process areas No. of SMEs following % of specific practices in process areas No. of SMEs following more than 75 % of specific practices in process areas Small Medium Small Medium Small Medium 1 Configuration management Measurement and analysis Project monitoring and control Project planning Process and product quality assurance 6 Requirements management Supplier agreement management

11 4.2 Trends to unofficial adoption of CMMI-based specific practices According to Table 5, ten medium-size enterprises have not achieved any level of CMMI certification but follow more than 75 % of the specific practices in case of at least 50 % of the process areas (at least four out of seven) of CMMI level 2. These medium companies have the full potential for CMMI-based SPI. There are 11 medium software companies that have not achieved any level of CMMI but implement % of specific practices in at least 50 % of the process areas of CMMI level 2. These medium companies are Semipotential companies for CMMI-based SPI. In our sample, there is no example of a small enterprise that has not achieved any level of CMMI but implements more than 75 % of the specific practices for at least 50 % of the process areas of CMMI level 2. Therefore, there is no small company in our sample that has the full potential for CMMI-based SPI. The 15 small companies have not achieved any level of CMMI but do implement % of specific practices for at least 50 % of the process areas of CMMI level 2. Such small companies are Semi-potential companies for CMMI-based SPI. The results show that six small enterprises do not meet the required criterion to become CMMI-potential companies. They are therefore called Non-potential companies for CMMI-based SPI. Thus, based on the extent or percentage of the informal usage of specific practices related to CMMI level 2 process areas, the SMEs can be divided into three categories: Fully potential, Semi-potential, and Non-potential. This provides the answer to RQ2. Table 3 shows these facts. Figure 1 shows the percentages of small and medium companies in the Fully potential, Semi-potential and Non-potential categories. 4.3 Process area-wise unofficial adoption trends impact on SMEs Section 4.3 provides the answer to RQ3. Less than 50 %, %, and above 75 % are the three categories used to classify the adoption of the specific practices of CMMI level 2 process areas. Figures 2, 3, and 4 depict, with respect to process areas, comparisons of the numbers of small and medium software companies that follow \50 %, %, and above 75 % specific practices, respectively. The seven process areas of CMMI level 2, namely Configuration Management, Measurement and Analysis, Project Monitoring and Control, Project Planning, Process and Product Quality Assurance, Requirements Management and Supplier Agreement Management, are denoted as CM, MA, PMC, PP, PPQA, REQM, and SAM, respectively. Figure 2 shows that for the \50 % category, the number of small companies is higher than that of medium companies for all of the seven process areas of CMMI level 2. This means that more small software companies adopt \50 % specific practices as compared Table 3 Fully potential, Semi-potential and Non-potential SMEs for CMMI-based SPI No. of Fully potential SMEs No. of Semi-potential SMEs No. of Non-potential SMEs Small Medium Small Medium Small Medium Total

12 Fig. 1 Small and medium companies contribution to the three SMEs categories 100% Medium Small 0% 90% 80% 42% 70% Percentages 60% 50% 40% 100% 100% 30% 58% 20% 10% 0% 0% SMEs' Categories No. of SMEs No. of Medium Companies No. of Small Companies CMMI Level-2 Process Areas Fig. 2 No. of SMEs following \50 % specific practices of CMMI level 2 process areas to medium software companies. Such small and medium software companies may be Nonpotential companies for CMMI-based SPI, but it is highly likely that a greater number of small companies are Non-potential in comparison with medium companies.

13 No. of SMEs No. of Medium Companies No. of Small Companies 2 0 CM MA PMC PP PPQA REQM SAM CMMI Level-2 Process Areas Fig. 3 No. of SMEs following % specific practices of CMMI level 2 process areas No. of SMEs No. of Medium Companies No. of Small Companies 0 CM MA PMC PP PPQA REQM SAM CMMI Level-2 Process Areas Fig. 4 No. of SMEs following more than 75 % specific practices of CMMI level 2 process areas Figure 3 shows how for the % category, out of seven process areas for four process areas (namely CM, MA, PPQA, and SAM), the number of medium companies is higher than the number of small companies. For the remaining three process areas (namely PMC, PP, and REQM), the number of small companies is greater than the number of medium companies. This shows that in the case of CM, MA, PPQA, and SAM process areas, more medium companies than small companies follow % specific practices. However, in the case of PMC, PP, and REQM process areas, more small companies adopt % specific practices as compared to medium companies. Such small and medium software companies may be Semi-potential companies for CMMI-based SPI. Figure 4 illustrates that for the more than 75 % category, the number of medium companies is higher than the number of small companies for all seven process areas of CMMI level 2. This proves that more medium companies follow the more than 75 % specific practices, as opposed to small companies. Such small and medium companies may

14 be Fully potential for CMMI-based SPI, and it is highly likely that a greater number of medium companies are Fully potential in comparison with small companies. 4.4 Usefulness of unofficial or informal adoption trends of CMMI-based SPI Figure 5 represents the percentages of Non-potential, Semi-potential and Fully potential SMEs. Figure 5 shows that 62 % of SMEs are Semi-potential and 24 % are Fully potential for CMMI-based SPI, whereas 14 % of SMEs have no potential for CMMI-based SPI implementation. Semi-potential SMEs follow % of specific practices in the case of at least 50 % of the process areas of CMMI level 2, and Fully potential SMEs adopt more than 75 % of the specific practices in the case of at least 50 % of the process areas of CMMI level 2. Therefore, in view of the potential SMEs criterion, 86 % SMEs (62 % Semi-potential? 24 % Fully potential) have the potential for CMMI-based SPI implementation. Table 3 indicates that out of 36 potential companies, 21 are medium and 15 are small, or 58 % potential companies are medium and 42 % are small. We know that 86 % of SMEs unofficially follow a sufficient number of CMMI level 2 process area-specific practices. Unofficial adoption can act as a catalyst for CMMI-based SPI implementation in SMEs. As a large portion of SMEs are already employing a suitable number of specific practices associated with CMMI level 2 process areas, this segment has real potential for fruitful and sustainable CMMI-based SPI. To prepare a pool of CMMIcertified software SMEs in the least possible time and to avoid wastage of resources, we recommend that this untapped potential in SMEs must be utilized. The relevant authorities should concentrate on the SMEs that informally follow an adequate number of CMMIbased SPI practices. Such potential SMEs can be identified through a survey similar to the one utilized in this study. For potential organizations, the CMMI appraisal or certification is straightforward and quick (Jackelen 2007). SPI has positive effects on the business of Non-Potential SMEs for CMMI-based SPI 14% Fully-Potential SMEs for CMMI-based SPI 24% Semi-Potential SMEs for CMMI-based SPI 62% Fig. 5 Percentages of three SMEs categories

15 software development SMEs in terms of financial and non-financial benefits (Clarke and O Connor 2012). This provides the answer to RQ4. The primary goal here is to make the relevant authorities aware that, for CMMI-based SPI initiatives, potential SMEs should be given additional attention to achieve quick and effective SPI. We are not claiming that potential SMEs, in this state, are sure to pass any assessment. Rather, if authorities focus on such SMEs for instance, in terms of financial aid or training it is highly likely that after the necessary preparations they will attain a certain level of CMMI-based SPI. 4.5 Differences between unofficial trends in small and medium software companies Chi-squared tests were performed to examine whether the differences between small and medium software enterprises trends for adopting specific practices of CMMI level 2 process areas, are statistically significant. Table 4 presents the results Measurement and analysis Statistically significant differences have been found between the trends of small and medium software enterprises with respect to following the specific practices of the process area Measurement and Analysis v 2 (2, N = 42) = , p \.01. Medium enterprises are more likely to follow % specific practices, and small enterprises follow \50 % specific practices. Measurement and analysis is aimed at establishing and sustaining the measurement capability to be used for obtaining the information needed for management. This involves the description of objectives and measures along with the specification and implementation of data analysis and data management techniques (Team CP 2010). The measurement and analysis process area renders management with the perceptibility necessary to formulate measurement guidelines that enhance efforts for project management (Procaccino and Verner 2006; Procaccino et al. 2005). Well-defined measurement guidelines assist project managers in identifying problems, their size and scope, and in finding alternative ways of dealing with them (Goldenson et al. 2003). The majority of project managers are of the opinion that using the right estimation techniques is important for the success of any project (Procaccino et al. 2005; Pereira et al. 2008). According to our findings, small software companies very frequently follow\50 % of the specific practices of the measurement and analysis process area. Therefore, such small software companies should concentrate more on measurement and analysis activities to manage projects effectively and increase their projects success rate. Medium companies frequently follow more than % of the specific practices of the measurement and analysis process area and thus exhibit better project management than small companies Project planning Statistically significant differences have also been observed with respect to following the specific practices of the process area Project Planning v 2 (2, N = 42) = 6.290, p \.05. Here, medium enterprises are more likely to follow more than 75 % of the specific practices, whereas small enterprises are more likely to follow more than 75 % of the specific practices in about half of cases, with further specific practices falling in the % range. The objective of project planning is to prepare and ensure the execution

16 Table 4 Results of Chi-squared tests to examine the differences between small and medium software enterprises trends for adopting CMMI level 2 process areas-specific practices S. No. Process areas No. of small companies No. of medium companies df = 2; a =.05 Following more than 75 % of specific practices in process areas Following % of specific practices in process areas Following \50 % of specific practices in process area Following more than 75 % of specific practices in process areas Following % of specific practices in process areas Following \50 % of specific practices in process areas v 2 p 1 Configuration management 2 Measurement and analysis 3 Project monitoring and control 4 Project planning 5 Process and product quality assurance 6 Requirements management 7 Supplier agreement management * ** * * * p \.01; ** p \.05

17 Table 5 Details of SMEs and unofficial adoption trends ID CT Configuration management (%) Measurement and analysis (%) Project monitoring and control (%) Project planning (%) Process and product quality assurance (%) Requirements management (%) Supplier agreement management (%) S 1 Small [ [75 \50 S2 Small \ [75 \50 S3 Small [ \ \50 S4 Small \50 \ [ S 5 Small \50 \ [ S6 Small \50 \ [ S7 Small \50 \ \50 \50 [ S8 Small \ \50 \50 [75 \50 S 9 Small \50 [75 [75 \50 [75 \50 S 10 Small \50 [75 [75 \50 [75 \50 S11 Small \50 [75 [75 \ \50 S12 Small \50 [75 [75 \ \50 S 13 Small \50 [75 [75 \ \50 S 14 Small \50 [75 [75 \ \50 S15 Small \50 [75 \50 \ \50 S16 Small \50 \50 [ \ S 17 Small \50 \50 [75 [75 \50 \50 \50 S 18 Small \50 \50 [75 [75 \50 \50 \50 S19 Small \50 \ [75 \50 [ S20 Small \50 \50 \50 [75 \50 [ S 21 Small \50 \50 [75 \50 [ M 1 Medium [75 [75 [75 [ M2 Medium [75 [75 [75 [ M3 Medium [75 [75 [75 [

18 Table 5 continued ID CT Configuration management (%) Measurement and analysis (%) Project monitoring and control (%) Project planning (%) Process and product quality assurance (%) Requirements management (%) Supplier agreement management (%) M4 Medium \50 [75 [ [ M5 Medium \50 \50 [75 [75 [75 [ M 6 Medium \50 \50 [75 [75 [75 [ M7 Medium \50 \50 [75 [75 [75 [ M8 Medium [75 [ [75 [75 M9 Medium [75 [ [ [ M 10 Medium [75 [ [75 [75 M 11 Medium [ [75 [ [75 [75 M12 Medium [75 [ [ M13 Medium [75 [ [ M 14 Medium [75 [ [ M 15 Medium [75 [ [ M16 Medium [ [ M17 Medium [75 \ M 18 Medium [75 \ M 19 Medium [ [75 [75 \ M20 Medium [75 \ M21 Medium \ ID, Identification No.; CT, Company Type; \50 %, less than 50 %; [75 %, more than 75 %

19 of plans that are required for defining project activities. Project planning includes project plan development, interaction with the stakeholders and attaining commitments to the project plan (Team CP 2010). Deficiencies in project planning are considered hazardous to the success of projects (Wallace et al. 2004; Huang and Han 2008). On the other hand, improved project planning is perceived as an efficacious tool for managing high-risk projects (Reel 1999). The results of our study show that in the case of the project planning process area, small software companies follow more than 75 % of specific practices about half the time and a further % of specific practices only occasionally. Medium software companies very frequently follow more than 75 % of the specific practices of the project planning process area. Therefore, as compared to small companies, such medium software companies are more capable of dealing with the project planning risks that can lead to project failure Process and product quality assurance Statistically significant differences also exist with respect to following the specific practices of the process area Process and Product Quality Assurance v 2 (2, N = 42) = , p \.01. In this case, medium enterprises are more likely to follow % of the specific practices, but small enterprises are more likely to follow \50 % of the specific practices. Process and product quality assurance activities are composed of the evaluation of processes and work products, the identification and documentation of noncompliance issues, the provision of feedback, and offering assurances that noncompliance matters are resolved (Team CP 2010). Process and product quality assurance is of the utmost importance for companies involved in software development or in purchase of the software (Niazi and Babar 2009). The quality assurance issue must be solved as soon as possible in the timeline of any project because in most cases, by the time a quality problem is recognized, it is too costly to resolve (Vivatanavorasin et al. 2006). Medium software companies follow % of the specific practices of the process and product quality assurance process area roughly half the time. As a result, such medium companies have intermediate capability of detecting quality problems as early as possible. Small software companies follow \50 % of the specific practices of process and product quality assurance process area very frequently. They are, therefore, more prone to quality issues Supplier agreement management Similarly, there are statistically significant differences with respect to adopting the specific practices of the process area Supplier Agreement Management v 2 (2, N = 42) = , p \.01. Here, medium companies are also more likely to follow % of the specific practices, whereas small companies are more likely to follow \50 % of the specific practices. The Supplier Agreement Management process area handles the procurement of products and related services from selected suppliers, via agreements with the suppliers (Team CP 2010). Failure to manage supplier agreement results in delayed product acquisition from suppliers and consequently contributes to rising unforeseen costs (Vivatanavorasin et al. 2006). According to our study, medium software companies follow % of the specific practices of the Supplier Agreement Management process area very frequently and therefore have intermediate capability of dealing with supplier issues. By contrast, small software companies have low competency in

20 handling supplier issues, as they frequently follow \50 % of the specific practices of the Supplier Agreement Management process area. For the remaining three process areas, small and medium software enterprises do not demonstrate statistically significant differences with respect to following the specific practices of these process areas. Small and medium software companies are more likely to follow % specific practices of the Configuration Management process area. For the Project Monitoring and Control process area, both small and medium software companies are more likely to follow more than 75 % specific practices and a further % specific practices of the process area. However, in the case of the Requirements Management process area, small and medium software companies are more likely to follow only more than 75 % specific practices. Thus, in the case of four of the seven process areas of CMMI level 2 (namely Measurement and Analysis, Project Planning, Process and Product Quality Assurance and Supplier Agreement Management ), there are statistically significant differences between small and medium software enterprises trends to unofficially adopt the specific practices related to these four process areas. This provides the answer to RQ5. 5 Threats to validity The threats to the validity of the results in this study are discussed below. 5.1 Construct validity CMMI is a well-established standard for SPI. This study is based on the unofficial adoption of specific practices related to CMMI level-2 process areas. It was anticipated that participants might have problems with precisely understanding the specific practices. To develop an effective survey, two rounds of pilot study were conducted. The recommendations given during the first round were accommodated in the second round. For maximum clarity, the details of subpractices related to specific practices were also provided. Therefore, we are confident that the specific practices under study were fairly known to our respondents. 5.2 Internal validity The respondents had a relevant working experience of 5 15 years and were nominated as representatives of the participating SMEs, where specific practices are being adopted informally. Therefore, we can say that these specific practices are related to the workspace of the respondents. To investigate the adoption trends, three percentage categories were framed: \50 %, %, and above 75 %, A potential SME must follow at least 50 % of the specific practices in the case of at least 50 % of the process areas of CMMI level 2. The percentages analogous to three categories and the 50 % criterion have already been applied in previous studies. Furthermore, our sample contains an equal number of small and medium software enterprises. The capabilities of the method employed are limited, since this method is based on only one form of objective evidence, which is gathered through questionnaires. Additionally, the method relies on the opinions, experiences, and observations of only one staff member per company. We attempted to handle this limitation by consulting with management to

21 select the most relevant available professionals in the companies. To ensure the reliability and accuracy of the results, the facts provided in the questionnaires were cross-checked. 5.3 External validity External validity was addressed by selecting a sample of 42 practitioners from 42 software development SMEs originating in two different countries, Malaysia and Pakistan. All the relevant professionals from the two countries may not completely agree with the results, but we believe that this sample is a true representative of the population. We believe this because, after the initial correspondence, 110 SMEs were approached, with 52 being selected for a second questionnaire survey, keeping in view the parameters of size, CMMI non-certification, and the selection of an equal number of small and medium software companies. Furthermore, a sample of 42 questionnaires was chosen for data analysis that considered the relevant experience of the respondents and the parameter of using an equal number of small and medium software companies. To adjust for the various SMEs standards (Australian, European, Malaysian and Pakistani), the concept of common range standard has been introduced. By following this new standard, a small or medium company according to any one standard (out of four standards under study) is also considered to be a small or medium company according to the three remaining standards. We surveyed the informal adoption trends only of the specific practices of CMMI level 2 process areas. Ideally, the specific practices related to the process areas of all CMMI levels would have been studied, but unfortunately this was not possible due to the time constraints on the respondents (CMMI-DEV, V1.3 contains 22 process areas and 167 specific practices). CMMI level 2 was selected for this study because most of the CMMI level 2 process areas are fundamental disciplines for any established software development organization. Another reason for concentrating on CMMI level 2 is that this level may be considered a starting point for CMMI-based SPI. If a SME follows the specific practices of CMMI level 2 process areas, then similar trends can be anticipated for other levels. There would therefore be a high likelihood of quick and effective CMMI-based SPI for that SME. In view of the purpose of this study, it offers no discussion as to the reasons why SMEs follow specific practices of CMMI level 2 process areas. The reasons were not explored in the survey. But a rationale for their tendency to follow CMMI practices is that the SMEs are compelled to do so, albeit unofficially, in order to meet the ever-increasing demands of this IT-based world. 6 Conclusion and future directions SPI is an obligatory phenomenon for overcoming software development challenges. This research has been conducted with the aim of assisting the successful implementation of sponsored CMMI-based SPI initiatives in SMEs. To apply a uniform SMEs standard and to allow for results that can be widely applied, this study presents the idea of a common range standard, which harmonizes Australian, European, Malaysian, and Pakistani standards on SMEs. By adhering to this standard, the study is able to include small software companies with employees and medium software companies with employees.