A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies

Transcription

1 A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies Muhammad Sulayman PhD Student Department of Computer Science The University of Auckland Supervised by: Assoc. Prof. Emilia Mendes

2 Abstract It is observed that in recent years small and medium Web companies have emerged very rapidly and thousands of such companies are in existence all over the globe. To cater the needs of such companies, a new field of research was created Web Engineering, given than Web engineering differs from traditional software engineering in numerous ways, which include the need of agile process models, extended modelling techniques (WebML), Navigational development techniques, different architectures and rapid application process along with different testing techniques. [12] [13] [15] [16 [25] [27]]. It has been observed that Software process improvement emerges as one of the biggest challenges for such companies [12]. A systematic literature review (SLR) has been conducted to identify and discuss the existing models and techniques used by small and medium Web companies. Important phases of our SLR included identification of the research questions to be investigated; primary and secondary database searches to identify relevant literature; data extraction from selected studies; data synthesis to formulate answers; and formal discussion to identify trends and research gaps. A total number of 88 studies were selected, after being filtered using an initial inclusion and exclusion criteria. Surprisingly, further inspection revealed only 4 relevant studies on the topic. A careful evaluation of studies was performed using qualitative as well as quantitative checklists; extracted data were further synthesized to answer the probed research questions. The identification of research gaps and possibilities of further research were explored.

3 1 Table of Contents Table of Contents Table of Contents Rationale of the Research Systematic Literature Review Process Formal Definition Motivation & Benefits The Process Formulation of Research Questions Identification of Relevant Literature Searching Procedure Searching Stages Primary Search Phase Primary Search Execution Inclusion & Exclusion Criteria Selected Studies Study Quality Assessment Checklist for Qualitative studies Checklist for Quantitative studies Studies Quality Assessment Results Data Extraction Data Synthesis & Results Research Question Research Question Research Question Research Question Research Question Drawbacks Found among the Existing Studies Research Gaps Conclusion References Appendix... 32

4 2 Rationale of the Research 1. Rationale of the Research Software processes play an important role in helping project teams in software development organizations and they use similar and sound practices [1]. Ideally, these processes should combine the need for rigor and discipline with the need for flexibility and creativity, but that balance is hard to achieve [1]. Formal processes emphasize the explicit command-and-control side of the organization due to their concrete nature, while informal team practices emphasize the mutual adjustment and explorations needed to accomplish tasks successfully [2] [3]. Almost all modern software organizations operate in a competitive market, under tight time and cost constraints [4] [58]. As an answer to their needs, organizations have started to undertake software process improvement (SPI) initiatives (see [5] for an overview of different approaches) aimed at increasing the maturity and quality of their software processes [6]. Investment in process improvement has had various business benefits i.e. improved product quality, reduced time to market, better productivity [6], increased organizational flexibility and customer satisfaction [7] [8][59]. Many researchers are focusing their attention to define the process and its relation to the quality of the products [9] [10]. While this remains important, many researchers are exploring the success factors and people issues that inherently play major roles in the adoption of new processes by software organizations [8]. In the current era, small and medium software development companies form a large population out of the total number of software companies. According to a recent survey 99.2% of the world s software companies are small and medium in context [11]. It is also observed that in recent years small and medium sized software development companies have emerged very swiftly and many of them are working in the domain of Web development [12]. However, there are differences between traditional software development and Web development [53][54], and even a new research field, Web Engineering was created to deal with the specific needs of Web companies and software companies that develop Web applications [22]. Web engineering is different from traditional software [17]. It requires Web agile process models e.g. RAD, SCRUM [13] [14] etc. so the development methodologies are very different [15] [16] [60] [61]. There is also a very strong focus on rapid application development and agility concepts in web software process [17] [18] [19]. Like traditional software the engineering of applications for the Web are supposed to adapt to the Model Driven Approach Agile Web Development approach [20] [60] [61]. The methodology for development is tried to be user centred due to rapid change of content and flexible nature [21]. Web application developers are different in attitude and approach from the traditional systems developers and have a strong focus on hypermedia context and continuously evolutionary approach [22]. Some examples of the Web development methods are OOHDM, SWM, OOWS and UWE & WebML as an extension of UML [23] [24] [50] [51 [25] [26] [27] [28]. Web requirements Engineering is moving from task orientation towards goal orientation [29] based on NDT (Navigational Development Techniques) [30]. Researchers are also focusing on devising special project management initiatives like WIPSE (Web Integrated Project Support Environment), Action minutes and PAWS (Project administration Web Site) for Web development companies [31] [32] [33]. Testing of Web applications is also different from

5 3 Systematic Literature Review Process the traditional systems and revolves around different quality dimensions [34]. RIA (Rich Internet Applications) is using the above specified Web Engineering practices [35]. The fact that the engineering of Web applications differs from the engineering of software applications motivated this work. As previously illustrated, many development methodologies and techniques were proposed specifically to tackle issues associated with Web applications development and project management. Therefore SPI for small and medium Web enterprises also seemed a relevant research topic to be investigated, which is the objective of this systematic literature review and automatically also the objective of this research. We focus explicitly on Web companies, which are characterized by companies that only provide Webrelated services such as Web application development, Web hosting and Web data management etc. The above mentioned studies and facts laid the foundation of our investigation. We observed different approaches for the various artefacts of engineering Web. In addition, a large number of Web applications are developed in small and medium-sized development companies. Therefore, the purpose of this systematic review (SR) is to gather evidence about process improvement initiatives observed for small or medium sized web companies. 2. Systematic Literature Review Process 2.1 Formal Definition A Systematic Literature Review (SR) is defined by Kitchenham [36] as, A systematic literature review (often referred to as a systematic review) is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest. 2.2 Motivation & Benefits Systematic reviews are used to gain effective insight into a problem and understand existing approaches. The main benefits that can be obtained by performing a SR are as follows [38] [37] : Identification of the particular research questions to be investigated. Identification of the desired population, intervention, context and outcomes. Helps in summarizing the existing research evidence. Lays a foundation for a disciplined search mechanism. Provides a case to assess the quality of studies. Helps in producing unbiased empirically validated results. Provides a mechanism to synthesize the research evidence.

6 4 Systematic Literature Review Process Largely benefits researchers, PhD students and industry practitioners by identifying the state of practice/research and gaps in the existing research used to plot new research efforts. 2.3 The Process SR is a detailed process divided into different tasks and activities that are listed as follows: Systematic Literature Review Study and Understanding: This Phase helps in developing and understanding of review concepts and to develop an understanding of the overall methodology. Formulation of Research Questions: This is an iterative phase where the important research questions to be investigated during the SR are identified. Development of a Study Protocol: This phase is very rigorous and also iterative. It covers the overall plan for the systematic literature review Identification of Relevant Literature: This phase encompasses the identification of primary and secondary studies and is a search phase. Determining Inclusion & Exclusion Criteria: During this phase a criteria is applied to select the studies for to be part of the SR. If a study fulfils the inclusion criteria it is selected otherwise it is discarded. Selection of Studies: This phase includes both primary and secondary studies. The studies are selected after the application of the inclusion criteria and are further filtered. Study Quality Assessment: Both qualitative and quantitative studies are assessed for quality in this phase based on the developed checklists and appropriate scores are assigned to each study. Data Extraction: Data are extracted from each study and based on the research questions. Data Synthesis: After extraction the data is aggregated, integrated and summarized for the further clarity and to answer the research questions. Report Write Up: A very important concluding phase that details and summarizes the results and findings of the overall systematic literature review process comes at last. All previous phases contributed to it. The following sections of this document will detail each phase of this SR in depth. The flowchart in Fig. 1 elaborates the complete process to further detail.

7 5 Formulation of Research Questions Fig.1 Systematic Literature Review Process Flow Chart 3. Formulation of Research Questions Identifying the valid research questions is an important component of any systematic literature review [36]. For the formulation of the research questions in this SR we have used the PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria defined by Petticrew and Roberts [38] as described in Table 1. Our research focus is not comparison. Therefore, it is not mentioned in the subsequent tables and sections.

8 6 Identification of Relevant Literature Table 1: Summary of PICOC Population Intervention Outcomes Context Small & Medium Web Development Organizations Using Software Process Improvement Various Practices of Software Process Improvement, Corporate Software Process Improvement Models i.e. CMMI, ISO, SPICE etc., Quality Assurance Activities Used by Small and Medium Web Organizations Measure of success showing how successful SPI approaches have been to small and medium Web organizations Industry, practitioners and consultants from Small & Medium Web Development Organizations, small & large-scale tasks The question identification process was iterative. Below are the research questions that we finalized and investigated after various revisions. Research Question 1 Which software process improvement models/techniques are followed by small and medium Web development organizations? Research Question 2 Which software process improvement models/techniques were successful to small and medium Web development organizations and how success is being measured? Research Question 3 Are there any software process improvements models that have been specifically made to measure for small and medium Web companies? Research Question 4 What are the important characteristics of small and medium Web organizations that follow software process improvement activities and practices? Research Question 5 What constitutes a small or medium Web organization for the studies investigated? 4. Identification of Relevant Literature Identification of relevant literature required exhaustive, rigorous and thorough searching of the relevant material. With its help gaps in the existing literature are found and it provides a context for the placement of new research [39]. The data sources to be used by this study included online databases, research journals, conferences and grey literature.

9 7 Identification of Relevant Literature 4.1 Searching Procedure Constructing the relevant search terms is also a very important task, part of the SR and the adopted approach is summarized below: a. Major search terms were derived from the research questions i.e. PICOC b. Keywords from the found articles also became relevant terms c. Synonyms of key terms were found by using thesaurus. d. Boolean OR was used with relevant terms e. Boolean AND was used to restrict the search. Tables 2 to 5 below elaborate the terms found from each approach. Table 2: Terms Derived from PICOC Population Intervention Outcomes Context Small & Medium Web Development Organizations, Software Process Improvement Software Process Improvement, Software Quality Assurance, Small and Medium Web Development Organizations/Companies, Software Process improvement Models, Software Process Improvement Techniques SPI Success, Measures of SPI success Industry, practitioners and consultants, small & large-scale tasks Table 3: Terms Derived from Synonyms Basic Term Software Process Improvement Quality Organization Assessment Measurement CMM / SW-CMM Web Small & Medium Web Development Organizations Alternate Term Software Process Enhancement, software Process Enrichment, Software Maturity Attitude Excellence Setting / Business / Organisation/ Enterprise/Company Evaluation, Appraisal, Review Estimation, Capacity, Capability CMMI, SPICE, ISO-15504, PRINCE II Internet / WWW / World Wide Web Small & Medium Web Companies/ Internet Companies Table 4: Concatenation of Similar Terms with Boolean OR (Software Process Improvement OR Software Process Enhancement OR Software Process Enrichment OR Software Maturity Attitude ) (Software Process Assessment OR Software Process Evaluation OR Software Process Appraisal OR Software Process Review) (Software Process Estimation OR Software Process Measurement OR Software Process Capacity

10 8 Identification of Relevant Literature OR Software Process Capability) (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets OR Software Process Improvement Purpose) (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets OR Software Process Improvement Purpose ) (CMMI OR SW-CMM SPICE OR ISO OR PRINCE II) (Web OR Internet OR WWW OR World Wide Web)) (Small and Medium Web Development Organizations OR Small & Medium Web Companies OR Internet Companies) Table 5: Concatenation of Similar Terms with Boolean AND (Software Process Improvement OR Software Process Enhancement OR Software Process Enrichment OR Software Maturity Attitude) AND (Software Process Assessment OR Software Process Evaluation OR Software Process Appraisal OR Software Process Review) AND (Software Process Estimation OR Software Process Measurement OR Software Process Capacity OR Software Process Capability) AND (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets ) AND (Measurement OR Estimation OR Assessment) (CMMI OR SW-CMM) AND (Web OR Internet OR WWW OR World Wide Web)) AND (Small & Medium Web Development Organizations OR Small & Medium Web Companies OR Small & Medium Web Development Organisations) 4.2 Searching Stages In order to retrieve the relevant literature there is a proper mechanism required [36]. The overall search phase consists of two components, also detailed in Fig. 2. Primary Search Phase Secondary Search Phase 4.3 Primary Search Phase The primary search phase comprises the search of the relevant literature in Online Databases, Search Engines, Individual Journals, Conferences, PhD/MS Dissertations and Theses. Details of the sources used in the primary search phase carried out as a part of this SR are listed below Online Databases ACM Digital library IEEE Xplore ISI Web of Science INSPEC Science Direct

11 9 Identification of Relevant Literature Springer Link SCOPUS The IEEE Computer Society Digital Library Online Search Engines Google scholar CiteSeer Agile alliance Individual journals Journal of Systems and Software (JSS) Empirical Software Engineering (EMSE) IEEE Transactions on Software Engineering (TSE) Journal of Software Process Improvement and Practice Information and Software Technology Journal of IEEE Software Software Quality Journal Journal of Software Maintenance and Evolution (SME) ACM Transactions on Software Engineering and Methodology (TOSEM) Conferences: International Conference on Software Engineering (ICSE) IEEE International Symposium on Empirical Software Engineering WOSQ (International Workshops on Software Quality) EuroSPI (European Conferences on Software Process Improvement) International SPICE Conferences on Process Assessments and Improvement Australian Software Engineering Conferences PhD Dissertation and Theses ProQuest Dissertations & Theses Auckland University Library The primary search phase used the above mentioned sources as they were accessible through the University resources and a previously conducted SR on the status of Web engineering research has used most of those sources [39].

12 10 Identification of Relevant Literature Fig. 2 Process of Identifying Relevant Literature from the Primary Search Phase 4.4 Primary Search Execution As mentioned above, different sources were considered during our primary search phase. Initially, during that phase, the comprehensive search string did not yield any results, therefore the Computer Science subject librarian at the University of Auckland, Ms. Liz Hardely, was consulted. She suggested the use of a simpler search string, which is also mentioned by Kitchenham [40]. The search string that we used in our primary search phase was therefore ("(software process improvement)" AND ("(small)" OR "(medium)")). The reason for excluding the term Web Company from the search string was because no results were returned when using it. Then, according to the recommendations of the subject librarian, we looked for the term Web companies inside the found studies. Initially the abstracts were collected and those studies

13 11 Identification of Relevant Literature which abstracts suggested the presence of Web companies were considered. The PhD student Mr. Muhammad Sulayman and his research supervisor Dr. Emilia Mendes both read the titles and abstracts of the found studies and agreement was reached on the inclusion or exclusion of studies. For the remaining studies a detailed inspection of their full text was conducted. The aim is to discard bias and improve internal as well as external validity [41]. In relation to the studies whenever we were unsure they were applied to the Web companies or not we contacted their authors for clarification. Authors of 61 studies were contacted, out of which 39 replied and 3 studies were included in the SR based on the positive replies of the authors. Very surprisingly, of the 88 studies initially shortlisted and finally only 4 studies met the inclusion criteria. Table 6 summarizes the complete search process. ACM and Springer Databases have demonstrated very interesting behaviour. ACM initially retrieved 255 studies as a result of our search string and similarly Springer retrieved 187 relevant studies. On detailed investigation, 238 studies from ACM and 182 studies from Springer did not satisfy our inclusion criteria and therefore were not considered. The authors of the remaining studies i.e. 17 from ACM and 5 from Springer were contacted by for clarification on whether their studies were applied to Web companies. 39 contacted authors that form 63.93% of the contacted authors responded to our queries. The studies for which we did not receive any clarifications were unchecked on our SR. None of the studies retrieved using Springer s database were included. None of the studies retrieved using the ACM database were included in our SR. Author of only one study from ACM confirmed that his study was applied on Web companies along with software companies but he was unable to provide any separate data for Web companies which made us not to select his study. Table 6: Summary of Primary Search Results Serial No. Database Name Number of Publications Found Relevant Irrelevant 1 IEEE Explore INSPEC Scopus ISI Web of Science Pro Quest Computer Database ACM 17/255 (Initially 255 found) Springer 5/187 (Initially 187 found) Google Scholar No new results Grey literature Inclusion & Exclusion Criteria The Inclusion criteria comprised the studies that focused on software process improvement activities in small and medium Web companies. The studies that did not explicitly focus on small

14 12 Selected Studies or medium Web companies, or did not investigate the use of SPI techniques and frameworks were excluded. 5. Selected Studies After careful investigation, the study selection process short listed the following 4 studies that satisfied the inclusion criteria. S1 L. Scott, R. Jeffery, L. Carvalho, J. D'Ambra, P. Rutherford, Practical software process improvement - the IMPACT project, Proceedings of Australian Software Engineering Conference, S2 A. El Sheikh, H. Tarawneh, A survey of web engineering practice in small Jordanian web development firms, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, 2007 S3 P. Allen, M. Ramachandran, H. Abushama, PRISMS: an approach to software process improvement for small to medium enterprises, Proceedings of Third International Conference on Quality Software, S4 R. Naidu, Software Process Improvement of Small & Medium Organizations, MSc thesis, Department of Computer Science, University of Auckland, Study Quality Assessment Two quality assessment checklists were developed to assess the quality of the selected literature and also of applicable to refine the study selection as well as to. Purpose of establishing two checklists was to assess the quality of quantitative and qualitative studies separately based on their criteria. [36] [38] [42] [43] The checklists were prepared using questions from Petticrew [38], Crombie [42] and Fank [43]. In addition some of these questions are adjusted to reflect our SR s context. The checklists are presented in the following sub sections. 6.1 Checklist for Qualitative studies Checklist in Table 7 is to review the qualitative studies. Table 7: Quality Assessment Checklist for Qualitative Studies No. Question Answer 1 Is the methodology used suitable to address the stated Yes/No/Partially research questions? 2 Does the article target the ideal population? Yes/No/Partially 3 Does the article use the research methodology adequately? Yes/No/Partially 4 Does the article discuss any of the previous work/literature? Yes/No/Partially 5 Is the study process specified in the article repeatable? Yes/No/Partially 6 Is the article biased towards one SPI framework model or Yes/No/Partially

15 13 Study Quality Assessment technique? 7 Do the findings address the original research questions? Yes/No/Partially 8 Does the article document any assumptions taken? Yes/No/Partially 9 Does the article document the procedure used to validate its Yes/No/Partially findings? 6.2 Checklist for Quantitative studies Checklist in Table 8 is to review the quantitative studies. Table 8: Quality Assessment Checklist for Quantitative Studies No. Question Answer 1 Are the aims of the research clearly stated? Yes/No/Partially 2 Is the research methodology used suitable to address the Yes/No/Partially research questions? 3 Does the article target the ideal population? Yes/No/Partially 4 Was the sample used random? Yes/No/Partially 5 Was the SPI technology/framework used clearly defined? Yes/No/Partially 6 Did the study account for confounding factors? Yes/No/Partially 7 Are the measures used in study fully defined? Yes/No/Partially 8 Are the measures used in the study relevant to answer the Yes/No/Partially research questions? 9 Are the data collection methods adequately defined? Yes/No/Partially 10 If different groups are treated, are they treated equally in the Yes/No/Partially study? 11 Was only relevant data used in the study? Yes/No/Partially 12 Are any of the statistical methods used for analysis of data Yes/No/Partially described? 13 Has the use of statistical methods been motivated? Yes/No/Partially 14 Are all main findings relevant to answer the research Yes/No/Partially questions? 15 Are the negative findings presented? Yes/No/Partially 16 Has the research ignored any significant factors, either Yes/No/Partially methodology or measures? 17 Are the results compared with previous results or is it clear Yes/No/Partially that there were no previous results? 18 Does the result adequately answer the research questions? Yes/No/Partially 6.3 Studies Quality Assessment Results A Score of 1 has been assigned for a yes, 0.5 for partially and 0 for a no for both qualitative and quantitative checklists. Based on the quality assessment checklists the maximum score a quantitative and qualitative study could obtain were 18 and 9 respectively. Higher achieved scores by a study decide its stronger evidence to the relevant context.

16 14 Data Extraction From the four studies selected, three were quantitative (S2 to S4) and one was qualitative (S1). Table 9 shows the summarized scores of each study, measured based on the quality assessment checklists. Table 8: Assessment Scores of Selected Studies Sr. No. Quality Assessment Score Study Type S1 6.5/9 (Qualitative) S2 10/18 (Quantitative) S3 8.5/18 (Quantitative) S4 13/18 (Quantitative) Studies S1 and S4 presented the highest quality based on our quality assessment. Average score found for study S2, however study S3 did not score very well. These scores represent our assessment regarding the relevance of each study to support evidence to be used to answer our research questions. 7. Data Extraction The purpose of the data extraction phase is to extract the relevant data, later to be used to prepare summary tables and quality scores, later to be used to answer SR s research questions. Data extraction was performed using two extraction forms created to extract the data needed to answer the SR s research questions and assesses the quality of each study. One of the forms stored the data extracted from for the qualitative study and the other stored data extracted from quantitative studies. Both data extraction forms are available in the appendix. The PhD student read all the related studies and filled out all the data extraction forms. The quality of data extraction was validated by the student s main supervisor, who also read a portion of the selected studies. Results were compared and points of conflict were also discussed. The purpose of this activity was to remove the bias related to the understanding of the studies. 8. Data Synthesis & Results In the data synthesis phase the results from all the findings were tabulated and summarized and each question was assessed individually against the findings. Tabulated results are also useful to identify current research gaps. The sections below elaborate on the synthesis process for each research question.

17 15 Data Synthesis & Results 8.1 Research Question 1 It states that Which software process improvement models/techniques are followed by small and medium Web development organizations? In relevance to this question some studies are proposing an applicable model for software process improvement of small and medium Web companies while others just rely on a set of techniques/practices believed to be useful for the cause. Models are established paradigms to perform certain tasks with an implication regarding the order of execution [47]. Techniques exist in isolation to perform a certain activity and can be implemented inside a model [48]. Models are more complete than techniques and also reside on a different level of complexity as they act as a framework or sometimes as a pattern based on experience [47]. Table 9 shows the data synthesized from each study regarding software process improvement models and techniques that are specific to small and medium Web companies. Table 9: Models and Techniques used by Small and Medium Web Development Organizations Sr. No. SPI Model SPI Techniques No. of Companies Salient Features S1 IMPACT Project A Generic (Plan-Do- Act-Check) Model 1 Iterative Approach, Model has two stages: project level & process level. Findings from the project level are applied at the process level for improvements. S2 Best practices are considered as techniques such as : Web metrics and automated tools for post project analysis, Control functions for the development process models, Project Management (PM) best practices, SPI & quality management initiatives, Change management & Project Tracking. 18 Provides some key practices and measures companies against them. S3 PRISMS Model based on Dynamic CMM 4 Iterative Approach, Definition of business goals, Key process areas (KPA) for improvement, Plan improvement, Post project Evaluation, Continuous Project

18 16 Data Synthesis & Results S4 Project Postmortem System Feedback, Revision of Process model, GQM--based measurements. 8 Iterative Approach, Context dependence approach, Assessment of state of Practice. Concerns more about Process Management, Requirements Management, Requirements Development, Verification & Validation, Configuration Management, Relationship Management, Product Innovation. The three different SPI models and techniques used by the studies in our SR will each be briefly described below. The IMPACT Model The IMPACT model was based on the methodology of continuous learning and improvement and uses the PLAN-DO-ACT-Check principle of project management. The model is based on the IDEAL model of software process improvement [44]. Fig. 3 provides a detailed illustration of the model. IMPACT is a two staged model that comprises two cycles, are process and project cycles. These cycles drive each other and observations from one cycle are used to continuously improve the other. The Project cycle is advocated by QIP (quality improvement paradigm) which has six stages. Since the model is iterative, so there is a strong focus on understanding the process and applying improvements. The processes and improvements are measured and results are stored in process guides that are a motivation for the next cycle or iteration. Fig. 3 The IMPACT Project Model [55]

19 17 Data Synthesis & Results The PRISMS Model The PRISMS model is illustrated in Fig. 4.The model focuses on the relationship between requirements goals and business goals. This model is also iterative and focuses on prioritization of key process areas with the help of metrics. It also encourages the involvement of stakeholders. This model is flexible and can be tailored to any model of software process improvement. The model demonstrates the development of key process areas (KPA) and its improvement plan is based on them. Key process areas for improvement are devised after consultation with the developers, process improvement initiatives and business goals. Process improvement plans are always evaluated and are revised on feedback. Plans are always influenced by the metrics. Fig. 4 The PRISMS Model [12] The Project Post-Mortem System Project post-mortem system of software process improvement for small and medium software organizations is generic to small and medium software companies but can also be adapted to Web companies. This model is motivated by the IDEAL model [44]. This approach is also iterative in nature and it uses a five steps. According to Naidu software organizations are classified into different categories according to their nature and in the first step the organization is to be identified according to a category e.g. software, Web, Embedded Systems etc. [56]. In the second step the focus is on the current state of practice and the analysis of the current software process improvement status of the organization. Based on the state of practice report this model identifies the improvement opportunity and development of the focus area that needs immediate attention. Focus areas can be prioritized according to the requirement of the organization. Process innovation is acquired in the next step of the model and it is assessed in the final step of the model. Then the post-mortem meeting is conducted and the next iteration is

20 18 Data Synthesis & Results planned based on the assessment of the data gathered from several questionarries. The model is very flexible in nature and more practical due to the support of generalizations, as illustrated in fig. 5. Fig. 5: The Project Post Mortem Model [56] SPI Techniques Mentioned in Survey of Jordanian Web Companies The survey of Web Engineering practices in small Jordanian Web firms is the study that focuses on some known general techniques of software process improvement and surveys them over a number of companies. These techniques include: Change Control Management Development of Standards and Procedures Application of Web metrics Development Process Control Proper Tools and Techniques for Configuration, Process and Change Management The survey also demonstrates that the use of Web metrics is the neglected area in most of the surveyed Web companies whereas organizations are focusing on the use of automated tools and techniques for improvement.

21 19 Data Synthesis & Results Discussion on Research Question 1 All the four considered studies propose a certain Model or Techniques for SPI of small and medium Web development companies and there are various commonalities found among them which are detailed in Table 10. Post project analysis and process management and measurement are the techniques that have been emphasized in all four studies; followed by project tracking, feedback analysis, change management, configuration management as well as mapping of business goals with SPI. All the suggested SPI models are using an iterative approach and influenced by CMM/CMMI and there is strong tendency for use of the IDEAL model. Similarly, all the studies are suggesting a measurement and management program for processes that govern the use of Web metrics. Table 10: Common Techniques/Features used by Small and Medium Web Development Organizations Sr. No SPI Techniques/Model Features Studies S1 S2 S3 S4 1 Iterative Approach 2 Automated tools for SPI 3 Post project analysis 4 Best Project Management Practices 5 Mapping of Business Goals or GQM with SPI 6 Motivation for the use of CMM/CMMI 7 Configuration/Change Management 8 Project Tracking and Feedback 9 Requirements Management 10 Process Management and Measurement 11 Use of IDEAL Model 8.2 Research Question 2 It states that Which software process improvement models/techniques were successful to small and medium Web development organizations and how success is being measured? This question comprises two parts. The first relates to the models and techniques that were considered successful for small and medium Web companies and the second investigates the success measures used in the studies. Table 11 summarizes the results extracted from the five studies.

22 20 Data Synthesis & Results Table 11: Success & Measures of Success of Different Models and Techniques found useful for Small and Medium Web Development Organizations Sr. No. Successful SPI Model Successful SPI Techniques Success & Measures of Success S1 IMPACT Project A Generic (Plan-Do- Act-Check) Model Comparison of current practices with standards, PLAN-DO-ACT- CHECK approach, process guides or accepted practices like CMM, RUP or ISO, Use of GQM. Success is measured by Increase in productivity, decrease in cycle time, feedback and experiences from discussion forums. S2 Best practices are considered as techniques such as: commitment by the organization, Web metrics and automated tools for post project analysis, Control functions for development process models, Project Management (PM) best practices, SPI & quality management initiatives, Change management & Project Tracking Does not state any measure of success. S3 PRISMS Model based on Dynamic CMM Assessment interviews from the customer and developers at the end of a business case workshop, Customized versions of CMM, GQM for assessment are measures of success. Project monitoring, Feedback from customers and developers and managers. The use of metrics for SPI coordination with business goals also contribute as success measures.

23 21 Data Synthesis & Results S4 Project Post-mortem System Questionnaire based for each AOI (Area of Interest) derived from customized CMM to scale the known outputs. Measures of success are Feedback from developers, customers, managers, Strategic value proposition, customer satisfaction, operational excellence, Reviewbased project tracking and monitoring. All the three SPI models mentioned in studies S1, S4 & S5 were considered successful, despite the use of different measures of success. Study S1 of IMPACT model measures success as: Use of Plan-Do-Act-Check Iterative Approach for Software Development Comparison of Current Practices with Known Standards, Process Guides or Accepted Practices like CMM, RUP or ISO Effective Linkage of GQM with Collected Measures Feedback and Experience of Process Participants Decrease in Cycle Time Increase in Productivity Study S4 of PRISMS model measures success as: Proper Definition of Business Goals Motivated by GQM Definition of Key Process Areas for Improvement Preparation of SPI Plan Metrics for SPI Mapping with Business Goals and their Evaluation on Feedback Project Tracking Activities Creation of Measurement Baselines Study S5 of Project Post-Mortem model measures success as: Creation of Focus Points based on Organizational Goals and Strategy Focus Pints Determine Areas of Interest (AOI). They include Process Management, Requirements Management, Requirements Development, Verification& Validation, Configuration Management, Relationship Management, and Product Innovation. Questionnaire Based Measurement Feedback from Developers, Customers, Managers Operational Excellence Review based Project Tracking and Monitoring

24 22 Data Synthesis & Results Studies S2 considered various techniques to be successful that include: Commitment by the Organization Towards the SPI Initiatives Definition of Set Standards and Procedures and their Application Use of Web Metrics Change Control and Configuration Management Use of Proper Automated Tools & Techniques However, Study S2 does not indicate any specific measures of success with respect to SPI. The four studies demonstrate certain measures of success and also exhibit certain common factors that are present in different studies. Table 12 exhibits the various measures of success found across the five studies. Increase in productivity, reduced time for development, client and development team satisfaction, operational excellence and feedback from discussion are considered as the most important measures of success by four of the five studies included in SR. Table 12: Comparisons of measures of success across selected studies Studies S1 S2 S4 S5 Sr. No. Measures of Success 1 Increase in Productivity 2 Reduced Development Time 3 Client Satisfaction 4 Development Team Satisfaction 5 Operational Excellence 6 Compliance with Existing Standards 7 Feedback from Discussion 8 Statistical Analysis/Use of Metrics 9 Alignment with Business Goals 10 Project Monitoring Through Reviews 8.3 Research Question 3 It states that Are there any software process improvement models that have been specifically made to measure for small and medium Web companies? None of the four studies considered are specific to the software process improvement of small and medium Web development companies. The previous studies applied conventional software SPI frameworks to Web companies without adaption to the Web context. All of them treated Web companies as a subset of small and medium software companies. However, the nature of the development processes employed in Web projects has led to the need of new processes specific to deal with these types of projects [15] [16] [24] [25] [61]. There are significant differences between conventional systems engineering and Web systems

25 23 Data Synthesis & Results engineering, some of them are also mentioned in Section 1 of this SR. Therefore, it is our argument that the way in which an organization identifies process improvement opportunities needs to be specialized for the Web. 8.4 Research Question 4 Question 4 states that What are the important characteristics of small and medium Web organizations that follow software process improvement activities and practices? None of the selected studies provided data to be used to answer this question. Table 13 identifies the data that was to be gathered across the selected studies. Table 13: Characteristics of Small and Medium Web Development Companies that follow SPI Sr. No. Web Projects Type Company s Age Target Market/s Total Turnover Average No. of Employees in Project Average Project Duration Average Project Cost Process Model (s) Used 8.5 Research Question 5 It states that What constitutes a small or medium Web organization for the studies investigated? In this SR term Web organization is considered same as a Web company which is also mentioned in section 4. Study S1 stated that small and medium companies are always concerned regarding the costs unlike the large SPI programmes where tangible improvement may not be seen for years, they do not seem convinced. According to S1 small companies have small budgets, require fast results with short deadlines and always want to avoid long term commitment. Study S2 asserted that small and medium Web companies have projects of different nature as compared to traditional software, tighter time frames, critical to strategy, increased visibility to the customers, more fine grained evolutionary maintenance and a less specific user group. They have also categorized small companies as having less than ten employees and medium companies as having between ten to fifty people. Study S3 suggested that small and medium companies typically operate on tight financial constraints; require low risk strategies and quick results.

26 24 Data Synthesis & Results Study S4 stated that small organizations constitute from three to twenty employees whereas medium constitutes from twenty to fifty employees. The average duration of a project for a small company is around six months and for a medium company is around a year. Table 14 summarizes the findings of the selected studies against Question 5. It indicates certain commonalities like, small and medium Web companies operate under tight budget constraints and with short deadlines. They like their strategies to be less risk based and they always demand quick results. Table 14: Characteristics of Small and Medium Web Development Companies Sr. No. Characteristic Small Company Medium Company S1 Budget Low Deadline Short Commitment Short Term S2 Time Frame Tight Strategy Intensive Visibility Increased for Customers Increased for Customers Number of Employees < S3 Financial Position Tight Strategy Low Risk Based Results Quick S4 Number of Employees Project Duration 6 Months 6 Months 1 Year 8.6 Drawbacks Found among the Existing Studies This section elaborates on what we believe are the shortfalls of the five selected studies. Studies S1 and S3 were at early stages of development and they themselves have stated that they have neither validated nor tested the results of their models properly. Studies S2 considers various techniques/practices to be useful for small and medium Web companies but they have not suggested any order or aggregation of these techniques/practices. The order is important since it will help in better planning and aggregation will help in synthesizing the overall improvement by defining a domain or framework. None of the studies are suggesting a new model for Web SPI but we believe that the nature of Web projects developed by the companies in our SR may have had an influence on that e.g. The Web companies that develop Web applications (e.g. some e-commerce applications, Internet banking etc.) very similar to conventional software applications may fit well with using a conventional SPI framework, as opposed to those Web companies that develop hypertext rich applications.

27 25 Research Gaps 9. Research Gaps The number of studies that met our inclusion criteria is very small, therefore the gaps identified are very wide and consequently represent a large research potential. Web Engineering is comparatively a new emerging discipline [17] where its technology also changes very rapidly as new standards, tools and protocols are being introduced [22] [49]. Therefore, investigation of SPI in the context of Web application development companies formulates an interesting research case. Selected studies S1, S3 & S4 indicate integration of existing SPI models for Web Companies but the integration phenomenon is not completely narrated and the studies do not indicate how to tailor new Web standards and procedures along with the existing SPI models. The systematic review did not identify any specific SPI model or technique made to measure for Web companies; therefore there is a clear research gap on the proposal of a specific SPI model for Web companies keeping in view the constraints and challenges. This can be achieved by extending some existing model or by proposing one from scratch. The evidence comes from the recent research where new development models have been proposed to deal specifically with Web projects e.g. new size measures for Web cost estimation [57], UML based Web engineering [51], OOHDM for Hypermedia Web applications [26], navigational design techniques(ndt) for Web [30] etc. Therefore, we argue that Web specific models or practice may also be applied to SPI frameworks that are specific to the Web. Therefore, there is a visible research gap to investigate result oriented, cheaper and lesser time consuming SPI strategy for small and medium Web companies. Selected studies S1 & S3 have indicated that most Web companies have tight financial position and they are relatively low budgeted companies. S2 and S3 have indicated that their strategies are intensive and they demand immediate results within limited time. Corporate SPI giants like SEI and ISO are also formulating focus groups to make their models more practicable and feasible for smaller sized companies [45] [46] but their explicit focus so far is not focused solely on Web companies. Therefore, there is a visible research gap to investigate the success factors for the SPI programmes targeted towards Web companies. The different context and nature of Web projects makes an interesting case to investigate how SPI should be tailored to them and what factors can be influential regarding its success. The outcomes of above research gaps would be some bridges that might help Web companies in addressing their immediate concerns and goals regarding the processes which will enhance the possibilities for longer term SPI commitment. In this case they might be able to reap the benefits of an SPI program since their very inception. 10. Conclusion Conducting a systematic literature review is a more exhaustive task as compared to traditional literature reviews [36]. It is performed by following the provided guidelines. The overall effort for conducting a systematic review is always very planned and disciplined in its case. Development of the review protocol for this systematic review was the most important and time

28 26 References consuming task as all other activities are based on the review protocol which streamlines the overall approach. Systematic reviews are an excellent source to investigate and synthesize the existing literature in order to obtain validated research gaps [39]. In this systematic review, we have investigated the current evidence of software process improvement in the context of Web companies. Due to our strict inclusion criteria, the number of relevant studies found was very small but the overall search process was very comprehensive and it was performed by following the recommended guidelines according in [36], to the best of our knowledge. Similarly data extraction and synthesis phase is also performed as prescribed by the practitioners with proper validation and quality assurance. Four studies were included in this SR and the number of investigated research questions was also five. Main objective of the SR was to investigate specific SPI models or techniques for small and medium Web companies. Found studies suggested did not suggest any specific model or technique made to measure for the SPI of Web companies. The SR also revealed the characteristics of some small and medium companies and suggested that they have tight budget constraints, have tight deadlines and they have a short term strategy. Investigated measures of success for small and medium Web companies include development team and client satisfaction, increase in productivity, compliance with standards and overall operational excellence. The review as mentioned earlier helped us in identifying the possible research gaps and directions. One of the research gaps lies in proposing a specific SPI model for Web companies, which keeps in view their characteristics and aims to help them measure their success and improve continuously. This can be achieved by either enhancing some existing SPI model or by proposing one from scratch. This is the line of our future work. 11. References [1] R.L. Glass, Software Creativity, Englewood Cliffs, New Jersey: Prentice-Hall, [2] L. Harjumaa et. al. Improving Software Inspection Process with Patterns Proceedings of the Fourth International Conference on Quality Software, 2004 [3] M. Lepassaar, T. Makinen, Integrating Software Process Assessment Models using a Process Meta Model, Proceedings of IEMC: Volume I, IEEE, [4] G. Cugola, and C. Ghezzi, Software Processes: A Retrospective and a Path to the Future. Software Process Improvement and Practice, 4, pp ,1998. [5] H.Thompson, and P.Mayhew, Approaches to Software Process Improvement. Software Process Improvement and Practice, 3 (1), 3-17,1997.