Understanding Software Quality Assurance In OSS

Understanding Software Quality Assurance In OSS Adeyemi Bolaji, Shikur Henok & Behutiye Woubshet Nema Abstract 1. Introduction 2. Overview of Software Quality Assurance 3. Quality Assurance in Open source Software 3.1 Different Perceptions of Quality assurance in OSS 3.1.1 Developers perspective 3.1.2 User perspectives 3.1.3 Organizational Perspective 4. Discussion 5. Conclusion References 1

Abstract There are lots of challenges confronting the open source software development, and one of the problems is quality assurance. And with regards to this fact, the paper investigates the responsible actors in assuring quality in open source software development. We meticulously explore, evaluate and interpret the existing literature as regards quality in Open Source Software (OSS) context. The principal findings of the study show that sustainable community participation, development processes, governance, systems and tools used to support development affect the quality assurance of open source software. The paper reflects the activities which principal stakeholders of OSS community should execute in order to achieve quality. The need to investigate on how to encourage developers adherence towards strict development processes in OSS was also proposed as future work. The rationale behind this is that, it will help address the discrepancy in development process which is a key source of quality issues in OSS. The study shows the key stakeholders that are responsible in ensuring quality in different phases of software development in OSS and the respective activities. 1. Introduction Open source communities have been rapidly attracting attentions across different fields. The underlying reason for this unprecedented development is still not completely known, although, many Researchers concluded that, the reason is due to low cost of software development, others (De, 2009; Nakakoji, Yamamoto, Nishinaka, Kishida, & Ye, 2002) argued that, because it is free, so it attracts lot of users. However, there are lots of challenges confronting Open Source Software(OSS) projects, in general. According to (A. Khanjani & Sulaiman, 2011a), identified one of the challenges to be in the area of quality of the software, and they also mentioned that, one of the biggest task of OSS is assuring quality, same conclusion is reflected in literature, for example (Nakakoji et al., 2002). Subsequently, OSS is yet to attain its full potential, as lot of critical projects do not consider using OSS mainly because there is no strict process in assuring quality, in other words, it seems no one is responsible for assuring quality. Further, Hedberg et al. (2007) predicted that, with the increase in commercial use of OSS, there will more people involved in OSS communities, particularly, the end users with cravings for quality products (Hedberg, Iivari, Rajanen, & Harjumaa, 2007). According to ISO 8402 quality assurance is define as All those planned and systematic actions necessary to provide adequate confidence that a product or services will satisfy given requirement of quality (ISO 8402). With regards to the aforementioned definition, it is logical to say that before quality can be attained there must be a strict process to be followed, and in the context of software engineering this process is called software testing process. Regrettably, Gacek and Arief in (2004) stated that OSS developers are less interested in both documentation 2

and testing (Gacek & Arief, 2004) processes vital in assuring quality. Herberg et al. (2007) also supported this claim using these words usually there is no testing plan, the test coverage is not addressed (Hedberg et al., 2007). Nonetheless, (Opensource, 2013) mentioned that OSS model is peculiar both in organizational structure and the development process. The OSS development process is not meticulously planned at the beginning of the project, it evolves in response to its users demands, as contrast to the proprietary software model. OSS software development involves distributed contributions of codes (A. Khanjani & Sulaiman, 2011a; Nakakoji et al., 2002). Thus, the software product of OSS is highly prone to defects or bugs, because the developers (contributors) may be following different development processes (modules). Although, some Researchers over the years has concluded that open source development has a greater chance of increasing the quality of product or reducing the bug, because the underlying model in open source is that, the more people that have access to the codes, the more chances of detecting bugs, contrast to the traditional closed source. In addition, OSS is unique in a distinctive way, because of its structure developer, co developer (who are regarded as contributor) and the end users, these are the stakeholders in any OSS project and they cluster around an OSS project as a community. However, without the presence of the aforementioned principal actors, there is high possibility that a given OSS project may fail (Nakakoji et al., 2002). In order to maintain a well focused literature review, we have drafted research questions as a guide to our review. Thus, the current study aim to investigate assuring quality in OSS by examining different stakeholders(participant) in the development stage, with a very focused research question of who is responsible for quality assurance in OSS projects?. We believe that, answering this question will help uncover the areas that need improvement in terms of strictly following development processes (developers) or encouragement for example increase user s participation. We intend to conduct this study, solely by analyzing literatures with the anticipation of bringing to light the gaps in literature as well as identifying, whose role it is to assure quality in the different phases of software life cycle in OSS. In addition, we assumed that, the result of the current literature review will give a broader and concise definition over who is responsible for software quality assurance in open source software development. As result, inform both research communities and practitioners. The primary search process was done using online databases (the ones Oulu University subscribed to), search engines, electronic journals, and conference proceedings. The common online resources includes; Google scholars, Web of science, Scopus, ProQuest Databases, ACM Digital library, IEEE xplore, SpringerLink, EBSCO, ScienceDirect, etc. Further, papers that discussed software quality, software defect management, and software quality assurance are the studies taken as the primary studies for the review. Additionally, the study classified the papers based on the year of publication, the research data, research method, and most importantly the findings. Furthermore, papers or scientific studies that are not free or in the database research support of university of Oulu were excluded, regardless the year of publication or the content of the paper. The papers with the above selection criteria were 3

written in English language, this is identified as one of the potential setbacks of the current study. The rest of this paper is organized as follows. Section 2 provides an overview of quality assurance in software engineering in general, as well as from open source software community perspective. Section 3 explains in detail the factors affecting quality in OSS as describe in the literature. Section 4 presents the discussion of the study. And finally, Section 5 presents the conclusion of the study. 2. Overview of Software quality assurance Many studies have been conducted in the context of assuring software quality (Gacek & Arief, 2004; Hedberg et al., 2007). Amidst these studies are definition of QA in software engineering, the best practices in assuring quality, and the factors that directly impact quality development process, process conformance, and improvement of processes as well as validating processes. Quality assurance is a way to assess whether the product has reached or satisfied customer expectation. In quality assurance process there are steps that should be followed such as: design, development or evaluation of the product by testing the result against the requirements. Although, these processes might change depending on the project and specific companies requirement. The whole theme of quality assurance is to provide good quality products and services for customers. (A. Khanjani & Sulaiman, 2011b)(A. Khanjani & Sulaiman, 2011a) Furthermore, quality assurance is a substantial part of software development as it can be related to correctness, usability, completeness and maintainability of software which affect economic and technical decisions made during software development. Software quality impacts a number of important factors in our daily lives such as economy, personal and national security, health and safety (Naik & Tripathy, 2011). Moreover, advancements in technologies and complexity of software systems also make it necessary to understand and achieve quality assurance in our products and development processes. This is also supported by (Otte, Moreton, & Knoell, 2008a), The execution of SQA activities requires the introduction of a quality model to obtain process, transparency, customer satisfaction, repeatable processes and methods. In proprietary software development process, quality assurance process includes different steps performed by developers and testers along the development process. During the process different standards have been applied to assure compliances with set of requirements. Performing quality assurance has been a problem in open source software development. The method used in traditional software development process is not pertinent to OSS development since OSS development life cycle is unstructured and disordered. (Ardagna, Banzi, Damiani, & Frati, 2009). Moreover, Software QA needs an integral collaboration of different elements (stakeholders) in the software development ecosystem and a follow up of systematic procedures to attain quality. It is highly important to ensure that processes and tools are utilized 4

in efficient manner to achieve quality. Tuteja and Dubey (2012), argue that quality cannot be improved by assessing an already completed product and highlight the importance of quality assurance measures targeted to improve software development procedures, standards and tools used in the development. In addition, arguing software testing techniques as measurements of software quality the authors depict that applying software testing methods in different lifecycle stages of software development contribute to ensuring software quality. (Tuteja & Dubey, 2012) Further, to achieve quality product, the conformance to the development process must be strict and followed also in distributed teams (Ardagna et al., 2009; Tuteja & Dubey, 2012). Sivert (1997) defines process conformance in the context of software development, informally as the degree of agreement between the software development process that is really carried out and the process that is believed to be carried out (Sørumgård, 1997). The study aimed to uncover the importance of process conformance in software engineering field. As a result the study helped to propose four key components to measure process conformance in software development. The components are model for definition, measurement based on the model, an alternative measurement and guidelines for modification of a process, so as to enable conformance measurement (Sørumgård, 1997). Rigidly, conforming to the processes of development, subsequently assuring quality of the software product. In addition, we can infer that, if software products have low quality the reason maybe that the process of development is not conformed to or the process of development requires some improvement. Bhandari et al. (1993) selected a narrowed aspect of software process improvement, which is defect based oriented. They were motivated by the inconsistency in implementation of processes in the context of software development. However, the study responded to a practical question by a developer; I know what I must produce; I know the process I must follow; and I have started development. How do I detect if things are going wrong, and what do I do to get back on track? Bhandari et al., (1993), also to create process improvement metric during development, further, the research method is a case study of a developing team of 25 people and the main conclusion of the study was that process improvement works (Bhandari et al., 1993). This implies that during development process, developing team (developers) could continually improve their processes, thus assuring quality. In terms of development processes, it is highly important to validate the process. A relevant study in this regard was conducted by Cook and Jonathan (1999), the study provides deeper insight in the relationship between a formal process model and process execution. As taken from the study; When process models and process executions diverge, something significant is happening. They investigated amidst the metrics involved in the processes, the human or behavioral aspects. Further, they proposed three metrics for process validation; Recognition Metric (REC), Simple String Distance Metric (SSD) and Non linear String Distance Metric. These metrics were demonstrated using the Test Unit task from (Kellner et al., 1990). This is unique in the sense that it uses the data in process validation activity (Cook & Wolf, 1994). In addition, a recent study by Dimech et al. (2013) aimed on improving process conformance as 5

regards to the requirements in the context of design driven development. They support the idea that initial requirement can be traced from high level design forms through the final product. Furthermore, the authors proposed an insight of handling issues relating the problem space among which is conformance using DiaSuite methodology (Dimech & Balasubramaniam, 2013) 3. Quality Assurance in Open Source Software Unlike proprietary software development model which bases on systematic and procedural testing, the open source software development highly utilizes post delivery bug reports in order to establish Quality assurance of software. Developing open source software products requires specific methods and techniques for quality assurance process to control the quality of the product (A. Khanjani & Sulaiman, 2011a). In addition, quick evolution of the software development process which is mainly driven by various actors including developers, users, firms and other participants is also another unique characteristic of the OSS model contributing to Quality assurance. As stated by Bonaccorsi and Rossi (2003), increased participation brings project momentum and frequent releases lead to bugs correction and generate feedback flows that improve the code and the understanding of the underlying problem, a practice in contrast to the conventional way of assuring quality in proprietary software (Bonaccorsi & Rossi, 2003). A survey research on applied quality assurance methods of OSS (Otte, Moreton, & Knoell, 2008b) found out that most successful OSS projects apply structured and organized development processes. The study also showed that engaging experienced project management, skilled developers and users in the OSS community is substantial for quality assurance. Moreover, it was also shown that it is essential to implement guidelines, documentations, standards and processes to improve quality. The findings support the indispensable inclusion of a wide and experienced OSS community together with adoption of systematic tools and development processes to achieve software quality assurance.(otte, Moreton, & Knoell, 2008b) Further, high Quality Open source software development depends on fundamental sector of sustainable community, code modularity, project management and test management. In order to attain software quality, OSS practitioners should fully understand the dependency between these sectors. Many researchers concluded that having a large sustainable community should be the main objective in OSS. A research on OSS project by Mark Aberdour (2007) concluded that a large contributor presence was one of the main factors for high OSS quality (Aberdour, 2007). Also, Code modularity is still one of the main factors for OSS quality which programmers can extend the program by working on separate modules, without the need to know the full understanding of the system. This minimizes the risk of new bugs being introduced in other modules. In addition, peer review and people management play a big role for OSS quality. Peer review mainly focuses on assessing whether the contribution quality compliance with the code base. Moreover, people management plays a great role on establishing an effective 6

environment and culture; it is seen as important as system design. (Aberdour, 2007) Furthermore, QA in open source software depends on two processes: code review and data testing. The accessibility of open source software allows anyone to analyze and make thorough assessment and get more bugs. Thus, having large community for knowledge sharing, paid and unpaid contributors, expert developers for controlling and managing bugs, etc. For example, open source software like GNU/Linux, Apache, Firefox, etc., receive more bugs due to the fact that different contributors share their idea that leads to the possibility of increased bugs to be released. (A. Khanjani & Sulaiman, 2011a) Referring to the study of Nakakoji, Kumiyo, et al. (2002), a full understanding of the evolution of an OSS system cannot be complete without understanding the evolution of the OSS community and its roles in driving the evolution of the system (Nakakoji et al., 2002). Hence, we aim to examine the role of each stakeholder as regard to quality of product in OSS project. The perceptions covered in this study includes; developers core and contributors, users and organization. Also, this classification is informed by the popular onion model of OSS pattern. Figure 1, adopted originally from the onion model, shows that sustainable community is constructed from small number of core developers and increasing number of contributing developers, bug reporters, users and organizations. Sustainable community is like multi layered onion with users in the outer layer and move towards core through bug reporters and contributing developers. A user increases their involvement on the basis of their knowledge through the process. As the users move towards the core, user becomes bug reporters, contributing developers, and core developers consecutively. Stakeholder involvements in different stages of open source software development are explained by drawing a metaphor based on the onion model as shown below. In order to achieve high quality Open Source Software, the OSS community should organize processes from various perspectives: organization perspective, user perspective, developers perspective and take measures to assure quality (Aberdour, 2007). 7

Figure 1. A metaphor based on the onion model (Aberdour, 2007), shows sustainable community in quality assurance from different stakeholder perspectives. A number of models and methods have been introduced to ensure software quality. Some of the recommended practices Hedberg et. al (2007), suggests organization and companies should follow stricter quality assurance methods, since the commercial aspect of OSS is increasing abundantly (Iivari, Hedberg, Rajanen, & Harjumaa, )(Hedberg, Iivari, Rajanen, & Harjumaa, 2007). In addition, it is stated that planning and documenting quality assurance process eases the communication problem between the increasing developers and users in OSS. It is also suggested that more emphasis should be given to testing and Test Driven Method (TDD) method should be used in the development process. Others (Wahyudin, Schatten, Winkler, & Biffl, 2007), also concluded that software testing and peer reviews help to reduce software defects hence software quality is ensured. User, developer and organizational participations are seen as the main stakeholders essential to maintain QA of OSS. The user participation and feedback constitutes one of the backbones of the open source model (Raymond, 2008) Different studies have showed that OSS projects that have the support of firms and a large scale of user community benefit from participation of these actors in assuring Software quality. Therefore, it is important to understand different aspects of stakeholder involvements and how each stakeholder s participation affects quality of the software. Hence, with a focused research question who is responsible for quality assurance in OSS? Sunindyo et al. (2013) study the business process in connections to OSS and proposed a framework that could help OSS project managers to efficiently manage the business process. 8

Owing to the fact that, it is the responsibility of the OSS project managers to manage the development process and delivering of product to the End users. Although, this role could be demanding, considering the collection and analysis of various data collected from different stakeholders. Further, the proposed framework consists of four levels; data source, data collection, data integration and the validation level. The result of the study shows that, Open source Model framework (OSMF) can assist managers handle business process more efficiently, as well as enhancing judgment on project quality. (Sunindyo & Ekaputra, 2013) Abdou et al. (2012) studied popular OSS projects; Apache HTTP server, Mozilla Web browser, and NetBeans IDE. Their focus was to analyse the test processes undertaken in those projects. The results revealed that, three activities were in compliance to the Test Process Standard ISO/IEC, although, significant dissimilarities were noticed in regards to the task associated with the test process practices. They proposed a framework for software test process in OSS called (OSS TPF), which was compared with ISO/IEC Test process to reveal the areas that requires improvement in achieving quality product from OSS software development. (Abdou, Grogono, & Kamthan, 2012) In Rashid et al. (2012), studied perception of quality of OSS product with the focus of investigating the process practices and problems associated with the development process from the perception of the developers. The result of the study revealed that, there is no coordination among the developers, because of the distributed nature of OSS developmental model. However, they highlighted the relevance of coordination between developers, thus, encourage developers to coordinate, since it makes the release process systematic. As a result assist to obtain the objective of software release process. (Rashid, 2012) 3.1 Quality assurance involvements in OSS Different sectors of the open source software community play roles in quality assurance process. Along community participation, governance, methodologies (development processes), systems and tools contribute to ensuring quality in the open source software platform. However, after reviewing processes and participations in this literature review only three key stakeholder involvements will be emphasized to describe quality assurance participation and roles. 9

3.1.1 Developer perspective Roberts et al., (2006) studied the motivations behind developer communities of the Open Source software platform and showed that there is a significant interest for ensuring quality besides regular contributions. The authors argue that developers interest in reputation in a specific knowledge domain as a driving force for contributions of quality work. In other words, developers intention to seek a status and reputation for specific skill can be related to achieving Quality in OSS. (Roberts, Adviser Hann, & Adviser Slaughter, 2006) Luyin and Sebastian (2002) studied OSS quality assurance techniques and perception of QA by OSS developers. The study shows that, in addition to high level of user participation, developers contribution is indispensable element in ensuring quality assurance under the OSS development model. Discussing practices of developers, it was also shown that often OSS developers act as users to validate functionality of software rather than following formal procedures of traditional software testing. (Zhao & Elbaum, 2003) 3.1.2 User perspective User participation in OSS software development is seen as crucial element that is needed to acquire quality and improved product. OSS projects follow models that encourage user involvement in order to facilitate development processes and assure quality. This is also supported by different literatures and theories. Raymond E. on the Cathedral and the bazar argues the importance of having users, Treating your users as co developers is your least hassle route to rapid code improvement and effective debugging. (Raymond, 2008) Furthermore, the level of participation of users can also be related to the success and failure of OSS projects. High level of user participation is seen as a common pattern and driving force of QA behind most OSS projects. Tobias et al. (2008) proposed a quality assurance process framework for ensuring quality in OSS. The key factors emphasized in the design of the framework included, balanced interaction of human factors, management skills and operating environments of organization and development measures etc. These factors are shown to contribute to quality assurance of OSS products. Moreover, the authors argue that QA is highly influenced due to user testing and debugging, therefore projects need to concentrate on enabling tasks, such as structured organization, good documentation and effective communication. User s skill in a specific knowledge domain together with the scale of an open source project can be used to determine the value of user participation in OSS software quality assurance. (Jullien & Zimmermann, 2009) show how users contribution can be affecting the quality of OSS by introducing the concept of dominant user skill. Emphasizing on user base perspective,users 10

skill level are used to show how product quality can be improved with bug reports, patches( fixes) and new features from experienced users. 3.1.3 Organisation perspective Currently, different organizations are on the process of changing their critical systems and applications to open source application, although, this decision is still highly restricted by the quality of software in OSS. However, the underlying motivation in the adoption of OSS can be justified by reduced costs and improved quality of software that exploits experiences of bigger community. Sarrab et al. 2013, demands for improved and additional features along maintaining product quality is also additional reason for organizational participation in Open source software.(sarrab & Rehman, 2013) Organizations involved in development and usage of complex and mission critical software applications find security pitfalls of the OSS as the main problem in integrating (adopting) OSS into their platforms. And as a result, companies and enterprises are interested in pragmatic solutions that enable assurance of OSS software. Hence, organizations will be involved in OSS projects by initiating and making contributions to assure quality aspects of a project. Claudio et al. (2009b) show how assurance plays a key part as requirements for telecom companies adopting OSS for their critical systems. In addition, the authors state that The criticality and complexity of Telco systems (e.g., for OSS/BSS systems) require the analysis and control of software quality during the whole development process (ensured by the quality or assessment levels of the factory or of the partner companies or vendors) (Ardagna et al., 2009b). This shows organizational participation and contribution have higher importance in ensuring quality of OSS projects. 4.0 Discussion In this literature review, a research question for determining principal actors of quality assurance behind open source software was established as a basis of the study. And hence, areas related to quality assurance, development processes and community interest of participation have been covered in the study to determine roles and participation in ensuring quality in OSS. The principal findings of the study show that sustainable community participation, development processes, governance, systems and tools used to support development affect the quality assurance of open source software. By emphasizing on interests behind participation and processes, three stakeholder involvements have been used to determine OSS quality assurance roles and responsibilities. However, the findings also showed that governance models, level of user and developer participation and scale of project affect the quality assurance of open source software as shown in Figure 2. These findings also align with Aberdour (2007), which argue that 11

High quality OSS relies on having a large, sustainable community results in rapid code development, effective debugging, and new features (Aberdour, 2007). Figure 2. Factors influencing quality in OSS development. The participation of different communities: users, developers and organizations together with governance, systems and tools used in OSS development play an important role in assuring quality process. In order to support the idea of sustainable community participation behind quality assurance in OSS, we can further analyze the relationship between different actors and activities involved in OSS projects. As it can be shown from Table 1, it is essential to have participation of organizations and developers in development and post release phases. In these stages, the actors (quality assurers) can enhance quality of software by creating coordination, utilizing efficient systems and tools (test processes, quality code, patches.), following governance models that encourage contributions etc. User participation (end user) is also shown in activities related to bug reports, new feature suggestions, detecting bugs which mainly occur in post 12

release phases of OSS projects. Moreover, the table depicts interdependence among the (quality assurers) actors in ensuring quality of OSS. QA Assurer Activities Involvement Developers (core developer,contributor) Govern code review peer review develop test process strict development process coordination bug report patch fixes Development & Post release phase Users (end users) Detect defect report bugs suggests features Post release phase Organizations Govern coordination develop process test Sponsor(Fund) Development,post release Table 1. Quality assurance activities in OSS and stakeholder involvements Table 1 above, shows the recommended activities that each stakeholder should perform in assuring quality in OSS. The core developers and contributors have more similarities in terms of activities recommended, however, the contributor is required to follow strictly the development process, as well report bug. Similarly, the End user is expected to also report bugs and suggest additional features. Organizations should also govern, coordinate and develop processes. As reflected in Table 1. There is a need to investigate how to encourage developers to follow development process strictly in OSS. This will address the discrepancy in development process which is a key source of quality issues in OSS. Even though the study findings show that mainly organizational, user participation and developer roles of involvement contribute to quality assurance of OSS, it lacks to support a detailed analysis of different community participations and organizational structures of OSS models. In addition, categorizations of the OSS community (stakeholder) involvements made in the study concentrate on developer, user and organizational participation. However, this comes as a drawback since there are OSS projects with defined QA teams, project management roles, 13

documentation roles, software testing team and support roles which are responsible to quality assurance to different degrees. 5.Conclusion In this review, we aim to answer the question who is responsible for assuring quality in OSS project? Toward that, we meticulously explore, evaluate and interpret the existing literature as regards quality in OSS context. Consequently, the existing knowledge revealed key factors that have direct influence in assuring quality in OSS, these factors include: developers, contributors, organization, and the End users. It is equally important to emphasize that, OSS project is different from traditional software development because it is situated within a community, and as such a holistic approach should be employed when investigating OSS project The major hindrance in assuring quality in OSS is the fact that there is no formal planning, along with developers lack of conformance to development processes, consequently, making assurance of quality uncertain. To this end, the existing literature suggested important activities that should be performed by the aforementioned stakeholders in the preceding paragraph. The findings are in accordance with the literatures explored in this review, that developers should provide a development plan, as well as encourage the contributor to strictly adhere to the process. Similarly, contributors in the other hand are advised to follow development process (if any), when developing software, they should also report bugs, fix patches. Finally, the End users are expected to report defects and suggest features. The contribution of this literature can be observed from two perspectives first, for practitioners this review enhances decision making process in terms of quality by identifying different activities that should be performed by the key stakeholder, as well as it uncover areas for improvement. Secondly, for academia, it provides deeper understanding of how quality assurance is practiced in reality, and also it suggests future work. Future research should focus on raising quality assurance awareness in the open source community and also encouraging uniformity of development processes among developers. 14

References Abdou, T., Grogono, P., & Kamthan, P. (2012). A conceptual framework for open source software test process. Computer Software and Applications Conference Workshops (COMPSACW), 2012 IEEE 36th Annual, 458 463. Aberdour, M. (2007). Achieving quality in open source software. Software, IEEE, 24(1), 58 64. Ardagna, C. A., Banzi, M., Damiani, E., & Frati, F. (2009). Assurance process for large open source code bases. Computational Science and Engineering, 2009. CSE'09. International Conference on,, 3 412 417. Ardagna, C. A., Banzi, M., Damiani, E., Frati, F., & El Ioini, N. (2009). An assurance model for OSS adoption in next generation telco environments. Digital Ecosystems and Technologies, 2009. DEST'09. 3rd IEEE International Conference on, 619 624. Bhandari, I., Halliday, M., Tarver, E., Brown, D., Chaar, J., & Chillarege, R. (1993). A case study of software process improvement during development. Software Engineering, IEEE Transactions on, 19(12), 1157 1170. Bonaccorsi, A., & Rossi, C. (2003). Why open source software can succeed. Research Policy, 32(7), 1243 1258. Cook, J. E., & Wolf, A. L. (1994). Toward metrics for process validation. Software Process, 1994.'Applying the Software Process', Proceedings., Third International Conference on the, 33 44. De, R. (2009). Economic impact of free and open source software: A study in india. Dimech, C., & Balasubramaniam, D. (2013). Maintaining architectural conformance during software development: A practical approach. Software architecture (pp. 208 223) Springer. Gacek, C., & Arief, B. (2004). The many meanings of open source. Software, IEEE, 21(1), 34 40. 15

Hedberg, H., Iivari, N., Rajanen, M., & Harjumaa, L. (2007). Assuring quality and usability in open soruce software development. Emerging Trends in FLOSS Research and Development,, 2 2. Iivari, N., Hedberg, H., Rajanen, M., & Harjumaa, L.Assuring quality and usability in open source software development. Jullien, N., & Zimmermann, J. (2009). Firms' contribution to open source software and the dominant user's skill. European Management Review, 6(2), 130 139. Kellner, M. I., Feiler, P. H., Finkelstein, A., Katayama, T., Osterweil, L. J., Penedo, M. H., & Rombach, H. D. (1990). Software process modeling example problem. Software Process Workshop, 1990.'Support for the Software Process'., Proceedings of the 6th International, 19 29. Khanjani, A., & Sulaiman, R. (2011a). The process of quality assurance under open source software development. Computers & Informatics (ISCI), 2011 IEEE Symposium on, 548 552. Khanjani, A., & Sulaiman, R. (2011b). The process of quality assurance under open source software development. ISCI 2011 2011 IEEE Symposium on Computers and Informatics, 548 552. Naik, S., & Tripathy, P. (2011). Software testing and quality assurance: Theory and practice John Wiley & Sons. Nakakoji, K., Yamamoto, Y., Nishinaka, Y., Kishida, K., & Ye, Y. (2002). Evolution patterns of open source software systems and communities. Proceedings of the International Workshop on Principles of Software Evolution, 76 85. Opensource, i. (2013). Open source. Retrieved 15.10, 2013, from opensource.org Otte, T., Moreton, R., & Knoell, H. D. (2008a). Applied quality assurance methods under the 16

open source development model. Computer Software and Applications, 2008. COMPSAC'08. 32nd Annual IEEE International, 1247 1252. Otte, T., Moreton, R., & Knoell, H. D. (2008b). Development of a quality assurance framework for the open source development model. Software Engineering Advances, 2008. ICSEA'08. the Third International Conference on, 123 131. Rashid, M. M. (2012). Release process on quality improvement in open source software project management. Journal of Computer Science, 8 Roberts, J. A., Adviser Hann, I., & Adviser Slaughter, S. A. (2006). Open source software participation: Developer motivations and performance. Sarrab, M., & Rehman, O. M. H. (2013). Selection criteria of open source software: First stage for adoption. International Journal of Information Processing and Management, 4(4), 51 58. Sørumgård, S. (1997). Verification of process conformance in empirical studies of software development. Department of Computer and Information Science, the Norwegian University of Science and Technology, Sunindyo, W. D., & Ekaputra, F. J. (2013). OSMF: A framework for OSS process measurement. Information and communicatiaon technology (pp. 71 80) Springer. Tuteja, M., & Dubey, G. (2012). A research study on importance of testing and quality assurance in software development life cycle (SDLC) models. International Journal of Soft Computing, 2 Wahyudin, D., Schatten, A., Winkler, D., & Biffl, S. (2007). Aspects of software quality assurance in open source software projects: Two case studies from apache project. Software Engineering and Advanced Applications, 2007. 33rd EUROMICRO Conference on, 229 236. Zhao, L., & Elbaum, S. (2003). Quality assurance under the open source development model. 17

Journal of Systems and Software, 66(1), 65 75. 18