QUALITY ASSURANCE UNDER OPEN SOURCE DEVELOPMENT MODEL. Presented by: Ashwini Guttal

Size: px

Start display at page:

Download "QUALITY ASSURANCE UNDER OPEN SOURCE DEVELOPMENT MODEL. Presented by: Ashwini Guttal"

Doris Harvey
8 years ago
Views:

1 QUALITY ASSURANCE UNDER OPEN SOURCE DEVELOPMENT MODEL Presented by: Ashwini Guttal

2 White Paper Link and Credits Quality Assurance under the Open Source Development Model Download Link Authors Luyin Zhao, Philips Research USA Sebastian Elbaum, Computer Science and Engineering, University of Nebraska-Lincoln

Luyin Zhao, Philips Research USA Sebastian Elbaum,

3 Luyin Zhao : He is currently a member research staff at the Healthcare Information Technology department of Philips Research USA and a part-time Ph.D. student at the State University of New Jersey Rutgers. He has received his Master of Software Engineering and Business Management from the J.D. Edwards Honors Program at University of Nebraska-Lincoln in 2001, a Master of Computer Science from Beijing University, and a Bachelor of Computer Science from Beijing University of Aeronautics and Astronautics. Sebastian Elbaum He is an Assistant Professor in the Department of Computer Science and Engineering at the University of Nebraska Lincoln

Edwards Honors Program at University of Nebraska-Lincoln in 2001, a Master of Computer Science from Beijing University, and a Bachelor of Computer Science from

4 What led to the study of QA under open source? Delivery of successful products such as Linux, Apache, Perl and SendMail etc. Unconventional principles followed under open source This model seemed to be yielding products in rapid succession and with high quality Need to capture the attributes that make open source products successful within the area of quality assurance

Unconventional principles followed under open source This model seemed to be yielding

5 QA under traditional software development model: Set of systematic activities to generate a software product that is fit to use The effectiveness of inspections and reviews, the efficiency of testing techniques, have all been extensively investigated User participation not to the levels observed in open source development environments

reviews, the efficiency of testing techniques, have all been extensively

6 Open Source Model Promotes free distribution and complete access to source code As Raymond repeatedly observes, more people looking at the code, will results in more bugs found, which is likely to accelerate software improvements [Raymond, 1999]. User participation is the major foundation in this model.

results in more bugs found, which is likely to accelerate software

7 Findings of the previous work on the QA activities of open source model: Based on the results of the preliminary survey on open source [Zhao 2000]: Planning is not regularly done Testing takes the significant portion of software life cycle User participation usually did not include looking at the code The findings and limitations of this study served as the motivators for the current work.

the significant portion of software life cycle User participation usually did not include looking

8 Key Goals Capture the quality assurance techniques used by open source developers and their perceptions about software quality Quantify how much the software user really contributes to the software evolution Determine the impact of attributes such as project size, maturity, and programming language on how quality assurance is carried

really contributes to the software evolution Determine the impact of attributes

9 Projects selection for the survey Limited the project selection to and freshmeat.net Boiled down the selection to the most active and evolving projects. Project selected for each combination group (8 observations in each group) with an unbiased distribution across programming languages, environments, and topics.

net Boiled down the selection to the most active and evolving projects.

10 Generic findings about the survey Majority of the projects fell into medium size category being developed by small groups of less than 5 Most of the projects having user groups of more than 50 users Only few number of projects were in the market for >3 years, while half of the projects were in the market for < 6 months Most responses from developers with several years of experience

50 users Only few number of projects were in the market for >3 years, while half of the

11 Key findings on the processes followed under open source Majority of the projects started out of personal needs while some with software community in mind and few for company needs Most used configuration management tools with CVS being the highly used Most projects used bug tracking tools provided by the host web sites Larger projects made more extensive use of the configuration management and fault tracking tools Documentation did not play such a dominant role

CVS being the highly used Most projects used bug tracking tools provided by the host web sites Larger projects made

12 Key Findings About Open Source Testing Some of the validation responsibilities transferred to user Over half of the projects spend some significant amount of time on testing Larger projects tend to spend less time in their testing phase compared with smaller projects. As projects get larger, the validation techniques tend to become more mature. 35% of the Huge projects use some kind of assertions Not all the projects use baseline test suite to support testing Lack of a baseline indicating the likely absence of regression testing, Jeopardize multiple reliable releases in a short time frame

As projects get larger, the validation techniques tend to become more mature.

13 Bug tracking usage by Project Size Capture

14 Documentation by project topic

15 Testing Time by project size

16 Role of user participation in open source User participation and feedback constitutes one of the backbones of the model [Raymond, 1999] Users finding hard bugs is higher in large and huge projects than in medium and small projects. User change suggestions was more welcome in small projects indicating higher flexibility in smaller projects than that of huge projects User contributions in terms of percentage of faults found increased for more mature products. Products in the market for longer periods of time are likely to have a greater user base Users find less number of faults in projects that spend more time in testing.

User change suggestions was more welcome in small projects indicating higher flexibility in smaller projects than that of huge projects User contributions

17 Bugs found by Users

18 Code Change by user suggestions

19 Conclusion Level of user participation in open source projects was notoriously high up to 20% of the changes for almost 50% of the projects discovering 20% to 40% of the faults in 20% of the projects Almost 60% of the projects started to meet the developer's personal needs, Use of configuration and bug tracking tools reached approximately 75% of the projects Documentation was not a high priority for most projects

Almost 60% of the projects started to meet the developer's personal needs, Use of configuration and bug

20 Conclusion Most developers seemed to invest no time in utilizing any testing techniques Most developers just attempt to informally imitate user behavior Only 5% employ any tools for test coverage Lack of regression testing in spite of the high release rate observed across most projects

user behavior Only 5% employ any tools for test coverage Lack of

21 Questions?

Quality Assurance under the. Open Source Development Model

Quality Assurance under the. Open Source Development Model Quality Assurance under the Open Source Development Model Luyin Zhao Philips Research USA Phillips Corporation luyin.zhao@philips.com Sebastian Elbaum Computer Science and Engineering University of Nebraska-Lincoln