A Social Networking Approach to F/OSS Quality Assessment

Transcription

1 A Social Networking Approach to F/OSS Quality Assessment Anas Tawileh 1, Omer Rana 2 and Steve McIntosh 3 1 School of Computer Science, Cardiff University 5 The Parade, Cardiff CF24 3AA, UK, m.a.tawileh@cs.cardiff.ac.uk, WWW home page: 2 o.f.rana@cs.cardiff.ac.uk WWW home page: 3 S.B.McIntosh@cs.cardiff.ac.uk Abstract. With the growing number of available Free and Open Source Software (F/OSS) applications, choosing between them becomes increasingly difficult. The concept of trust in social networking has been successfully applied to facilitate choice in similar situations. We propose a social networkbased approach to quality assessment and evaluation of F/OSS applications. The proposed system utilises the community formed around F/OSS projects to produce meaningful recommendations based on specific user preferences. We suggest that such an approach would overcome some of the difficulties complicating user choice by making useful suggestions and can fit seamlessly within the structure of the majority of F/OSS projects. The main focus of this work is on the end users of free and open source software and not on the developers of the software. The social network-based approach would apply differently to these different user classes. Keywords: Social Networks, Free and Open Source Software, Quality Assessment, Trust. 1 Introduction The number of software applications developed within the Free and Open Source Software (F/OSS) movement has been growing at a significant rate. On April 24, 2008, 175,470 F/OSS projects were hosted on Sourceforge ( and 44,532 projects on Freshmeat ( While this variety offers choice, picking the most suitable software becomes a real challenge, especially as F/OSS projects vary in their quality and features. For each particular requirement, dozens of projects may be suitable. Choosing the most appropriate software becomes a difficult task, and quality assessment approaches for F/OSS become a necessity. The F/OSS software development paradigm differs fundamentally from commercial approaches. F/OSS software is built by a large number of volunteer contributors, communicating with each other using online tools and platforms. Development involves a transparent process where all source code is kept open to

2 facilitate peer review and bug discovery [1]. It has been argued that these distinct characteristics of the F/OSS paradigm enable the creation of higher quality software that rivals its commercial counterparts [2, 3]. These claims are usually supported by examples of very successful and popular F/OSS projects such as the GNU/Linux operating system and the Apache Web server. However, little empirical research has been done to confirm this [4, 5]. Although many publications acknowledge the potential of the F/OSS development processes to produce high quality software, this has not yet been effectively demonstrated. Moreover, the concept of quality itself is vague and subjective. Quality may be perceived differently by different people, and varies considerably according to the context in which it is applied. This makes the achievement of a consensus on a universal concept difficult. A new approach to F/OSS quality assessment based on social networks is presented. The problem area is briefly discussed in order to lay the foundation for the work, followed by a review of related work. The proposed system is then described with an illustrated example of its applicability. A prototype of the proposed system is then presented, concluding with a brief discussion of the implications of the suggested approach for quality assessment of F/OSS projects, and possible directions for further research. 2 Background The problem of quality assessment and evaluation of Free and Open Source Software (F/OSS) is increasingly attracting the attention of both researchers and practitioners [2,3,4,7]. An increase in the number of available F/OSS projects presents users with a real challenge in choosing the most suitable software for their own requirements [6], as openness and choice come at a price. Practical solutions and approaches to facilitate choice are acutely needed. Different aspects of the F/OSS development paradigm make the objective evaluation of software artefacts difficult. The F/OSS development process is open, potentially involving a large number of developers submitting contributions that may have significant variation in quality. Moreover, there is not a single, well understood set of processes that are applied in a F/OSS project. Michlmayr et al. [7] argue that software processes adopted by F/OSS projects are diverse and vary in their maturity, which eventually affects the success of these projects. Another issue is the subjective nature of the notion of quality itself. The International Standards Organisation, for instance, defines quality as: the totality of features and characteristics of a product or service that bear on its ability to satisfy specified or implied needs. [8] Another definition of quality is proposed by Juran as being fitness for user [9], and Crosby considers quality to be conformance to requirement [10]. These definitions clearly link quality to the context within which the concept is applied, and acknowledge the need to consider the perception of the user in assessing quality. The increasing number of lines of code of F/OSS projects, and contributions from developers, coupled with the voluntary, and consequently volatile nature of F/OSS participation make a rigorous testing regime out of the reach of many projects.

3 The same distinct features of F/OSS that facilitate peer review and rapid development of software may also complicate quality assurance. Due to the open nature of the development process, most of the testing is done informally through frequent cycles of release, defect discovery, bug reporting and fixing. Although some F/OSS projects have implemented systematic testing and quality control processes, most of the projects lack these capabilities [11]. In addition, as F/OSS projects employ a variety of different programming languages and styles, there is a clear shortage of testing tools. This diversity of styles and scarcity of tools limits the utility of any approach to quality assurance that is based solely on testing - thereby making an objective comparison between different projects difficult. Someone who is looking for a suitable content management system (CMS), for instance, may find a dozen different projects utilising different technologies and programming languages. Furthermore, each project may have specific strengths in particular areas and weaknesses in others. The context within which the software will be used and the required functionality play a significant role in deciding the most suitable project for a user s needs. A useful approach to quality assessment of F/OSS should acknowledge the specificities of this distinct development paradigm. It should account for the diversity of practices, processes and technologies employed by F/OSS projects, the difficulties of applying strict quality assurance mechanisms, and the open and transparent nature of the development processes. 3 Related Work Several attempts were made to address the issue of software quality assessment in general, and within F/OSS in particular. The online communication platforms and tools used in the development process (Concurrent Versioning System CVS, Mailing Lists, Bug Tracking Systems Bugzilla, and online discussion forums) contain considerable data about the software project throughout its development phases. F/OSS software repositories such as Sourceforge provide instant access to this information and offer specialist tools for mining this data. This provides a useful first step to derive information about a particular project. The Open Source Maturity Model (OSMM), developed by Navica [12], helps in determining the maturity level of a F/OSS software application through a formal assessment process. Such evaluation is achieved using a description of the software, support, documentation, training, product integration and professional services. OSMM consists of three phases. In the first phase, the product s elements are evaluated through a 4 step process. The second phase entails weighting the evaluations of elements obtained in the first phase so that they reflect user needs. An overall maturity score for the software is calculated in the last phase -- to assess the suitability of this software for the user. Another approach is the CapGemini Open Source Maturity Model [13], which suggests 27 indicators within 4 groups (product, integration, use and acceptance). An overall score can be calculated using these indicators to determine the maturity level of any open source artefact. A feedback phase where the effectiveness of the utilised

4 indicators can be evaluated is also mandated. The model suggests that a continuous evaluation of such indicators would allow evaluation within a changing environment, which is particularly important for the development approach adopted in the F/OSS community. Atos Origin has developed the Method for Qualification and Selection of Open Source Software (QSOS) to qualify, select and compare Free and Open Source Software in an objective, traceable and argued way [14]. The method consists of 4 interdependent steps: the definition of software families, licenses and communities, evaluation of the proposed software, qualification of the software based on the specific user context and the selection of the particular piece of software that fulfills user requirements. The Open Business Readiness Rating, initiated by Intel Corporation, the Centre for Open Source Investigation at Carnegie Mellon University and SpikeSource [15] is another approach for the evaluation of F/OSS which is intended to enable the entire community (enterprise adopters and developers) to rate software in an open and standardized way. The rating weighs a number of factors, such as: functionality, quality, performance, support, community size and security. A recent effort in the evaluation and quantification of F/OSS project quality is the Open Source Software Quality Observation (SQO-OSS) [16]. This system is based on the automated analysis of the available data sources of the project (such as CVS, mailing lists and bug management databases) to derive a quality metric for a software product. Although the frameworks identified above provide a quality assessment based on information sources associated with a F/OSS project, they still have significant limitations. The variety of methods and inputs they use confirms our argument about the difficulty of applying well defined methods to the loosely defined and highly diverse nature of F/OSS projects. How these frameworks compare with each other and how close their results might be is an interesting area for further research. Furthermore, the use of any of these frameworks (except the SQO-OSS) requires considerable effort and resources. Sharing the results of evaluation is also an issue. Tawileh et al. [17] proposed an evidence-based approach to quality management in F/OSS, primarily focusing on communicating quality related information to users outside the developers community. The suggested approach called for higher levels of user participation and the integration of feedback and rating systems into the F/OSS software quality evaluation process. In this paper, we propose an approach to quality assessment of F/OSS based on the idea of social networks. Social networks involve the creation of interconnected structures where individuals can form trusted relationships based on their associations with others [18]. The focus of this paper is on quality issues as perceived by users associated with an open source project, and not on the interactions (or social network) within the developers of the project. A somewhat related effort is the identification of social networks when analysing a developer community. Wagstrom [34] et al. focus on automatically deriving the social network within the developer community by analysing communication logs (such as mailing lists, web logs, etc) between individuals associated with a particular project. Their analysis is based on the Advogato.org repository, and used as the basis to infer the quality that one may associate with a developed artefact and the general cohesiveness within the

5 community associated with the project. Gao and Maddy [35] follow a similar approach, essentially investigating the topology of interaction between developers within the Sourceforge.net community. They use network analysis approaches, such as path analysis and clustering that can be observed between developers and projects within Sourceforge. Their intention is to better understand how such developer networks evolve over time. 4 F/OSS Quality Evaluation While most of the online repositories of F/OSS software store substantial amount of data about software projects, these resources are rarely utilised to support the evaluation and assessment of project quality. Although such repositories provide useful tools to facilitate project development and communication, little functionality exists to aid the user in evaluating and selecting the most appropriate software from the tens of thousands of those available. Freshmeat.net provides a rating metric for projects listed in its website. However, the presented rating is based on the average value of all ratings of a particular project, which is less indicative of the real virtue of a project than a trust aware recommendation. We propose a social networks based quality assessment and evaluation to complement on-line F/OSS repositories. Social networks have been utilised successfully in other contexts, for instance Epinions.com and Amazon.com tap into social networks formed between visitors to derive recommendations [19]. We believe that by extending the same approach to online F/OSS repositories, users can choose the most appropriate software in a much faster and more efficient manner. Wei et al. [20] propose a system where users are requested to rate the recommendations provided by a system in order to establish a user profile that could be used to enhance future recommendations. Two different approaches can be utilised to derive such recommendations. In content-based filtering systems [21, 22] items are compared to the user s previous ratings to determine their similarity and decide whether these items will be of interest to the user. On the other hand, collaborative filtering systems [19, 23, 24] use the historical ratings of different users who have similar profiles to determine whether a specific item may interest the user. The recommender system has several limitations: for instance, it may not be able to accurately establish the profile of new users and therefore is incapable of predicting their preferences because they have not rated enough items yet [25]. New items face the same problem as well. This may cause the system to become ineffective if it does not have enough users. A trust-aware approach [26] was proposed to overcome these limitations. In this approach, the user identifies another user in whose ratings she has some confidence; recommendations are then made by calculating the trust level based on the structure of the social network between these two users. The following example illustrates the application of a trust aware quality assessment system. A search for the terms Content Management System on Sourceforge yielded projects. This number clearly demonstrates the problem of selecting the most appropriate project for a specific requirement. For the purposes of this example, we chose Exponent a feature-rich CMS written in PHP. It was

6 selected because it has an established and active community (consisting of 1 project manager, 11 developers and 3 testers). As of 7 January 2008, the different forums of the project contain 7,071 posts. The bug tracking system contains 254 entries. The hierarchy of the project illustrated in Figure 1 resembles the structure reported by Crowston and Howison [27], which they suggest is a good indicator for a healthy community. Fig. 1. Exponent CMS Project Hierarchy (adapted from Crowston and Howison [27]) Trust establishment relies on users rating specific projects from the repository. Other users will read these ratings and evaluate their usefulness based on their own experiences. They will then rate other user s ratings. The more positive ratings a user gets, the more trustworthy he becomes. In order to maintain a persistent identity of raters, most online portals provide a registration process to establish a unique user identity (although this does not prevent the user having multiple accounts). Sourceforge establishes user identities through a user name and a unique user id. r u (p,c) represents the review left by user (u) of project (p) within the context (c). Context in this regard may either refer to the category of the project under consideration (such as content management systems, graphics, etc) or to a specific feature of the project (such as usability, security, scalability, etc). For the purposes of this paper, we take context as the category under which the F/OSS project has been classified in the repository. The basic principle of trust-aware recommendations is the level of trust people can place on a certain rating or recommendation. Trust is a relative concept; it is very context-specific and it changes over time. For any trust-aware recommender system to work effectively, it must take into account these principal characteristics of trust. Members of the social network formed around a particular F/OSS project usually possess specific skills and expertise, qualifying them to occupy certain roles within the network. Because of the diversity of technologies utilised in F/OSS development, members of the network may be more qualified to rate and recommend projects based

7 on a limited subset of these technologies. Categories are usually used by online community portals to enable combining members of similar background or interests into specific groups in order to produce more sensible interactions. Fig. 2. Social Network of a F/OSS Trust-Aware Recommendation System A useful feature of trust derived from social networks is the transferability of trust between different projects, but within the same context. For instance, a recommender who is trusted for Content Management Systems may not be trusted for Graphics software. Ratings and reviews from different users differ in their quality and usefulness. We propose a structure for the rating system (Figure 2) similar to the structure of a F/OSS project. For instance, in F/OSS projects, users who submit consistently high quality contributions are considered to be part of the core development team. Members of the core development team can commit their contributions faster, without waiting for others to review their code. They also play a larger role in steering the project. In other words, members of the core development team are more trusted in the context of the development of this particular project. Likewise, our proposed rating system takes into consideration the quality and quantity of ratings provided by a user. Users who submit reviews that are consistently found to be valuable acquire greater trust. We propose the designation of an Expert status to be granted to certain users based on their contributions. This status would be awarded within a specific category, and re-evaluated over time to account for the possible changes in the user s recommendations or the frequency of contributions. Users may support their ratings with empirical evidence. They can use testing tools, for example, to back their rating of a particular project. The use of testing tools will result in higher weighting for the rating. The weight assigned to each tool depends on the community acceptance of the tool. This value will depend on the

8 quality of the tool as seen by the users community, and the ratings the tool is receiving. A good indication about the community s acceptance of a particular testing tool can be deduced from the relative number of ratings in which the rater used this tool, compared to the total number of ratings in the category in which any testing tool is used at all. Ratings should be weighted to reflect the expertise of raters and whether their reviews are supported by any tools. The weight of the rating of any project within a specific category can then be calculated as follows: we i α1 = r ar w v nu + α 2 + α 3 + β α 4 aw av nt α + α + α + α where: r: number of ratings submitted by the user ar: average number of ratings within the same category w: number of members in the rater s web of trust aw:average size of the web of trust within the same category v: the average vote of the rater by other users av: average vote of all raters within the same category β: binary variable that equals 1 if testing tools are used and 0 if not nu: number of ratings in which the tool is used nt: number of ratings in which all tools are used Where coefficients α1, α2, α3, and α4 are calculated based on the weightings assigned by the user in her profile according to what criteria she places more emphasis on. For users who do not have a profile (either members of the portal who did not complete their profile or for visitors who do not have a user account), the coefficients are given default values. These default values can be initially selected based on reported research findings and analysis of similar social networks and recommender systems. Further calibration can be done as more data becomes available on the experimental system. Analysis of the most commonly used application in e-commerce (Epinions.com) indicates an average of 14 reviews per user and an average of 10 members in each user s web of trust [33]. Some rating systems enable users to rate the ratings of other users, and possibly leave comments on specific reviews left by others. Users who disagree with a certain review may comment on it describing their reason for disapproval. While this feature may prove to be useful in stimulating discussion, it tends to produce large volumes of data that are of highly qualitative nature. Consequently, these data offer little value to the automated analysis required to generate recommendations. Our approach assumes that users are entitled to their views, and the web of trust of each user becomes the critical factor that establishes the level of confidence users place on the ratings and reviews of each other. The web of trust of a particular rater is composed of other users who said they trust this rater, and thus are considered members of the rater s web of trust. Evaluation of the web of trust can be achieved using two metrics: the number of members within the

9 web of trust, and the average trust these members placed in the user. In our system, we use the number of members within the rater s web of trust as an input to weight these user ratings. A scale of ratings based on the specific context (category) of the project (following Allen and Appelcline [28]) may be proposed, where a rating of 10 implies One of the best, and a rating of 1 implies Very Poor with other semantic interpretations associated with ratings between these two extremes. In order to infer meaningful recommendations from a users ratings, a certain level of trust should be established in these ratings. The social network between users is used to facilitate the deduction of trust values. Each user can decide, based on the usefulness of another user s ratings, whether she trusts this user (the ratings of this user are consistently valuable) or not. She can also determine the level of trust she places in this user. The user can specify different trust levels t(ui, uj) for the same user in different categories (a user whose ratings might be very trustworthy in database systems may not be so for operating systems). We propose a scale of 1-10 for trust levels between users. When making a recommendation, the system encounters three distinct scenarios: 1. a casual visitor to the site who has not registered and therefore does not have any trust levels established with any members of the website (Outsider in Figure 2). 2. a new member who has joined the site recently but has not yet established any trust relationships (user D in Figure 2). 3. a member of the site who has established trust relationships with other members (user C in Figure 2). As the system has limited information about casual visitors and new members, it cannot make a recommendation based on their particular preferences and trust relationships. Therefore, the system provides users with two ratings of any particular software: an average rating (based on weighted ratings of other users), and a recommended rating (based on the established trust relationships between users). Users who want to increase their confidence in the provided recommendations must establish trust relationships with other users. Average rating is calculated by aggregating the weighted ratings of each user who rated the project under consideration: avr( p, c) n i i= 1 = n ( we r ( p, c)) where: avr(p,c) the average rating of a specific F/OSS project within the context (c) we i weighting of user i s rating r i (p,c) user i s rating of the project p within the context c n number of users who rated project p In the third scenario where the user is a member of the site and has established trust relationships with other members, the system can refine its recommendations based on this additional information. In order to make appropriate recommendations, the system constructs the social network connecting the user to the users who left i= 1 i we i

10 ratings for the particular project. The TidalTrust [29] algorithm is used to infer the trust level between members of the social network. First, all trust relationships between users are established. Secondly, the algorithm looks for raters who are directly connected to the user within this network. If the user does not have any direct connections with other users who rated the project, the system searches for raters who are two hops away from the user. This process is repeated until a path connecting the user and at least one rater is found. The system then calculates the recommended rating of the project by aggregating the ratings made by users at this particular depth in the social network and weighs the ratings according to the inferred trust level t ij in each rater [30]. Based on the calculated trust levels, the weighting of each rating we i is adjusted to reflect the confidence a particular user places on each rater. A compound rating wt i is calculated using the following formula: wt we j = γ 1 tij + γ 2 j This approach for constructing social networks to infer personalised recommendations has been successfully implemented in different applications, such as FilmTrust [31] for movies and NetTrust for websites [32]. The recommended rating ru i for project p within context c is calculated as follows: ru ( p, c) i n j j= 1 = n ( wt r ( p, c)) where: ru i (p,c) the recommended rating of a specific F/OSS project for user i t ij trust between user i and user j wt j weighting of user j s rating based on expertise and trust r j (p,c) user j s rating of the project p within the context c n number of users who rated project p Coefficients γ 1, γ 2 are calculated based on the user s preferences for the weightings of the rater expertise and trust level as defined in the user s profile. If the user does not have a profile in the system, default values are used for these coefficients. The system can then generate a list of recommended F/OSS projects based on the user s search criteria and ordered according to their respective ratings. The user is presented with the opportunity to provide feedback on the recommendations, and to explore the individual ratings and reviews left by a subset of users and rate these suggestions. The following example illustrates the process: a user wants to select an appropriate F/OSS content management system of a reasonable quality. For the sake of clarity we will limit our discussion to the ratings of a single project (say Project X), however the same logic may be applied when ranking several projects. Referring to figure 2, assume that user A is an expert user, who has submitted 24 ratings, his web of trust contains 8 people and the average of his votes from other users is 9. The second user B submitted 5 ratings, has 3 people in his web of trust and his average vote is 6. We j= 1 wt j j

11 will suppose the following ratings for Project X (for illustration purposes only): average rating for the project is 8 out of 10, the average size of the web of trust for all users who rated the project is 5 and the average vote for all raters is 5. The project is built using PHP and unit testing can be performed using the PHPUnit tool. PHPUnit has been used twice while all other testing tools in the site are used 12 times. User A rates Project X at 7 and user B gives it a rating of 4. User A used PHPUnit to support his rating while user B did not. The system can calculate the average rating for Project X in this scenario as follows: we A = = 4.15 we B = = avr ( p, c) = = Consider user C who is a member of the social network and has established trust relationships with both A and B. She expresses her trust in A as 3 and in B as 8. The system can recommend the rating of Project X for user C as follows: wt A = = wt B 7.15 = = * 4 ru C ( p, c) = = Although user A is an expert user, and has used testing evidence to support his rating, the system s recommendation to user C is skewed towards user B s rating as C trusts B more than A (she find B s ratings to be consistently more valuable than A s). 5 Experimental Platform: TrustedSource To evaluate the utility and the effectiveness of the proposed approach in generating useful recommendations on the quality of F/OSS applications, we developed an online experimental platform: TrusedSource (available at In the experimentation phase, the website will list Free and Open Source Content Management Systems (CMS). The website enables users to rate and review the available CMS projects and to establish a network of trusted friends, based on the level of confidence they have in others ratings. The system then uses our proposed

12 rating aggregation technique to make recommendations, combining user trust and empirical testing evidence. Figure 3 shows a screenshot of the TrustedSource project. An invitation to test the platform was sent to several Free and Open Source Software user communities. The data collected from user interactions will be analysed to test the usefulness and relevance of the recommendations made by the system. An important goal will be to compare the social network s trust-aware recommendations to those produced by traditional systems. We aim also to explore user acceptance of the system through the analysis of the evolution and the structure of the users social network. Fig. 3. TrustedSource Website 6 Conclusion and Future Work An approach for quality assessment of F/OSS projects based on social networking is presented. The proposed recommendation system utilises the social network of user formed around F/OSS projects, in addition to the open repositories of project data hosted on sites like Sourceforge ( to make recommendations according to user preferences and the trust relationships within the social network. The idea of using trust in social networks to produce meaningful recommendations has been successfully implemented to filter a large number of options. We believe that a recommendation system for the F/OSS community based on these ideas would enable users to evaluate the available choices in a much more effective and informed manner. Such a system can be a useful addition to online F/OSS repositories and can complement previous F/OSS quality evaluation frameworks. An experimental platform has been developed to evaluate the effectiveness of the proposed approach. TrustedSource ( website will be used to

13 collect empirical statistics about user ratings in order to assess the usefulness of social networks for generating trust-based. Further work will include the collection of usage data from the TrustedSource website and evaluate the system to examine the relevance and accuracy of the social network based recommendations. The system will also be extended to include ratings along different dimensions of F/OSS software features (such as security, usability and stability) and to consider different user profiles to refine its recommendations. The approach presented is specifically aimed at the F/OSS community, as the community structure described in section 4 is specialised for this domain. Our subsequent model also utilises the connectivity between users within such a domain. However, we believe that the use of social networks to support automated recommendations is an important emerging research area as explored in [18, 33], especially with availability of a number of social networking sites on the Internet (such as facebook, which currently boasts 70 million active users (April 24, 2008)). References 1. Raymond, E. S. (2000), The Cathedral and the Bazaar (accessed April 24, 2008) 2. Halloran, T. J. & Scherlis, W. L. (2002), High quality and open source software practices, In: Proceedings of 2nd Workshop on Open Source Software Engineering. ICSE, May 25, 2002, Orlando, USA. 3. Schmidt, D. C. & Porter, A. (2001), Leveraging open-source communities to improve the quality & performance of open-source software, In: Proceedings of the 1st Workshop on Open Source Software Engineering, May 15, 2001 Toronto, Canada. 4. Glass, R. (2001), Is Open Source Software More Reliable? An Elusive Answer, The Software Practitioner 11 (6). 5. Harrison, W. (2001), Editorial: Open Source and Empirical Software Engineering. Empirical Software Engineering, 6, Golden, B. (2005), "Succeeding with Open Source". Addison-Wesley, Boston. 7. Michlmayr, M., Hunt, F., Probert & D. R. (2005), Quality Practices and Problems in Free Software Projects, In: Proceedings of the First International Conference on Open Source Systems, July 11-15, 2005, Genova, Italy. 8. International Standards Organisation (2002), (accessed April 24, 2008). 9. Juran, J. (1979), Quality Control Handbook, 3rd edition. McGraw-Hill 10. Crosby P.B. (1979), Quality is Free: The Art of Making Quality Certain. McGraw-Hill. 11. Michlmayr, M. (2005), Software process maturity and the success of free software projects, In: Proceedings of the 7th Conference on Software Engineering, KKIO October 18-21, 2005, Krakow, Poland. 12. Open Source Maturity Model (2003),

14 (accessed April 24, 2008). 13. Duijnhouwer, F. W. & Widdows, C. (2003), Capgemini Expert Letter Open Source Maturity Model, Capgemini, (accessed April 24, 2008). 14. Method for Qualification and Selection of Open Source software (QSOS) (2005), Atos Origin (accessed April 24, 2008). 15. Business Readiness Rating (2005), (accessed January 5, 2007). 16. Groot, A. Kugler, S., Adams, P. & Gousios, G. (2006), Call for Quality: Open Source Software Quality Observation, In: Proceedings of The Second International Conference on Open Source Systems, June 8-10, 2006, Como, Italy. 17. Tawileh, A., Rana, O. F., Ivins, W. & McIntosh, S. (2006), Managing Quality in the Free and Open Source Software Community, In: Proceedings of the 12th Americas Conference on Information Systems, August 4-6, 2006, Acapulco, Mexico. 18. Alshabib, H. Rana, O. F., Ali, A.S. (2006), Deriving ratings through social network structures, In: Proceedings of the First International Conference on Availability, Reliability and Security (ARES 06), April 20 22, 2006, Vienna, Austria. 19. Resnick, P., Iacovou, N., Suchak, M. Bergstrom, P. & Riedl, J. (1994), GroupLens: An open architecture for collaborative filtering of netnews, In: Proceedings of The ACM Conference on Computer-Supported Cooperative Work, November 6-10, 2004, Chicago, USA. 20. Wei, Y. Z., Moreau, L. & Jennings, N. R. (2003), Market- Based Recommendations: Design, Simulation and Evaluation, In: Proceedings of the 5th International Bi-Conference Workshop on Agent- Oriented Information Systems AOIS, July 14, 2003, Melbourne, Australia. 21. Lang, K. (1995), Newsweeder: Learning to filter netnews, In: Proceedings of the Twelfth International Conference on Machine Learning, July 9-12, 1995, Lake Tahoe, California, USA. 22. Pazzani M., and Billsus, D. (1997), Learning and Revising User Profiles: The identification of interesting websites, Machine Learning 27, Goldberg, D., Nichols, D., Oki, B., & Terry, D. (1992), Using collaborative filtering to weave an information tapestry, Communications of the Association of Computing Machinery 35(12): Hill, W., Stead, L., Rosenstein, M., & Furnas, G., (1995), Recommending and Evaluating Choices in a Virtual Community of Use, In: Proceedings of ACM CHI 95, May 7-11, 1995, Denver, USA. 25. Nkweteyim, D. (2005), A Collaborative Filtering Approach To Predict Web Pages Of Interest From Navigation Patterns Of Past Users Within An Academic Website. PhD Thesis, University of Pittsburgh. 26. Massa, P. & Bhattacharjee, B. (2004), Using trust in Recommender Systems: an Experimental Analysis, In: Prococeedings of 2nd Int. Conference on Trust Management, March 29- April 1, 2004, Oxford, UK. 27. Crowston, K. & Howison, J. (2006), "Assessing the Health of Open Source

15 Communities," Computer, vol. 39, no. 5, Allen, C. & Appelcline, S. (2005), Collective Choice: Experimenting with Ratings, (accessed January 5, 2008). 29. Golbeck, J. (2005) "Computing and Applying Trust in Web-Based Social Networks," Ph.D. Dissertation, University of Maryland, College Park. 30. Golbeck, J. (2005), Personalizing Applications through Integration of Inferred Trust Values in Semantic Web-Based Social Networks, In: Proceedings of Semantic Network Analysis Workshop, November 7, 2005, Galway, Ireland. 31. FilmTrust (2005), (accessed April 24, 2008). 32. Camp, L. J. & Genkina, A. (2006), (accessed April 24, 2008). 33. Massa, P. & Bhattacharjee, B. (2004), Using Trust in Recommender Systems: an Experimental Analysis, In Proceedings of the 2nd International Conference on Trust Management. 34. Wagstrom, P., Herbsleb, J., & Carley, K. (2005), A Social network approach to free/open source software simulation. In Proceedings, First International Conference on Open Source Systems, Genoa, Italy, July Gao, Y. & Madey, G. (2007) "Network Analysis of the SourceForge.net Community", The Third International Conference on Open Source Systems (OSS 2007), IFIP WG 2.13, Limerick, Ireland, June.