Crowdsourced Software Testing for Enterprises. August 2013

Transcription

1 August 2013

2 At a glance 2

3 Crowdsourced Software Testing for Enterprises 3

4 Table of contents 1. Introduction 5 2. IT Landscape today 5 3. What is crowdsourced software testing? 7 4. Exploratory and enterprise crowdtesting 8 5. What applications are good candidates for crowdsourced testing? 9 6. How does it work? Onsite, outsourced & crowdtesting What to choose? Build a community or establish partnerships? Security & Governance Ensuring the quality of testers Choosing a crowdtesting company Summary About the author 16 Figures 1. Crowdsourcing trends since Regional interest in crowdsourcing 6 3. Crowdtesting VS traditional in-house methods 7 4. Enterprise crowdtesting 8 5. Crowdtesting ROI apps with high exposure to market 9 6. Application criticality and defect exposure factor Crowdsourced testing process

5 Introduction Over the past few years, the crowdsourcing paradigm has evolved from its humble and isolated custom initiatives to major success stories, such as Wikipedia and Amazon Mechanical Turk. In the field of information technology, crowdsourced software testing has witnessed increased interest and adoption, especially among startups and smaller companies. However, adoption has been quite slow among larger businesses, largely due the paucity of published business literature and use cases. To help facilitate crowdtesting by large businesses, we shall discuss some important issues such as make-or-buy decisions, building an internal testing community or partnering with external agencies, application suitability, governance, tester quality and will conclude by providing some guidelines for successful crowdtesting engagements. IT landscape today In recent years, Information Technology (IT) has assumed such strategic importance that many companies have already made substantial investments in ERP, CRM and other business applications, demonstrating the critical nature of enterprise IT. These companies also have consumer-facing components, such as websites and mobile applications that should not only have good design, but also deliver uninterrupted functionality. It is almost impossible to predict the users of such systems and their tastes a priori. Crowdsourced software testing helps achieve a high probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree. Fig. 1 Crowdsourcing trends since 2007 With the growing trend in cloud computing, BYOD, etc., IT services are delivered over multiple channels and consumed by diverse stakeholders, whose IT systems may come in different configurations. 1 Truly robust software testing is time consuming and ensuring that every possible permutation and combination of features, localizations and platforms works as intended is nearly impossible. For example, if a web application does not render in a particular browser or a particular software tool fails to deliver a critical functionality, it may lead to disruption in business. Through crowdtesting, companies could effectively eliminate the problems associated with in-house software testing team missing critical elements during the internal testing phases. 1 Real-World On-Demand Apps Testing: 5

6 Fig. 2 Regional interest in crowdsourcing According to Paul Herzlich, a software-testing analyst at Ovum, an independent IT industry research institution, If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it. Thus, crowdtesting offers a simple, elegant and cost-effective solution to what would otherwise be an intractable quandary. What is crowdsourced software testing? Crowdtesting is a software testing methodology that leverages a community of external expert software testers with diverse backgrounds and demographics from all across the globe. It differs from the traditional approach in that testing is carried out by a larger number of testers from different places, rather than by a limited number of in-house testing professionals. These communities range from a few hundred to several thousand testers globally. 6

7 Fig. 3 Crowdtesting VS traditional in-house methods Crowdtesting subjects the application to a set of realistic scenarios, loads and user paths, which cannot be replicated by an internal testing team. Knowing an application s performance under real-world conditions in advance makes refactoring easier and cheaper. Such testing also increases reliability and decreases the time required to develop and deploy. Usually, almost all noticeable bugs or issues are detected and fixed before a full-fledged deployment. Exploratory and Enterprise Crowdtesting Presently, much of the focus is on exploratory crowdtesting services, where testers analyze software for issues, bugs or defects based on generic guidelines and test cases. With free hand exploratory testing, the outcomes cannot be quantitatively predicted as it is difficult to qualify the range, area and number of possible defects in advance. Testers usually identify issues in the most obvious places, but highly skilled testers may uncover defects hidden under layers of the software. Exploratory testing is most suited to test less mission critical applications with a large number of users to ensure reproducibility of a set of issues under different configurations or to simulate a typical load on an application (Kaner 2008). Enterprise crowdtesting puts more rigor and structure into the software testing process, while simultaneously retaining the spirit and agility of exploratory testing. Thus, enterprise crowdtesting incorporates the best practices and concepts of both informal and formal software testing methodologies. The degree of professionalism involved in enterprise crowdsourced software testing is higher. 7

8 Fig. 4 Enterprise crowdtesting In this approach, testers are scrutinized and deployed based on comprehensive selection criteria. The software testing process involves clearly identified scope and is guided by extensive guidelines, tailored to fit the needs of the project. The team has clearly identified roles and responsibilities; a single community point of contact for the client organization; formalized reports and reporting structures; and extensive documentation and project tracking, etc. Furthermore, enterprise crowdtesting entails formal processes built over existing software testing methodologies, such as Agile and SCRUM. As a result of such alignment, the client organization experiences a smooth synchronization between their internal development and external software-testing cycles. This allows the client to reduce project management overheads and maximize the benefits from both teams. What applications are good candidates for crowdsourced testing? Crowdsourced testing methodology is most suitable for applications that are user-centric. It is frequently used in mobile applications and game development projects by releasing cheap or free versions of products in beta stage. Large corporations would also benefit from crowdsourced testing by simulating a large user base to understand usage patterns and improve on the feedback, while ensuring their applications run smoothly on a number of different devices, operating systems, browsers and language versions. 8

9 Fig. 5 Crowdtesting ROI apps with high exposure to market For example, Microsoft released the beta version of its Office 2010 productivity suite, which was downloaded and tested by 9 million people who provided 2 million valuable comments and insights, resulting in substantial product improvements. As can be seen from the following matrix, applications with a high Defect Exposure Factor i.e. criticality as a measure of immediate customer exposure post release, are good candidates for crowdtesting. Fig. 6 Application criticality and defect exposure factor 9

10 How does it work? Most crowdsourced testing companies provide the platform and the project management framework, including governance and legal structures, for the testing cycles. The crowdtesting company also allocates a qualified Project Manager, who oversees the testing process. On the consumer side of crowdsourced software testing services, the clients specify the type and battery of tests they wish to have performed; the skillset and demographic information of testers; and the types and configuration of devices the software must be tested on, etc. The client company could also consult the crowdtesting company to ensure testing adequacy. Fig. 7 Crowsourced testing process 10

11 On the supplier side, the testers complete a profile, indicating their background and qualifications, the devices and platforms they have access to and other relevant details. Testers may sometimes be required to participate in assessments or trial projects as a pre-condition for community membership. Before a project starts, testers are provided with detailed test plans, sample scenarios, tools, scripts and instructions. During execution, the testers document their observations and are rated based on the volume and quality of their reports, which ties directly with their compensation and incentive structures. The community combines aspects of collaboration and competition, as members work to find solutions to the stated problem. Forums facilitate knowledge management, networking and discussion of bugs or relevant issues, while rating systems allow for recognition of a job well done, which helps participants to gain credibility and improved career opportunities. Crowdtesting should be considered as additional, complementary testing prior to production release rather than a replacement of an organization s internal testing team. It is most effective in catching residual UI and configuration defects prior to production release. Onsite, outsourced & crowdtesting Onsite testing is the established practice of deploying an internal team for software testing purposes. This approach has limits to its scalability and is ineffective in reproducing realistic usage situations. In outsourced software testing, the process is rendered by an external service provider, on a per-project or agreement basis. Outsourced software testing has the obvious cost benefits of highly skilled resources at a fraction of costs through labor arbitrage. In crowdtesting, the testers voluntarily test software, motivated by their inherent drive to detect software defects, establish their standing in a community and gain peer recognition, etc., despite the risk of not getting paid if no issues are detected. This self-organizing feature, shared community goals and intrinsic motivation are strong forces for crowdtesting s success. What to choose? The onsite team, due to its proximity to customers, delivers value by focusing on technological and business challenges requiring immediate attention. In an outsourced testing project, the testing team(s) could be either captive or non-captive to the outsourcing company. In a captive model, the testing team is part of the company and performs software testing as an ongoing activity from which, the captive team gains experience and efficiency levels through an enhanced understanding of the most common internal applications. 11

12 Non-captive teams are external to the company and work with several clients on multiple projects. From these engagements, non-captive teams gather experience and expertise across several areas, domains, functions industries and technologies. The client company could leverage such diversity in projects that require thinking outside the box and special testing efforts (Brabham 2008). A crowdtesting team s laser focus, single-minded efforts, inherent motivation and scalability deliver tremendous value by detecting issues before go-live. The crowd s knowledge is diffused among testers, but the collective testing knowledge base is larger than that within any company. Given the complexities of modern systems, an optimum testing organization should comprise of onsite, outsourced and crowdsourced testing teams in the right proportion. Converging on the right proportion is an element subject to experience and experimentation. Build a community or establish partnerships? While some companies, such as Netflix and Amazon, have built their own crowd capabilities internally, this is not an activity that most companies could achieve on their own or makes tactical business sense. Crowdtesting is a complex activity requiring deep technical, functional and business knowledge. So, unless the work is so strategic or highly confidential to the business, companies should look for a crowdtesting company that delivers all the necessary ingredients to deploy the crowd almost immediately. These ingredients include experienced testers, devices, configurable architectures, participation mechanisms, ground rules, engagement methodologies, legal constructs, support mechanisms, etc. Security & Governance More often than not, confidential customer information is exposed to the crowd during testing cycles. Crowdtesting firms have realized the need for security and confidentiality; hence they have implemented several layers of screening, filters, governance mechanisms, obfuscation tools, data management frameworks, etc., to assure clients that their data and information is processed with the utmost confidence. For projects requiring high security, a pre-screened list of white hat test engineers that have a longstanding credibility and professional reputation are proposed and usually accepted. Test data management ensures security by obfuscating sensitive information or creating test only data. This helps minimize the impact of potential security breaches during large-scale testing. Testers are also required to sign a general Non-Disclosure Agreement (NDA) when they join the community, which forbids them from discussing their customers, products, etc., both offline and online on social networks, personal blogs and elsewhere outside the private testing platform. Customers also have the freedom to choose their testers from the pool and furnish the chosen ones with customized NDAs, which testers must sign before viewing the customer s project. 12

13 Furthermore, standardized communication channels, access restrictions, etc. help secure data and gain confidence in testing efforts, which results in a seamless transition across the various stages of the crowdtesting project lifecycle. Ensuring the quality of testers To ensure the quality of testers, formal and informal tests are periodically administered to individuals based on their experience, available material and concepts mastered. The testers and project managers are continually monitored to ensure both quality and integrity. Through constant filtering, applicants without formal training and significant professional experience are eliminated. Increasingly, crowdtesting companies are also sponsoring professional certifications for experienced testers, so their standing and responsibilities grow in proportion to their contributions. To avoid potential conflict of interest among testers, performance-based compensation mechanisms are used to ensure that compensation is commensurate with their level of engagement and actual contribution to software quality improvement. This means that testers who detect a larger number of less impactful defects are compensated less than those that detect a few, high-impact defects. These mechanisms also help prevent any flame wars, jockeying, etc. within the community and increase testers focus, ownership, loyalty and affiliation to the community, thereby creating a congenial and respectful working environment. Choosing a crowdtesting company Choosing the right testing company is of course by far the most important stage of the crowdtesting process. It is essential to perform adequate due diligence, extensive project evaluation and understand the capabilities and scale of the testing company, etc. before selecting a crowdtesting platform. In general, a proven and experienced crowdtesting platform will makes the operational activities much easier, as there is usually less maintenance and governance required during the operational stages because the testing company has already addressed many of the relevant issues from past experiences with other clients. Therefore, it is important to evaluate crowdtesting platforms on at least the following criteria to ensure that the platform contains at the minimum functionality and security required: 1. It is important to ascertain the crowdtesting company s level of experience, delivery capability, skill diversity and global outreach to accommodate the evolving requirements of enterprise customers. 2. A credible pedigree demonstrated through organic growth and experience with both corporate and mid-scale companies. It is relatively easy for a small crowdtesting platform to offer mobile application and website testing services, but very few companies have the maturity to test 13

14 enterprise applications and provide turnkey testing services. 3. Financial stability built over a period of time is particularly important if client companies aspire to undertake long-term projects without worrying about the service provider exiting the market. 4. Many crowdtesting platforms operate merely as matchmaking services between clients and testers rather than providing a managed service. These platforms do not assure the number of software testers, amount of testing work done by individual testers, the quality of testers or test results, nor any reliable outcomes. Such platforms are best avoided. 5. An assiduous review of the Terms and Conditions is mandatory to understand their scope of services and SLAs provided. 6. A sourcing hierarchy of constraints and criteria document should be used for evaluating platform operators for every major crowdtesting project. This document should address various elements, such as: Price Structures Quality, Safety, Security and Confidentiality Professional Services Portfolio Capabilities and Processes Delivery Methodology Service Infrastructure Compensation Structures Social Responsibility Convenience and Scalability Functionality and Quality of the Platform and Tools Risk Management Structures Summary Having discussed the various elements of crowdtesting, it could easily be inferred that large enterprises could expect substantial cost savings and scalability by partnering with established and experienced crowdtesting companies. Crowdtesting tools, processes, frameworks and methodologies have matured to such a high degree that crowdtesting could easily be incorporated into the normal development lifecycles at organizations. Crowdtesting decreases the likelihood of human error by diversifying software testing activities to a much larger team. By subjecting the application under realistic conditions, actual performance data and execution paths could be traced, which could be plowed back into re-engineering or improvements, leading to further quality improvements. Besides these direct benefits, large companies could also experience faster time to market, free up resources to focus on core operations, gain access to a large pool of talents and increase flexibility, etc. Getting started is easy and those organizations that dare to take the plunge are assured of the benefits in both the short and the long run. 14

15 About the author Dieter Speidel (53) is founder and CEO of PASS Group, a leading Swiss provider of software and system test services. PASS Group is owner and home of passbrains.com, a global platform for on-demand crowdtesting services, with offices in Zurich, Berlin, Boston, Belgrade, Bangalore and Mumbai. Being active as an entrepreneur in the IT outsourcing, software development and testing business since more than 30 years, Dieter Speidel owns strong expertise with global delivery models, including near-/offshoring and crowdsourcing. Since 2011, he specialised further in the application of crowdsourcing models for IT related services and developed the passbrains platform and community for crowdtesting and knowledge services. PASS Group, owner of passbrains.com, is a leading global provider of On-Demand Managed Application Testing Services. Their delivery centers are located across the U.S., Europe and India. PASS offers a unique service model which integrates the strengths of crowdsourcing and outsourcing solutions to optimize the software testing life cycle to a wide range of different industries. PASS operates its own platform and community for crowdsourced application testing ( Through passbrains, customers can get their web-, mobile- and enterprise applications tested on-demand by selected testing professionals and users with specific demographics, test configurations and mobile devices. The passbrains community counts thousands of registered testers with a wide range of skills and domain expertise from more than 100 countries. Visit us online at and www. passbrains.com. Copyright 2014 PASS Group AG PASS Switzerland International Headquarter Dufourstrasse 91 CH-8008 Zurich Phone: info@passbrains.com PASS India Vaishnavi Badri, #48 5th Main, Jayamahal Extension Bangalore Phone: india@pass.ch PASS USA West Olympic Blvd, Suite 200, Los Angeles, CA Phone: usa@pass.ch 15