March The Magazine for Professional Testers CROWD TESTING

Transcription

1 25 March 2014 The Magazine for Professional Testers CROWD TESTING

2 By Dieter Speidel, Mayank Mittal & Mithun Sridharan Enterprise Crowd Testing The information technology world is shifting from one driven by corporations to one driven by consumers, who demand more flexibility as they carry their devices through a myriad of environments and ecosystems. This lays out a clear challenge to software designers and their quality assurance teams. High quality design and functionality are absolute pre-requisites for building customer loyalty. Crowd testing lets real users test the software under real-world conditions. This allows companies to gather real insight, feedback, and defects quickly and cheaply, leading to significant quality and design improvements, and improving the chances of software adoption by customers and end users. Using the crowd to test software raises the bar over other common options like outsourcing or automation when it comes to levels of quality, flexibility, speed, and cost. Today, well-designed and high quality software components are pre-requisites for building customer loyalty. Extensive testing is obligatory to ensure that software meets customers exacting design, utility, and performance expectations. Software has to perform on the various permutations of devices and usage scenarios that cannot be predicted in advance. In-house testing with limited test resources cannot meet such extensive test coverage requirements. Crowd-sourced software testing subjects the software to testing under real-world conditions by real users, allowing companies to gather real insights, feedback, and Figure 1: Crowdsourcing Trends Since 2007 (from Google Trends) Figure 2: Regional Interest in Crowdsourcing (from Google Trends) defects quickly and cheaply, leading to significant quality improvements. 1. Introduction Over the past few years, the crowdsourcing paradigm has evolved from its humble and isolated custom initiatives to success stories, such as Wikipedia and Amazon Mechanical Turk. In the field of information technology, crowd-sourced software testing has witnessed increased interest and adoption, especially among startups and smaller companies. The adoption has, however, been quite slow among larger businesses, mainly due the paucity of published business literature. To facilitate crowd testing by large businesses, we shall discuss some important issues, such as make-or-buy decisions, building an internal testing community or partnering with external agencies, application suitability, governance, and tester quality, and conclude by providing some guidelines for successful crowd testing engagements. 2. Today s IT landscape In recent years, information technology (IT) has assumed such strategic importance that many companies have already made substantial investments in ERP, CRM, and other business tools, demonstrating the critical nature of enterprise IT. These companies also have consumer-facing components, such as websites and mobile applications, which should not only be well designed, but also deliver uninterrupted functionality. It is almost impossible to predict the users of such systems and their tastes a priori. Crowd-sourced software testing helps achieve a high probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree. With the growing trend in cloud computing, BYOD, etc., IT services are delivered over multiple channels and consumed by diverse stakeholders whose IT systems may come in different configurations. Truly robust software testing is time consuming, and it is nearly impossible to ensure that every possible permutation and combination of features, localizations, and platforms works as intended. For example, if a web application does not render in a particular browser, or a particular software tool fails to deliver a critical functionality, business disruptions may ensue. Through crowd testing, companies could effectively diminish the likelihood that the internal software testing team might have missed critical elements during the internal testing phases. According to Paul Herzlich, a software-testing analyst at Ovum, an independent IT industry research institution, If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it. Thus, crowd testing offers a simple, elegant, and cost-effective solution to what would otherwise be an intractable quandary. 34 Testing Experience 25/2014

3 3. Exploratory and enterprise crowd testing Presently, much of the focus is on exploratory crowd testing services, where testers analyze software for issues, bugs, or defects based on generic guidelines and test cases. With free hand exploratory testing, the outcomes cannot be quantitatively predicted, as it is challenging to qualify the range, area, and number of possible defects in advance. Testers usually identify issues in the most obvious places, but some highly skilled testers may uncover defects hidden under layers of the software. Exploratory testing is best suited to testing less mission critical applications with a large number of users to ensure reproducibility of a set of issues under different configurations or to simulate a typical load on an application. Enterprise crowd testing incorporates more rigor and structure in the software testing process, while simultaneously retaining the spirit and agility of exploratory testing. Thus, enterprise crowd testing incorporates the best practices and concepts of both informal and formal software testing methodologies. The degree of professionalism involved in enterprise crowd sourced software testing is higher. In this approach, testers are scrutinized and deployed based on comprehensive selection criteria. The software testing process involves a clearly identified scope, and is guided by extensive guidelines tailored to fit the needs of the project. The team has clearly identified roles and responsibilities, a single community point of contact for the client organization, formalized reports and reporting structures, extensive documentation and project tracking, etc. Furthermore, enterprise crowd testing entails formal processes built over existing software testing methodologies, such as Agile, SCRUM, etc. As a result of such alignment, the client organization experiences a smooth synchronization between its internal development and external software-testing cycles. This allows the client to reduce its project management overhead and maximize the benefits from both teams. Díaz & Hilterscheid GmbH In the Mobile App Testlab your app is tested on-site on a multitude of devices. Our experienced staff members help you with optimizing your app and perfecting the user experience. Testing Experience 25/

4 4. What applications are good candidates for crowdsourced testing? Most crowd-sourced testing companies provide the platform and the project management framework for the testing cycles, including governance and legal structures. The crowd testing company also allocates a qualified project manager who oversees the testing process. The crowd-sourced testing methodology is the most suitable for applications that are user-centric. It is frequently used in mobile applications and game development projects by releasing cheap or free versions of products in the beta stage. Large corporations could benefit from crowd-sourced testing by simulating a large user base to understand usage patterns and improve on feedback, while ensuring their applications run smoothly on a number of different devices, operating systems, browsers, and language versions. In other words, applications with a high defect exposure factor, i.e. criticality as a measure of immediate customer exposure post release, are good candidates for crowd testing. For example, Microsoft released the beta version of its Office 2010 productivity suite, which was downloaded and tested by 9 million people who provided 2 million valuable comments and insights, which resulted in substantial product improvements. 5. How does it work? On the consumer side of crowd-sourced software testing services, the clients specify the type and battery of tests they wish to have performed, the skill set and demographic information of testers, the types and configuration of devices the software must be tested on, etc. The client company could also consult the crowd testing company to ensure testing adequacy. On the supplier side, the testers complete a profile, indicating their background and qualifications, the devices and platforms they have access to, and other relevant details. Testers may sometimes be required to participate in assessments or trial projects as a pre-condition for community membership. Before a project starts, testers are provided with detailed test plans, sample scenarios, tools, scripts, and instructions. During execution, the testers document their observations and are rated based on the volume and quality of their reports, which links directly to their compensation and incentive structures. The community combines aspects of collaboration and competition, as members work to find solutions to the stated problem. Forums facilitate knowledge management, networking, and discussion of bugs or relevant issues; rating systems allow for recognition of a job well done, which helps participants gain credibility and improve their careers. 1 HOW IT WORKS EXAMPLES OF CROWDSOURCED TESTING PREPARATION PHASE Assess client s business needs and goals Design the Test Plan incl. required skills and capacities Fix price and deliverables Crowd testing should be considered additional complementary testing prior to production release rather than as a self-contained activity and replacement to an organization s internal testing team. It is effective in catching any residual UI and configuration defects prior to production release. 6. Onsite, outsourced, and crowd testing 2 INITIATION PHASE Define test cases and configurations Recruit crowdtesters from a global pool of professionals Establish secure test environment Onsite testing refers to the established practice of deploying an internal team for software testing purposes. This approach has limits to its scalability and is ineffective in reproducing realistic usage situations. 3 EXECUTION PHASE Initiate testing across required devices and software Manage and supervise test team Monitor and ensure performance 5 COMPLETION PHASE EVALUATION PHASE Present in-depth results to client Refine crowdtesters classification ratings Close test cycle Validate results and conduct quality assurance Prepare client deliverables Process testers compensation 4 In outsourced software testing, the software testing is rendered by an external service provider, on a per-project or agreement basis. Outsourced software testing has obvious cost benefits highly skilled resources at a fraction of costs through labor arbitrage. In crowd testing, the testers voluntarily test software, motivated by their inherent drive to detect software defects, establish their standing in a community, gain peer recognition, etc., despite the risk of their not getting paid if no issues are detected. This self-organizing feature, shared community goals, and intrinsic motivation are strong forces for crowd testing s success. 7. What to choose? The onsite team, due to its proximity to customers, delivers value by focusing on technological and business challenges requiring immediate attention. 36 Testing Experience 25/2014

5 In an outsourced testing project, the testing team(s) could be either, captive or non-captive to the outsourcing company. In a captive model, the testing team is part of the company and performs software testing as an ongoing activity from which the captive team gains experience and efficiency levels through an enhanced understanding of the most common internal applications. Non-captive teams are external to the company and work with several clients on several projects. From these engagements, non-captive teams gather experience and expertise across several areas, domains, functions, industries, and technologies. The client company could leverage such diversity in projects that require thinking outside the box and special testing efforts. A crowd-testing team s laser focus, single-minded efforts, inherent motivation, and scalability deliver tremendous value by detecting issues before go-live. The crowd s knowledge is diffused among its tester base, but the collective testing knowledge base is larger than that within any company. Given the complexities of modern systems, an optimum testing organization should comprise of onsite, outsourced, and crowdsourced testing teams in the right proportion. Converging on the right proportion is an element subject to experience and experimentation. 8. Build a community or establish partnerships? While some companies, such as Netflix and Amazon, have built their own crowd capabilities internally, this is not an activity that most companies could achieve on their own or that makes tactical business sense. Crowdtesting is a complex activity requiring deep technical, functional, and business knowledge. So, unless the work to be done is so strategic or highly confidential to the business, a company is well advised to seek a crowdtesting company that delivers all the necessary ingredients to deploy the crowd almost immediately. These ingredients include experienced testers, devices, configurable architectures, participation mechanisms, ground rules, engagement methodologies, legal constructs, support mechanisms, etc. ones with customized NDAs, which testers must sign before viewing the customer s project. Furthermore, standardized communication channels, access restrictions, etc. help secure data and gain confidence in testing efforts, which results in a seamless transition across the various stages of the crowd-testing project lifecycle. 10.Ensuring the quality of testers To ensure the quality of testers, formal and informal tests are periodically administered to individual testers based on their experience, available material, and concepts mastered. The testers and project managers are continually monitored to ensure both quality and integrity. Through constant filtering, applicants without formal training and significant professional experience are eliminated. Increasingly, crowd testing companies are also sponsoring professional certifications for experienced testers, so their standing and responsibilities grow in proportion to their contributions. To avoid potential conflict of interest among testers, performance-based compensation mechanisms are deployed to ensure that testers compensation is commensurate with their level of engagement and contribution to software quality improvement. Testers who detect a larger number of less impactful defects are compensated less than those who detect a few high-impact defects. These mechanisms also help prevent any flame wars, jockeying, etc. within the community and increase testers focus, ownership, loyalty, and affiliation to the community, thereby creating a congenial and respectful working environment. 11. Choosing a crowd-testing company Among the various stages of the crowd-testing process, choosing the right testing company is by and far and away the most important. It is mandatory that the client company performs adequate due diligence, extensive project evaluation, capabilities and scale of the testing company, etc. before choosing a crowd testing platform. 9. Security and Governance More often than not, confidential customer information is exposed to the crowd during testing cycles. Crowd-testing firms have realized the need for security and confidentiality; hence, they have implemented several layers of screening, filters, governance mechanisms, obfuscation tools, data management frameworks, etc., to assure their clients that their data and information is processed with the utmost confidence. For projects requiring high security, a pre-screened list of white hat test engineers who have long-standing credibility and professional reputation are proposed and usually accepted. Test data management ensures security by obfuscating sensitive information or creating test only data. This helps minimize the impact of potential security breaches during large-scale testing. Testers are also required to sign a general non-disclosure agreement (NDA) when they join the community, which forbids them from discussing their customers, products, etc., both offline and online on social networks, personal blogs, and elsewhere outside the private testing platform. Customers also have the freedom to choose their testers from the pool and furnish the chosen In general, selecting a proven and experienced crowd testing platform makes the operational activities much easier, as there is usually less maintenance and governance required during the operational stages because the testing company has already addressed many of the relevant issues from past experiences with other clients. Therefore it is important to evaluate crowd testing platforms on at least the following criteria to ensure that the platform contains at least the minimum functionality and security required: 1. It is important to ascertain the crowd testing company s level of experience, delivery capability, skill diversity, and global outreach to accommodate the evolving requirements of enterprise customers. 2. A credible pedigree demonstrated through organic growth and experience with both corporate and mid-scale companies. It is relatively easy for a small crowd-testing platform to offer mobile application and website testing services, but very few companies have the maturity to test enterprise applications and provide turnkey testing services. Testing Experience 25/

6 3. Financial stability built over a period of time is particularly important if client companies aspire to undertake long-term projects without worrying about the service provider exiting the market. 4. Many crowd-testing platforms operate merely as matchmaking services between clients and testers rather than providing a managed service. These platforms do not assure the number of software testers, amount of testing work done by individual testers, quality of testers or test results, nor any reliable outcomes. Such platforms are best avoided. 5. An assiduous review of the Terms & Conditions is mandatory to understand their scope of services and SLAs provided. 6. A Sourcing Hierarchy of Constraints & Criteria document should be used for evaluating platform operators for every major crowd testing project. This document should address various elements, such as: Price structures Quality, safety, security and confidentiality Professional services portfolio Getting started is easy, and those organizations that dare to take the plunge are assured of the several benefits in both the short and the long term. Bibliography 1. Brabham, Daren C. Crowdsourcing as a Model for Problem Solving. SagePub, Gerrard, Paul. Crowdsourcing and Testing: Can it work? UK Tester Forums, Hoßfeld, Tobias, et al. CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation. 4. Kaner, Cem. A Tutorial in Exploratory Testing. Department of Software Engineering, Florida Institute of Technology, PASS Technologies AG. Crowdtesting and the Future of Software QA. May 2, PASS Technologies AG. Testing Applications for the Real World Capabilities and processes Delivery methodology Service infrastructure Compensation structures Convenience and scalability Functionality and quality of the platform and tools Risk management structures Summary Having discussed the various elements of crowd testing, it could easily be inferred that large enterprises could expect substantial cost savings and scalability by partnering with established and experienced crowdtesting companies. Crowd-testing tools, processes, frameworks, and methodologies have matured to such a high degree that crowd testing could easily be incorporated into the normal development life cycles at organizations. Crowd testing decreases the likelihood of human error by diversifying software testing activities among a much larger team. By subjecting the application to realistic conditions, actual performance data and execution paths can be traced which can be plowed back into re-engineering or improvements, leading to further quality improvements. Besides these direct benefits, large companies can also experience faster time to market, free up resources to focus on core operations, gain access to a large pool of talents, increase flexibility, etc. 7. Ziegler, Wibke. Crowdtesting Bugtracking via Crow d. Volksbank Bühl, > about the author Dieter Speidel (53) is founder and CEO of PASS Group, a leading Swiss provider of software and system test services. PASS Group is owner and home of passbrains. com, a global platform for on-demand crowd testing services, with offices in Zurich, Berlin, Boston, Belgrade, Bangalore, and Mumbai. Being active as an entrepreneur in the IT outsourcing, software development, and testing business for more than 30 years, Dieter Speidel has in-depth expertise in global delivery models, including near/offshoring and crowdsourcing. Since 2011, he has specialized further in the application of crowdsourcing models to IT related services and developed the passbrains platform and community for crowd testing and knowledge services. Mayank Mittal is a strategic business leader in the application testing industry. During his career spanning over a decade, he has established large scale managed test centers and executed QA organizational transformations while working with leading organizations such as Oracle, CSC, and Cognizant, delivering testing solutions to many Fortune 100 customers across the globe. Mayank works as Director Corporate Strategy and Business Development at PASS Group AG. Mithun Sridharan (33) is a Business Development Manager with Passbrains, based in Eschborn, Germany. He brings over ten years of international experience in business development, marketing, global delivery and consulting. He holds a Master of Business Administration (MBA) and Master of Science (M.Sc). He is a Project Management Professional (PMP) and a Certified Information Systems Auditor (CISA). He also serves as the Communication Chair for the German Outsourcing Association. 38 Testing Experience 25/2014