Crowd Testing Mike Bartley - CEO

Transcription

1 Crowd Testing Mike Bartley - CEO Test and Verification Solutions Delivering Tailored Solutions for Hardware Verification and Software Testing

2 EMPLOYEES TVS - Global Leaders in Test and Verification Germany UK USA France India China South Korea Singapore Continuous geographical expansion Number of Employees by quarter Q3-13 Q4-13 Q1-14 Q2-14 Q3-14 Q4-14 Q1-15 Q2-15 Q3-15 Q4-15 Q1-16 Q2-16 (Est.) CALENDAR YEAR Q3-16 (Est.) Q4-16 (Est.) Copyright TVS Limited Private & Confidential Page 2

3 The rise of the crowd Users rate a service or products Airbnb, Amazon, travel websites,. Using the Crowd as an Innovation Partner Apple use large numbers of users and developers around the world to create apps and podcasts. Crowdfunding In 1997 American fans of Rock band Marillion used the internet to raise $60,000 so they could play in the US. In 2012, there were over 500 crowdfunding platforms Global crowdfunding market to reach $34.4B in 2015 (Massolution 2014 prediction The crown as testers utest incorporated in August 2007 with $2.3M in seed funding Copyright TVS Limited Private & Confidential Page 3

4 99 Tests Partnership deal with T&VS Over 15K testers Over 50 countries Over 200 clients worldwide Over 85K bugs found Copyright TVS Limited Private & Confidential Page 4

5 Crowd Testing Basics No matter who you are, most of the smartest people work for someone else Joy s Law by Sun Microsystems co-founder Bill Joy Given enough eyeballs, all bugs are shallow Linus Law by Eric S Raymond in The Cathedral and the Bazaar Copyright TVS Limited Private & Confidential Page 5

6 Crowd Testing Basics Testing by anonymous crowd of testers Testers are rewarded by identifying bugs quickly, first to find the bug gets paid. Testing can be limited by device, or open to any platform Testers are keen to work out of hours 24/7 Limit testing by geographical region, or global Pay for the bugs only Copyright TVS Limited Private & Confidential Page 6

7 Crowd Testing Advantages - Speed Test quickly and conveniently, 24/7 if required Drive down testing times Ramp up and down test teams quickly Have test teams the right size for each stage of development Copyright TVS Limited Private & Confidential Page 7

8 Crowd Testing Advantages - Costs Reduce your test costs Improve Cash Flow Ramp up and down test teams quickly Have test teams the right size for each stage of development No need to invest in different platforms and devices Copyright TVS Limited Private & Confidential Page 8

9 Crowd Testing Advantages - Quality Improves Test platform coverage Testing by real users in the real world Testing in multiple languages and cultures Pay Testers for Testing, Developers for Developing Standard bug reporting Remove internal bias Copyright TVS Limited Private & Confidential Page 9

10 Crowd Testing Disadvantages Mainly for customer facing applications Limited control of staff experience Not for Unit or Integration testing May need manuals to be ready earlier May need support teams in place Control of Piracy Management of the Services Copyright TVS Limited Private & Confidential Page 10

11 Crowd Testing Hidden Costs Management of the Crowd Test Process Control of builds Regression testing Earlier deployment Support teams Better documentation may be required Copyright TVS Limited Private & Confidential Page 11

12 Some questions to consider Do you need Unit Testing? Do you need Integration Testing? Do you need a Strategic Test Plan? How will you manage retesting? How bug-free does it have to be? What will you do with Test Results? Copyright TVS Limited Private & Confidential Page 12

13 Test Results Testing Approach To test the web application, exploratory testing approach was used. Testers from various locations accessed the web application and tested the whole application using different exploratory approaches. To ensure coverage of the major features We shared the critical workflows of the website Copyright TVS Limited Private & Confidential Page 13

14 Test Results Copyright TVS Limited Private & Confidential Page 14

15 Test Results Copyright TVS Limited Private & Confidential Page 15

16 Test Results Copyright TVS Limited Private & Confidential Page 16

17 Enterprise v Small companies Enterprise Small/Startups Flex testing teams without having to employee additional staff Reach out to a broader cross section of users Reduce the testing cycle Test Apps and Web Products on every conceivable device in days No need to divert development teams to carry out User Testing Add independence to the testing cycle Control testing costs Copyright TVS Limited Private & Confidential Page 17

18 Major Companies are using it Crowdtesting has already been used by likes of ebay, Amazon, GE, Microsoft, Google and Facebook to develop user-centric products. Google regularly deploys crowd testing for 14 of their major product lines. Copyright TVS Limited Private & Confidential Page 18

19 Typical questions to answer Combination of devices i.e. OS & Browser versions, mobile platforms, Tablets to be tested. Testing timeframe, number of days per cycle or total duration for testing being allocated. Number of crowd sourced testers would you like testing your site, egg 25 50, Estimate the number of testing hours you would like to achieve in the days allocated Estimated number of defects you believe will be found as a result of testing List bugs that you are already aware of 19 Copyright TVS Limited Private & Confidential Page 19

20 Typical questions to answer Do you require a Test Plan to be written? Will you provide Test Cases to be executed? Would you like the crowd to provide you with: Fully Detailed Steps - screen shots of each tests performed by Crowd (Screenshots / Video capture of tests) Screen Capture - attached to defects only Select specialist testers with a specific product / domain skills and expertise? Confidentiality agreement / NDA with the Crowd testers required? Do you require Usability testing of your site? Formal Crowd actively is testing for usability and raising defects Basic a Questionnaire is filled in at the end of the cycle to gauge testers feeling about usability Defects will not be raised. 20 Copyright TVS Limited Private & Confidential Page 20

21 Case Study of a Large MNC: Intuit Intuit provides accountancy packages for small companies, mainly its flagship application Quickbooks Intuit has lots of Small Teams building software products. Each team is given limited resources. i.e. their critical resource for them is time. So, each team is tasked with creating innovative software product that can potentially reach millions of customers. Crowd Testing frees up time for these teams. Thus running a Crowd Testing campaign gives them ability to compress their release cycles & also improve the quality of their application. 21 Copyright TVS Limited Private & Confidential Page 21

22 Case Study of a medium MNC: Teligenz Teligenz is a successful technology company that provides a range of creative services from web designing to App development. The successful delivery of their projects depends greatly on the quality of the insights gained through App testing. We leveraged on our talent pool of professional crowd testers from across the globe to implement effective functional testing for their Apps. The test cycles created were executed by professional crowd testers who were assigned to various projects after matching their skill set with the project requirements. The efficient strategy enabled the entire test cycle for an App to be covered in just 3-5 days. The quick turnaround time helped the development team at Teligenz to make real-time decisions and focus product development in areas that needed their attention. They were able to improve their App quality exponentially. 22 Copyright TVS Limited Private & Confidential Page 22

23 Case Study of a small MNC: Reverie Reverie s cloud-based language-as-a-service platform enables businesses to connect more easily with their target audience by delivering dynamic and static content for apps in real-time in the languages that they need. Testing the usability for language feature is not an easy task, especially when dealing with highly user-centric E-commerce Apps. Reverie wanted to test the usability of the App for specific target demographics comprising of Tier 2 and Tier 3 Indian cities whose native language was Hindi. We provided Reverie with testers across different demographics factoring in parameters like age, city etc. from all over India to gain a real life, end user perspective. The focus on the human approach, helped developers gain valuable insights on how their target demographics would react to the App. 23 Copyright TVS Limited Private & Confidential Page 23

24 Case studies in progress MovieR a UK startup providing a social media multi-user video experience. ios application being tested by 50 users over a 48 hour window Pogo a UK startup providing a photo comparison monitor for social networking. ios application being tested by 50 users over a 48 hour window 24 Copyright TVS Limited Private & Confidential Page 24

25 Questions? Test and Verification Solutions Delivering Tailored Solutions for Hardware Verification and Software Testing