Organising Massive Test bol.com. Jeroen Ruijgers and Viktor Clerc

Transcription

1 Organising Massive Test bol.com Jeroen Ruijgers and Viktor Clerc 1

2 Introducing 5 million customers 9 million products #1 online retailer in the Netherlands and Belgium 1999 bol.com becomes separate (and Started by Bertelsman Online only) entity employees Started with Scrum > 35 scrumteams > 900 employees 2

3 Introducing A leader in Software Delivery Automation Headquartered in Boston, Massachusetts Sales in US, EMEA and APAC 150 premier customers Growth over 100% y/y REVENUE 110% 140% 100% (projected) 100% Renewals in 2014! Customer Satisfaction 100%

4 What s up? Positioning Test Automation as part of Continuous Delivery Practice at bol.com History of testing and test automation at bol.com Organizing it all: making sense of test results Practice at bol.com

5 First Principle Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

6 Continuous Delivery The three base principles Base principle 1: Rigorous Automation Base principle 2: Extreme Feedback Base principle 3: Continuous Change

7 Transformation to Test Automation SPECIFY DESIGN BUILD TEST INTEGRATE REGRESSION USER USER ACCEPTANCE RELEASE Development = Test Test = Development Automate ALL Acceptance Driven Testing User Acceptance Test effort

8 CD & bol.com Focus on test automation Special team with focus on improvements for all Scrum teams Teams do TA themselves Running tests as often as possible Not only once a day, but also after deployments (to test) Focus on fast test! Fast feedback! Earlier insight in regression Development of MayFly (soon to be open sourced!) User story-based Creates separate dev and test environments per application More Goto Amsterdam

9 History of bol.com Where did we come from? All testing done manually until started automating with QFTest 2012 evaluated other tooling and started working with around that time also developed in-house tool for testing databases 9

10 History of bol.com Where are we now? FitNesse is mostly used It is Java based and we are a Java company The wiki format makes easy reading for business New tools are introduced Spock 10

11 FitNesse implementation Started with 1 project containing all tests Sharing knowledge Structured the same as our use cases, i.e. WebshopSuite.BusinessAccountSuite.UseCase1500 Nightly runs from the beginning Indication by labels (nightly) First sequential per application WebshopSuite Later parallel by split running of suites on the lowest level (a.k.a. suites with only tests) WebshopSuite.BusinessAccountSuite.UseCase

12 Running & Reporting Nightly structure Maven is used to bootstrap FitNesse In combination with a JUnit Runner Specialized Jenkins job for parallel execution Knows the structure of FitNesse and fires corresponding jobs The tests are executed by Jenkins Slaves Specialized Linux and Windows hosts with browsers Chrome, Firefox, IE8 and IE11 Suites are run and results are sent to the reporting tool Data stored in MySQL database Reports generated by BIRT 12

13 Running & Reporting Issues with the setup 1 FitNesse project is too big More smaller projects need more maintenance BIRT was getting very slow To much data to process Reports did not look very fancy Rerunning (failed) tests was not intuitive No links in reporting Need to know where to find it in Jenkins 13

14 Running & Reporting Introducing XL Test at bol.com One endpoint for reports XL Test takes over orchestration from Jenkins Custom scripting options for parallel execution of suites Custom graphs for bol.com We need a dashboard with all tests for a day, per team & application Added analytics for test stability Reports on duration and flakiness Developed in close collaboration between bol.com and XebiaLabs Tailored to the needs of bol.com Ideal kick-start of the development of (now) XL TestView 14

15 Introducing The first test results management and analysis tool that allows you to define and execute tests across your full spectrum of test tools, then aggregate, and visualize results to provide insight into your application s quality. XL TestView serves as your central hub for quality assurance, spanning test tools to offer a configurable dashboard, history and reports for all test results. Make data-driven release decisions and optimize your entire testing efforts. Think of it as your central nervous system for quality within your Continuous Delivery pipeline or as the fitness tracker (FitBit) for your software s health.

16 Test Results in Perspective

17 More Test Results! ;-)

18 Aligning Results for Go-Live Recommendations

19 Never Change a (Winning) Jenkins Setup

20 XL Test at bol.com: Running & Reporting 20

21 XL Test at bol.com: Running & Reporting 21

22 XL Test at bol.com: Running & Reporting 22

23 XL Test at bol.com: Running & Reporting 23

24 XL Test at bol.com: Running & Reporting 24

25 Key take-aways Continuous Delivery needs Test Automation Test automation done right needs structure and oversight! XL Test helped bol.com and XL TestView can help you make sense of your test results 25

26 Thanks! till next bol.com Jeroen Ruijgers Viktor Clerc Software Engineer Product Manager