Organising Massive Test Automation @ bol.com Jeroen Ruijgers and Viktor Clerc 1
Introducing 5 million customers 9 million products #1 online retailer in the Netherlands and Belgium 1999 bol.com becomes separate (and Started by Bertelsman Online only) entity 2001 2008 2015 6 employees Started with Scrum > 35 scrumteams > 900 employees 2
Introducing A leader in Software Delivery Automation Headquartered in Boston, Massachusetts Sales in US, EMEA and APAC 150 premier customers Growth over 100% y/y REVENUE 110% 140% 100% 2012 2013 2014 (projected) 100% Renewals in 2014! Customer Satisfaction 100%
What s up? Positioning Test Automation as part of Continuous Delivery Practice at bol.com History of testing and test automation at bol.com Organizing it all: making sense of test results Practice at bol.com
First Principle Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software
Continuous Delivery The three base principles Base principle 1: Rigorous Automation Base principle 2: Extreme Feedback Base principle 3: Continuous Change
Transformation to Test Automation SPECIFY DESIGN BUILD TEST INTEGRATE REGRESSION USER USER ACCEPTANCE RELEASE Development = Test Test = Development Automate ALL Acceptance Driven Testing User Acceptance Test effort
CD & TA @ bol.com Focus on test automation Special team with focus on improvements for all Scrum teams Teams do TA themselves Running tests as often as possible Not only once a day, but also after deployments (to test) Focus on fast test! Fast feedback! Earlier insight in regression Development of MayFly (soon to be open sourced!) User story-based Creates separate dev and test environments per application More information @ Goto Amsterdam 2015 8
History of testing @ bol.com Where did we come from? All testing done manually until 2010 2010 started automating with QFTest 2012 evaluated other tooling and started working with around that time also developed in-house tool for testing databases 9
History of testing @ bol.com Where are we now? FitNesse is mostly used It is Java based and we are a Java company The wiki format makes easy reading for business New tools are introduced Spock 10
FitNesse implementation Started with 1 project containing all tests Sharing knowledge Structured the same as our use cases, i.e. WebshopSuite.BusinessAccountSuite.UseCase1500 Nightly runs from the beginning Indication by labels (nightly) First sequential per application WebshopSuite Later parallel by split running of suites on the lowest level (a.k.a. suites with only tests) WebshopSuite.BusinessAccountSuite.UseCase1500 11
Running & Reporting Nightly structure Maven is used to bootstrap FitNesse In combination with a JUnit Runner Specialized Jenkins job for parallel execution Knows the structure of FitNesse and fires corresponding jobs The tests are executed by Jenkins Slaves Specialized Linux and Windows hosts with browsers Chrome, Firefox, IE8 and IE11 Suites are run and results are sent to the reporting tool Data stored in MySQL database Reports generated by BIRT 12
Running & Reporting Issues with the setup 1 FitNesse project is too big More smaller projects need more maintenance BIRT was getting very slow To much data to process Reports did not look very fancy Rerunning (failed) tests was not intuitive No links in reporting Need to know where to find it in Jenkins 13
Running & Reporting Introducing XL Test at bol.com One endpoint for reports XL Test takes over orchestration from Jenkins Custom scripting options for parallel execution of suites Custom graphs for bol.com We need a dashboard with all tests for a day, per team & application Added analytics for test stability Reports on duration and flakiness Developed in close collaboration between bol.com and XebiaLabs Tailored to the needs of bol.com Ideal kick-start of the development of (now) XL TestView 14
Introducing The first test results management and analysis tool that allows you to define and execute tests across your full spectrum of test tools, then aggregate, and visualize results to provide insight into your application s quality. XL TestView serves as your central hub for quality assurance, spanning test tools to offer a configurable dashboard, history and reports for all test results. Make data-driven release decisions and optimize your entire testing efforts. Think of it as your central nervous system for quality within your Continuous Delivery pipeline or as the fitness tracker (FitBit) for your software s health.
Test Results in Perspective
More Test Results! ;-)
Aligning Results for Go-Live Recommendations
Never Change a (Winning) Jenkins Setup
XL Test at bol.com: Running & Reporting 20
XL Test at bol.com: Running & Reporting 21
XL Test at bol.com: Running & Reporting 22
XL Test at bol.com: Running & Reporting 23
XL Test at bol.com: Running & Reporting 24
Key take-aways Continuous Delivery needs Test Automation Test automation done right needs structure and oversight! XL Test helped bol.com and XL TestView can help you make sense of your test results 25
Thanks! till next bol.com Jeroen Ruijgers Viktor Clerc Software Engineer Product Manager jruijgers@bol.com vclerc@xebialabs.com