How Spotify Tests World Class Apps Alexander Andelkovic, Spotify

Size: px

Start display at page:

Download "How Spotify Tests World Class Apps Alexander Andelkovic, Spotify"

Lilian Patrick
10 years ago
Views:

1 How Spotify Tests World Class Apps Alexander Andelkovic, Spotify January 17, 2014

2 Agenda Introduction Background Approval process Future Summary Questions

3 Introduction About me - Lives in Stockholm, Sweden years testing experience - Have a background in telecom and Med-Tech - Test lead at Spotify - Passionate about advanced test techniques

4 Background Spotify Apps - Apps(HTML5 & JS) integrated into Spotify music client(c++) - Launched 30 Nov 2011 (at launch13 apps, today over 100 apps) - Different categories of apps (discovery, concert, review, social, lyrics) - Manual QA process

launch13 apps, today over 100 apps) - Different categories

5 Background QA Challenges - Spotify App = mini application - Release high quality in no time - Testing needs to scale over time(partners need to test) - Stable test environment(new API, backend) - Dependant on external development(partners)

over time(partners need to test) - Stable test

6 Approval process - Introduction - Steps that help partners release high quality apps - Approval steps: Concept > Development > Release > Update - Average app development time 3 months - Approval team around persons

Concept > Development > Release > Update - Average app

7 Approval process - Team - Product owners(prioritize tasks, final calls) - QA(manual testers) - Test automators(regression test) - Designers(concept, prototype) - Release manager(silent/big bang release)

8 Approval process - Concept - An app shall bring valuable content or something unique to the Spotify experience. - A good concept will save development time - Summary, Mockups, Wireframes, Screenshots, Alpha version - Designers gives valuable feedback - No development should be started until concept is approved

- A good concept will save development time - Summary, Mockups, Wireframes,

submited for approval when production ready - QA gives thorough

9 Approval process - Development - App development can start after concept approved and agreement signed - App should be submited for approval when production ready - QA gives thorough feedback on app quality - An app is approved when all issues are resolved

10 Approval process - Release - When an app is approved a release date and market(s) are decided - PR involved(internal, external) - Release manager inform partner when app is live - Live app needs to be maintained

11 Approval process - Update - Bugfixes, tweaks, small features - Requires new App Submission - App goes through the approval phase for Development again - New version treated as a new app concept(approval phase concept)

through the approval phase for Development again - New

12 Approval process - Kanban - Just-in-time approval process - App submission enter pipeline - Approval process(concept > Development > Release > Update) inside pipeline - App submission approved/not approved emerges from the other end of the pipeline - Effective development process for identifying bottlenecks

inside pipeline - App submission approved/not approved emerges from the

13 Approval process - Support - Stack overflow - Internal tech support - Keeps guidelines updated - Best practises posted to developer blog

14 Approval process - Partners - One man shows - Web developers - App developers - Inhouse development

15 Approval process - Checklist - Approximately 50 teststeps(manifest, navigation, performance, security etc..) - Should be run through by partner before app submission - Session based testing used(charter, timeboxed, debrief) - Test session divided into smoke test and exploratory test(et) phase - ET mindset crucial - Pair testing performed on first submission to find as many bugs as possible(reduces re-submissions)

timeboxed, debrief) - Test session divided into smoke test and exploratory test(et) phase - ET mindset

16 Approval process Test automation - Regression vs bug finding test automation(ta) - Robust TA a challenge(false positives) - Empower partners to perform own TA - App TA tests production environment

17 Approval process - Security - App security tests performed by Security Penetration Testers - All ways of input needs to be tested - Are HTML5 and javascript apps a security risk? - Partner homepages

18 Approval process - Metrics - Performance metrics(daily, weekly, monthly goals) used to tweak approval process - Enables greather understanding of user app behaviour - Drives A/B testing of appfinder - Continous app content tweaking

Enables greather understanding of user app behaviour -

19 Future - Quantity vs quality - More test automation(bug finding, crawler) - More advanced apps(webgl) - App SDK s

20 Summary - Developer performing tests early reduces app development time - Tweak it mindset crucial for an efficient approval process - App develpment a new dicipline - Apps are fun and easy try it out

21 Questions

22 January 17, 2014 Thank you!

UX for Successful Products

UX for Successful Products CRAFTING THE BEST EXPERIENCE FOR YOUR USERS Our 360 solution enhance your product experience. We create digital experiences users want to interact with, which is the core of