Speed to Market in Mobile Development

Speed to Market in Mobile Development Finding the right solution with continuous integration on real devices Tina Su, Director of Development, Intuit

About me Nearly 20 years of industry experience in R&D, quality and engineering management, with a strong passion for automation. Engineering Director at Intuit, Developer Productivity Sr. Engineering Director at Yahoo, Mobile Platforms Sr. Engineering Manager at Sun, Java Standard and Mobile Editions Compiler Developer at MIPS and Tandem Tina Su, Intuit

Agenda Mobile trends & developers challenges Speed & quality through continuous integration - Intuit Virtual Device Lab - Intuit mobile test automation Takeaways

Aggressive mobile momentum Will surpass desktop by 2014 Proliferation of platforms, OS versions & device types It should just work! 2013 Internet Trends, KPCB

Developer challenges Immense configuration matrix - OS platforms (Android, ios, Windows, phone vs. tablet, ) - Inputs (touch/gesture, real buttons, voice, camera, GPS receiver) - Outputs (portrait/landscape, screen resolution, language) Application delivery through the markets - App signing, provision (app & device), submission Offline experience - Network, different network bandwidth/latency, no-network - HTML5 offline app Media challenges - Some codecs don t work on a device - Streaming, sound Security & privacy

Intuit mobile app portfolio today 22 Apple 15 Google Amazon 11 3 SMS 491 mobile apps for financial institutions 0 5 10 15 20 490

Ramping up Intuit s innovation engine Pain: Takes days per build to test combinations of device/os configurations Platforms Devices OS versions Builds physically & manually, one by one QA can t keep up with the changes X X X Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices Strategy: Speed & quality through mobile Apple Google Amazon iphone 4 iphone 4S iphone 5 ipad 2 ipad 3 ipad Mini ios 5.0 ios 5.1 ios 6.0 ios 6.1.x ios 7.0 Daily builds continuous integration

Continuous integration (CI) What is CI? Automate build, test & deployment process Integrate & build early, often, daily, for every check-in Best practice of Agile development Jenkins is widely adopted, flexible & extensible Benefits of CI in a mobile world Instant feedback on quality, functionality & system impact Less complex, no integration hell Detect and address device configuration issues early Involve stakeholders early Ultimately achieve shippable release any time

Developers/ IDE Speed and quality through mobile CI Check in Test results Process starts here SCM: source repository Submit the app to app store if passed Analyze test results Key enablers: Check-in triggers automated CI Mobile device lab Test automation Automatic test run Run tests on real devices in parallel Jenkins Trigger testing in mobile device lab Real devices Test processor Install the test app to real devices in parallel

3 rd -party vendor solutions didn t work for us Couldn t handle E2E automation of testing pre-release apps - Dependency on internal resources & infrastructure Vendor lock-in: customized CI & test automation solutions - No flexibility to use our existing tools & frameworks Disadvantages of vendors shared devices - Security concern on testing on shared devices - No access to corporate network - Cost-prohibitive to reserve devices Private cloud (on-site setup) with local devices not practical - With sensitive customer data, we need private cloud - Extensive use of devices for daily testing of hundreds of mobile apps is expensive

Why test on real devices? Most mobile features can be tested on simulators, but simulators have limitations: Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density Real-world experience over the real connection (3G, 4G, crappy Wi-Fi, etc.) True performance, security tests Testing on non-default Web browsers (e.g., Opera Mini) Not all simulator/emulators have good quality (don t reflect real rendering/behavior, bugs) Testing SMS & app integration

Intuit Virtual Device Lab

Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser User machine 3-click access Intranet User machine Always on Globally available VDL User machine Integrated with dev infrastructure Manufacturers/ devices: iphone Galaxy Droid Optimus Nexus One ipad Captivate Milestone Desire Moto Incredible

VDL architecture Android Internet/ intranet Device interaction app Windows PC USB connection Image transfer Tomcat Java image transferring application Browser & Java applet Developer s laptop VDL

VDL architecture ios Internet/ intranet VNC server Mac with Xcode installed USB connection Port forwarding Tomcat Java VNC client Browser & Java applet Developer s laptop VDL

Open source tools & technologies used in VDL Key capabilities Android ios Installing mobile applications into the real device adb install <APK file> Fruitstrap https://github.com/ghughes/ fruitstrap Transferring device screen images, controlling device Screencast tool http://code.google.com/p/ androidscreencast Veency VNC app Fetching device local files adb pull <filename> sftp command (jailbroken devices only)

VDL key features & functionality today Control the device through standard browser/ RESTful APIs Change orientation to landscape or portrait Copy local file to the device from your PC Get file from the device onto your local PC Execute an ADB command on the device See current running logs in your browser Download history logs to your local PC

Continuous integration how it works Trigger file launches test Monitor for new builds Test execution engine Resource manager VDL device inventory Product builds Results Select a device

Mobile continuous integration Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously. 1 Select devices & tests 2 Tests run automatically across selected devices with each build Submit, build, test in minutes vs. hours 3 See the results!

Next step: over-the-air replaces USB connection VDL Fast onboarding App installation Test execution Internet/ intranet Scalable Any device Results retrieval

Mobile Test Automation

Mobile test automation tools Instrumentation-based ios MonkeyTalk Android Frank UISpec Calabash (LessPainful) TestDroid (BitBar) Robotium UIAutomation Soasta Android SDK ios SDK ios Driver Appium Non-instrumentation MOET Sikuli MonkeyRunner Cross-platform solution DA/ PerfectoMobile eggplant Assessed as of 2012

Mobile test automation trends Abstraction with domain-specific language (DSL) - Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET) - Support behavior-driven development (BDD) Client/server-based approach (remotely driven tests via HTTP) Instrumentation & non-instrumentation solutions complement each other Mobile cloud testing environment - From hardware (cradle) to software approach to drive tests on real devices Automated OTA app installation & tests

Intuit automation framework of choice/recommendation Native apps: - Cucumber + Calabash for BDD on ios & Android Mobile Web: - Selenium/WebDriver

What is Cucumber? It lets stakeholders (customers, QE or developers) describe how software should behave in plain text. BDD: Written in business domain-specific language (DSL) - Good abstraction over different device types One format for multiple purposes - UI feature specification - Automated tests Cucumber itself is NOT a test automation tool - Add-on to any test automation framework - Examples: Calabash, Sikuli & Frank mobile test frameworks

What is Calabash? Open source cross-platform mobile test framework - ios + Android, simulator + real device, native + hybrid Supports Cucumber for BDD Application needs instrumentation that embeds a Calabash HTTP server - Tests run from a remote client (in a Jenkins slave) - Tests can be driven via Wi-Fi (no USB wire) Ruby clients in API level (Java clients in experiment) Works better in touch simulation & real device testing Mobile cloud test execution available by LessPainful

Intuit shared test library rapid test development Common test library Product tests High-level test domain-specific language (Cucumber steps) Teams focus on creating Mobile front-end Native/hybrid test framework Use shared test lib Tests Tests Tests Tests Globalization test framework (I18N/L10N) Mobile back-end Security test framework Performance/load test framework Contribute and then contribute to common test library Internal open source repository

Stunning results: reduced iteration cycle from hours to minutes From: manual Deploy 10 5 Code build Notify email Get devices Connect device * 50*5 = 250 Test Consolidate report Upload report = 6 hours 15 30 20 30 10 15 Results To: automated CI Code build Trigger Deploy on devices Execute tests in parallel Consolidated report in archive = 15 mins 0 5 10 0 Tested on 5 devices Release quality increased TIME SAVED: 95%

VDL cost saving over vendors private clouds (today) Assumptions: 50 native Annual cost ($) 3 rd -party vendor 1 local device in cradle apps & daily check-in Note: As vendor pricing 3 rd -party vendor 2 local device w/o cradle (Software approach) Savings Savings changes, so do the savings Intuit Virtual Device Lab 25 150 # of devices Private cloud setup behind corp firewall

Key takeaways Building custom solution vs. 3 rd -party vendor solution Understand your technical/platform/global spread Many vendor solutions provide a larger set of devices/os versions in the global market Quite a few offer mobile test services might be tightly coupled with their device access solution & automation framework Connecting to your corporate network is a challenge some provide private cloud (on-site setup within your network) but it s cost-prohibitive Invest in automation and CI Automate tests with robust automation framework Enable rapid test development through shared test library Adopt CI in mobile development for speed & quality

More mobile innovations at Intuit Mobile crash reporter Mobile components & design patterns Mobile analytics SDK & gateway Visit booth #601 Accelerate Development Demos & conversations Contest Mobile device charging station Foosball We re hiring! Win a weekend with a TESLA!

Thank you!! Tina_Su@intuit.com