Open Source in Mobile Test Automation Ru Cindrea - Altom ru@altom.ro
About me software tester since 2002 BS in Computer Science 7 years of mobile application testing testing consultant and managing partner at Altom since 2008 member of Association for Software Testing and instructor for AST Black Box Software Testing online course currently using test automation on a customer project for a mobile phone application running on Android and ios
My context and my views working with Maaret Pyhäjärvi on FAST Test Lab concept (open space organized for practicing testing with guided exercises, where all tools are open source) => mobile testing and mobile test automation will be part of it strongly believe that: TESTED = CHECKED + EXPLORED and that: Results of automated checks are not binary (passed/failed), but at least fourfold: probably fine, miss, potential problem, false alarm. (@ilarihenrik)
In this presentation look at common concerns I have noticed regarding open source tools in general analyze these concerns in the mobile test automation context, with focus on Android and ios platforms show actual examples of one open source tool used for developing behavior-driven tests on iphone provide some options and insights for testers working on mobile application development projects
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Mobile automation options Robotium (http://code.google.com/p/robotium/) - most commonly used test framework for Android, Java based UISpec (http://code.google.com/p/uispec/) - ios behavior driven test automation framework modeled based on RSpec MonkeyTalk a.k.a Fonemonkey (https://www.gorillalogic.com/monkeytalk) - supporting both ios and Android Frank (http://moredip.github.com/frank_at_selenium_slides.html) - ios support based on UISpec NativeDriver (http://code.google.com/p/nativedriver/) - implementation of WebDriver for mobile platforms, currently supports ios, Android with plans for multi-platform Calabash (http://github.com/calabash/) - both ios and Android, behavior driven test automation framework - to be discussed and showed in more detail TDriver (http://projects.developer.nokia.com/testabilitydriver/wiki/introduction_to_tdriver) - for QT mobile applications, open sourced by Nokia
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Support with open source tools the short answer: Google it! (it works!) most open source tools have community online groups where you get answers in less than 24 hours and even fixes from others who are actually committed and interested in the tool a lot of the open source solutions are supported by a company that offers professional paid support for the tool as their main business different commercial packages: -> initial training -> initial test automation setup -> general support services -> continuous support services customized for your own application support isn t exactly free with commercial tools either since you pay for the tool in the first place
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Example with Calabash used for Calabash Wordpress Mobile -> functional test automation framework for Android and ios -> open-sourced by lesspainful.com, who offers support and customized testing services using Calabash -> focuses on Cucumber (https://github.com/cucumber), an open source behavior driven development tool that uses Gherkin - a business readable domain-specific language used to define tests in a non-technical way, using examples -> but also supports any Ruby-based test framework -> really good documentation on Google Groups and GitHub
Example with Calabash used for Wordpress Mobile Calabash installation + configuration (ios) (explained in the Fast Track installation instructions here: https://github.com/calabash/calabash-ios) Installation done! Configuration done!
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Example with Calabash used for Wordpress Mobile Sample app: Wordpress Mobile -> mobile client for popular Wordpress blogger platform for ios + Android:
Let s try it! Feature: Editing a post on a self-hosted blog As a Wordpress user with a self-hosted blog I want to be able to edit my posts So I can easily manage my blog from my phone Scenario: Edit a post as author Given I login with "mobileapp" to "http://thesoftwaretestlab.org" When I select the "What is the Test Lab?" post Then I should be able to edit it
Commercial is easier commercial only looks easier because: -> sales focus on ease of use of the tool by non-technical testers -> record and playback is emphasized in practice: -> test automation involves programming and is not done by non-technical testers -> scripts produced with record and playback are hardly ever usable -> the more technical details you understand about the app and technology you re testing the better still: -> tools like Calabash are extremely easy to use -> MonkeyTalk for example offers record and playback
Most Common Concerns there are not that many common open source tools used in mobile test automation open source tools don t offer the same level of support as commercial tools open source tools have little documentation and are difficult to configure you might need to invest too much time to understand how to use open source tools and they have no record-playback functionality (commercial is easier) open source tools might disappear or its contributors might lose interest
Open source projects are often abandoned real risk (also a risk with commercial tools) however if this happens: -> you most likely have only spent time (and no other money) on the tool -> you still have the source code and others might still get involved -> you can switch to using another open source tool without the hassle of explaining why money was spent on something useless testing tools like Jenkins, Selenium, Cucumber, Robot Framework, JUnit etc. give us hope imagine the software world without Linux, Apache, Git, Eclipse, MySQL, Firefox, Android etc.
Questions?
Thank you! For any questions or thoughts, email me at ru@altom.ro