Building Mobile App Test Automation into a Cloud-based Continuous Integration Pipeline Mark Pedersen YOW! Connected 2015
QA shifting from being concentrated in specialist testing roles back to being diffused throughout the software lifecycle
Assembling the Solution
Issues with Test Automation Test Automation = Software Development Maintainability = Sustainability Platform Fragmentation is not your friend
How much device coverage do you actually need?
Barriers to Mobile App Test Automation Platform control ios is strictly controlled, making automation tricky Apps either need to be instrumented, or ios must be jailbroken OR. You can use Appium Android apps less restricted, but root access may be needed for full test automation Object recognition Native object recognition not always reliable Must find a reliable element to recognise Embedded Web View elements require special handling
Appium Architecture
Why Cucumber? Requirements and test execution are inherently linked
Automation in BDD Given I have some test scenarios written in a cucumber style When I process those scenarios with a cucumber parser Then the corresponding pieces of automation code will be executed
Writing Scenario's in Cucumber Scenarios are organized together into features Each feature is represented as a plain text file. Feature files must have the.feature extension Each feature can contain many scenarios
Step Annotations import cucumber.annotation.en.*; Each Gherkin step keyword has a Annotation @Given("^I have a test written in a cucumber style$") public void methodname() { }
Capturing Arguments @Given("^an owner with a pet called \"Sly\"$") public void an_owner_with_a_pet_called() { } @Given("^an owner with a pet called \"([^\"]*)\"$") public void an_owner_with_a_pet_called(string name) { }
Cross Platform Scripting Approach Framework component Feature layer Given Open App Example Step Definition layer View layer public void Open_app(){ if(system.getproperty("platform").equals("android")) objstepview.androidopenapp(); else if(system.getproperty("platform").equals("ios")) objstepview.iosopenapp(); } public void AndroidOpenApp(){ WebDriverWait nwait = new WebDriverWait(objStepBase.getDriver(), 360); nwait.until(expectedconditions.presenceofelementlocated(objhomepage.getnotnow())); objutilities.takescreenshot(); objhomepage.clicknotnowlink(); } Page layer public By getnotnow(){ By mysearchcriteria = null; if (System.getProperty("platform").equals("android")){ mysearchcriteria = By.name("Not now"); } else if (System.getProperty("platform").equals("ios")){ mysearchcriteria = By.xpath("//UIAApplication[1]/UIAWindow[1]/UIAButton[1]"); } return mysearchcriteria; }
Structuring Features Cross platform scenarios Given the underlying step definitions, views, page objects support cross platform logic, When a single script is tagged with both @Andoid and @ios It will be run on both platforms by the relevant maven task Alternatively, individual platform scripts can be maintained One feature file can contain multiple scenarios Individual scenarios shouldn t be be too long Max executions limit on your device cloud can be easy to hit!!
TestDroid: Devices in the Cloud Other Device Clouds: SauceLabs Xamarin TestCloud AWS Device Farm Android only Google Cloud Test Lab Not yet launched
Connecting with Test Droid if(objconfig.getproperty("test.os").equals("android")) { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setcapability("platformname", objconfig.getproperty("device.platformname")); capabilities.setcapability("testdroid_target", objconfig.getproperty("test.testdroid.target")); capabilities.setcapability("devicename", objconfig.getproperty("test.testdroid.devicename")); capabilities.setcapability("testdroid_username", objconfig.getproperty("test.testdroid.username")); capabilities.setcapability("testdroid_password", objconfig.getproperty("test.testdroid.password")); capabilities.setcapability("testdroid_project", objconfig.getproperty("test.testdroid.project")); String strtestrun = objconfig.getproperty("test.testdroid.testrun") ; Calendar cal = Calendar.getInstance(); strtestrun = strtestrun + "_" + cal.get(calendar.date) + (cal.get(calendar.month) + 1) + cal.get(calendar.year) + "_" + cal.get(calendar.hour_of_day) + cal.get(calendar.minute); capabilities.setcapability("testdroid_testrun", strtestrun); capabilities.setcapability("testdroid_device", objconfig.getproperty("test.testdroid.device")); capabilities.setcapability("testdroid_app", objconfig.getproperty("test.testdroid.app")); } androiddriver = new AndroidDriver(new URL(objConfig.getProperty("test.testdroid.server")+"/wd/hub"), capabilities);
Connecting with TestDroid test.testdroid.platformname=android test.testdroid.target=android test.testdroid.devicename=android Device test.testdroid.username=mark.pedersen@kjross.com.au test.testdroid.password=magicpudding test.testdroid.server=http://appium.testdroid.com test.testdroid.project=ju_appiumandroid2 test.testdroid.testrun=androidrun test.testdroid.device=samsung Galaxy S V SM- G900F (Europe) test.testdroid.app=latest test.testdroid.package=android.abcapplication test.testdroid.activity=android.abcapplication.activities.preloaderactivity test.testdroid.waitactivity=android.abcapplication.activities.preloaderactivity
Tester Integrating with Bamboo
Integrating with Bamboo
Integrating with Bamboo Bamboo Bamboo Plan Configuration 1 stage 2 Jobs (can run in parallel) Run Android Tests Run ios Tests Each job: Check out code from repository Run Maven 3 clean test with options Maven options: - Denv: local testdroid - Dplatform: android ios - Ddevice: <device name> - Dcucumber.options: - - tags @Android etc
CI Integration Recommendations Scheduled test runs rather than trigger on check- in Create one Bamboo job for each device/platform to be tested Create and enable one maven task in each job You may end up creating a set of maven tasks for each device and just enabling the right task for the configuration you want
Challenges Getting the structure of the automation abstractions right takes time: You must balance ease of use by script writers (who may be non- technical product owners) with ease of maintainability by technical team (test automators / developers) Device Clouds are shared resources: TestDroid device availability frequently caused timeouts You need to either invest in hardening your local execution methods against timeouts Or pay more to get server- side execution / dedicated devices Re- writes of apps means updating automation Pick the right time to start! The cultural challenge of getting features files to be the single point of truth for requirements is possibly biggest challenge we re still working on that.
QUESTIONS? mark.pedersen@kjr.com.au @mark_j_pedersen