Increase Your Mobile Test Automation ROI

Transcription

1 Increase Your Mobile Test Automation ROI Make your Mobile Test Automation more mature and dependable. Take your framework to an all new level of robustness by executing your tests on real devices and at the same time increase your ROI This white paper should help you in understanding the nuances of mobile test automation, its challenges and how to derive more from your test automation effort without having to compromise on cost or quality About the Author Aniket Deshpande Pune, India Master in Business Administration (Information Technology) Aniket Deshpande is a Test Architect at AFour Technologies. He has around 10 years of experience in testing varied applications across different platforms including mobile applications and has formulated testing strategies of various mobile applications. As a Test Architect, he has led teams in diverse domains like Social Engineering, Health & Fitness, E-Commerce, etc. His primary interests lie in conceptualizing & implementing mobile automation frameworks and environments and setting up mobile device labs.

2 Introduction It is always risky not to test on real devices Which devices would you choose? Need to manage your automation testing cycles across these devices Choosing the right tool is very important Integrating your automation runs with the build cycles Mobile Application testing can be a tricky proposition. With the plethora of devices, operating systems and their versions, further compound the problem. As gatekeepers of quality, we QAs cannot be satisfied with testing only on simulators and emulators. It is always risky not to test on real devices for various reasons, above all being, end users would not be using the application on simulators but on real world devices. Hence, your mobile test automation should also sign-off the build on real devices. Should you choose to run your automation on real devices, which devices would you narrow down on, from the ever-growing list of mobile devices? Using your short-listed devices, you would need to manage your automation testing cycles across these devices cleverly to be able to derive more from your investment. For planned automation initiatives, choosing the right tool is very important for maximizing returns on your automation effort. You do not want your tool to throw you ugly surprises later in your framework development efforts. Integrating your automation runs with the build cycles gives you fantastic ROI as you may have already realized with your normal web and desktop automation frameworks. Setting up such a process on your mobile automation will not only fetch optimal returns, but also make your automation a truly hands-off process. This White Paper will give you insights into how we achieved increased ROI at minimal cost while implementing a mobile automation framework for one of our clients. The solution includes choosing a tool, automating the tests, setting up of a mobile device lab for test execution and integrating the automation runs with the build process. 2

3 Trends One in every 5 people now own some kind of a smart phone Enterprises want faster time-to-market Gartner: By 2015, 80% of all application development activities will be for smartphones & tablets More mobile apps are being downloaded than ever before and the number is increasing yearly. The year 2013 has seen an increase of more than 50% in apps being downloaded from the respective Stores 1. By the end of year 2013, there were more smartphone owners in the world than PC owners. One in every 5 people now owns some kind of a smartphone 2. This further goes on to show the demand for mobile apps is ever increasing and there are more smartphone users than PC users. This would mean enterprises would want to quickly reach out to users on the mobile platforms than traditional websites approach and would want faster time-to-market, which in turn would demand process innovations in QA cycles. Businesses with mobile apps have a customer conversion rate of more than 5% as against less than 1% for businesses with no mobile interface 3. Gartner says that, by 2015, around 80% of all application development activities will be for smartphones & tablets. And having a mobile application, systems & devices are one of three top CIO priorities 4. All this points to a huge upsurge in mobile application development. Many of our customers have release cycles as small as 1 week. The reason being the ease of submitting the app to the Store and application upgrade on the user s device. As QAs we need to be ahead of the curve and make sure we are prepared to handle such demanding timelines. More than technological challenges, there is a need for upheaval in the QA process itself, which can help in making the process sleeker and agile. 3

4 About the App: The mobile application under test is a Health & Fitness application, which is freely downloadable from App Store (ios) and Play Store (Android). The application has implemented WebViews for user interactions and REST API calls for server communication. Problem 1. Tool Selection: This, by far, was the most vital of all our activities for implementing the solution. We had to find a tool that we could use across both ios and Android platforms, because using 2 different tools is just not the right way to go. Among other things, maintaining 2 separate code bases and technologies would be an overhead later in the automation cycles. Also, we wanted a tool that would seamlessly work with a CI server so that we could integrate it with the development build cycles, which would ensure a truly hands-off automation execution. The tool should also have rich reporting. We could trade-off this feature by implementing our own reporting module, but an in-built reporting engine is always more preferable than a custom one. And importantly, the tool should work smoothly on real devices and not only on emulators & simulators. Reasons for running on real devices: The end-user downloading the application from the Store, will be using the application on an actual device and not on a simulator. Features like GPS, for location services, can be tested reliably on a real device than on a simulator. Simulators & emulators use the host machine s hardware and processor architecture, which will be different than the actual device. RAM, Disc space, Battery usage, etc. can cause performance issues on a device but not on simulators. We have had many instances on other projects where we found a bug on the device but not on the simulator or vice-versa, which made the test results unreliable. Using a device, we could test applications on cellular data bandwidth as against wireless/lan in case of simulators, which is obviously always reachable and faster than cellular data. Most importantly, for Android devices, the operating systems released by OEMs contain many customizations on middleware and drivers, which may impact the application. You cannot install other applications on simulators, like browsers, or if your application uses social media, then dependency on the social app needs to be tested. This cannot be done on a simulator. 2. Mobile Device Lab: The reasons mentioned above justified setting up an in-house Mobile Device Lab. We could have used the services of a third-party device lab like DeviceAnywhere, but it would mean sharing data with third-party vendors. Also, with such device labs, the devices are not exclusive and are shared. You always run the risk of a rogue application sitting on the device listening to your activities. Since these shared devices are cleaned & reset after use, running automation on it would require you to install and create all pre-requisites every time you want to trigger an automation run. Another issue was, if you find a bug on that device and if later, the developer wants to reproduce it, then the device may no longer be available or would have been reset. 3. Continuous Integration: We are working with a team in diverse locations. Setting up a continuous integration server would make the automation execution truly hands-off. You have various options of scheduling or triggering your automation runs based on events that happen in your build cycles. Moreover, it mandates that your environments are standardized and encourages the team to automate the pre and post tasks as well. This approach also helps in managing your test cycles & reporting. If the triggers / schedule is setup cleverly, you do not have to wait for a QA to fire the runs. It becomes a 24/7 system. Stitching It All Together: The above challenges are inter-related with each one having an impact on the other. The UI automation tool selection, device lab design and CI tooling & strategy all need to be tied up together to create a tight harness. Lets see how the solution was implemented. 4

5 Solution 1. Tool Selection: Looking at the different problems and the importance of choosing the right tool, a detailed PoC (Proof of Concept) was carried out. The litmus test of the tool was automating 3 end-to-end test cases. We identified these test cases based on the complexity of the flow and the various controls encountered on the screen. For the PoC we considered, Calabash, Ranorex, Appium, Robotium & Sikuli. We evaluated these tools on the following features: Others Platform Support: Support for both ios & Android was mandatory. Maintaining separate code bases for both platforms will be an overhead in the long run. Since WebViews are used throughout the application, application flows across both platforms are similar. We could take advantage of this by writing a common code base for both platforms. The above criterion allowed us to focus our attention on Calabash & Appium. Tool Cost: Both Calabash & Appium are Free & Open Source tools. Other tools like EggPlant were also on the list, but the cost was too high to be considered as an option. Test Cases Automated: The ability of the tool to automate the 3 identified test cases. We found that Appium was suspect when it came to automating WebViews, while Calabash was quite stable. Also, Appium did not fully support ios7 at the time of the PoC. Ease of coding: Calabash uses Ruby+Gherkin, while Appium uses Java. It is easier to add new test cases to Calabash due to Gherkin BDD (Behavior Driven Development) implementation, but on the other hand, it is easier to find Java resources than Ruby ones. Hence, both scored average in our rating. Code reusability: Both support modular framework design hence code becomes highly reusable. Sikuli and Ranorex have image processing engines for object identification. This makes the code highly unreliable and unstructured. Tool Support: Both are open source and have ample support on various forums. Upgrades for new operating systems were also fairly quickly available. Source Code Requirement: For ios, Calabash adds its own library to the application source code and then builds and deploys the application. This is a huge negative for Calabash if your client is not comfortable sharing the source code. Appium requires just the UDIDs of the Apple device be included in the build. Both however, do not require source code for Android automation. Reporting: Calabash has its own reporting mechanism while Appium uses TestNG. Both have powerful reporting and Exception handling. They both conformed to our reporting requirements (Error logging, screenshots, run summary, etc.) Execution on real devices: Both Appium & Calabash can run automation on real devices. The process of device detection & test execution for both was quite simple and easily configurable. The execution speed was comparable to that of simulators. CI integration: Both have strong command-line execution processes and can be easily integrated into a CI server. We tried firing automation runs on a device for both Calabash & Appium through a Jenkins job and both ran successfully. 5

6 Both Appium & Calabash scored relatively equally on our PoC. In the end, the inability of Appium to support WebViews & the client agreeing to share the ios source code tilted the decision in Calabash s favor. You can find out more about the tool at The Framework: Next, we had to create an automation framework that will leverage the advantages of the Calabash tool, foremost being, able to create and maintain a common code base for both ios and Android automation. Although, both applications load the same WebViews, the method of referencing objects for both are a little different. We created an object repository where we maintained the object reference of each object for both Android & ios. It looks something like this: Android Repository: id=# password=#password loginbutton=#login ios Repository: id=webview css:'# ' password=webview css:'#password' loginbutton=webview css:'#login' In the code, we reference the object name: def Login( ,passwd) end if elementexists ($ids[ id ],30) end typetext($ids[ id ], ) if elementexists($ids[ password ],30) end typetext($ids[ password ], passwd) if elementexists($ids[ loginbutton ],30) end touch($ids[ loginbutton ]) Exception handling is done in the above code to capture failures and log them. Before execution begins, the respective Object Repository is automatically loaded based on the execution environment (ios / Android). Finally, the test cases are written in Gherkin (BDD): Scenario: Login with valid credentials Given I open the App and login with "[email protected]","spiderman" And I touch Profile tab Then I see that the screen changes to profile screen And step definitions are written in Ruby: Given /^I open the App and login with "([^\"]*)","([^\"]*)"$/ do , password homepage = Home.new homepage.checkwelcomescreen() homepage.login( , password) end By designing such a framework, we were able to create & maintain a single code base for both Android & ios. 6

7 The dedicated Mobile Device Lab at AFour Technologies 2. Mobile Device Lab: To support the application testing we had to test on multiple devices. Testing on iphones is quite straightforward since their devices are standardized and common throughout the world. But when it comes to Android there are a wide range of devices, resolutions & OEMs we need to consider. To finalize the list of devices that we would use for testing, we sought statistics about the active users of the application on both Stores. We found the top 5 devices & resolutions that had users actively using the application. Apart from the 4 iphones (4, 4s, 5 & 5s), we also procured the top 5 Android devices as per the statistics, to be part of our mobile lab. Collectively, they made up around 80% of the customer base. We created a separate physical space for these devices in our server room with dedicated power supply and Wi-Fi support. By using services like VNC, we were able to remotely connect and control these devices over LAN. We also setup a VPN for this lab for our client to connect to our network and access these devices over VNC. Now that the remote access problem was resolved, we had to solve the automation problem, where we had to trigger automation runs remotely on these devices. Calabash had very good support for this. We setup a separate Windows machine for the Android devices and a Mac Mini for the ios devices. Using Calabash s command-line arguments, we could trigger automation runs on specific devices using the device s UDIDs and IP addresses. 7

8 3. Continuous Integration: Having an automated job that will build and deploy applications on the mobile, trigger automation runs, notifications, etc. would really help in achieving a 24/7 service. You would not need a person to trigger off this complicated set of steps and everything would be a hands-off process. These are just some of the benefits that a CI server offers. We decided to use Jenkins as our CI Server where we would create jobs, which automate the entire run execution process. Common automation execution steps would include: Jenkins Picking up the specified.apk file (for Android) from a common repository Or, pickup a specific ios code base from a repository Build the code and/or deploy the application on the specified device Pickup the automation code from the repository Trigger the automation the execution report to the stakeholders We created various parameterized jobs in Jenkins and configured all the above steps in it. By starting the job, all the above steps would be carried out and the final execution report is ed to the specified people. Now, our client has integrated this into their build server. So, whenever new code is checked into their object repository, the Jenkins jobs are triggered through their integration server using the very useful Jenkins REST API calls. They can even specify the specific application build number against which they wish to fire the automation and the rest is taken care by Jenkins which would also notify them when the automation run finishes by ing the execution report. The Windows machine and the Mac PC are configured as Jenkins slaves and the jobs have been configured to run from these machines. 8

9 Benefits 1. Common Code Base By choosing Calabash that can be run across both platforms (Android & ios), we reduced our coding & code maintenance efforts by nearly 40%. And we will keep reaping its rewards in the future as we add more test cases or update existing application flows. The same automation engineer can write scripts for both Android & ios. 2. Automation Transparency Now everyone in the chain knows exactly what test cases are being executed and the test data being used, because Calabash mandates you to write your test cases in Gherkin (Given-When-Then) format which are easily readable and can be understood by everyone involved in the software development chain. Anyone can easily change the test case and test data or even add new automated test cases to the test suite. We, as QAs, have started writing all our test cases in a standardized Gherkin format instead of the old & traditional test case writing format. Eventually, we can just pick these tests and plug them into the framework. Again, saving us precious time & effort of re-writing automation scripts. 3. Reporting With Calabash s rich HTML reporting system with screenshots and activity logs, our automation engineers can spend their valuable time in writing quality code rather than worrying about how should the error be reported. On encountering an exception, Calabash will automatically report the error with screenshots and activity logs. 4. Device Security Since these devices are physically located in our secure server room, physical access is restricted, making the lab secure with specific access rights to authorized personnel, thus making the devices immune to unscrupulous software or malware installations. Remote access to the devices is audited and activity log is maintained. Data and software being accessed on the devices are limited to the device making client data & other collaterals secure. 5. Dedicated Access All the stakeholders in the software development cycle have dedicated 24/7 access to the devices making bug reproduction and testing more dependable and convenient. We seldom require resetting the devices since they are dedicated and we can securely install third-party application (like Social Apps) to check compatibility. 6. Testing on Real Devices Not testing on real devices is always a risky proposition as already discussed in this paper. Our testing results are more reliable since we test on devices. We can report issues in the software encountered for low bandwidth, security, performance, etc. We can also reliably sign-off on builds with the knowledge that builds are not misbehaving when installed on specific OEM operating systems. 7. Continuous Testing Since the Jenkins environment is integrated with the client s build & integration environment, automation runs on real devices can be kicked off immediately without any manual intervention or waiting for the QA team to start the process. Abstracting the mobile environment layer under Jenkins provided the client with a more user-friendly access to the mobile lab. By automating the entire pre & post steps required for automation runs, we have been able to save 1 full resource day spent in running automation daily on all 9 devices. 8. Scalability & Standardization The Mobile Device Lab mandates a particular process for adding devices to the lab. Such a process standardizes the devices and their inclusion into the device lab. It also makes the lab easily scalable. Common processes can be set for tasks such as resetting devices, installing / uninstalling software on the devices, software or OS upgrade, etc. We can now quickly scale up our mobile lab by just duplicating the VMs & Jenkins jobs and they become immediately available for automation execution. Parallel execution of test suites can be easily configured and fired under Jenkins. Although parallel execution-ability of the test suite depends largely on the design of the framework and test data, if done correctly, it helps in drastically reducing the time required to get the test results. 9

10 Future Planned Enhancements Code Coverage Statistics Using tools, we can find the code coverage statistics of an automation run. These statistics can be a useful indicator for the impact of the automation run. Using tools like Emma for Android and internal XCode configurations in ios, we can get a detailed report of the code that was traversed during an automation run. We can then tweak or add test scenarios to increase this coverage. Suite Execution Controller A controller that will execute customized test suites where the user should be able to provide details of the tests to run and the Jenkins job should execute only the selected test cases. Enhanced Reporting Dashboard Create a dashboard where the user can view current and past execution reports. Implement live reporting where the user can see the results of the test being executed. Create a mechanism where the user can select test cases and choose to execute them from the Reporting Dashboard itself, which will internally call the Jenkins API for job execution. Page Load Statistics Creating a simple logging mechanism, which will record the page load time during automation execution. This should provide us valuable information over various builds about the degradation of page load times among other things. This should also help us in finding performance bottlenecks. 10

11 Conclusion The customer saw great benefit in the solution that we provided them. They could now integrate their development builds with testing and get an immediate view on the health of the build to take a go/no-go decision. In the end, it s the entire turnkey solution that counts and that is what we have been successful in achieving. It was an interesting challenge to execute since there was a merging of multiple technologies and platforms. Watching the automation get executed on devices, which gets automatically triggered from a remote build completion was pure pleasure. There were many small challenges in the way and, for us, to work around those challenges made this a memorable solution to work on. The key to realizing the various benefits was to understand the pain-points of the customer and the QA team and then, design a foolproof solution that would satisfy all the needs. This solution has been able to address many such problems. 11

12 References