Automated testing for Mobility New age applications require New age Mobility solutions

Transcription

1 Automated testing for Mobility New age applications require New age Mobility solutions

2 Executive Summary Today, mobile phone has transformed from its former role as a mere medium of communication to that of a multifunctional personal gadget. Proliferation of devices and technological advancements has made it more difficult for the application developers and hardware manufacturers to develop and introduce new products. The number and variety of enterprise and consumer mobile applications has grown a lot in the last few years. Organizations need to make sure that each application is of high quality so as to prevent loss of revenue, productivity and brand reputation. However, testing mobile applications is different and even more complex than traditional desktop and web application testing. There are varieties of software platforms and versions, various hardware and form factors on which mobile applications need to be tested upon. There is frequent introduction of new devices and customers expect quick upgrades of the applications which require additional test cycles. This whitepaper investigates industry best practices for mobile testing and showcases effective ways of managing quality of mobile applications through test automation. As with established applications, there is an enhanced requirement to automate mobile applications testing to improve efficiency and scale. Hardware manufacturers and software developers are able to reduce effort and time to market of the applications with well defined automation strategy. Automated Mobility Solutions Device Specific User Interface and Visual Design Accessibility Solution Cross Browser Automation and Performance Native, Middleware, Hybrid Applications Fig 1: Various types of testing for Mobile Devices and Applications 2

3 Mobile Application 30% Project Effort SMS Mobile Internet + WAP JAVA + Phone Apps USSD < 1 Service < 2 Menu Authentication Security Database Access External Systems Customer Profiles and Data Customer Scenarios Complex Systems Integration Device Detection Web Services Performance /Load 70% Project Effort Fig 2: Mobile complexity is Below the Glass i.e. below the device screen in the operating systems, browsers, interfaces, network connectivity etc. Magnitudes of Mobile apps testing Before going deep into the complexity of testing mobile apps, it is necessary to know and understand the different characteristics of an application which are to be tested. Distribution channel (e.g. Google Play, Apple App Store, direct distribution) The kind of application (gaming, banking, social or business) Kind of Target audience (enterprise, consumer) and volume 3

4 Aspect Performance Functionality Security Network Compatibility Conformance Usability Installation and provisioning Areas/Types of UI responsiveness Peak load performance completion time(s) of transaction Longevity Transaction of User Interaction Transmission Security Retention of data on device Type of Network (Wi-Fi, 2G,3G,4G) effect on Connectivity issues Compatibility for mobile platform Compatibility for device model Backward Compatibility (with prior app version) Compliance of marketplace guidelines Compliance of enterprise policy User experience Installation procedure Un-installation procedure Provisioning for User and de-provisioning There is up gradation of mobile applications for various reasons: User expectation of rapid feature upgrades and fixing bugs. Increasing usage of agile development methodologies by the users cause frequent releases of applications with new features. Updating applications for taking advantage of new versions of mobility platforms like Android or IOS. Each upgrade requires an additional test cycle. Also, additional test cycles are required on release of a new version of mobile platform (like IOS or Android) to ensure compatibility. Likewise, additional test cycles are required on arrival of a new profile device in the market. The scope of testing for every test cycle is dependent on the evident changes. Some small changes may require testing only key application functionality called as sanity test or smoke test, while other changes may need full regression testing. Mobility - Key Challenges Mobility testing is different and more complex and challenging than testing web apps on desktop due to: 1. Different range of mobile devices: Mobile devices come with varied screen sizes and hardware configurations like virtual keypad, hard keypad and trackball etc. Also, various ranges of mobile devices are available in the market like Samsung, HTC, Apple and Nokia etc. This results in huge diversity of environments which mobile apps face. 2. Device fragmentation and various OS platforms: Various OS platforms are available like Android, Symbian, Windows, Blackberry and IOS. Also there are different versions of the OS like IOS 5.x, IOS 6.x, BB5.x, BB6.x etc. So, providing consistent user experience across diversity of OS and versions is a great challenge. 4

5 3. Different mobile network operators like GSM and CDMA. 4. Diverse mobile App types like Native, Web and Hybrid app. 5. Multiple test interfaces like emulators and simulators. 6. Various test automation tools: There are many open and commercial solutions available in the market. Choosing a single tool for complete mobile test automation requirements is a challenge. Understanding the real time challenges for the mobile testing is quite important: Challenges Multitude of Mobile Devices: Enormous number of mobile devices availability ranging from handsets, to smart phones, to tablets, to ipad and wearable technology provides a huge diversity of environments which your mobile app faces. Device Fragmentation & Various OS Platforms: Providing consistent user experience with diversity of mobile operating systems and versions Diverse Mobile App Types: Native, web, hybrid app Multiple Test Interfaces: Emulators & Simulators. Various Tools: Many open and commercial solutions available in the market Points to Ponder Is testing your application only on few devices good enough? What about if your customers are using other device types and brands? As mobile application has to be deployed across various devices with different operating systems & various versions, testing becomes more challenging. of each app type is different from another as their implementation is different from one another. Benefits of these tools are limited in scope and should be in tandem with in-the-wild testing to fetch best results. Choosing a single tool for entire mobile test automation needs is a challenge. So, what is that ideal solution for your mobility testing requirements? What are different steps you need to take? Let s have a look at some of them: 1. Mobile Test Lab: If your mobile testing needs are more and frequent, a good idea is to invest in organising your own mobile test lab or you can also look for solutions to hire an outside test lab or seek the cloud mobile lab solutions. You need this lab to make sure that you have not only tested your app on a simulator or emulator, but on actual device. Also you have taken care of the variability of OS, fragmentation, device type, memory, screen and other factors which can effect functioning of your application on the mobile. 2. The right testing solution: What you require in this is an IDE so as to reduce scripting effort, a keyword driven procedure so that even manual testers can make use of it effectively, integration with the mobile cloud, tool which should help you to identify objects based on both property and image. Many solutions are available in the market like Appium, Robotium from Open Source, and Eggplant, Perfecto etc. in commercial tools category. 5

6 Mobility - Key Challenges Mobile applications are getting more sophisticated, significantly enhancing the requirement for functional testing. To tackle this, test organizations are searching for alternatives for primary manual testing. Automated testing is a highly efficient method to mobile app testing which can offer significant returns, provided it is put into action using right architecture and tools, factoring in cross-platform difficulties. The following particular types of testing also need to be automated: Cross-platform compatibility testing: This is compulsory as there are increasing number of handsets and platforms. User-experience testing: Most testing companies have limited experience with execution and design of usability tests. Field or network testing: should be done in a geographically distributed environment so as to take into account a variety of network types. Structural challenges: Integrating testing tool sets for mobile into present IT systems is an important hurdle. Test Automation Mechanisms for Mobile Apps A mobile website testing can be automated in any of the following three ways: User-agent based automation: This method leverages user agent add-ons that are present along with popular browsers like Google chrome and Mozilla Firefox. The user agents help render the web content that would show on the device onto a regular desktop browser. The automation of the functionality can be carried out by utilizing popular tools like Selenium, QTP or RFT- each of which supports all desktop browsers. Cloud based automation: Cloud based test automation sellers support web based QA across platforms. Cloud based automated solutions are very effective as they are provided as a unified package. Lab based automation: This method comprise of setting up tools in a test environment and giving power to simulators or physical devices to automate utilizing popular open-source tools like Selenium. Web driver supported device browsers and simulators are best applicable here. Mobile Application Tools/Platform-specific Tools: Device platform-specific tools trustworthiness is a high priority. Platform sellers supply tools like Robotium for Android and ios instruments for IOS to offer this facility. However, to make use of this facility, it demands additional programming skills. 6

7 Mobile application testing tools selection criteria There are many applicable techniques which make testing applications on mobiles more efficient and effective. It is important to ensure functional and non-functional compliance of the applications on devices and mobile automation. It is also important to establish labs for testing the mobile applications. Some basic criteria for mobile application testing comprise the use of multiple browsers, emulators, various operating systems and the unique types of testing needed for GUI, filed and interruption testing. There are around 10 key principles that needed to be considered while choosing the right automation tool. Ten Key principles for choosing the right automation tool It is a good practice to initiate a proof of concept (PoC) on important platforms before identifying a tool set or a tool. One tool might not gain cross-platform automation aim. During evaluation of test automation tools, companies must not overlook the way they integrate with test management environments. One beginning stage is to find reusable scenarios across platforms and cluster them accordingly. This provides the production of an apt, time saving framework. For native apps, select critical test cases which are apt across releases as automation candidates and make sure that there is multi-browser compatibility and cross-platform reusability. Always pay attention to various form factors while identifying test coverage, especially for the mobile web. Consider various connectivity aspects for running automation. Additionally, the below recommendations will be helpful regarding your test automation procedure: Perform feasibility tests for tool, as mobile platforms and technology vary. Choose tools that support both platform devices and simulators, as you can mix and match simulators and devices to optimize automation on various platforms. Goal should be to achieve automation in non-functional places - hardware scenarios and interruption scenarios that comprise battery state changes. Always optimize on platform support: At some places, one or more tools might be needed for performing automation. Find out various device support and versions support. Find the usage of utilities and reusable functions which increase the value of automation. Ensure ROI per every mobile platform and execution duration as part of the tool consideration procedure. Look for chances for early automation, as most mobile projects are agile. Integration should be achieved with application lifecycle management and test management tool. Pay attention for data driven automation support: Iterations in the procedure will enhance coverage and ROI. 7

8 Good candidates for automation are test cases with repeated business logic, data driven tests or functional flows. Data driven tests can be like smoke test to build acceptance testing, synchronization testing, regression testing, and compatibility testing, GUI testing and testing non-dynamic UI components, along with cross-platform functional testing. Conclusion Fig 3: Mobility Eco system Although there are challenges in mobile application testing, but careful selection of target devices, tools that maximise automation and connectivity options can offer a cost effective mobile testing procedure. A right selection of target devices and combining usage of physical devices and simulators can maximise test coverage, eliminating the need to test every feature on each device. The usage of Wi-Fi networks for most of the testing combined with network simulation tools can minimize the cost and complexity of testing of different cellular networks. The best way to reduce the test procedure along with long term costs is to maximise automation. While selecting automation tools, factors like script usability, support for applicable mobile platforms and total cost of ownership should be considered. The challenges of mobile application testing can be effectively addressed by combining the solutions to mobile specific aspects of testing the application along with traditional best practices. TechArcis Solutions, Inc. Quality Assurance l Transformation l Outsourcing