Mobile Automation: Best Practices

Transcription

1 contents A U T H O R : M a n i s h B h a g a t Mobile Automation: Best Practices Abstract... 2 Executive Summary... 3 QA Challenges of Mobile Application Testing... 4 Mobile Application Automation Challenges... 5 Approaches for Mobile Automation Testing... 7 Automation Framework Design & Considerations Automation Governance Conclusion References Mobile Automation Best Practices Page 1 of 14

2 Abstract Mobile applications evolved from previous Desktop/Web applications which provide faster communication, organize and manage work in few taps are one of the powerful means of new-age communication. A recent research report from Forrester has predicted that half of all corporate-issued devices will be mobile by Mobile application for business is most affordable and effective way to target the desired market segment. Every Corporate is aiming to capture the market with an aim to reach its target audience as fast as it can, which has resulted in the development of mobile applications at an irrepressible pace and has positioned Mobile Application Testing as a challenging activity in a software lifecycle. Some of these challenges are device and platform diversity, short release cycles, immature testing tools, variety of network dependencies, results in cost overruns and missed deadlines. The Mobile testing strategy that includes streamlined delivery, optimized selection of devices, and effective combination of manual & automation testing to cover both functional and non-functional testing is considered essential. This paper will discuss various elements of effective mobile automation testing strategy and will also cover optimized automation model. Mobile Automation Best Practices Page 2 of 14

3 Executive Summary The number of mobile users has increased exponentially in recent years. Today, mobile usage is not limited just to making calls and sending text; in fact, it has become a powerful tool of communication and information sharing. Today every industry is trying to connect with consumers through the medium of mobile, and making big investments in the mobile technology. Hence, there is huge potential in developing new applications and better scope in maintaining the quality of mobile apps. Testing Mobile Application is different and more complex than testing traditional desktop /web application. Mobile applications need to be tested on variety of platform versions, diverse hardware and screen size, under different network conditions. Frequent updates of Mobile OS (operating system) eventually needs updates for mobile applications, which is responsible for additional test cycles to meet the customer expectations. This whitepaper describes QA(Quality Assurance) challenges in mobile application testing and discusses mobile testing strategies, mobile testing types and automation tools. Mobile Automation Best Practices Page 3 of 14

4 QA Challenges of Mobile Application Testing The goal of any application testing exercise is to understand the quality and performance of features been offered. There are, however, some critical factors which make mobile testing a far greater challenge than desktop and web-based testing. These factors are as follows. Device Variation When the application is designed to run on multiple OS, on various screen sizes, it becomes challenging for the QA team to guarantee if the application works well on given devices, it will also give 100% on another device even if it s from the same family. Mobile testing tools availability Various desktop testing tools available currently in the market are still making huge investments in making their tools compatible with Mobile devices. However, we cannot say that these tools are 100% compatible with mobile devices/applications without tweaking them. Need skilled QA specialist in Automation testing The quality team needs a skilled QA specialist who understands mobile device architecture as well as OS-specific features in order to find the bugs quickly,to access the automation tools available and apply to application according to its needs, to further validate the application quality as quickly as possible. Cloud Computing Cloud computing offers web-based environment, where testers can deploy, automate and test the mobile application on diverse devices. This approach helps the cost of project under development. Following are common problems faced by testers. Subscription model High Cost Internet Connectivity issues Automation is OCR-based, time consuming & reduces test coverage Automation cannot deployed outside framework Mobile Automation Best Practices Page 4 of 14

5 Mobile Application Automation Challenges Application Challenges Smaller screens bring out different considerations and challenges in automation because of different screen resolutions for different device patterns. Different Platforms have different look & feel and interactions. Mobile apps are tightly coupled with specific Mobile Platforms. Technical Challenges Simulating Touch Screens, (Tapping) which is new way of interacting with application rather than keyboard and mouse inputs. Lack of Object Repository identification tools. Source Code dependency Building common automation framework for different mobile platforms. (Android, ios, Blackberry, Windows). Network Challenges Applications need to be tested on different Network types (eg. GSM, GPRS,3G,4G,WiFi,WIfi Max). Connectivity across geographies Multiple Network Operators with customized network features. Business Challenges App needs to be supported for different platforms versions, different devices available in markets. Due to shorter iteration cycles features cannot be tested on multiple devices and version platforms. Test Lap provisioning with Real Devices could be expensive. Test Challenges Test Combinations platforms and versions End-to-End automation API to device UI, external database and systems. Testing with Physical Devices, Networks. Tool Limitations Application Code dependency Lack of Object Recognition Techniques Tools limited to execution of test specific to platforms. Lack of debugging and logging support Limited to specific application type (Native/Hybrid/Web) Lack of Detailed reporting formats. Mobile Automation Best Practices Page 5 of 14

6 Supports Running test on device rather than Desktops Tools integrations with existing Framework or New Framework. Mobile Automation Best Practices Page 6 of 14

7 Approaches for Mobile Automation Testing Successful implementation of Mobile Automation can be achieved by selecting precise strategy within correct timeframe to realize expected ROI (Return On Investment) using automation testing. Device Selection Strategy This strategy plays a key role as in the real world, we cannot test applications on each devices that are in the market, and hence, we need to select devices using RBT(Risk-Based Testing) methodology that would reduce the effort of executing test cases on each device and also the investments made in collection of actual devices or cloud-based subscriptions. Tool Selection Criteria Current Trend of Device Successful implementation of mobile automation depends on the approach in selecting a right automation tool. Some of recommendations for selection of right tool are as follows. The tool should - Support application types (Native/Hybrid/Web) Support major diverse Mobile Platforms (Android/iOS/Windows/BB) Aim for automation in functional & non-functional areas interruptions and hardware scenarios that include memory and battery state changes. Depending on need should support executions private/public Cloud. Support Data-driven testing. Ensure ROI per each mobile platform and duration of execution as part of tool consideration process. Look for early automation scenarios, because most mobile projects are Agile. Allow scripting test without having access to Application Source code. Support remote executions. Automation Environment Automation environment and setup depends on type of testing approach been decided. Two major automation testing approaches are: Device Lab Testing : All the devices on which automation testing needs to be performed are setup in premises and by using Device Management tools, testers can access those devices through secured network. Executions are rapid, good control over application and devices can be achieved. Major mobile automation tools(open source/proprietary) can be used for automation, hard to maintain device bank, required secure network setup. Mobile Automation Best Practices Page 7 of 14

8 Cloud-based Testing: Cloud Test automation vendor provides web-based automation platforms that can be accessed from anywhere in the world with good internet connectivity. They also provide keyword-driven web-based framework, tester without any scripting language can write the automation scripts. Easy to script, devices bank can be easily maintained, works on both OCR as well as native object identification methods, slowness is automation executions, full capability of automation cannot be achieved, subscription model test lab setup require huge investments. Mobile Automation Methodology Automation Methodology is no different from the way we used in Desktop Application; the process remains same and a few more components are added that are required for Mobile Automation. Requirement Analysis: Initial phase identifies the requirements, analyzes and categorizes them based on the application type (Web App, Hybrid App, Native App). Identify Automation Scope: Based on the requirement categorization, identifies the scope of each requirement. Mobile Automation Best Practices Page 8 of 14

9 Device Identification: Identifies different device types, versions and platforms that would meet the requirement criteria. Tool Identification: Based on devices, requirement categorization and scope, identifies an Automation tool that will best fit in terms of devices and scope of automation. POC: Filters the best tools by performing POC(Proof of Concept) using different tools that would proof that requirements can be automated. Go-No Go with framework design is dependent on the POC developed. Framework Design and Development: Designs the framework architecture, and evaluates it against the requirements that are identified in the initial phase. As far as setting up the development environment is concerned, a version-controlling setup starts here followed by development of reusable components, framework and plugins. Test Automation Strategy: This strategy can be carried out in parallel to framework design. Test Automation Strategy should state a technical scope for automation of the requirements, test environment setup details, scope for running tests on Emulator as well as device, and any network emulator required for n/w packet simulation, types of testing. Automation Test Design : Based on priority of manual test cases, starts writing test scripts, re-usable functions and reviews the test scripts as per test governance mentioned in later sections of whitepaper. Script Executions: Once the design is completed,it integrates all modules and executes as single suite that should cover all critical business flows. Test Results Reporting: Test results are generated and can be automatically uploaded through Test Management tools. Mobile Automation Best Practices Page 9 of 14

10 Automation Framework Design & Considerations Usage of standard design patterns and OOP(Object-oriented programming) concept allows easier decoding of changes and provides robustness to your Automation Framework. Encapsulates the actual implementation of core automation components and provides generic wrappers. Test data maintenance should be simple and should allow user to configure different sets of data. Identifying the resource for maintenance of Test data plays an important role in the Automation frame work design. Test data identified should be reusable. Build Pluggable components. Integration of static code analysis to identify early bugs in Automation framework.dedicated test lab environment for automation Test Scripts, Test Data and Framework Level configurations should be isolated from the framework. Object Repository: The Object Repository contains the objects for GUI(Graphical user interface) based validations. Any changes in GUI objects is modified at central location, thereby avoiding rework in automation scripts. Major objects for mobile application can be native objects, web objects or OCR-based object identifiers. Function Library: Plays important role in basic building of framework and defines common set of reusable functions. Function Library must contain generic set of reusable functions that are usable across the applications as well as business functions specific to application. Driver Library: Driver scripts should be written to support different tools in the framework. Different driver scripts should be written to integrate automation tools, if any new tools should be added framework, the user has to write simple driver scripts to integrate to framework, hence rest of the scripting is not affected. Mobile Automation Best Practices Page 10 of 14

11 Test Scripts: They contain actual code of logic to perform validation of business or functional scenario. Scripts should be properly designed and commented and closely related to manual business or functional scenario. Test Data: Application-specific data used to test application under test. This data should be collected in single format and structure that passed as an input to automation test scripts, wherever possible try to regenerate test data rather than manually entering as an input to scripts during run time. Recovery Scenarios: Define and guarantee the robustness of test scripts. They are implemented to handle unexpected exceptions where scripts executions may come to halt. Recovery scenarios help the automation scripts flow to follow a definite execution flow or path whenever unexpected expectations arise during the script executions. Configuration Scripts: Global settings and applications access parameters are maintained in configuration scripts and files. If an changes in environment setup occur, we may change the parameters accordingly instead of making changes in scripts and data. Reporting Mechanism: Reporting and Logging mechanism should be configured so that test results that are generated during and after execution are easily accessible and understood. It has becomes very important to generate and debug information at various points in test scenarios as this information can be used in finding bug areas and easily fix them. Mobile Automation Best Practices Page 11 of 14

12 Automation Governance Automation Governance is a critical aspect of Test Automation Framework design. Automation governance will ensure that all the areas of testing methodology are kept consistent, regardless of individual developing of the scripts, or executing the test. Key automation governance parameters are: Test Suite Structure Script Structure Script Versioning Coding Standards Script Naming, object naming conventions, identifying mandatory properties, should be under governance for maintainance Error and Exceptions handling Independent testing of automation framework Framework for Test Data creation and management Mobile Automation Best Practices Page 12 of 14

13 Conclusion The Principles and success factors of test automation for mobile applications are much different than those traditionally applied to conventional applications. The attributes of device environments, tools sets and coverage add multiple layers of complexity. It requires insight and a customized approach to build an automation strategy. In the end, it s the entire turn-key that counts and that we have been successful in achieving it. It is an intresting challenge merging of multiple technologies and platforms. Watching automation gets executed on real devices, which gets automatically triggered from a remote build. There are many challenges in the way; it is prudent to overcome those challenges wisely, offer memorable solutions and provide motivation to work in a challenging environment. The key to realizing the various benefits was to understand the concerns of the customer and the QA team and then design full-proof solutions that would satisfy major needs. Abbreviations and Acronyms UFT GSM/GPRS Appium Selenium OCR Unified Funtional Tool Global Systems for Mobile/ General Packet Radio Service An opensource test automation tool An opensource web test automation tool Optical Character Recognization References mation.html Mobile Automation Best Practices Page 13 of 14

14 About the author(s) Manish Bhagat works as Senior Tester at L&T Infotech. He has over 6 years of extensive experience in Automation Testing with expertise in Enterprise Mobility. About L&T Infotech Larsen & Toubro Infotech Ltd. (L&T Infotech), one of the fastest growing global IT services company, is ranked by NASSCOM as the 8th largest software & services exporter from India and among the top 20 IT BPO employers in It is a wholly-owned subsidiary of USD 14.3 billion Larsen & Toubro (L&T), India s largest technology, engineering, construction, manufacturing and financial services conglomerate, with global operations. L&T Infotech provides end-to-end IT solutions and services to Banking & Financial Services; Insurance; Energy & Process; Hi-tech & Consumer Electronics; Utilities, Engineering & Construction; Consumer Packaged Goods, Retail & Pharmaceuticals; Auto & Aerospace; Media & Entertainment; Healthcare; Plant Equipment & Industrial Machinery and Travel & Logistics industries. Headquartered in Mumbai, India, L&T Infotech delivers IT-enabled business solutions to its clients, leveraging its strong domain-expertise and in-depth technology know-how, through its service lines including Testing; Mobility; Infrastructure Management Services; BI/DW; SAP; Oracle & Microsoft; Enterprise Integration; and Manufacturing Execution Systems. L&T Infotech is differentiated by its three-pronged value proposition encompassing Businessto-IT Connect, Engage the Future and Execution Excellence. For more information, visit us at or us at [email protected] Mobile Automation Best Practices Page 14 of 14