Mobile Testing Automation

Transcription

1 Mobile Testing Automation HARNESS THE MAGIC OF MOBILE TESTING AUTOMATION An InfoStretch White Paper August 2014 Elevating confidence 3200 Patrick Henry Drive, Suite 250 Santa Clara, CA InfoStretch Corporation. All rights reserved

2 TO SUMMARIZE The need for automation is imminent. Why automation? Release cycles need to be shorter and faster to meet market demands. There has to be structured method in dealing with fragmentation, considering the proliferation of devices phones, tablets, phablets, wearables. In this paper, we cover the parts of testing that can be automated, what tools can be used to do so and our approach Need a cost-benefit analysis of automation ROI? We ve got you covered there, too. Read on InfoStretch Corporation. All rights reserved 2

4 Software testing has evolved in a new dimension with the entry of mobile devices and is changing the status quo of the testing game. Mobile applications have to be made available quickly in the market, and tested effectively with shorter QA cycles. Companies today want to rapidly push applications in the marketplace. A traditional QA cycle which would take 3-6 weeks is now being condensed to just a few days. If the application does not meet the quality standards end users expect, more cycles will be spent on fixing those issues, negatively impacting the budget and the brand. Mobile testing is challenging and what makes it so are the seemingly endless combinations of devices, OS s, carriers, and networks. As it is an immature market, there is a lack of established best practices or industry standards around it. Mobile applications have become increasingly complex and to test the performance of the app on the server, device and network can be quite inundating. There are very few tools in the market to perform automation, and additionally, the automation strategy needs to be driven by specialists who can develop custom scripts for associated business need. Traditional manual testing cannot feasibly cover all of the bases especially when the app to market life cycle has to be short. There has to be some mechanism to test an app quickly and effectively, without wasting cycles. As a result, mobile automation has become a popular choice in overcoming the challenges in traditional mobile testing in order to maintain coverage InfoStretch Corporation. All rights reserved 4

6 Automation is essential and it needs to be approached with a plan. Evaluating the what, when and how to automate becomes important. What Regression tests can be easily automated to ensure that the basic functionality is maintained from the previous release. OS specific test cases can be defined and that case can be executed in multiple environments for that OS. Automation works best with features that are stable and are not undergoing changes. Not all features can be automated and require manual testing. So, there needs to be a balance between manual and automated testing. When QA managers must choose the right time for automation. Automating very early in the applications lifecycle might require too many changes in future and can significantly raise automation costs and effort. Only after manual testing has established with repeatable processes and stable test cases, should automation be considered. How There are many automation tools in the market which provide the automation engine through a public or a private cloud platform. Some very well-known automation tools are Perfecto Mobile, Device Anywhere/Keynote, ZapFix, Eggplant, Selenium, Robotium and Jamo. Here s a list of automation tools that can be used for depending on the platform. Note: This is a basic list, and new tools get added frequently. InfoStretch Selenium Test Automation Framework InfoStretch s Test Automation Framework For Mobile Web, Native Apps, and Hybrid Apps Easy Integration with TestNG, Selenium, Autoit, and ReportNG Enterprise-Grade, Cloud-Based Mobile Application Quality Management Platform, with a Suite of Mobile Testing and Monitoring Products For Mobile Web, Native Apps, and Hybrid Apps [PROPRIETARY TOOL] [SERVICE PARTNER] Write and Execute Automated Acceptance Tests For Hybrid Apps, Android and ios Native Apps Write Functional, System, and Acceptance Test Scenarios For Android Native and Hybrid Applications Open Source Cross-Platform Functional Testing Tool for SOA and Web Service Testing Functional and Regression Test Automation for Software Applications and Environments Mobile Test Automation by Integrating Perfecto Mobile 2014 InfoStretch Corporation. All rights reserved 6

8 It is necessary to understand the overall scope and strategy in order to identify what to automate and justify efforts and ROI. Next is to identify the right tools and technologies for automation and define reusable test assets. Parameterizing select test cases by leveraging reusable assets to enable data driven test automation is essential. Execution of regression test cases and enhancements based upon the objectives and goals in addition to maintenance of reusable frameworks assets and automated tests with changes in application is critical InfoStretch Corporation. All rights reserved 8

9 Costs Prior to quantifying the benefits of test automation, it is important to understand the cost of automation. It is key to begin the conversation by discussing the different variables affecting the cost of automation, granted that the environment for automation has already been set up. The cost of automating test processes is tangible however the net present value also includes many intangible factors. The best approach is to determine, with as much precision as possible, what the hard-dollar costs are and then compare them to the hard and soft dollar benefits of automating the test efforts. Hard dollar costs to consider in an ROI calculation include: ACQUISITION COSTS DEVICE COSTS LABOR COSTS TRAINING COSTS Test automation tools can be expensive, and are usually employed in combination with manual testing. In the long term, it can become cost-effective, particularly when used repeatedly in regression testing. The cost of mobile handsets required for testing. With the Android fragmentation, there are many variations in the devices with manufacturers such HTC, Samsung, Motorola, Sony embedding Android into their devices. Factor in the cost of trained personnel to script test cases or build manual tests. Be sure to include the costs of recruiting, hiring, paying, and retaining skilled QA engineers. Depending on which test automation product is selected, the user may need considerable training to become proficient at writing scripts for automated testing. Alternatively, companies may elect to hire professional services firms to handle the initial automated test creation. When weighing the potential rewards of automation, it is important to consider intangible benefits, such as higher morale and job satisfaction among testers, improved customer satisfaction and loyalty, and an enhanced reputation for reliable software among end users InfoStretch Corporation. All rights reserved 9

10 Among many other intangible benefits, automation can help to improve employee morale and productivity, because it can take on the repeated tests, allowing manual testers to focus on more challenging tasks. Discovering bugs early in the testing cycle is another intangible benefit that cannot easily be quantified. The later in the development cycle defects are found, the more expensive they are. A defect identified in a production environment can be greater than 100 times more expensive to fix than the same defect identified earlier in the design phase. Automation is the key to improving the speed, accuracy, and flexibility of the software testing process, enabling companies to find and fix more defects earlier. Detailed Benefits of Automation: REDUCED COST OF TESTING DUE TO EFFORT REDUCTION REDUCED TIME TO PRODUCTION Hours to create automated scripts Hours to maintain and enhance script Reduction in Engineer's time requirement due to automation for test execution Hours for 1st cycle testing Hours for subsequent cycles testing when automation is available First year person-hours required Subsequent year person-hours required First year cost of testing Subsequent cost of testing per year Elapsed testing time Automation time reduction (1st year and subsequent years) Production rate Total production recaptured Margin per unit IMPROVED PRODUCT QUALITY INTANGIBLE - QUALITATIVE BENEFITS Average number of defects in product Defect-finding efficiency Average cost of fixing defects prior to and after release Cost of fixing defects pre-production Cost of fixing defects post-production Annual cost of fixing defects Increased consumer confidence Increased ability to support the change in application functionality Higher level of product quality and reduction in support costs 2014 InfoStretch Corporation. All rights reserved 10

13 Selecting the Right Test Cases TOOL AND RESOURCE CAPABILITY STABILITY/CHANGE CAPABILITY/COMPLEXITY/PRIORITY System/Peripheral Interaction Interaction between multiple apps / OS-App / Multi-Domain Location aware, peripheral, battery life, system alerts, etc. Best Practices for other GUI Automation apply UI components that will change less Clear understood business processes Less graphically intensive, more data and business scenarios Medium complexity test cases: High Low Smoke: High Medium Low The Automation Architecture BROWSER OPEN BROWSER GO TO SIGN ON MAKE PAYMENT TEXT VERIFY CONFIRM PAYMENT SIGN OFF QTP/SELENIUM MAKE PAYMENT TEST CASE ADDRESS ATM SEARCH BROWSER OPEN SELECT ACCT VIEW SCHEDULE PAYMENT SYNC TEXT QUICK LINKS NICKNAME BROWSER GO TO SIGN ON SIGN OFF ZIP ATM SEARCH MAKE PAYMENT SYNC IMAGE CONFIRM PAYMENT FRAMEWORK COMPONENT LIBRARY ios ANDROID HTC ARIA ANDROID HTC DROID BB TORCH BB 8520 LG CU620 LG VX9000 DEVICES DEVICE IMPLEMENTATION FOR SIGN ON (10+) 2014 InfoStretch Corporation. All rights reserved 13

14 In Conclusion Quantifying the benefits of test automation is a complex task. In organizations where decisions are made purely on the expected ROI, managers must keep several criteria in mind while making computations. In many cases, the intangible benefits of automation will supersede the tangible benefits. Hence, ROI computations might not reveal the real value of automation. The best results are obtained when costs and benefits are computed in terms of opportunity costs related to test automation and manual testing. This will provide decision makers the visibility into some of the intangible costs and benefits of automation. Test Automation has worked its magic for many enterprises. Contact us to see how it can for you InfoStretch Corporation. All rights reserved 14

15 ABOUT INFOSTRETCH InfoStretch is a leading provider of mobile and enterprise QA services and solutions. Our offerings range from enterprise QA, mobile application development, testing, and automation to certification and sustenance. We ve been providing expert solutions and services to various large and strategic players in the mobile ecosystem including enterprises, carriers, app stores, VAS, and ISV s to actualize their mobile and QA strategy very successfully. We offer the most cost effective, secure, and scalable models to service our clients across various verticals such as banking, healthcare, education, and entertainment to name a few. Our innovations have allowed us to provide an SLA of 24 hours and the ability to scale and test over 2,000 apps a day. CONTACT US SANTA CLARA OFFICE InfoStretch Corporation 3200 Patrick Henry Drive Suite 250 Santa Clara, CA GENERAL INQUIRIES e: info@infostretch.com p: (408) f: (408) All registration marks are the property of their respective owners InfoStretch Corporation. All rights reserved