Mobile Application Testing

Transcription

1 White Paper Mobile Applicatio Testig Device ad platform diversity, short release cycles, lack of mature testig tools ad the variety of etwork coectivity optios result i frequet cost overrus ad missed deadlies i today s mobile applicatio testig eviromet. A comprehesive mobile testig strategy that icludes device ad etwork ifrastructure, optimized selectio of target devices, ad a effective combiatio of maual ad automated testig tools to cover both fuctioal ad o-fuctioal testig is essetial for gettig your mobile applicatios to market o time ad withi budget. This paper will discuss various elemets of a effective mobile testig strategy ad suggest ways to optimize testig of mobile applicatios.

2 About the Author Tushar Pradha Educatio Product Maagemet Haas School of Busiess, Uiversity Of Berkeley B.E. (Computer Sciece) VJTI, Mumbai Uiversity Summary of Experiece Tushar Pradha is product maager for Hy5 Test - a automated solutio for testig mobile applicatios. He has more tha 15 years of experiece i the mobile space i product maagemet ad egieerig roles, ad has led a variety of mobile products ad services icludig mobile test tools, cloud services for developers ad device maagemet solutios at Adobe, DeviceAywhere ad Iopath. His primary iterests lie i mobile developmet tools ad ecosystems especially for Adroid ad ios. 2

3 Table of Cotets 1. Executive Summary 4 2. The Challege of Testig Applicatios 4 3. Mobile Applicatio Testig Strategy 8 4. Coclusio 12 3

4 Executive Summary The umber ad variety of cosumer ad eterprise mobile applicatios has grow expoetially over the last few years. Orgaizatios eed to esure that every applicatio meets a high quality bar i order to prevet reveue loss, lost productivity ad damage to brad reputatio. Testig mobile applicatios is differet ad more complex tha testig traditioal desktop ad web applicatios. Mobile applicatios eed to be tested o a variety of software platforms ad versios, o diverse hardware ad form factors, ad uder differet etwork coectivity coditios. Moreover, the rapid pace of mobile OS updates, the frequet itroductio of ew devices ad the customer expectatio of quick upgrades require additioal test cycles. A comprehesive mobile applicatio testig strategy is essetial for gettig your applicatios to market o time ad withi budget. Key elemets to cosider for effectively testig applicatios are Target Device Selectio Create a optimal mix of simulator testig ad physical device testig o differet models to maximize test coverage. Test Automatio Select a effective test automatio tool ad maximize the use of automatio to reduce the cost of regressio testig. Network Eviromet Cosider testig primarily o Wi-Fi etworks ad usig etwork simulatio tools to simulate cellular coectivity ad various etwork coditios. Types of Testig Cosider differet types of testig required (fuctioal, performace, security, ad compliace) The challege of mobile applicatio testig ca be effectively addressed by a test strategy that combies these elemets with traditioal best practices ad processes for testig. The Challege of Testig Applicatios Smartphoe applicatios have experieced explosive growth sice The combied umber of applicatios i the Apple App Store ad Google Play already exceeds a billio. Additioally, every major eterprise is rapidly addig applicatios for iteral use. With so may applicatios competig for users attetio, meetig a high quality bar is essetial for the success of a app. Poor-quality applicatios ot oly hamper user adoptio but also ca cause reveue loss ad irreparable brad damage. Defective eterprise applicatios ca lead to lost productivity. Why Mobile App Testig is Differet Testig mobile applicatios is more complex ad time cosumig compared to traditioal desktop ad web applicatios. The majority of desktop applicatios eed to be tested o a sigle domiat platform Widows. The lack of a similar domiat platform for mobile apps results i may apps beig developed for ad tested o Adroid, ios ad sometimes eve more platforms. 4

5 Adroid 3.1 Adroid 3.2 Adroid Adroid Adroid Adroid 4.1 Adroid 1.5 Adroid 1.6 Adroid 2.1 Adroid 2.2 Adroid 2.3 Figure 1 - Adroid OS Versio Fragmetatio The slow pace of OS updates o Adroid devices (see figure) ad the resultig OS fragmetatio results i the eed to test apps o various versios of Adroid. Ulike the desktop world, where PCs are established as stadardized referece hardware, the wide variety of device form factors (e.g. phoes ad tablets of various scree size) adds aother layer of complexity i testig mobile apps. Device diversity is a especially acute problem for Adroid devices eve the official 1 Adroid device gallery icludes over 60 devices of various scree sizes, resolutios ad form factors. The ease of upgradig apps over the air combied with icreased user expectatios about quicker releases (both for bug fixes ad ew features) result i frequet applicatio releases. Addig multiple major ad mior OS updates o top of this, test teams are cotiuously tasked with testig ew app features or recertifyig the app agaist a ew OS versio. Mobile apps operate i a uique eviromet where applicatio behavior ca be affected by chages i etwork coditios (badwidth chage, dropped coectios), alerts ad otificatios, as well as touch scree resposiveess. This uique eviromet requires additioal testig to esure acceptable app behavior i real world coditios

6 All these factors are resposible for the high cost of testig mobile applicatios. At TCS, we regularly hear from customers that are tryig to reduce the cost of mobile app testig which sometimes exceed developmet cost. Dimesios of Testig Mobile Apps I order to uderstad the complexity of testig mobile apps, it is importat to uderstad various aspects of a applicatio that eed to be tested. Some of these aspects are specific to mobile applicatios while others are applicable for testig ay type of software. The specific types of testig required for each applicatio depeds o various factors such as: The type of applicatio (bakig, gamig, social, or busiess) Target audiece type (cosumer, eterprise) ad volume Distributio chael (e.g. Apple App Store, Google Play, direct distributio) Aspect Fuctioality Performace Network Security Compatibility Coformace Usability Istallatio ad Provisioig Areas / Types of testig User Iteractio Testig Trasactio testig UI resposiveess Trasactio completio time(s) Peak load performace Logevity Network type (Wi-Fi, 2G, 3G, 4G) Impact of Coectivity Issues Data Retetio o device Trasmissio Security Mobile Platform Compatibility (e.g. ios 6, ios 5.1.1, ios 5.1.1) Device Model Compatibility Backward compatibility (with previous app versio) Marketplace guidelies compliace (e.g. Apple App Store policies) Eterprise policy compliace (e.g. prohibited cotet) User Experiece Istallatio process U-istallatio process User provisioig ad de-provisioig 6

7 Applicatio Lifecycle ad Testig Mobile applicatios are upgraded frequetly for a variety of reasos Mobile users expect rapid feature upgrades ad bug fixes As more ad more developers use agile developmet methodologies, frequet releases with icremetal feature updates are becomig commo Applicatios are updated to take advatages of ew versios of mobile platforms (such as ios or Adroid) Each applicatio upgrade warrats a additioal test cycle. Moreover, additioal test cycles are eeded whe a ew versio of a mobile platform (such as ios or Adroid) is released to esure applicatio compatibility. Similarly, additioal test cycles are eeded whe a ew high profile device is itroduced i the market. The scope of testig for each test cycle depeds o the uderlyig chages. Some mior chages may require testig oly key applicatio fuctioality (ofte called smoke test or saity test ) while other chages may require full regressio testig ad testig ew features. Mobile Test Tools Ladscape Effective testig tools are essetial for addressig the challeges ad complexities of mobile applicatio testig. Remote Access Solutios for Maual Testig I the pre-smartphoe era, it was importat to test applicatios o a large umber of devices o major operator etworks. Vedors addressed this eed by providig a remote access (aka device cloud ) solutio that allowed testers to access devices hosted i various data ceters over the Iteret through a browser or cliet applicatio. These services were evetually exteded to iclude smartphoes. Users typically pay for these services based o the duratio of device usage. Today s smartphoe market is domiated by ios ad Adroid. Typical applicatios eed to be tested o 5-7 ios devices (iphoes ad ipads with a mix of latest ios versios) ad Adroid devices (most popular Adroid phoes ad tablets represetig various Adroid versios, scree sizes ad resolutios). Moreover, the majority of testig ca be carried out o Wi-Fi etworks rather tha carrier s cellular etworks. Tools such as TCS WANem eve allow simulatio of cellular etworks (icludig badwidth variatios ad packet drops) over a Wi-Fi etwork. The average hourly usage cost of remote access solutios is $25-$30. With average ios ad Adroid device costs of $500-$600, remote access solutios may ot be cost effective compared to testig with live i-house devices. 7

8 Automatio Tools Automatio tools for mobile applicatio testig use oe of two major techologies: Object based tools (such as TCS Hy5 Test ad Jamo Solutios) achieve automatio by mappig elemets o the device scree ito objects ad maipulatig them. This approach is idepedet of scree size or resolutio ad provides a high degree of script reusability. This ca be especially importat for Adroid devices where variatios i scree sizes ad resolutios are widespread. Some object-based tools (e.g. Jamo Solutios) require chages to applicatio source code while others (e.g. TCS Hy5 Test ) do ot require ay source code chages. Image based or bitmap based tools (such as Perfecto mobile ad DeviceAywhere) create automatio scripts based o scree coordiates of elemets. For example, tappig a butto o the scree is achieved by tappig the coordiates (e.g. x=35, y=40). While this approach is agostic to the type of applicatio (ative, web, hybrid) ad provides useful image matchig capabilities, it typically requires jailbroke or 2 rooted devices ad scripts to be re-writte for each device with a differet scree size or resolutio (due to chage i coordiates). Jailbroke devices are typically icompatible with eterprise MDM security policies ad hece tools that rely o jailbroke devices may ot be suitable for testig eterprise apps. Some of the curret automatio tools are characterized by very high iitial ivestmet ad/or ieffective ROI. Ufortuately, oe of the curret tools provide a cross-platform, cost effective test automatio solutio that combies the beefit of all available automatio techologies. TCS is ivestig heavily i its Hy5 Test test automatio tool to meet this gap. Mobile Applicatio Testig Strategy Various elemets of a effective mobile applicatio testig strategy are cosidered i this sectio. Recommedatios i this sectio are based o TCS extesive experiece i developig ad testig scores of mobile applicatios. Selectig Target Devices Target devices for testig a mobile applicatio should balace the eed to use a represetative sample of the expected device populatio with the eed to optimize duratio ad cost of testig. 2 Jailbreak (ios) or rootig ( Adroid ) is the process of hackig ito a device to istall ad execute applicatios prohibited by OS / device vedor. 8

9 Simulators vs. Physical Devices Use of device simulators ca be highly effective i the early stages of product developmet whe features are uder developmet. Simulators are useful to familiarize the test team with various applicatio features. Figure 2 illustrates various coditios uder which testig with simulators ad real devices ca be effective. Simulators ca be effectively used for testig basic applicatio fuctioality. Testig o physical devices is essetial to uderstadig applicatio behavior such as touch respose ad user experiece. Stable, defect-free (based o simulator testig) features ca the be tested o physical devices. Device usage ca be optimized by distributig test cases across various types of devices. Note that this strategy requires that the test tools beig used support testig with simulators as well as devices. Device Testig (i-etwork) Recommeded oly for small subset of etwork-depedet use cases Device Testig (O Wi-Fi Networks + Network Silmulatio Tools) Use for stable features Distribute features across device models ad form factors Importat for UX Testig Combie with WANem or similar tools for simulatig etwork coectivity coditios Simulator Testig Ideal i developmet phase ad for feature demos Suitable for Basic Fuctioality Testig Figure 2- Coditios for Testig with Devices / Simulators ad Differet Coectivity Optios Device Model Selectio Key factors i decidig specific device models for testig are OS Versio Applicatios should be tested o all major OS versios i the curret istalled base. For Adroid devices, OS distributio of various Adroid versios provided by Google at ca be used to determie the mix of Adroid versios to be targeted. 9

10 For ios devices, previous experiece shows that 90% of ios devices ca be expected to be upgraded to the last major versio (see For example, whe the latest available versio is 5.1.1, over 90% of devices ca be expected to be o versio 5.0 or later. Form Factor Applicatios may behave differetly (especially i terms of user experiece) o smartphoes ad tablets. If the test applicatio supports both smartphoes ad tablets, it should be tested o both form factors Display Desity The scree size ad scree resolutio of a device (aka display desity) ca also affect applicatio user experiece (e.g. look ad feel of UI, rederig of web pages). Device models selected for testig should iclude a mix of differet display desities. A variety of other factors (keyboard, CPU type, memory) ca affect specialized applicatios that rely o specific device features ad should be take ito accout whe ecessary. Selectig specific ios device models ca be relatively easy due to the small umber of device models available. A commoly used ad effective method for selectig Adroid test devices relies o selectig the most popular devices based o market data ad the adjustig for the above parameters. TCS maitais a recommeded list of target devices for testig that is periodically updated based o market chages. Whe testig eterprise applicatios, a user survey or device model data from iteral systems ca be a good source of iformatio to determie the mix of target devices for testig. Coectivity Optios Almost all applicatios rely o etwork coectivity to provide useful fuctioality. I test eviromets, Wi-Fi etworks ca be easy to set up ad ca be cost effective, ulike cellular coectivity that ca be both ureliable ad expesive. Network simulatio tools such as TCS WANem provide a cost effective ad easy to use optio to simulate cellular etworks usig a WAN/Wi-Fi coectio. These tools ca be used to simulate various etwork speed/badwidth optios (2G, 3G, ad 4G), coectivity issues, ad badwidth variatios. O-locatio testig o specific cellular etworks should be cosidered oly for specialized applicatios that rely o carrier features such as free usage allowace or two way SMS. Maual vs. Automated Testig Automated testig provides a mechaism to cosistetly repeat a test procedure ad verify applicatio results. It ca be effective both for regressio testig as well durig developmet (to esure that ew features are ot resultig i uiteded chages to existig features). However, automatio requires a sigificat iitial ivestmet (i a test tool as well as scriptig) ad the ROI is realized whe the same automated test is executed multiple times with egligible icremetal cost. Therefore, it is importat to judiciously select cadidate test cases for automatio. 10

11 The followig guidelies are recommeded for automatio of mobile applicatio testig Features that are expected to chage i the ear future (especially i terms of UI flow) should be avoided for automatio. Some test cases may be extremely expesive to automate due to techical challeges or uique use cases (e.g. hadlig real time data from multiple sources), egatig the beefits of automatio. Such test cases should be avoided for automatio. Every possible test case should be automated uless it meets the above criteria for avoidig automatio. To achieve the maximum beefit from automatio, all automatable test cases from a release should be automated before the ext release. Key use cases for automatio i mobile applicatio testig are Verifyig applicatio compatibility whe a ew ios, Adroid (or other applicable) OS versio is released Verifyig backward compatibility whe the applicatio is upgraded Selectig a Test Automatio Tool As described i sectio 1, the lack of a ideal test automatio tool makes the selectio of a test automatio tool a complex decisio. The followig table lists key criteria that should be cosidered whe selectig a test automatio tool. Criterio Multi-Platform Support Script Reusability Areas / Types of testig Cosider your curret ad future target platforms ad esure that the tool ca support them. Object based tools such as TCS Hy5 Test provide a high degree of script reuse. Image based tools geerally have lower script re-use resultig i higher cost. Jailbreak Requiremet Source Code Chages Lead Time for New OS versio / Device If the tool uses jailbroke or rooted devices, it may ot support latest OS versios ad may be icompatible with MDM policies required for eterprise apps. Sharig source code may ot be always possible ad chagig the source code icreases the risk of missig some defects. How soo ca the tool support ew ios/adroid/other OS versios? 11

12 Criterio Test Workflow Results ad Reports Itegratio Capabilities Deploymet Price Service ad Support Areas / Types of testig Does the tool provide a easy to use workflow? Does the tool archive results ad provide reports? Ca the tool itegrate/iteroperate with other test maagemet/bug trackig/build maagemet systems? What deploymets optios are available? Ca it be deployed ihouse/i a cloud model? How much does it cost? What services (e.g. script developmet) ad support are offered? Coclusio Despite the challeges i mobile applicatio testig, careful selectio of target devices, coectivity optios, ad tools that maximize automatio ca esure a cost effective mobile testig process. A optimal selectio of target devices ad usig a mix of simulators ad physical devices ca maximize test coverage without the eed to test every feature o each device. The use of Wi-Fi etworks for the majority of testig i combiatio with etwork simulatio tools ca reduce the cost ad complexity of testig o various cellular etworks. Maximizig automatio is a effective way of expeditig the testig process ad reducig log term testig costs. Factors such as support for applicable mobile platforms, script reusability ad total cost of owership should be take ito accout whe selectig automatio tools. Combiig the solutios to mobile specific aspects of applicatio testig with traditioal best practices ad testig processes ca effectively address the challeges of mobile applicatio testig. Refereces [1] Adroid Versio Distributio: [2] ios Versio Distributio: 12

13 About TCS Mobility TCS Mobility delivers best i class mobility services ad solutios with complete mobility lifecycle cosultatio ad developmet service alog with customizatio o top of pre-built solutios to provide the best i class beefits to our customers. Our passio for providig the very best ad comprehesive mobility services ad solutios to our customers is realized through our deep expertise i mobility gaied through the experiece of a strog team that has a career log experiece i mobile techologies ad a dedicated mobility user experiece desig team that is committed to leveragig the uique ative capabilities of each device platform. Cotact For more iformatio about TCS Mobility cotact [email protected] Subscribe to TCS White Papers TCS.com RSS: Feedburer: About Tata Cosultacy Services Ltd (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig ad TM assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at IT Services Busiess Solutios Outsourcig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2011 Tata Cosultacy Services Limited TCS Desig Services I M I 10 I 12