Mobile testing challenges

Mobile testing challenges Smart ways to do handheld testing Written by Ankit Agarwal (ankit.agarwal@hexaviewtech.com) Shobhit Gupta (shobhit.gupta@hexaviewtech.com) C-56/31, Sector-62, IIIrd Floor, Noida, Uttar Pradesh -201301 Page 1

Abstract Technology today has become an essential part of every human being s life. With technology improving at each second at a lightning fast pace, people want the best return for their money. Gadgets have become more than just a source of entertainment or business. Mobile phones are now not just a means to talk or text; with so many platforms (Symbian, Android, iphone) and so many applications, today we find endless kinds of handsets, touch screens, social networking apps, Emails, etc. all residing in one mobile phone. The market has become very competitive as a result, and just like consumers want the best, the manufacturers compete to make the best. A major challenge is posed to the manufacturers in the context of testing these devices and their apps. To come out with the best, it s important that the applications are tested on different platforms. The available tools today do not provide cross-platform support. Significant testing like regression, functional and automated testing are still need to be addressed. A tool which can be used to test mobile applications that run on different platforms is surely very much needed. Also, the tool should be able to do a fair amount of automated testing of the applications across different platforms. To achieve a tool that can test mobile applications across different platforms, we need to understand the application s GUI and functionality across different platforms. For example, let s take a simple application that shows the image of a person on entering a name in the textbox field. Now different platforms might give different look and feel, but to test the text entry field, a threshold can be set with respect to the pixel strength of the text box (from the image). Once we have identified our area of testing in the application, we can go ahead and give different values for the said field. Now this test is usable across different platforms, as we have generalized the functionality across different platforms. All we need is identifying the specific area for test in any application. This tool can be effectively used for substantial amount of functional testing, and a fair amount of automated testing as well, across different platforms. Page 2

Introduction Devices are exploding, applications are exploding, and technologies are evolving very fast. How does a test engineer keep a track of the fast changes and make sure that applications developed still have the quality that one is used to seeing in the desktop space. This paper is an overview of the effort done by Hexaview Technologies in the mobile testing space and is a combined effort of two factors: a) Study of the current handheld market and its pain-points b) Proposing generic solutions which are scalable to the challenges posed by explosion of devices Based on our experience with testing handsets and devices, we soon realized that doing testing the manual way won t solve the problem; one would need to go under the hood and develop a tool which reads user actions and can repeatedly perform them on various handsets. Our Aim Is to create a collection of tools to record the set of actions performed by tester manually and playback it whenever required. The idea is to create a tool to find out GUI bugs quickly in order to make Android applications more reliable. Why Android and not iphone The answer is simple: CHOICES! IPhone hardware and software is developed and maintained by one company Apple whereas ANDROID OS is used by many mobile manufacturing companies and offered by any carrier. It s a fact that currently there is more apple apps out there than Android Apps but the rate of growth is definitely in Android s favor. For example in October, 2010 the difference in the number of apps on IPhone AppStore and that on Android Market was 185,000 whereas in March 11 it reduced to about 100,000. Android App Development is open source; any phone manufacturer can use Android without expensive license fee. Because it is Open, manufacturer can modify Android without restriction, allowing it to fit the device they are making - total freedom. It s the future: Your car dashboard, refrigerator can run on Android OS. Therefore the reach is quiet high. The Need Need for a new tool for automating the testing of Android applications is because the android application structure is very different from conventional web-based/desktop applications. For example: Android applications are centered on activities, broadcast senders & receivers, content providers. This is different from a 3-tier web based application or event based desktop GUI application. Also, due to variety of devices having different screen specifications, different power consumptions, different processors & memory, it becomes difficult to consider all these different specifications before starting on building an app. Usually, an app starts being built on emulator with standard specifications and later imported/tested on various devices. Therefore it becomes more & more important to create/use an automation framework which performs the test on various devices (with no or minimal changes in scripts) and can produce results faster. Page 3

Other issues pertaining to Android platform, is that the OS itself is being changed frequently and Google is coming up with new features every day, in order to compete with iphone, BlackBerry & Windows. Now in order to verify your application on all new upcoming OS versions, automation tool is the key Objectives a.) To be able to create the script once (either writing manual script or using recording facility) and run it across multiple devices of different specifications. b.) To be able to run this tool remotely, i.e. the tool is there on a workstation residing at one location (like India) and the device to be test upon is at another location (like US, Europe). c.) To be able to perform automated UI and functional test cases. d.) Perform regression testing by comparing screenshots with the set of screenshots which are correct. Main Things to test a.) Various Activity lifecycle states The application states are saved in onpause() and ondestroy() methods and recreated in oncreate(bundle). So we should check that the state is saved and retrieved correctly under all circumstances. b.) File System& Database usage Saving data in database or file system is common activity in any android app. Content Providers is used for storing and accessing data. They should be checked and also we can do some lower level validation by using Android mock objects. c.) Device characteristics Android app should be checked against various device characteristics such as i. Screen size ii. Screen resolution iii. Screen density iv. External storage v. GPRS/WIKI/Network Connections vi. Keypad/trackball/touch screen d.) Performance Tests - To measure performance characteristics of components in an android application. The performance tests are performed mostly on actual devices because emulator cannot provide the correct results. e.) Testing memory leaks To get memory & process information related to each activity within the application. Technologies Used The tool is being built using all open-source technologies. It uses Python/Jython script on the host machine which captures all screen events like button click, touch events etc. The tool can be used to test in any of the following 2 ways a.) Test with only the apk file. b.) Test using the application source code. Tool in Action a.) Connect the Device on which app is to be tested. We can test on emulator as well b.) Run the recorder script to present an application screen where user can perform manual testing of application one time. Page 4

c.) Once testing is complete for each activity, a script is generated which is saved and can be run anytime later. d.) Please ensure to include delays in between the various events while recording. This is needed to ensure that proper time is provided for the event to complete (i.e. event listeners are called properly and have completed their action) The script created is for specific device/emulator configuration. What user also does using this tool is the following: a.) Record a script for one emulator configuration b.) Then depending upon the various parameters for each device, like screen size and resolution, the tool can automatically adjust the script to perform more or less correctly on this new device. Therefore the same script can be run on multiple devices with minimal input parameters. Shortcoming There are certain shortcomings due to the wide variety of mobile device hardware and carrier limitations. a.) Like, in case, there is a drastic variation in mobile device processor speed and/or network speed/connection then the script might not function properly as the delay provided in the script is not sufficient enough as per the device specifications. Therefore we need to alter the script to increase the delays between the recorded events in order to run it successfully. b.) If any UI layout changes, the script has to be modified. c.) Different scripts needs to be created for different orientations (landscape, portrait) Advantages a.) No need to manually write script. Therefore no or minimal scripting knowledge is required. b.) Once recorded, can be played back anytime therefore increasing our testing efficiency. c.) The script can also take screenshots to provide to end customer. d.) The screenshots can also be compared with earlier screenshots to enable us perform regression testing. This is done using some Image comparison tool like ImageMagick e.) The script can be run on multiple devices with varying configurations with minimal input from the tester. f.) The script can be run on various android versions. g.) Multiple devices can be tested in one go, i.e. multiple devices can be connected to a workstation and testing can be performed on each one of them sequentially, without user intervention Roadmap Ahead a.) To add remote device capability, wherein mobile devices can be connected from remote location and automotive testing can be performed. b.) To add functionality to support other android based touch devices like tablets. c.) To enhance reporting wherein customer can get customized reports along with screenshots. d.) To integrate it with other open source tools (such as Hudson) to provide Continuous Integration. Examples / Illustrations of existing tools in the market Case study 1 ROBOTIUM It is a tool, using which one can write strong and robust black-box test cases for Android applications. Test cases can include functions etc., which can spread across multiple Android activities. Robotium test is a subclass of junit.framework.testcase. Solo object is created (using Robotium library), which allows access to different views within the activities of the application. Page 5

Clicking on buttons, entering text, getting result from UI components, all can be accomplished by specifying the activity Solo object also helps to retrieve the values from Solo functions. The finalize method should always be called from within the teardown method Case Study: Calculator Application We would now take up a simple functionality of calculator Application build on Android 2.2 version and perform testing using Robotium. To use Robotium we would be requiring robotium-solo jar file (as an external jar) Here is a source code for the Android calculator application. publicclass Main extends Activity { EditTextFirstValue; EditTextSecondValue; TextViewResult; Button Calculate; floatnum1, num2; @Override publicvoidoncreate(bundle savedinstancestate) { super.oncreate(savedinstancestate); setcontentview(r.layout.main); FirstValue = (EditText) findviewbyid(r.id.edittext01); SecondValue = (EditText) findviewbyid(r.id.edittext02); Result = (TextView) findviewbyid(r.id.textview01); Result.setText("0.00"); Calculate = (Button) findviewbyid(r.id.button01); //Adding listener to button Calculate.setOnClickListener(newView.OnClickListener() { publicvoidonclick(view v) { //Getting first & second values and passing to show result showresult(firstvalue.gettext(), SecondValue.getText()); ); //Showing multiply results protectedvoidshowresult(editable first, Editable second) { float num1 = Float.parseFloat(first.toString()); float num2 = Float.parseFloat(second.toString()); float result = num1 * num2; Result.setText(String.valueOf(result)); Page 6

An.apk file for the Calculator application would be created now, which should have the same signature as the test project. To resign the.apk file with one s own key, re-sign.jar file can be used. It is a java tool, which is used to sign an application. Next we will create an Android JUnit test case for the above application. Here is the source code for the same: importcom.jayway.*; importcom.jayway.android.robotium.solo.solo; publicclasstestmainextends ActivityInstrumentationTestCase2<Main> { private Solo solo; publictestmain() { super("com.calculator", Main.class); @Override protectedvoidsetup() throws Exception { super.setup(); solo = new Solo(getInstrumentation(), getactivity()); @Override protectedvoidteardown() throws Exception{ try { solo.finalize(); catch (Throwable e) { e.printstacktrace(); getactivity().finish(); super.teardown(); publicvoidtestdisplayblackbox() { //Enter any integer/decimal value, we will enter 10 in first editfield solo.entertext(0, "10"); //Enter any integer/decimal value, we will enter 20 in second editfield solo.entertext(1, "20"); //Click on multiply button solo.clickonbutton("multiply"); //Verify that resultant of 10 x 20 asserttrue(solo.searchtext("200")); Next the robotium-solo jar file is loaded into our test project. To run the test case, we would run our project as Android JUnit test. The selected Emulator would be launched, and the test scenarios would be implemented automatically. The Android Calculator Application here is implementing the Multiplication functionality of the calculator. Our test case gives two values and checks whether the application is giving the right answer. Refer screenshots below: 1) The report generated after the test execution, looks like shown below. Numbers of Errors, runs, failure Trace are all shown in the test report. Page 7

There is another way to run the test case when the Eclipse is offline. We accomplish this by using DOS commands. We utilized adb commands to run the test project on DOS. Here com.calculator.test is test application target name. Pros and Cons of the Tool: Robotium benefits: With little knowledge of the application being tested, one can develop good (powerful) test cases The framework makes possible to handle multiple activities automatically The GUI components are binded at run-time One does not require the source code of the application, presence of the.apk file is enough Test execution speed is smooth and fast Page 8

Cons: It does not allow record-playback feature One cannot take screenshots (without proper security tag) If the layout of the application changes, one needs to edit the test accordingly A single test case is not possible for testing 2 applications Reporting (test results) not very much detailed. Makes difficult to understand the error Case study 2 Less Painful Some features of LessPainful include: 1) Cloud-based start-up doing testing for mobile 2) Used for Android only 3) Tests can be run concurrently on multiple devices. 4) Tests for Android and ios can be written in same language 5) Tests can be executed using Cucumber, or using a programmatic Ruby, Java or Clojure API. 6) Support for several features, touch, swipe etc. 7) Tests can be run on multiple devices with different settings, for e.g.: regional language etc. Source: http://blog.higher-order.net/files/automated_testing_geeknight_aarhus_07_09_2011 Pros and Cons: Pros: Record and playback feature supported Screenshots can be taken Timing can be controlled Page 9

Cons: e.g. Waiting for specific button to be clicked or text to be entered Automation possible on ios platform Work on real Android and ios devices in cloud, featuring report back results. UI automated testing doesn t seem to be very productive, since it has problems during integration. User has to give his consent of testing, but could not perform testing himself. LessPainful is more or less used like a service to test applications on real devices in cloud. Hence hands on experience not so exposed. Case study 1 ROBOLECTRIC It is a unit test framework, which removes a part of Android SDK jar, so that one can test his Android application. The tests are executed inside the JVM of the machine. The framework works by shadowing the Android objects, so that the business logic and functionalities of the application can be tested. The features of the framework provide access to the resources and to stand on the state of views. The framework is able to handle the views, resource of strings etc. Since the tests are run inside the JVM, there is as such no requirement of the Emulator. The main focus here is on the behavior of the application rather than the implementation of the Android. Eclipse usage: To run a test in Eclipse using Robolectric, we would require a separate test project, and the testing would be accomplished using Eclipse JUnit launcher. Just like using Robotium, we require a robotium-solo jar file; here also we would be requiring a robolectric jar file. The working directory should be made the root directory and not that of the test project. Pros and Cons: Pros: It is much simpler and faster than robotium. Open source framework Emulators are not required to execute the tests. Real Android objects are hidden (their shadow used instead), so that the business logic can be easily tested. Hence lesser need of extra steps or abstractions Allows testing of the whole code, not just the Android codes Test coverage is more Very useful for continuous integration Easy to maintain codes Helpful for testing bigger applications Cons: Fails to cover all the Android activity. E.g. Activity Manager is not covered under it. Execution is not done on Emulators, which might result in few scenarios getting left out from being tested. Does not test actual graphic contents Page 10

Since it uses shadow classes, updating of Android framework with new classes, will in consequently cause supporting the Android framework in an ad-hoc manner Page 11

Conclusion The world is getting androidified.the need of the hour is to develop scalable and end-to-end solutions which can lead to better tested handheld applications. References http://www.androidpolice.com/2011/03/14/number-of-apps-in-android-market-rapidly-gaining-on-applesapp-store-but-android-tablets-arent-so-lucky/ http://developer.android.com/guide/index.html Page 12

Biography Ankit Agarwal works as Head of Development (Productivity tools)& Partner at Hexaview Technologies Pvt. Ltd. Prior to starting Hexaview Technologies, Ankit has worked with many esteemed service & product based organizations like Infosys, Oracle &GlobalLogic in various domains such as Data-Warehousing, ERP & Finance (Wealth Management). Other key highlights include: - 10 years of experience in complete product lifecycle from Requirement Gathering toofficial Release to Production Deployment on customer site - Key team member for campus Recruitment and lateral hiring - Worked for key U.S banks like Bank of New York Mellon, PNC, Legg Mason - Proven ability in managing fixed bid projects within the time scope - His present role requires him to drive the product development and services efforts of his company Ankit holds a Engineering degree in Information Technology from University School of Studies, Delhi and MBA (finance) from ICFAI Hyderabad. Email: ankit.agarwal@hexaviewtech.com Shobhit Gupta works as a Software Test Engineer at Hexaview Technologies Pvt. Ltd. Prior to working in Hexaview Technologies, Shobhit has worked with Infosys Technologies Ltd for 2 years in Software Testing domain as a Test Engineer. Other key highlights of his career include: - Strongly believes in Knowledge Management activities in the corporate world. - He has hands on experience in Black Box Testing & Automation Shobhit holds an Engineering degree in Electronics & Communications from Krishna Institute of Engg. & Tech., Ghaziabad. Email: shobhit.gupta@hexaviewtech.com Page 13