An Approach of GUI Test Automation on Mobile Devices

Transcription

1 1 Xiaojie Lin, 2Lu Lu, 3Kai Su, 4Yuping Yan Department of Computer Science and Engineering, South China University of Technology, Guangzhou, Guangdong, China, *2, Corresponding Author Department of Computer Science and Engineering, South China University of Technology, Guangzhou, Guangdong, China, 3,4 Information Center, Guangdong Power Grid Corporation, Guangzhou, China 1, First Author Abstract Aimed at the complex and increasing test issues of mobile phone GUI application, this paper presents an event sequences-based platform called MobileAutomation. The platform consists of three primary modules: mobile automation agents, framework and IDE. The agents capture user actions as event sequences, turning them into corresponding keyword-driven scripts which constitute a reusable test case. The automation framework enables application to execute the test case on mobile devices. The IDE is an eclipse based tool that users can take full control of test procedure like capturing, editing, playing back test scripts and managing test suites. A case study demonstrates how the three parts of MobileAutomation platform interact with each other to reduce the complexity of mobile test automation. Keywords: Mobile Test Automation; GUI Test; Event Sequences; Test Case 1. Introduction As the wireless and mobile networks develop rapidly, a huge number of mobile applications emerge. It has brought vast changes in mobile commerce, social networking, multimedia, games and locationbased services. Application delivery channels like Apple App Store, Google Play have provided end users with access wherever and whenever they need. Given so many options to fetch and enjoy the mobile applications, users consider that the applications be reliable and stable. More complex software results in difficult quality assurance. Therefore, clear and nimble test should be applied to make sure that the software functionality lives up to the actual design during the whole development process. There are various functional test automation tools and frameworks contributing to improving the quality of mobile applications. They automate the steps of test using operation simulation, feedback, verification and validation of GUIs [1]. However, some of them don t make it fully automatic while the others don t support testing directly on mobile devices [2] [3]. Due to the diversity of mobile environment, performing the same test in different mobile devices is imperative. To provide a more efficient testing practice, the MobileAutomation platform is dedicatedly designed and implemented. It uses an event-sensitive approach to automate the procedure of scripts generation. Aspect-oriented programming technique is also applied to simplify the development of event handling model and improve the scalability of the system. It allows both real devices and emulators for mobile test automation. On one hand, testers can generate test suites for different device types by simply loading the appropriate device profile, creating the emulators in their own powerful PCs. On the other hand, testers can reuse their test suites, performing test automation on multiple devices. The MobileAutomation platform has supported Android (1.6 and higher) version of AUT in the present implementation. The development of windows phone and ios version is ongoing. The following section includes an overview of mobile test automation frameworks and tools. It also presents a critical analysis of related works and draws forth the objectives of the MobileAutomation platform. The architecture of the platform is described in section 3. Section 4 gives details about design Supported by Guangdong Production, Education & Research Project (2012B ), Guangzhou Production, Education & Research Project (2011Y5-0004) International Journal of Digital Content Technology and its Applications(JDCTA) Volume7,Number9,May 2013 doi: /jdcta.vol7.issue

2 and implementation of the platform. In section 5, a case study is introduced. Section 6 draws a conclusion of this paper. 2. Related works 2.1. Mobile test automation techniques Former research has shown that the popular GUI (Graphical User Interface) test methods are Record/Playback, Capture/Playback, Particulars-based and Beta test method [2]. Record/Playback is the method recording the events occurred in GUI selected through user s mouse and keyboard input before replaying the low-level events stream. So it cannot adapt to the change of the application under test (AUT). Capture/Playback captures interactions between users and GUI applications, and then translates them into a test case that can be automatically replayed. It is the accomplishing way of writing complicated scenario of GUI. Particulars-based method requires GUI particulars before test; however, getting these particulars makes constant efforts. Beta test technique is used regularly by merchandising the software of beta version. But the tasks of training common users are arduous. Moreover, it is hard to understand which will affect the test result Mobile test automation frameworks and tools Many researchers have tried to develop a universal test automation framework and a set of tools. The papers [3] [4] [5] [6] [7] have discussed about the construction of them. MobileTest [3] is an SOA based framework, with extendable script interpreter, universal communication interface and agent-based testing mechanism. It is specifically designed for mobile device testing. Considering the implementation of test result verification, it still relies on image comparison. QTP [4] (HP QuickTest Professional) is the most advanced automated functional testing solutions that support for all major software applications. The interfaces of QTP are quite complex to use because they are designed for test automation mainly on desktop applications. Specific plugins have to be installed so as to enable mobile test automation. Selenium [5] is a portable web test framework. The Selenium-IDE (Integrated Development Environment) is a tool to develop Selenium test cases. Currently, it only supports mobile browser test by using the web drivers. MoGuT [6] framework uses image flow to describe the event change and the relevant screen feedback. It is suitable for testing nimble application, but lacks of flexibility due to not supporting object series. Robotium [7] makes it easy to write powerful and robust automatic GUI test cases for Android applications. With the support of Robotium, test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities. On Android platform, there are generally three types of test automation tools in the market. Using different ways to identify GUI object, they are divided into graphic, digital signature, and source code based automation frameworks. Graphic based frameworks like PerfectoMobile identify GUI elements by image comparison. It can also collaborate with QTP as a plugin of mobile test automation. Digital signature based frameworks like DroidPilot monitor AUT through in-process data sharing. Source code based frameworks control AUT by instrumentation. Testdroid is such a tool recording user operations and then generating test scripts of Robotium or MonkeyRunner accordingly. In this paper, the Android version of the MobileAutomation platform takes advantage of this source code instrumentation technique. It has been observed that test automation is performed with the mobile device connected to PC. The test scripts reside on local PC and requests are sent to the targeted mobile for test execution via a communication module. None of the frameworks perform the automation testing of the mobile applications solely on the mobile device. While some frameworks communicate with PC to perform the test on the device, others run on the emulator installed on PC [4]. On the basis of existing frameworks and tools, the MobileAutomation platform is designed and implemented in order to meet the following objectives: ⑴Support unit test and concurrent test on multiple devices. ⑵Support 451

3 simulators as well as real devices on mobile test. ⑶Support test case generation, visualization of execution and test suite management. 3. The architecture of MobileAutomation This section will provide an overview of the MobileAutomation platform and show how this automatic GUI test platform works for mobile devices. It uses modular design to reduce the complexity of the whole system. This paper elaborates each module and illustrates their interaction. The platform is composed of four modules as shown in the Figure 1. Figure 1. The MobileAutomation Platform Modules The Mobile Automation IDE provides users with eclipse style workspace to perform mobile test automation on PC. Using http protocol, it communicates with mobile device connected via USB or Wi- Fi. The server listens into http requests on specific port. It is actually a daemon process running on both mobile device and PC, maintaining a message queue. The agents interpret commands passed by the server process. They enable the applications to capture and play the commands of user actions or verification steps. The mobile automation framework is to implement application instrumentation code. Running with instrumentation turned on, AUT will be instantiated by the framework, allowing users to monitor all of the interaction between the platform and the AUT. The testing process of MobileAutomation platform is depicted in Figure 2. Figure 2. The MobileAutomation Platform Testing Process 452

4 Tester sends a capturing command by pressing the capture button in the IDE console, waiting for the connection established between the console and agent. The agent runs with the AUT, capturing the user actions on device as event sequences. The http server here is to forward commands. Receiving these sequences, the console interprets them as commands and generates test cases. The IDE provides proper environment for test cases/suites management, allowing users to modify and reuse test scripts. Completing the test cases/suites, users can execute them through MobileAutomation framework. The framework is able to set up the AUT and perform test on device. It also supports for verification as well as assertion during the test automation. Finishing execution, the IDE will display the test result and log generated by the framework. 4. The design and implementation Before describing the detailed design of the MobileAutomation platform for Android test automation, this section presents some basic techniques. And then it takes an example, showing how this platform works by applying these techniques Basic techniques GUI test automation GUI Test Automation mimics how humans interact with the GUI by its widgets and corresponding events [8] [9]. There are three approaches verifying a GUI: ⑴bitmap comparison; ⑵optical character recognition (OCR); ⑶ Application Programming Interface (API) [7]. Test case using bitmap comparison is hard to maintain because of various shape of target devices. The OCR approach is sometimes slow to execute and even inaccurate. Verifying the GUI through an API can be much more maintainable, allowing small changes in the GUI. Keyword-driven testing method is a convenient approach to design and implement a GUI test. It abstracts low-level UI operation by keywords. Keywords reflect basic user events, including pressing hardware keys, tapping, dragging objects on the screen and selecting an item from a menu. They also verify the state of AUT by examining a desired text or GUI object found on the screen. During the execution of test cases, keywords are transformed into concrete UI operation. By this approach, a GUI is modeled as a set of widgets W = {w1, w2, w3 wm} (e.g. buttons, text fields, panels) that constitute the GUI, a set of identifiers D = {d 1, d2, d3 dn} (e.g. id, text, index) that identify a component in a GUI. For a set of actions A = {a 1, a2, a3 ap}, an event sequence of a GUI is specified as a set S of triples (wi, dj, ak) where wi W, dj D, ak A. It represents all possible sequences of actions that a user can perform on the GUI [10] [11] Activity lifecycle management An activity represents a single screen with a user interface. It is one of the four types of application components (activities, services, content providers and broadcast receivers), which serve distinct purposes. It also has a distinct lifecycle that defines how the component is created and destroyed. It is crucial to manage the lifecycle of the activities by implementing callback methods. An activity can exist in essentially three states [12]: ⑴Resumed, also known as running, means that the activity is in the foreground of the screen and has user focus. ⑵Paused, refers that another activity is visible on top of current activity. A paused activity is completely alive. That is, a paused activity object is retained in memory. It maintains all state and member information, and remains attached to the window manager, but can be killed by the system in extremely low memory situations. ⑶Stopped, means that the activity is completely obscured by another activity. A stopped activity is also still alive. But it is no longer visible to the user. It can be killed by the system when memory is needed elsewhere. The following skeleton activity includes the fundamental lifecycle methods which developers can override to do appropriate work when the state of activity changes. public class ExampleActivity extends Activity { public void oncreate(bundle savedinstancestate) {super.oncreate(savedinstancestate);} 453

5 } protected void onstart() {super.onstart();} protected void onresume() {super.onresume();} protected void onpause() {super.onpause();} protected void onstop() {super.onstop();} protected void ondestroy() {super.ondestroy();} Figure 3. The Activity Lifecycle oncreate: The activity is being created. onstart: The activity is about to become visible. onresume: The activity has become visible (it is now "resumed"). onpause: Another activity is taking focus (this activity is about to be "paused"). onstop: The activity is no longer visible (it is now "stopped"). ondestroy: The activity is about to be destroyed. Figure 4 illustrates the states transformed above (more details shown in [12]). The rounded rectangles represent these callback methods. They run through the whole lifecycle of mobile application activities. Figure 4. The Activity State Transition Aspect-oriented programming (AOP) AOP is a good candidate for weaving capture module into AUT. It improves concerns separation in test automation platform by adding crosscutting functionalities without changing the original application [13] [14]. Figure 5 shows the basic structure of aspect-oriented coding. aspect OnExample { pointcut func() :(execution(*somefunc (*))); 454

6 before(): func() {execution I.} after(): func() {execution II.} } Figure 5. The Structure of AOP It defines an aspect named OnExample. Before the execution of method that matches the regular pattern *SomeFunc (*), the execution I will be invoked. And after it successfully returns, the execution II will be invoked, too. Thus, the AspectJ compiler weaves these methods into AUT at runtime dynamically MobileAutomation testing algorithm This paper chooses an instance of mobile application to demonstrate how the MobileAutomation platform works. The mobile application is a well-known notepad application. It is chosen because it contains common GUI components (e.g. button, text field, menu, menu item, dialog and etc.) as well as clear business logic. Based on the techniques listed above and the common approach of Capture/Playback GUI test, an algorithm is summarized below (Figure 6), which describes the detailed procedure of mobile test automation. 1) 2) 3) 4) 5) 6) 7) 8) 9) Initialize widget_map with components for automation and activity_map with loading activity. For all of the w widget_map, Chain listeners of w and weave event handlers. Start capture daemon, initializing the command_list and oracle_list with empty array. Capture user actions by the event handlers of components, interpreting them into command_list as sequence. Get the current activity of the application and its hierarchical component view view_set. Save the context of application and jump back to Step 4 until the capture is done. Set up the AUT on mobile device when starting playback. Initialize the test environment, including the deployment of test code. Verify if the item of command_list is valid in the current context. (e.g. w view_set, which means that the component is in the current activity). If valid, it is translated into function call f using MobileAutomation API. Otherwise, the test is failed. Assert the result of execution and put it into the oracle_list. Compare the returned value of the call f with relevant o (o oracle_list). If they are equal to each other, the test is passed. Otherwise, it fails. Tear down the AUT on mobile device; generate test result and exit. Figure 6. The MobileAutomation Testing Algorithm Description The essential mobile test automation is identifying components in the current screen, preparing oracle for each function under test, and executing calls to decide whether the test case is passed successfully or not. The MobileAutomation testing algorithm requires the initialized activities and widgets for automation. When the activity of application is created, the oncreate method is called, triggering the matched pointcut through the AOP. MobileAutomation platform captures this activity and adds it into the activity_map, which is a hash map storing name as well as class of each activity; and meanwhile, the program adds listeners to the components, which offer the capturing functionality. It traverses the components hierarchy of current activity by Java reflection technology, generating a tree-like structure, view_set. The view_set contains all the components in the activity. By attaching event handlers to them, the components are ready for capture. They are encapsulated into the agent in the aspect path of AUT. Thus every action generates an event sequence as soon as the AUT is started. The sequences then make up a command_list, which composes a reusable test case. The test case is readable and executable. It runs as an Android JUnit test so that testers are able to put the verification and assertion in and set up the AUT from it. 455

7 4.3. Agent capture module The capture module of the MobileAutomation platform works with the AUT by monitoring its event handlers (e.g. onclick, onchanged, onscroll). A GUI event will trigger the capturing module, which translates the event sequence into keyword-driven script. The grammar of keyword-driven script follows the pattern: Component ID Action Arguments Timeout (ms) Think Time (ms). Table 1. The Grammar of MobileAutomation script Class type of the component Component ID Action Arguments Timeout Think Time Identifier of the component User action(e.g. enter text, click, scroll) Position or Text (Optional) Milliseconds allowing retry Milliseconds before executing the action 4.4. Instrumentation framework, IDE playback module Based on the keyword-driven script, the playback module of the MobileAutomation platform generates a test case using MobileAutomation framework. The test case is shown in the IDE, which allows user to add or modify functionality. The test case is also managed as an Android JUnit Test which can be started in the IDE Server command module Since capture module runs as agent in mobile device and playback module integrates eclipse IDE in PC, a command module is needed for communication between PC and mobile device. Using http socket, it maintains a blocking queue to capture user event sequences. When the capture of AUT is on, event sequence is put into the queue and sent to the host successively. And the capture daemon loops to take the elements out of the queue and produce the scripts. 5. Case study The MobileAutomation platform has been tested on Nexus S and HTC One X which are 3G smart devices with Android platform. In this case study, a sample application provided by Android SDK, named Notes, has been tested. It is a simple editor for user to manage their note lists, including adding, editing, deleting notes as well as changing titles. There are 4 activities in Notes application totally: NotesList, NoteEditor, TitleEditor and NotesLiveFolder. Among them, NotesList is the main activity to set up the whole application. Figure 7 shows 12 different screenshots when running Notes application on mobile device. Considering the static activities as vertexes, Figure 8 is a directed graph for them, which takes event as the transforming edge. Test scenarios (Table 2) have been devised to test the application. For each of the test scenarios, the test cases (Table 3) are designed and successfully executed. No. TS1 TS2 TS3 TS4 TS5 TS6 TS7 Table 2. MobileAutomation Platform Test Scenarios Test Scenarios Description Launch mobile application with the agent. Perform manual testing on mobile application. View scripts and export them to external data file. Perform automated test on mobile application. Create new test data and run new automated test with them. Assert the state of the mobile application. Verify the MobileAutomation framework API and its output results. 456

8 Table 3. MobileAutomation Platform Test Cases No. Test Case Test Scenarios Description TC1 Add agent to the aspect path and start the AUT. TS1 TC2 Start capture and operate the AUT in the mobile device. TS2 TC3 TC4 Test with generated scripts. Test with modified scripts. TS3 TC5 View the scripts in IDE and export them to external data file. TS4 TC6 TC7 Run test case on one mobile device. Run test case concurrently on different types of mobile devices. TS5 TC8 Define new test data in the table and run automated test with them. TS6 TS7 TC9 TC10 TC11 TC12 TC13 TC14 Assert the current activity. Assert the properties of components in the current activity. Set the properties (e.g. text, color, etc.) of certain component. Find certain element (e.g. button, textarea, etc.) in the current activity. Wait for certain activity to load. Verify the display (e.g. text, color, etc.) of certain component. Figure 7. The Sample AUT 457

9 The particular executing path is that: user opens the Notes application, saves an item of notes and then edits the notes. According to the activity diagram (Figure 9), the path is: start a1 a2 a3 a4 a5 a6 a7 a9 a10 a6 exit. The keyword-driven script is: Device * menu Menu * select "Add note" TextArea note entertext "Test" Device * menu Menu * select Save Table * SelectIndex 1 TextArea note entertext "Test Edit" Device * menu Menu * select Save Device * back Figure 8. The Keyword-driven Script of AUT Figure 9. The Activity Diagram of AUT Interpreting the commands (Figure 8) through the command module, the playback module generates the following test case: device.pressmenu(); findelementbytext("add note").doclick(); findelementbyid("note", TextViewElement.class).setText("Test"); device.pressmenu(); findelementbytext("save").doclick(); findlistelementbyindex(0).doclick(); findelementbyid("note", TextViewElement.class).setText("Test Edit"); device.pressmenu(); findelementbytext("save").doclick(); device.pressback(); Figure 10. The Android JUnit Test Case of AUT The test case can be run as Android JUnit test, and the result displays as JUnit test report (Figure 11), which can be also exported as XML format. 458

10 Figure 11. The Test Result of the MobileAutomation Platform 6. Conclusion There are three important points for GUI test using the MobileAutomation platform: (i) automatic test case generation reduces the hand writing expense; (ii) test cases are executable and reusable; (iii) integrated environment makes development and test management easier. With the IDE of different PC platform (win32, win64, Linux gtk 32 and gtk 64), it can automate the whole process of GUI test. The platform attains user action through aspect code, adopting an event-based approach to generate the feasible test case. Using Java reflection technology, it can identify components and generate their hierarchical views precisely. It can be enhanced to enable mobile games test, other applications with complicated business logic and even web-based applications test. The fault seeding technique and test suite generation of GUI application will be integrated into this platform to detect more faults out of the test cases in the future. 7. References [1] Liu Zhifang, Liu Bin, Gao Xiaopeng, Test Automation on Mobile Device, In Proceedings of the 5th Workshop on Automation of Software Test, pp.1-7, [2] Sun-Myung Hwang, Hyeon-Cheol Chae, Design & Implementation of Mobile GUI Testing Tool International Conference on Convergence and Hybrid Information Technology, pp , [3] Jiang Bo, Long Xiang, Gao Xiaopeng, MobileTest: A Tool Supporting Automatic Black Box Test for Software on Smart Mobile Devices, Second International Workshop on Automation of Software Test, [4] Leckraj Nagowah, Gayeree Sowamber, A Novel Approach of Automation Testing on Mobile Devices, International Conference on Computer & Information Science, pp , [5] Andreas Bruns, Andreas Kornstädt, Dennis Wichmann, Web Application Tests with Selenium", IEEE Software, IEEE Computer Society, vol.26, pp.88-91, [6] Kwon, O.H. Hwang, S.M., Mobile GUI Testing Tool Based on Image Flow, 7th IEEE/ACIS International Conference on Computer and Information Science, pp , [7] Tommi Takala, Mika Katara, Experiences of System-Level Model-Based GUI Testing of an Android Application, ICST, pp , [8] Xun Yuan, Atif M. Memon, Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback, IEEE Transactions on Software Engineering, IEEE Computer Society, vol.36, no.1, pp.81-95, [9] Xun Yuan, MB Cohen, Atif M. Memon, GUI Interaction Testing: Incorporating Event Context, IEEE Transactions on Software Engineering, IEEE Computer Society, vol.37, pp , [10] Jaymie Strecker, Atif M. Memon, Accounting for Defect Characteristics in Evaluations of Testing Techniques", ACM Transactions on Software Engineering and Methodology, vol.21, no.3, pp.1-43, [11] Atif M. Memon, An event-flow model of GUI-based applications for Testing, Journal of Software: Testing, Verification and Reliability, John Wiley & Sons, pp , [12] App Components, [13] Colyer, Adrian M., Aspect-oriented programming with AspectJ, IBM Systems Journal, vol.44, pp , [14] L. Madeyski, Ł. Szała, Impact of Aspect-oriented Programming on Software Development Efficiency and Design Quality: An Empirical Study, IET Software, vol.1, no.5, pp ,