Testing Mobile Software

Transcription

1 Page 1 by Hans Schaefer Hans.Schaefer@ieee.org Some tips and tricks Some decisive test techniques 2014 Hans Schaefer Slide no. 1 What determines app success? The decisive fact: User experience Usability Reliability (Some advice on functional testing) Security Power use 2014 Hans Schaefer Slide no. 2

2 Page 2 General Test Approaches Emulation based Device based Cloud based Crowdsourcing based 2014 Hans Schaefer Slide no. 3 General Approaches 2014 Hans Schaefer Slide no. 4

3 Page 3 Tool Overview 2014 Hans Schaefer Slide no. 5 "Software Test Attacks to Break Mobile and Embedded Devices" by Jon Hagar. Developer attacks Static analysis White box test coverage Control system Forcing unusual bugs (the unusual 80% outputs) Long duration tests HW/SW attacks Interfaces and integration testing (normal integration) 2014 Hans Schaefer Slide no. 6

4 Page 4 "Software Test Attacks to Break Mobile and Embedded Devices" by Jon Hagar. Mobile and embedded AW attacks Breaking digital communications (simulators needed) Bugs in the data Bugs in system-software computation Using simulation of real situations Time attacks Interrupt handling and priority inversion General time related bugs (with distributed systems) Time related tours, scenarios Performance testing 2014 Hans Schaefer Slide no. 7 "Software Test Attacks to Break Mobile and Embedded Devices" by Jon Hagar. Usability, Install-ability, Help Smartphone / App attacks Bugs in apps due to Network communication Data exchanged within the device Interference from other apps or the OS Restricted size Restricted UI Restricted functionality (unmet needs) General coding and design bugs Game attacks Cloud dependencies 2014 Hans Schaefer Slide no. 8

5 Page 5 "Software Test Attacks to Break Mobile and Embedded Devices" by Jon Hagar. Security attacks Big area, for specialists Generic attacks Combination tests Attacking functional bugs (usual functional testing) 2014 Hans Schaefer Slide no. 9 "Software Test Attacks to Break Mobile and Embedded Devices" by Jon Hagar. How to use the book? As a checklist Lot of advice Lots of tricks Some attacks are expert level Understand criticality! 2014 Hans Schaefer Slide no. 10

6 Page 6 Tips and Tricks The next 13 pages 2014 Hans Schaefer Slide no. 11 Taking Screenshots on a Mobile 1. Press the power and volume down button at the same time and hold them 2. On/off button and at the same time, press the home button 3. If you don't want to use or don't know how to use in-built functions of your Smartphone to take screenshot, you can use simple & free screen capture apps available on app-stores for ios, Android, OVI, etc. and tune them to take automated screenshots on particular intervals. Save the screen capture as a picture in the local picture gallery. Depending upon the device, you will know you've done so if the screen quickly flashes to blank and then back, or a ring around your display appears and disappears, or if your sound is on - you hear the "click" like after using an old-fashioned camera. Then, , save, toss, etc Hans Schaefer Slide no. 12

7 Page 7 Installing Test-apps on a Mobile Example: Google Drive. New builds from the continuous integration server to a shared Google Docs folder. (This is for Android, which is less restrictive than ios when it comes to installing apps). From the test device, download the build through the web browser, install. Opposite way also possible, for example pasting the contents of a debug console on the mobile app to a Google doc and the analyzer and handle it on your desktop machine. Generate test data on the laptop, then paste it into a Google doc, copy it from the doc on the device, and paste it into the app und test Hans Schaefer Slide no. 13 Testability: Logging Ask your developers to provide several levels of log output. Utilize the most detail for learning how the app ACTUALLY works as opposed to what the dev team or others tell you what the app should be doing. But if you utilize this method, BE CAREFUL to test without the high detail level of logging turned on, to be sure the detailed logging doesn't affect PERFORMANCE BEFORE reporting any bugs, turn your logging off, and slowly increase the level of detail for retesting the presumed bug Hans Schaefer Slide no. 14

8 Page 8 Simulating Wifi Speed The tool Linktropy can be used to mimic good to poor WIFI connections as well as 3G/4G. With this device you can mimic a WIFI connection via wired Ethernet but also schedule drops in speed or connection and packet reordering. Linktropy and Netropy WAN emulators simulate bandwidth, latency, jitter, loss, congestion, and other important network impairments to test the performance of applications under a spectrum of real-world conditions. Older / more primitive ways: Faraday cages (metal boxes) and modified APIs) Hans Schaefer Slide no. 15 Testing Functionality with Variable Context Ref: Schulte, M. and Majchrzak, T.A.: Context-Dependent Testing of Apps. In TestingExperience no. 19. testingexperience.com 2014 Hans Schaefer Slide no. 16

9 Page 9 What is What Test case A sequence of blocks with actions and corresponding result checks (like a use cases with data) Context One or more parameters that influence a test case i.e. a platform configuration, kind or speed of connection, change of location, memory state, battery state etc Hans Schaefer Slide no. 17 Why context based testing Run the same test case many times with different constant or variable context Possible to automatically generate test cases 2014 Hans Schaefer Slide no. 18

10 Page 10 How to Set up (usage based) test cases Blocks containing actions Result checks for every block Set up contexts for testing For every block: Result checks for different contexts (if available) Generate many concrete test cases By changing contexts between blocks 2014 Hans Schaefer Slide no. 19 Concept for context-based testing 2014 Hans Schaefer Slide no. 20

11 Page 11 A test case with two context changes 2014 Hans Schaefer Slide no. 21 The Tests Should be Automated Execution Generation Checking 2014 Hans Schaefer Slide no. 22

12 Page 12 Using Fiddler Proxy the device under test through your desktop, while running a tool like Fiddler ( to capture all http traffic (headers, requests, responses, payloads, sizes, server response time, sequences, caching status, cookies, and other attributes of the communication) to and from the device. Fiddler also interfaces to other tools making it possible to use proxies for IP addresses from different geo-locations. Next level: repeat the tests with new sessions/new devices/new builds, and capture the new traffic. Then, diff the request/response contents to identify what changed between sessions, requests, devices, etc Hans Schaefer Slide no. 23 Developments in Automated Test Running Improving the Accuracy of Automated GUI Testing for Embedded Systems Yang Dar Lin et al., IEEE Software Jan/Feb They used a newly developed tool Smart Phone Automated GUI (SPAG) to test automatically from a PC driving an Android device. Trouble: More load on the device makes replay of events uncertain and unreliable. Alternative is MonkeyRunner (part of Android software development kit). SPAG uses Android screencast and Sikuli. Android screencast couples the Android device screen to a PC and lets the tester drive the device from the PC. Sikuli is a capture-replay tool for GUI tests allowing to program verification points. Most trouble with gestures, because each gesture is composed of many events. And both displacements (i.e. distance between touches) and time difference (i.e. how fast the user moves) is interesting Hans Schaefer Slide no. 24

13 Page 13 End of Tips and Tricks 2014 Hans Schaefer Slide no. 25 Testing Usability 2014 Hans Schaefer Slide no. 26

14 Page 14 The Dimensions of Usability Learnability: How easy it is to learn to use it first time Efficiency: How many steps to complete a task Memorability: Does it get easier to repeat the task? Error recovery: How graceful is it? Accessibility: Easiness and simplicity of accessing key tasks or usability with handicaps Mapping: Do UI responses match user expectations? Visibility: Are important informations the most visible? Feedback: Is the user in control of the UI? Kept informed of what happens? Consistency: Are like items displayed and act the same way? Responsiveness: How responsive is UI interaction? 2014 Hans Schaefer Slide no. 27 Learnability: How easy it is to learn to use it first time Measure with sample user population Time until user is able to do use cases without help Online help / manual (?) Typical use cases Special use cases 2014 Hans Schaefer Slide no. 28

15 Page 15 Efficiency: How many steps to complete a task How many gestures / button presses / selections / menus? How many fields to fill in? How long text? Need to look up / refer to? 2014 Hans Schaefer Slide no. 29 Memorability: Does it get easier to repeat the task? Hotkeys? User defined shortcuts? Auto complete? Defaults? Remembering vs. other apps / devices? 2014 Hans Schaefer Slide no. 30

16 Page 16 Error recovery: How graceful? User should not lose control! No sudden (dangerous) actions Auto save / recovery 2014 Hans Schaefer Slide no. 31 Accessibility: Ease and simplicity of accessing key tasks or usability with handicaps Most important tasks on main screen Hotkeys Shortcuts Keys not in remote location For handicaps: Special area! 2014 Hans Schaefer Slide no. 32

17 Page 17 Mapping: Do UI responses match user expectations? Scenario testing with typical users Computer / Mobile / Domain literacy What do they expect? Test this early! Prototypes! 2014 Hans Schaefer Slide no. 33 Visibility: Is important information the most visible? What is most important? For whom? Visibility by font / size / contrast / placement Where do people look first? 2014 Hans Schaefer Slide no. 34

18 Page 18 Feedback: Is the user in control of the UI? Kept informed of what happens? Touching controls makes things happen User information especially if things take time Hans Schaefer Slide no. 35 Consistency: Are like items displayed and act the same way? Review screen vs. screen 2014 Hans Schaefer Slide no. 36

19 Page 19 Responsiveness: How responsive is UI interaction? UI control speed Repetition speed Reaction speed and info Does a user feel slowed down? 2014 Hans Schaefer Slide no. 37 How to test usability? Usability lab Low fidelity (on prototype, manual observation) High fidelity (on final product) User population Find a selection of people (different maturity and characteristics) Crowdsourcing International users on any platform Utest.com 2014 Hans Schaefer Slide no. 38

20 Page 20 Testing Battery Use 2014 Hans Schaefer Slide no. 39 What Drains the Battery? Communication (with cloud, server) -> minimize data use -> caching! Display Brightness, colors, changes (local) Computation and memory access 2014 Hans Schaefer Slide no. 40

21 Page 21 How to Measure it? Remove battery, replace by power supply with measurements Use an app like PowerTutor Typical use cases Repeat them many times, measure (high frequency sample), remove outliers Minimize uncontrollable background activity! 2014 Hans Schaefer Slide no. 41 PowerTutor Tool An app for Android Measurement exactness about 5% Measures any app s power use Repeat, sample! 2014 Hans Schaefer Slide no. 42

22 Page 22 PowerTutor Screens 2014 Hans Schaefer Slide no. 43 Thank you! 2014 Hans Schaefer Slide no. 44