How To Test On A Mobile Device

Transcription

1 Mobile Testing April 2013

2 Mattias Bergströmner Mattias has 12 years of experience within the QA and test domain. Mattias helps and support leading companies in various questions and challenges related to quality assurance and testing. Past Ericsson Software Technology AB Ericsson Mobile Platforms AB System verification Capgemini Sweden Present Mattias is head of the business area QA & Testing in Sogeti Sweden. Mattias belongs Sogeti s global team of experts within QA & Testing. You reach Mattias by mattias.bergstromner@sogeti.se Tel:

3 Agenda Mobility Your Business. Always On. Mobile Testing Priorities and Challenges What is different with Mobile Testing 10 Mobile Testing Success Factors Test Tools for Mobile Testing Summary

4 This is Sogeti QA & Testing Practice 9500 test professionals worldwide, 400 in Nordics and 200 in Sweden Test service portfolio that ranges from test strategy development to outsourced testing operations across the whole world Consultants trained in own methods TMAP & TPI and also ISEB/ISTQB certified Alliance partners with HP, IBM-Rational, Microsoft, Oracle and Microfocus Analysts such as Nelson & Hall, Ovum and IDC confirms our position as the leading company within testing in the world

5 Our Methods, Tools And Accelerators Create Immediate Efficiency And Quality Benefits Client dedicated test lines Market leading methodologies and accelerators for testing and test process improvement

6 Mobile Solutions and Testing From mobile solution design to mobile testing Mobile Solutions 1,500+ mobile experts world wide Mobile Solutions Center of Excellence, Mumbai Testing World leading testing provider Test Professionals Mobile testing is a joint initiative between the Mobile solutions and Testing global service lines Mobile Testing Center of Excellence, Mumbai

8 Mobility Your Business. Always On. Enterprises ask: How can we enhance and extend our products and services? Making your business mobile is no longer a matter for debate. Today s mobile users your customers, your employees your suppliers even your shareholders expect interaction with you at their fingertips Enterprise mobility goes beyond mobile apps it influences how you run your business Mobility has reset expectations of what constitutes application quality robust performance and usability on the move adding a further layer of complexity One of Europe s largest travel operators generates 200 million Euro per year in mobile channel Mobile bookings reaches 15% of total bookings by end of 2012 Source: Wall Street Journal Business travelers: the past 12 months, 26% have used an app to check flight schedules 11% have used an app to book flights Source: Forrester Research

9 Business Critical Mobility Mobile and consumerization of IT Majority of mobile phones sold are app centric >50 billion app downloads >1 million apps in App Stores Mobile business is in exponential growth Transactional features in responsive web sites and apps Products and services Accelerated internal processes Mobile process management including field and sales service, approval and time management Improved social collaboration using social enterprise platforms Mobile access to business intelligence solutions

10 Mobile evolution

12 Quality Assurance and Mobile Testing The latest annual World Quality Report * research found that: Organizations are struggling to manage the mobile challenges, with only one-third (31%) of those surveyed currently formally testing their mobile applications * Based on 1,554 telephone interviews conducted for report sponsors Capgemini, Sogeti and HP across 25 countries

13 Mobile Testing Priorities: Performance WQR Survey: Performance testing is a market priority (64% of respondents) Mobile Transactions Survey*: +1 second delay: Up to 8% of users stop using the app 85% of users expect the mobile experience to be better than or equal to online using a laptop or desktop computer *Source: Harris Interactive Mobile Transactions Survey, 2011

14 Mobile Testing Priorities: Usability App users expect an easy to use, beautiful user interface, which is both distinctive and enjoyable 47% expect the mobile experience to be better than the in-store experience 85% expect the mobile experience to be better than or equal to online using a laptop or desktop computer

15 Mobile Testing Priorities: Compatibility Challenges of proliferation and fragmentation: 3,997 Android models* 52% of respondents do not have the necessary devices available for testing (WQR ) Companies consider compatibility testing almost as important as functional testing (46% vs 48%) *Source: opensignalmaps.com study, May 2012

16 Mobile Testing Priorities: Security Explosion of mobile applications creates new security challenges such as: Lost or stolen devices Mobile malware Targeted attacks on devices Security issues include: Confidentiality: Does your app keep your private data private? Integrity: Can the data from your app be trusted and verified? Authentication: Does your app verify you are who you say you are? Authorization: Does your application properly limit user privileges? Availability: Can an attacker take the app offline? Non-Repudiation: Does your app keep records of events?

17 Summary Mobile Testing Challenges Lack of devices and operating systems for test Rapidly changing technology landscape, such as device types and operating systems, leading to continuous need for updates Lack of methods, solutions and experts specific for mobile testing Lack of the right tools Lack of re-usable assets and frameworks Performance, usability and security require specific attention

19 Mobile Testing is business as usual...

20 ... but Mobile Testing adds additional hot spots

21 ..Screen sizes today

22 Device fragmentation creates a need to perform device level testing & UAT provisioning for mobile apps Testing cycle for mobile application Testing cycle for desktop application Requirements Analysis Requirements Analysis Test report analysis Test Plan Test report analysis Test Plan Device provisioning & device UAT Test Case Creation Execute regression testing / automation Execute new features testing / manual testing Test case creation / automation scripting Device Level validation / manual testing Execute regression testing / automation Execute new features testing / manual testing Unlike the desktop, the mobile space includes multiple platforms and form factors. These complexities lead many organizations to test on real devices, and most do this manually Source: Forrester - Strategies For Testing Mobile Apps

23 The complete eco system must be considered in your mobile testing strategy

25 10 Mobile Testing Success Factors Mobile Platform Solution Connectivity Physical characteristics Location Fragmentation User experience Integration Performance Test automation Third parties Security

26 Success Factor #1: Testing for Connectivity Mobile solutions depend on network connectivity Solution design and testing need to address: variable bandwidth offline and flight mode scenarios user sessions moving between different network conditions To ensure consistent behavior, it is recommended to conduct a combination of: automated network simulations manual testing in real network conditions

27 Success Factor #2: Understanding Physical Characteristics Physical characteristics differ significantly from non-mobile solutions Mobile testing needs to take several characteristics into consideration: Screen size Touch and gesture capabilities Orientation (vertical or horizontal) and movements in 3 dimensions Camera GPS

28 Success Factor #3: Location Simulation An increasing number of solutions utilize location data and GPS integration Features may depend on location or distance to other locations Successful mobile testing ensures quality across different GPS implementations and enables efficient location simulation Make use of devices in the cloud, located in different parts of the world Simulate localization scenarios in emulators

29 Success Factor #4: Test Automation Test automation using scripting or recording makes mobile testing significantly more efficient Smart test automation utilizes parameterized scripts for both device types and user interaction, and is enabled to run on devices in-house and in the cloud Some sophisticated user interactions cannot be fully automated, hence there is need for planning for what tests can be automated and what tests must remain manual

30 Success Factor #5: Addressing Fragmentation Fragmentation of operating systems and device types continues to challenge solution design and testing Most solutions need to support hundreds of device types and several versions of operating systems A relevant mobile testing tool box includes: physical access to most major combinations of device type and OS ability to run manual and automated tests across both physical devices and emulators access to cloud based platforms to maximize number of test devices

31 Success Factor #6: User Experience User experience is a main criteria for success Testing user experience is an art that consists of both objective and subjective measurements Key elements include: User interaction level of intuitiveness and efficiency Navigation adherence to platform rules and guidelines Signup and login clarity and ease of use Layout and user interface design look and feel, aesthetics, precision in layout Exception handling graceful, returning user friendly messages

32 Success Factor #6: User Experience

33 Success Factor #6: User Experience Not so good!

34 Success Factor #6: User Experience

35 Success Factor #7: Engaging with Third Parties Most apps are distributed through open and public app stores, each with its own set of guidelines Apps that fail to adhere to guidelines may be rejected Given the third party review process, there is an inherent bug fix latency in updates Operating systems are frequently updated. New releases may break existing apps. Guidelines: Use common testing checklists based on latest app store rules and guidelines Rapid testing cycles for updates in order to minimize impact of bug fix latencies Test existing apps on beta versions of operating systems

36 Success Factor #8: End-to-End Integration Testing Transactional apps integrate with back end servers and applications System integration is often the most challenging aspect of any solution design Mobile testing becomes highly relevant as it provides a foundation for end-to-end integration testing Mobile Testing should be considered as an integrated activity in the overall solution testing

37 Success Factor #9: Performance Mobile solutions can overload servers, impacting performance for all users, for several reasons: Rapid growth of number of users Average life span of transactions increase due to variations in bandwidth and latency There are significant performance differences across devices. Tests should be conducted on high-end and low-end devices. Successful mobile testing focuses on network capabilities, system integration and back end layers, as well as the app itself

38 Success Factor #10: Security Mobile solutions drive an increasing number of scenarios where sensitive systems are accessed, and private data is in transit and at rest, i.e. stored in mobile devices Given the physical nature of mobile devices, they are more easily forgotten, lost, or stolen Mature mobile testing aims at Securing confidentiality and integrity of data Validating whether authentication and authorizing is secure Verifying that systems are keeping records of events (non-repudiation)

40 Perfecto Mobile Test Automation (UFT Mobile) Keyword-based scripting ScriptOnce - Patented device agnostic automation Device Replacement takes minutes not weeks Native HP ALM Extension

41 Performance Test Tool Performance Testing End-to-end testing Simulation of user types Network virtualization Partnership with Neotys

43 Summary Mobile testing needs to be addressed in the company s overall Mobile strategy As apps become increasingly business relevant and business critical do not neglect the importance of quality assurance Make sure you understand why mobile testing is different and apply the right actions to meet them Make sure your testing efforts take the device s characteristics into consideration Mobile apps is Everywhere and IMMEDIATLY Your business is always on

44