Mobile Test Strategy. Shankar Garg. Senior Consultant - Testing

Transcription

1 Mobile Test Strategy Shankar Garg Senior Consultant - Testing

2 Scope of Mobile Testing Why Quality is important Challenges in Mobile Testing Best Practices for Mobile Test Strategy Mobile Testing Tools Conclusion

3 Scope of Mobile Testing is: Web Apps : Web applications opened on mobile browsers ( chrome, safari, opera etc.) Native Apps : application program that has been developed for use on a particular platform or device.( android, ios etc.) Hybrid Apps : Native Apps + Web Apps

4 Why Quality is important Back in the day, one bad customer experience would spread via word-of-mouth and reach few Customers. Thanks to Social Media platforms like Twitter, Facebook etc. however, that same word-of-mouth now travels at the speed of light and reaches million users. The other major difference is App Store ratings. Imagine if a poor review were stamped directly on the packing of a product. These Social Media Ratings and Likes can prove to be a make or break for the brand and product.

5 CHALLENGES

6 Hardware - Device Variation Software - OS Variation Network Vendor and Bandwidth Variation Interrupts Alerts and Notifications Waterfall Software Development Process Updates Quicker Release Cycles Data Security

7 Hardware - Device Variation Mobile application testing is difficult because of : Various Manufacturers ( Samsung, Apple etc. ) Various Screen Sizes ( Phone and Tablet etc.) Keypad (Virtual or hardware) Variations in RAM, CPU, OS utilization etc. Touch Screen Responsiveness. No Guarantee that application tested on one device will run successfully run on other device also.

8 Hardware Fragmentation in Mobile phone_subscriber_market_share

9 Screen Fragmentation in Android Source:

10 Vendor Fragmentation in Android

11 Software - OS Variation Mobile application testing is difficult because of : Various OS ( ios, android, Windows etc. ) Various Versions of OS ( ios 4 to ios 7.1, android to android 4.4 etc.) OS like ios does not allow the device to be down graded to lower ios once upgraded to higher ios.

12 OS Fragmentation in Android Source:

13 OS Fragmentation in ios Source:

14 Network Network Carrier ( Airtel, Vodafone etc.) Bandwidth ( 2G, 3G, Broadband etc.) GSM, GPRS, CDMA and LTE? No Network, Airplane Mode Network Connectivity (low, high etc.) Geo-graphical location of the target Audience. Replicate diverse locations? Test GPS-based Operations?

15 Network Speed vs App function

16 Interrupts Mobile Application behavior is affected by: Alerts and Notifications from other apps installed on the machine. Very difficult to simulate While accessing internet if a calls comes than Internet connection is interrupted. (2G)

17 Waterfall Waterfall Software Process is a big challenge for mobile development. By the time the design the app is freezed, new version of ios is released. By the time development is finished, new line of Smartphones is released. In the examples mentioned above it would be very difficult for the app developed in waterfall model to handle these changes.

18 Updates The ease of upgrading apps over the air combined with increased user expectations about quicker releases (both for bug fixes and new features) result in frequent application releases. App are also updated frequently to handle multiple major and minor OS updates. Test teams are continuously tasked continuously tasked with testing entire applications faster and more often.

19 Data Security Mobile Apps have access to more personal Data as compared to Web Sites Users expect their personal data to remain personal. Users expect their apps to be 100% free of viruses, malware and other threats.

20 BEST PRACTICES

21 Best Practices for Test Strategy Select Target Devices Emulators vs. Real Devices Cloud Computing Network Options Mobile Testing Tool Test Automation Agile Testing Security App Store Guidelines and International Standards

22 Select Target Devices Points to keep in mind: OS Version Android, ios, windows etc. Which all versions are supported ( don t forget the minimum version). Smartphone or tablet Display Density Low, high and High Definition Misc. CPU, RAM and Memory etc.

23 Emulators vs. Real Devices Advantages Developers can verify certain functionality that is not specific to any device, carrier or operating system. Usage of Emulators is very Cost effective in early stages of the development To familiarize the team with basic features of a phone. Disadvantages Touch Responsiveness can not be tested on Emulators. Ex. Tap and Gestures etc. Actual user behavior can not be tested with emulators. Ex. ( Nokia Advertisement :- person jumping off bridge with phone) Camera and Mic can not be tested with Emulators. Ex. Instagram. Access to multiple phones at no cost at all.

24 Emulators and Real Devices Emulator Emulator + Device 1 Used in early development/dem os Distribute features across device and Emulators Device Used only for network dependent test cases 2 Basic Functionality Testing Network Connectivity 3 Can be used in automation testing Stable UI Features

25 Cloud Computing Cloud Computing is a web based environment where testers can deploy, test and automate their mobile applications on real devices or emulators via internet. User provides the input to the Web Interface via internet Server receives the request and sends it to the device/emulator Mobile Devices resolves the user request and sends the response to server which in terms send the response to user

26 Cloud Computing Advantages 1 Per Hour Rate/ Usage Based Disadvantages Internet Connectivity Issues 2 Device logs are available Network latency 3 Large number of devices are available 4 Tests can be run in parallel on many devices Latest Device availability Issues Over Crowding issues ( device waitlisted) 5 Build Integration Automation Test Failure 6 Incoming call and SMS facility also available 7 Automation test execution is stored as video

27 Cloud Computing - Examples

28 Network Points to keep in mind for selecting real network: Geo-graphical Economical Real Cellular Network Testing should be done on some of the selected devices and selected operators. User Experience - basements

29 Agile Testing Agile Testing Approach automatically handles most of the problems faced in Mobile Testing: Consider the example where a new ios has been introduced, then changes for that ios can be included as early as next sprint onwards. Agile allows the developers to change course on a moment s notice.

30 Data Security Make sure to include this in your testing scope: Does your mobile app store saves information properly? What about data deletion or worse unintended data deletion? App does not allow unauthorized access to personal information.

31 App Store Guidelines App Store Guidelines Instead of being the last mile in Testing, should be followed through the project life cycyle Using Apple s graphics in your own apps Icons Don t use private APIs Avoid copying core functionality of the OS s pre-installed apps If you re collecting user information, be sure to note it somewhere in your app No Offensive Language

32 User Reviews on App Stores

33 Mobile Testing Tools mobiready Android Developer Stat Counter Appium BrowserStack Robotium Instrumentation Responsive Crash Reports Console Logs

34 Conclusion Despite of all the challenges mentioned if we follow these best practices then we can ensure a cost effective and Comprehensive mobile testing process. Follow App Store Guidelines and Standards Target Device Selection ( using a mix of real devices and emulators) Network Throttling Use of Cloud Computing Solutions Taking Care of Data Security Appropriate Testing and Automation Tool Selection Be Agile and Follow Agile Maximizing Automation wherever possible Applying traditional best practices and methods of Desktop Testing to mobile Testing

35 picture by -bast- Q & A

36 References ndex.html ode/unittests/introduction/intro.html ation/ides/conceptual/ios_simulator_guide/testi ngontheiossimulator/testingontheiossimulator.ht ml

37