Contents. Executive Summary 3. A Summary of Mobility Challenges 4. Mobile Platform Capability: 6. Mobile Functional Testing 8. Mobile Performance: 10

Transcription

1 by Laura Paterson

2 Contents Executive Summary 3 A Summary of Mobility Challenges 4 Mobile Platform Capability: 6 Mobile Functional Testing 8 Mobile Performance: 10 Summary 11 2

3 Executive Summary The increasing number of mobile platforms and mobile application usage all over the globe is dramatically affecting the way we do business. A study done by emarketer in 2014 predicts that there will be close to 2.16 billion smartphone users in 2016, which is a 12.6% increase from the predicted 1.91-billion smartphone users in They also estimated that by 2018 over one third of the world s population will be using smartphones. As the use of mobile platforms increase they become an integral part of the market platform for many businesses. Many organizations are now being forced to develop applications for mobile use while ensuring that there is no loss of performance or functionality on devices that have limited computing resources. Not only are businesses developing complex applications for devices that can have limited capabilities, they also have to keep up with the ever increasing demands and expectations of their consumer base. Mobile applications are expected to run as efficiently, if not better than, the Web and Desktop-based applications that organizations have become comfortable with. In this day and age people expect the same usability and performance for their business applications on devices that are generally used for fun. These challenges are also compounded by the ever-changing scape of mobile platforms, as new devices, browsers, operations system versions and form factors continue to multiply. Mobile applications are swiftly becoming critical to enterprise operation and the unique set of circumstances found in the market present new challenges and risks, and prove that a comprehensive and successful mobile application development strategy is more important than ever before. 3

4 A Summary of Mobility Challenges The major issues encountered in mobile application development are unique and can be summarized as below: Interface Issues: Mobiles have limited computing resources when compared to PCs and this means less processing power, smaller screen sizes and less memory capacity. Growing Expectations: Expectations have increased dramatically when it comes to user experience and usability. Users now expect mobile applications to be simple to use, reliable and problem free over a variety of different devices and platforms. Mobile Device Proliferation: There is already a massive variation of mobile devices and their configurations are ever growing and evolving. This makes it impossible to develop and test every device on the market. Bandwidth Variations: Performance and functionality of a mobile application can be affected by lower bandwidths of some mobile connections. Mobile Carriers and Manufacturers: Mobile devices are often created by manufacturers in partnership with certain mobile carriers that could mean any number of preinstalled and native devices that could affect the functionality of your application. Interruptions: On a mobile device there are a variety of interruptions such as phone calls, reminders, notifications and low battery. Application development needs to take these interruptions into consideration and ensure they don t affect performance and functionality. Security and Privacy: Mobile applications need to be developed with both hardware and software mechanisms that protect the user s private details and offer a secure service. Many businesses have a limited amount of resources when it comes to dealing with the challenges in developing a mobile application- given this- how do you begin to assure the quality of your mobile application in this market? This white paper covers the essential building blocks of a broad and effective QA testing strategy that should be employed to successfully test newly developed mobile applications. 4

5 Critical Aspects in a mobile QA and Test Strategy There are eight critical dimensions that make up any mobile QA and test strategy: Platform Compatibility Performance Functionality Integration Usability and User Experience Skills Test Automation Network Variances When creating a comprehensive QA and mobile test strategy it is imperative in each stage that you keep the above critical dimensions in mind. The sub-sections below discuss these critical QA and test strategy dimensions: 5

6 Mobile Platform Capability: There are many different types of mobile applications and it is important that when you begin to test your mobile platform capability and mobile functionality that you bear in mind the type of mobile application that you are testing: Pre-Installed Mobile Applications: Every mobile device comes with some pre-installed applications and programs that are built into the device and its operating system. The most common pre-installed mobile applications that are found on devices include SMS, ringtones, , internet access, calculator, contact/phonebook, camera and video recorder etc. You need to make sure that you application is functional in both singular and spontaneous use with the other pre-installed applications on the device. Native Mobile Applications: These applications are similar to pre-installed mobile applications where they may be inherently manufactured into the operation system of the device, but they are also applications that can be downloaded from a variety of different mobile application stores such as Apple App Store and Android Market. Most of these mobile applications will be tested for their inbuilt functionality and may also be tested by their intended store, and will need to pass certain criteria and submission guidelines to be accepted. These applications are situated in the device and will generally work in conjunction with the other inherent features such as contacts, and camera. Mobile Hybrid: These mobile applications reside in the device in the same capacity as a native application but will also have access to the Internet and require a higher degree of user interaction. Mobile hybrid applications are extremely popular and some of the most renowned are applications such as Facebook and Skype. Mobile WebApp: WebApps that are mobile-based need Internet access to function. These applications are different to pre-installed and native mobile applications in that information and codes are stored on a server and not on the device. These applications are popular with consumers as they can be used across many different platforms, but this does mean that thorough testing has to be done to ensure crossplatform compatibility. These applications require a regular stream of updates to keep them at optimal functionality. Mobile WebSites: You don t only have to test standard websites on mobile devices, but also sites that are optimized for mobile web use, as they need to conform to Wireless Application Protocol (WAP) and HTLM5. TOOLS USE EXAMPLE BATTERY CONSUMPTION Simulate low battery power Windows Mobile Emulator SCREENSHOT Screenshots from the device itself DDMS from Android RANDOM EVENT GENERATORS LOG CAPTURES Simulate keys being pressed randomly Capture log files to understand what is happening with each event Monkey from Android Logcat for Android NETWORK SPEEDS Measure network speed Speedtest by Ookla DATA ENTRY TOOLS Simulate data entry Zap-fix LOCATION CHANGE Teleport your phone to any place inthe world Fake GPS Location MOBILE NETWORK SIMULATE Simulate network conditions Facebook Augmented Traffic Control SYSTEM INFO CHECK Provide all information about your device - hardware, operating system, processor, memory, GPU, network interface, storage and battery, including OpenGL powered visual representation in real time System Monitor Ultimate SCREEN RECORDER Record all operation on mobile screen RecScreen 6

7 No matter the type of application that you are testing- there are a few characteristics unique to mobile application development that need to be regarded: various screen resolutions and how they affect rendering and navigation, user interface, privacy and security, optimum performance and browser/platform compatibility. Functional testing should be done according to the different types of mobile application that is being developed but regardless of the type of application there are two essential characteristics that need to be tested on all mobile applications: Interruption Testing: One of the unique challenges faced in developing a mobile application is to ensure that the various other applications on the device do not interfere if they are in use. Users are doing many things on their devices and applications need to be able to work simultaneously with other applications that the user might open. Interruption testing should be done with voice calls, SMS, low battery, memory shortages and reminders etc. Key Press Testing: Your application s stability should be tested by ad hoc testing, which imitates random keypad entries. Accidental touches and key presses can happen in any number of environments and these should be simulated and tested for your application. Like when testing any software application, there are a number of tools and services that you can use to augment your efforts. Almost all Software Development Kits will provide you with tools, but there are many extra tools that can assist you and make the testing process more efficient. For example: Windows Mobile Emulator is a great tool to simulate low battery power that will enable you to test for battery consumption and network speed is easily measured and tested through the Speedtest by Ookla. 7

8 Mobile Functional Testing Mobile applications must be congruent with the native and preinstalled features on mobile devices and they should not interfere with any of the device s inherent functions. There is a proliferation of mobile device/os platforms on the market and each comes with its own set of abilities and limitations, therefore an effective mobile QA needs to take into account and test across all these platforms to distinguish between application flaws and platform restrictions. It s imperative to create a strategy that will prioritise your testing sequence based on market share of certain devices or the objectives of your particular products. This prioritisation is necessary because depending on the type of application combined with network capabilities and ever changing OS versions- it is possible to have a choice of 3 or 4 testing options. There are three main options for testing functionality acrossplatform: Real Device Testing Mobile Device Emulators Remote Device Access With the large number of different combinations and configurations it is highly unlikely that you will be able to test your application using only one system and it is advisable that you use a combination of test devices, remote devices and simulated devices in addition; to comprehensively test your application. This means that you should use real, emulated and remote devices in your functional testing to ensure that your application is functional across the board. Using Mobile Device Emulators In an ideal world- all mobile applications would be tested on the real devices as they represent the authentic software and situations experienced by both the device itself and the end user. However because the number of mobile devices continues to grow and new platforms and network capabilities are constantly in flux- it is not feasible to test mobile applications on every device that they could be used on. This is not just because of the cost of acquiring all of these devices but also because of the time constraints in the mobile applications development industry. Device emulators and simulators offer an attractive alternative to testing on real devices. Although using an emulator is often a cheaper and faster option than using actual devices- they cannot be relied upon solely as they do come with limitations and should be using in conjunction with and to compliment testing on real devices. This is by no means a comprehensive list of all the great mobile emulators that can be found on the market, but the following examples are some of the most popular mobile device emulators: Google s Android SDK includes AVD (Android Virtual Device) Apple s ios simulator for ios use on Mac Windows Phone Emulator by Microsoft Blackberry Simulators from RIM MobiReady Responsive PX Screenfly ipad Peek ipadian Remote Device Access Services Simulators are a great alternative to buying all the devices that your application is going to be used on, but while they create a good substitute for economical reasons, they cannot be completely relied upon to comprehensively test and implement a mobile application. Remote Device Services (RDA) provide real access to a device over the Internet, which permits you to load and test your appli- Application SMS VOICE CALL WAP (text/image) WAP (+video/audio) VIDEO CALL 2G G G G

9 cation on an actual device from a remote location. DeviceAnywhere, MobileLabs and Perfecto Mobile are three of the most renowned remote device access services. Options will vary depending on which service provider you choose and some will give you the option of selecting different network carriers to test across. Some of the more advanced providers will provide automation testing features and your service provider will also manage the level of control that you will have over the devices. To create an all-inclusive mobile application testing strategy, you should incorporate real device based testing with both emulators/simulators and RDA. Mobile testers need to focus on the different browser, devices and OS combinations and the available testing tools when creating a test approach. This three-pronged approach is particularly effective when defects are found in just one device rather than across all the platforms- the combination of real device, emulators and RDA will help your pinpoint whether the fault lies within your application or the device that it is being tested on. Network Variances One of the unique aspects of performing mobile application QA and testing is that it includes more than the device itself and the application being tested. Mobile devices are used all over the world, and therefore there are fewer and fewer completely local mobile applications being developed. There are a variety of different network configurations found throughout the globe and these differences could have a huge impact on the performance and functionality of your application. It is important to consider who your end users are going to be when you are developing your test environment and strategy. Different locations and countries will have different network services and variations, for example: if your end users are going to be more prevalent in a country such as Brazil rather than somewhere like Denmark- then you should expect some network variations and limitations that could affect the usability and user experience of your applications. It is important to include network variations such as 2G, 3G 4G, LTE, CDMA, GPRS, GSM, HSCSD, SIM and WAP in the correct test matrix for the base of your target market. Some places may have low network strength or coverage and this can significantly impact the use of your application. Switching networks and negative testing must be a central part of testing for your application, particularly if it is location or GPS dependent. 9

10 Mobile Performance: The performance of mobile applications can be more complex as the parts are fragmented and often badly integrated. Mobile applications depend on three main elements: server, network and mobile device. You need to keep these three components in mind when you are developing your testing strategy. Along with these three elements, you must also examine the functioning and performance of your application on a device that is under limited resource conditions such as low memory or network. Mobile users have extremely high expectations when it comes to mobile application performance and thus performance testing is one of the most important parts of mobile application testing. Some tips for performance testing include: Do lab testing first Anticipate your peak load by basing it off the worst case scenario Test across your user profiles to precisely model the user environment and behaviour Test in different locations Mobile Test Automation Mobile applications are being developed in a market of evergrowing complexity and competitive stress, and while manual testing is a significant part of any testing strategy, sometimes it is just not economically feasible or doable under incredibly tight time constraints. Automated testing provides a cost-effective solution for providing fast, reliable and strong mobile applications. There are many hundreds of automation tools available for automated testing that have been developed over the last few years. The market is constantly changing however and it can be hard to keep up with the latest tools to use in your QA and testing but some of the best tools right now are: Appium eggplant MITE Mobilerunner MonkeyRunner NativeDriver Robotium TestiPhone UIAutomation Integration Depending on the type of your mobile application and what its intended use is, you may have to test it with regards to integration of other devices such as wearables. The last few years have seen a dramatic increase in the development and implementation of wearable devices and this is only expected to surge in the coming years. Nearly all wearables rely on smartphones as their centre of communication and processing. Ant+, WIFI, Bluetooth and NFC are all used by wearables and their functionality is often dependent on it. For example: the newer Garmin watches for exercise have an inherent GPS system that can transfer the user s location, but this is reliant on ANT+ to create a connection with the corresponding smartphone. Almost all the new fitness and health wearable devices are dependent on smartphones to upload data to their commensurate applications. In some cases payment can be done through NFC (Near Field Communications) which means that all interfaces need high levels of testing to ensure that not just performance and functionality are up to scratch, but also that privacy and security are of the highest level. Usability and User Experience This is perhaps the most imperative part of the whole testing process and one that should never be over looked or done in a rush. There is no point in having an innovative and functional application that doesn t transfer to an enjoyable user experience. Applications should be designed and tested with the user in mind from the very beginning. You need to keep in mind that many of your users, while tech savvy, may not be complete technological geniuses and therefore you need to keep your design clutter free and make it simple and enjoyable to use. Your application cannot be perfect for everyone who uses it but it is important for it to be everything for the market that you are targeting. Basically, no matter how useful your application is, or how many functions it has- the bottom line is that if it is too difficult to use then it s not going to be used at all. 10

11 Summary This whitepaper was written to assist you in planning and implementing a comprehensive and successful mobile application testing strategy. It is important that you incorporate all the components covered to ensure that your mobile applications testing and QA is effective. The main components to keep in mind are: Functionality Platform Compatibility Network Variances Performance Test Automation Integration Usability and User Experience Mobile application development has a unique set of risks and challenges attached to it and the process is only made more complex by the exploding number of devices on the market and the constantly evolving technology associated with mobile smartphones. However, if you cover all of the above mentioned components in your mobile application test strategy, then you should have created a plan that will comprehensively cover all the most important aspects of a successful mobile application, which will be a great fit for the devices it is used on and an enjoyable experience for the users themselves. 11