Mobile Application Quality

Transcription

1 Mobile Application Quality CHALLENGES, APPROACHES AND STRATEGY An InfoStretch White Paper July 2013 Elevating confidence 3200 Patrick Henry Drive, Suite 250 Santa Clara, CA InfoStretch Corporation. All rights reserved

2 TABLE OF CONTENTS To Summarize The Mobile Boom is Here to Stay Mobile Testing Could be a Rocky Road Designing the Right Testing Strategy for Your Enterprise What You Need to Know about Different Testing Types A Sneak Peak into Automation In Conclusion InfoStretch Corporation. All rights reserved 2

3 TO SUMMARIZE The mobile market is booming, the app economy is exploding and the need for mobile testing continues to grow. But mobile testing is not easy. It is very different from traditional enterprise, web and packaged application testing. What makes it different are the various devices, numerous form factors across these devices, fast changing mobile OS, unpredictability of simulators and growing customer demands to support mobile applications globally. Each phase of mobile quality like planning, definition, execution, automation, release and sustenance brings its own set of challenges to be addressed. For example, in the planning phase, there needs to be a focus on device selection. In the definition phase, types of tests and processes need to be well defined. In the execution phase, remote access and conditioning issues for devices need to be covered. In the automation phase, one needs to account for ROI, tools, types of test cases etc. And in the last phase of mobile quality, post release support and upgrades etc. need to be measured and addressed. But to do all this, the overall testing strategy needs to be well defined with the right tools, framework and technology. This white paper provides a detailed view of how mobile testing is different, how an optimal mobile testing strategy can be created with risk mitigation plans and optimized processes. This paper also takes you through the various types of testing; the attributes that one needs to look out for when performing these tests, and the types of tests that become important based on a specific business scenario. It briefly talks about mobile automation and how it can be leveraged in the overall mobile solution for accelerated time to market initiatives InfoStretch Corporation. All rights reserved 3

4 The Mobile Boom is Here to Stay The mobile landscape is changing real time - new devices, new versions of operating systems, new carriers and of course, new applications. The challenge is to make your application work and work well across this constantly evolving ecosystem InfoStretch Corporation. All rights reserved 4

5 KEY TAKEAWAY The Mobile Economy Will Continue to Explode! This means: Need for global testing coverage Need for cross-platform testing capabilities Need for ongoing support and user experience management 380K /4 Billion 10+ Billion 435K The ios app store is going to have 435,000 new apps in 2013 Google Play s revenue were up 311% overall in 2012 Three quarters of a billion ios and Android devices across 220 countries generated over $10 billion in revenue in 2012 In 2012, the market for paid application downloads reached $8 billion in the top five app platforms, up 27% from 2011 ~ 54Billion Mobile application market size is going to be $54 billion by 2015 Forrester 2013 InfoStretch Corporation. All rights reserved % 27% 8Billion

6 Mobile Testing Could be a Rocky Road Mobile testing is different from traditional enterprise QA. With a zillion device/os combinations challenging testing coverage, security issues across enterprises around apps and varied connection speeds across various devices altering the performance of apps, mobile testing does become the largest elephant in the room InfoStretch Corporation. All rights reserved 6

7 There are some basic attributes within performance, compatibility, security and network connectivity where the traditional enterprise QA principles differ from mobile QA. Apart from adding device provisioning and device level validations, there are issues related to memory, network bandwidth, cross platform, screen size and resolution compatibility. There are functional issues related to swipes, UI and localization. There are security concerns such as caching, uploading personal information to servers, virus scanning, memory corruption etc. All this makes mobile QA very different. Below is the list of the different phases of mobile quality and the various attributes within each phase that needs to be addressed to create a solid mobile strategy. Phases of Mobile Quality Attributes in Consideration 1 PLANNING User Profile Device Selection Do we really know end users? Which OS/device combination is ideal for my app? 2 DEFINITION Types of Tests Processes Which ones provide me high coverage? How do we manage all the testing processes? 3 EXECUTION Devices Remote Access Conditioning How do we acquire and maintain them? How do we do this securely and exclusively? How do we expose the device to potential defects? 4 AUTOMATION Automation Is there ROI? Which tools will ensure required coverage? 5 RELEASE AND SUSTENANCE Post Release Support Identifying Defects End User Feedback & Usage How do we manage the new OS s and devices How do we respond and identify defects to ensure High customer satisfaction? How do we measure real end user-experience? 2013 InfoStretch Corporation. All rights reserved 7

8 Mobile Testing Challenges At A Glance INDUSTRY CHALLENGES BYOD Mobile Device Management Enterprise Level Security Testing on the Cloud Consumer and Enterprise Mobility Native vs. Hybrid vs. mweb vs. HTML5 Mobile Test Automation, Security Tools NETWORK SERVICE PROVIDER Network Delay and Jitter Network Disruption Testing Network Performance Testing Network Congestion Testing Multi Network/Trace Testing Enterprise Carrier Network Switching NETWORK CONNECTIVETY AND VARIYING SPEED 2G, 2.73G (GPRS, EDGE), 3G, 4G WiFi (Secured, Unsecured) WiMax VPN, Hotspot, Tethering Cellular Internet, Telecom Carriers Data loss over network Bandwidth, Signal Loss DEVICE DIVERSITY Device Makers, Models Smart Phones, Phablets, Tablets Device Density (ldpi, mdpi, hdpi, xhdpi) Device Screen Sizes (Small, Normal, Larger, Extral Large Screens) Device Orientation Challenges Device Browsers, Rendering Differences DEVICE OS FRAGMENTATION ios (3, 3GS, 4, 4GS, 5.x, 6.x) Android (2.x, 3.x, 4.x) Blackberry (4.x, 5.x, 6.x, 7.x) Windows Phone (2003, 5.x, 6.x, 7.x) Symbian, BREW, J2ME, BADA, others DEVICE HARDWARE Limited Processing Capability/Power Locations and Languages Limited Memory size Availability of external memory SD card Device Comm Protocols (WAP, HTTP) Device Configuration DEVICE DATA INPUT METHODS Touch Screen Keypad Non Touch, Soft/Physical Keypad Track Balls, Qwerty Keypad Voice (SIRI, Skyvi) Split Keypad Copy and Paste Blink of an eye DEVICE CAPABILITY & APP FUNCTIONALITY Bandwidth Connectivity Camera Phone/SMS/MMS Contact list OTHERS Rapid Rollouts, Deployment Velocity Mobile Context, Usability, Performance Monitoring for energy consumptions Testing with Emulators/Simulators Device Interruption Scenarios Field Testing, Test Execution Challenges 2013 InfoStretch Corporation. All rights reserved 8

9 KEY TAKEAWAY Mobile Demands a Comprehensive Testing Strategy This means: Interaction with Device/OS native capabilities Optimal combination of devices and OS Effective combination of manual and automated testing Need for faster agile adoption On-time and off-line connectivity A Mobile Native App is Different... The strategy to test a mobile application depends on the type of application we re talking about. If it is a native application, then there are certain conditions for which it needs to be tested. The tester needs to ensure that the app can be successfully downloads and executes on the device and interacts with the backend infrastructure. Apps render differently across devices, so it needs to be thoroughly tested on all relevant ones.there is also a need to ensure backward compatibility with selected older generation of devices and OSs. Creating test plans for manual testing on each of these devices can be a cumbersome process. From Mobile Web and Hybrid Mobile web applications pose a different set of challenges. These applications are viewed by users across the globe and that presents its own challenges in terms of networks, scripting and devices. No matter what type of app it is - native, hybrid or web, manual testing of these apps is more rampant. Mainly because testing on simulators does not completely request real user experience. And, testing on all device/os combination is not practical. The other issue which we have already touched on is the lack of a standardized mobile SDLC, which results in an inconsistent testing methodology InfoStretch Corporation. All rights reserved 9

10 Designing the Right Testing Strategy for Your Enterprise A well defined and executed mobile solution is critical to the success of any business. And defining the right strategy requires time tested expertise supported by the right infrastructure and enabling technology InfoStretch Corporation. All rights reserved 10

11 TESTING STRATEGY PLANNING DESIGN Mobile Testing Methodology Having a right mobile testing methodology is critical for your business and you need a partner who understands intricate details and brings required knowledge as well as infrastructure ANALYSIS REPORTING ENVIRONMENT EXECUTION TECHNIQUES METHODS 2013 InfoStretch Corporation. All rights reserved 11

12 KEY TAKEAWAY 2013 There is a Need to Establish a Link Between Business and Quality Goals This means: Understanding the quality dimensions Aligning them with a tactical testing plan Selecting the right service partner Quality Dimensions for Mobile Testing PERFORMANCE INTEROPERABILITY CONFORMABILITY RELIABILITY USABILITY MAINTAINABILITY InfoStretch Corporation. All rights reserved

13 Deliverables from Mobile Testing Activity: TEST STRATEGY Define key policies, components, review criteria, approach, testing types, scope, communication and resource planning necessary to control and manage your risks throughout the Software Testing Program. TEST PLAN Address key areas such as test approach, scope, test types, review criteria, test data, incident management, resources and supple mentary materials. TEST CASE / SPECIFICATION Identify and outline key objectives of the test validation, related requirements and testable items. All required test pre-conditions and processing will be detailed and present with associated supplementary materials. TEST SUMMARY REPORT Provide dynamic snapshot of progress throughout the software testing life cycle, providing visibility to management and stakeholders. The overall summary, scope deviation, requirements coverage, analysis/design progress, execution progress, review criteria and out standing defects are areas that will be addressed throughout the life of the project for review, approval and sign-off. TEST AUTOMATION Define repeatable process and help you streamline your testing to support on demand releases and critical software changes. A hybrid automation framework is deployed for immediate use, eliminating development ramp-up time. TRACEABILITY MATRIX Provide a relational model of your software testing validation framework. Requirements and User Stories are mapped to Test Cases and Test Plan and are executed against target software build releases, configurations and environments. TEST RESULTS Each of the test result is well supported with tester comments and screenshots along with the required logs for the developer to under stand the issue, and be able to reproduce the same without any trouble. TEST ASSESSMENT Provide an overall review of the efficiency and effectiveness of testing activities. Assessment methodology uses interviews with key testing leaders and an analysis of your testing deliverables in a short time frame to provide recommendations InfoStretch Corporation. All rights reserved 13

14 What You Need to Know about Different Testing Types There are different types of testing that need to be addressed independent of the platform to ensure the mobile app quality. Here s a snapshot of the various types of tests and test cases InfoStretch Corporation. All rights reserved 14

15 FUNCTIONAL TESTING Scenarios Sign-up & Login Menu Options Keys Data Handling Connection Speed / Carrier Description Make sure that the application is easily accessible to the users. Test to make sure that menu items like Help, about, etc. are easy to find and navigate. Make sure keys functionality is clear and consistent. Also, we assure that to cross-check if the app will run with a keyboard and/or touch-screen. Make sure to manage data handling (important for games in particular) and include this in the testing scope. Make sure o take care of data deletion or worse unintended data deletion? We make sure to include this in our testing scope. Test to make sure that the most basic app tests include some of the most popular and likely carriers, as performance can vary greatly from one carrier to another. Operating System Screen Size Make sure to test the mobile OS running the app as it will have a huge impact on the relative performance. While this applies mainly to tablets, screen size discrepancies should be a top consideration while testing mobile app. Interruptions Make sure to understand how the mobile app would behave when the device battery is at full strength, medium strength and low strength? This is just one of the many variables to consider when testing, but it s also one of the easiest items to check for. Others include incoming calls, SMS, MMS and video calls. Error Messages The error messages should be clear, concise and actionable InfoStretch Corporation. All rights reserved 15

16 Scenarios USABILITY TESTING Description Connectivity effect Small screen size effect Different Display Resolutions effect Check for the slow and unreliable wireless network connection with low bandwidth is a common hindrance for mobile applications. Effect of various network conditions is taken into consideration Connectivity testing. Check for physical constraints of mobile devices, especially small screen size, these can significantly affect the usability of mobile applications. Low resolution can degrade the quality of multimedia information displayed on the screen of a mobile device. Check for low resolution that can degrade the quality of the multimedia information. Test Case LAYOUT AND DESIGN Description Page Layout Color Schemas Findability Localization Touchscreens Trackballs, track wheels and touchpads Soft Keyboards Hard Keys Short cuts Evaluate the overall effectiveness of the page layout. Evaluate use of colors in the background, text, links, icons, buttons. Evaluate accessibility of key items like Help, About, Instructions, search. Evaluate that the app is consistent in terms of text, message and symbols in all supported languages. Make sure the test cases cover all different rouchscreen scenarios Multi-touch vs single touch, long touch vs short touch, button size, workflow, other input methods if supported. Confirm that the user is able to move from object to object using this and screen navigation is as painless as possible for the user. Check user interaction with the keyboard eg: if the user s select to enter some text the soft keyboard automatically appears. Check the device available hard keys Eg: Start, Home, Menu, Back interact with the application in similar way. Check if there are any expected shortcuts for the device and test their use within the application InfoStretch Corporation. All rights reserved 16

17 Test Case LOCALIZATION TESTING Description General localization testing Check for user interface, content files, keyboard, text filters, hot keys, spelling rules, sorting rules, upper and lower case conversion rules. Format testing Check for date formats, currencies as per country Local market compliance Check if it complies with the local laws and regulations. Content testing Check video content, static and dynamic content like catalogs, search results, metadata. Characters Check the different languages which have different character sets. Postal codes Verify if the target countries support only number alphanumeric. Language Direction Verify the direction for the way language is written left to right or right to left. Currency conversion Test for currency conversion for the internet retailers applications. Tax calculation verification Test the rules around the calculation of VAT, sales tax. Translation validation Test for on-demand translation validation and accuracy. For companies that have global footprint, it coves accessibility from any market location InfoStretch Corporation. All rights reserved 17

18 SECURITY TESTING Web applications Format testing Local market compliance Make sure that applications are tested wit coomon web security and testing tools such as Firefox web developer. Make sure that web proxy is used to intercept all mobile device traffic to monitor data and test for security issues. Make sure to test for password/usernames being left uncrypted on device is a real one that can taken advantage of by other users. Make sure the footprint and fingerprint analysis can be used to make hash files. Make sure to research the permissions configuration on the device and design testing to ensure that your application can get access to the device areas it needs in order to function properly, and it provides useful error messages directing the user to set the permissions properly if it cannot. make sure to verify that your application can be successfully installed at all expected device security level settings. Make sure that after installation, it is possible to change the security settings of your device or application permissions. You should include some test cases to cover these scenarios as well. LOAD AND STRESS TESTING Test infrastructure management and reporting Make sure there is enough research on application to decide upon the test infrastructure on live or simulated load. Make sure to feed the mobile app under test with the largest tasks it can operate with. Make sure to increase the load on the system via automated tools to simulate real time scenario with virtual users. Make sure to report management using internal and external bug reporting and test management tools. INTERRUPTION TESTING Incoming calls and SMS, battery and cable removals Make sure to test for: Incoming and Outgoing SMS and MMS Incoming and Outgoing Calls Battery Removal Cable Insertion and Removal Network Outage and Recovery Media Player On/Off Device Power Cycle 2013 InfoStretch Corporation. All rights reserved 18

19 A Sneak Peak into Automation Because mobile testing must be performed on many different combinations of devices, browsers, and operating systems, it is time-consuming and expensive. Test automation can bring efficiencies to this process by reducing the costs and accelerating the go-to-market plan InfoStretch Corporation. All rights reserved 19

20 Why Automation? The Device OS Maze - The virtually endless combination of devices, OS models, and device sizes mean that in order to cover as much as possible and the same functionality is ran on different devices. The Changing Industry Trends - The industry is constantly changing with OS updates, upgrades, new devices, and new manufacturers, etc. In order to ensure that the application is up-to-date and supported by new trends, testing it in a timely manner becomes important. The Rapid Upgrades to Applications - Applications are regularly and rapidly updated with new features in order to keep up with demand. In order to introduce upgrades into the market quickly, testing using automation is crucial. Approach to Mobile Automation Simulate device request using browser proxy plug-in leveraging user-agent string and other device profiles Ability to run manual/automated test for functional testing BROWER PLUG-IN NATIVE API/ TOOLS/SIMULATORS Leverage ios and Android APIs and tools Ability to run automated test for functional and regression testing Defined a data driven test automation framework using remote device access tools, native tools and traditional test automation tools. END-TO-END AUTOMATION REAL DEVICE AUTOMATION Setup private cloud with handsets Created virtual function library to increase reuse and reduce script maintenance 2013 InfoStretch Corporation. All rights reserved 20

21 In Conclusion Mobile testing can be complex but mobile app quality is critical to stay current and successful. The traditional rules of testing do not apply to mobile due to the unique features of mobile devices, limited bandwidth, unreliability of wireless networks as well as other changing contexts in mobile InfoStretch Corporation. All rights reserved 21

22 There is a need to develop mobile specific test methodologies and guidelines and therefore a need to rely on a testing partner. This partner would provide insights to modern approaches in testing and would suggest the right strategy and coverage based on business needs. After defining the strategy, a testing partner like InfoStretch would provide a well-defined plan on what types of testing and what attributes need to be tested, validated and verified. InfoStretch has executed testing strategies for enterprises across many verticals retail, travel, banking and insurance, and digital entertainment. It has acquired the necessary know-how and intelligence with enabling technology, best practices and experts to provide top class service to customers. InfoStretch s testing services are enabled by its private mobile cloud with more than 3000 real and pre-market devices in the cloud and servers distributed across several geographies. What should you be looking for when selecting a vendor? The ability to use the right combination of emulator and real device testing to offer predictability. Emulators with scripting help to test many aspects at an effective cost The ability to provide intelligence behind use of select devices for your application The ability to provide automation and agile solutions for faster time to market by building the right user stories and increasing the sprints and creating script reusability The ability to have access to the real devices remotely when you cannot have a slew of mobile device to test on physically The ability to increase the star rating of your app by closely monitoring it and providing reactive and proactive remediation The ability to offer sustenance related services for your application post-launch 2013 InfoStretch Corporation. All rights reserved 22

23 ABOUT INFOSTRETCH InfoStretch is a leading provider of mobile and enterprise QA services and solutions. Our offerings range from enterprise QA, mobile application development, testing, and automation to certification and sustenance. We ve been providing expert solutions and services to various large and strategic players in the mobile ecosystem including enterprises, carriers, app stores, VAS, and ISV s to actualize their mobile and QA strategy very successfully. We offer the most cost effective, secure, and scalable models to service our clients across various verticals such as banking, healthcare, education, and entertainment to name a few. Our innovations have allowed us to provide an SLA of 24 hours and the ability to scale and test over 2,000 apps a day. CONTACT US SANTA CLARA OFFICE InfoStretch Corporation 3200 Patrick Henry Drive Suite 250 Santa Clara, CA GENERAL INQUIRIES e: info@infostretch.com p: (408) f: (408) All registration marks are the property of their respective owners InfoStretch Corporation. All rights reserved