DEPLOYMENT. ASSURED. SEVEN ELEMENTS OF A MOBILE TEST STRATEGY An Olenick & Associates White Paper July 2013
Contents Executive Summary... 1 Mobile Vision... 3 QA Methodology and Process... 4 Deployment Process... 5 Information Security... 6 Mobile Development and Test Environment... 7 Test Tooling... 8 Team and Skills... 10 Conclusion... 11 About Olenick & Associates... 11 Exhibits EXHIBIT 1 SEVEN ELEMENTS OF A MOBILE TEST STRATEGY... 2 EXHIBIT 2 MOBILE VISION DECISION POINTS... 3 EXHIBIT 3 QA METHODOLOGY AND PROCESS DECISION POINTS... 4 EXHIBIT 4 DEPLOYMENT PROCESS DECISION POINTS... 5 EXHIBIT 5 MOBILE SECURITY DECISION POINTS... 6 EXHIBIT 6 MOBILE TEST ENVIRONMENT DECISION POINTS... 7 EXHIBIT 7 TEST TOOLING DECISION POINTS... 8 EXHIBIT 8 MOBILE TEST AUTOMATION TOOLS... 9 EXHIBIT 9 STAFFING DECISION POINTS... 10 Copyright 2013 Olenick & Associates, Inc. All rights reserved. 205 W. Wacker Drive, Suite 2000 Chicago, IL 60606 312 332 0096 http://www.olenick.com @OlenickIT Additional copies of this white paper can be downloaded from http://www.olenick.com/ Seven Elements of a Mobile Test Strategy
EXECUTIVE SUMMARY While the importance of mobile applications rapidly grows for both consumers and enterprises, mobile information technology presents significant new risks for IT professionals and their organizations. An explicit mobile quality management strategy can offset these risks. CIOs and IT managers cannot assume that traditional quality practices will achieve the same stability and predictability for mobile apps as they have for traditional systems. Most organizations will have to stretch to design effective mobile apps and then establish entirely new development environments, processes, and players to produce them. But that isn t all that s needed. An effective, organization-wide mobile test strategy can offset the new risks of mobile IT and help to realize the promise of this transformative technology. This white paper offers a checklist for planning your mobile testing initiative: 1. Mobile technology vision 2. Quality assurance methodology and process 3. Deployment process 4. Mobile development and testing environment 5. Information security 6. Tooling 7. Team and skills These elements, listed in Exhibit 1, reflect lessons we ve learned in testing many mobile apps and working with our clients as they do the same. In the following sections, the key decision points for each element are discussed. Your answers to them will outline a mobile testing strategy for your organization. Seven Elements of a Mobile Test Strategy 1
Mobile Vision QA Methodology & Process Deployment Process Mobile Environment Information Security Tooling Team & Skills User needs Business objectives Product roadmap QA methodology QA process Metrics, standards, templates Development process Deployment process Mobile devices and connectivity Server-side environment Maintenance and support process Information security policy Mobile device policy Mobile device management Configuration managmenet Testing and test management Deployment management Staff experience Staff core skill set Training program Exhibit 1 Seven Elements of a Mobile Test Strategy Seven Elements of a Mobile Test Strategy 2
MOBILE VISION A mobile vision defines key business drivers for using mobile technology. The mobile vision defines a general approach to mobile application development and support, a mobile application portfolio, and targeted mobile platforms and devices. A mobile test strategy based on it will be aligned with the company s direction as well as current and future testing needs. Exhibit 2 lists key decision points to consider in defining a mobile technology vision. Users Business Objectives Mobile Vision Product Roadmap Mobile Vision Decision Points 1 What are the business drivers for developing mobile application(s)? 2 What mobile applications are planned? 3 What platforms will be supported and why? 3 How will the mobile applications be developed and supported? 4 Who are the users and stakeholders for the mobile applications? 5 What are the stakeholders success criteria for the mobile applications? 6 What level of risk is the organization willing to accept with the deployment of mobile applications? 7 How critical is the quality of the applications? Exhibit 2 Mobile Vision Decision Points Seven Elements of a Mobile Test Strategy 3
QA METHODOLOGY AND PROCESS Your organization s current quality assurance (QA) methodology and processes should be reviewed in light of how you plan to develop, test, deploy, support, and maintain mobile apps. In response to the urgent need to build and deploy mobile applications, many enterprises are relying on new vendors and technologies. Existing QA methodology and processes should be revised to align your approach these new resources. Mobile projects can be measured, monitored, and reported using existing test management tools, reports, and dashboards. However, some adjustments will probably be needed to get meaningful feedback about mobile app quality. For example, you may want to closely monitor the stars your apps garner in Google Play or itunes, in addition to traditional quality metrics. You may want to closely monitor the stars your apps garner in Google Play or itunes, in addition to traditional quality metrics. QA Methodology and Process Decision Points 1 What is the development process: agile, waterfall, or your own? 2 What are the QA processes that are currently in place? 3 What testing standards and governance models need to be followed? 4 5 Are your QA methodology and processes aligned with the mobile development process? If vendors are being used, are their processes integrated or aligned with the organization? Exhibit 3 QA Methodology and Process Decision Points Seven Elements of a Mobile Test Strategy 4
DEPLOYMENT PROCESS An organization s traditional build and deployment process will need to be extended to support mobile applications. In particular, it is important to define an efficient and scalable build/ deployment process for systems that span server-side components and mobile platforms. A mature mobile application deployment process effectively supported by application development and complete platform testing is vital for: Managing and controlling application updates. Improving the user experience. Increasing user adoption of the application. Reducing application user support. Define an efficient and scalable build/deployment process for systems that span server-side components and mobile platforms. Deployment Process Decision Points 1 How are developers managing and delivering code? 2 Will this process accommodate mobile applications or is a new approach and infrastructure needed? 3 What is the current deployment process? 4 5 Can it be adjusted to accommodate the mobile applications or is a new process and related infrastructure required? For native mobile applications, how will they be deployed to mobile devices in the test and in the production environments? Exhibit 4 Deployment Process Decision Points Seven Elements of a Mobile Test Strategy 5
INFORMATION SECURITY Traditional enterprise applications run on enterprise desktops / laptops and are typically secured following an enterprise IT security policy. The expansion of enterprise applications into the mobile space and flexibility now given to employees to connect with mobile devices presents significant new challenges to information security. Consider security policy, implementation, and verification early in the mobile development and test planning process. Support for BYOD (bring your own device) and access to core enterprise applications outside of a secure corporate network creates exposure to many new kinds of attacks. The physical security that fixed location desktop and server computers provide cannot be assumed. Mobile devices are routinely lost, stolen, or surreptitiously compromised outside of an organization s premises. Mobile malware and mobile-specific social engineering now pose threats at the same level as web and desktop exploits. It is therefore essential to consider security policy, implementation, and verification early in the mobile development and test planning process. The National Institute of Standards and Technology provides many recommendations for securing mobile technology. * We see three additional device management policy issues: How will Mobile Device Management (MDM) support authentication, auto-lock, application installation, and remote restore/clean for a fleet of corporate and BYOD mobiles? What is data management policy for data encryption and persistence? What is data access policy including roles, logging, auditing, and remote wiping? Mobile Security Decision Points 1 Does the IT security policy address mobile applications and devices? 2 What are the security concerns associated with the current mobile applications? Future? 3 Will the enterprise applications be allowed on personal devices? Exhibit 5 Mobile Security Decision Points * Related publications may be downloaded from http://csrc.nist.gov/publications/pubstc.html#communications & Wireless Seven Elements of a Mobile Test Strategy 6
MOBILE DEVELOPMENT AND TEST ENVIRONMENT A mobile test environment has different characteristics than traditional desktop and web application test environment(s). Not only does the mobile environment need to integrate with existing systems for end-to-end business flow testing but it also needs to accommodate mobile compatibility and connectivity requirements. Exhibit 6 lists mobile testing environment decision points. We ve found that a hybrid approach is practical. Mobile Test Environment Decision Points Physical Devices Actual physical devices. Device Emulators Mobile Web Browser Emulators Cloud Services Crowd-sourced Services Software that simulates actual devices. Browsers that simulate a mobile browser environment. Mobile cloud service vendors provide fee-based remote desktop access to a wide range of devices and carriers. Crowd-sourced services provide rapid feedback from thousands of users over many devices and locales. Exhibit 6 Mobile Test Environment Decision Points It is usually not possible to test even a small subset of every combination of device type, form factor, platform (OS), locale, and carrier. We ve found that a hybrid approach is practical: provide your own basic set of device form factors and platforms, then as necessary, use cloudbased device sharing and crowd-sourced mobile user experience services. Seven Elements of a Mobile Test Strategy 7
TEST TOOLING Many new design, programming, and deployment tools are needed for mobile apps. A wide range of new mobile-specific tools and services are available to automate functional and performance testing. These tools are typically platform-specific, so any organization developing for multiple mobile platforms (ios, Android, mobile web, etc.) will have to resolve questions of coverage and integration. An incremental approach with proof of concept and pilot projects is a good way to find the best fit. Existing test management tools can support hands-on functional testing and user-experience evaluation. Configuration and process management tools like Jenkins and Jira or their commercial counterparts may be used to good effect for mobile app development. Testing tools newer mobile tools in particular tend to be technology islands. To establish a complete environment, it s useful to consider their fit with your overall tool chain, as Exhibit 7 suggests. Well-defined criteria to evaluate tooling for functional and performance testing can lead to a configuration that best supports your mobile vision. We ve found that an incremental approach with proof of concept and pilot projects is a good way to find the best fit. Tooling Decision Points 1 What tools will be used to support requirements, be they user stories or line items? 2 What tools will be used for test management and bug tracking? 3 What tools will be used to support the configuration management process? 4 What tools will be used to support the deployment process? 5 For automation and performance, what are the current supported platforms? Future? Exhibit 7 Test Tooling Decision Points Seven Elements of a Mobile Test Strategy 8
There are many choices to make once you ve settled on an environment. Exhibit 8 lists mobile functional testing tools that we have used in our mobile testing lab. Borland SilkMobile Tool Hosting Mobile Platforms Testing Supported Desktop Android, ios, Blackberry, Windows Phone Functional Device Anywhere Desktop and Cloud Android, BlackBerry, ios, Windows Phone Functional, Performance Jamo M-uex Desktop Android, BlackBerry, ios, Windows Phone Functional Perfecto Mobile Cloud Android, ios, Blackberry, Windows Mobile, Symbian Functional, Performance Ranorex Desktop Android, ios Functional SOASTA Cloud and local Android, ios Functional, Performance ZapFix Cloud Mobile Web, Android, BlackBerry, ios, Windows Phone Functional, Performance Exhibit 8 Mobile Test Automation Tools Seven Elements of a Mobile Test Strategy 9
TEAM AND SKILLS To define, manage, and implement a robust and scalable mobile strategy, it is critical that the right people are involved and given necessary training. Mobile testing requires strong software testing skills along with a mobile-savvy mindset. We ve found it is equally important to get the right cross-section of users involved. When assembling a testing team, tester background, skill, and testing specialties should be taken into account. If testing will be automated, testers adept in test environment design, development, maintenance, test data selection, and experienced in the selected automation tool should be included in the team. We ve found it is equally important to get the right cross-section of users involved. They can communicate the broadest spectrum of user scenarios to drive effective functional, performance, and usability testing. Staffing Decision Points 1 What level of experience does your staff have with mobile testing? 2 How technical is the existing testing staff? 3 What is the training approach and budget? 4 Do you have a mobile test strategist? Exhibit 9 Staffing Decision Points Seven Elements of a Mobile Test Strategy 10
CONCLUSION CIOs and enterprise IT managers can establish a broad and robust strategy to manage risk as mobile apps are added to an IT portfolio. A comprehensive mobile test strategy can enable your organization to deliver, test, and support high-quality and high-value mobile apps. This can also help to right-size mobile testing budgets and staffing. A mobile test strategy should address process, technology, and staffing requirements. It should guide and empower project teams to plan effective application-specific functional and performance test plans. It should also define an IT organization s development, deployment, and environment management approaches for mobile apps. The seven key elements and their decision points discussed in this white paper provide a framework for such a mobile testing strategy. ABOUT OLENICK & ASSOCIATES We prepared this white paper to help CIOs and IT managers understand how to set an approach for mobile app testing. In our experience, many managers underestimate this challenge. To find out more about how Olenick & Associates can support your mobile testing initiatives, please contact us at 1 (312) 332-0096 or via email at info@olenick.com. To learn more about our full range of deployment assurance services, please visit us at http://www.olenick.com/. Seven Elements of a Mobile Test Strategy 11