"It's a Phone First! How to Test Your Five-star Mobile Apps"

Transcription

1 BW4 Concurrent Session 11/7/ :15 AM "It's a Phone First! How to Test Your Five-star Mobile Apps" Presented by: Will Hurley Quality and Security Services Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL sqeinfo@sqe.com

2 Will Hurley Quality and Security Services A practice leader of quality management and security for Idea Integration Corporation, Will Hurley has been at the crest of numerous software development waves during his twenty-four year career. Will represented the US Air Force in the DoD s first joint service system architecture, led CMM assessments, and supported the development of the CMMi. After leaving the Air Force, he consulted for industry-leading companies, helping them adopt and apply new development methods. In his current position, Will consults with clients in North America on achieving desired system and product lifecycle characteristics.

3 It s a Phone First! How to Test Your Five star Mobile Apps 2012 Idea. All rights reserved 1 Today s discussion Idea and I Acknowledgements Five Star Mobile Application improving mobile project outcomes and time to market Idea. All rights reserved 2 1

4 Since 1986, we ve helped some of the world s largest and most respected brands thrive in the digital space Idea. All rights reserved 3 The Presenter Will Hurley Practice Executive Quality Mgmt. & Security Idea William.Hurley@idea.com (781) USAF Systems Engineer TBMCS Core Systems Lead Joint Service Architect GCCS USAF Data Administrator Consultant Small Business Owner Practice Leader 2012 Idea. All rights reserved 4 2

5 Acknowledgements This discussion captures mobile testing insights developed at Idea s Global Solution Centers in Nashua, NH and Houston, TX. Idea s insights are supported by more than 10 years of mobile development and testing. Insights are not SDLC or tool specific. Data is updated on the quarter through a verity of sources. ComScore Netbiscuits Harris Interactive OPA Crittercism 2012 Idea. All rights reserved Star Mobile applications 2012 Idea. All rights reserved 6 6 3

6 5 Star Mobile radar chart 2012 Idea. All rights reserved 7 Ecosystem insights Understanding the mobile ecosystem and trends increase test effectiveness and product quality characteristics. o Capture more than 80% of smartphone users with ih3 network carriers. o Capture more than 85% of smart phone users with Android and ios platforms. o Application use is growing faster than browser use. o Android users spend more time on carrier networks. o ios users spend more time on wireless networks Idea. All rights reserved 8 4

7 Ecosystem carrier US Carrier Smart Phone Market Share Verizon 31.60% As of Aug 2012, 114 million Americans own smart phones. 60 percent of all mobile devices acquired in the U.S. are smartphones. Other 15% Verizon market share has grown fastest driven by ios. Tracfone 7.10% T Mobile 9.50% Sprint 10.50% AT&T 26.30% Consider market share when signing contracts for cloud services Idea. All rights reserved 9 Ecosystem oem Other, 15.8% RIM, 6.3% US Smartphone OEM Samsung, 25.6% HTC, 6.4% Motorola, 11.2% LG, 18.4% Apple, 16.3% Samsung Galaxy S III leading 2012 sales may be passed by iphone 5. iphone 4 took the top US device sales spot in iphone 3GS took 2 nd place. iphone 4S took 3 rd place. BlackBerry Curve th place. HTC Evo 4G 5 th place. HTC is the most active OEM for Android smartphones Idea. All rights reserved

8 Ecosystem platform US OS Share 66 percent of Americans ages own a smartphone. Other 0.5% Symbian 0.8% Android 52.2% Apple and Android market share are growing. Market share for all other platforms is shrinking. Microsoft 3.6% RIM 9.5% Apple 33.4% Android devices out number iphones by 3-to-2 in the U.S. market Idea. All rights reserved 11 Platform fragmentation Fragmentation is a real but manageable issue for all platforms. Android faces more issues since it resides in an open ecosystem. Android OS adoption is slower than ios. Trades reporting >½ of all iphone users that can upgrade to ios5 have. ios6 expected to drive largest upgrade population o to date. Crash data are normalized as crashes as a percentage of each application launch Idea. All rights reserved 12 6

9 Ecosystem tablet Tablet Share (Platform) Android 42% Other 12% Apple 46% An estimated 74.1 million consumers are using tablets. ipads account for more than 70% of tablet network traffic. More than 80% of tablet traffic is wireless, not cellular. Tablet adoption is balanced between males and females, trending older with households earning $50K or more. Kindle Fire 28% ipad 2 31% 2012 Idea. All rights reserved 13 Customer Experience insights Mobile users expect more. o Good design is not accidental. Get help if you need dit. o Good design is sticky. o Unstructured or unsolicited feedback has more action value than a survey. o Analytics portray the customer voice Idea. All rights reserved 14 7

10 Experience components How easy is it for your customer to use your application. The application does what your customer wants it to do. How does the application make your customer feel Idea. All rights reserved 15 Customer experience Social net or Blog, 35.70% Music, 24.50% Games, 31.80% Smartphone Usage Text message, 74.60% Used browser, 48.60% Used App, 48.60% Four in ten mobile users are disappointed with applications from their favorite brands. Three-quarters of users surveyed believe a company's mobile app should be easier to use than its Web site. Users spend more time with mobile applications than their mobile browser. 82% of ios and Android Apps crash once out of every 10 loads Less than 5% of mobile apps are still in use 3 months after downloading Idea. All rights reserved 16 8

12 Experience themes Based on conversations, we found several common themes: All purchasing because it s required by their apartment complex. All participants were referred to site via a brochure. Participants pleased overall with the simplicity of the online process Liked being able to select from presented options in order to simplify and speed up the process. Appreciated the ability to get additional details where necessary. A majority didn t use these features. All participants pay annually for their policy Idea. All rights reserved 19 Experience maturity o Functional requirements are only part of the customer experience. o Key moments in customer interaction demand good design. o If you don t have experienced designers on your team bring them in. o Understand accessible and emotional goals at every key moment. o Don t predict how people will respond to a new experience, test it. o Allot the time to make incremental changes on all key moments Idea. All rights reserved 20 10

13 Functional insights It s a phone first. o Test while preforming common smart phone tasks. o It s cheaper, faster and easier to test in emulated environments first. o Testing types are the same as Web based n- tier applications o Manage functional and non-functional insights across projects. o Create a mobile test strategy document. o Reduce tool and process resistance between groups Idea. All rights reserved 21 Manage scope Carriers Operating Systems OEM Devices Com & Sensors 2012 Idea. All rights reserved 22 11

14 Manage knowledge 2012 Idea. All rights reserved 23 Manage modularity Display and Graphics Input Devices Sensors Permissions Intents Phone and messaging Deploy and Upgrade Analytics Your application s key moments 2012 Idea. All rights reserved 24 12

15 Emulation(Simulation) testing Does not require the physical device. Can still use network services. Emulation requires experience with specific platform development tools. Static security and performance testing can be performed on source code. Emulation is best for: Investigating architectures and application layers Agile testing of key features and interactions Evaluating code quality characteristics Research HTML5 characteristics. Planning for automation 2012 Idea. All rights reserved 25 Device testing Does require a physical device. Testing on the device is best for: User Experience the last t100 miles. Acceptance testing working with gestures, sensors and hardware. Soak Testing how will the device perform over time. Provisioning installation and removal. Cloud providers and plugins for popular automation tools support both functional and performance testing. No need to run out and buy all platforms Idea. All rights reserved 26 13

16 Automated testing Browser automation Tools can be used for closest match, browser testing on or off the device. Closest match browser automation possible on some devices Safari 3 for the iphone driven on a PC. User Agent header data to drive compatibility testing logic. Work closely with development team to manage and control Web 2.0 features. Third party libraries can speed implementations. Physical devices and Cloud services can be used to test physical platforms when defects are detected Idea. All rights reserved 27 Automated testing Bluetooth/ Near Field Application automation Tools can be used in emulated environments and on the device. Installation of some type of automation controller is required on the device. Controllers consume resources. Plug in and Cloud support is growing and stable. Cloud services may not provide object recognition. Additional fixtures may be required for sensor (GPS, Accelerometer) and hardware (camera, battery) automation. Applications may incorporate browser engine Idea. All rights reserved 28 14

17 Functional details Create a mobile test strategy document. Tight control of manifest and OS support. Identify development and testing tool stack. Explain how you plan to manage fragmentation OS, device, sensors, UI, marketplace. Test teams should be working in emulated, physical and cloud based environments. Test teams should manage insights across releases. HTML5 platform insights Carrier and Device insights OS / Platform insights Security insights Marketplace 2012 Idea. All rights reserved 29 Security insights Protect your brand. o Address security issues programmatically and start now! o Static analysis is a highly effect approach to address coding issues. o Dynamic analysis, Security Penetration testing, is your last line of defense. o SD memory should be tested 2012 Idea. All rights reserved 30 15

18 Security testing Static Application Security Testing (SAST) Static analysis tools can address unsafe practices earlier in the SDLC. Clang; AppValidator; AnalytiX SAST should be integrated with build and reporting practices. Evaluate application permissions. Don t build on or deliver on compromised platforms. Dynamic Application Security Testing Mobile devices are susceptible to common browser exploits and more. Emulators or devices can be used to ensure applications don t bleed sensitive data on to the device or to the network. Keep up to date with known exploits and add test cases to qualify the impact Idea. All rights reserved 31 Performance insights Protect your brand. o Static analysis is a effect approach to address performance issues. o Dynamic analysis, Security Penetration testing is your last line of defense before delivery. o SD memory should be tested Idea. All rights reserved 32 16

19 Performance testing Static Application Performance Testing (SAPT) Static Analysis tools find issues earlier in the SDLC and improve coding practices. Tools address code that slow application performance. In a 53K line project we found 328 instances of code that would contribute to performance issues. SAPT should be integrated with build and reporting. Performance Testing Load, stress and longevity testing are critical to the customer experience. Load and stress testing are focused on middle and backend service layers. Longevity testing should look at both backend services and the mobile device. Performance tools should support bandwidth throttling and common RESTful protocols Idea. All rights reserved 33 Final thoughts o Cloud based testing services are valuable but o Will not achieve expected ROI without proper investigation. o Lead to Scrumfall application lifecycles. o Can slow delivery times. o Functional, performance, security and test automation should be considered for each mobile project. o Simplify the testing space through active management. o Diagram test environments. o Support agile approaches for all team members Idea. All rights reserved 34 17