Testing the Mobile Application's Performance

Size: px

Start display at page:

Download "Testing the Mobile Application's Performance"

Silas Bryant
8 years ago
Views:

1 Testing the Mobile Application's Performance Case Study on Communicator Mobile Rama Krishna Pagadala Microsoft Corporation

2 Agenda Performance Metrics for Mobile Applications Communicator Mobile 2007 R2 Product Performance Strategy Performance Testing Strategy Architectural improvements Takeaways from this case study

3 Performance Metrics Time - UI Response Time (Mobile) Users tend to be less patient Time Battery Network Memory & CPU Binary size Very demanding

4 User: Download is too slow! How can we make the download faster? Better compression, faster connection, smaller file size, etc. Operational Management Most compelling and tangible, but expensive How can we make the download appear faster? Provide progress indication, diversion, etc. Perception management. Inexpensive, effective for mainstream usage context How can the download be more tolerable? Offer completion notification, provide emergency exits, etc. Tolerance management. Best remedy when perception is rigid. Inexpensive Designing and Engineering Time by Steven C. Seow

5 Performance Metrics Battery critical Time Battery Network Memory & CPU Binary size Drains non-linear Mitigations Minimize the radio wakeups System events instead of polling

6 Cell Radio (Data)

7 Performance Metrics Time Battery Network Bandwidth Low speed networks Pay-per-use ($) Roaming ($$$) Network Memory & CPU Binary size Impacts Time User Experience Battery

8 Performance Metrics Time Battery Network (no Virtual) Memory & CPU Device sluggishness Responsiveness Application crashes Memory & CPU Binary size

9 Performance Metrics Binary size (footprint) Time Battery Network Memory & CPU Application load time RAM usage Download/Install time Binary size

complements other UC endpoints Contacts, Presence, Work

10 Communicator Mobile 2007 R2 Communications application for enterprise users Always-on Completes UC* story and complements other UC endpoints Contacts, Presence, Work calls, Conferencing, Search, Windows marketplace *Microsoft Unified Communications

11 Product Strategy Customer feedback for CoMo 2007 Other communication apps Personas 3 Power Users Performance goals Ex: Login < 10 seconds Design for Performance

Testing Strategy User Profiles Baseline/Email Power

12 Testing Strategy User Profiles Baseline/ Power User User Profiles Scenarios Tools Test Execution Reporting

13 Testing Strategy User Profiles Communicator Mobile Power User User Profiles Scenarios Tools Test Execution Reporting

14 Testing Strategy User Profiles Scenarios End-to-End scenarios (User centric) Script Specific scenarios (Dev/Test teams) Scenarios Tools Test Execution Reporting Login Send/ Receive messages Work calls / Conf Logout

Reporting GPRSMon (network/application level) In

15 Testing Strategy 3 rd party Tools PowerMon (device level) User Profiles Scenarios Tools Test Execution Reporting GPRSMon (network/application level) In house Tools NetMon (network level) Memory/CPU (per process)

16 Testing Strategy User Profiles Scenarios Tools Test Execution Daily/Weekly? Manual vs. Automated? Test configuration: PocketPC vs. SmartPhone? 3G vs. Edge?, Test Execution Reporting Debugging Is hard Inconsistent issues Logging impacts the performance Standard variation

17 Testing Strategy User Profiles Scenarios Reporting Weekly report ( ) Trends, charts Automated results (daily) Product team analysis Tools Test Execution Reporting

18 Test Matrix Wide array of devices Carriers Networks - 2.5/Edge/3G Cellular Data vs. WiFi Platform intricacies (WM 6.x power management)

19 Architectural improvements Poll vs. Push Presence updates Contacts in view Collapse Contact Groups improves sign-in time! Piggy back the data Keep-alives with Presence updates ActiveSync ( /Calendar)

20 Takeaways Performance is key to mobile application's adoption Think/Design/Test early on Set clear goals for select-few targeted users Performance testing is key to is mobile challenging application's mobile adoption platforms Think/Design/Test Metrics are inter dependent early on Set Tools, clear variety goals of for platforms, select-few targeted users App usage patterns are quite different +/ % Variation External factors affecting perf: signal strength, carrier networks, Network type (3G/HSPDA),

21 Takeaways Performance is key to mobile application's adoption Think/Design/Test early on Set clear goals for select-few targeted users Performance testing is challenging on mobile platforms Metrics are inter dependent Tools, variety of platforms, Usage patterns are quite different +/ % Variation? Consider External factors affecting perf: signal strength, carrier networks, Network type (3G/HSPDA), Manual (UI) vs. Automated (UI/API) Absolute (engineers) vs. Perceived (users!)

22 Credits Communicator Mobile Product Team Windows Live Messenger Team Designing and Engineering Time: The Psychology of Time Perception in Software by Steven C. Seow Power Monitor SPB GPRSMon

23 Thanks! Questions

Best Practices for Performance Testing Mobile Apps

Best Practices for Performance Testing Mobile Apps Lee Barnes, CTO Utopia Solutions Apology I don t believe in best practices I do believe in guidelines and better approaches for a given situation Key