A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS. How Best-in-Class App Developers Use Data to Develop Amazing Mobile Apps

Transcription

1 A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS How Best-in-Class App Developers Use Data to Develop Amazing Mobile Apps

2 EVALUATING MOBILE STRATEGY QUANTIFYING GOALS ON MOBILE HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN TESTING ANALYTICS SEGMENTATION 00 CONTENTS PERSONALIZATION COMPARING UX & OPTIMIZATION: WEB VS. MOBILE UX MEETS CRO ON THE WEB UX MEETS CRO ON MOBILE CASE STUDY: TASK RABBIT - WEB VS. MOBILE TECHNICAL DYNAMICS ON THE WEB VS. MOBILE CRO ON MOBILE: ACQUISITION OPTIMIZING FOR APP STORES 06 OPTIMIZING FOR REGISTRATION/KEY ACTIONS CRO ON MOBILE: RETENTION TECHNICAL CONSIDERATIONS PRODUCT CONSIDERATIONS CONCLUSION 2

3 A majority of Americans now own a smartphone of some kind positioning mobile as one of the most significant opportunities for consumer and B2B technology companies. 1 It s no surprise that the app market is growing, with more than one million apps both in the Apple App Store and Google Play Store. How do you make your app stand out from the crowd? The first step is to craft a best-in-class user experience. Optimizely has already built a market-leading platform for creating best-in-class user experiences on the web. With users spending an increasing amount of time on mobile devices, it s become critically important for businesses to be able to deliver amazing user experiences that span the web and mobile. Delivering on this, however, is easier said than done. Many budding app developers will soon learn that while their web and mobile WHILE WEB AND MOBILE USERS ARE OFTEN THE SAME PEOPLE, THOSE PEOPLE HAVE VERY DIFFERENT GOALS AND BEHAVIORS WHEN ENGAGING ON MOBILE VS. WHEN ENGAGING ON THE WEB. users are often the same people, those people have very different goals and behaviors when engaging on mobile vs. when engaging on the web. That s why we wrote this guide to provide you with the most comprehensive and in depth resource to learn: (1) What makes an awesome mobile user experience (2) How to evolve your mobile strategy to create highly engaging products (3) How to optimize that user experience for conversions

4 01 EVALUATING MOBILE STRATEGY With smartphone penetration rates as high as 74%, your company s mobile strategy is more important than ever. Mobile-only companies like Instagram, WhatsApp, and Snapchat have forged a path to billion-plus dollar valuations. Not to mention, today s most innovative firms like Eventbrite, an event management platform, and Percolate, software for content marketers, are investing heavily in developing mobile-focused solutions. 2 What s your mobile strategy? How can your brand make the strongest impact possible?

5 Years ago, a successful mobile strategy meant having an app. But the landscape has quickly changed. Hollywood success stories are becoming less and less common. Just-for-fun apps are everywhere which makes it tough for app developers to stand out. The secret to a successful mobile app is no secret at all. Plain and simple, top mobile apps delight users by 1: making their lives easier and/or 2: providing entertainment. JUST-FOR-FUN APPS ARE EVERYWHERE WHICH MAKES IT TOUGH FOR APP DEVELOPERS TO STAND OUT. Consumers crave elegant user experiences, workflow tools, and highly engaging games. They re looking for resources to be more productive, opportunities for brain breaks, and touch points to stay connected with friends and family. That s why analytics, A/B testing, and conversion rate optimization are extremely important to the app design process. You need to be learning about your users, testing your assumptions, and exploring areas for improvement. QUANTIFYING GOALS ON MOBILE Before building an app, product owners will need to clearly define their revenue objectives. This process starts with identifying key metrics that align with an engaging and profitable mobile user experience. Start by thinking about the metrics that define success on a desktop. Some of the user experience objectives (i.e. user engagement and monetization) will be similar. You ll want your customers to be engaging with the product in a way that drives revenue. At the same time, you need to approach your products goals with a mobile-first mindset, understanding that you re reimagining your products for a completely new device. Pay attention to the following metrics when designing for mobile: 1. DAILY ACTIVE USERS (DAUS): DAU measures your app s engagement levels how many unique users visit the app daily? A high DAU count is a sign that your app is something that people need (or really love). For instance, users might love your game and decide to play it, every day, on their morning train commutes. Users might rely on your app to organize their notes (like Evernote), book vacation rentals (like HotelTonight), and organize their content marketing programs (like Percolate). 5

6 2. AVERAGE RATING & REVIEWS: This metric quantifies what users think of your app and directly impacts performance in app marketplaces. App reviews are also a great place for developers to research user concerns -- which can directly influence your brand positioning and future product releases. So how do you get users to review your app? Well, you ask them. But here s the thing. Simply telling users to leave a great review won t be enough. In addition to prompting users to review your app, you need to think about how to position these requests within your user experience. Most apps will generally prompt a user to leave a rating or review upon launching the app - interrupting the general user flow. A better way to prompt the user is after they have finished doing what they need to do within your app -- or after they ve accomplished Apple s App Store uses app ratings as a factor in calculating Top Chart rankings. It s tough for apps to bounce back from a low ranking -- even if they re downloaded fairly often. something that gives them a sense of achievement (like reaching a high score within a game). For example, FIFA 2012 prompts you to leave a review for the app after you have won a major tournament. 6

7 3. DOWNLOADS: Downloads of an app are the equivalent of an account sign-up on the web and play a crucial role in the app s ranking. It s important to monitor downloads with DAU to ensure that both are trending positively. If you see a spike in downloads, do you see a similar spike in DAU? If not, you may have to revisit your retention features (and usage levels) or test different push notification messages. As an example, the Carrot To-Do app does a great job of using humor in its push notifications to pull users back into the app. 4. RETENTION: Retention is calculated as DAUs divided by number of downloads. It s a key driver for apps with in-session monetization opportunities. Are your DAUs increasing as downloads increase? If not, it is worth digging deeper into why users are abandoning your app. Apps using the freemium model as their main source of revenues must be sure that they have the proper mechanics in place to maximize retention and upselling. As an example, Clash of Clans (a popular mobile game) offers virtual currency and cleverly timed push notifications. Some app companies have even determined the length of time it takes to monetize a user and will even serve ads to competing games when a user has past that time frame and still has yet to monetize. Virtual currency and push notifications in Clash of Clans Carrot To-Do in the App Store 7

8 5. APP CRASHES / FAILURES: While difficult to track, it is important to have crash analytics built into your app. Unfortunately, the default Google Play and Apple analytics do not capture all crashes, so it s important to use an additional crash analytics tool. To circumvent a crash or feature request from turning into a negative review, app developers are implementing feedback functionality so users can provide performance issues or new feature suggestions directly within the app. Negative bug-related app review left by a Tweetbot user For example, Tweetbot, a popular Twitter client, implements a feedback mechanism within their settings page. Tweetbot s in-app feedback mechanism 8

9 02 Conversion HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN rate optimization describes the structured and systematic approach to improving the performance of your website or mobile app through analytics and user feedback. Simply put, it s the practice of running experiments to 1: find out why users aren t doing what you want them to do, 2: test your assumptions, 3: iterate, and 4: test again. CRO is a long-established process that began in the early days of the Internet -- long before smartphones entered the picture. The process starts with understanding your company s unique objectives and key performance indicators (KPIs). Techniques such as A/B testing and qualitative research allow business owners to use data to make informed user-driven decisions. To understand how CRO fits in with mobile, you need to understand its application on the web. Before we jump back into mobile-specific workflows, let s start with a quick refresher by reviewing the following nuts and bolts of CRO. 9

10 TESTING A/B tests allow companies to compare page versions (or elements) against one another. Each page variation is then measured by its effectiveness in comparison to the other. For example, a marketer may compare a page with a green call-to-action (CTA) to a page with a red CTA. A/B tests focus on only one element at a time. ANALYTICS Analytics empower you to assess your current conversion rate, identify trouble areas, and design an execution action. Pay attention to the following metrics: CONVERSION RATE: This metric bridges the connection between website traffic and revenue. Are users taking the action that you would like them to? If not, analyze your conversion flow and start testing the page(s) with the biggest drop off. This metric will help you make sure that users are completing desired goals within your app (making purchases, returning quickly, encouraging friends to download the app, etc.) BOUNCE RATE: This metric captures the percentage of visitors who leave your website after viewing a single page. If your web pages have a high bounce rate, it could be that users are expecting something different than what you are presenting. A high bounce rate could be a symptom of a confusing, cruddy, or less-than-intuitive app design. Bounce rates will tell you what s happening, but you need to dig deeper to understand why. Qualitative research -- with a service like UserTesting -- can help you understand what users are expecting from your website or mobile product. EXIT RATE: This metric captures the percentage of visitors who leave your website after viewing a specific page. Each page on your website will have its own exit rate. Pay close attention to the exit rate on your crucial pages such as the homepage, pricing page, and checkout flow. If you see a high exit rate on a page during the checkout flow, it may be worth testing that page to see if you can improve user retention. AVERAGE TIME ON SITE: This engagement metric gives you an idea of how long people are staying on your website. Are visitors who spend more time on your site more likely to convert? Or are users spending a lot of time on your site, but your conversion rate is very low? Optimizing these metrics will reveal whether you need to make changes or test different call to actions. Focus your attention to variables that are likely to improve user engagement. AVERAGE PAGE VIEWS: This engagement metric tells you how many pages the average visitor goes through before leaving your website. Average page views is another metric to measure in your conversion optimization. Are visitors who see more pages on your 10

11 site more likely to convert? Are visitors who read 3 or more blog posts likely to convert into trial users? If so, it is worth testing a call to action at the end of each blog post. Know your KPIs. These metrics will help you develop laser-focused optimization experiments. On the web, these goals will typically include leads, user account sign-ups, e-commerce checkouts, and direct sales. KNOW YOUR KPIS. THESE METRICS WILL HELP YOU DEVELOP LASER-FOCUSED OPTIMIZATION EXPERIMENTS. Although a completely different platform, on mobile, you will find similar goals that relate to user engagement, retention, conversion, and revenue. Example metrics to optimize include downloads, conversions from free users into paid customers, and retention rates. 11

12 SEGMENTATION No two visitors are alike. One visitor may be new to your brand while the other may be a long-time customer. A well-optimized website ensures that your brand aligns with both user experiences. Businesses will typically design segments based on the following characteristics: Where is the user in the sales funnel New versus repeat visitor The type of customer (small business vs. enterprise) Decision maker or influencer in purchase decision Demographic variables like gender and age Levels of technical sophistication PERSONALIZATION Personalization can be thought of both from a technical perspective and also from a marketing or promotional perspective. From a technical point of view, personalization involves displaying information about the user that has been saved from prior visits. App developers can then create algorithms to behaviorally target users with app features (or offers) that they ll find relevant. Amazon has done a great job of personalizing each user s experience. Simple things such as displaying the visitor s name, showing the products previously viewed by the user, and present products a visitor may be interested in based on past data. Personalization can also be incorporated in your static marketing or promotional copy. You can use surveys to find common objections about your service or features that customers find most valuable. You can use these findings to craft a more personalized, human message. Speak in a language that resonates with your audience. Your audience will see that your brand fully understand their needs and that you have the right product to solve their problems. The bottom line of personalization is to show the right marketing message to the right audience at exactly the right time -- in a human, engaging, and personally relevant way. Present each user with a unique experience on your website. 12

13 03 COMPARING UX & OPTIMIZATION: WEB VS. MOBILE User experience (UX) involves more than just aesthetics. This concept encompasses the totality of how your user engages with your website, product, or app. What s important to keep in mind is that user experiences span multiple touch points, especially for customers who engage with your brand across multiple platforms. You need to pay attention to the following: How easy your site or app is to use How fast your product is How much friction exists when users complete key actions UX and CRO go hand-in-hand. Across platforms, you need to guide your audiences through the conversion funnel. There are a few major differences when it comes to designing conversion optimized user experiences on the web versus on mobile. In this section, you will discover the key differences and how to best optimize for each screen. 13

14 UX MEETS CRO ON THE WEB On the web, we are primarily stationary with our keyboard and mouse. Key distractions include other browser tabs, IMs, notifications, and other software programs. User objectives could range from finding case studies to learning more about a product to making a purchase. While business objectives include sign-ups to filling out a form to a phone call. On the web, you ll be focusing primarily on two aspects of UX: Visitors to your website come from different sources with varying levels of brand engagement. Each visitor has her own goals in 1. Reducing friction in the mind and it s up to you to map form of wasted clicks, excess pages, slow page loads, and other friction points that cause users to give up. 2. Reducing doubt and indecision from the user s mind that may cause them to not convert. DESIGN FLOWS NOT PAGES IT S BETTER TO HAVE 100 ENGAGED, HIGH-CONVERTING USERS THAN 1,000 WHO USE YOUR APP ONCE AND THEN DISAPPEAR. each user flow to the appropriate conversion funnel. It s important to prioritize quality over quantity of users. In other words, it s better to have 100 engaged, high-converting users than 1,000 who use your app once and then disappear. Too often, we prematurely turn our focus to page design and High-quality users are especially important when using paid information architecture, when we should focus on the user flows acquisition channels. You want to make sure you are attracting that need to be supported by our designs. users that you can ultimately monetize. When starting your next web project, consider starting with a detailed look at the objectives of the user and the business. This practice will allow you to scheme the various flows that address both parties. SOCIAL MEDIA SHARE LANDING PAGE SIGNUP Focus your efforts on the user flows that will have the biggest impact on your visitors and result in the biggest gain for your business. 14

15 BEST PRACTICES FOR UX DESIGN While there is no exact formula for the perfect landing page, there are common elements that you should monitor to maximize your conversions. MAIN HEADLINE The landing page headline should complement the source of the traffic. This is especially true when you are driving paid traffic to your landing page. SECONDARY HEADLINE Try to address a specific point that is related to the content of the landing page rather than having a vague or uninteresting headline. COMPLIMENTARY VIDEO OR PHOTO Customer testimonial videos, product demos, or strong visuals can have a positive impact on visitors. SOCIAL PROOF Build trust with your visitor by incorporating trust symbols such as testimonials, press mentions, number of customers, and guarantee banners. STRONG CALL TO ACTIONS The primary call to action button should stand out from the rest of the page and contain words that your visitors might be searching for such as free, buy, download, etc. STREAMLINED LINKS Simplicity is key to an optimized landing page. Keep links to other parts of your website to a minimum, as they distract your visitors and can have a negative impact on conversions. CONTINUOUS TESTING Continue to test different headlines, call to actions, and videos to see what resonates with your visitors. In addition, consider testing two completely different site designs. This comparison can have a big impact to your long-term success. UX MEETS CRO ON MOBILE Although you are working with a much smaller screen size, the same basic web UX principles apply to mobile. It is still important to start with the objectives of the user. Starting with a goal-first approach, focus on designing your user flows before thinking about what you re crafting on each screen. You also want to reduce friction in the form of wasted taps, excess ads or screens, big app sizes, and complicated user flows. DESIGN IS ESSENTIAL ON MOBILE Apple s culture revolves around design excellence. It s no coincidence that Apple s apps have surged in popularity -- leading some to argue that design is the most critical component in building a successful app. Here are some tips to get you started: 1. POSTPONE SIGN-UP TO AVOID ABANDONMENT: App developers frequently struggle to create interfaces with continuous user engagement. That s because consumers are non-committed. In fact, the average ios and Android user only downloads three to five apps per month. On top of that, 26% of the apps are only opened once and never used again, and another 48% are opened 10 times or less. Moreover, mobile users have become weary of apps that automatically post activity to social media. To earn consumer trust, app developers are moving towards a new trend, requiring users to sign up after they ve become dedicated users. Forcing a user to sign up too early may result in one-star reviews and complete abandonment. 15

16 Key Gestures on mobile Devices 2. CONSIDER GESTURES: The Apple ios Human Interface Guidelines emphasize that an app should allow users to directly manipulate on-screen objects. By simply allowing users to tap, pinch, or swipe the screen you will find that your users are more engaged. It will be easier for users to experience the results of their actions. In an ios app, people experience direct manipulation when they: Rotate or otherwise move the device to affect on-screen objects Use gestures to manipulate on-screen objects Can see that their actions have immediate, visible results 3. ENFORCE CONSTRAINTS: One of the biggest challenges in mobile design is undoing the years of designing on the desktop and the web with its large screen sizes. On the web, it makes total sense to create long pieces of content or forms with many fields -- users are sitting at a desk and have the patience (and computing power) to absorb more information. On mobile, however, every new page or field can add more complexity to the app and make for a poor user experience. Product owners need to approach app design with a mobilefirst mindset. In other words, stop thinking like you re designing for the web. Mobile presents a completely new user experience. You need self-control when designing mobile apps. Here s what we mean by enforcing constraints : Developers can avoid the content and information overload problem by stripping the app its screen and elements to the bare essentials. Great mobile design empowers users to do more with less. For example, your app can allow users to perform all the necessary features within the app with just the use of one thumb. Or remove all navigation within your app and allow the user to use gestures to navigate through your app. 16

17 Let s take the two to-do list apps in the screenshot as examples. The one of the left appears cluttered and hard to navigate. And it is not completely clear how a user can mark a task as complete. In addition, there is wasted screen space at the bottom where the Home and Add buttons reside. But the one on the right, Clear, shows all the tasks in a neat manner and uses the iphone s built-in gestures to manipulate your to-do. Users can pull down on the app to add a new to-do list or swipe to the left to mark a task as complete. 4. RESPECT THE TAP: Every swipe, tap, and pinch that a user takes will have a lasting impact on UX. Each small gesture should help a user make progress. Think: lots of return for very minimal effort. A cluttered to do list app (left) vs. a simplified, gestures-first to do list app (right) Sam Shank, CEO of Hotel Tonight, illustrates that booking a hotel on the app requires only three taps and a swipe totaling about eight seconds. Compare those stats to the apps of Priceline (52 taps, 102 seconds) and Hotel.com (40 taps, 109 seconds). It is no wonder that Hotel Tonight has exploded as a mobile only company. 17

18 CASE STUDY: TASKRABBIT - WEB VS MOBILE DESKTOP HOMEPAGE MOBILE HOMEPAGE Let s take a look at TaskRabbit and the key differences between their UX on the web and on mobile. HOMEPAGE: On the web, TaskRabbit has a lot more screen real estate, so they have decided to promote becoming a TaskRabbit, posting a task, and TaskRabbit for Business. On mobile the company focuses on having users sign up or log in immediately upon launch. DESKTOP TASKS SCREEN MOBILE TASKS SCREEN TASKS SCREEN: In this screen, you will notice that the web version has a list of all the different categories of tasks. On the web, TaskRabbit can also use more visual content along with a clear call to action. 18

19 POST A TASK SCREEN: The web version of this screen is the basic form that most audiences are accustomed to seeing. In the mobile version, each section of the form Location, Supplies, Deadline, etc. has its own screen and users have to navigate back to complete the entire form. While this UX adds more taps and screens to the overall flow, the app varies the gestures of each screen so users are engaged with each section. TECHNICAL DYNAMICS ON THE WEB VS. MOBILE Eric Ries in his book The Lean Startup popularized the term minimum viable product (MVP) - a development technique in which a new product or website is created with the few features necessary to satisfy early adopters. The final, complete set of features is only designed and developed after considering feedback from the product s initial users. DESKTOP POST A TASK SCREEN MOBILE POST A TASK SCREEN While the concept of a minimum viable product has exploded on the web, it is still very difficult to build minimum viable products on mobile. On the web, changes can be implemented and pushed to production in a matter of minutes. On mobile, however, developers must account for each app store s review process, time for users to upgrade their apps, and the nuances of each mobile platform. To further explain the technical dynamics of developing and designing for the web versus mobile, we asked someone who has familiarity building products in both spaces. 19

20 Steve P. Young led Product Marketing for SmartShoot - an online marketplace of professional photographers and filmmakers. He was the lead product manager of SmartShoot s bid platform that allowed anyone to submit a photo or video brief and receive quotes from the network. He also is an app developer with more than 10 apps in the App Store and Google Play Store. Furthermore, he hosts a podcast, Mobile App Chat, where he s interviewed over 100 app developers including Tweetbot, Shazam, Facebook Messenger, and many more. Q&A WITH STEVE P. YOUNG OPTIMIZELY: What are the key differences in building on the web versus mobile? from the developer. Also if an app crashes, users are generally quick to leave a negative review or simply delete the app. STEVE: The main difference is speed of development. On the web, there are many tools that already exist that allow us to quickly build prototypes and get it in front of users. Some tools are so easy to use that marketers these days can test their business hypotheses without the help of engineering. However, on mobile those tools do not exist. Development requires time to code, publish on the app stores, and analyze the results. In addition, on mobile you have to test on different devices with varying screen sizes and computing power. And with the recent release of ios 7, many app developers had to overhaul their entire design. Can you imagine Chrome or Firefox dictating how a website should look and operate? OPTIMIZELY: What about the key differences in user experience? STEVE: On the web, users are a little more forgiving. When you reach a 404 page, you can just hit the back button and find your way around a website. Moreover, many websites provide an online chat feature, so users can immediately get answers in case they are lost. This is certainly not the case on mobile. If an app isn t intuitive and its design confuses the user, there s no way to get an immediate response OPTIMIZELY: From your experience what have you learned about the mobile space? STEVE: It s definitely an exciting space to be in. Things are constantly changing and you are hearing more success stories from indie developers. Through my 100+ interviews, here are three key lessons about building apps: Be a feature: Some of the best apps in the world solve one specific need. On the web, we like our tools to have a lot of features. However, on mobile we are happy when our apps to just do one thing really well. Don t distill the web onto mobile: With such a small screen size, don t try to distill what you have on the web onto a mobile screen. The experience should be completely different and you should take advantage of the touch screens and incorporate taps and swipes into your interface. Build a minimum viable experience/value: It is incredibly hard to rapidly prototype on mobile, so think about creative ways you could prototype the experience or value of your app. Don t start with interface of your app, start with the experience you d like the user to have with your app. 20

21 04 CRO ON MOBILE: ACQUISITION Thanks to Optimizely, A/B testing and optimization on the web requires little technical assistance and presents few barriers. Mobile is a different story. Marketers on the web have easier access to data and with the help of tools can make changes on the fly without engineering help. On mobile, marketers and product managers are more reliant on engineering teams and do not have the freedom to make changes like they do on the web. 21

22 Fortunately, there are some non-technical hacks that marketers and product managers can do to help optimize adoption and engagement with their apps. Before we do that let s take a look at a typical conversion funnel for an app: Awareness App Store Search OPTIMIZING FOR APP STORES App Store Optimization (ASO) is the equivalent of search engine optimization (SEO) for mobile apps. The goal is to make it easy for users to find your app in the App Store through careful selection of keywords that specific user intents. The nice thing about ASO is that it requires very little technical knowledge and often times can be optimized without the need of an engineering team. However, it s important to note that ASO is not a real-time A/B test. You need to compare data from before and after the change. In fact, when performing ASO, you are analyzing past results with current results to see if the change you made had an impact. Download Open App App Store Optimization tests can involve: APP ICON: It s important to test your icon even before releasing the app. You can buy some ads on mobile advertising networks such as Admob or online advertising through Google Adwords and Facebook. What you want to find out is which app icon is getting the highest click through rate. Registration/ Key Action Usage & Retention APP NAME: According to Nielsen, 63% of Android and ios users have utilized search to discover new apps. That s why it s important to optimize your app name with highly sough after keyword phrases. ASO tools such as Straply and Sensor Tower are great places to start. In the online marketing world, having keywords in the title of your web page helps with SEO. In the app store, the same principle applies with your app name