A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS. How Best-in-Class App Developers Use Data to Develop Amazing Mobile Apps

Transcription

1 A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS How Best-in-Class App Developers Use Data to Develop Amazing Mobile Apps

2 EVALUATING MOBILE STRATEGY QUANTIFYING GOALS ON MOBILE HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN TESTING ANALYTICS SEGMENTATION 00 CONTENTS PERSONALIZATION COMPARING UX & OPTIMIZATION: WEB VS. MOBILE UX MEETS CRO ON THE WEB UX MEETS CRO ON MOBILE CASE STUDY: TASK RABBIT - WEB VS. MOBILE TECHNICAL DYNAMICS ON THE WEB VS. MOBILE CRO ON MOBILE: ACQUISITION OPTIMIZING FOR APP STORES 06 OPTIMIZING FOR REGISTRATION/KEY ACTIONS CRO ON MOBILE: RETENTION TECHNICAL CONSIDERATIONS PRODUCT CONSIDERATIONS CONCLUSION 2

3 A majority of Americans now own a smartphone of some kind positioning mobile as one of the most significant opportunities for consumer and B2B technology companies. 1 It s no surprise that the app market is growing, with more than one million apps both in the Apple App Store and Google Play Store. How do you make your app stand out from the crowd? The first step is to craft a best-in-class user experience. Optimizely has already built a market-leading platform for creating best-in-class user experiences on the web. With users spending an increasing amount of time on mobile devices, it s become critically important for businesses to be able to deliver amazing user experiences that span the web and mobile. Delivering on this, however, is easier said than done. Many budding app developers will soon learn that while their web and mobile WHILE WEB AND MOBILE USERS ARE OFTEN THE SAME PEOPLE, THOSE PEOPLE HAVE VERY DIFFERENT GOALS AND BEHAVIORS WHEN ENGAGING ON MOBILE VS. WHEN ENGAGING ON THE WEB. users are often the same people, those people have very different goals and behaviors when engaging on mobile vs. when engaging on the web. That s why we wrote this guide to provide you with the most comprehensive and in depth resource to learn: (1) What makes an awesome mobile user experience (2) How to evolve your mobile strategy to create highly engaging products (3) How to optimize that user experience for conversions

4 01 EVALUATING MOBILE STRATEGY With smartphone penetration rates as high as 74%, your company s mobile strategy is more important than ever. Mobile-only companies like Instagram, WhatsApp, and Snapchat have forged a path to billion-plus dollar valuations. Not to mention, today s most innovative firms like Eventbrite, an event management platform, and Percolate, software for content marketers, are investing heavily in developing mobile-focused solutions. 2 What s your mobile strategy? How can your brand make the strongest impact possible?

5 Years ago, a successful mobile strategy meant having an app. But the landscape has quickly changed. Hollywood success stories are becoming less and less common. Just-for-fun apps are everywhere which makes it tough for app developers to stand out. The secret to a successful mobile app is no secret at all. Plain and simple, top mobile apps delight users by 1: making their lives easier and/or 2: providing entertainment. JUST-FOR-FUN APPS ARE EVERYWHERE WHICH MAKES IT TOUGH FOR APP DEVELOPERS TO STAND OUT. Consumers crave elegant user experiences, workflow tools, and highly engaging games. They re looking for resources to be more productive, opportunities for brain breaks, and touch points to stay connected with friends and family. That s why analytics, A/B testing, and conversion rate optimization are extremely important to the app design process. You need to be learning about your users, testing your assumptions, and exploring areas for improvement. QUANTIFYING GOALS ON MOBILE Before building an app, product owners will need to clearly define their revenue objectives. This process starts with identifying key metrics that align with an engaging and profitable mobile user experience. Start by thinking about the metrics that define success on a desktop. Some of the user experience objectives (i.e. user engagement and monetization) will be similar. You ll want your customers to be engaging with the product in a way that drives revenue. At the same time, you need to approach your products goals with a mobile-first mindset, understanding that you re reimagining your products for a completely new device. Pay attention to the following metrics when designing for mobile: 1. DAILY ACTIVE USERS (DAUS): DAU measures your app s engagement levels how many unique users visit the app daily? A high DAU count is a sign that your app is something that people need (or really love). For instance, users might love your game and decide to play it, every day, on their morning train commutes. Users might rely on your app to organize their notes (like Evernote), book vacation rentals (like HotelTonight), and organize their content marketing programs (like Percolate). 5

6 2. AVERAGE RATING & REVIEWS: This metric quantifies what users think of your app and directly impacts performance in app marketplaces. App reviews are also a great place for developers to research user concerns -- which can directly influence your brand positioning and future product releases. So how do you get users to review your app? Well, you ask them. But here s the thing. Simply telling users to leave a great review won t be enough. In addition to prompting users to review your app, you need to think about how to position these requests within your user experience. Most apps will generally prompt a user to leave a rating or review upon launching the app - interrupting the general user flow. A better way to prompt the user is after they have finished doing what they need to do within your app -- or after they ve accomplished Apple s App Store uses app ratings as a factor in calculating Top Chart rankings. It s tough for apps to bounce back from a low ranking -- even if they re downloaded fairly often. something that gives them a sense of achievement (like reaching a high score within a game). For example, FIFA 2012 prompts you to leave a review for the app after you have won a major tournament. 6

7 3. DOWNLOADS: Downloads of an app are the equivalent of an account sign-up on the web and play a crucial role in the app s ranking. It s important to monitor downloads with DAU to ensure that both are trending positively. If you see a spike in downloads, do you see a similar spike in DAU? If not, you may have to revisit your retention features (and usage levels) or test different push notification messages. As an example, the Carrot To-Do app does a great job of using humor in its push notifications to pull users back into the app. 4. RETENTION: Retention is calculated as DAUs divided by number of downloads. It s a key driver for apps with in-session monetization opportunities. Are your DAUs increasing as downloads increase? If not, it is worth digging deeper into why users are abandoning your app. Apps using the freemium model as their main source of revenues must be sure that they have the proper mechanics in place to maximize retention and upselling. As an example, Clash of Clans (a popular mobile game) offers virtual currency and cleverly timed push notifications. Some app companies have even determined the length of time it takes to monetize a user and will even serve ads to competing games when a user has past that time frame and still has yet to monetize. Virtual currency and push notifications in Clash of Clans Carrot To-Do in the App Store 7

8 5. APP CRASHES / FAILURES: While difficult to track, it is important to have crash analytics built into your app. Unfortunately, the default Google Play and Apple analytics do not capture all crashes, so it s important to use an additional crash analytics tool. To circumvent a crash or feature request from turning into a negative review, app developers are implementing feedback functionality so users can provide performance issues or new feature suggestions directly within the app. Negative bug-related app review left by a Tweetbot user For example, Tweetbot, a popular Twitter client, implements a feedback mechanism within their settings page. Tweetbot s in-app feedback mechanism 8

9 02 Conversion HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN rate optimization describes the structured and systematic approach to improving the performance of your website or mobile app through analytics and user feedback. Simply put, it s the practice of running experiments to 1: find out why users aren t doing what you want them to do, 2: test your assumptions, 3: iterate, and 4: test again. CRO is a long-established process that began in the early days of the Internet -- long before smartphones entered the picture. The process starts with understanding your company s unique objectives and key performance indicators (KPIs). Techniques such as A/B testing and qualitative research allow business owners to use data to make informed user-driven decisions. To understand how CRO fits in with mobile, you need to understand its application on the web. Before we jump back into mobile-specific workflows, let s start with a quick refresher by reviewing the following nuts and bolts of CRO. 9

10 TESTING A/B tests allow companies to compare page versions (or elements) against one another. Each page variation is then measured by its effectiveness in comparison to the other. For example, a marketer may compare a page with a green call-to-action (CTA) to a page with a red CTA. A/B tests focus on only one element at a time. ANALYTICS Analytics empower you to assess your current conversion rate, identify trouble areas, and design an execution action. Pay attention to the following metrics: CONVERSION RATE: This metric bridges the connection between website traffic and revenue. Are users taking the action that you would like them to? If not, analyze your conversion flow and start testing the page(s) with the biggest drop off. This metric will help you make sure that users are completing desired goals within your app (making purchases, returning quickly, encouraging friends to download the app, etc.) BOUNCE RATE: This metric captures the percentage of visitors who leave your website after viewing a single page. If your web pages have a high bounce rate, it could be that users are expecting something different than what you are presenting. A high bounce rate could be a symptom of a confusing, cruddy, or less-than-intuitive app design. Bounce rates will tell you what s happening, but you need to dig deeper to understand why. Qualitative research -- with a service like UserTesting -- can help you understand what users are expecting from your website or mobile product. EXIT RATE: This metric captures the percentage of visitors who leave your website after viewing a specific page. Each page on your website will have its own exit rate. Pay close attention to the exit rate on your crucial pages such as the homepage, pricing page, and checkout flow. If you see a high exit rate on a page during the checkout flow, it may be worth testing that page to see if you can improve user retention. AVERAGE TIME ON SITE: This engagement metric gives you an idea of how long people are staying on your website. Are visitors who spend more time on your site more likely to convert? Or are users spending a lot of time on your site, but your conversion rate is very low? Optimizing these metrics will reveal whether you need to make changes or test different call to actions. Focus your attention to variables that are likely to improve user engagement. AVERAGE PAGE VIEWS: This engagement metric tells you how many pages the average visitor goes through before leaving your website. Average page views is another metric to measure in your conversion optimization. Are visitors who see more pages on your 10

11 site more likely to convert? Are visitors who read 3 or more blog posts likely to convert into trial users? If so, it is worth testing a call to action at the end of each blog post. Know your KPIs. These metrics will help you develop laser-focused optimization experiments. On the web, these goals will typically include leads, user account sign-ups, e-commerce checkouts, and direct sales. KNOW YOUR KPIS. THESE METRICS WILL HELP YOU DEVELOP LASER-FOCUSED OPTIMIZATION EXPERIMENTS. Although a completely different platform, on mobile, you will find similar goals that relate to user engagement, retention, conversion, and revenue. Example metrics to optimize include downloads, conversions from free users into paid customers, and retention rates. 11

12 SEGMENTATION No two visitors are alike. One visitor may be new to your brand while the other may be a long-time customer. A well-optimized website ensures that your brand aligns with both user experiences. Businesses will typically design segments based on the following characteristics: Where is the user in the sales funnel New versus repeat visitor The type of customer (small business vs. enterprise) Decision maker or influencer in purchase decision Demographic variables like gender and age Levels of technical sophistication PERSONALIZATION Personalization can be thought of both from a technical perspective and also from a marketing or promotional perspective. From a technical point of view, personalization involves displaying information about the user that has been saved from prior visits. App developers can then create algorithms to behaviorally target users with app features (or offers) that they ll find relevant. Amazon has done a great job of personalizing each user s experience. Simple things such as displaying the visitor s name, showing the products previously viewed by the user, and present products a visitor may be interested in based on past data. Personalization can also be incorporated in your static marketing or promotional copy. You can use surveys to find common objections about your service or features that customers find most valuable. You can use these findings to craft a more personalized, human message. Speak in a language that resonates with your audience. Your audience will see that your brand fully understand their needs and that you have the right product to solve their problems. The bottom line of personalization is to show the right marketing message to the right audience at exactly the right time -- in a human, engaging, and personally relevant way. Present each user with a unique experience on your website. 12

13 03 COMPARING UX & OPTIMIZATION: WEB VS. MOBILE User experience (UX) involves more than just aesthetics. This concept encompasses the totality of how your user engages with your website, product, or app. What s important to keep in mind is that user experiences span multiple touch points, especially for customers who engage with your brand across multiple platforms. You need to pay attention to the following: How easy your site or app is to use How fast your product is How much friction exists when users complete key actions UX and CRO go hand-in-hand. Across platforms, you need to guide your audiences through the conversion funnel. There are a few major differences when it comes to designing conversion optimized user experiences on the web versus on mobile. In this section, you will discover the key differences and how to best optimize for each screen. 13

14 UX MEETS CRO ON THE WEB On the web, we are primarily stationary with our keyboard and mouse. Key distractions include other browser tabs, IMs, notifications, and other software programs. User objectives could range from finding case studies to learning more about a product to making a purchase. While business objectives include sign-ups to filling out a form to a phone call. On the web, you ll be focusing primarily on two aspects of UX: Visitors to your website come from different sources with varying levels of brand engagement. Each visitor has her own goals in 1. Reducing friction in the mind and it s up to you to map form of wasted clicks, excess pages, slow page loads, and other friction points that cause users to give up. 2. Reducing doubt and indecision from the user s mind that may cause them to not convert. DESIGN FLOWS NOT PAGES IT S BETTER TO HAVE 100 ENGAGED, HIGH-CONVERTING USERS THAN 1,000 WHO USE YOUR APP ONCE AND THEN DISAPPEAR. each user flow to the appropriate conversion funnel. It s important to prioritize quality over quantity of users. In other words, it s better to have 100 engaged, high-converting users than 1,000 who use your app once and then disappear. Too often, we prematurely turn our focus to page design and High-quality users are especially important when using paid information architecture, when we should focus on the user flows acquisition channels. You want to make sure you are attracting that need to be supported by our designs. users that you can ultimately monetize. When starting your next web project, consider starting with a detailed look at the objectives of the user and the business. This practice will allow you to scheme the various flows that address both parties. SOCIAL MEDIA SHARE LANDING PAGE SIGNUP Focus your efforts on the user flows that will have the biggest impact on your visitors and result in the biggest gain for your business. 14

15 BEST PRACTICES FOR UX DESIGN While there is no exact formula for the perfect landing page, there are common elements that you should monitor to maximize your conversions. MAIN HEADLINE The landing page headline should complement the source of the traffic. This is especially true when you are driving paid traffic to your landing page. SECONDARY HEADLINE Try to address a specific point that is related to the content of the landing page rather than having a vague or uninteresting headline. COMPLIMENTARY VIDEO OR PHOTO Customer testimonial videos, product demos, or strong visuals can have a positive impact on visitors. SOCIAL PROOF Build trust with your visitor by incorporating trust symbols such as testimonials, press mentions, number of customers, and guarantee banners. STRONG CALL TO ACTIONS The primary call to action button should stand out from the rest of the page and contain words that your visitors might be searching for such as free, buy, download, etc. STREAMLINED LINKS Simplicity is key to an optimized landing page. Keep links to other parts of your website to a minimum, as they distract your visitors and can have a negative impact on conversions. CONTINUOUS TESTING Continue to test different headlines, call to actions, and videos to see what resonates with your visitors. In addition, consider testing two completely different site designs. This comparison can have a big impact to your long-term success. UX MEETS CRO ON MOBILE Although you are working with a much smaller screen size, the same basic web UX principles apply to mobile. It is still important to start with the objectives of the user. Starting with a goal-first approach, focus on designing your user flows before thinking about what you re crafting on each screen. You also want to reduce friction in the form of wasted taps, excess ads or screens, big app sizes, and complicated user flows. DESIGN IS ESSENTIAL ON MOBILE Apple s culture revolves around design excellence. It s no coincidence that Apple s apps have surged in popularity -- leading some to argue that design is the most critical component in building a successful app. Here are some tips to get you started: 1. POSTPONE SIGN-UP TO AVOID ABANDONMENT: App developers frequently struggle to create interfaces with continuous user engagement. That s because consumers are non-committed. In fact, the average ios and Android user only downloads three to five apps per month. On top of that, 26% of the apps are only opened once and never used again, and another 48% are opened 10 times or less. Moreover, mobile users have become weary of apps that automatically post activity to social media. To earn consumer trust, app developers are moving towards a new trend, requiring users to sign up after they ve become dedicated users. Forcing a user to sign up too early may result in one-star reviews and complete abandonment. 15

16 Key Gestures on mobile Devices 2. CONSIDER GESTURES: The Apple ios Human Interface Guidelines emphasize that an app should allow users to directly manipulate on-screen objects. By simply allowing users to tap, pinch, or swipe the screen you will find that your users are more engaged. It will be easier for users to experience the results of their actions. In an ios app, people experience direct manipulation when they: Rotate or otherwise move the device to affect on-screen objects Use gestures to manipulate on-screen objects Can see that their actions have immediate, visible results 3. ENFORCE CONSTRAINTS: One of the biggest challenges in mobile design is undoing the years of designing on the desktop and the web with its large screen sizes. On the web, it makes total sense to create long pieces of content or forms with many fields -- users are sitting at a desk and have the patience (and computing power) to absorb more information. On mobile, however, every new page or field can add more complexity to the app and make for a poor user experience. Product owners need to approach app design with a mobilefirst mindset. In other words, stop thinking like you re designing for the web. Mobile presents a completely new user experience. You need self-control when designing mobile apps. Here s what we mean by enforcing constraints : Developers can avoid the content and information overload problem by stripping the app its screen and elements to the bare essentials. Great mobile design empowers users to do more with less. For example, your app can allow users to perform all the necessary features within the app with just the use of one thumb. Or remove all navigation within your app and allow the user to use gestures to navigate through your app. 16

17 Let s take the two to-do list apps in the screenshot as examples. The one of the left appears cluttered and hard to navigate. And it is not completely clear how a user can mark a task as complete. In addition, there is wasted screen space at the bottom where the Home and Add buttons reside. But the one on the right, Clear, shows all the tasks in a neat manner and uses the iphone s built-in gestures to manipulate your to-do. Users can pull down on the app to add a new to-do list or swipe to the left to mark a task as complete. 4. RESPECT THE TAP: Every swipe, tap, and pinch that a user takes will have a lasting impact on UX. Each small gesture should help a user make progress. Think: lots of return for very minimal effort. A cluttered to do list app (left) vs. a simplified, gestures-first to do list app (right) Sam Shank, CEO of Hotel Tonight, illustrates that booking a hotel on the app requires only three taps and a swipe totaling about eight seconds. Compare those stats to the apps of Priceline (52 taps, 102 seconds) and Hotel.com (40 taps, 109 seconds). It is no wonder that Hotel Tonight has exploded as a mobile only company. 17

18 CASE STUDY: TASKRABBIT - WEB VS MOBILE DESKTOP HOMEPAGE MOBILE HOMEPAGE Let s take a look at TaskRabbit and the key differences between their UX on the web and on mobile. HOMEPAGE: On the web, TaskRabbit has a lot more screen real estate, so they have decided to promote becoming a TaskRabbit, posting a task, and TaskRabbit for Business. On mobile the company focuses on having users sign up or log in immediately upon launch. DESKTOP TASKS SCREEN MOBILE TASKS SCREEN TASKS SCREEN: In this screen, you will notice that the web version has a list of all the different categories of tasks. On the web, TaskRabbit can also use more visual content along with a clear call to action. 18

19 POST A TASK SCREEN: The web version of this screen is the basic form that most audiences are accustomed to seeing. In the mobile version, each section of the form Location, Supplies, Deadline, etc. has its own screen and users have to navigate back to complete the entire form. While this UX adds more taps and screens to the overall flow, the app varies the gestures of each screen so users are engaged with each section. TECHNICAL DYNAMICS ON THE WEB VS. MOBILE Eric Ries in his book The Lean Startup popularized the term minimum viable product (MVP) - a development technique in which a new product or website is created with the few features necessary to satisfy early adopters. The final, complete set of features is only designed and developed after considering feedback from the product s initial users. DESKTOP POST A TASK SCREEN MOBILE POST A TASK SCREEN While the concept of a minimum viable product has exploded on the web, it is still very difficult to build minimum viable products on mobile. On the web, changes can be implemented and pushed to production in a matter of minutes. On mobile, however, developers must account for each app store s review process, time for users to upgrade their apps, and the nuances of each mobile platform. To further explain the technical dynamics of developing and designing for the web versus mobile, we asked someone who has familiarity building products in both spaces. 19

20 Steve P. Young led Product Marketing for SmartShoot - an online marketplace of professional photographers and filmmakers. He was the lead product manager of SmartShoot s bid platform that allowed anyone to submit a photo or video brief and receive quotes from the network. He also is an app developer with more than 10 apps in the App Store and Google Play Store. Furthermore, he hosts a podcast, Mobile App Chat, where he s interviewed over 100 app developers including Tweetbot, Shazam, Facebook Messenger, and many more. Q&A WITH STEVE P. YOUNG OPTIMIZELY: What are the key differences in building on the web versus mobile? from the developer. Also if an app crashes, users are generally quick to leave a negative review or simply delete the app. STEVE: The main difference is speed of development. On the web, there are many tools that already exist that allow us to quickly build prototypes and get it in front of users. Some tools are so easy to use that marketers these days can test their business hypotheses without the help of engineering. However, on mobile those tools do not exist. Development requires time to code, publish on the app stores, and analyze the results. In addition, on mobile you have to test on different devices with varying screen sizes and computing power. And with the recent release of ios 7, many app developers had to overhaul their entire design. Can you imagine Chrome or Firefox dictating how a website should look and operate? OPTIMIZELY: What about the key differences in user experience? STEVE: On the web, users are a little more forgiving. When you reach a 404 page, you can just hit the back button and find your way around a website. Moreover, many websites provide an online chat feature, so users can immediately get answers in case they are lost. This is certainly not the case on mobile. If an app isn t intuitive and its design confuses the user, there s no way to get an immediate response OPTIMIZELY: From your experience what have you learned about the mobile space? STEVE: It s definitely an exciting space to be in. Things are constantly changing and you are hearing more success stories from indie developers. Through my 100+ interviews, here are three key lessons about building apps: Be a feature: Some of the best apps in the world solve one specific need. On the web, we like our tools to have a lot of features. However, on mobile we are happy when our apps to just do one thing really well. Don t distill the web onto mobile: With such a small screen size, don t try to distill what you have on the web onto a mobile screen. The experience should be completely different and you should take advantage of the touch screens and incorporate taps and swipes into your interface. Build a minimum viable experience/value: It is incredibly hard to rapidly prototype on mobile, so think about creative ways you could prototype the experience or value of your app. Don t start with interface of your app, start with the experience you d like the user to have with your app. 20

21 04 CRO ON MOBILE: ACQUISITION Thanks to Optimizely, A/B testing and optimization on the web requires little technical assistance and presents few barriers. Mobile is a different story. Marketers on the web have easier access to data and with the help of tools can make changes on the fly without engineering help. On mobile, marketers and product managers are more reliant on engineering teams and do not have the freedom to make changes like they do on the web. 21

22 Fortunately, there are some non-technical hacks that marketers and product managers can do to help optimize adoption and engagement with their apps. Before we do that let s take a look at a typical conversion funnel for an app: Awareness App Store Search OPTIMIZING FOR APP STORES App Store Optimization (ASO) is the equivalent of search engine optimization (SEO) for mobile apps. The goal is to make it easy for users to find your app in the App Store through careful selection of keywords that specific user intents. The nice thing about ASO is that it requires very little technical knowledge and often times can be optimized without the need of an engineering team. However, it s important to note that ASO is not a real-time A/B test. You need to compare data from before and after the change. In fact, when performing ASO, you are analyzing past results with current results to see if the change you made had an impact. Download Open App App Store Optimization tests can involve: APP ICON: It s important to test your icon even before releasing the app. You can buy some ads on mobile advertising networks such as Admob or online advertising through Google Adwords and Facebook. What you want to find out is which app icon is getting the highest click through rate. Registration/ Key Action Usage & Retention APP NAME: According to Nielsen, 63% of Android and ios users have utilized search to discover new apps. That s why it s important to optimize your app name with highly sough after keyword phrases. ASO tools such as Straply and Sensor Tower are great places to start. In the online marketing world, having keywords in the title of your web page helps with SEO. In the app store, the same principle applies with your app name

23 KEYWORDS: On the web, keyword meta-data no longer influences search rankings. An app s keyword data, however, actually has an impact on its search rank. Along with your app s name, having relevant, high-traffic words in your keyword meta-data can help improve your apps search ranking. SCREENSHOTS: While the term may imply a screen capture of your app, you should think of your screenshots as a banner advertisement. Both the Google Play and Apple App Store prominently display screenshots rather than an app s description. Besides the app icon, your app s screenshot is the strongest visual representation of your app, so you want to draw the user to download it. Again, just like the app icon, you can test using traditional banner ads to see which screenshot yields the highest click throughs. OPTIMIZING FOR REGISTRATION/KEY ACTIONS A recent study from Compuware shows that 80-90% of apps are deleted after only one use. Mobile customers are unforgiving. If you do not make an immediate first impression, it s likely that your app will be deleted. Without an optimized onboarding experience, you are not only losing users, but also an opportunity to monetize your app. Let s unpack this a little more. Unlike the web where we have multiple tabs open and going to a website takes a matter of seconds, on mobile we have to go through the trouble of finding your app, downloading it, and then remembering to actually open it. WITHOUT AN OPTIMIZED ONBOARDING EXPERIENCE, YOU ARE NOT ONLY LOSING USERS, BUT ALSO AN OPPORTUNITY TO MONETIZE YOUR APP. The user has gone through three steps just to see what your app has to offer, so retaining as much of that 90% as possible is critical. How do you ensure you make the best first impression possible? Make sure your app does not make some of these common mistakes we see from app developers during the onboarding process: REGISTER FIRST, TRY LATER: Apps that require registration before using it, lose up to 56 p ercent of its users. Mobile users want to see and engage with your app before deciding to join. LENGTHY REGISTRATIONS: The mobile screen is small enough, so requiring a lot of information from your users will have a negative effect on their usage. You may want to use multiple screens to allow the user to enter in bite-size information. For example, having three screens with 2-3 fields each is much better than having one screen with 9 fields. LONG LOAD TIME: Mobile customers have a low tolerance for unstable and buggy apps. Apple has emphasized design and UX within its default apps, so users have come to expect elegant designs and intuitive user flows. 23

24 05 CRO ON MOBILE: RETENTION As outlined in this paper, mobile app optimization can yield huge benefits for app developers. At the same time the reality of today s marketplace is that users are interacting with brands and products across desktop web and native mobile apps. However given the technical constraints of executing an A/B test or a personalization campaign within a native mobile app, many app developers will simply not test at all. This leaves valuable valuable opportunities on the table for better conversion rates, higher engagement, and increased revenue. 24

25 That s why Optimizely is investing in product enhancements that enable customers to optimize end-to-end experiences across all devices, websites, and apps their visitors are using. Our goal is to reduce the friction required to run in-app experiments that lead to optimized user experiences that achieve the goals app developers have set out to accomplish. USERS MUST UPGRADE APP: The landscape changed a bit when Apple introduced automatic updates with ios7, but it is still no guarantee that users have this featured turned on. Some power users have actually turned this feature off to save battery life. Even with the feature turned on, it is hard to predict when the update will reach every user. TECHNICAL CONSIDERATIONS Before deciding on a mobile app optimization strategy, it s important to consider the technical challenges mentioned above. There is no mobile app equivalent to the DOM (Document Object Model), which can be manipulated by JavaScript. This introduces the complexity of requiring engineering support to test changes that would be much more straightforward to implement on the web. Moreover companies are becoming increasingly reliant upon third party SDKs for analytics, monetization, and optimization, all of which require technical knowledge. USERS MAY NOT BE ONLINE: We have all had connection problems here and there. While this may not be a huge concern, it is still something to consider when releasing an update. You can t always guarantee that users will have Internet connection, so you need to ensure that your app has the proper messaging in place to address offline issues. Users can always upgrade when they re back online, but in the interim, they may feel frustrated. This one moment of frustration has the potential to wreak havoc on user engagement. As a marketer, you may think -- well, they ll hop online later. Keep in mind, however, that consumers want instant results and gratification. Here are some key technical considerations. While not specific to A/B testing and optimization, they are important to keep in mind: APP STORE REVIEW: Each app store has its own review cycle. On Google Play it could be a matter of hours, where on the Apple App Store, it could be anywhere from 5-7 days. This delay makes it hard for developers to push changes. Control remains squarely in the hands of the platforms. While tools like Parse can make it possible to release updates quicker, it is still difficult for a non-technical marketer or product owner to have control over an A/B test or feature change. PRODUCT CONSIDERATIONS Technical considerations aside, we don t see much of a difference between A/B testing and optimization on the web vs. mobile. As mentioned above, Optimizely is expanding its already market-leading web optimization platform to support mobile app A/B testing and optimization. We built Optimizely for mobile apps using the same philosophy as we did when building our web product. Of course, it s also important to recognize and account for the fact that there are tactical differences when optimizing across devices. 25

26 When choosing your native mobile app optimization strategy, it s important to keep these tenets in mind: CROSS-PLATFORM. Users are the same across web and mobile, but their behaviors and motivations are different. That s why it is important to choose a native app optimization framework that works across both the web and mobile devices. This framework should allow you to craft different experiences to cater to device-specific needs, while still carrying over information between web and mobile to make sure the experiences is consistent no matter where the user is engaging. USERS ARE THE SAME ACROSS WEB AND MOBILE, BUT THEIR BEHAVIORS AND MOTIVATIONS ARE DIFFERENT. FAST. The app development space is one of the fastest-paced that exists today. It s never been more important to build apps that react to the needs of your users and to the market in general. Any app optimization platform you choose should take this into account. It should be easy to roll out changes without a code push. It should minimize dependence on developer time as much as possible. ANALYTICS AND INSIGHTS. Experimentation and optimization is no good without concrete data to take action on. When choosing an optimization platform, it s critical to pick one that will give you the insights an analysis you need to make decisions using the data that s being generated. SPEED, STABILITY, AND SECURITY. Mobile apps are different than websites in that they re ultimately end-to-end programs that users are installing on their devices. Once an app is installed, it needs to be able to function and scale independently. It s important that any solution you choose does not compromise the fidelity of your app, because that will ultimately reflect on your app s rating and, by extension, its future success. INTEGRATES YOUR ECOSYSTEM. If your app is like most apps, it s already leveraging third party SDKs for things like analytics and crash reporting. For something as fundamental as A/B testing and optimization, it s critical to choose a platform that integrates with the most commonly used mobile app SDKs out there. Your team shouldn t have to relearn a new set of tools or define a new workflow just to optimize your app. Barriers like that can prevent you from being able to implement a culture of experimentation within your organization. 26

27 CONCLUSION It s never been more important to deliver compelling experiences that delight and engage users on mobile devices. For years, marketers and product managers have embraced a culture of experimentation and testing, and as a result, web experiences are becoming increasingly effective at improving users experiences while also helping companies accomplish their goals. However this culture of experimentation has not yet made it s way into the realm of mobile app development. This is largely due to the reality that implementing a mobile app optimization strategy typically requires a prohibitively high level of involvement from engineering teams who could otherwise be working on feature development. In developing its mobile app optimization framework, Optimizely s goal is to reduce the necessity to involve engineering teams to the extent that it can be adopted in the mobile app community as heavily as it has been adopted in the web community. Learn more about Optimizely s mobile app optimization platform at 27

28 ABOUT THIS GUIDE A Blueprint for A/B Testing and Optimization on Native Mobile Apps Written By: Sean Oliver Senior Product Marketing Manager, Designed By: Jon Thank you to Ural Cebeci, Steve Young, and Ritika Puri. To learn more about Optimizely for Mobile Apps, visit OPTIMIZELY.COM/MOBILE ABOUT OPTIMIZELY Optimizely is the world s leading experience optimization platform, providing A/B testing, multivariate testing, and personalization for individuals and organizations with and without technical expertise. The platform s ease of use empowers organizations to conceive of and run experiments that help them make better data-driven decisions. With targeting and segmentation using powerful realtime data, Optimizely meets the diverse needs of any business looking to deliver unique experiences to their visitors. [email protected] Phone: Facebook: facebook.com/optimizely Twitter: twitter.com/optimizely San Francisco Office 631 Howard Street, Suite 100 San Francisco, CA Amsterdam Office Nes KE Amsterdam