A/B Testing and Analysis with Cascade Server

Transcription

1 A/B Testing and Analysis with Cascade Server Hannon Hill Corporation 950 East Paces Ferry Rd Suite 2440, 24 th Floor Atlanta, GA A high-level overview of implementing a hosted web analytics tool with Cascade Server to maximize effectiveness in online marketing and boost profitability. Tel: Tel: Fax: Hannon Hill. All rights reserved. Cascade Server, Hannon Hill, and the logos are registered trademarks of Hannon Hill Corporation. All other trademarks are owned by their respective owners.

2 Table of Contents Introduction...3 The A/B Test...3 A/B Testing and Cascade Server...7 Real World Example...7 Conclusion...12 Confidential, Hannon Hill Corporation, Atlanta, Georgia 2

3 INTRODUCTION As online commerce has become an increasingly booming industry, its traditional paradigm has shifted from a problem of information scarcity to information overload. Companies have saturated search engine listings offering generic products and services that their competitive counterparts also offer. The end result is a site visitor that has become more fickle with his or her online habits. Very similar to the way consumers are free to channel surf their way out of watching a commercial or quickly change his or her mind about going into a store inside of a shopping mall, online consumers are more inclined to skip right over what your organization is looking to offer than they are to actually spend time listening to your sales pitch. The solution to the problem is a relatively simple in concept but much harder to implement. In any given situation, online consumers have a problem they would like to solve with a product, service, or information. Due to the vast wealth of information on the Internet, the average person looks to find item immediately and in the format that appeals to him or her the most. Much like the traditional rules of advertising, an effective campaign on the Internet must be able to garner attention from the user quickly. The correct implementation of content on an organization s site can become a series of costly trial and error exercises that ultimately end up with a less-than-effective result. Implementing the correct combination of content and formatting on a site to increase visitors and conversions is the problem that carefully planned market testing is designed to solve. More explicitly, through a process called the A/B Test, metrics can be used to evaluate the different variations of a site s content and presentation formatting. The remainder of this article will show you how Cascade Server can offer you the benefits of market testing right within the content management system. Moreover, we will touch upon the idea of an A/B Test in more detail, and demonstrate how Cascade Server can perform the same test for you at a fraction of the cost. THE A/B TEST A/B testing strives to solve the problem of trial and error tests that are neither very accurate nor scientifically efficient. The A/B test is more of an experiment than it is a Confidential, Hannon Hill Corporation, Atlanta, Georgia 3

4 marketing technique. Similar to how a lab test would be conducted, a control variable exists the A version of a site page and the variable being tested the B version of that same page. For instance, suppose we have a page that is featuring a product that is sold through the site s online shopping cart. The page consists of a large product image in the left-hand column, and a brief description of the product in the right-hand column. At the top of the right column, next to the product image is a large Buy button for interested buyers to click on when they want to purchase the product. The marketing team suggests that the site would convert more visitors into customers if a more appealing image of the product were displayed instead of the current product image being used on the page. A simple A/B test could be constructed for the product page with the current implementation on the site serving as our control or A variable. For the B test variable, a new implementation of the product page would be constructed with a new image, completing the setup for the first part of the experiment. Now that we have two versions of the product page, we will run the test by randomly serving the two different versions up to new visitors on the site for a period of time (i.e. couple weeks, month, etc.). In order to ensure that the results of the experiment are more accurate and compelling, the application routines responsible for serving up the pages would be designed such that that the two different versions of the page are served up randomly and also stay close to a 50% distribution such that half of all visitors will either see the control variable A or the B test variable of the product page. The last crucial part of the A/B Test is the collection of the conversion page usage data and the choice of metrics that will be run against that data to produce the results of the experiment. Various packages exist that allow site usage data to be tracked. The objective of this experiment is to measure how many visitors viewed the product page and how many of them converted into buyers by choosing the Buy button that was placed in clear view, next to the product image. These measurements would logically need to be performed so that they can be reported for both variations of the product page. The variation of the page with the most converted visitors to buyers is considered by the test to be the more effective product page. Confidential, Hannon Hill Corporation, Atlanta, Georgia 4

5 Below is a diagram (Figure 1) of how the process works. Figure 1 example diagram from OfferMatica.com Before we continue, an obvious fact exists that there is more than just one variable that could factor in the conversion rate of a given page. The product description, the placement of the Submit button, and its wording are equally valid variables that could factor in a person s choice to buy the product or leave the page unsatisfied. The product/landing page itself and the choice to lay out the page in a two-column format could also be considered as well. The A/B test, much like all tests, has limitation to it that must be considered when evaluating the results that it produces. The existence of Confidential, Hannon Hill Corporation, Atlanta, Georgia 5

6 a scientific approach, however, is better than no scientific approach at all, especially when time and costs are taken into account with trial and error tests. The costs to an organization that chooses to serve up one variation of a page for a month to all visitors and another version the next month, for instance, are likely to be much higher due to the timing of product offerings. For example, a company that sells outdoor pools in August could lose a significant amount of income if they were to implement a variation of their product page that was not as good as September s version of the page. More revenue could have been generated if both versions of the page were served up during that month in a 50/50 fashion while affording management the opportunity to see a side-by-side comparison of the success rates of the pages. Returning to our example of a simple product page, we are able to view the success rate of the two pages via metrics performed on the usage data. If version A our control variable, proves to be more successful in turning visitors into buyers, then the recommendation is to take the logical step to evaluate another variable on the page that could affect user buying behavior. If version B is more successful, consider the new product image a step in the positive direction. Similar to the situation when the control variable A was more successful, there could be additional factors that should be tested for version B in addition to the new product image. With each new variable that needs to be tested another A/B test to be set up and performed. The choice of the control variable for new experiments is up to the person conducting experiment. If there was a noticeable margin of conversions with a particular page version in the previous experiment, it would make sense to use that version as the control in the new A/B test. With the ability to view a comparison of conversion rates between the two pages, the A/B Test provides a simple way to evaluate the effectiveness of how a site presents products, services, and information to online consumers. In the next section, we ll take a look at how the Cascade Server product, when used in conjunction with a simple web heuristics API, can allow an organization to perform an A/B Test without additional costs. Confidential, Hannon Hill Corporation, Atlanta, Georgia 6

7 A/B TESTING AND CASCADE SERVER Performing an A/B Test can become a time-consuming process to undertake, especially for the non-technical business user. The typical business user is only interested in obtaining the results of the experiment rather than writing the code routines to perform the experiment correctly. Traditionally, the choices have been to produce a testing environment in-house, use a third-party, choose the traditional method of trial and error, or the most common, don t do anything at all. Fortunately, the Cascade Server product comes with support to conduct A/B Testing along with a variety of other online marketing of features. By making use of the structured data definition technology included with the Cascade Server product, a business user can easily create variations of content to publish out into an A/B test that can be tracked and measured by simple web statistic programs such as Web Trends or WebSideStory. Moreover, Cascade Server provides an excellent framework that allows an organization to take a structured approach to providing the right content in the best format possible. Real World Example In order to provide a clear picture of how Cascade Server can be used to create an A/B Test, we will use a search engine landing page from the Hannon Hill corporate website to demonstrate. In Figure 2 below, the landing page for a search engine is displayed from the Hannon Hill site. Confidential, Hannon Hill Corporation, Atlanta, Georgia 7

8 Figure 2 The figure above shows a screenshot of the current landing page that search engine users see when they click on a Cascade Server advertisement. The page can be broken down into various regions including a header, menu/navigation, default content area, and right-column callouts. The purpose of this page is to generate more sales leads for users who are seeking a CMS to handle their content management needs. The default content area displays a form where users can enter in their information to receive a free content management buyer s guide. In the top of the right-hand callouts column is a promotional item that provides an additional incentive for the user to fill out Confidential, Hannon Hill Corporation, Atlanta, Georgia 8

9 the form. Our A/B Test that we are going to set up will be based on the hypothesis that more users will choose to fill out the form if the promotional item were replaced with another that might appeal to more users. Our current page displays an offer to be entered into drawing to win a free MP3 music player. This current page will serve as our control variable or A version of the page. The B version of the same page will look very much the same but will display a $100 gift card offer to Amazon.com as our test variable. Figure 3 displays the same page again but with the change reflected in the promotional item area. Figure 3 Confidential, Hannon Hill Corporation, Atlanta, Georgia 9

10 Both content variations of this promotional item region are populated by information entered into a structured data definition that is attached to this landing page inside of Cascade Server. Our structured data definition technology is a form-based area where users can enter content quickly and safely without the worry of breaking template design while maintaining structure to the information that is entered. The data definition for our landing page is easily accessible by clicking Edit on the blue action bar. Figure 4 displays the data definition being used to collect the two different versions of our promotional content that we re going to use for our A/B Test. Figure 4 The content in the data definition above should look familiar to the content that is displayed in the promotional regions in the previous figures 2 and 3. There are fields for text content such as the title and main paragraph content as well as a field that allows a user to point and click on a logo image that he or she would like to use in the promotional offering. Additionally, there is a field at the top of each group that Confidential, Hannon Hill Corporation, Atlanta, Georgia 10

11 specifies a unique name for a hidden parameter that will allow us to track the page that users see when they submit the form on the landing page. With our two different versions of promotional region content entered into the CMS, we are now ready to attach a stylesheet to the page that will take the data definition content that we have entered and turn it into JavaScript routines 1 that will manage the randomization of our promotional region content each time the page is served up to a user from our web server. The last step for us to do is to re-publish out our new dynamic landing page to the web server and allow data to be collected from user interaction for an established time period that we would like our experiment to run. During the period of time that our A/B Test is running, our web statistics program will also be running on the web server that will collect information on the user interaction with our landing page. For instance, the web statistics package will be able to track the number of all unique visitors that come to the site while also tallying the number of visitors that viewed which version of the page as well. Lastly, of the number of unique visitors that we can track, we will also be able to track the number of users who submit the form and which version of the page they viewed as well. The end result of our data collection is our ability to see how many visitors initially saw a version of a page and how many who saw that page then chose to submit their information to the form. For example, if version B had 100 visitors, and 10% of those visitors chose to submit the form while version A also had 100 visitors and only 5% of those visitors chose to submit the form, then we would be able to say that version B was more successful than A. Since both versions of the landing page was served up randomly in a 50/50 distribution, and only one variable was changed between the two, we could say that the promotional item on version B was more enticing to users. 1 The details regarding the attachment of the stylesheet and its JavaScript routines are beyond the scope of this article and have been omitted for the sake of clarity. Confidential, Hannon Hill Corporation, Atlanta, Georgia 11

12 CONCLUSION Market testing is a valuable tool at the disposal of an organization to judge the best way to offer up the right content in the most acceptable form. The A/B Test provides a simple framework that allows organizations to perform such market testing. Cascade Server comes standard with a framework that allows an organization to conduct their own A/B Test efficiently and effectively. LEARN MORE Please contact Hannon Hill to learn more about managed A/B Tests for your site. In addition, view the Hannon Hill page described in this document online at: Confidential, Hannon Hill Corporation, Atlanta, Georgia 12