Optimize Windows 8 Campaign Website Using Big Data and Online Survey Results

Transcription

1 Optimize Windows 8 Campaign Website Using Big Data and Online Survey Results Methodology Challenges Web analytics as a discipline is established yet changing as the technology and privacy regulations evolve. However, it is still exclusively focused on behavioral metrics. This is sufficient for Web sites that are transaction driven as their success can be evaluated by behavioral metrics. For brand-building Web sites with limited or no e-commerce mandate, current Web analytics offerings fall short as consumption or visitor path are not indicative of brand perception. Site evaluation surveys can assess the overall perception of site visitors, but these surveys typically don t integrate with site action data. At the same time, although selected Web sites incorporate click-through information from current online campaigns, such information is restricted to the last click and can t incorporate campaigns running off the internet. In addition, perception measurement from site evaluation surveys can t adequately isolate the impact of site visitation from other factors. Empirical research indicates prior exposure to campaign activities, primary reasons for visiting the site, and site actions all have an impact on perception measurement. A special methodology has to be devised to isolate the impact of the site visitation on perception changes. The combined realities above present challenges to the Microsoft Windows Online team. The team routinely creates and updates its Web sites to support campaign activities in order to increase perceptions of Windows software and Windows PCs. The Web site doesn t have an e-commerce section that directly sells Windows PCs to consumers or businesses. At the same time, a significant portion of Windows campaign activities are TV driven. Current Web analytics and site evaluation tools can t adequately answer the fundamental questions What is Windows Online s contribution to the perception building campaign? What can the site team do to maximize such contribution, while at the same time maintaining a positive customer experience (or journey) online? Due to lack of existing established methodology to answer the above questions, the Windows Online team has tried various approaches to arrive at the answers. Integration of survey data and action data captured by ATLAS indicates that the types of actions taken by visitors on the site are not predictive of perception. Since action data alone can t satisfy the need for optimizing the site content and perception, the Windows Online team worked with comscore piloting a comprehensive

2 methodology to answer these complex questions in 2011, and applied the same design for the Windows 8 launch campaign landing environment (CLE) in Understand the impact of campaign activities prior to Web visitation Researchers all agree that visitors intention and campaign exposure prior to visiting the site impact visitor perceptions. In order to account for online campaign exposure, online creative and paid search campaigns are trafficked in a way that the click-through URLs carry pertinent information of the campaign. These notations allow us to separate visitors who came from organic search and those who came from SEM. The types of online creative that visitors clicked on are also coded in the click-through URLs to distinguish from standard banners, rich banners, and online videos. Specific creative names are passed through the URLs. Some of those who entered the site via search results links might have been exposed to online campaign creative prior to visiting the site. In order to account for this, campaign creative assets are tagged with a comscore vce tag. All exposure history to the specific campaign are then integrated with survey results to ensure such exposures are accounted for and recorded. A large proportion of the campaign budget is typically dedicated to TV advertisement. Therefore, prior TV exposure will have to be taken into consideration when understanding impact. Although the standard opportunity to see a set of questions can be used, it is not feasible to add these questions in an online intercept survey that is open to all visitors, without incentive. Rather, a collage of TV commercials is used in these studies to record recognition of TV commercials among site visitors as a proxy for TV commercial. 2

3 With all online and TV campaign activities accounted for, survey data can be cut to understand the impact of these campaign activities in combination with site actions on perceptions: 1. TV alone 2. TV with online campaign 3. TV with search and online campaigns 4. Online campaign alone 5. Online and search campaigns 6. Search alone 7. No prior recorded campaign exposure Isolate the impact of Web visitation One challenge to overcome is how to isolate the impact of Web visitation from other compounding variables: how arrive at the site, reasons for visiting, campaign exposures, landing pages, and site actions. The key is to collect perception before and after visiting the Web site. Previous attempts to use campaign target audience who have not visited the Web site proved inefficient. In order overcome this difficulty, two survey tracks are created on the site. Once a visitor clicks on a paid search link, a display unit, or comes to the site via other means, they could be invited to take one of the two surveys running on the campaign landing pages: 3

4 1. On-entrance survey the click to the site within the advertisement takes a random subset of visitors to a survey invitation page before displaying the site content. Those who agree to take the survey will answer perception, visit and visitor profile questions before going back to the campaign Web site. Those who don t want to take the survey are taken directly to the campaign landing page where they originally clicked. Campaign exposure data will be analyzed among those who take this survey. 2. On-exit survey a small survey invitation is set to appear on the landing page for a random subset of visitors. Those who do not wish to take the survey will visit the site as normal and those who accept the survey invitation will see a survey displayed after they leave the site. Campaign exposure and site action data are integrated with the survey data. Perceptions are compared between the on-exit survey and the on-entrance survey in order to understand the contribution from the campaign Web site in lifting perception in isolation. Respondents are balanced on similar campaign exposure, purpose of visit, and visitor profile. Incorporate the web analytics data with survey data comscore and the Windows Online team have integrated various Web analytics data sources and leveraged various platforms to analyze combined data. For this type of research, survey data and Microsoft s proprietary WEDCS data are integrated seamlessly as part of WEDCS set up WEDCS data carry unique survey IDs from Web intercept surveys. WEDCS data, campaign exposure data, and survey data are then ingested into comscore Digital Analytix platform for analysis. The Digital Analytix platform is used to analyze multiple data sets, and provides the flexibility to merge, code, segment, and report key metrics for analysis. Perception impacts of site actions are analyzed by looking at actions taken and campaign exposure among those who took the on-exit survey. 4

5 Sampling frame All pages where campaign activities can land are subject to survey intercept. All visitors who land on the US campaign pages of the site are eligible for recruitment. The intercept frequencies are monitored and adjusted to reach 6,000 completed on-exit surveys and 2,500 on-entrance surveys every month while the campaign is running. Questionnaire On-exit and on-entrance surveys have very similar questions. The questions focus on brand and competitive perceptions of Windows 8 on various form factors. In addition to perception measurements; PC purchase timeline, preference, stage in the purchase cycles, and TV collage recognition are also included in the questionnaire. The differences between on-exit and on-entrance surveys are site evaluation questions. Research Results, Optimization Actions, and Business Impact The Windows 8 CLE (a collection of campaign landing pages to market Windows 8) is set up as an element of the overall Windows 8 campaign to help its visitors find out more about Windows 8, become familiar with it, and to include Windows 8 in their consideration set for their next PC purchase. Analyses one month into the campaign indicated the need to focus on showcasing Windows 8 on laptop/desktop given the negative impact on campaign KPIs. The data below were collected between October 29, 2012 and November 25,

6 The big data analysis (survey + action data) also indicated the need to re-evaluate the video strategy/content on the Web site. There is a large portion of visitors (41%) who viewed at a least a video, but they have lower purchase intent than those who didn t view a video. The high engagement metric is only indicative of interest in video, but not the impact of viewing the video. Relying only on action data could lead to the wrong optimization decision. In addition to the quantitative data, verbatim comments from the survey provided the explanations and specific directions for optimization. Visitors didn t want just to watch another commercial they have already seen somewhere else. They want to see Windows 8 in action, particularly how Windows 8 will work on their desktop/laptop. They want to understand the rationale to upgrade to Windows 8, particularly on laptops/desktops. In addition, consumers want a direct link to download Windows 8 to their current PC from the campaign landing pages. 6

7 The Microsoft team took the learning from the research seriously and optimized the CLE on December 10, 2012 to include the following changes: o New video in the hero that showcases Windows 8 features instead of showing another commercial o Content that highlights what s new with Windows 8 vs. Windows 7 o Content that more explicitly explains why consumers should upgrade to Windows 8 instead of the original soft-sell approach, especially for desktop/laptop o Disruptor that prominently features the $39.99 upgrade offer so that customers don t have to dig deeper into the page or site to find it As more appropriate videos for the Web site were under development, the Microsoft team also removed the emphasis of video on the landing pages in another round of optimization later in December. The payoffs of the optimization series were quantified in the second round of in-depth analyses with combined survey and Web action data. First, the negative impact on Windows 8 for laptop/desktop was reversed without adversely affecting the positive perception of Windows 8 for tablet/convertibles. The data below were collected from the Web site between December 11, 2012 and January 6, Second, the improved CLE did a better job at convincing visitors to purchase their next PC sooner. 7

8 Although the changes in video presentation in the site didn t have an immediate positive impact on perception of those who viewed the videos, the proportion of those who viewed a video reduced from Wave 1 (41%) to Wave 2 (20%). Meanwhile, new videos are in production to better satisfy visitor s needs. 8

9 The series of optimizations not only improved the outcome of the Web site visitation, but also how the Web site content works in concert with the overall campaign activities, either online or offline. Understanding the interaction between campaign activities and how they work together with the Web site is important as the Web site is the extension of the campaign. The Web site should work with the campaign content to further advance visitors down the purchase funnel. In Wave 1, there was a negative impact on consideration for Laptop/desktop among those who visited the Web site via search, either paid or organic. The optimizations after Wave 1 improved the CLE s relationship with search to produce positive consideration lifts. Furthermore, the improvements with search did not come at the expense of other media as the positive significant lifts were sustained in wave 2. Conclusions and Suggested Future Research/Quality Guideline 9

10 Using a big data platform such as Digital Analytix to combine and analyze survey, site action, and advertising data can yield timely insights for optimization decisions that enhance a perception driven campaign. With a larger volume of data set than what was used for the monthly analyses presented here, more in-depth analyses can be done to look at: Campaign exposure combined with action Online creative type impact on perception Impact of Microsoft-owned and operated campaign assets Impact of Microsoft-generated organic traffic The key to successful data integration is to ensure integration is done at the respondent level between site actions and perceptions, as well as between survey and advertising exposure data. The big data platform should be flexible enough to handle data sources other than Web action data, such as survey and advertising exposure data. Also, the platform needs to be easy to use, such as drag and drop interface, so a researcher without programming experience can analyze the data via the interface to increase speed and efficiency of analysis. From an analytical point of view, the most important preparation work prior to the commencement of the analyses is to balance the on-entrance survey data with on-exit survey data in order to remove any potential biases in the data. For this analysis, we checked and balanced all data that could potentially skew respondents preference in operating system and demographic profile. Authors and Contact Information Kate Boatman, kateboat@microsoft.com Senior Marcom Manager for the Windows Consumer Division, Microsoft Roslyn Ku, rku@comscore.com Vice President, comscore 10