Adding Big Data Booster Packs to Survey Data

Transcription

1 Adding Big Data Booster Packs to Survey Data Scott Porter Carlos G. Lazaro, Ph.D TIME SPENT PRICE CHANGES REVIEWS COMMENTS TWEETS CLICKS

2 Many of our clients have undertaken concerted efforts to improve the data they collect as part of doing business, in order to enable more informed and hopefully better decisions. The increased quality and quantity of non-survey data opens up exciting opportunities for analysis. We wanted to take a moment and focus on a particular opportunity improved analysis by combining survey data with this non-survey data. This paper contains case studies illustrating ways that survey data can be integrated with data from other sources to greatly enhance analysis in effect adding booster packs to the survey data. Examining cases with different analysis goals and using different approaches for blending the data illustrates the wide variety of possibilities for combining survey and non-survey data. We present four cases. Each demonstrates a different way of combining survey with other kinds of data: by ad by time by respondent as a contextual backdrop Case Summaries Case 1: Making comparisons by ad. Direct response advertising data combined with copy test survey data. Our clients using direct response advertising have always been focused on data-driven measures of Return on Investment, and many are now investing in better warehousing of even more granular response data (such as clicks and calls) for each ad. Standard approach: On its own, response data demonstrates which ads perform better, which informs allocation of spend among ads. How we blend the rocket fuel: We model in-market performance (by ad) predicted by copy test results (also by ad) which allows us to make predictions about future performance, enabling our clients to make earlier decisions. The modeling also identifies aspects of the ad related to above- and below-average in-market performance, so clients can make optimizations to current ads and improvements to future ads. CASRO Digital

3 Case 2: Making comparisons in trends over time. Competitive intelligence and sales performance data combined with brand tracking survey data. Tracking competitive pricing information has always been crucial, but now with many categories having a large portion of their sales conducted within online channels where pricing is publicly viewable, it is possible to warehouse very granular competitive intelligence data for future analysis. Standard approach: Using this competitive intelligence and sales data, it is possible to build models to understand effects of competitor pricing changes on sales over time. How we blend the rocket fuel: The impact of competitor pricing varies over time. This is sensible because we presume that perceptions of the competitive set change over time. By combining the modeling results about price effects (over time) with brand tracking data (also over time), we better understand which brand perceptions relate to reduced sensitivity to competitive price drops (in other words, which perceptions enable the client command a price premium). Case 3: Making comparisons by respondent. Consumer behavior data from website activity and transactions combined with survey data capturing perceptions, attitudes, life events, and offsite behavior trends. Our clients track a multitude of onsite behaviors, as well as every interaction and transaction with their clients. Standard approach: By building models using just the behavioral data, we are able to understand different pathways leading to preferred outcomes. How we blend the rocket fuel: When we use respondent-level models to relate customer perceptions (from survey data) to behaviors for the same customers (from client data), we are able to better understand the whys behind behaviors, and prioritize areas for improvement based on understanding of people s needs. CASRO Digital

4 Case 4: Analysis with multiple levels of data. Using survey data as a wide (but thin) overview of the market, to contextualize the deep (but narrow) pockets of non-survey data. Digital marketing efforts are making attribution analyses possible at a different scale because we have consumer-level data on both touchpoints (which marketing efforts are seen, when, and how often, etc.) and responses (visitation, purchase, etc.). Standard approach: As deep and as valuable as this data is, it has blind spots. Some marketing efforts remain offline and analog, and we would like to handle attribution across all efforts. How we blend the rocket fuel: Survey data remains a great way to build proxies for all of the campaign activities and the consumer responses, filling in the gaps for activities where the digital behavioral tracking has blind spots, and serving as a skeleton or framework for attribution modeling. In areas where there is digital tracking, we can enhance or deepen the model, within the big-picture framework provided by the survey data. Each of the cases will remain blinded to protect individual client details, but we will provide information about the analysis goals and the method of comparing/connecting the survey data to the non-survey data so the cases can serve as a source of inspiration for other researchers. Any example data points have been simulated; no actual client data has been included in the paper, but the examples are realistic stand-ins for cases we have encountered. Although all of our cases are from the marketing domain, the cases are informative for any survey researcher trying to plan ways to improve analysis by leveraging survey data in combination with non-survey data. CASRO Digital

5 Case Details Case 1: Making comparisons by ad. Direct response advertising data combined with copy test survey data. Both direct response advertising data and copy test data are valuable on their own for analyzing ad effectiveness. Direct response advertising allows for the audience to respond directly to the proposed offer 1. In our case, the ads were video ads and the methods of response provided were phone or web. Using the response data, we can understand how many people responded to particular offers/ads. By comparing how many people viewed the ad to the volume of responses, we can calculate how efficient each ad is at generating response. Observed Expected Time Figure 1: Performance of an individual ad in a specific market compared to expectation given type of ad, market, and advertising pressure. Just using the direct response data, we are able to build a model for the estimated response for an ad based on its offer type, the audience being 1 direct response advertising: An approach to the advertising message that includes a method of response such as an address or telephone number whereby members of the audience can respond directly to the advertiser in order to purchase a product or service offered in the advertising message. Direct response advertising can be conveyed to members of a target market by a wide variety of advertising media, including television, radio, magazines, mail delivery, etc. Source: CASRO Digital

6 advertised to (geography, media, etc.), and how long the ad has been on air (ads tend to wear out and generate relatively less response over time). We can then compare the actual response of an ad to the modeled expectation to see if the ad is over- or under-performing versus this expectation. This kind of analysis informs the best allocation of budget between ads, and can alert if ads/offers need early replacement if they begin to wear out sooner than expected. Figure 2: Aggregated performance for an ad across all markets compared to expectation, to evaluate wear out of the ad. Copy research is an approach for understanding ad quality by testing audience reaction to the ad. 2 In copy research, we collect audience reactions in a survey conducted at or before the time the ad airs on media. Sections of the survey are standardized, and the audience composition is held constant from test to test, to make it possible to make comparisons between ads on various metrics. For example, we can find out how interesting the audience finds the ad and compare that to other ad tests 3. Copy research can include a wide variety of metrics that can be used in diagnosing a direct response ad, including perceptions about the brand, product or offer after viewing the ad, intent to respond to the offer, etc. 2 copy research:the testing of audience reactions to advertising messages while the advertising is being developed (called pre-testing) or after the advertising has been produced in final form (called post-testing.) Source: 3 For a common measure like this that is included in virtually every copy test we will have tens of thousands of previously tested ads to serve as comparison points and benchmarks for evaluating new ads. More specific metrics that only apply to certain product categories or certain types of ads will have smaller comparison sets: for example, only ads from the same category or only other direct response ads. CASRO Digital

7 Although the direct response data gives us a detailed way to understand the performance of the ad after airing, the copy test results have the advantage of being collected before the ad airs. The earlier the client can make certain tactical changes, the more they benefit. If a new ad outperforms an older ad in driving sales, the earlier the better-performing ad can be rotated in, the greater the economic benefit to the client. Also, early detection of problems such as an ad wearing out sooner than expected will alert the client to accelerate development of new ads to replace it. Sample Timeline Storyboard Finished Spot Spot Airs Expected Begin Wearout Copy Test (Pretest) Sufficient Copy Test response (Posttest) data Figure 3: Copy test data can be collected before any response data is available. We use predictive models to combine the copy research data with response data, looking for aspects of the copy test that predict ads that performed above or below expectation. An additional benefit to prediction of performance based on the copy test is that it illuminates why certain ads perform above or below average. Different models may be appropriate depending on what aspect of the response data we wish to predict, for example, predicting early performance of the ads as opposed to detecting ads that wear out earlier than normal. Certain copy test results, such as an indication that the ad could be annoying to part of the audience, may not be as important for predicting early performance of the ad, but very important for understanding the ad s longevity. CASRO Digital

8 Figure 4: Just knowing whether or not we have a clear winner or it is too close to call based on the copy test is important information. In the cases with clear winners, we can make preparations for changing out ads sooner, while in cases too close to call, additional decisions can be made after response data is available. Although in-market response data is obviously a more exact measure of ad effectiveness, predictions based on copy test data are beneficial. Even a model that only roughly predicts ad performance may have high enough fidelity for practical decisions. For example, for selecting the best ad out of two available ads, the copy test model really only needs to either 1) identify an ad as a clear winner/loser, or 2) indicate that the copy test results between the ads are too close to call, in which case response data will need to be collected before we can confirm the higher performing ad. Rather than making decisions on one or the other set of data, decision makers can use a fusion of the two. Early decisions are made based on the copy test results, later ones augmented with the in-market response data as it is collected. Both the copy test and response-data-based models can continue to be refined as we collect more information. CASRO Digital

9 Case 2: Making comparisons in trends over time. Competitive intelligence and sales performance data combined with brand tracking survey data. In many categories, a large portion of sales are conducted online in channels where pricing information is publicly viewable. These digital sales channels potentially include large amounts of competitive intelligence data that change on a more or less continuous basis, and can be warehoused for future analysis. Combining this competitive intelligence and sales data, it is possible to build models to understand effects of competitor pricing changes on client sales over time. We built such a model for one of our clients. In our client s category, it is common for competition to manage volume targets by changing price. Of course it is not always clear what is driving price drops by competition, but in many cases there is evidence pointing to reasons underlying the price shifts ranging from simple oversupply to competitors trying to compensate for image or word-ofmouth problems and still meet volume targets. Our client s positioning allows them to maintain a higher price point. But competitive shifts in pricing still impact their sales, and are important for them to monitor. The impact on sales due to competitive price shifts has varied over time and understanding when sales are more versus less affected by competitive price changes was a key focus for our clients. In addition to their investment in competitive intelligence and sales data, this client has also invested in survey-based brand tracking. This is a continuously fielded survey among decision makers in their category that collects perceptions about the client brand and competitor brands (both of which continuously change in response to advertising, word of mouth, and industry news), as well as other relevant metrics about the category. CASRO Digital

10 Figure 5: Simplified version of our situation. Sales have a strong seasonality component and are also affected by market factors such as frequent pricing changes by client brand and competition. We decided to connect the survey data to the pricing and sales data by time period. We looked to see if we could understand which brand perceptions predict the periods of time when the client brand was more resilient to competitive price changes versus periods of time when the client brand was more affected by competitive price changes. By doing this, we were able to identify perceptions that were important in maintaining their resilience to price changes by competition. CASRO Digital

11 Figure 6: Sales after accounting for known market drivers such as price and seasonality can then be compared to brand tracking metrics to look for brand effects. Both the pricing/sales data and the brand-tracking data were useful on their own for informing client decision making. However, by combining the two data sets, we helped the client zero in on a particular goal: understanding which brand perceptions were important for maintaining their price premium. Neither data set in isolation was sufficient to answer this particular question. CASRO Digital

12 Case 3: Making comparisons by respondent. Consumer behavior data from website activity and transactions combined with survey data capturing perceptions, attitudes, life events, and behavior trends offsite. This case illustrates the area that is probably most exciting about the increased focus on keeping and leveraging data from business operations for analysis purposes: individual-level data. Analyzing individual data allows us to identify relationships between variables that might be missed when analyzing aggregate data. Our client in this case is a large online retailer. They currently collect and already successfully leverage a large amount of data on every transaction conducted by their users: when they visit, if the visit was in response to an action by the client, how long they spend on various activities, what they look at, what they buy, etc. The particular business question we explored was why a group of their customers demonstrate a lapse in purchase behavior, while continuing to visit on occasion. Past research they had conducted had been in two veins: analysis of the behavioral data, and surveys of their user base. Examining the behavioral data had been able to confirm the trend, and indicate what categories of items were more affected. Surveying their users brought several potential issues to light, but it was unclear how much of the problem to attribute to each potential issue, nor how to prioritize areas for improvement. Combining both data sources allowed us to achieve more actionable insights. We began by more careful modeling within the behavioral data. This revealed that there were actually two behaviors at work: 1) decreased visitation, and 2) decreased purchase conversion even among those who visit. CASRO Digital

13 We then fielded a survey among the client s users, in such a way that we could append key behaviors from the behavioral data to the survey data. We built models to understand the relationship between perceptions, attitudes, and background variables measured in the survey and behaviors from the client data such as decreased visitation and decreased purchase conversion. With the individual-level data, we were able to better attribute which potential causes were responsible for what portion of the problematic behaviors among this group of their users. Figure 7: Using the model, we simulate the impact of reasonable improvements on a wide variety of contributing factors. We used the model to predict which potential areas for improvement, if addressed, would generate the largest reduction in the problematic behaviors, and this was used to prioritize their development strategy. The detailed level of the modeling allowed us to provide clear and actionable direction for the client. Because we were simulating specific improvements, the client could assign rough costs/time to each development effort, and then trade off the benefit with the cost. In this way, we were able to identify an affordable set of developments that still represented a sizeable improvement in the problem (see Figure 8 for details). CASRO Digital

14 Figure 9: Expected impact of the top 10 prioritized improvements represents a reduction in the problem by 30-40%. Modeling the impact enables prioritization of intervention efforts by allowing selection of innovation opportunities within budget constraints while still representing a sizeable improvement. The question could not have been answered as completely using just the survey data or just the behavioral data alone. It would not have been as reliable to ask respondents their past behavior directly. We already had detailed information on visitation and purchase, and even in cases in which we could count on respondents memories to recount their behaviors accurately, we would have only been able to ask very high-level behaviors in a short survey. The behavioral data, while detailed, is missing the background information on the why driving their behaviors. By asking respondents about their brand perceptions, changing interests, etc. and then relating that to the behaviors, we are able to attribute various causes to the behaviors. CASRO Digital

15 Case 4: Analysis with multiple levels of data. Using survey data as a wide (but thin) overview of the market, to contextualize the deep (but narrow) pockets of non-survey data. As clients get more used to the benefits of the deep wells of individual-level data we have in activities such as digital marketing, it can be disconcerting to then realize that there are still areas of the business where our new and improved data warehouses still leave us relatively blind as to what is happening. The wide differential in terms of data available could be because certain markets or channels may rely more on traditional or analog methods of marketing. Also, we don t always know exactly how our wells of deep digital data fit into the larger picture of the overall market. Although the areas where we have deep, digital tracking continues to expand and these blind spots will likely shrink, there is still great advantage to a single overview of the market that contextualize the deep pockets of data we have by projecting the incidence of behaviors and the relation of those behaviors to other behaviors and attitudes. Survey data can be used to build a view of the entire market landscape, including: proxies for online activities attitudes and intentions that are only available by asking respondents proxies for offline behaviors CASRO Digital

16 Can potentially be tracked online Digital Buzz Site visits Brand Association Awareness Behaviors OnlineVideo TV Attitudes Figure 10: Survey data can be used to understand wider set of relationships, while digital tracking can simultaneously be used to dig deeper into relationships between digital behaviors by examining individual-level data. After building a landscape of the market using survey data, understanding of certain relationships can then be refined using the more granular behavioral data. For example, understanding how exposure to a multi-media campaign relates to attitudes and offline behaviors might be best understood at the market landscape level, while understanding the social sharing of the digital portion of the campaign can be deepened by digging into the well of specific comments and behaviors that are digitally tracked at an individual level. This approach is useful even if the wider landscape is a completely separate analysis simply used as a backdrop for contextualizing the deep dive data. But the integration can be taken even further and analysis of the different levels of data can approached jointly when building models. Information from the deep dive into the individual digitally-tracked data can be used when building the landscape model and vice versa. Our clients who invest in survey-based landscapes to track and understand their entire market are also invested in deep digital tracking and attribution. By leveraging both, we can understand the big picture of how the pieces of the strategy are working together while still leveraging insights on how to optimize individual pieces using the deepening data available. CASRO Digital

17 Summary Each of these examples has illustrated a different way to blend survey and nonsurvey data, and each has a different strategic focus. However, there are commonalities: In each case, the question was better answered by the combination of both the survey and non-survey data than could have been accomplished by either alone. The non-survey data is generally used to get to a more detailed answer to what than would have been possible in a survey and the survey data is generally used to explore evidence about why (although this is a fuzzy distinction, and some pieces of evidence fall in the alternate category). Predictive modeling is used to enable the blending of data in a way that makes it strategically actionable. o By combining data sources using models, we are able to get to extremely focused answers, for example: Instead of simply identifying important ad attributes, discovering which ad attributes lead to early wear out Instead of simply identifying important brand perceptions, clarifying which brand perceptions enable the brand to command a price premium. Instead of simply identifying improvements that should be made, selecting a tactically feasible list of improvements that still makes a desired level of impact. We explored several different ways of blending data: By ad (and a similar approach can be used to analyze by product, program, etc.) By time By individual Multiple levels of data As the quality and quantity of non-survey data collected by us and our clients continues to improve, we expect that survey research organizations need increased focus on the competency of integrating survey and non-survey data. One of the first questions we should ask when kicking off a survey-based research project is: what other data is available? Then, we should decide how that might influence our research design, and how we might leverage that data, together with our survey data, to better answer our clients questions. CASRO Digital