Conventional Survey Methods

Transcription

1 Single Attribute Random Assignment: A True Measurement of Customer Experience Marketing Science for Business Decisions Single Attribute Random Assignment design represents a new way of thinking about measuring and improving customer loyalty and sales in the emerging social media, online and mobile retail markets. By Steve Runfeldt, VP Analytical Services, Ryan Wright, Associate Consultant, Schwartz Consulting Partners, and Alan Runfeldt, Design Architect, ASK3.com Emerging technology has changed everything in business today, except our way of thinking about market research methodology. Single Attribute Random Assignment design (SARA) represents a new way of thinking about measuring and improving customer loyalty and sales in the emerging social media, online and mobile retail markets. Single Attribute design, as implemented in Schwartz Consulting s ASK3 applet, respects customers time and intelligence while providing marketers with a true measure of their loyalty. Increasingly we are shopping online or using our smart mobile phones to locate the best deals or the nearest stores. Half of all young families no longer have landline telephones in their homes. Not to be left behind, traditional retailers are ramping up their online and mobile experience, tying in-store sales to online support and instant mobile coupons. Yet, the majority of market research studies rely on methods developed in the 1950s that require consumers to take 10 to 20 minutes to provide tedious and redundant answers to endless and seemingly meaningless questions. Conventional Survey Methods Like having a clerk on every checkout page to ask, Is there anything else we can do for you today? Conventional methods of measuring consumer attitudes and relationships were developed in the mid 1900s when the telephone and television were emerging as the primary methods of communication in the marketplace. Prior to this time if a retailer wanted to know what his customers thought, he would just ask them; Is there anything else we can do for you today? With the advent of mass marketing it became impracticable for retailers to talk to every customer. So market researchers turned to the science of sampling and statistics in order to measure customer opinions. The in-depth opinions of a few hundred survey respondents came to represent the views of millions of customers. Technological innovation today allows a return to the day when retailers can talk and listen to every single customer. Instead of asking a small number of customers to take 20 minutes out of their busy schedules to complete long and tedious surveys, applications like ASK3 that use the SARA design allow marketers to engage every customer with a few short questions, taking no more than 10 seconds of their time and providing a truer measure of their loyalty. 1

2 Conventional methods of measuring consumer attitudes and relationships were developed in the mid 1900s when the telephone and television were emerging as the primary methods of communication in the marketplace. Conventional survey methods typically ask batteries of probing questions designed to measure the customers attitudes and opinions about a business s products or services. Customer satisfaction and loyalty questionnaires are designed to do several things, including: Identify market demographics: Who are our customers? Measure overall customer experience: How are we doing? Determine those factors that drive a positive customer experience: What are we doing right or wrong? Identify areas where improvement will most effectively improve the customer experience and sales: Where do we need to change? In order to accomplish these goals customer satisfaction and loyalty surveys typically include between 30 and 50 questions from within three critical question sets, including: 1. Conventional customer surveys typically begin with a measure of overall satisfaction or loyalty. This is typically a 5 or 7 point scale question, Please tell us how satisfied you are overall with our product? Overall measures questions may also ask about the customers likelihood to return or recommend the product or service. 2. Conventional surveys then typically include a set of probing questions about specific attributes of the company, their products or services. Attribute questions typically use the same scale as the overall measure and may include ratings of: Prices Product quality Product durability Delivery Ease of finding what the customer was looking for Customer services quality Product variety Shipping or delivery Ability to meet customer needs Other aspects of the purchase or service experience 3. Finally, conventional surveys ask a set of demographic and behavioral questions designed to identify target markets, including: Gender Age Income Purchase behaviors Media habits Other relevant facts about the respondent 2

3 With Internet and emerging media, it is now not only possible, but preferable, to ask customers just two or three questions at a time instead of 20 or 30. And, it is now both possible and preferable to ask every customer to participate. Analysts examine the relationships between these three question sets in order to identify the drivers of customer loyalty among different customer segments. A True Measure of Customer Loyalty A True Experiment is one in which each subject is randomly assigned to one specific treatment rather than being given a battery of tests on different topics. In the past, the true experimental method was impossible to implement in market research. The expense was simply impractical. But with Internet and emerging media, it is now not only possible, but preferable, to ask customers just two or three questions at a time instead of 20 or 30. And, it is now both possible and preferable to ask every customer to participate. In a Single Attribute Random Assignment study each customer is asked just two rating questions at a time; one measure of the overall customer experience and one probing attribute. A third demographic/behavior question is added to provide segmentation and focus. The questions are selected randomly from a question array based on a traditional customer loyalty questionnaire. Each subsequent customer is asked a different set of three questions. The three question applet may be inserted within the checkout page of a shopping cart. From the customer s perspective the experience is much like being asked Is there anything else we can do for you today? at the cash register. Rather than being an imposition, the ASK3 applet serves as an enhancement of the customer experience, providing consumers with an opportunity to give feedback. The ASK3 applet is small enough to be sent to a mobile phone, complete in three questions on one mobile screen. Unlike other methods, ASK3 feedback is incorporated as a part of the online shopping experience. The ASK3 applet uses no obtrusive popups, no links to other web pages, no invitations. The customer is never distracted from the transaction. How it Works The ASK3 applet may be inserted into a web page, wherever customers expect to interact. Ideal locations may be within the checkout page of an online shopping cart or a social media page. The applet is small enough to be sent to a mobile phone, complete in three questions on one mobile screen. This method is also ideally suited for other types of interactive media, for example ratings of on-demand films or programming where more than a single answer response is desired. 3

4 Instead of asking a small sample of customers or survey panelists to complete a 20 minute survey, ASK3 provides every customer with the opportunity to give feedback in about 10 seconds. The ASK3 applet inserted into the checkout page of an online travel website. The feedback process becomes a natural part of the transaction. The customer is not inconvenienced. No special incentives or rewards are required as the cost to the customer in time and inconvenience is minimized. Each customer is asked three short questions located within an inline frame. When the customer clicks Submit the text within the frame changes to a Thank You message, leaving the rest of the page unchanged as the customer continues to the checkout without interference. Instead of asking a small sample of customers or survey panelists to complete a 20 minute survey, the Single Attribute Random Assignment method provides every customer with the opportunity to give feedback in about 10 seconds. Response rates are typically in the 90% range. The feedback process becomes a natural part of the transaction. The customer is not inconvenienced. No special incentives or rewards are required as the cost to the customer in time and inconvenience is minimized. The next customer sees the same page, but with a different attribute and demographic questions. 4

5 Each three-question set always includes one overall measure, one probing attribute and one demographic or behavior question. Each set of questions is selected from a full questionnaire array according to an algorithm that ensures that: Each three-question set always includes one overall measure, one probing attribute and one demographic/behavior question. Each permutation of overall, probing attribute and demographic/behavior question appears exactly once per rotation. The order of the question sets is determined according to a balanced randomized algorithm. What About Sampling? The key to sound quantitative market research is sampling. Research sampling methods are based on theories developed during the early 20 th century. It was simply not possible to conduct a census of every customer, so statistical methods were developed to enable researchers to project results from a small sample of people onto entire markets. Initially respondents were sampled randomly in door-to-door and mail surveys. Later, random digit dialing was introduced to recruit statistically valid samples of telephone respondents. The Internet has proven to be a much more convenient medium for surveys than the telephone, except for sampling purposes. Random selection of respondents from the Internet population is extremely difficult to achieve. Most researchers rely on panels of people who have opted-in to participate in surveys, raising the issues of self-selection and professional respondents. Most panel members are compensated for completing their online surveys. Websites advertise Make money taking surveys. Professional market research panel providers take precautions to reduce respondent fatigue, straight-lining (the practice of rushing through a survey without reading the questions), and outright fraud. Online sample providers have developed methods for river sampling whereby banners, popup ads and links on networks of websites capture respondents for near-random sampling. Sampling methodology was originally developed because the cost of surveying every customer (a census) was simply prohibitive. With emerging technology that cost is no longer based on the logistics of reaching the customer. Survey costs are no longer based on the rate of pay for the telephone interviewer, but on the rate of compensation to the online panel respondent. The cost is now borne by the respondents in terms of inconvenience and the time it takes them to answer our questions. By randomly selecting three questions at a time, ASK3 minimizes the cost to the respondent to about 10 seconds per completed response. 5

6 By randomly selecting three questions at a time, ASK3 minimizes the cost to the respondent per completed question set to about 10 seconds per completed response. ASK3 randomly samples the questions, not the customers. With single attribute random assignment design the entire market is the sample. When an online retailer places an ASK3 applet on the checkout page of their shopping cart, they receive feedback on every customer transaction. Single attribute random assignment design randomly samples the questions, not the customers. Customers are randomly assigned to the different question set groups. Questions are combined according to a balanced random design so that every probing attribute is asked in random order once every N a customers. Every demographic question is asked in random order once every N d customers. Every attribute is paired with every demographic question once every N a x N d customers. So, if the study calls for 10 attributes and 5 demographic questions, every 50 customers will receive a different combination of questions; each attribute will be rated 5 times and each demographic will be asked 10 times. For the first 100,000 customers, each attribute will be rated 10,000 times, and each demographic will be asked 20,000 times. Sample cell sizes vary by question depending on proportions. In the above example, if the market is 50% female and gender is asked as a demographic question, the sample would include results for approximately 10,000 women with each attribute being answered by about Because ASK3 is a census, statistical reliability at the broadest level is near 100%. Within the smallest cell reliability would be approximately 97%. Analysis of Single Attribute Random Assignment Results Traditional customer loyalty studies typically employ multivariate statistical tools to delineate the key drivers of satisfaction and loyalty and to identify those areas of marketing, sales and service where additional efforts are most likely to produce the greatest improvement. Analysis typically involves multivariate methods such as Structural Equation Modeling (SEM), Principle Components Analysis (Factor), Multiple Regression, Discriminant, and Cluster, among others. One difficulty faced when analyzing customer loyalty survey results concerns a statistical hurdle called multicollinearity. Multicollinearity occurs when survey respondents are asked to rate multiple attributes at the same time. Often the differences between the attributes are less than the differences between the people. When a person is asked to rate 20 attributes at one time, it becomes difficult to say how much of each response is due to the person and how much is due to the attribute being measured. Some attributes, like price and value, may be naturally related to one another. Others, like sales person courtesy and product quality may not be intrinsically related, but because they are being rated by the same groups of people, even these seemingly unrelated 6

7 A true measurement method produces zero multicollinearity sideeffects. Single attribute random assignment design shows respect for the consumer by asking for no more than seconds of their time, yet the method yields a rich and true measure of consumer opinions. variables are often statistically correlated. Research analysts employ special statistical techniques to tease the real differences between the attributes apart from variance due to differences between respondents. Single attribute random assignment design is a true measurement method, producing zero multicollinearity side-effects. Since each customer is asked about only one attribute at a time, the ratings of each attribute are statistically independent from one another. The relationships between the overall experience question and each attribute can be determined by simple correlation or proportional analysis. Because they are measured independently, each attribute rating will be a truer measure of that attribute s impact upon customer loyalty and sales. Single attribute random assignment design represents a new way of thinking about online and mobile market research. It employs the best of established quantitative research methodology while resolving the central problems produced by the over-surveying of the online and mobile markets. This method shows respect for the consumer by asking for no more than seconds of their time, yet it yields a rich and true measure of the consumer experience. For more information on ASK3 and Single Attribute Random Assignment methodology, please call Schwartz Consulting Partners at , go to or ASK3@jettsurvey.com. 7