Today s Tracking Studies New Techniques and Best Practices by Chris DeAngelis, Vice- President of Sales, North America, SSI Viktor Bergh, Sales Director, Nordics, SSI January 2016 Survey Sampling International, 2016
ABOUT THE AUTHORS Chris De Angelis is Vice President of Sales at SSI. He has almost three decades of experience working closely with SSI's customers to provide sampling solutions for their research challenges. He has provided consultation, expertise and sampling solutions for a wide variety of commercial, government and academic research projects. Viktor Bergh is Sales Director, Nordics at SSI. He has over a decade of experience conducting online market research projects and working with a wide range of clients and different types of projects. Viktor joined SSI at the start of 2015 and is helping our Nordic clients to optimize and fulfill their online research projects. Prior to SSI he worked in several European markets for local companies gaining extensive knowledge of the differences and nuances between markets. Offices worldwide info@surveysampling.com surveysampling.com 2
WHY DO TRACKERS? Some people have been predicting the death of trackers for years now but at SSI we see that while sometimes conducted less often, trackers are alive and well because when done right they deliver deep, valuable insights to provide management with relevant, accurate, reliable and current information. Marketers and brand managers need reliable information to understand today s complex and fast- changing environments. Trackers repeated studies designed to understand changes over time can provide vital information to help steer business decisions. Tracking studies may last for just a few days, or many years. They may be repeated daily, weekly, annually, or at any regular interval. The critical thing about a tracking study is that it is designed to measure real world changes, which might be changes in awareness, in behavior, or in purchase intent. Therefore consistency in the way the tracker is run is paramount to ensure that changes seen are real market changes, not changes caused by differences in question wording, sample or some other aspect of the study. As well as measuring brand performance over time, trackers help increase profitability by identifying new opportunities to grow a brand, change consumer perceptions about it or even target new customer segments. Trackers can often be very large and therefore expensive to run, so today s trackers must earn their keep. They must be executed more efficiently, and produce data which is reliable, relevant and recent. The key to a tracking study s success is that it has to add value with high- quality actionable information. HOW TO PLAN AND EXECUTE A SUCCESSFUL TRACKER There are four key ingredients in a successful tracker: 1. Meticulous planning 2. The right sample 3. A well- designed questionnaire 4. An experienced, proactive delivery team 1. Planning Everything starts with clear planning and goals. What is the goal? If you had all of the data in front of you now, what would it look like? What would you do with it? What are the underlying business issues you want to understand? 2. The sample It is essential to fully understand the feasibility of the tracker throughout its lifetime, especially in small markets or with narrow population targets. Even in large markets there may be limited numbers of 3
participants in your key demographics. If feasibility is tight, think about the population segments you will be analyzing. Is it necessary to have an equal number of men and women? Rich and poor? Young and old? Depending on the topic being studied it may not be. Will the population being targeted become scarcer in the future? This could happen if the target is users of a product which may become less popular over time a study of home PC purchasers for example. If feasibility is tight you might consider shortening the lock- out period. Research done by SSI and others has demonstrated how difficult it is to condition people. It may be possible to have some of the same people take the survey again after a few weeks or months have passed. To further help with feasibility, make sure your questionnaire is mobile- friendly, so as not to exclude people who will want to take it on a mobile device. SSI sees over 25% of our panelists choosing to take a survey on a mobile device. Without these people, both feasibility and representivity could be at risk. This is particularly true in international research. It is important to understand not just whether the sample is large enough to sustain your project, but what the sample consists of. You will want to understand the sample sources and whether they are proprietary panels, river or web intercept sample, (sometimes called just in time sample) or a mixture of these. Has your sample been validated by a third party for consistency? If multiple sources are used, how are they combined? At SSI, to maximize consistency we blend sample based on broad personality and behavioral traits of individual people, not on individual sources because sources can and do change over time. (For example the profile of Facebook users a decade ago was mostly US college students very different from today s Facebook user population.) 3. The questionnaire Tracker studies are especially at risk for poor data quality because many are so long that participants get tired and become inattentive before the end. Many trackers suffer from having been designed by committee with everyone s favorite question included. SSI s research across many different countries and cultures and over more than a decade shows that participant engagement starts to diminish after about 15-20 minutes. Our studies rotated question blocks within a longer questionnaire and found that participant behavior changed and answers were different depending on where in the survey the question was asked. For example, after 20 minutes, people were more likely to hurry through the questions, more likely to decide not to move a dial but leave it in a default position, and used fewer words in open end questions. We also found that people were less likely to agree that they had done things like going to the movies. The reason might be that participants have learned that if they say they have done an activity the survey will ask them even more questions or it may simply be that at the 25 or 30 minute mark it is just too difficult for people to make the effort to recall whether or when they did an activity. There are a multitude of ways that questionnaires can be shortened. And it is important to think about this in terms not just of number of minutes, but also length in terms of cognitive burden. 4
Submit a first draft to rigorous review and be ruthless. Because shorter surveys product better quality data and cost less to execute ask yourself and the team for every question: 1. Could we find the same information by other means? 2. What will we do with this information 3. Will the answer to this question ever change? 4. If the answer to this question changes, is there anything we can do about it? 5. Is this question just nice to know? or is it essential? 6. Does everyone have to answer every question every time? 7. Can the sample provider supply demographic and geographic information to reduce the number of those questions asked? 8. Can we reduce the number of words in the questions or use a graphical solution to reduce cognitive burden? While it is important to pare down a questionnaire to the essentials, we must also ensure that every aspect of the research problem is addressed, because adding questions later can make comparisons with previously- collected data difficult. Check and re- check for spelling and name errors. If a brand name is wrongly- spelt it may place all the data in that section in doubt because the participant might have been led astray by the question. We highly recommend doing a test run to be sure the questions elicit the data you expect. A trial run can also allow you to see if some questions are confusing to participants or identify other questionnaire design issues. We recommend using a brand list that is small enough to fit on a single page i.e. whichever device the participant is using, they should not need to scroll to view the entire list. SSI research has also shown that instructions which limit people s ability to answer accurately such as Select three brands that you would not like to purchase should be avoided. People may be willing to consider all the brands, or may only have one or two brands they would not purchase. If forced to list three they will tend fill in the boxes with little- known brands as a way to get through the question. Let the participant choose freely how to answer the question, whenever possible. 4. The support team Written standard operating procedures and processes are important, but a skilled sample, field and data delivery team can play a big role in ensuring that a tracker executes successfully in field. They become your eyes and ears, alerting you to anything unusual in the data or even to external events which might impact the study. They will get to know your preferred communication pattern, and know how to clean and analyze the data on time to your specifications. GLOBAL TRACKERS Market research is a truly global industry, with the greatest growth areas now in APAC and the Middle East. It is important to understand different cultures and values in the countries that you will research and the legal constraints in each market. Working on a multi- country tracking study can be a daunting task. It is essential to understand cultural 5
nuances and ensure the questionnaire is not just translated but also localized. For example, can you ask about ethnicity in France? What about income in Spain? Is it okay to ask about religion in Holland? Do they have social class in Sweden? Is the distribution channel for your client s product the same in every country under study? Are the logos or brand names different? A presentation at the 2015 ESOMAR Congress discussed how differently colors and symbols are interpreted in countries across the globe. We also know that numbered scales may be interpreted differently, with 10 signifying a high score in some countries, a low score in others. Scales should have verbal labels to avoid confusion. When it comes to interpreting the data it is important to know for example that there is an acquiescence bias in some cultures, and that people in China often report product ownership aspirationally, especially for luxury brands: This is the car I want to have, or expect to have rather than the car currently owned. WHAT IF CHANGE IS NEEDED? Consistency is the Golden Rule with trackers, but sometimes a change is necessary, especially for a long- running tracker. For example, when more and more people are choosing to take surveys on mobile devices, we will want to include those people, and will need to redesign the questionnaire to work correctly on a smartphone. Demographic quotas may have changed if the target market is aging for example. Competitor brands may be different, necessitating changes to brand lists. When changing methodology, we recommend a three- step approach: 1. Begin by mapping out potential changes and their relative importance. Some changes might be essential, others less important. 2. Do a parallel test. A full test may not be required just testing certain key questions could be sufficient. Usually there will be data differences when changing methodology. 3. At this point you can decide if the differences are acceptable and adjust benchmarks accordingly. A second parallel test may then be required to confirm findings and ensure confidence in the data before making the changes. Expect changes and prepare to make adjustments. Changes are not necessarily a bad thing but you need to understand them up front and make adjustments. Finally, when planning trackers, always think mobile. SSI TRACKER CASE STUDY A large pharmaceutical company in the healthcare sector, came to SSI with a problem. They wanted to create a harmonized brand tracker across as many markets as possible and needed to collect data on both a corporate and local level. The goal was to understand how marketing activities were impacting the brand and product usage. The company wanted to keep the study in the field continuously across all the categories in 40 different markets over many months. Some of the markets had low brand penetration and the markets were very different culturally. Potentially valuable local data about the brand was incomplete, and not available in a convenient format. The budget and timeline were tight and the stakes were high: 6
the client would use this data for key corporate decision- making. The solution SSI delivered was a continuous data collection tool across the 40 countries. The questionnaire was reviewed together with the stakeholders and harmonized so it provided value across the many internal clients. A single solution and single global support team gave the client the consistency they required and allowed delivery of data which mapped accurately to their Key Performance Indicators. The team also significantly improved and simplified the communication and project management of this very large, complex project by having a single, focused team approach to the questionnaire, the sample sizes, sample definition, quotas and data delivery. ABOUT SSI SSI is the premier global provider of data solutions and technology for consumer and business- to- business survey research, reaching respondents in 100+ countries via Internet, telephone, mobile/wireless and mixed- access offerings. SSI staff operates from 30 offices in 21 countries, offering sample, data collection, CATI, questionnaire design consultation, programming and hosting, online custom reporting and data processing. SSI s 3,600 employees serve more than 2,500 clients worldwide. Visit SSI at www.surveysampling.com. 7