Full report. The 2012 Confirmit Market Research Software Survey. Ninth annual survey by meaning ltd

Transcription

1 The 2012 Confirmit Market Research Software Survey Full report Ninth annual survey by meaning ltd Report written by Tim Macer and Sheila Wilson, meaning ltd independent advice on technology for research

2 Contents 1 Background to the survey The Sample Demographics 8 2 Part 1: New questions 9 3 Survey length Survey length: the shift to shorter interviews Methods adopted to shorten surveys Natural survey length Natural survey length: distribution of responses 15 4 The future for mobile apps Preferences for apps or browsers for mobile surveys Predicting the future for apps and browsers 19 5 Panel rewards and quality control Prevalence of incentivised panel surveys Distribution of incentivised panel use Incentives used Survey quality measures applied Survey quality measures at a detailed level Use of independent panel verification services 27 6 Communities Who runs communities? Change in uptake since Number of communities operated Predicted growth for communities Software used to administer communities Required capabilities in community software tools Satisfaction with community software capabilities 35 Part 2: Trends 36 7 Research activities Research mix 6-year trend Company revenues from research in Confirmit 2012 Annual MR Software Survey: Full report 2 of 63

3 7.3 Volume of work Research modes offered mainstream modes Research modes offered minority modes Changes in the amount of work Reported smartphone usage in web surveys 45 8 Sources of online sample Proportion of online sample in use - trend Proportion of online sample in use in Online sample in use - trend 49 9 Mixed mode Importance of mixed mode Use of integrated platforms Level of mixed mode support required Analysis and results reporting Distribution methods in use Distribution methods in use, by company size Change in demand for high tech delivery Importance of producing cross tabs in volume Changing software Planning on changing software in Planning on changing software - trend Software type to change Reasons for changing software 63 Confirmit 2012 Annual MR Software Survey: Full report 3 of 63

4 List of tables Table 1 Sample composition: company size by region Table 2 Sample composition: responsibility and seniority Table 3 Descriptive statistics for the estimates of the natural interview length for surveys Table 4 Incentives used frequently by global region and company size Table 5 Incentives used frequently or occasionally by global region and company size Table 6 Quality control measures by global region and company size Table 7 Proportion of quantitative research by mode List of charts Figure 1 Changes reported in survey length Figure 2 Methods adopted to shorten surveys Figure 3 Natural limit for interview length for telephone, online and mobile surveys Figure 4 Distribution of the stated natural limit of different interview modes, in minutes Figure 5 Preference for app or browser when delivering mobile surveys Figure 6 Predictions for which method will dominate in future Figure 7 Extent to which panel participants are incentivised Figure 8 Distribution of the prevalence of incentivised panel as a sample source Figure 9 Incentives used frequently or occasionally Figure 10 Chart title Figure 11 Adoption of independent panel verification services (among panel owners) Figure 12 Communities operated in 2012 by region and company size Figure 13 Number of communities operated in 2009 and 2012 Figure 14 Predicted growth for communities Figure 15 Software used to administer communities Figure 16 Required capabilities in community software tools Figure 17 Satisfaction with community software capabilities: ranked from good to poor Figure 18 Proportion of company revenues that derive from research activities, annual trend38 Figure 19 Proportion of company revenues that derive from research activities, by region and company size 39 Figure 20 Volume of work by mode Figure 21 Percentage of market research firms using the main modes of research Figure 22 Percentage of market research firms using the minor modes of research Figure 23 Anticipated changes in the amount of work over next three years Confirmit 2012 Annual MR Software Survey: Full report 4 of 63

5 Figure 24 Percentage of web survey participants taking surveys on smartphones and other small format mobile devices: 2011 and Figure 25 Proportion of online sample in use Figure 26 Proportion of online sample in use, by region and company size Figure 27 Online sample sources in use Figure 28 Importance of mixed mode Figure 29 Use of integrated platform versus switching between different platforms Figure 30 Level of mixed mode Figure 31 Distribution methods in use Figure 32 Distribution methods in use, by company size Figure 33 Anticipated change in demand for high tech delivery methods Figure 34 Importance of being able to produce volumes of printed cross tabs Figure 35 Planning on changing software, by region and company size Figure 36 Planning on changing software, by year Figure 37 Software type wishing to change Figure 38 Reasons for wishing to change software Confirmit 2012 Annual MR Software Survey: Full report 5 of 63

6 Acknowledgements We are greatly indebted to Confirmit for the assistance they have provided in supporting the survey financially, and also technically, for providing us with the use of their software for this survey, and for their significant contribution they have made to the content and design of the survey. We are also most grateful for the support provided by CASRO ( JMRA ( and Quirk s Publishing ( in publicising this survey among their own members and subscribers, and inviting participation. This has been especially helpful in increasing the coverage of this survey among groups that had proved harder to reach in the past. We also thank Ascribe ( for their generosity in coding the open-ended questions in this survey, and Lloyd Irving of Magnify Solutions for programming the survey for us. Tim Macer and Sheila Wilson, meaning ltd London, 2013 Confirmit 2012 Annual MR Software Survey: Full report 6 of 63

7 1 Background to the survey Meaning ltd has been carrying out this annual software survey since 2004, making this the ninth successive year of the survey. The survey provides a unique set of information and insights into the interplay of technology and methodology within the market research industry. It provides a snapshot of current usage and attitudes and predictions from practitioners, and identifies trends from a number of tracking questions that are asked repeatedly each year. In addition to identifying general trends, each year, the survey also focuses on several areas of topical interest. For the 2012 survey, these topics are: Survey length (and the impact of mobile surveys on this) Attitudes to apps versus browsers for mobile surveys Survey quality initiatives and the use of incentives in panels Research communities: uptake and software requirements This year (as in several previous years), the survey software company Confirmit has kindly sponsored the survey and provided financial and technical assistance that support its administration. This is acknowledged in the survey s full title, which is the 2012 Confirmit Market Research Software Survey by meaning. 1.1 The Sample The 2012 survey comprises a sample of 250 market research companies globally, and only of individuals who are responsible for, influential in or aware of technology decisions within each company. The sample is managed to ensure representation of three global regions: North America, Europe and Asia Pacific according to the relative amount of research carried in these regions from data published by ESOMAR. The survey consists of a self-completion interview on the Web, comprising around sixty questions and timed to last approximately fifteen minutes. As it is difficult to identify the target individual within a research company, sample is obtained from a variety of sources: Participants who agreed to be re-contacted from the previous year s survey Sample compiled by meaning ltd Sample provided by the survey s sponsor, Confirmit Sample from 2010 and 2011 surveys ed invitations by CASRO (Council of American Survey Research Organizations) and by JMRA (Japan Market Research Association) to an open survey link. Several links inviting participation from mailshots and on the homepage of the research publication Quirk s Marketing Research Review to an open survey link. Confirmit 2012 Annual MR Software Survey: Full report 7 of 63

8 All ed invitations are de-duplicated across all sources prior to invitations being sent, and de-duplication is again applied to completed interviews for the small number of cases where more than one response has been received from the same company within that country. Responses from open survey links accounted for 6% of the sample. These were additionally asked to provide company information, and this was used to verify their eligibility for the survey i.e. that they were a research company. The nature of the sampling method, and its eligibility criteria (i.e. senior IT decision makers within research organisations) makes it impossible to provide a meaningful estimate of response rate, because invitations are sent to individuals without prior knowledge of their eligibility, and those who are not eligible are unlikely to respond to the survey invitation. 1.2 Demographics This report is based on 250 participants in unique research companies within each country and covers 37 countries. The composition of the demographics referred to in this report is as follows: Size Total N America Europe Asia Pacific Small % 55 56% 72 64% 18 45% Medium 63 25% 26 27% 23 21% 14 35% Large 42 17% 17 17% 17 15% 8 20% Total % % % % Table 1 Sample composition: company size by region Company size is defined by the declared revenue of the organisation. A company is considered small if its revenue is below $5 million or fewer than 50 employees, and large if it exceeds $25 million or 500 employees. Small is intended to reveal the different needs of companies unlikely to have specialist in-house technical staff. The smaller proportion of companies in the large (or over $25m turnover) category is only a reflection of the pyramid that exists of company size, with a smaller number of large companies globally. The survey is only asked of senior individuals who are involved in IT decision-making within their respective company. The level of responsibility and role of the participants is shown below: Role N % Level of responsibility N % Research 88 35% Board Member 67 27% IT 25 10% Primary IT decision maker % Data processing 32 13% IT Decision influencer 77 31% Business or Operational % Table 2 Sample composition: responsibility and seniority Confirmit 2012 Annual MR Software Survey: Full report 8 of 63

9 2 Part 1: New questions Each year, the survey also focuses on several areas of topical interest. For the 2012 survey, four related topics have been explored. Topics in this section Survey length Mobile survey apps versus browsers Panel rewards and survey quality measures Communities Confirmit 2012 Annual MR Software Survey: Full report 9 of 63

10 3 Survey length Questions 1. Are surveys getting any shorter? 2. What, if anything, is being done to shorten surveys? 3. What is the natural length of a survey on the phone, web or mobile? Key findings Only a minority report that surveys are getting any shorter Despite the fact that survey length is well understood to be a significant factor in poor response and poor engagement by respondents, it is only a third of companies that are reporting any overall reduction in survey length For the majority, it s business as usual, including some who say length continues to increase. but many companies are at least taking action Half of the companies surveyed have a company policy that seeks to limit survey length. A minority of firms are using other measures such as adaptive surveys and semistructured questions to replace large grids or batteries of questions Seven or eight minutes is the limit for a mobile survey There was broad agreement across the whole sample on the natural limit for different types of survey 20 minutes for telephone and online, but just 8.3 minutes for mobile. A small minority consider surveys of one or two hours duration are OK. The median, which is less affected by these outliers, gives optimum interview lengths of 18 minutes for telephone surveys, 15 minutes for web and 7 minutes for mobile. But you may get a bit longer in Asia Pacific Companies in Asia Pacific cited slightly longer, times for all modes, especially mobile. Confirmit 2012 Annual MR Software Survey: Full report 10 of 63

11 3.1 Survey length: the shift to shorter interviews à There is a weak trend towards shorter surveys à A third say they are getting shorter while 1 in 8 say they are still getting longer à 50% see no change à Small firms and Asia Pacific lead the way on shorter surveys The question was introduced with this preamble: Survey methodologists have identified excessive survey length as an important factor in diminished participation in surveys by the public. The situation that emerges is very mixed the largest group see no change; beyond that, more than twice as many report a decrease in length as report an increase. Q: To what extent have you observed any decrease in the length of the surveys that your company administers? N = 250 Large! 7%! 31%! 48%! 12%! 2%! Medium! 5%! 24%! 54%! 16%! 2%! Small! 12%! 26%! 48%! 12%! 2%! Asia Pacific! 20%! 20%! 50%! 10%! 0%! Europe! 7%! 25%! 53%! 13%! 2%! North America! 8%! 30%! 46%! 13%! 3%! Total! 10%! 26%! 50%! 13%! 2%! Most of our surveys are now shorter! No overall change! Not sure! Figure 1 Changes reported in survey length Some are now shorter! Surveys are getting longer! Confirmit 2012 Annual MR Software Survey: Full report 11 of 63

12 3.2 Methods adopted to shorten surveys à 50% of firms have a policy or best practice guide to limit survey length à The adaptive survey is cited as a popular method to reduce survey length à Breaking projects into smaller surveys or using semistructured questions are less popular methods Our interpretation of these data is that firms are not doing as much as they could to address the issue of survey length. Professional codes of ethics or conduct all state the importance of collecting no more data than is necessary and many also speak of respondent burden, so we would expect virtually all companies to have policies that reflect this. It is possible that they do, and those participating in this survey are unaware of these. However, the other more technical methods of limiting survey length are uncommon, with the exception of adaptive surveys using intelligent routing to ensure no single participant is faced with an excessive number of questions. Q: Which of these methods, if any, does your company use, to limit survey length? N = 250 By adopting policy or best practice guidance! 50%! Adaptive surveys! 42%! Break up projects into several smaller surveys! 24%! Substitute semi-structured/open questions for question batteries! 21%! Other measures! 13%! Figure 2 Methods adopted to shorten surveys Confirmit 2012 Annual MR Software Survey: Full report 12 of 63

13 3.3 Natural survey length à The mean average suggested is 20 minutes for phone and web; 8.3 minutes for mobile à Slightly longer surveys tolerated in Asia Pacific à Small firms are the most cautious about survey length This question was asked as an open numeric question. Most responses were rounded to multiples of 5 minutes. Overall, there is very little difference in the mean average survey length proposed by participants. Behind this, there is of course, a wider variation. The maximum survey length proposed was a mind-numbing 120 minutes for telephone and web, and an optimistic 34 minutes for mobile. It is our general experience that the actual interview lengths on a great many commercial market research projects for telephone and online, far exceed the optimum lengths stated here. Confirmit 2012 Annual MR Software Survey: Full report 13 of 63

14 Q. What do you consider is the natural limit for these kinds of surveys, in interview length telephone, online and mobile? Asia Pacific! 9.5! 21.3! 21.4! Europe! 8.0! 19.2! 19.4! N America! 8.2! 20.2! 19.8! Total! 8.3! 19.9! 19.9! Large! 9.4! 21.4! 23.9! Med! 8.6! 21.4! 21.2! Small! 7.9! 18.8! 18.1! Telephone survey! Online survey! Mobile survey! N = 250 Figure 3 Natural limit for interview length for telephone, online and mobile surveys Confirmit 2012 Annual MR Software Survey: Full report 14 of 63

15 3.4 Natural survey length: distribution of responses à Median lengths probably give a better estimate: 18 mins for phone, 15 for web and 7 for mobile à 49% think 5 minutes is the natural limit for mobile à Agreement is strong, with the majority of responses for all modes fairly close to the mean The difference in the shape of these three curves are interesting. Phone and online have a much longer tail than mobile because there are several outliers where atypically large lengths were given. This has the effect of making the mean a bit higher than the media, and shows there is a minority that think surveys can be very much longer than the majority do. Q. What do you consider is the natural limit for these kinds of surveys, in interview length telephone, online and mobile? N = %! Telephone! Online! Mobile! 40%! 30%! 20%! 10%! 0%! <5 mins! 10! 15! 20! 25! 30! 35! 40! 45! 50! 55! 60! Don't know! Figure 4 Distribution of the stated natural limit of different interview modes, in minutes Telephone and online can be considered to follow a positively skewed curve (i.e. the centre of the curve is to the right of where it might be expected to be on the chart, if it were a normal distribution). On the other hand, the distribution for mobile is asymmetric it is only one half of a curve, with the peak on the extreme left of the chart. Confirmit 2012 Annual MR Software Survey: Full report 15 of 63

16 The values are more tightly clustered than a normal distribution, with 87% of the responses for telephone lying within one standard deviation (SD) of the mean; 89% for online and 74% for mobile (in a normal distribution 68% of values would be expected to fall within one SD of the mean and 98% within 2 SDs.) This can also be observed in the high Kurtosis scores. These and other descriptive statistics for these distributions can be seen in Table 3, below. Telephone Online Mobile Mean Median Std dev Skew Kurtosis Std err Table 3 Descriptive statistics for the estimates of the natural interview length for surveys Confirmit 2012 Annual MR Software Survey: Full report 16 of 63

17 4 The future for mobile apps Questions 1. Which is better for mobile surveys today: app or browser? 2. Is this likely to change in the future? Key findings Mobile browser surveys have the edge for the moment Participants in this survey are fairly divided on whether the app or browser is the most viable way to administer mobile surveys, with most considering either equally valid. Among those favouring one or the other, it is the browser that has the edge today. Apps are likely to catch up, or even surpass the browser in the future The gap between apps and browsers closes when respondents were asked for their future prognosis on which will dominate. Among some sections, apps are predicted to edge ahead of browsers Mobile research is now considered viable by the vast majority There was broad agreement, especially in Asia Pacific, that mobile research is now a viable method only one in ten consider that mobile as a whole is not viable. Confirmit 2012 Annual MR Software Survey: Full report 17 of 63

18 4.1 Preferences for apps or browsers for mobile surveys à The browser is slightly more popular than the app à Apps hold more appeal in Asia Pacific and among large firms, but even here they don t eclipse the browser à Apps hold the least appeal in North America à One in ten still think that mobile research isn t viable yet but far fewer in Asia Pacific Bets are fairly well hedged on whether using an app or the browser will get better results, with one third of the sample thinking both have equal merit a view we share. But among those favouring one or the other, it is the browser that has more credibility at present. Q: Which method do you consider is most viable at this time for administering mobile surveys - mobile app or mobile browser? N = 250 Large! 24%! 38%! 29%! 10%! Medium! 13%! 40%! 43%! 5%! Small! 19%! 39%! 32%! 10%! Asia Pac! 25%! 35%! 38%! 3%! Europe! 21%! 37%! 33%! 9%! N America! 11%! 43%! 35%! 11%! Total! 18%! 39%! 34%! 9%! App! Browser! Both equally viable! Neither viable at this time! Figure 5 Preference for app or browser when delivering mobile surveys Confirmit 2012 Annual MR Software Survey: Full report 18 of 63

19 4.2 Predicting the future for apps and browsers à A reversal is predicted in several sectors: Europe, Asia Pacific and small firms are backing apps to dominate à North America, along with mid-size firms think the browser will be the more dominant à Overall, and specifically in large companies, the gap is predicted to close Mobile apps, and mobile generally, have not penetrated into much of research yet, as can be seen from our tracking questions on the methods being used. However, the mobile app is expected to have an important part to play in the future and not necessarily at the expense of browser-delivered mobile surveys. The most likely outcome is that there will be a place for both technologies along side each other even though the data do not explicitly show this, because the neither will dominate group is just 10% overall, and is fairly consistent across all groups but a large group, by stating not sure, seem to be saying it is too close to call at present. Q. Which method do you think will dominate in the future, as the technology matures? N = 250 Large! 43%! 45%! 2%! 10%! Medium! 25%! 46%! 19%! 10%! Small! 36%! 31%! 8%! 25%! Pac Rim! 50%! 33%! 8%! 10%! Europe! 38%! 35%! 11%! 16%! North America! 24%! 42%! 10%! 25%! Total! 34%! 37%! 10%! 18%! App! Browser! Neither will dominate! Not sure! Figure 6 Predictions for which method will dominate in future Confirmit 2012 Annual MR Software Survey: Full report 19 of 63

20 5 Panel rewards and quality control Questions 1. To what extent do surveys across the industry come from incentivised panels? 2. What incentive methods are used, and with what frequency? 3. What quality control methods are applied? 4. How much do panel owners use independent panel verification services to control for professional respondents with multiple panel memberships? Key findings Penny for your thoughts most online surveys today are incentivised Nearly six out of 10 (57%) of research companies are using incentivised panels for between two-thirds and 100% of their samples. Only a few (7%) are not using rewards at all. Independent panel verification is the exception not the norm Around three-quarters (76%) of panel operators do not subscribe to independent panel verification services. Even among large companies 58% do not do this. Most MR companies run simple fraud prevention checks on online responses Most companies are checking for speeding by respondents (73%) and nearly twothirds (63%) look for straightlining : two quality control methods that many data collection tools make easy to apply. More thorough respondent fraud checks are largely shunned Just over a half surveyed (52%) use challenge questions, and fewer still some of the more high tech methods. Consumerist rewards are the norm Incentives to participate in surveys appeal to a consumerist appetite, with vouchers being the most frequently used reward (75%). Donation to good causes, gifts, cash payments, entries to sweepstakes or points that can be exchanged for prizes are all rewards that research companies very commonly use. The communitarian approach of rewarding participation by enhanced status, or visibly though special recognition and kudos awards is currently used by only 16% of companies. Confirmit 2012 Annual MR Software Survey: Full report 20 of 63

21 5.1 Prevalence of incentivised panel surveys à Around two-thirds of all survey responses now come from incentivised panels à There is little difference in behaviour by company size à N America makes very slightly more use of incentives than other regions The differences observed between company tiers and global regions are very slight so a highly consistent picture emerges that virtually two out of every three surveys fielded today are drawn from an incentivised panel. Q: How much of your quantitative survey work uses panels or other sample sources where participants receive a reward? [0-100%] N = 250 Large! Medium! Small! Asia Pacific! Europe! N America! Total! 64%! 61%! 64%! 63%! 61%! 66%! 63%! Figure 7 Extent to which panel participants are incentivised Confirmit 2012 Annual MR Software Survey: Full report 21 of 63

22 5.2 Distribution of incentivised panel use à Practices vary widely between different firms à The largest sector uses incentivised panels for most of its work (and many of these for all work) à Another large group use panels sparingly or not at all à North America is the most dependent on panels à Asia Pacific is the most diverse in its practices Q: How much of your quantitative survey work uses panels or other sample sources where participants receive a reward? [0-100%] N = 250 Large! 5%! 17 %! 24 %! 26 %! 29 %! Medium! 5%! 24 %! 18 %! 37 %! 18 %! Small! 8%! 19 %! 14 %! 31 %! 28 %! Asia Pacific! 8%! 25 %! 8 %! 23 %! 38 %! Europe! 8%! 20 %! 21 %! 28 %! 24 %! N America! 5%! 17 %! 15 %! 40 %! 22 %! Total! 7%! 20 %! 16 %! 32 %! 26 %! 0%! 33%! 67%! 100%! None! Up to 1/3! Between 1/3 and 2/3! Over 2/3! All! Figure 8 Distribution of the prevalence of incentivised panel as a sample source In the graph above, the percentages given have been grouped into terciles, with the exception of those stating 0% or 100%, which are grouped separately. If panel usage were randomly (and therefore evenly) distributed, the green, orange and blue sectors would align with the 33% and 67% vertical grid lines. Instead, this shows that behaviour tends to be clustered at either end of the spectrum, with the middle tercile relatively sparsely populated. Confirmit 2012 Annual MR Software Survey: Full report 22 of 63

23 5.3 Incentives used à Vouchers dominate but points and cash are used the most frequently à Cash is more common with large firms and in N America à Donations and lotteries widely used but not as a major incentive method à Awarding kudos points is still a rarity Research companies appear to be using the full spectrum of incentive methods available, with the sole exception of awarding enhanced status or privileges which is a method that is popular on social media sites and which, though especially relevant to communities, is also applicable to panels. Q. Which of these incentives do you use, and to what extent? N = 250 Vouchers! 27 %! 48 %! Donation to a good cause! 16 %! 53 %! Non-monetary gifts! 12 %! 51 %! Cash! 28 %! 32 %! Sweepstakes/lotteries! 21 %! 38 %! Points for redemption! 32 %! 22 %! Enhanced member status or privileges! 2 %! 14 %! Frequently! Occasionally! Figure 9 Incentives used frequently or occasionally There are interesting differences between the methods that are widely used (regardless of how frequently that use is) and the underlying core group for each that use them frequently. Vouchers are widely used, but among those users, just over a quarter them frequently; even more so with donations, which 69% of firms use, but less than a fifth use them frequently. The reverse applies to redeemable points, which overall 54% use, but a majority of these use them frequently. Confirmit 2012 Annual MR Software Survey: Full report 23 of 63

24 N Asia Total America Europe Pacific Small Med Large Vouchers 27% 29% 25% 28% 24% 32% 29% Donation to a good cause 16% 13% 18% 15% 13% 21% 17% Non-monetary gifts 12% 9% 12% 23% 9% 13% 24% Cash 28% 36% 22% 28% 30% 21% 33% Sweepstakes/lotteries 21% 25% 19% 18% 21% 18% 24% Points for redemption 32% 28% 35% 38% 29% 32% 45% Enhanced status or privileges 2% 0% 3% 5% 1% 2% 7% Table 4 Incentives used frequently by global region and company size N Asia N=250 Total America Europe Pacific Small Med Large Vouchers 75% 81% 69% 80% 72% 81% 76% Donation to a good cause 69% 75% 66% 63% 63% 78% 76% Non-monetary gifts 64% 59% 64% 73% 58% 70% 74% Cash 60% 75% 49% 58% 56% 60% 76% Sweepstakes/lotteries 59% 69% 53% 50% 57% 60% 62% Points for redemption 54% 45% 62% 55% 52% 54% 62% Enhanced status or privileges 16% 14% 13% 28% 10% 21% 29% Table 5 Incentives used frequently or occasionally by global region and company size Some methods also appear to be more popular with different company sizes and global regions. Perhaps living up to a cultural stereotype, North America is into cash as a reward method, both as a frequent and occasional method, much more than other parts of the world. It is cited less frequently in Europe, where points are more popular (points are less popular in North America). Companies in North America also seem to use slightly more incentive methods then the other regions. On average, they mentioned 4.2 methods, against 3.8 in Europe and 4.1 in Asia Pacific. Large companies on average mentioned 4.6 methods, against 3.7 mentioned by smaller firms. Large firms also seem to be ahead in offering enhanced status or other kudos rewards which may relate to their greater engagement with research communities, where this model is particularly relevant. Confirmit 2012 Annual MR Software Survey: Full report 24 of 63

25 5.4 Survey quality measures applied à Simple, immediate measures like speed checks and straight-lining detection are the most popular à More effective but more costly measures like digital fingerprinting or ID verification are less common à Most firms seem to be using a combination of methods Research professional bodies such as ESOMAR and CASRO have recently highlighted many quality issues arising from panels and the effect of incentives on survey quality leading to interview fraud on a grand scale. Having established the degree of incentive usage, we also wished to examine the extent to which research companies were using technology to detect fraud using the kinds of quality control measures that technology can facilitate relatively easily. Q. Which of these quality control methods do you apply to prevent inappropriate or fraudulent participation from participants taking incentivized surveys in your panel research? N = 250 Speed detection! 73 %! Straight-lining detection! 63 %! Challenge questions! IP address location checking! Digital fingerprinting to identify multiple registrations! Analytics on aggregated responses! ID verification at the time of registration! 29 %! 52 %! 49 %! 46 %! 43 %! No method used! 9 %! Figure 10 Chart title Confirmit 2012 Annual MR Software Survey: Full report 25 of 63

26 5.5 Survey quality measures at a detailed level à Larger companies apply more QC measures than smaller ones à Europe behind N America and Asia Pac on stronger fingerprinting, IP checking and ID verify measures There are several differences in detail, at a company size and global regional level, in the both the actual quality control (QC) measures applied and the number of measures applied. We calculated the number of measures by counting the number of mentions, after excluding no method used. Large firms are taking the lead with the number of measures they apply (on average more than four were cited), and also the extent to which they use IP address checking, digital fingerprinting and ID verification, all of which are technically advanced methods. These may require additional software to achieve, and usually involve some additional effort and knowhow to achieve. There also appears to be slightly more attention given to quality control measures in North America (average of four methods cited) against Europe (3.2) and Asia Pacific (3.4). Europe lies behind both North America and Asia Pacific on the three more high-tech and therefore stronger fraud checking measures of IP address location checking, digital fingerprinting (which identifies multiple registrations) and ID verification. The 5% of large and medium-sized companies that report no checks possibly indicates that there is some under-reporting in this question, as it is unlikely that the organisations involved in fact apply no such checks it serves to remind us that a limitation of this survey of companies is that it depends on the extent of knowledge of the individual completing it about activities which other teams may be performing. North Eur- Asia N = 250 Total Am. ope Pac Small Med Large Speed detection 73% 76% 71% 70% 67% 83% 79% Straight-lining detection 63% 77% 57% 48% 61% 65% 67% Challenge questions 52% 58% 48% 45% 46% 60% 57% IP address location checking 49% 55% 40% 58% 43% 51% 64% Digital fingerprinting to identify multiple 29% 35% 22% 33% 24% 32% 41% registrations Analytics on aggregated responses 46% 49% 45% 43% 43% 43% 60% ID verification at the time of registration 43% 49% 38% 43% 38% 51% 50% No method used 9% 6% 11% 10% 12% 5% 5% Average number of measures (excl. none) Table 6 Quality control measures by global region and company size Confirmit 2012 Annual MR Software Survey: Full report 26 of 63

27 5.6 Use of independent panel verification services à Overall, only a quarter of research companies with their own panels use independent verification services à Large firms are more than twice as likely to use them than small firms à Only one in five firms in Europe use IPV services While the quality control measures discussed previously can help to detect fraudulent behaviour in the moment (either at the moment of the survey, or at the time of joining the panel), they do not help to combat the professional respondent problem, identified in studies by ESOMAR and CASRO, amongst others. For this, individuals need to be checked for membership of other panels using measures such as digital fingerprinting, to recognise multiple registrants. Use of these services could eliminate this problem to a large extent, although there is an additional cost associated, and some additional effort involved. This survey identifies the same trend observed with other fraud reduction measures that the more powerful measures, which are also more costly to apply, are used less frequently. In this case, they are used much less frequently. Q. Do you subscribe to any independent panel verification services? N = 110 Base: those who operate their own panels Large! 42%! Medium! 21%! Small! 17%! Asia Pac! 28%! Europe! 19%! North America! 29%! Total! 24%! Figure 11 Adoption of independent panel verification services (among panel owners) Confirmit 2012 Annual MR Software Survey: Full report 27 of 63

28 6 Communities Questions How many communities are research firms operating today (versus 3 years ago)? Is more growth in research community work anticipated? What kind of software do community operators use to run them? What particular software features are sought in a community platform? How good is existing software support for these features? Key findings Communities are still rare As in 2009, only one in six companies (17%) are operating a community. Even among large companies, only 31% run communities. Companies that run communities don t run many of them Of the 41 companies in our sample who are running communities, over half run 4 or fewer communities: figures broadly similar to those we obtained in Still optimistic about growth, despite lack of growth Those running communities, and especially large companies, mostly predict that they will increase their number of communities over the next year. Discussion boards and polls are most important requirements for community software 79% said they need discussion forums and 75% need polls within their community platform. Over 60% also said they required diaries, blogs and real-time groups, making the requirements for community platforms quite diverse. Existing software supports most important features The simpler and more established capabilities like polls, diaries and forums seem well supported, but poor support was identified for co-creation tools, text analytics and mobile community apps Four out of ten community operators wants to have a mobile app for their community members to use There is a place for specialist community software Around a quarter of firms use the same software for panel and community administration, but the majority either use specialist panel tools, or blend these with their panel platform. Confirmit 2012 Annual MR Software Survey: Full report 28 of 63

29 6.1 Who runs communities? à Communities are a specialised activity among a small minority of firms à Communities much more likely to be found in N America and among large companies à Uptake is lower in Asia Pacific and among small firms The number of firms operating communities is probably fewer than is expected, and remains very much the preserve of specialist operators rather than a mainstream activity offered by a majority of research firms. In fact, the substantial majority do not offer communities, nor do they show any intention of doing so. Q: Does your company operate any online research communities? N=250 Large! 31%! 17%! 52%! Medium! 24%! 11%! 65%! Small! 9%! 21%! 70%! Asia Pac! 8%! 23%! 70%! Europe! 15%! 22%! 63%! North America! 21%! 10%! 68%! Total! 16%! 18%! 66%! Operate at least one community! Developing or plan to introduce! No current plans! Figure 12 Communities operated in 2012 by region and company size Confirmit 2012 Annual MR Software Survey: Full report 29 of 63

30 6.2 Change in uptake since 2009 à No overall growth detected in communities operated à Fewer companies now planning to introduce communities than three years ago This same question was asked in 2009 and in 2012, which allows direct comparison of the responses. Although the sample size was smaller in 2009, the sampling method and sample composition is broadly similar. The proportion of firms that offer communities is virtually the same then and now. Those with no plans to introduce communities have hardened since that time, from just over half of firms to two-thirds now. Q: Does your company operate any online research communities? N=188 (2009) and 250 (2012) 2009! 17%! 27%! 56%! 2012! 16%! 18%! 66%! Operate at least one community! Developing or plan to introduce! No current plans! Confirmit 2012 Annual MR Software Survey: Full report 30 of 63

31 6.3 Number of communities operated à The majority of firms run 4 communities or fewer à Since 2009, slow growth in the number of communities firms are running à A very small number of niche players have achieved scale with 20+ communities Q: How many communities do you currently run? 1-4! 56%! 63%! 5-9! 13%! 17%! 10-19! 12%! 16%! 20-49! 9%! 10%! 2009 N=32! 50+! 0%! 5%! 2012 N=41! Figure 13 Number of communities operated in 2009 and 2012 Confirmit 2012 Annual MR Software Survey: Full report 31 of 63

32 6.4 Predicted growth for communities à Weak growth anticipated overall à A minority anticipate strong growth à Growth predictions now more cautious than in 2009 This question was also asked in 2009 as well as 2012, which allows direct comparison. As this question was asked only of those operating or planning to operate communities, the sample size is too small to analyse it by region or company size. Compared to 2009, the sentiment in 2012 is more cautious, with fewer predicting strong growth and more considering modest or no growth. Q: Do you expect the number of communities you run to increase over the next 12 months? Base: Those currently or planning to operate communities Increase strongly! 20%! 34%! Increase slightly! 50%! 60%! Remain the same! 7%! 11%! Decrease! Cannot say! 2%! 4%! 6%! 6%! 2009 N=82! 2012 N=85! Figure 14 Predicted growth for communities Confirmit 2012 Annual MR Software Survey: Full report 32 of 63

33 6.5 Software used to administer communities à A quarter of community operators use panel management software to run their communities à Most use specialist community software à Some combine specialist software with panel management software From a technical perspective, communities and panels share many characteristics that make their requirements distinctive from that of conventional survey data collection. These include having a database of members or participants, repeated contacts to the same individual for different purposes, and the accumulation of response history over time. For that reason, some panel software offers community capabilities as an extension to panel management capabilities. This question aimed to discover whether this hybrid approach is popular among community operators, or if they prefer to use tools dedicated to managing communities. It appears that only a minority use the same tools as for panel management exclusively; some supplement their panel management software with specialist community tools, while the majority only use specialist software for communities. Q: Do you use the same software or different software for communities as for managing panels? N = 85 Use a mixture! 18%! Use the same software! 27%! Use different software! 55%! Figure 15 Software used to administer communities Confirmit 2012 Annual MR Software Survey: Full report 33 of 63

34 6.6 Required capabilities in community software tools à Simple panel plus tools are those most sought à Around 60% want diaries, real-time groups and blogs à 45% want a mobile app for community members to use There are many different forms of engagement possible with a community, from the addition of simple discussion forums and polls to a panel (which could be considered panel plus rather than a full community) to more intensive and unconventional research activities. This survey set out seven such methods that are frequently cited. Of these, only co-creation tools were requested by less than half of those responding (by 41%). Communities tend to generate a lot of text, which can be painstaking to analyse. Despite this, only 35% saw this as a requirement. However, almost half (45%) want their community tool to offer a mobile app so that community members can participate from smartphones, tablets and the like. Q: Which of these specific capabilities do you require in your research community software platform? N = 85 Discussion boards or forums! Polls! 75%! 79%! Diaries! Real-time groups! Blogs! 62%! 61%! 61%! Mobile app (for participants)! Co-creation tools! 41%! 45%! Text analytics! 35%! Other capabilities! 8%! Figure 16 Required capabilities in community software tools Confirmit 2012 Annual MR Software Survey: Full report 34 of 63

35 6.7 Satisfaction with community software capabilities à Blogs, discussion boards and diaries appear to be wellsupported à Support for text analytics and co-creation tools considered least satisfactory à Mobile app support considered poor by nearly a fifth As these questions relate to features required by users and not all companies require these features the base sizes are inevitably quite small for these responses, and are clearly shown in the axis of the chart. A mixed picture emerges with some features well supported, and others less so. Blogs, discussion boards (or forums) and diaries all fare reasonably well almost all rated the support good or fair. The more technically demanding requirements of real-time focus groups and mobile apps have a significant minority finding them poor, however. As do the more novel and unconventional options for text analysis and the more nebulous co-creation tools. These also give greater cause for dissatisfaction, with only 30% and 23% respectively rating them good. Q: How would you describe the support for these capabilities in the software tools that you currently use? Blogs (N=52)! 6%! 42%! 52%! Discussion boards (N=67)! 8%! 40%! 52%! Diaries (N=53)! 9%! 43%! 47%! Mobile app (N=38)! 18%! 42%! 40%! Real-time groups (N=52)! 15%! 39%! 46%! Text analytics (N=30)! 17%! 30%! 53%! Co-creation tools (N=35)! 23%! 26%! 51%! Good! Fair! Poor! Figure 17 Satisfaction with community software capabilities: ranked from good to poor Confirmit 2012 Annual MR Software Survey: Full report 35 of 63

36 Part 2: Trends There is a central core of questions in this survey that we have been asking for several years and some since the inception of this project in With several years of data, several clear and stable trends have emerged. This shows continued consistency in the survey as a whole, since the sample this year is made up largely of different companies and individuals than in the previous year. Topics covered in this section Research activities Sources of online sample Mixed-mode Analysis and results reporting Changing software Confirmit 2012 Annual MR Software Survey: Full report 36 of 63

37 7 Research activities Questions Volume of work that derives from quantitative, qualitative or other research activities? Volume of work by research mode (CATI, CAWI etc)? Anticipated changes in the amount of work? Key findings Mobile research growing rapidly The number of companies offering mobile research (self completion, but not including SMS) is increasing rapidly. When we first measured this in 2009, 6% of companies offered this as a service. In 2012 it is 18%. Most other modes are fairly stable, except for paper and CAPI Paper appears to be declining and CAPI increasing. Volumes of mobile research may be increasing too in previous years, the volume was 1% or less, in 2012 it is 2%. Tablets may be fuelling CAPI growth The number of companies offering mobile CAPI appears to be stable, but there is growth in the number of companies offering laptop/tablet CAPI in 2011 and Significant minority of Web surveys completed on smartphones Companies now report that 13% of those taking part in their online surveys are using their smartphones. Volumes of paper and CATI research dwindling, Web strong Web research represents just over half (51%) of all quantitative research and this proportion may now be stabilising. Research companies expecting Web and mobile research to be main growth areas The participants in this survey say that mobile and Web research are the area they expect to grow the most (the latter seemingly contradicting the finding above) Confirmit 2012 Annual MR Software Survey: Full report 37 of 63

38 7.1 Research mix 6-year trend à Almost a quarter of revenues derive from qualitative à Qualitative vs. quantitative split is very stable over the years à Research firms do very little non-research work Over the years, the proportion of quantitative research revenues is to qualitative is remarkably stable. Quantitative represents around 70% and qualitative between 20% and 24%. There is a discontinuity in the chart, at 2009, because non-research activities was added as a new answer category. Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities. 80%! 70%! 60%! 50%! 40%! 30%! 20%! 10%! Quantitative research! Qualitative research! Other! Other research! Nonresearch activities! 0%! 2007! 2008! 2009! 2010! 2011! 2012! N = 250 Figure 18 Proportion of company revenues that derive from research activities, annual trend Confirmit 2012 Annual MR Software Survey: Full report 38 of 63

39 7.2 Company revenues from research in 2012 à Smaller companies do more qualitative work, compared to larger companies à The mix of work varies little between global regions These results are much the same for all regions. However large companies do more quantitative and less qualitative work than small and medium-sized companies. Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities. 3%! 4%! 2%! 4%! 4%! 5%! 6%! 4%! 4%! 2%! 4%! 4%! 5%! 5%! 23%! 24%! 22%! 25%! 26%! 24%! 14%! 69%! 69%! 71%! 66%! 66%! 68%! 79%! Total! Americas! Europe! Asia Pacific! Small! Medium! Large! Quantitative research! Qualitative research! Other research! Non-research activities! N = 250 Figure 19 Proportion of company revenues that derive from research activities, by region and company size Confirmit 2012 Annual MR Software Survey: Full report 39 of 63

40 7.3 Volume of work à Web + CATI + paper = 92% of work à Web was growing until 2010 but may now have reached a plateau à Steep decline in paper, gradual decline in CATI The volume of Web research has risen by 11 percentage points between 2006 and In the same time, paper research has declined by exactly the same amount. The volume of CATI work has also declined a little over the years, from 27% in 2006 to 21% in Despite the large increase in the number of companies offering laptop/tablet CAPI, it remains at 6% of total quantitative research volume in In 2006, it was 5%. For details on the more minor modes, see the table overleaf. Q: Focusing on your quantitative research activities, please indicate the approximate proportion of your work represented by each of these interviewing modes or combinations. N = %! 50%! 40%! 30%! 20%! Web! CATI! Paper! laptop/tablet CAPI! Mixed mode! mcapi! 10%! 0%! 2006! 2007! 2008! 2009! 2010! 2011! 2012! Figure 20 Volume of work by mode Confirmit 2012 Annual MR Software Survey: Full report 40 of 63