New survey recruiting strategies: Online Panel vs. Mobile Advertising General Online Research Conference 2015 GOR15 18-20 March 2015, Cologne University of Applied Sciences, Germany Holger Lütters Hochschule für Technik und Wirtschaft Berlin Suggested citation: Lütters, Holger (2015): New survey recruiting strategies: Online Panel vs. Mobile Advertising. General Online Research (GOR) Conference, Cologne. Data Report available under http://goo.gl/k6xh4t Contact: holger.luetters@htw-berlin.de @luetters This work is licensed under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/)
Empirical study in comparison of two market research recruiting approaches: Online Panel vs. Mobile Ad 1 Study set-up and field report
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 3 Berlin based market researchers teamed up for the research project Berlin Market Research Initiative Field report Questionnaire development Online Panel Mobile Ad Questionnaire Technology by 3029 contacts 1732 complete answers 8 minutes questionnaire with questfox using Gyroscale Random distribution of demographics to determine differences in sampling Quota on devices Survey in July 2014
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 4 Two recruiting approaches competing with one identical questionnaire GapFish (Classic) Online Panel www.entscheiderclub.de 110.000 participants Germany, Switzerland, Austria Advertising sampling approach Global reach (200 million potential users) All languages possible (need of developed mobile advertising market structures) Invitation on third party ad platform Generic invitation Real-time screening Re-Direct to third party survey (Non-)Monetary Incentive
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 5 Methodological question: What do participants in studies of (mobile) market research surveys expect from research? Methodological issues: Effects of different sources on the data? Effects on non-reactive data (response time, willingness to participate, etc.)? Effects of different methods used on different devices (AHP, PSM, GyroScale) General interest in Expectations of users towards mobile surveys Decision-making factors of survey participation Expectations of compensation for participation in (mobile) surveys
Online Panel vs. Mobile Ad Recruiting 2 Sociodemographics, Devices and Non-Reactive Data
Target per approach n=800 spread across four device categories with a quota of 200 each Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 7
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 8 The two approaches reached a similar, structurally representative regional population pattern without quota setting Baden-Württemberg Bayern Berlin Brandenburg Bremen Hamburg Hessen Mecklenburg-Vorpommern Niedersachsen Nordrhein-Westfalen Rheinland-Pfalz Saarland Sachsen Sachsen-Anhalt Schleswig-Holstein Thüringen 10,7% 9,9% 13,1% 12,5% 16,2% 15,5% 7,8% 7,2% 4,2% 4,7% 2,5% 3,0% 0,5% 1,0% 0,8% 2,4% 2,6% 2,2% 7,2% 5,7% 7,5% 2,0% 2,7% 2,0% 5,2% 9,7% 9,7% 3,6% 4,7% 5,0% 1,6% 1,2% 1,2% 6,2% 4,9% 5,0% 3,5% 2,1% 2,8% 5,4% 3,5% 3,5% 3,5% 2,8% 2,7% Online Panel Mobile Ad BRD 22,9% 21,9% 21,8% Same questiontype for all device categories; Launch from Berlin Online Panel GapFish (n=888) Mobile Ad Dalia Research (n=811); Data BRD DESTATIS 2012
Mobile Ad approach is significantly less representative for sociodemographics, reaching younger and more male participants Age Distribution Gender Female Male Online Panel Female Male Mobile Ad Average Age Online Panel Mobile Ad Female Male Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); no quota settings Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 9
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 10 Both approaches are not representative in terms of education. Mobile Ad approach achieves higher educational levels Higher Education A-Level vocational school comprehensive school secondary school Without formal degree No answer Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); no quota in question
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 11 Couch surfing as a standard pattern in mobile research 4 out of 5 participants in the study are mobile but at home At Home Work / School / University Public Place On the road None of these places
Technographics: Mobile Ad participants have a slightly more modern equipment, use a broader variety of browsers but do not seem to be more tech savvy in general Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 12
Mobile participants tend to speed through a questionnaire Panel participants take much more time on a mobile device Online Panel Speeder Mobile Ad Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); Information in seconds for complete interviews Response time in seconds Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 13
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 14 The two approaches show different behavioral patterns Criteria Online Panel Mobile Ad Contacts 1912 1032 Complete Interviews 888 811 Average Completion Rate 79,5% 82,9% Quota reach Complete Incomplete Duration 7 days 13 days Response Time Completes 497 seconds 441 seconds Response Time including drop-outs 296 seconds 383 seconds Overall time used in seconds 565.952 395.256 Drop-out pattern Decision in the first minute (independent from device used) Slightly higher drop-out during the entire interview (independent from device used) Very good and good consistency AHP (CRH< 0,2) 29,5% 29,2% Inacceptable consistency AHP (CRH> 0,5) 24,0% 29,0%
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 15 Conclusion non-reactive data: Strong differences in response behavior: Mobile participants respond faster (even on the desktop) Online Panel Short overall project time Intensive use of panelists Slightly more drop-outs Exact quota reach Longer interview duration Smartphone interview much longer Mobile Ad Double overall project time Used less participants Slightly less drop-outs Quota not reached for classic devices Speeding behavior: All interviews are answered quicker on all kinds of devices
Expectations of participants when taking surveys 3
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 17 The online panelists have much more experience with surveys of market research compared to the mobile participants Mean: Online Panel 6,63 Mobile Ad 3,72 Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research(n=811) und n=31 professionelle Marktforscher Mittelwerte einer Skala von 0 (überhaupt keine Erfahrung bis 10 extrem viel Erfahrung)
¾ of the respondents prefer the (classic) online survey Participants of the mobile recruitment are less enthusiastic about online surveys (and could live without any kind of survey) Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 18 Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research (n=811) und n=31 professionelle Marktforscher
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 19 Fair compensation for entertaining but short - interviews are demanded. The Mobile Ad group has lower or no expectations Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research (n=811) und n=31 professionelle Marktforscher Mittelwerte einer Skala von 0 (trifft überhaupt nicht zu bis 10 trifft voll und ganz zu)
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 20 Compensation and duration are the most important factors influencing the willingness to participate in a survey Questions asked in pairwise comparisons with Analytic Hierarchy Process
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 21 The majority of online panelists expect the same compensation for a participation in mobile market research Frage nur an n=879 Teilnehmer Online Panel GapFish und n=30 Teilnehmer MaFo Profis Frage nach Kompensation/Kosten im Vergleich zu klassischer Online-Befragung
The mobile participants expect a form of compensation derived from the context of their recruitment into the study Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 22
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 23 Price Sensitivity Meter to measure a fair compensation for participating in a mobile study Imagine the following invitation to a survey: Topic: Shopping; Duration: 4 minutes; participation only to mobile device (smartphone / tablet)
Participants express high hopes for the incentive of a 4 minute mobile interview which is not backed-up by the willingness to pay of Market Research Professionals Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 24 Optimal Price Point (VanWestendorp) Online Panel: 3,56 Mobile Ad: 4,52 MR Professionals: 0,94
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 25 The identical survey is evaluated very differently by the different approaches Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research(n=811) Measurement Net Promoter Score 0=would not recommend at all.. 10 totally would recommend NPS: 0-6 Detractor; 7-8 Neutral; 9-10 Promotor
Mobile participants are shockingly honest about lying... who is less honest? Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research(n=811) und n=31 professionelle Marktforscher; Mittelwerte einer Skala von 0-10 unter Anwendung NPS: 0-6 gelogen; 7-8 neutral; 9-10 total wahrheitsgemäß) Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 26
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 27 Mobile research is happening anyway. We better find ways to manage it appropriately Mobile Ad approach works Dalia Research developed a new technology which offers more options to the researcher The approach can be as appropriate as it can be wrong for a specific topic The reach of fresh and unexperienced target groups is an interesting option Complicated approaches with special methodologies might not work (due to the screen size and due to the user) Things to consider when using the Mobile Ad recruiting approach Mobile takes more time in creation, testing and surveying Might be more expensive (overall) Mobile users have different expectations which are not necessarily the same as before Researchers should not lower their standards in order to please the new target groups
Holger Lütters New survey recruiting strategies #GOR15 Cologne 20 th March 2015 @luetters 28 Thank you for your interest in the study Data live report available under http://goo.gl/k6xh4t Author: Holger Lütters HTW Berlin Thanks to the supporters of the study: www.gapfish.com Contact information Prof. Dr. Holger Lütters Hochschule für Technik und Wirtschaft Berlin Fachbereich 3 Wirtschafts- und Rechtswissenschaften Treskowallee 8 D 10318 Berlin -Germany Raum TA VG 733 www.daliaresearch.com www.pangealabs.com Gyroscale is a development of pangea labs GmbH Switzerland and is available in the SaaS survey tool questfox www.questfox.com Holger.Luetters@htw-berlin.de www.luetters.com Twitter @luetters xing.com/profile/holger_luetters de.linkedin.com/in/luetters/ The project was one of the interactions during the HTW Berlin / HWR Berlin Project InnoRadar. InnoRadar is supported by the Berlin Institute of Applied Research (IFAF) www.ifaf-berlin.de.