ESOMAR 26 Questions to help research buyers

Transcription

1 ESOMAR 26 Questions to help research buyers The use of internet access panels for sampling and questioning is a relatively new research technique. It will take some time before uniform quality standards can be defined and adopted across the industry. In the early stages of a methodological development there will be many different approaches attempted by research suppliers. Research buyers who wish to get a more thorough understanding of the methodology offered by a research provider can use the following set of 25 questions as the basis of a discussion of methodology. Reputable suppliers with established methodologies and active panel management policies will be able to provide answers to these questions. The decision to go ahead or not can then be made on an informed understanding of the approach being offered and its likely quality. COMPANY PROFILE 1. What experience does your company have with providing online samples for market research? Surveytalk Panel1 has been operating since conducting Face to Face, CATI, Medical and Business to Business research. Online operations began in Panel1 Online Research and a proprietary market research only panel have been in operation since SAMPLE SOURCE 2. Please describe and explain the types of source(s) for the online sample that you provide (are these databases, actively managed panels, direct marketing lists, web intercept sampling, river sampling or other)? The sample source is an actively managed panel. Members opt-in to join for the purpose of completing online surveys. Invitations are sent to individual members for each project 3. What do you consider to be the primary advantage of your sample over other sample sources in the marketplace? Our members are kept actively engaged with the panel through a variety of poll and incentive related activities. Multiple recruitment sources are used including online and offline methods. Data is manually screened after each survey to remove poor data. The database is regularly cleaned of members who provide non-serious responses to surveys and inactive members. These factors combine to give high response rates and quality data.

2 4. If the sample source is a panel or database, is the panel or database used solely for market research? If not, please explain The panel is used only for the purpose of conducting market research. 5. How do you source groups that may be hard-to-reach on the internet? Some targets, such as young males, are particular difficult to find in representative numbers whether online or offline. To reach the widest possible audience, Panel1 employs a variety of recruitment methods and incentive programs. 6. What are people told when they are recruited? Members are asked to opt-in to participate in online surveys and polls in order to have their opinion heard. In return, members receive points that may be exchanged for cash or prize draw entries. PANEL RECRUITMENT 7. If the sample comes from a panel, what is your annual panel turnover/attrition/retention rate and how is it calculated? The activity levels of members are measured through a combination of: Dates of last participation Consecutive non-participation Non-response to s / address issues (bouncebacks) If a member is determined to be potentially no longer viable, they are given an inactive status and will not be placed in any samples. If after further testing, the account does not respond, it is deleted. The rate of occurrence of inactivity (churn) is 10 15% per year. 8. Please describe the opt-in process. A member will reach the website either through a search engine, via a link from an affiliate site, by invitation or by offline invitation. A new member fills in a registration form on the website where they provide basic contact details, including address. A link to a 3 to 5 minute profile survey is sent to the address provided; thereby confirming the address is correct and is accessible by the person who provided it. The new member may then complete the profile survey when they choose to. 9. Do you have a confirmation of identity procedure? Do you have procedures to detect fraudulent respondents at the time of registration with the panel? If so, please describe. Double-opt in is used to verify addresses. Duplicate addresses are not allowed. An algorithm holds aside members who may be potential duplicates based on similar information such as name or DOB. These potential duplicates are then manually screened to ensure they are unique.

3 10. What profile data is kept on panel members? For how many members is this data collected and how often is this data updated? Basic demographic information and data on the most common topics for online surveys are collected from all members. This includes: age, gender, location, occupation, income, education. In addition, panel members are profiled on a range of topics including: technology, medical, mobile phones, travel and automotive. These additional profile surveys are optional. Data on the additional profile topics ranges from 20% to 50% of the entire panel. Periodic, six monthly updates occur to member profiles. Members are able to update their details at any time. 11. What is the size and / or the capacity of the panel, based on active panel members on a given date? Can you provide an overview of active panellists by type of source? 55,000 active members (April 2010). Source data is collected for each member, it is possible to view reports based on respondent source. PANEL AND SAMPLE MANAGEMENT 12. Please describe your sampling process including your exclusion procedures if applicable. Can samples be deployed as batches/replicates, by time zones, geography, etc? If so, how is this controlled? Samples are selected based on cells of demographic criteria. Exclusions may be made on general participation levels, participation on specific topics or membership of specific samples. These exclusions may be filtered over a time period e.g. filtering out all respondents who have participated in automotive surveys in the past 6 months. deployment may be scheduled in batches at particular times to specific target groups e.g. geographic locations, respondent status and time zone. 13. Explain how people are invited to take part in a survey. What does a typical invitation look like? invitations are used. Both html and text s are employed depending on the context. All s contain brief details of the survey such as topic, length and points earned. Company logos and colour schemes are displayed on graphical s. Information is provided on helpdesk contact, privacy and membership termination. 14. Please describe the nature of your incentive system(s). How does this vary by length of interview, respondent characteristics, or other factors you may consider? Points are earned for joining, profiles and survey participation. Points may be exchanged for cash or prize draw entries. The amount of points earned is directly related to the length of the survey. There may be special consideration given if a survey is considered to be particularly tedious or if the target group is difficult to reach. 15. How often are individual members contacted for online surveys within a given time period? Do you keep data on panelist participation history and are limits placed on the frequency that members are contacted and asked to participate in a survey? Members are contacted between once and twice per month, on average. Participation history is automatically monitored and may be used as a means of excluding members from a sample if activity is excessive.

4 POLICIES AND COMPLIANCE 16. Is there a privacy policy in place? If so, what does it state? Is the panel compliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US? What other research industry standards do you comply with e.g. ICC/ESOMAR International Code on Market and Social Research, CASRO guidelines etc.? The privacy policy is accessible on all communications to members as well as on the website. The privacy policy states compliance with all Australian privacy laws. Panel1 is compliant with all AMSRS and ESOMAR privacy guidelines. 17. What data protection/security measures do you have in place? The database is protected, maintained and backed up by an international software provider (Nebu). Member privacy is highly regarded and sensitive information on individuals is never passed on to a third party, this includes addresses. 18. Do you apply a quality management system? Please describe it. In addition to regular filtering of inactive members (see 7), the data for each project is manually screened. Surveys that are completed faster than is realistic as well as surveys containing inappropriate or non-serious responses will be excluded. These members are then removed from the database. 19. Do you conduct online surveys with children and young people? If so, please describe the process for obtaining permission. For some projects, surveys may be conducted with children. In all cases where children will be participating, permission is sought from the parent. An invitation is first sent to the parent explaining the nature of the project and some basic information such as survey length, after permission is granted, the child may complete the survey. PARTNERSHIPS AND MULTIPLE PANEL MEMBERSHIP 20. Do you supplement your samples with samples from other providers? How do you select these partners? Is it your policy to notify a client in advance when using a third party provider? Do you de-duplicate the sample when using multiple sample providers? If sample from the internal panel is not sufficient, an external sample source is used. The preference is normally for larger panels which may use direct marketing, however if a client prefers, an alternate source may be used. When using an external supplier, sample is always de-duplicated. 21. Do you have a policy regarding multi-panel membership? What efforts do you undertake to ensure that survey results are unbiased given that some individuals belong to multiple panels? Higher levels of representation are attained by using a variety of recruitment methods. Panel1 recruitment includes offline (CATI & F2F address collection, flyers) and online (lists, affiliates, search engine optimisation, search engine advertising) methods. Multipanel membership is the norm rather than the exception. Previous industry studies suggest that if a person begins participating in online surveys, and they enjoy doing so, they will seek out other sources for online surveys. The primary motivators are expressing an opinion and reward for participation. Members are not excluded based on membership of other panels, however it is important to maintain multiple recruitment methods.

5 DATA QUALITY AND VALIDATION 22. What are likely survey start rates, drop-out and participation rates in connection with a provided sample? How are these computed? The following statistics are available for all projects: Start rate, Response Rate (can be complete, screened out or quota full), Completion Rate, Drop Out Rate, Incidence Rate, Screened Out Rate and Quota Full Rate. The most salient rate is Response Rate. Given ample time for reminders and survey completion, Response Rates between 30% and 45% are standard. 23. Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your panelists? Are you able to supply your client with a per job analysis of such individual level data? The following individual level data is collected: membership start/end date, date of last participation, date of last login, number of logins, consecutive non-responses, samples a member of and status for each survey invited to (started, complete, screened etc), membership status. This information can be used to generate comprehensive participation level information. Certain data on participation levels are available to clients on request. 24. Do you use data quality analysis and validation techniques to identify inattentive and fraudulent respondents? If yes, what techniques are used and at what point in the process are they applied? Data for each project is manually screened. Surveys that are completed faster than is realistic as well as surveys containing inappropriate or non-serious responses will be excluded. These members are then removed from the database. 25. Do you measure respondent satisfaction? Respondent feedback is available for all surveys. This feedback can be particularly helpful in designing surveys that are more user-friendly and in maintaining a website that users want to be a part of. 26. What information do you provide to debrief your client after the project has finished? The following statistics are available on request for all projects: total invites, start rate, response rate, completion rate, screen out and quota full rate, interview length. Clients may view or recommend changes to the standard invitation. We are happy to provide any further information required on the fieldwork process and panel. Contact: e: info@panel1.com.au t: