STRATEGIES FOR GROWTH SM "CONSULTANTS TO THE SERVICES INDUSTRY" Comparative Advantages/ Disadvantages of Phone, Mail & E-Mail/Web-Based Surveys Prepared by: William K. Pollock President Strategies For Growth SM Westtown, Pennsylvania 2004 Strategies For Growth SM
Comparative Cost Moderate to High - dependent on number of interviews, respondent incidence rate, length of questionnaire, number of openended questions, degree of customer segmentation, etc. Low to moderate - varies by complexity/sample design; heavily dependent on number of pages/ inserts (i.e., weight); cost of outbound/ inbound postage and questionnaire/ envelope printing is high, but is sometimes offset by internal printing/ posting capabilities; return postage from outside U.S. adds cost. Low - majority of costs are related to questionnaire design/programming, software for mass distribution and receipt of completed e-questionnaires; but virtually no cost for paper, printing or postage. Data processing costs may actually be lower through the use of embedded data collection/processing tools. Geographic Coverage No problem covering all time zones in North America - covering French-speaking Canada and Spanish-speaking Mexico would add some incremental cost for additional native-language trained interviewers. Must be conducted in native languages, at local times. No problem mailing to all of North America - mailing outside of U.S. adds cost for return postage; mailing to French-speaking Canada and Spanish-speaking Mexico would add some cost for translated questionnaires. Generally must be conducted on an individual country basis. No problem e-mailing to targeted respondents on a worldwide basis; generally conducted on a language- (and cultural) specific basis, rather than on a geographic basis. Turnaround Time Fast turnaround time once the questionnaire has been programmed and the sample has been processed and entered into the system - also allows for interim topline printouts at any time during the course of the survey. Slow turnaround time - preparatory design, printing and envelope stuffing may take a few weeks or more; may then take up to 4-6 weeks on the street before returns diminish; may require a reminder mailing (adding time/expense). Fast turnaround time once the questionnaire has been preprogrammed, the sample has been processed and entered into the system, and the data processing specs have been developed - generally allows for ad hoc detailed printouts at any time during the course of the survey. Screening of Respondents Best method for screening and managing respondent quotas in terms of respondent relevance, as well as per category/cell. Limited screening capability for a small sample mailout (e.g., no guarantee that the questionnaire will reach the targeted respondent); however, not a problem if the entire universe (e.g., 20,000 customers) is targeted. Limited screening capability - due to the nature of the methodology, questionnaires can only be disseminated to potential respondents that have valid e-mail addresses; this may add significant skewing and, as a result, may not be applicable in many situations. Page 2 of 5.
Sample Respondents Good, for named respondents with current telephone numbers - not as good when the interviewer must use a generic screen to reach unnamed respondents. Good, for named respondents with current home/business addresses - but not good at all for unnamed respondents (i.e., low likelihood of having the mail questionnaire passed on to the appropriate individual). Good only for named respondents with current e-mail addresses - but not good at all for those without PC access, e-mail addresses, or any unnamed respondents (i.e., low likelihood of having the e-mail questionnaire passed on to the appropriate individual). Overall Response Rate Best of all methodologies - typically about 50% of the qualified customer base responds (depending on the overall quality of the customer list); provides the most management control in terms of meeting individual segment/cell quotas. Very low percentage - typically on the order of 15% - 25% (depending on the availability of a clean / qualified customer list). Very low percentage (but highly dependent on the specific target segments) - typically on the order of 10% - 20% (depending on the availability of a clean / qualified e- customer list). Highest likelihood of response comes from segments that reflect the highest continuous use of PCs, the Internet, and e-mail. Non-Response Bias Small - reducible by prior notice/alerts, callbacks, etc.; controllable by interviewer. Moderate to large - reducible by incentives, but would involve a high expense for large mailings; respondent can selectively skip questions, or misread skip patterns, leading to incomplete responses. Large - due to the fact that not all targeted respondents have e-mail addresses, or easy access to a PC or the Internet; therefore, the nonresponse bias will generally be significant. Validity of Response Relies on memory - little chance to look it up while interview is underway; however, calls made soon after the event are more likely to obtain current information. Relies on memory - but may be aided by easily accessible/handy records/ memos, where applicable; depends on question, and ease of access to back-up information. Relies on memory - but may be aided by easily accessible/handy records/memos, where applicable; depends on question, and ease of access to back-up information. Accessibility of Respondent Variable - depends on availability and contact data; better controlled through having respondent name and phone number available, and through scheduled callbacks. Variable - depending on quality and accuracy of the customer list (i.e., name, title, address, etc.). Variable - depending on quality and accuracy of the e-customer list (i.e., name, title, physical address, e-mail address, etc.). Page 3 of 5.
Type of Questionnaire No limitation beyond interviewer skill - can ask scripted probing follow-up questions such as Why do you say that?, How often has that happened in the past 12 months?, For what reason?, etc.; however, limited ability to ask complex questions (e.g., attribute rankings, responses totaling to 100%, etc.). Generally limited to check-off and fillin responses with only limited openended questions; no further ability to probe or clarify response. Generally limited to check-off and fillin responses with only limited openended questions; no further ability to probe or clarify response (unless built into the web-based or e-questionnaire through a series of intricate skip patterns ). Questionnaire Revisions Highly flexible - open to immediate changes on the fly based on preliminary responses. None, once mailed - requires pilot interviews and pre-tests. Somewhat flexible for Web-based; but not so for e-mail surveys - Webbased surveys are open to impromptu changes on the fly ; e-mail surveys are fairly fixed once e-distributed. Alternate Question Sets Full opportunity to add questions, or skip patterns, as required, at any time. Not recommended - adds too much complexity/pages and skip patterns to the printed questionnaire. Full opportunity to add questions, or skip patterns, as required, for webbased (but not for e-mailed) surveys. Question Order Totally controlled by Computer- Aided Telephone Interviewing (CATI) system (including staggered starts for attribute testing, etc.). No control - respondent can skip questions and come back later, or change answers to earlier questions based on responses to later questions. Highly controlled for web-based surveys, but not for e-mail surveys - respondents must answer web-based surveys in sequence, but may answer e-mail surveys in any order. Length of Interview 15 minutes is the conventional maximum for business interviews. 4-8 pages is the conventional maximum for business questionnaires (but, regardless of length, must always look extremely professional, and should be printed on a single sheet of paper (e.g., 11x 17, 11x 25.5) or in a bound booklet (i.e., not stapled). Equivalent of 4-8 pages is the conventional maximum for business questionnaires (but, regardless of length, must always look extremely professional, and should be easy to follow, back-up and correct, pause, etc. (i.e., must be user-friendly). Interviewer Bias Exists to a small degree - but generally not an issue. None - no interviewer is involved. None - no interviewer is involved. Page 4 of 5.
Pace of Interview Set by interviewer - usually attempts to move as quickly as possible to avoid respondent disconnect /termination. Set by respondent - some risk that multiple starts and stops will lead to a termination. Set by respondent - should be programmed not to terminate if respondent pauses too long (e.g., to take a telephone call, etc.). Ease of Callback/ Follow-up Easiest - quick, generally inexpensive and can be used to increase final response rates. Difficult, if not impossible on a respondent-specific basis - too much of a time lag exists; also, some risk that the respondent may not believe his/her responses have been kept confidential if he/she is recontacted, or contacted by telephone. Not very difficult - if you adequately track the e-mailouts (or e-invitations to participate) and responses; easy to follow-up with e-reminders, but some risk that respondent may not believe his/her responses have been kept confidential if recontacted by e-mail. Supervisory Control Centralized, direct and immediate - remote monitoring is available for listening in during live calls. None - respondent is on his or her own. None - respondent is on his or her own. Interviewer Training Requirements Full interviewer training and specific project briefing is conducted - continuous technical assistance for interviewers is available, as needed. None - not applicable. None - not applicable. Capturing Sensitive Information More difficult than by mail - since only limited anonymity may be perceived by respondents. Best mode for sensitive information - entirely anonymous (if return envelope contains no tracking info). Mixed mode for sensitive information - since e-mail address (or respondent code/id number) is traceable. Interviewer/Respondent Error Depends on interviewer skill and complexity of questionnaire - however, errors are greatly minimized through the use of the CATI system. Respondent error may be moderate to high - for respondents answering without reading/following instructions, answering too quickly, or skipping questions, etc. Respondent error may be moderate to high - for respondents answering without reading/following instructions, answering too quickly, or skipping questions, etc. Respondent Instructions/ Assistance Interviewer provides respondent with initial instructions, scales, etc., but may not lead or prompt respondent in any way. Cover letter and instructions must be very clearly articulated - additional assistance may be made available through an 800 number or on the web. Cover e-letter and instructions must be very clearly articulated - additional assistance may be made available through an 800 number or on the web. Page 5 of 5.