The Effect of Pre-Notification on Mail Survey Response Rates: An Experimental Result

Transcription

1 The Effect of Pre-Notification on Mail Survey Response Rates: An Experimental Result Malcolm Wright This study investigates whether pre-notification by mail increases response rates or reduces survey cost when it is used in conjunction with a survey involving reply paid envelopes, two reminder letters, and a replacement questionnaire - as suggested by Brennan (1992). It also compares the effects of prenotification and reminder letters as method of increasing the number of respondent contacts. These results confirm that, contrary to the findings of the previously published meta-analyses, prenotification by mail does not increase response rates when it is used in conjunction with reply paid envelopes, two reminder letters, and a replacement questionnaire. All pre-notification does is increase total survey cost. Keywords: mail survey, response rates, pre-notification Introduction Although mail surveys are often assumed to have low response rates, techniques for improving mail survey response have been known for some time. Kanuk and Berenson (1975) reviewed a number of articles in this area and found that follow-up letters and monetary incentives were particularly effective. More recently Brennan (1992), in an examination of a number of commercial and experimental results, demonstrated that it is possible to achieve response rates in excess of 60% routinely in mail surveys in New Zealand. Brennan (1992) claimed that the most basic approach to achieving high mail survey response rates was to use at least two reminders, include a copy of the questionnaire with each reminder, and supply reply paid envelopes. He also reported on a variety of other techniques for improving response rates, the most effective of which was a 50c monetary incentive. One technique which was not assessed by Brennan (1992), but which has received considerable attention elsewhere, is the use of preliminary notification letters to facilitate response. At first glance the international evidence in favour of the use of pre-notification seems conclusive. Major meta-analyses of mail survey response rates have consistently found significant effects from pre-notification. Yu and Cooper (1983) and Fox, Crask, and Kim (1988) both reported increases of 8% in response rate, while Yammarino, Skinner and Childers (1991) reported an average increase of 28.5%. These meta-analyses suffer from some hidden problems. Only Fox, Crask, and Kim (1988) distinguish between mail and telephone pre-notification in their meta-analyses, yet it seems reasonable that these two methods will differ in both cost and effectiveness. Furthermore, many of the studies used in the analyses are rather dated and, given the evidence of reducing survey participation (Brennan, 1991), it seems likely that respondent behaviour has changed since much of the original research took place. Most importantly, an examination of the original literature on pre-notification by mail suggests that pre-notification will not have any effect beyond that which would be achieved Page 1 of 6

2 by the follow up techniques which are now standard practice. Studies by Heaton (1965), Stafford (1966), Ford (1968), Pucel et al. (1971), Walker and Burdick (1977), and Jones and Lang (1980) all reported noticeable increases in response rates with the use of prenotification, but all also had control group response rates of well under 50%. By contrast, studies by Parsons and Medford (1972), Childers and Skinner (1979), and Jobber and Sanderson (1983) reported nil or negative effects on response rate when pre-notification was used, but all had control group response rates in excess of 50%. Thus, when response rates are low, it seems reasonable to expect that any sort of additional contact will increase them, although Kanuk and Berenson (1975) have commented that while pre-notification has a positive effect, the use of follow ups seems to be a better investment. This study investigates whether pre-notification by mail increases response rates or reduces survey cost when it is used in conjunction with a survey involving reply paid envelopes, two reminder letters, and a replacement questionnaire - as suggested by Brennan (1992). It also compares the effects of pre-notification and reminder letters as method of increasing the number of respondent contacts. Method The study used a commercial survey (in fact a census) of local government representatives, recently elected for the first time, to investigate the effects of pre-notification by mail. A preliminary list of 316 councillors was developed for the survey, similar to a list which had been used approximately eighteen months earlier for a survey on a related topic. The list was ordered alphabetically and systematically sampled (three names at a time) to give an experimental group of 156 people and a control group of 160. The experimental treatment involved a pre-notification letter which referred to the earlier survey, announced the current survey, and invited respondents to inform the researcher if they did not wish to take part. Two weeks later both the experimental and control groups were sent identical questionnaires and covering letters (which also referred to the earlier survey), and reply paid envelopes. These were followed at fortnightly intervals by two waves of reminder letters to non-respondents. To reduce costs, only the second reminder letter contained a replacement questionnaire and reply paid envelope. Both the original covering letter and the second reminder offered a summary of the survey results as a response incentive. The earlier survey on a related topic does represent a potentially confounding factor, as it may have affected the overall response rate for the survey. However it should make no difference to the findings on pre-notification; as the experimental and control groups had equal exposure to the earlier survey, any resulting change in response rate would apply equally to both groups, and therefore would not confound the difference between the experimental and control group. Results and Discussion Approximately 25% of respondents were outside the census population, and the sample size was further reduced by assuming that non-respondents were ineligible in the same proportion. Also, 15% of list members proved to be "gone - no address". These factors reduced the size of the experimental group to 94 and the control group to 107. The response rates for each of these groups are shown in Table 1, in both absolute and percentage terms. Page 2 of 6

3 Table 1. Response rates Mailout Pre-notification 1 Control Group 2 n % n % Questionnaire Package First Reminder Second Reminder Total Note: 1. N = N = 107 As Table 1 demonstrates, the addition of a pre-notification letter had a very small positive effect on overall response rate in this census - an additional 3.9%. However this small advantage may well be due to the higher number of "gone no address" list members identified by the additional letter (27 "gone no address" responses came from the pre-notification group, and 19 from the control group). If "gone no address" responses are added back into the analysis, the response rates become 61.2% for pre-notification and 63.5% for the control group. Both results are consistent with the earlier observation that pre-notification by mail was ineffective in studies with response rates over 50%. Pre-notification by mail did accelerate response, producing a response rate to the questionnaire package of 56% compared with 41% from the control group. This is consistent with what one would expect given the established relationship between the number of respondent contacts and response rate. However, contrary to the conclusion of Kanuk and Berenson (1975), pre-notification by mail was as effective as a single follow up letter; two contacts with the experimental group (pre-notification and the main survey mailing) led to a response rate of 56.4%, while two contacts with the control group (the main survey mailing and one reminder letter) led to response rate of 57.0%. When a third contact took place (the first reminder for the experimental group, the second reminder for the control group), the 66.0% response rate of the pre-notification group became noticeably inferior to the 74.8% response rate of the control group, although the control group's second reminder did include an additional questionnaire. The fourth contact required under pre-notification to achieve a comparable response rate to the control group outweighed any savings made to that point, and made pre-notification a more expensive option (savings were accrued from identifying some of the list to be "gone - no address", ineligible, or refusers before the main questionnaire mailing, and from reducing the size and cost of reminder letter mailings due to the acceleration of response). The costs for each method are analysed in Table 2 and Table 3. This analysis is based on the return rates and mailing costs of this study. It excludes the cost of reply paid postage. To enable easier comparison of mailouts within each treatment, the data has been standardised to reflect mailings per hundred original list members. To avoid random variation in "gone - no Page 3 of 6

4 address" and ineligible responses confounding the analyses, it has also been assumed that 30% of the list will be identified as "gone no address" or ineligible in both treatments, half on the first contact and half on the second contact (this is close to the average survey results). Table 2. Per Hundred List Members - Pre-notification Mailout Production & postage cost Mailing size Returns, GNA, & ineligible Cumulative Pre-notification $ $ $ Questionnaire $ $ $ First reminder $ $ $ Second reminder $ $ $ Table 3. Per Hundred List Members - Control Mailout Production & postage cost Mailing size Returns, GNA, & ineligible Cumulative Questionnaire $ $ $ First reminder $ $ $ Second reminder $ $ $ A comparison of Table 2 and Table 3 demonstrates that pre-notification by mail, in conjunction with two reminder letters, actually increased the cost of the survey by $16.88 per hundred list members. Even after two mailouts, using pre-notification was $3.96 per hundred list members more expensive than using a reminder letter. After three mailouts, prenotification by mail did reduce the cost of the survey by $21.20 per hundred list members, but at the expense of a response rate 8.8% lower than that of the control group (see Table 1). The lower response rate after three mailouts for the pre-notification group can be partly attributed to the lack of a replacement questionnaire in the third mailout. If the third mailout to the group had included a replacement questionnaire, however, the cost per hundred list members would have escalated to $219.84, which is $5.32 more expensive than the control treatment. Furthermore, it is questionable whether including a questionnaire in the third mailout to the pre-notification group would have increased the response rate to a level comparable with the control treatment. Page 4 of 6

5 Conclusions This study has confirmed that, contrary to the findings of the previously published metaanalyses, pre-notification by mail does not increase response rates when it is used in conjunction with reply paid envelopes, two reminder letters, and a replacement questionnaire. All pre-notification does is increase total survey cost. This research involved people in responsible public positions and was sponsored by an organisation with which they had an association, so it is possible that response rates were affected by the type of respondent or by the salience of the survey. While the research was likely to have been particularly salient to the respondents, it seems likely that this would only affect the overall response rate. There was no reason why salience should have differed between the experimental and control groups, especially since up to two reminders were sent to non-respondents in both groups. Furthermore earlier research on the effect of prenotification involving business executives (Jobber & Sanderson, 1983) and religious leaders (Parsons & Medford, 1972) gave similar results to research on pre-notification conducted with recent graduates (Parsons & Medford, 1972) and insurance agents (Childers & Skinner, 1979). These earlier results suggest that the effect of pre-notification by mail does not vary with the type of respondent, although the effect on a more general population sample has not been investigated; this remains an area for further research. Despite the issue of generalisability to a broader population, the implications of this study seem clear; pre-notification by mail is an unnecessary expense which adds little to the response rates already achieved by existing mail survey methodologies. References Brennan M (1991). Survey participation and attitudes towards surveys in New Zealand. New Zealand Journal of Business, 13, Brennan M (1992). Techniques for improving mail survey response rates. Marketing Bulletin, 3, Childers TL & Skinner SJ (1979). Gaining respondent cooperation in mail surveys through prior commitment. Public Opinion Quarterly, 43, Ford NM (1968). The advance letter in mail surveys. Journal of Marketing Research, 4, Fox RJ; Crask MR & Kim J (1988). Mail survey response rate: A meta analysis of selected techniques for inducing response. Public Opinion Quarterly, 52, Heaton EE Jr. (1965). Increasing mail questionnaire returns with a preliminary letter. Journal of Advertising Research, 5, Jobber D & Sanderson S (1983). The effects of a prior letter and coloured questionnaire paper on mail survey response rates. Journal of the Market Research Society, 25, Jones WH & Lang JR (1980). Sample composition bias and response bias in a mail survey: A comparison of inducement methods. Journal of Marketing Research, 17, Page 5 of 6

6 Kanuk L & Berenson C (1975). Mail surveys and response rates: A literature review. Journal of Marketing Research, 12, Parsons RJ & Medford TS (1972). The effect of advance notice in mail surveys of homogeneous groups. Public Opinion Quarterly, 36, Pucel DJ; Nelson HF & Wheeler DN (1971). Questionnaire follow-up returns as a function of incentives and responder characteristics. Vocational Guidance Quarterly, 19, Stafford JE (1966). Influence of preliminary contact on mail returns. Journal of Marketing Research, 3, Walker BJ & Burdick RK (1977). Advance correspondence and error in mail surveys. Journal of Marketing Research, 14, Yammarino FJ; Skinner SJ & Childers TL (1991). Understanding mail survey response behavior. Public Opinion Quarterly, 55, Yu J & Cooper H (1983). A quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, Malcolm Wright is a Lecturer in the Department of Marketing, Massey University. Page 6 of 6