A Predictive Test Of Pre-Testing Fundraising DM Materials Margaret Faulkner and Rachel Kennedy Ehrenberg-Bass Institute for Marketing Science, University of South Australia Abstract This research examines whether pre-testing as typically used in advertising, can identify the most effective fundraising direct mail. It replicates a study undertaken by the Advertising Research Foundation validating pre-test measures in a new environment, and extends it to confirm that false winners are not identified. The initial tests show promise for pre-testing in a fundraising DM setting, though further validations are recommended. Keywords: Advertising, Pre-testing, Fundraising, Direct Mail Introduction This research investigates if advertising pre-test measures can identify the potential response to fundraising direct mail (DM) materials. Pre-tests are commonly used to predict the advertisement(s) that will evoke the greatest in-market response from a set of ads. However, validating the best pre-test measures with sales performance has been difficult. Validation difficulties include isolating the communication effects, controlling for time and that only the winning concept is usually produced and put to market. The Advertising Research Foundation (ARF) overcame these and provided empirical support for pre-test measures in the only industry supported validation study, the Copy Research Validity Project (CRVP) (Haley and Baldinger 1991). It remains the most significant study of its kind (Ambler 2003) with the resources associated with the study making replication highly unlikely. Many of the validation problems are overcome when the arena is changed from pre-testing advertising to pre-testing DM. DM results are available for multiple versions tested with experimental designs (split-run tests). Changing the medium from advertising to one where split-run tests are easily performed increases the likelihood of identifying pairs with different response rates to the communication tested. The cost of putting alternative concepts to market is negligible and common practice. Typically DM practitioners compare actual results from split-run tests (Nash 2000; Sugarman 1998) letting them know the best performing piece of DM, but they lack diagnostic information (ie why one piece of DM worked or did not work) (Tapp 2000, Bird, 2000 #8773). The idea of combining market research methods with split-run tests is not new (Bodenberg and Roberts 1990), however, the literature contains no mention of pre-testing research being undertaken in this area. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 39
With fundraising DM materials, the validation task is simpler than with mass advertising as DM typically requires a response, using the device included, immediately following exposure. This means it is much easier to validate pre-test results at both aggregate and individual levels. The printed medium also makes it easier to control respondent exposure to the campaign. Testing can be controlled at the individual household level with different campaigns being sent according to the experimental design, allowing for simultaneous testing. Most respondents receive one exposure only with a short timeframe between exposure and response. DM differs from advertising in that an immediate response (i.e. a donation) is observed. With fundraising DM the link between communication and response is much clearer than with mass advertising. Thus collecting pre-test measures for fundraising DM and validating the results against split-run test results provide the opportunity to determine if pre-tests can select the winner. Research Questions The specific research questions addressed in this paper are: Can a pre-test instrument indicate which of more than one DM campaign will evoke the best response? What measure(s) identify effective DM material? Research Methodology The methodology involved a number of phases. The instrument was developed through: Exploratory interviews with 20 fundraisers and 16 donors; and Six pilot studies involving 423 respondents that tested the measures and a range of approaches (ie face to face/self completion, pick-any /scales, different samples, generic vs specific pieces of DM, etc). The range of competing measures used in the final instrument, were grounded in the literature, based on respondent s use of the measures, and the importance placed on them by fundraising experts. The original diagnostic measures taken from the literature required the most modifications for use with fundraising DM materials and are thus shown in Table 1. A 5-point agreement/disagreement scale was used to rate them, with a mid-point indicating when the adjective did not apply. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 40
Table 1: List of Diagnostic Items Measures (and their attribute from the literature) Alienating Irritating Relevant News Worth remembering Entertaining Familiar Empathy Confusing Brand Reinforcing Appealing Seen-a-lot Gentle Confusing For-a-good- Cause Wasting Funds Well done Warm (not) Easyto-forget An-oldformat Easy-toread (reverse) Reinforces- the- <charity> True-to-life Engaging Sensitive Believable Convincing Informative Meaningful Relevant Itinvolvedme Important-tome Told-mesomethingnew One of the Best letters sent Ittouchedme The pre-tests were conducted in the environment where fundraising DM material is received (i.e. at home) using real DM pieces. Guidelines for conducting pre-testing were followed where possible (ie one exposure is sufficient, with recipients spending as little time, or as much time, as their interest requires) (Rossiter and Percy 1987). Instruments followed a funnel sequence where general questions were asked about support for charities, followed by more specific questions about the specific charity and its DM (Dillon et al. 1993). Order rotation was used within groups of questions to control for bias due to order effects. Respondents were screened to ensure they read DM material before evaluating alternatives. The CRVP was used as the template for the design of this study with pre-test measures administered on fundraising DM materials mailed in two split-run tests. Test A: Winner/Loser was a replication of the CRVP with a known difference in sales response, allowing us to examine if pre-test measures were able to identify the winner in this new DM setting. Based on the 2% difference in response rate, a 32% increase in the funds raised would have occurred from selecting the more effective DM material in this case. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 41
Test B: No difference 1/No difference 2 extended the CRVP by pre-testing paired DM where no difference in response occurred, to check that winners were not falsely identified. Data collection consisted initially of 261 computer assisted telephone interviews with regular donors to the charity whose DM was being tested. Donors who reported they read DM materials sent by the relevant charity were asked to participate in a second interview. Of these, 62 agreed to evaluate letters face-to-face in their homes. While a smaller sample than ideal these numbers do reflect the smaller samples typically used in to pre-test advertising (Cramphorn 1998), thus it remains a useful test. Each respondent evaluated three letters, which were rotated across the interviews. One of the strengths of this research is that in-market results are available, at both the individual and aggregate level, on the response for each DM material. Thus each measure was analysed for its ability to determine if it was able to identify the winner more often than would occur by chance. Repeating this process for each letter at aggregate and individual level provided insights into the most appropriate measures to be used for pre-testing fundraising DM materials. The key limitation is that respondents had been exposed once, a year prior to this research, to the communications from Test A. Results for the Main Quantitative Phase The findings of this research support the use of pre-test measures for fundraising DM materials. In accordance with actual results, most pre-test measures identified differences for Test A, but not for Test B as summarised in Table 2. The five-point liking scale validated in the CRVP successfully distinguished the more effective DM from Test A (mean of 4.0 vs 3.2) but found no significant difference for Test B. Table 2: Performance of Key Measures in Identifying Split-Run Test Results Measure Bandaid : Spring National : Regional Verbal reaction to DM material Spring < negative comments (H1 supported) Bandaid > positive comments (error reverse H1) Action taken H1 supported H1 supported Type 1 error Global Liking H1 supported H0 supported as not significantly different Probability of donating H0 supported (type II error) H0 supported, as not significantly different Thirteen of the 27 diagnostic measures successfully identified the winner as significantly better than its paired DM material. They were: Easy to Read, Warm, Seen a lot, Gentle, Well Done, Appealing, One of the Best Letters sent, Sensitive, Reinforces the (charity), An Old Format, It Touched Me, Confusing and Engaging. All of the diagnostic measures (except engaging and involved me) showed no difference for Test B, thus demonstrating that the measures tend to identify differences that reflect actual response. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 42
The mean scores of all adjectives were used to form an index score of seven main attributes used in advertising (detailed in table 1). The score for all attributes indicated that the Winner DM was described as more fitting the characteristics important for effective communication. An exception was the attribute familiarity, which while associated with worn out in advertising is a more positive element for fundraising DM materials. Conclusions While very summarised in this paper, this research extends the CRVP study (Haley and Baldinger 1991). It offers new evidence on the validity of pre-tests by confirming results against actual response at both individual and aggregate levels for fundraising DM. It is important to use multiple measures when pre-testing advertising with the choice of measure related to the specific brand-situation (Baldinger 1991; Jones 1996; Moult 1995; Rossiter and Bellman 2005; Walker 1991). Predicting the winning campaign through pre-testing for fundraising DM materials also requires the use of multiple measures and a modified instrument to suit the new environment, however, that now exists. Not only does pre-testing fundraising DM predict the winning piece of DM when compared against actual results, it also does not falsely identifying a winner where there was no difference in actual results. Clearly further testing of the tool is needed and refinement of the instrument to remove unnecessary measures, but a strong basis for pre-testing fundraising DM is now available. The research findings suggest that a familiar style may be useful for existing donors to identify the charity. This is contrary to its impact on advertising effectiveness, where it is a negative attribute (Schlinger 1979). Familiarity is theorised to reduce the liking of ads, being the opposite of the entertaining attribute (du Plessis 1994). Future research of acquisition DM is recommended to examine familiarity in this setting, particularly its impact on gaining the attention of potential donors. In advertising the sample sizes for pre-testing typically range from 75 respondents (Cramphorn 1998) up to 150 (Eagleson and Rossiter 1994), with 200 per target group per execution recommended (Rossiter and Bellman 2005). 175 respondents have been recommended to pre-test ad liking (Kennedy et al. 2000) and the results provided in this study lead us to calculate that sample sizes of 200 are recommended for future DM pre-tests. Fundraisers can reduce research costs by sampling groups of 50-100 respondents progressively, and ceasing pre-testing when differences are identified with 95% confidence levels (Rossiter and Bellman 2005). ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 43
References Ambler, Tim and Goldstein, Scott (2003), Copy testing: practice and best practice: A review of UK advertising research procedures: World Advertising Research Centre. Baldinger, Allan L (1991), "The Case for Multiple Measures: Is One Ever Enough?" in Copy Research The New Evidence - Eighth Annual ARF Copy Research Workshop. New York: Advertising Research Foundation. Bodenberg, Thomas M. and Mary Lou Roberts (1990), "Integrating Marketing Research into the Direct-Marketing Testing Process: The Market Research Test," Journal of Advertising Research, October/November, 50-60. Cramphorn, Michael (1998), "When is enough, enough? A question of sample size," Admap, December. Dillon, William R, Thomas J Madden, and Neil H Firtle (1993), Essentials of Marketing Research. Homewood: Irwin. du Plessis, Erik (1994), "Understanding and Using Likeability," in ARF/ESOMAR. 583-601 ed. South Africa. Eagleson, Geoff and John R. Rossiter (1994), "Conclusions from the ARF's Copy Research Validity Project," Journal of Advertising Research, 34 (3, May/June), 19-32. Haley, Russell I. and Allan L. Baldinger (1991), "The ARF Copy Research Validity Project," Journal of Advertising Research, 31 (April/May), 11-32. Jones, John Philip (1996), "Look Before You Leap," Admap (November), 1-4. Kennedy, Rachel, Byron Sharp, and Cam Rungie (2000), "How Ad Liking (L A ) Relates to Branding & the Implications for Advertising Testing," Australasian Journal of Market Research, 8 (No. 2, July), 9-19. Moult, William H (1995), "Selective Success Amid Chaos: Advertising in the 1990's," European Society for Opinion and Marketing Research. Nash, Edward (2000), "Research," in Direct Marketing. 4th ed. New York: McGraw-Hill. Rossiter, John R. and Steven Bellman (2005), Marketing Communications: Theory and Applications. Frenchs Forest: Pearson Education. Rossiter, John R. and Larry Percy (1987), Advertising and Promotion Management. Singapore: McGraw-Hill. Schlinger, Mary Jane (1979), "A Profile of Responses to Commercials," Journal of Advertising Research, 19 (No. 2, April), 37-48. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 44
Sugarman, Joseph (1998), Marketing secrets of a mail order maverick (1 ed.). Las Vegas: DelStar Books. Tapp, Alan (2000), Principles of Direct and Database Marketing (second ed.). Essex: Pearson Education. Walker, David (1991), "Efficiency and Effectiveness: Calibrating Copy Research to Sales," in Copy Research The New Evidence - Eighth Annual ARF Copy Research Workshop. New York: Advertising Research Foundation. ANZMAC 2005 Conference: Advertising/Marketing Communication Issues 45