The Library

Transcription

1 The Library Educating Professionals, Creating and Applying Knowledge, Engaging our Communities This article is 2016 Emerald Group Publishing and permission has been granted for this author accepted manuscript (post print) to be posted to an institutional repository. The link to the ROR record is available here: Emerald does not grant permission for this article to be further copied/distributed or hosted elsewhere without the express permission from Emerald Group Publishing Limited. The final published version (version of record) is available from the following link:

2 Generalisability of Advertising Persuasion Principles Author details Byron Sharp, University of South Australia Business School, Ehrenberg-Bass Institute for Marketing Science, Adelaide, South Australia, Australia Nicole Hartnett, University of South Australia Business School, Ehrenberg-Bass Institute for Marketing Science, Adelaide, South Australia, Australia

3 Abstract Purpose: To reflect on the generalisability of the predictive validity test of the Persuasion Principles Index (PPI) conducted by Armstrong, Du, Green, and Graefe (2015). Approach: Different aspects of the test are considered, such as the sample of ads, the dependent variable, and the comparability of the methods used to predict effectiveness, in terms of how relevant these are to real world advertising testing. Findings: The sample of ads and the testing procedure may have contributed to the success of the PPI predictions over the other copy testing methods. The sample of print ads does not bear a close resemblance to current advertising. The competing copy tests do not represent modern advertising copy testing. Research implications: More research is needed to test the validity of the principles and the predictive accuracy of the PPI across a range of conditions (e.g. different ads, media, products, cultures, etc.). Testing against advertising sales effectiveness would be the ideal next step. Practical implications: It certainly seems the index method has the potential to help advertisers make better decisions regarding what executions to support, for highinvolvement products at least. Given the accessibility of the software it should be easy and cost effective for advertisers to trial the PPI. Originality/value: This commentary directs researchers to the real world conditions under which advertising pre-tests need to be evaluated.

4 Generalisability of Advertising Persuasion Principles Advertising is becoming more accountable. Creative decisions, though, remain very difficult to justify. Armstrong s (2010) evidence-based persuasion principles are a landmark effort to bring together evidence to support creative decisions. In this invited commentary, we ask: What does this test of the Persuasion Principles Index (PPI), and by association the persuasion principles themselves, reveal about the big, wide world of advertising effects? Our conclusion is: Less than we would like. Armstrong, Du, Green, and Graefe (2015) set out to test how useful Armstrong s (2010) persuasion principles might be to advertising professionals to evaluate and improve their advertising creative. Could rating ads based on these principles be easy and cheap, but most importantly, could they predict advertising sales effectiveness better than traditional copy testing evaluations? They created a checklist, called the Persuasion Principles Index (PPI), to assess how ads apply the persuasion principles; either applied well, could be improved, violated, or not used. They had 17 raters (novices) use the PPI to score 96 pairs of full-page print ads with five raters assigned per pair. This took raters many hours but was not expensive; we estimate it was about $28 per pair including the fixed cost of the paid self-training sessions. In each pair of ads one ad had a higher day-after recall score 1 than the other, and the PPI was assessed on its ability to correctly identify the better-recalled ad. So, sales 1 Brand prompted day-after recall results were provided by Gallup and Robinson. Their method was to deliver the current issue of a magazine to people s homes and asked participants to read the magazine the same day. They were then interviewed via telephone the following day. According to one of the Which Ad Pulled Best editions: Respondents were read a list of ads appearing in the magazine and asked which ones they remembered. Then, for each of the ads that respondent claimed to remember, they were asked to describe elements from the ad to prove they were remembering the ad of interest, i.e. You may be familiar with other ads for, but thinking only of this issue, please describe the ad as you remember it. What did the ad look like or say? Sample size for a typical survey was 150 adults, who had read previous issues, but not the current issue, of the specific magazine that included the ad.

5 effectiveness, unfortunately, was not evaluated directly. How often the PPI identified the better-recalled ad of the 96 pairs was compared to alternative methods for judging the potential effectiveness of the same advertising creative executions. Before commenting on the findings, it is necessary to put this test into perspective. In hindsight, the following facts turn out to load the dice somewhat in favour of the PPI. The Which Ad Pulled Best (WAPB) sample of ads is not ideal to test the predictive validity of the principles because the same sample features heavily in providing supporting evidence for the principles as well, see Armstrong (2010). The PPI is, therefore, weighted towards principles based on WAPB ads and their relation to dayafter recall. As such, this sample creates a weaker test, somewhat akin to testing a model to a holdout sample. The test is also purposely confined to ads for high-involvement utilitarian products (e.g. cars, cold remedies, electronics) where we expected the persuasion principles to be more useful for predicting the effectiveness of ads. As already mentioned, the test looks at the ability to predict superior day-after recall, not the effects of creative on sales. Even if we can predict recall, and do so better than other methods, recall is still a reasonably poor predictor of in-market sales response (Haley and Baldinger, 1991; Lodish et al., 1995). What of the results? Armstrong et al. (2015) compared the performance of their PPI to other approaches for evaluating advertising creative, such as the unaided judgment

6 of different groups of people. Please refer to the table below for a summary of how the competing approaches performed. None of the evaluation methods did especially well given that random chance would result in a score of 50% and each method did similarly well/poorly looking at individuals ratings, judgments, or intentions. Table: Percent of better-recalled ads correctly identified by different methods Evaluation method Individual predictions % Combined predictions % Random chance Trained novices apply the Persuasion Principles Index (PPI) Unaided judgement of novices, including university students and regular people recruited via M-Turk Unaided judgement of 16 advertising professionals Unaided judgement of people who claimed to have advertising experience recruited via M-Turk Copy testing (purchase intentions scores) with regular people recruited via M- Turk were asked to imagine they were in the market for this type of product n/a Combining judgments for each pair lifted the success rates for all methods. For the unaided judgments, this meant taking the modal forecast, whereby more people said Ad A was more effective than Ad B for Pair X, therefore, the combined judgment predicted Ad A, correctly or incorrectly. Novice judgments improved from 54% to 62%. Experts from 55% to 63%. Armstrong et al. (2015) provided us with the combined judgments for the 16 advertising professionals, not reported in their paper, which also showed a subtler improvement from 60% to 64%. It should be noted, however, that this data refers to only 20 pairs of ads and combines three judges per pair rather than five.

7 Combining the judgments of the PPI was not as straightforward. Rather than combining the judgments of the trained PPI raters, where 3 or more of the 5 raters agreed on which ad was superior, Armstrong et al. (2015) used consensus scores across the principles to come to a single consensus judgment. Where there was not consensus for a particular principle that is, whether the principle applied to the pair and how it was used (well, could be improved, violated, not used) for Ad A and/or Ad B that principle was not included in calculating the index score for one or both of the ads in any pair. Raters who consistently disagreed with other raters (they were considered unreliable), which would have lowered the number of principles for which there was a consensus, were also dropped and replaced by more agreeable raters. The conclusion that the PPI exceeds other methods in predictive ability hinges on the fact that this combining procedure improved the predictive accuracy of the PPI from 61% to 75%. We do not really know if the other approaches might also benefit from similar theory-based enhancement, for example, by removing outliers, removing judges who appear to be guessing, and so on. Returning to our original question about what this says about advertising in the real world, we find that the methods that were pitched against the PPI look mostly atypical, not the sort in everyday use. The copy testing, which had to ask respondents to imagine that they were in the market at the time the ad was run (in the 1980s and 1990s), bares little resemblance to the commercial copy testing services that we know of. What is more, the samples of respondents are rather unconventional, advertising practitioners

8 from one marketing department in China or recruits from Amazon s Mechanical Turk (M-Turk). This test consequently does not tell us much about the PPI s superiority or inferiority to the techniques routinely used today, such as Millard Brown s Link, IPSOS- ASI s Connect, or Nielsen s Brand Effect, nor of the unaided judgment of senior managers who control large advertising budgets. It is possible that these commercial approaches would perform similarly to the PPI or that the PPI would outperform them, we just do not know. We also do not know when, where, under what conditions, for which product categories, in which media, or for what styles of advertising. Nor do we know if the persuasion principles have any ability to predict the all-important sales effect of the advertising. The WAPB sample is further limited to print ads, most of which are at least 20 to 30 years old. Print ads have changed over this period. One such change is including fewer words and more pictures. Content analysis of the very same WAPB sample demonstrated this trend with a marked increase/decrease in dominant pictures/substantive body copy, particularly from 1996 to 2002, which are the two most recent editions of WAPB (McQuarrie and Phillips, 2008). Depending on which editions the 96 pairs were taken from, which is not discussed in detail by Armstrong et al. (2015), it is possible that the ads evaluated in this test differ somewhat from contemporary print ads. Aside from the specificity of this unusual sample of print ads, this is, of course, a single test. From one test it is not possible to know if the results might replicate for another set of ads, for video ads rather than print ads (note, there are different sets of principles for print and motion ads), for another media, for different sorts of products, or

9 in different countries, and so on (see Hubbard and Armstrong, 1994; Stern and Ehrenberg 1997). The authors do suggest that the principles are invariant across cultures or language, something which cannot be known from the available evidence. In closing, Armstrong (2010) spent more than a decade systematically vetting and documenting the extensive body of empirical research on advertising s creative effects. What he found, he has distilled into reasonably simple principles, many of which include conditions (i.e. if/then type statements). This was no small feat. There are almost 200 principles, which reflects just how complex creative advertising is. There is much more to be done, however. Some of the principles have considerable (predominantly experimental) supporting evidence, whereas a few have almost none. None of the principles are likely to be infallible even with one or two conditions, hence, more research is needed to refine and extend the principles as they stand. Fortunately, the index approach makes it quite easy to incorporate new discoveries when they are made and these new discoveries may improve its effectiveness and/or applicability to a wider range of advertising situations. Armstrong and his colleagues have done exemplary work to make their index accessible and user-friendly (an Excel spreadsheet via as well as being transparent about how their index is calculated (there are no black box algorithms). We hope that this encourages others to conduct and share their own tests of the index s generalisability.

10 References Armstrong, J.S. (2010), Persuasive Advertising: Evidence-based Principles, Palgrave MacMillan, Hampshire, UK. Haley, R.I. and Baldinger, A.L. (1991), The ARF copy research validity project, Journal of Advertising Research, Vol. 31 No. 2, pp Hubbard, R. and Armstrong, J.S. (1994), Replications and extensions in marketing: Rarely published but quite contrary, International Journal of Research in Marketing, Vol.11 No. 3, pp Lodish, L.M., Abraham, M., Kalmenson, S., Livelsberger, J., Lubetkin, B., Richardson, B. and Stevens, M.E. (1995), How T.V. advertising works: A meta-analysis of 389 real world split cable T.V. advertising experiments, Journal of Marketing Research, Vol. 32 No. 2, pp McQuarrie, E.F. and Phillips, B.J. (2008), It s not your father s magazine ad: Magnitude and direction of recent changes in advertising style, Journal of Advertising, Vol. 37 No. 3, pp Stern, P. and Ehrenberg, A.S.C. (1997), Replication means extension, paper presented at 26th European Marketing Academy Conference, May, (place of conference), available at: (accessed ).