Direct Mail Testing: Innovations and Insights for Challenging Markets

Transcription

1 1

2 Direct Mail Testing: Innovations and Insights for Challenging Markets Richard Varey, Vice President, Consumer Marketing Financial Times Chris Law, Senior Director, Marketing and Analysis THD Gordon Bell, President LucidView Tuesday, October 16,

3 Scientific DM Testing Opportunity Perhaps one time in fifty a guess may be right. But fifty times in fifty an actual test tells you what to do and avoid. (Claude Hopkins, My Life in Advertising, 1927) The more you test, the more profitable your direct mail will become. Innovations, provided you test them, can work wonders. (David Ogilvy, Ogilvy on Advertising, 1983) Test everything has been a rallying cry in the marketing and advertising industry throughout the 20th century. (Ledolter & Swersey, Testing 123, 2007) 3

4 Scientific DM Testing Challenge Improving upon success The secret of all effective originality in advertising is not the creation of new and tricky words and pictures, but one of putting familiar words and pictures into new relationships. (Leo Burnett) Constant change in the marketplace It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change. (Charles Darwin) Tight budgets The man who stops testing to save money is like the man who stops the clock to save time. Risk No one tests the depth of a river with both feet. (African proverb) 4

5 Scientific Testing = structured framework beneath each campaign 5

6 But in the marketplace it can get messy 6

7 Strategic Testing approach 1. Robust statistical design underlying each test 2. Clear, consistent execution 3. Fast tests shorttermterm metrics 4. Strategic cycle of tests Screen Refine Adapt 7

8 Strategic Testing test designs Splitrun tests Scientific multivariable tests Key decisions Number of test elements Levels for each element (optimal = 2) Effects of interest: main effects and/or interactions Number of recipes (test cells) Statistical complexity 8

9 THD: Approach to DM testing Philosophy: Testing is paramount to a successful and evergrowing file. Live by the adage: Test, test more, then test again Become intimately involved in your client s data Gain a crystal clear understanding of all that has occurred in the past and use that learning to develop future testing strategies. Everything is imminently testable: Creative, Offer, Message, Ask, Audience, etc.. Apply same vigilance and methodical approach in all facets of the program 9

10 Case study #1: Nonprofit fundraising Test was for the American Lung Association Holiday name sticker renewal campaign 3.1 million pieces mailed in the campaign One 15element test within this campaign Mailed in 16 different recipes 500,000 mailed within this test Challenge: improve upon ongoing creative tests 10

11 Test Elements Test Elements () Control () New Idea A Address label on envelope Window Mailing label on closed face B Envelope graphic and teaser Control (label graphic and teaser) No art, new teaser C "Urgent" stamp on envelope No stamp "Urgent" stamp on OE D Copy theme Control copy "Urgent" copy theme E Campaign name Generic, unbranded appeal Holiday Drive campaign name F Add deadline No deadline "Please respond by" date G Gift ask Control ask array One gift ask H Accentuate gift ask Control reply slip Highlight gift ask J Check boxes on reply slip Control (check boxes on front) Move check boxes to back K Backer color Control (black) Red and black L Label design Control label design New label design M Monograms on labels Labels with monograms No monograms N Additional labels Control (6 panel form) Add more labels O Lift note None Add lift note P Reply envelope CRE ("place stamp here") BRE (postage paid) 11

12 Creatives control package (recipe 2) 12

13 Creatives control package (recipe 2) 13

14 Recipe #8 outer envelope (A, B, C, L) ( versus control OE) A 14

15 Recipe #8 outer envelope (A, B, C, L) ( versus control OE) B 15

16 Recipe #8 outer envelope (A, B, C, L) ( versus control OE) C 16

17 Recipe #6 outer envelope (A, B, C, L) ( versus control OE) AL 17

18 Recipe #3: reply slip and copy (D, E, F, G, H, J) G H J F D 18

19 Test Design Address label on envelope Envelope graphic and teaser "Urgent" stamp on envelope Copy theme Campaign name Add deadline Gift ask Accentuate gift ask Check boxes on reply slip Backer color Label design Monograms on labels Additional labels Lift note Reply envelope (disguised data) Recipe A B C D E F G H J K L M N O P Volume Response Avg rate Gift % % % % % % % % % % % % % % % % 500, % 19

20 ALA test results: 15 main effects (and interaction) ALA holiday test: Response Rate optimal = 30.1% lift N: Additional labels (yes) P: Reply envelope (CRE) L: Label design (control) B: OE graphic/teaser (control) H: Accentuate gift ask (yes) A: OE address label (new) E: Campaign name (yes) M: Label monograms (yes) C: OE urgent stamp (yes) K: Backer color F: Add deadline G: Gift ask J: Check boxes on reply slip D: Copy theme O: Lift note Significant effects (beyond line) LM twoway interaction ti (response rate) L: Control L: New Design Label Design M: Include monograms M: No monograms 20

21 ALA test results: Net Revenue ALA holiday test: Net/M P: Reply envelope (CRE) E: Campaign name (yes) B: OE graphic/teaser (control) H: Accentuate gift ask (yes) L: Label design (control) M: Label monograms (yes) G: Gift ask (one) C: OE urgent stamp (yes) D: Copy theme A: OE address label F: Add deadline K: Backer color N: Additional labels O: Lift note J: Check boxes on reply slip 21

22 Response Rate versus Net Revenue ALA holiday test: Response Rate optimal = 30.1% lift ALA holiday test: Net/M (in same order as response) N: Additional labels (yes) P: Reply envelope (CRE) L: Label design (control) B: OE graphic/teaser (control) H: Accentuate gift ask (yes) A: OE address label (new) E: Campaign name (yes) M: Label monograms (yes) C: OE urgent stamp (yes) K: Backer color F: Add deadline G: Gift ask J: Check boxes on reply slip D: Copy theme O: Lift note Significant effects (beyond line) 22

23 Test Design pros and cons Pros Tested 16 out of 32,768 possible combinations Yet could predict outcome for all other combos Quicker to market confidence in results allows for larger rollouts Each main variable tested to over half of test volume or 250, donors In contrast, splitrun testing: Limits the number of variables tested or else requires sample size > 4.7 million or 15 drops for equal confidence Cannot analyze interactions Cons Greater cost and effort than splitrun testing Complex statistical test design requires guidance 23

24 THD: Agency Insights Don t be afraid of the water It wasn t as bad as it looks! Production staff needs to be relied upon much more than current process Need a capable vendor network Data processing, printing, & lettershop 24

25 Financial Times: campaign test strategy The Financial Times (FT) is a daily newspaper offering business news and analysis to a global audience, with a third of its circulation in the US. For subscription growth, direct mail is still the primary DTP source. Direct mail strategy relies on database marketing (modeling, optimization) and continual testing, historically on a splitrun basis. The prime control package has been a voucher or statement of benefits format for 5 years, with a hard offer of $49 for 26 weeks (cash with order or billme). A second soft control, a different package with a 4 week riskfree trial offer, is used selectively on some lists. In this period we increased mail volume, and improved efficiency, but to quicken the process, decided to try testing with LucidView. 25

26 Case study #2: Subscriptions DM test The Financial Times Hard offer direct mail package 1 creative test 1 price test grid Challenge: allow adequate sample size with relatively low response rates Challenge: how to improve alreadystrong control 26

27 Creative test elements Test Elements () Control () New Idea A Live stamp Meter Stamp B Teaser on envelope (front and back) No teaser Add teasers (front and back) C Picture of newspaper No Yes, add graphic of newspaper D Benefits copy Control New benefits copy E Reply slip layout Control New layout F Reply slip creative Control (Prof Rate Offer) New (green Savings Certificate) G "Reply by" date Control Hype deadline H Calltoaction Control (no URL) Add URL J Request address No Yes K Brochure Control (mininewspaper) New brochure L Letter from Robin No Yes, add letter 27

28 Live stamp Teasers on envelope Picture of newspaper Benefits copy Reply slip layout Reply slip creative "Reply by" date Calltoaction Request address Brochure Letter from Robin Recipe A B C D E F G H J K L Creative test design & recipes Package A C D E Recipe 1 (control) Recipe 2 Recipe 3 Recipe 4 A: Live stamp Meter Stamp Stamp Meter B: Teasers on envelope C: Picture of newspaper No teaser No Add teasers (front and back) Yes, add graphic of newspaper No teaser Yes, add graphic of newspaper No teaser D: Benefits copy Control Control New benefits copy New benefits copy E: Reply slip layout F: Reply slip creative G: "Reply by" date H: Calltoaction J: Request address K: Brochure L: Letter from Robin Control New layout Control New layout Control (Prof Rate Offer) New (green Savings Certificate) New (green Savings Certificate) No New (green Savings Certificate) Control Control Control Control Control (no URL) Add URL Control (no URL) No No No Control (mininewspaper) Control (mininewspaper) New brochure Add URL Yes, request address Control (mininewspaper) No No Yes, add letter Yes, add letter 28

29 Creatives control package (recipe 1) 29

30 Recipe #4 (C, D, E, F, G, H, J) and #7 (C, D, E, F, G, H, J) 30

31 FT creative test: 11 main effects FT hard offer Creative Test: Gross response rate F: Reply slip creative (control) 11.3% drop in response versus the control B: Teasers on envelope (none) 9.8% drop D: Benefits copy E: Reply slip layout C: Picture of newspaper H: Calltoaction 8.4% lift Positive, but not significant opportunity for refining A: Live stamp G: Reply by date K: Brochure L: Letter from Robin Significant effects (beyond line) J: Request address 31

32 Price test elements Test Elements () Control () New Idea A Price $49 for 26 weeks $99 for 12 months B Longer term (25%) No Yes (32 weeks / 15 months) C Shipping i & handling fee No Yes, $4.95 D Streamline price box No Yes, redesign price box and (3 lines vs 5) reply slip 32

33 Price Longer term (25%) S&H (delivery) fee Streamline price box Recipe A B C D Price test design & recipes A P R S Recipe 1 (control) Recipe 2 Recipe 3 Recipe 4 A: Price $49 $99 $49 $99 B: Longer term (25%) C: S&H (delivery) fee D: Streamline price box No ($49 for 26 weeks) No ($99 for 12 months) Yes ($49 for 32 weeks) Yes ($99 for 15 months) No No No No Yes, redesign price Yes, redesign price No price box box and reply slip box and reply slip T U V W Recipe 5 Recipe 6 Recipe 7 Recipe 8 A: Price $49 $99 $49 $99 No B: Longer term (25%) C: S&H (delivery) fee D: Streamline price box No ($49 for 26 weeks) No ($99 for 12 months) Yes ($49 for 32 weeks) Yes ($99 for 15 months) Yes, $4.95 Yes, $4.95 Yes, $4.95 Yes, $4.95 Yes, redesign price box and reply slip No No Yes, redesign price box and reply slip 33

34 Creatives control (recipe 1) vs. all plus recipe 8 34

35 FT price test: main effects & interactions FT hard offer Price Test: Gross response rate optimal = 32.7% lift B: Longer term (yes, 25%) 18.9% increase in response vs the control C: Shipping & handling fee (no) 17.2% drop BC interaction 13.8% further increase in response if... BC Interaction D: Streamline price box A: Price AC interaction AB interaction B: Current (26 wk / 12 mo) B: Longer term (25%) Subscription Term C: No S&H fee C: Add $4.95 S&H fee 35

36 Overall Test Results 5 significant main effects 1 interaction 41% overall increase in net response Creative test: 13% increase in net response (calltoaction: add URL) 7% increase in % payup Price test: 28% increase in net response If these 15 elements were tested with splitrun techniques Not one effect would have been statistically significant Interactions would be impossible to see Only 3 or 4 elements could be tested with equal confidence 36

37 The Financial Times: Insights & advice Faster deeper learning: we raised the number of test panels nearly 3fold to 19, but tested 2,064 combinations. LucidView gave scope to analyze main effects, comparative effects and multiple interactions with confidence. Initial investment (consultancy & production) pays off on rollout. out Get brainstorming participation from a wide group, not just the direct marketers, and think bold. Use LucidView s flexibility to test a wide range of designs. Scale the experiment to your budget. Expect a few variables to make a positive difference, a few negative and most none. Continue: next steps to confirm and refine tests (Fall 07). If you are not a statistical genius, consult an expert, ideally with the relevant industry experience. 37

38 Direct Mail Testing: Innovations and Insights for Challenging Markets Richard Varey, Vice President, Consumer Marketing, Financial Times Chris Law, Senior Director, Marketing and Analysis THD Gordon Bell, President LucidView Tuesday, October 16,