Advanced Topics in Direct Mail Fundraising, Part 3 A workshop with MAL WARWICK AFP Winnipeg Winnipeg, Manitoba, September 27, 2007 Copyright 2007 by Mal Warwick
In this session... 1) Test your creative head! 2) Testing, Testing, 1, 2, 3 3) 6 guidelines for successful testing
1) Test your creative head
Ask amount test Will a lower entry-level gift boost response without comparably depressing the average gift amount? Credit: Bill Rehm Mal Warwick Associates for Save-the-Redwoods League
Be careful what you ask for Longstanding control package: $20 Annual Membership offer Test: $10 Individual Member offer A/B split, two panels of 30,000 names each
Which Ask won? $20 Individual Membership? $10 Introductory Membership?
Ask amount test Package # gifts Response Avg. gift $20 Ask 188 0.62% $28.47 $10 Ask 246 0.82% $24.02
Package enclosure test Will a bold enclosure delivering big news lift response over that of a simple letter package containing the news within its text? Credit: John Genette - Black Mountain Communications, for United Way of Greater Los Angeles
Convey breaking news In the main letter copy? or In an enclosure ( Emergency Bulletin )? Mailed: 49,000 with bulletin 49,000 no bulletin Bulletin added cost of $25/m
Which version won? With the enclosure? Without the enclosure?
Package enclosure test Package # gifts Response Net/new Bulletin 302 0.61% ($26) No bulletin 355 0.72% ($8)
Cost efficiency test Will emphasizing an NGO s cost-efficiency boost response? Credit: Susie Fink DeCarlo - Adams Hussey & Associates, for Project HOPE
Creative test design OSE teaser changed to read, 94% of All Contributions To Project HOPE Go Directly To Those In Need! Inside OSE, new copy, dollar bill graphic All other package elements the same Control mailed to 784,000 names Test mailed to nth select of 35,000 names Rollout cost of $270/M same for both packages
Control package
Test package
Which version won? With the dollar bill? Without it?
Cost efficiency test Package Response Avg. gift Cost/donor Control 0.84% $19.89 $12.15 Test 0.87% $27.24 $3.71
Makeover test Will an improved and redesigned package outpull a weak control without requiring a completely new package? Credit: Bill Rehm - Mal Warwick & Associates, for the Center for Victims of Torture
The challenge Dynamic organization, compelling cause Limited creative budget Longstanding (but flawed) control package Declining acquisition results
The response A package makeover Essentially the same contents Same copy Subtle changes in format, design A/B split, 40,000 packages each
Control
Test
Which version won? The control package? The makeover?
Makeover test Package # gifts Response Avg. gift Control 325 0.81% $19.91 Test 269 0.67% $19.81
Insert Test Will a simple insert boost acquisition results? Credit: John Genette - Black Mountain Communications, for Mama s Kitchen
Is this insert necessary? New client; new acquisition package Included a mission-related insert Strong results, but... Insert adds $78/M Next mailing: 15,659 control (with insert) 15,657 test (no insert)
Which version won? With the insert? Without it?
Insert test Package # gifts Response Net/new Control 192 1.23% ($9) No insert 201 1.28% ($2)
New-package test Credit: Susie Fink DeCarlo - Adams Hussey & Associates, for NARAL Pro-Choice America
Control Who decides, the govt. or women? #10 white standard window OSE 4-page letter, petition reply, small insert Message is institutional, focus on mission 50,000-name nth select
Control
Test #10 personalized OSE with large back window showing letter and clinic ad 4-page letter Focus on fake clinics the deceptive practice of luring pregnant women into clinics for support and then forcing them to read anti-choice materials 50,000-name nth select
Test
Which version won? The control: govt vs. women? The test: fake clinics?
New-package test Package Response Avg. gift Cost/donor Control 0.57% $22.62 $35.49 Test 0.20% $28.77 $218.06
2) Testing, testing, 1, 2, 3
Why should you test? Everybody tells you so? It s really cool? You don t have anything better to do?
Testing is... What makes direct response unique The way to gain incremental improvements A straight path to frustration Not within reach for most small mailers
Types of package tests A/B split 3-, 4-, or 5-way split Multivariate testing
Technical stuff Test results require interpretation! All is not what it seems. The confidence level is key. Cheat-sheet: www.malwarwick.com
3) Six guidelines for successful testing Ignore them at your peril!
(1) Select statistically valid test panels Compare apples to apples Compare against a control ZIP code order, not alphabetical Nth name select
(2) Mail to all panels on the same day from the same place Can t compare summer and winter Can t compare one mailing with another Variations in seasonality, days of the week, postal efficiency, current events
(3) 100+ responses per panel Not 10% or 20% of a list! No fixed quantity Depends on response rate Statistical validity requires either: Large numbers of responses, or Huge differences in results
(4) Don t t trust other mailers results Question authority! Don t believe everything you hear. Study others work, but verify. Cast around for ideas but test!
(5) Above all else, test lists! Package testing is fun but secondary. The list is the most important element. Constantly test new lists and retest!
(6) Retest if confidence level is below 95% 80% confidence is meaningless. 90% confidence is suggestive. 95% confidence gives confidence. Then a rollout will probably work.
My 5 testing priorities 1) New packages 2) Telephone follow-up 3) Offer tests: premiums 4) Offer tests: involvement devices 5) Ask amounts... but ALWAYS test lists!
About list testing... If membership numbers are key, then % If net revenue is key, then Acquisition Cost If Long-Term Value is key, then compare Acquisition Cost to Long-Term Value
Your turn now! For more information, contact: Mal Warwick Associates 2550 Ninth Street, Suite 103 Berkeley, CA 94710-2516 1-(510)-843-8888 info@malwarwick.com www.malwarwick.com Copyright 2007 by Mal Warwick