A/B Testing, Behavior-Based Campaigns, Data Reporting & Analysis. Have Fun!

Transcription

1 A/B Testing, Behavior-Based Campaigns, Data Reporting & Analysis Have Fun!

2 Don t Forget: Segment and personalize communications prior to A/B testing.

3 What is an A/B test: A/B Testing Method of market testing. Send two versions of your communication to two test audiences. Track which version performs best.

4 What is an A/B test: A/B Testing A classic direct marketing tactic. The most simple form of testing. Also called split testing or bucket testing. Used to improve response or conversion rates.

5 A/B Testing How Much Better are A/B Tested Campaigns? In a comparison of the marketing stats of A/B testers against the average marketing stats across all customers. Average open rate across all lists: 21.7% Average for winning A/B groups: 24.1% Average click rate across all the lists: 4.7% Average for winning A/B groups: 5.5% *MailChimp analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (only included s sent to over 500 recipients).

6 A/B Testing What is the A/B test process: Send each to at least 1000 subscribers. A small audience (500 or less), may not be statistically significant. Best to wait at least 24 hours before deciding winner.

7 A/B Testing What is the A/B test process: Design for a target segment. Save the original and then change one item. Send original to 50% of target and the alternative to the other 50%. Review the results. Keep testing, reviewing and refining.

8 A/B Testing The Informz Association Marketing Benchmark Report: Key findings from analyzing 600 million s sent by over 700 large and mid-sized associations. Day of the week sent had little effect. s sent in the morning had the highest open rates at 39.2% while mid-afternoon had the highest click rates at 21.0%. Desktop software usage continued to decline, while mobile usage increased to 17%. In this study, iphone was the leader in the mobile device category at 12% while ipad reported at 4% and Android at 3%. Survey-related s had the highest combined open and click rates with 36.60% average opens and 33.39% clicks. Keep in mind: This is just one of many studies and includes data from the United States, Australia, New Zealand, Canada and the United Kingdom. Use only as a guideline.

9 A/B Testing Executing a Simple A/B Test SELECT SEGMENT OR TARGET LIST SEND 10% OF TARGET 1 SEND 10% OF TARGET 2 SELECT SEGMENT OR TARGET LIST WAIT WINNER SELECTED CAMPAIGN CONTINUES REMAINING 80% OF SELECT SEGMENT OR TARGET LIST RECEIVE WINNING Initial 10 and 10 test. Remaining 80% to receive winner. OR SEND 50% OF TARGET 1 WAIT CAMPAIGN COMPLETED WINNER SELECTED SEND 50% OF TARGET 2 Full 50/50 split. The technology automates the campaign and selects the winning communication.

10 A/B Testing Why do A/B subject line testing: Most people don t read all their s. Many determine which s to open by subject lines. Testing subject lines is a small component of marketing strategy, and also one of the more important.

11 A/B Testing A/B testing to increase open rates: Open rates are significantly influenced by sender s and subject line. Segmentation and relevance also influences open rates. Entice and engage with clear sales propositions. Unclear or confusing headlines could lower current and future open rates. Best to include event name, type and price.

12 A/B Testing A/B testing to increase click rates: Click through rates are significantly influenced by body copy and effective communication of offer. Segmentation and relevance also influences click through rates. Many variables within the body can be tested to assess what s most effective in driving customers to click through and purchase.

13 A/B Testing What items can be tested with an A/B test: Subject lines Calls to action Wording size, color and placement Headline or event description Testimonials s length Layout or style of Product pricing and promotional offers Images and graphics Sign-up button color, style and size Timing and frequency

14 A/B Testing What can be assessed with an A/B test: Does delivery time of day affect overall click rate. Which day of the week, or time of day, obtains the highest open rates? What wording or phrases work best for promotional campaigns (act now, 24hrs only, etc.) What subject line style works best? Hard or soft sell?

15 A/B Testing SELECT SEGMENT OR TARGET LIST SEND 10% OF TARGET SUBJECT LINE A SEND 10% OF TARGET SUBJECT LINE B Executing a Simple A/B Subject Line Test WAIT WINNER SELECTED REMAINING 80% OF SELECT SEGMENT OR TARGET LIST RECEIVE WINNING Pre-automated set-it and forget-it technology.

16 A/B Testing WYSIWYG / PLUG & PLAY TECHNOLOGY Technology flow of an A/B test.

17 A/B Testing SEGMENT 1: SEGMENT 2: SPECIFIC SEGMENT OR PREFERENCE 10% SUBJECT LINE A 10% SUBJECT LINE B WAIT GENERAL OR EVERYONE ELSE 10% SUBJECT LINE A 10% SUBJECT LINE B Executing a Combination Segmentation Assessment and A/B Subject Line Test REMAINING 80% OF SEGMENT 1 TO RECEIVE WINNING WINNER SELECTED REMAINING 80% OF SEGMENT 2 TO RECEIVE WINNING Assess how unique changes affect specific targeted segments compared to other more general lists and populations.

18 A/B Testing Consideration for creating subject lines: Some providers have character limits. AOL has a limit of 60 characters and some forms of Yahoo and Hotmail only allow just 47 characters. Many fans will also view the communications on smartphones, which could limit initially viewable characters to less than 30.

19 A/B Testing Creating subject line test options: Item to be A/B Tested A Subject Line Example Character Count B Subject Line Example Character Count Inclusion of ticket price versus no $46 Tickets: Earth, Wind & Fire 53 Earth, Wind & Fire "Guiding Lights 40 ticket price "Guiding Lights Tour" Tour" Includes event date versus no June 16 th : Earth, Wind & Fire 51 Earth, Wind & Fire "Guiding Lights 40 event date "Guiding Lights Tour" Tour" Deal/Price at front versus deal/price at end $46 Tickets: Earth, Wind & Fire "Guiding Lights Tour" 53 Earth, Wind & Fire $46 Tickets: "Guiding Lights Tour" 53 Inclusion of only main event versus inclusion of additional acts (or show) Event type listed versus no event type listed Call to action versus no call to action Day versus no specific day Earth, Wind & Fire "Guiding Lights Tour" Concert: Earth, Wind & Fire "Guiding Lights Tour" Buy Tickets for Earth, Wind & Fire "Guiding Lights Tour" This Saturday 8PM: Earth, Wind & Fire "Guiding Lights Tour" 40 Earth, Wind & Fire And Earth, Wind & Fire "Guiding Lights Tour" 56 Earth, Wind & Fire "Guiding Lights Tour" 59 Earth, Wind & Fire "Guiding Lights Tour" Use of specific phrases or words Reminder: Earth, Wind & Fire 50 Earth, Wind & Fire "Guiding Lights 40 "Guiding Lights Tour" Tour" Note: This list in not inclusive of all the different possible subject line tests. Organizations should try different creative options to assess what works best with their fan base. Text initially seen by smartphone users

20 Data Reporting and Analysis Understanding the data: Opens and clicks Total versus unique How opens and clicks relate to purchases Calculating increased effectiveness and ROI

21 Data Reporting and Analysis Calculating Increased Effectiveness and ROI: Q. As CMO or senior marketing executive in your organization, how important are the following factors in helping you determine and communicate the value of marketing programs? YES 32% NO 68% Marketers with a method for quantifying marketing ROI are in the minority.

22 Data Reporting and Analysis Calculating Increased Effectiveness and ROI: Return - Investment (or cost) Divided by Investment Basic ROI calculations INVESTMENT RETURN PROFIT $ 1, $ 4, $ 3, to % $ $ $ % $ $ $ to 1 100%

23 A/B Testing Data Reporting and Analysis Calculating Increased Effectiveness and ROI: Examples of performance differences of A/B tests ITEM A B DIFFERENCE Opens 11.2% 13.3% 18.75% Clicks 2.1% 3.2% 52.38%

24 Examples and returns: A/B Testing A Subject line: International Irish Band Celtic Woman Pre-sale B Subject line: Celtic Woman Pre-sale Distinct opens 6.76% Distinct opens 8.00% Distinct clicks 0.67% Distinct clicks 0.73% A, which included the text International Irish Band : Generated 173% more ticket sales. Produced 197% more revenue. B had higher open and click rates: 18.34% greater opens and 10.08% more clicks. This did not translate to higher ticket sales.

25 Examples and returns: A Subject line: The Professional Bull Riders Are Back - Advance Price Saver Tickets Start at $15* A/B Testing B Subject line: The Professional Bull Riders Are Back - Get Your Tickets In Advance And Save Distinct opens 38.57% Distinct opens 35.03% Distinct clicks 11.43% Distinct clicks 7.54% Estimation of forwards: 1.57% Estimation of forwards: 0.22% For the segmented past purchasers, the inclusion of price in A: Generated 300% more ticket sales Produced 240% more revenue Performed 10% better in distinct opens (35.03% to 38.57%) Achieved 51% higher distinct clicks (7.54% to 11.43%) Led to significantly higher forwarding (0.22% to 1.57%)

26 A/B Testing Keep in mind: Testing doesn t have to be limited. You can also test multiple other elements. For example, to assess best day for most opens: 1 would send to 1,000 random on Monday 2 would send to 1,000 random on Tuesday 3 would send to 1,000 random on Wednesday 4 would send to 1,000 random on Thursday Results could enable you to align campaigns with when YOUR fans open rates are the highest.

27 Keep in mind: A/B Testing There are no hard and fast rules. The rules and technologies are always changing. What worked in the past, may not work in the future (and vice versa). Continued A/B testing is necessary to see what works.

28 A/B Testing Don t forget: Keep a running list. Sometimes data will be inconclusive. Look at results tests to identify elements that consistently tend to produce the greatest improvements. A/B tests must reach an audience of sufficient size. Customers are increasingly reading s on various web-enabled devices.

29 A/B Testing Additional resources:

30 A/B Testing Discussion What was effective for you? Any interesting insights? Copy and Content Considerations Data Reporting and Analysis

31 Behavior-Based Campaigns A Highly Effective Retargeting Tactic Don t Forget: Segment and personalize all communications in behavior-based campaigns.

32 Behavior-Based Campaigns Example flow of basic behavior-based campaign.

33 Behavior-Based Campaigns WYSIWYG / PLUG & PLAY TECHNOLOGY Technology flow of basic behavior-based campaign.

34 Behavior-Based Campaigns Provided fans multiple opportunities to purchase Automated communications based on fan s response Personalized with link to accounts

35 Behavior-Based Campaigns EVENT ADDT'L TICKET SALES FROM 2 ADDT'L REVENUE FROM 2 ADDT'L TICKET SALES FROM 3 ADDT'L REVENUE FROM 3 ADDT'L TICKET SALES FROM CAMPAIGN ADDT'L REVENUE ROM CAMPAIGN Monster Truck 350% 299% 94% 113% 775% 747% Manheim Steamroller 182% 97% 5% 9% 195% 114% Winter Meltdown 42% 38% 35% 34% 92% 85%

36 Behavior-Based Campaigns Data Reporting and Analysis Calculating Increased Effectiveness and ROI: Examples of behavior-based campaign performances Item 1 2a 2a % increase 2b 2b % increase 2a+b total 2 % increase 1+2 totals % increase 2+3 total 2+3 % increase revenue % % % % % tickets % 48 74% % % % Item % increase 1+2 total % increase 2+3 total 2+3 % increase revenue % % % tickets % % 97 89%

37 Behavior-Based Campaigns Examples and returns: EVENT ADDT'L TICKET SALES FROM 2A ADDT'L REVENUE FROM 2A ADDT'L TICKET SALES FROM 2B ADDT'L REVENUE FROM 2B ADDT'L TICKET SALES FROM 2A & 2B ADDT'L REVENUE FROM 2A & 2B ADDT'L TICKET SALES FROM CAMPAIGN ADDT'L REVENUE FROM CAMPAIGN Monster Jam 16% 21% 21% 21% 36% 42% 41% 58% Rodney Carrington 270% 110% 87% 87% 357% 197% 370% 210% The Monster Jam campaign added 5% in incremental ticket sales and an additional 16% in revenue for other events that were advertised.

38 Behavior-Based Campaigns Examples and returns: EVENT ADDT'L TICKET SALES FROM 2A & 2B ADDT'L REVENUE FROM 2A & 2B ADDT'L TICKET SALES FROM 3 ADDT'L REVENUE FROM 3 ADDT'L TICKET SALES FROM CAMPAIGN ADDT'L REVENUE ROM CAMPAIGN Sesame Street 53% 93% 22% 14% 87% 120% Paul Brandt 53% 47% 7% 8% 64% 59% The Paul Brandt behavior-based campaign added 31 incremental ticket sales and an additional $1,242 in revenue for other events that were advertised.

39 Behavior-Based Campaigns Simple campaigns: All targeted receive similar communications. All targeted receive same frequency. Re-targeting based open or click in communications. More targeted campaigns: Different targets receive tailored communications. Different targets receive different frequencies. Re-targeting based on prior purchases, known data, or open and click behavior.

40 Behavior-Based Campaigns Segment and Retarget: Example Strategy for WWE Event Wrestle-Maniacs Attended 4-6 WWE events Target fans with copy that highlights their love for the event Send 3x notice regardless of open and click activity Families Previously attended family shows Target families by detailing how WWE is great for kids Focus on the sport and activity, and stay away from mention of violence Send behavior-based campaign: Up to 3 s sent according to interest and open activity Sports Fans Previously attended sports event Target sports fans by detailing WWE as an athletic event Send behavior-based campaign: Up to 3 s sent according to interest and open activity Don t Forget: Exclude all known ticketholders from segments.

41 Copy and Content Considerations

42 Copy and Content Considerations Best Practices Review all communications to ensure: has clear call-to-action, above the fold. Includes an offer or incentive, if possible. Incorporates a sense of urgency by including dates and deadlines. Final copy was read out loud to review tone and proofreading.

43 Copy and Content Considerations Best Practices Communications should also be reviewed for the following: Subject lines kept short. Written in a conversational tone. Written for the scanner. Only short paragraphs five lines or less for . Key message is very high in the , and focused on customer benefits.

44 Copy and Content Considerations Example Template 1 st of a behaviorbased campaign.

45 Copy and Content Considerations Best Practices Communications should also be reviewed for the following: Important benefits placed in hot spots. Kept it simple - average person reads at an elementary school level. Used personalization fields. Told reader what you wanted him or her to do and how to do it, for example, Reserve your seats click here! Used the word you twice as often as I or we. If it s important, said it more than once, but stated it differently.

46 Copy and Content Considerations Example Template 3 rd of a behaviorbased campaign.

47 Copy and Content Considerations Best Practices Communications should also be reviewed for the following: Kept your copy to a minimum. Duplicated important links. Used press or testimonials to add credibility. Included a guarantee if possible. Ensured all links and calls-to-action are also text-based. copy was read out loud again yes a 2 nd time!

48 Final Discussion Behavior-Based Campaigns Target and frequency differences Event and fan type Copy and Content Considerations Data Reporting and Analysis

49 Thank you for your time! Bernie Turner Marketing Manager and Digital Strategist FanOne Digital Fan Marketing