Read Me! Click Me! Innovations in Invitation Design for Today s Digital World. Ted Saunders. Alexa Kessler

Transcription

1 Read Me! Click Me! Innovations in Invitation Design for Today s Digital World Ted Saunders Manager - Digital Solutions MaritzCX Alexa Kessler Analyst - Quality, Loyalty & Brand Research Market Intelligence Nissan North America, Inc. February 11, 2015

2 Agenda 1 Background 2 Responsive Design 3 In-Invite Question Design 4 Study Design 5 Results and Conclusions 2

3 Background

4 Mobile Phone Usage Continues to Grow Across Web Studies Trend of MaritzCX Web Survey Starts by Device Type 4

5 Many Companies Have Established Mobile Survey Design Best Practices Programming Question Presentation Survey Length 5

6 There Is A Gap Between Our Mobile Survey Start Metrics & Industry Metrics 53% of total opens occurred on a mobile phone or tablet in Q This is an increase from the 48% percent seen in Q % of consumers now read their exclusively on mobile devices. -Experian Quarterly benchmark report (Q3 2014) -Yesmail Compass: The mobile Effect (2013) By 2018 time, 80% of worldwide users are expected to access their accounts via a mobile device. -The Radicati Group Statistics Report, Source: Litmus Analytics 6

7 Ignoring Mobile Users Can Be a Costly Mistake Source: BlueHornet 2013 Consumer Views of Marketing Report 7 7

8 Design Has Evolved Desktop-centric Mobile Aware Responsive 8

9 Responsive Design

10 Responsive Design This Some approach things that uses may media be different queries in to the determine mobile template screen size versus of device the desktop viewing template are: If media query is supported by application and a small screen (max width ~600px) is detected a mobile-specific format is rendered Font size, colors Amount of text and placement of call to action Images used / use of images Single column of text (versus two or more columns in some PC designs) 10 10

11 Marketers Have Tested Responsive Design Career Builder Tested responsive design on a marketing message; changes included Streamlined design from thee columns to one Increased font Moved call to action up in the These changes resulted in a 15%-17% increase in open rate and a 21%- 25% increase in click rates 11 11

12 Quick Start In-Invite Question

13 New Invitation Options: In-Invitation Question Currently referred to as Quick Start Involves inserting a simple question into the body of the invite Important that the question require little cognitive effort and have few response options (ideally two) This approach is already being used on several studies with positive results 13

14 Original Invitation New Quick Start Invitation 14

15 Study Design

16 Study Design 2x2 design to test for effectiveness of responsive design and inquestion surveys starts Test used over 10,000 invitations randomly assigned to each of the four cells (over 2,500 invitations per cell) Cell One (control) Standard Invitation Format Standard Survey Start Cell Three Standard Invitation Format In-Invite Question Cell Two Responsive Invitation Design Standard Survey Start Cell Four Responsive Invitation Design In-Invite Question Analysis to compare click rates, unsubscribes, undeliverables, data comparability and data quality measures across cells 16

17 Survey Background Nissan defector study Mobile-aware invitation design Mobile-optimized survey Quick Start Option required the first survey question to be re-written 17

18 Cell One (control) Standard Invitation Format Standard Survey Start 18

19 Cell Two Responsive Invitation Design Standard Survey Start 19

20 Outlook Threw Us a Curveball 20

21 Work-Around for Outlook 21

22 Cell Three Standard Invitation Format In-Invite Question 22

23 Cell Four Responsive Invitation Design In-Invite Question 23

24 Test Results

25 Click Rates (Unique Clicks / Initial Invitations Sent) * 30% 25% 20% Click Performance Relative To Control Standard Start / Responsive Design In- Question/Standard In-Question / Format Responsive Design Desktop -18% +37% -4% Mobile +29% +163% +138% OVERALL +6% +100% +67% 17.2% 15% 15.6% 10% 6.5% 8.4% 5% 6.6% 5.5% 9.1% 6.4% 0% Control Standard Start / Responsive Design In-Question/Standard Format In-Question / Responsive Design Mobile Click Rate Desktop Click Rate * Through three weeks of field; study is still open 25

26 Overall Survey Response Rate (Completed Surveys / Initial Invitations Sent) * 12% 10% 8% 6% Performance Relative To Control Standard Start / Responsive Design In- Question/Standard In-Question / Format Responsive Design Desktop -12% +25% -1% Mobile +64% +125% +135% OVERALL +19% +65% +54% 5.4% 5.6% 4% 2.4% 3.9% 2% 3.5% 3.1% 4.4% 3.5% 0% Control Standard Start / Responsive Design Survey Response Rate - Mobile Completes In-Question/Standard Format In-Question / Responsive Design Survey Response Rate - Desktop Completes * Through three weeks of field; study is still open 26

27 Unsubscribe / Undeliverable Summary 5% Unsubscribes By Blast 4% 3% 2% 1% 0% Control Standard Start / Responsive Design In-Question/Standard Format Initial Invitation Reminder #1 Reminder #2 In-Question / Responsive Design We also looked at undeliverables and saw no significant differences across the test cells 27

28 Data Comparability Only one question consistent across all survey versions No significant differences at 95% confidence level when compared mean scores to the overall satisfaction questions across all four tests Looked at consistency of answers from the invitation to the first question in the survey Only one response (less than 0.2% of completes) differed from the invitation to the survey 28

29 Data Quality Metrics Open End Response Rate and Mean Survey Length 94% 92% 90% 88% 86% 84% 82% 80% % Control Standard Start / Responsive Design In-Question/Standard Format In-Question / Responsive Design 6.6 Open-end Response Rate Mean Survey Length (minutes) 29

30 Conclusion / Next Steps

31 Responsive Design Conclusions Next Steps We tested limited aspects of responsive design (font and text order) and experienced significant improvement in click rates and survey completes Much of the bump was tied to less text being presented in the message We continue to refine our approach and plan to continue testing Mobile respondents who started the survey through a responsive invitation were significantly more likely to complete the survey Standard Start / Responsive Design In- Question/Standard Format In-Question / Responsive Design Control Mobile Completes / Clicks 36.8% 46.7% 31.4% 36.2% Desktop Completes / Clicks 53.2% 57.0% 48.5% 54.9% Responsive design involves significant labor (at least initially) and significant testing is required Budgeting for additional cost and time is recommended when implementing a responsive design 31

32 In-Invite Question Conclusions Next Steps In-Invite questions leading to double the number of clicks and 65% more survey completes that the control group Much larger impact on mobile but improvements to mobile and desktop Mobile respondents who started the survey through the in-invite question were less likely to complete the survey, responded to open ends at a lower rate and spent less time taking that survey This leads us to believe that there may be a curiosity factor around this design and we question if the increases we are seeing are sustainable Plan to monitor click and survey response rates on programs using this technique Currently recommending this approach for projects that need a nearterm bump in completes but caution against baking initial survey response rates from this technique into future analysis plans 32

33 APPENDIX

34 Useful Information for Designing & Testing Invitations 35

35 Client Categorizations Mobile Webmail Desktop Andriod 4.x (Native client) AOL Mail (Chrome) Apple Mail AOL Mail (Android browser) AOL Mail (Firefox) Lotus Notes 8.5 AOL Mail (iphone browser) AOL Mail (IE) Outlook BlackBerry OS7 Gmail (Chrome) Outlook 2007/2010/2013 BlackBerry Z10 Gmail (Firefox) Outlook 2011 (Outlook for Mac) Gmail (Andriod browser) Gmail (IE) Thunderbird Gmail (iphone browser) Outlok.com (Chrome) Gmail app (Android) Outlook.com (Firefox) Gmail app (ios) Outlook.com (IE) ios 6.x (Native client) Yahoo! Mail (Chrome) Mailbox (ios) Yahoo! Mail (Firefox) Outlook.com (Android browser) Yahoo! Mail (IE) Outlook.com (iphone browser) Windows Mobile 7.5 Windows Phone 8 Yahoo! App (Andriod) Yahoo! App (ios) Yahoo! Mail (Andriod browser) Yahoo! Mail (iphone browser) 36