Social Media Measurement & Metrics Northrop Grumman Communications Meeting December 11, 2009 Katie Delahaye Paine CEO kdpaine@kdpaine.com www.kdpaine.com com http:/kdpaine.blogs.com Member, IPR Measurement Commission www.instituteforpr.org SNCR Fellow www.sncr.org
Why Measure? The main reason to measure objectives is not so much to reward or punish individual communications manager for success or failure as it is to learn from the research whether a program should be continued as is, revised, or dropped in favor of another approach James E. Grunig, Professor Emeritus, University of Maryland If we can put a man in orbit, why can t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires. Ralph Delahaye Paine, Publisher, Fortune Magazine, 1960 speech to the Ad Club of St. Louis 2
First, some numbers 1. Want to know how fast social media is growing: http://ow.ly/vo3t 2. In a week, the Red Cross raised $35 million for Haiti relief via texting 3. Years to Reach 50 million Users: Radio (38 Years), TV (13 Years), Internet (4 Years), ipod (3 Years) Facebook added 100 million users in less than 9 months iphone applications hit 1 billion in 9 months. 4. The fastest growing segment on Facebook is 55-65 year-old females 5. Ashton Kutcher and Ellen Degeneres have more Twitter followers than the entire populations of Ireland, Norway and Panama 6. 80% of Twitter usage is on mobile 7. Facebook USERS translated the site from English to Spanish via a Wiki in less than 4 weeks and cost Facebook $0 8. 78% of consumers trust peer recommendations, only 14% trust advertisements 9. Only 18% of traditional TV campaigns generate a positive ROI 10. 90% of people that can TiVo ads do 3
Signs that it s the end of measurement as we know it 1. 48% of respondents to a PRWeek study said they were moving $$ out of advertising budgets into Social Media. Only 18% said they were taking $$ away from PR. 2. Procter & Gamble is now paying for engagement, not eyeballs 3. Sodexo cut $300K out of its recruitment budget using Twitter 4. Immunize BC measured SM success via share of discussion, increased awareness and shots given 5. BMC Software measures communications effectiveness based on contribution to EPS 6. HSUS generated $650,000 in new donations from an on-line photo contest on Flickr 7. The Red Cross measures the effectiveness of Twitter via lives saved and property lost 8. IBM predicts the ends of advertising as we know it. Also receives more leads, sales and exposure from a $500 podcast than it does from an ad 9. 11 Mom s turned around Wal-Mart's image and delivered measureable increases in sales.
Everything you know about measurement is obsolete Old School PR 21 st Century Role of PR The definition of timely has changed The definition of reach has changed GRPs & Impressions are impossible to count (an irrelevant) in social media The definition of success has changed The answer isn t how many you ve reached, but how those you ve reached have responded Page 5
Old School Metrics AVEs Eyeballs HITS (How Idiots Track Success) Couch Potatoes # of Twitter Followers (unless you re a celebrity) # of Facebook Friends/Fans (unless they donate money) Page 6
New School Metrics Influence = The power or ability to affect someone s actions. Engagement= Some action beyond zero Advocacy = engagement driven by an agenda Sentiment = contextual expression of opinion regardless of tone ROI: Return on Investment no more no less. End of discussion
A measurement timeline MSM Online Social Media Eyeball counting HITS Engagement
The Engagement Decision Tree Awareness Consideration Preference Trial Purchase Find Observe/ Lurk Participate Engagement Purchase/Act /Link/WOM
The Five Levels of Engagement Level 1 Searchers % Unique Visitors Metrics Level 2 Lurkers % Repeat visitors > 3-5/month Level 3 Casuals % Comments, friends, followers Level 4 Actives % repeat comments, retweets, participation in threads Level 5 Defenders % Advocating, recommending, defending the brand
What ROI isn t: Reach Frequency Hits Friends Followers
First some definitions Influence = The power or ability to affect someone s actions. Engagement= Some action beyond zero Advocacy = engagement driven by an agenda Sentiment = contextual expression of opinion regardless of tone ROI: Return on Investment no more no less. End of discussion
Defining ROI 1. ROI is an economic term, not necessarily an actionable measure of success 2. Only 18% of traditional TV campaigns generate a positive ROI 3. 48% of respondents to a PRWeek study said they were moving $$ out of advertising budgets into Social Media. Only 18% said they were taking $$ away from PR. Why do we want to compare ourselves to a declining industry? 13
ROI is an equation Revenue minus cost = ROI Revenue = $$ in or $$ not spent HSUS Flickr Revenue: $650,000 Contest Cost $1000 ROI = $649,000 Sodexo Twitter costs: $30,000 Cost savings: $300,000 Net savings (ROI): $270,000
Conversation Type by Message Saturation Expressing support 40 221 Conversation Type January February March Making an observation 142 152 46 Making an observation 22 24 Expressing support 20 40 261 Offering an opinion 18 1 1 Rallying support 4 4 5 Asking a question 5 Contains no message Contains One+ Message Advertising something 10 Asking a question 1 2 6 Rallying support 4 Distributing media 4 2 0 50 100 150 200 250 300 Making a suggestion 2 3 Giving a shout-out 1 2 1 Mentions Conversation Type by Tone Calling for action 1 2 Expressing criticism 1 1 Expressing support 178 83 Disclosing personal information 1 1 Putting out a wanted ad 1 Making an observation 12 34 Expressing agreement 1 Recruiting people 1 Answering a question 1 Expressing surprise 1 Asking a question Rallying support Positive Neutral Negative Grand Total 206 213 321 0 50 100 150 200 250 300 Mentions 15
Moving conversation from observation to support
Media Engagement & Online Giving 35,152,789 OTS 6,253,852 OTS Red line indicates media impressions
Goals for Social Media 1. Marketing/leads/sales/recruitment 2. Mission/safety/civic engagement 3. Relationship/reputation/positioning To fix this Or get to this
Goals drive metrics, metrics drive results Goal Reputation/ Relationships Get the word out Sales Relationship scores % hearing Engagement Index Cost per customer acquisition Metrics Recommendations % believing Web analytics Positioning % acting Sales leads Engagement Marketing Mix Modeling 19
Goals, Actions and Metrics Goal Action Output Metric Outtake Metric Outcome Metric Recruitment Twitter campaign Number of tweets % inclined to support % improvement in perception of organization Increase in support Flickr photo contest Number of entrants % likely to support the cause % increase in online requests for information % increase traffic to recruitment site Improvement in relationship scores Message consistency Blogger/Speaker training % postings/articles containing one or more key messages Share of positioning on key issues % hearing message % believing message % consistency in messaging between external and internal communications Market share Ratio of on-message to offmessage quotes 20
The 7 steps to Social Media ROI 1. Define the R Define the expected results? 2. Define the I -- What s the investment? 3. Understand your audiences and what motivates them 4. Define the metrics (what you want to become) 5. Determine what you are benchmarking against 6. Pick a tool and undertake research 7. Analyze results and glean insight, take action, measure again
Step 1: Define the R -- What Return Do you Want? What return is expected? Define in terms of the business or mission. What problems is Social Media supposed to solve? What were you hired to do? What difference are you expected to make? If you are celebrating complete 100% success a year from now, what is different about the organization? If your Social Media is eliminated, what would be different? 22
Step 2: Define the I What is the investment? Personnel Agency compensation Senior Staff time Opportunity cost 23
Step 3: Define your audiences and how you impact them There is no audience. There are multiple constituencies How does a good relationship impact your mission? List every stakeholder Where do they go for information? What s important to them? What is the benefit of having a good relationship with that stakeholder group? What s important to them? Where do they go for information? Should you blog or Twitter? Don t ask me, ask your customers 24
Step 4: Define your Key Performance Indicators (KPIs) You become what you measure Don t measure what you can t change Don t get lost in the weeds All that matters is the KPIs 25
Step 4: Typical KPIs Cost savings Efficiency Cost per message communicated Cost per new recruit/lead/customer acquired Productivity: Increase in employee engagement/morale Lower turnover/recruitment costs Engagement: Ratio of posts to comments % of repeat visitors % of 5+min visitors % of registrations Trust: Improvement in relationship /reputation scores with customers and communities (Loyalty/Retention) Thought leadership: Share of quotes Share of opportunities Message penetration Positioning on key issues Improvement in favorable/unfavorable ratio Improvement in Optimal Content Score (OCS) 26
Step 5: Define your benchmarks Emerging benchmarks Past Performance Peer Stretch goal Whatever keeps the boss up at night 27
Consumer organizations are far more likely to see Undesirable Discussion than non-profits or educational institutions. *For each mention, we determine whether it leaves the reader more or less likely to donate to, partner with, volunteer for or otherwise support the efforts of Non-Profit Organization. If it leaves the reader more likely, we consider it positive. If it leaves the reader less likely, we consider it negative. If it doesn t sway the reader one way or the other, we consider it neutral. 28
Key Message penetration by segment Contains a Message Does not Contain a Message 27% 2% 29% 73% 71% 98% 0% 20% 40% 60% 80% 100% Consumer Education Non Profit 29
Key messages must be 140 characters or less Average Number of Words Per Key Message 2 6 10.5 Number of Messages 6.5 8 Tracked 12.75 0 5 10 15 Consumer Education Non Profit Recommendation: Less is more when trying to get your messages across. We recommend reducing the number of key messages and simplifying and shortening the key messages that are tracked. 30
Step 6: Pick a tool 1. Content Analysis 2. Survey 3. Web Analytics
Step 6: Selecting a measurement tool Objective KPI Tool Increase inquiries, web traffic, recruitment Increase awareness/preference % increase in traffic #s of clickthrus or downloads % of audience preferring your brand to the competition Web Analytics: Google Analytics, Omniture, Web trends Survey: Online -- SurveyMonkey, Zoomerang or Mail Engage marketplace Conversation index greater than.8 Web analytics or Content Analysis: Rankings TypePad, Technorati % increase in engagement Omniture, Google Analytics Communicate messages % of articles containing key messages Total opportunities to see key messages Cost per opportunity to see key messages % aware of or believing in key message Media content analysis Survey 32
Content Analysis requires a content source: Free: Google News/Google Blogs, RSS feeds, Technorati, Social Mention, Twazzup, $500+ Radian 6, Techrigy, Sysomos, Crimson Hexagon, Visible Technologies, Scout Labs, Cyberalert, CustomScoop, e-watch 33
Assessing Influence Free tools: Twitter Grader, Tweetlevel, Twitalyzer
Share of conversation in social media
Popularity of topics in Defense Industry discussions
Need #2: A way to analyze content Automated Human: Census vs. random sample Sentiment vs. Topics The 80/20 rule Measure what matters because 20% of the content influences 80% of the decisions 37
Typical analysis includes coding for: Tonality Nature of discussion What messages were communicated How you re positioned on key issues Dominance/Prominence/Visibility Authority Subject of the article/posting Who was quoted? Products, events, initiatives, battles mentioned Optimal Content Score
Why an Optimal Content Score? You decide what s important: Benchmark against peers and/or competitors Track activities against OCS over time Positive: Mentions of the brand Key messages Positioning Visibility Negative Omitted Negative tone No key message 39
How to calculate Optimal Content Optimal Content Score Quality score +1 0-1 Score Score Score Tonality Positive 3 Neutral 0 Negative -3 Positioning Contains 2 Doesn't contain Positions the competition favorably or positions Sargento 0 negatively -2 Messaging Contains 3 partially contains Does not contain or miscommunicates key 0 message (neg mess) -1 Quotes Contains 1 Does not contain -1 Competitive mention Does not mention Competition 1 Competition mentioned prominently -3 Total Score 10 0-10 Visibility Score +1 0-1 Score Score Score Brand Photo Contains 3 Doesn't contain Contains competitive 0 photo -5 Dominance Focal point 3 Not a focal point -1 Visibility Headline mention 2 Top -20% of story 0 Minor mention -2 Target publication Top Tier 2 2nd tier 0 Not on target list -2 Total Score 10 0-10
OCS quickly identifies best practices CEO to retire, Bush to replace
Analyze for what matters
Look for the bad stuff, too
Tool #2: Surveys require: A defined sample A list a way to get to that sample Agreement on what questions you need to answer A survey instrument/questionnaire A test A way to analyze data SPSS SAS 44
Aspects of relationships Control mutuality Trust Satisfaction Commitment Exchange relationship Communal relationship 45
Components of a Relationship Index Control mutuality In dealing with people like me, this organization has a tendency to throw its weight around. (Reversed) This organization really listens to what people like me have to say. Trust This organization can be relied on to keep its promises. This organization has the ability to accomplish what it says it will do. Satisfaction Generally speaking, I am pleased with the relationship this organization has established with people like me. Most people enjoy dealing with this organization. Commitment There is a long-lasting bond between this organization and people like me. Compared to other organizations, I value my relationship with this organization more Exchange relationship Even though people like me have had a relationship with this organization for a long time; it still expects something in return whenever it offers us a favor. This organization will compromise with people like me when it knows that it will gain something. This organization takes care of people who are likely to reward the organization. Communal relationship This organization is very concerned about the welfare of people like me. I think that this organization succeeds by stepping on other people. (Reversed)
How to implement relationship metrics Step 1: Conduct a benchmark relationship study Step 2: Implement PR program Step 3: Conduct a follow up relationship study Step 4: Look at what s changed
Tool #3: Web Analytics Google Analytics/Web Trends/Omniture Unique URLs Data delivered in parallel with content analysis Ability to correlate and integrate data SPSS/SAS 48
Step 7: Analysis - -Research without insight is just trivia Look for failures first Check to see what the competition is doing Then look for exceptional success Compare to last month, last quarter, 13-month average Figure out what worked and what didn t work Move resources from what isn t working to what is 49
What matters, what doesn t & what to do about it Engagement High engagement/ Low Influence High Influence/ High engagement Influence Low engagement/low Influence High influence/low engagement
For all institutions, most postings were simply making an observation or distributing media. Share of Conversation Types Share of Engagement by Conversation Type - Institutional Blogs Showing dismay 1 Showing dismay 100.0% Recruiting people 3 1 Recruiting people Rallying support 1 Rallying support Playing a game 16 1 Playing a game Offering an opinion 2 11 2 4 1 Offering an opinion 49.5% 10.8% 39.7% Making an observation 14 46 6 18 9 Making an observation 30.9% 23.1% 10.9% 35.1% Making a suggestion 15 12 3 6 8 Making a suggestion 72.7% 27.3% Giving a shout-out 5 12 2 2 Giving a shout-out Giving a heads-up Expressing surprise 29 203 1 4 13 17 Arizona State Giving a heads-up 6.5% Michigan State Expressing surprise 26.9% 66.6% Arizona State Michigan State Expressing support 1 2 2 3 Penn State Expressing support Penn State Expressing criticism 3 1 Purdue UniversityExpressing criticism 100.0% Purdue University Distributing media Disclosing personal information 36 7 24 787 4 13 235 5 University of Michigan Distributing media Disclosing personal information 53.9% 44.2% 1.6% 46.1% 38.7% 15.5% cx University of Michigan Calling for action 1 2 Calling for action Augmenting a previous post 6 1 Augmenting a previous post 100.0% Asking a question 1 2 Asking a question 100.0% Answering a question 6 7 1 Answering a question Advertising Something 3 12 2 1 1 Advertising Something Acknowledging receipt of information 2 Acknowledging receipt of information 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Page 51
The vast majority of discussion in external blogs is neutral. Share of Tone Share of Engagement by Tone - External Blogs 30 100% 29 90% 94% 25 80% 83% 20 20 70% 60% 71% 15 12 14 Negative Neutral Positive 50% 40% 58% 42% 58% 42% Negative Neutral Positive 10 30% 5 0 8 5 4 4 3 2 1 University of Michigan Purdue University Penn State Michigan State Arizona State 20% 10% 0% 29% 14% 6% 3% Arizona State Michigan State Penn State Purdue University University of Michigan Page 52
Overall Comparison of Georgia Tech Social Media Outlets Based on 2007 data, Georgia Tech outperformed its peers in Facebook presence, but significantly lagged peers on other social media. Post-2007 media monitoring has not included a social media dimension due to funding constraints, but this will be important to trend as feasible in the future. Share of All Coverage Share of Youtube 25% 20% 15% Share of External Blogs 10% Share of Social Bookmarking 5% 0% Georgia Tech Competitor Average Share of Institutional Blogs Share of Facebook 53/17 Definitions: YouTube: a video sharing site. Social Bookmarking: a site where members can display media they have found on the web. Facebook: a social networking site. Institutional Blogs: blogs hosted and owned by schools studied. External Blog: any blog post that is not hosted by an institution.
Change the conversation, improve your reputation Improve your reputation Listen first, then respond Stop doing stupid things
Negative coverage over time 25 1 20 3 1 2 15 4 2 Entries 10 10 2 14 18 21 1 12 10 15 14 1 5 10 2 5 0 9 3 4 2 4 2 7 6 5 1 4 4 2 2 2 2 1 2 2 1 1 2 1 1 2 2 2 1 2 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr 2006 2007 2008
Thank You! For more information on measurement, read my blog: http://kdpaine.blogs.com or subscribe to The Measurement Standard: www.themeasurementstandard.com For a copy of this presentation go to: http://www.kdpaine.com Follow me on Twitter: KDPaine Friend me on Facebook: Katie Paine Or call me at 1-603-868-1550