Social Media Monitoring Tool Buyer s Guide

Transcription

1 Social Media Monitoring Tool Buyer s Guide A hands-on test and performance analysis 13 September 2011 Web Liquid 11201

2 What s In Here? Introduction... 3 Who Is This Buyer s Guide For?...4 Methodology...4 Vendor Selection...4 Sales Demos...5 The Trial...5 Performance Summary... 6 Similar Feature Sets...7 Two Approaches to Data Capture...7 Data Quality...8 The Tools... 9 Alterian...9 Brandwatch MutualMind Radian Synthesio Appendix Matrix of Grading Criteria Vendor Selection Stages Definitions of Performance Summary Columns Confidential 2

3 Introduction As social media adoption continues to grow, the value in the conversations that consumers are having online is becoming ever more apparent; after all, the web grew out of a desire to be able to communicate more efficiently. Today it has blossomed into a vast digital salon where people are sharing their opinions on a variety of subjects and, thanks to the ubiquity of products and services in our everyday lives, a great deal of this conversation is in some way related to brands and companies. To a marketer, this is online Word- of- Mouth (WOM). Online Word- of- Mouth is manifest in a variety of ways for instance, the way in which people search for products and services on a search engine but the most obvious is in social media. On blogs and in forums, Twitter and Facebook, even in the opinion columns of traditional media's digital outlets, people are levying judgments about people, brands, products, services and companies - for better or worse. More often than not, they re giving reasons for why they feel the way they do. Herein lies the value of social media monitoring. Yet for all of its potential and despite the possibilities that it affords marketers, social media monitoring is a relatively new form of consumer research. However new, it is a source of insight that is becoming increasingly important to marketers: according to a recent study conducted by Web Liquid in collaboration with RSW/US, 95% of marketers view monitoring of online Word- of- Mouth as an important aspect of their marketing initiatives. (Click here to download a copy of the 2011 Social Media Monitoring Survey.) As with many new fields, there is a variety of new organizations vying to capitalize on this increasing desire to tap in to online Word- of- Mouth, making it difficult to separate the wheat from the chaff, as it were. That inspired us to try them out and share our findings. Confidential 3

4 Who Is This Buyer s Guide For? While companies have traditionally used social media for marketing, we are increasingly seeing businesses attempt to leverage the channel for functions that span the organization - human resource departments use LinkedIn to look for job candidates. Brands like JetBlue and Best Buy have become veritable poster children for how Twitter can be applied as a customer service channel. For some organizations, social media platforms have even become a critical step in the conversion process. But whether you re using platforms like Facebook and Twitter to manage your online reputation, or leveraging a blog to achieve social media PR objectives, monitoring the online conversation and mining the data made available by social media monitoring tools presents brands with an unprecedented opportunity: to build more meaningful relationships with more customers. We created this Social Media Monitoring Buyer's Guide to make the task of choosing the best social media monitoring tool a little bit easier for you, and we understand that there are a number of factors that go in to the selection process when it comes to choosing the right solution for your business. As such we refrain from telling you which tool to go with. We are confident that our experience providing clients with social media monitoring solutions since 2004 has provided us with enough insight and knowledge in to what makes a good social media monitoring tool, such that we can make credible recommendations. Methodology Vendor Selection We began our process with a list of over 40 social media monitoring tool vendors (see the appendix) some were paid tools, while others were free; some tools focused on the traditional social networks (e.g., Facebook and Twitter), while others took a broader approach by attempting to capture as much of the web as possible. In order to qualify for consideration in the 2011 Social Media Monitoring Buyer s Guide, we had two main criteria that narrowed that list down: 1) The tool must be Enterprise level. That is, no free tools. Check out our blog post on Google Alerts vs. Brandwatch to find out why. Additionally, vendors must provide tech support and/or account management teams to users. 2) Tools must have cross channel coverage. While in some instances it wasn t entirely evident how many channels the tool covered, we excluded any tools whose focus was solely on Facebook and/or Twitter. Based on these criteria and our experience, we were able to whittle our initial list down to a group of twelve, who were then invited to conduct sales demos. Confidential 4

5 Sales Demos During the sales demos, the social media monitoring tools were considered on a variety of criteria including stated features such as sentiment accuracy, sources monitored (e.g., Blogs, Social Networks, Forums, News sites, etc), metrics used to determine influence, query structure, raw data access, the degree to which data may be segmented, and a variety of other questions that have proven valuable in our experience. Each social media monitoring tool had to meet, in some form, the following criteria to be considered for a trial: ability to setup and define a query through the user interface (UI), ability to manually validate the data and an ability to export raw data. After receiving the live demos, we narrowed our list down to five finalists for our hands- on trial: Alterian, Brandwatch 1, MutualMind, Radian6, and Synthesio. The Trial In choosing a brand to monitor as the basis for this evaluation, we wanted a brand that: 1. Was sufficiently well known, to ensure a statistically significant sample volume of online conversation. 2. Had a somewhat common name, in turn requiring a sophisticated query that would allow us to fully test the capacity of each tool to process the data captured (but not overwhelm the dataset with irrelevant content). 3. Was susceptible to spam, although not so much that data management would become overwhelming. Ultimately, we chose Tesla Motors. As a darling of a niche industry (Electric Vehicles), the discussion would be active enough to demonstrate the social media monitoring tools capabilities, while still maintaining a degree of manageability. It also didn t hurt that the brand shares a name with one of history s most famous inventors, as well as a solid 90 s Hair Rock band. We tested these social media monitoring tools over a period of seven days during the week of April 18 th, Web Liquid currently uses Brandwatch as our primary monitoring tool. Confidential 5

6 Performance Summary Tools are ranked relative to each other 2, where 1 is the best and 5 is the worst. Raw Data Data Data Usability/ Tool Query Data Quality Quantity Management Interface Output Alterian Brandwatch * 2 3 MutualMind * 5 1 Radian Key Advantages Help Resources Flexible data management options Competitive Facebook Page monitoring Highly customizable dashboard Synthesio Data Quality *indicates a tie, see appendix for detailed definitions of column headers In conducting the trials for this Social Media Monitoring Buyer s Guide, there were a few things that stuck out to us above all else: Since the feature sets offered by each of these vendors are relatively uniform across the board, the real differentiator lies in each feature s functional nuances. There seem to be two distinct approaches when it comes to data capture. They all overstate their capabilities in terms of sentiment accuracy, relevancy and spam mitigation, therefore requiring human validation for reliable data accuracy. If there is one thing that you take away from this Social Media Monitoring Buyer s Guide, above anything else, it should be that none of these monitoring tools are perfect. The technologies that they employ are not nearly sophisticated enough for their analyses to be taken at face value. Human validation of the data collected by these tools should be a key component to any social media monitoring program and it appears that the vendors agree with this; indeed, the 2 The lack of an overall score for each tool is intentional. The relative importance of solid query writing vs. data quality, for instance, is better left to the reader; there are no winners and losers, only strengths and weaknesses Confidential 6

7 tools selected for evaluation made the cut partly because they allowed for manual validation of the data, from within each monitoring tool s respective UI. Similar Feature Sets Generally, the social media monitoring tools on which we conducted trials (and the ones for which we received sales demos) offered the same features. For instance, all of the tools have rather sophisticated, robust user interfaces (UI) that visualize the data nicely, and ostensibly provide at- a- glance insight into high- level trends in the conversation. Along the same lines, users of any of these tools have the capability to filter their data in a number of ways (the most standard of which being by tag and category). Varying degrees of batch functionality can be found in all of the tools trialed. Raw data access, a criterion for consideration in this trial, comes in variety of formats ranging from.csv to.xml. All of the monitoring tools used logic as their query language of choice, and gave the option to enter the query as free text or have the tool assist you in structuring a query. (MutualMind was the only tool that did not offer this option, opting instead to go with structured query entry only). Two Approaches to Data Capture The most insightful takeaway from conducting this trial was the discovery that there seem to be two approaches to data capture. Ultimately, this comes down to a question of quality vs. quantity. The first of these is geared toward crawling and capturing as much of the Internet as possible. Attempting to organize so much unstructured data and making it useful to the end user is certainly an admirable task, and one that would presumably result in a far deeper understanding of the conversation. However, in our analyses, we noticed that achieving such breadth is not without its perils as we will see later. The second methodology takes a less ambitious approach, focusing less on breadth and instead working off a white list of sites that are known to be legitimate that is, sites that are not spam farms. While the data these tools are able to collect is much cleaner, concessions are made with respect to the breadth of data captured. However they collect their data, all of these tools maintain vast databases of all the online Confidential 7

8 conversations they have captured. This gives users the ability to backdate their queries; in other words, individuals using the tool can access historical data and subject it to the same analyses as the data they collected upon initial launch of the query. All of the monitoring tools trialed provide this in some capacity, typically at an additional cost. The implications of these two different approaches, needless to say, manifest themselves in the quality and quantity of the data. Looking at the table below, we can see that Brandwatch and Radian6 (the two tools in our trial that employ the first methodology) achieved the broadest coverage, but suffered dramatically in the amount of spam they managed to capture. Conversely, Synthesio, MutualMind and Alterian all work off of a white list of sites. We can see that this approach results in a much cleaner data set, while simultaneously restricting the total number of domains crawled. Data Quality Tool Total Domains Total Mentions % Spam Mentions Twitter Mentions % Marked Neutral 3 Sentiment Accuracy Brandwatch % % 80% Alterian % % 90% Radian % % 83% MutualMind % % 77% Synthesio % % 81% The above table was created using data from each tool, inclusive of spam; from instance, the Total Mentions column would be read as such: Brandwatch captured a total of 2,851 unvalidated mentions. 3 This is the percentage of total mentions, across the unvalidated data set which were marked neutral. Confidential 8

9 The Tools There are many considerations that go in to choosing the social media monitoring tool that best suits your business s objectives, so instead of telling you which tool to use we ve summarized our experiences with each tool below, highlighting the pros and cons of each so that you can make your own decision. We also gave the vendors a chance to respond to our evaluations, which you can find at the end of each tools section along with pricing information. Alterian As one of the monitoring tools that limits itself to crawling a finite list of sites across the web, Alterian s data quality was among the best of the tools we trialed. One thing that stood out about the tool was the plethora of help resources available, many of which were available in video format. From a more technical perspective, Alterian currently offers a fairly standard set of search options, from location and language targeting to limited proximity search within the query itself. One nice aspect unique to Alterian is the explicit approach to backfilling while setting up the query, users have the option of specifying exactly how far back in time they would like to begin collecting data. Data management proved to be cumbersome in some respects. Despite giving users the ability to manually validate data, the interface requires navigation between tabs to change sentiment, delete mentions, flag items as spam, and so on. Pros Cons Raw data access (HTML, XLS, TXT, XML) Inefficient data validation Batch operations (tagging, deleting, categorize) Breadth of data available based on bucketed pricing structure Intuitive UI Myriad help resources, including video tutorials on many features Flexible query entry Relatively clean data capture Manual data validation possible Confidential 9

10 Query Detail Alterian s Statement and Pricing Alterian provides actionable insights and foresights through social media data. We provide a range of consulting, research, and training services alongside SM2, our social media our social monitoring and listening platform Since the timing of Web Liquid s initial test during the week of April 18 th, 2011, SM2 released version 5.0. In addition to several new and enhanced features, this release introduced a new information architecture which dramatically improves many aspects of SM2 including overall system performance, improved content collection and near real- time results of data. The architectural changes from this release would have certainly had a positive impact on the Data Quality results. In the next coming releases, SM2 will be focusing on enhanced data access including: new visualizations of social data, more available metrics and measures, integration with the other Alterian product lines as well key external integrations and a drastically improved user interface. Alterian did not provide us with pricing information. Confidential 10

11 Brandwatch Of all the tools tested, Brandwatch s data contained the largest share of spam - 45% of non- validated mentions. Data management is relatively easy, however due to the data quality, it does take more human resources to do so. At the domain level, users are able to create Site Groups, which can then be filtered out. This gives users the ability to cut out a substantial amount of spam from the database (a full 50% of Brandwatch domains tracked were found to be spam domains). At the mention level, batch delete and tagging options also aid in the data management process. Pros Cons alerts High percentage of spam captured Create categories and tag mentions inline UI hangs from time to time, when retrieving data Metadata is exported with raw data Not all batch operations are available across all aspects of the tools Ability to create site lists Substantial breadth of data Flexibility in query entry Batch tagging operations Easy to use, intuitive UI Stock and custom filters allow for various segmentation of the data Query Detail Confidential 11

12 Brandwatch s Statement and Pricing First, we'd like to thank Web Liquid for including us in the Social Media Monitoring Buyer s Guide. As you demonstrate, it's a very competitive marketplace and we are very proud to have made the final five tools. Being tied for best Data Management is a result we are particularly pleased with. It is an area we believe to be very important and something we've focused a lot of resources on so would expect to perform well there. We also take Data Quality very seriously and while we often receive compliments on the quality of our data compared with competitors, we recognized a few months ago that too much spam was getting into the system. For most of our users this was not a problem as there are many quick ways a user can reduce the number of low quality mentions (better query definition, using filters, our bulk- delete facility, etc). However, we wanted to eliminate the spam in the first place and so around the time this report was put together, we added a 'report spam' button for users to identify (and blacklist) spam and upgraded our spam filters. We are confident that these measures have already much improved our data quality. Here are some other developments to Brandwatch that we would like to update you on: Live now: We have recently upgraded Brandwatch to include traffic data for sites and specific engagement metrics for blogs and forums (comments, thread views etc.) Coming: Batch- categorization and batch- sentiment classification are features that are currently in development, scheduled for release in a few weeks. Coming: We will also be gaining access to the full Twitter fire hose within approximately 2 months, which will mean no tweets will be missed. Basic Plan: 400/Month 10k mentions/month Unlimited queries Upgrades Advanced plan: 1500/Month You Get: Unlimited mentions 15 queries Purchase up to 6 months of backdated data Upgrades Confidential 12

13 MutualMind MutualMind was another of the tools tested which monitors a white list of domains, with a focus on the non- traditional side of digital media - blogs, forums, social bookmarking sites and the major social networks like Facebook and Twitter. Users are also given the option of connecting their Facebook and Twitter accounts and, perhaps more interestingly, explicitly monitor the Facebook pages of competitors. This is different from engagement capabilities, which MutualMind also allows for. Instead the unique point of differentiation here is that users are able to compare their Facebook fan growth, for instance, against that of their competitors. Pros Cons Data validation is quick and efficient Highly structured query entry requirements; restrictive set up process Robust dashboard Data can only be filtered by one parameter at a time Batch operations Filtering is only inclusive API access Raw data is exported without metadata Manual data validation Raw data can only be exported by day Raw data access (XLS, CSV) Heavily focused on social media sites (Facebook, Twitter, blogs, etc); mainstream/traditional media sites not tracked Query Detail Confidential 13

14 MutualMind s Statement & Pricing Product update: Expanded monitoring coverage significantly including review sites, mainstream/traditional media have been added (as of July 1, 2011) Product update: Advanced boolean search within monitoring data allows for enhanced filtering with multiple parameters (as of August 15, 2011) Note: Blogs, forum and news posts are also ranked for influence Note: Multiple categories of tags (labels) can be applied to posts in batch Upcoming enhancement: Boolean query support (scheduled September 5, 2011) Upcoming enhancement: Data export with metadata for custom date range (scheduled September 12, 2011) Entry price is $500/campaign per month, which comes with unlimited results and user accounts. Full pricing at: and- pricing/ Confidential 14

15 Radian6 Like Brandwatch, Radian6 captured an exceptional amount of data over the week that we conducted this trial, with spam accounting for 40% of all un- validated mentions. However, unlike Brandwatch the user interface (UI) for Radian6 comes with a steeper learning curve. It s not so much that the options offered for manual validation are overwhelming, but that the verbiage used to describe the options are either disjointed or vague. Despite this, once users get a grasp on how to set up a social media monitoring dashboard for validation, the tool offers a suite of options for tagging, categorizing and flagging up spam and irrelevant mentions. All of these options come in the form of one- click operations, however the limited batch functionality of the tool adds to the amount of time that must be spent for human validation of the data. Pros Cons Exceptional breadth of data Limited batch operation functionality Modular UI creates a variety of ways to Terminology used in help sections is vague look at the data Vast array of targeting options in the query (domain, URL, Page Title) 5000 mention limit when exporting data, which the tool does not express General filtering capabilities (language, geography, URL, author, site metrics) in the UI Raw data exports to a variety of formats Variety of help resources Manual data validation Ability to categorize and tag mentions inline Query Detail Radian6 s Statement and Pricing After numerous requests, Radian6 did not provide a response to our findings, or pricing information. Confidential 15

16 Synthesio Synthesio had the highest quality data of all the social media monitoring tools that we trialed. With only 3% of all mentions identified as Spam, it s clear that once a user writes their query and the tool starts to collect data they should feel confident enough to dive right in to analysis. In writing the query Synthesio offers the option of breaking it up and assigning categories to various components of the query, such that when a piece of conversation on the internet matches, the mention is automatically categorized as such. There are also a variety of targeting options (geographic, site, etc), both within the query and outside of the query. With limited resources required for data management, one can move immediately to analysis. There does, however, seem to be a risk of missing some of the conversation due to the limited number of monitored domains. Pros Cons Exceptional data quality Metadata is missing from raw data export Intuitive interface, with a variety of options for drilling down Inability to categorize on the fly may result in lost segmenting capabilities Dashboards segment data for you in a Inability to batch tag plethora of ways You can identify influencers at a variety of levels Raw data export only comes with page and mention titles, not the actual mentions Robust ability to segment the data by a variety of parameters Manual data validation, with the ability to tag individual mentions. Custom solutions for each client Flexible query writing options Confidential 16

17 Query Detail Synthesio s Statement and Pricing Synthesio provides global social media monitoring and engagement tools combined with customized services to fine- tune our solutions for the needs of each individual client / project, as well as the option of human analysis in over 30 languages. While we provide full custom end- to- end monitoring, engagement, and analysis solutions, even our monitoring tool alone, evaluated here, features a basic layer of included custom services: Custom Sourcing: Before we begin each project, we research the brand and the market to ensure that all the most relevant sites, blogs, forums, etc. (social & mainstream media) are covered for each industry and country monitored. Clients may let us know at any time which specific sites they want to include, and which to remove, so they can focus on the data that matters most. Meanwhile, our technologies are also constantly scanning the web looking for new relevant websites for our clients. Specialty sites are covered including - YouTube, Foursquare, Orkut, Renren, Weibo and more. This results in the largest quantity of relevant and accurate data for our clients. Human Sentiment Analysis: Our network of analysts is available to provide human sentiment analysis in over 30 languages, providing industry- leading levels of accuracy, client- relevance, and value for each client far higher than any automated technology. Unlimited, Personal Service: We maintain close communication with each client with included consultation and ongoing support. Assistance in areas like setting up the optimal query, making updates on the fly, or integration with client- side platforms are always available. Confidential 17

18 Appendix Matrix of Grading Criteria Vendor Alterian Brandwatch MutualMind Synthesio Radian6 Query Set up Ease **** *** **** **** ** Flexibility **** ***** ** **** *** Data Validation Spam Batch Batch Batch Singular Noo Delete Batch Batch Batch Singular No Sentiment Singular Singular Batch Singular No Batch (cat.) Batch (tagging only) Batch Query- dependent Batch Tagging/Categories categorization; Singular tagging Data Sources News Yes Yes Yes Yes Yes Twitter Yes Limited Limited Yes Full Facebook Yes Yes Yes Yes Yes Blogs Yes Yes Yes Yes Yes Forums Yes Yes Yes Yes Yes Review Sites Yes Yes Limited Yes Yes Video Yes Yes Yes Yes Yes Photo Yes Yes Yes Yes Yes Data Set History 4 years 1 month No history 4 years No Domain Yes Yes Yes Yes Yes Sentiment Yes Yes Yes Yes Yes Language Yes Yes Yes Yes No Country Yes Yes Yes Yes No Age Yes No Yes No No Gender Yes No Yes No No Influencers Yes No Twitter only - Klout Synthesio Rank Yes Filters Tags/Categories Yes Yes Yes Only one Yes Yes Domain No Yes Yes Yes No Language Yes No No Yes No Country Yes No No Yes No Sentiment Yes Yes Yes Yes No Age No No No Yes Yes Gender No No No Yes No Influencers No No No Yes No UI Ease of Use ***** ** *** **** ** Flexibility ***** *** * *** *** Data Access Export Up to 7 days Full Daily Yes Yes API Full Limited Limited Full, at cost Full Account Support FAQ No No Yes Yes Yes Twitter Yes Yes Yes Yes Yes Live Chat No No No No No Dedicated A/C Management Yes Yes Yes Yes Yes Confidential 18

19 Vendor Selection Stages Confidential 19

20 Definitions of Performance Summary Columns Query: refers to the flexibility users are given with regards to both the writing and entering of a query into the tool; for instance, can you write your own query and enter it as free text, or are you required to build your query within the tool? Are you given an option for either? What query operators does the tool accept? Data Quality: the percentage number of mentions in a given tool s unvalidated data set that is made up of spam; also, the percentage of mentions whose sentiment was accurately marked Data Quantity: Based on the total number of mentions captured by each tool Data Management: evaluates the efficiency, ease and variety with which data may be cleaned, segmented, categorized, tagged, etc. Raw Data Output: evaluates the tool based on the variety of file formats in which data is available for download, and the metadata included in each output (full mentions, page URLs, sentiment, etc Usability/Interface: evaluates how intuitive the user interface of each tools is, relative to the efficiency with which data can be managed and analyzed within the tool Confidential 20