IntelliTraffic - Your key to getting your data back and the insights you need to be successful

Transcription

1 Unlocking (not provided) IntelliTraffic - Your key to getting your data back and the insights you need to be successful Author: Laurence O Analytics SEO 1 Page

2 Contents 1.0 Executive Summary A Short History of (Not Provided) Why You Need Your Keyword Data How To Get This Data Back Yourself Our (Not Provided) Solution Benefits of an Integrated (Not Provided) Solution Possible Shortcomings In Our Approach What You Can Now See In Analytics SEO Summary Annex A Comparison of Competitive Solutions Page

3 1.0 Executive Summary Thank-you for taking the time to download and read this ebook. We care passionately about supporting our customers to grow their online businesses and the recent launch of our IntelliTraffic algorithm is an important step forward for our clients in the battle for better insights and higher search engine rankings. The culmination of all the changes Google (and others) have made to make users search queries secure since 2011 have effectively made keyword visit and conversion reporting in Google Analytics and other web analytics packages a waste of time. Our recent research of web analytics traffic data from over 4,000 websites shows that nearly 9 out of 10 queries from Google now show up as (not provided) in our clients web analytics data. This is a significant problem for search marketing and content marketing professionals everywhere. With so much data missing, how on earth do you plan your SEO or Content Marketing Strategy, report on the effectiveness of the content you are writing, the website changes you are making or the relationships and links you are building? Without a good understanding of how users are finding your website and what terms they are using, you are left with little to no idea of whether you are attracting the right audience or not. You are left in the dark and have no alternative other than to hope that the work you have done has taken you in the right direction. We have been tracking the growth in (not provided) since it first reared its ugly head in Q When Google announced that it was moving to secure search as the default position for all users in Q we saw (not provided) rates accelerate sharply. It was at this point that we accelerated our own efforts to implement a solution that recovered this data for marketers everywhere. We recently announced the launch of our IntelliTraffic algorithm as an integrated part of the Analytics SEO platform. It uses our unique proprietary database of millions of keywords, consumer search modelling and behavioural 3 Page

4 analysis and your Google Analytics and Google Webmaster Account data to forensically piece back together your missing data on a daily basis. Unlike some competitors, we are not making spurious claims that it is 100% accurate or that it reclaims 100% of your missing keyword data. Nobody (except maybe Google) has the real data to compare their estimates and projections against. But we do believe our solution offers much more than most of our competitors (we ve even compared their solutions to ours in Annex A and we d encourage you to do the same) as our algorithm picks the best combination of data on a daily basis, and if you integrate Google Webmaster Tools data then we can pull this into our platform and store this data for you. SEOs and content marketers can now see whether the new pages or websites they are building are attracting the right audience they are no longer left in the dark. Google turned off the lights on keyword data for marketers; we ve just switched them on again! IntelliTraffic is available from May 2014 to all current customers (charges may apply) and will be available for all new customers from June 2014 (any new customers joining in May 2014 will join a priority waiting list). Whether you are a customer or not, I hope you find our research into the Not Provided issue and our advice and checklists a useful tool when considering how you are going to out manoeuvre the competition in 2014 and beyond. I hope you find this ebook helpful, if you do please share it liberally. All feedback, positive or otherwise is welcome. Your comments are always read and replied to. Best wishes for even faster online growth in Laurence 4 Page

5 2.0 A Short History of (Not Provided) This has been covered extensively on industry websites and blogs but here s a quick summary of what s lead to this position and our latest Not Provided research study. 1. October, 2011 Google announced support for secure search (SSL) for those searchers who are logged into their Google accounts, meaning that this keyword visit data was encrypted and began to show in Google Analytics accounts as (not provided), instead of showing visits data for each organic keyword. 2. October 2011-September 2013 Google Analytics users (and SEOs) began to see a steady increase in the level of inaccessible keyword data in their Google Analytics accounts, as more and more data began to show as (not provided). 3. September 2013 Google switched secure search on by default, meaning all searches conducted through Google now show in your browser bar as (not and the keyword data in Google Analytics really began to dry up even quicker than before. 4. April Following complaints that Google was still providing keyword visit data to their PPC customers, Google announced that it would restrict this data too. We have been anonymously tracking the (not provided) percentage across thousands of international websites which have connected their web analytics data to our platform since its inception in October It was clear from day 1 that Google s estimates of single-digit not provided rates were way off. 5 Page

6 We have continued tracking (not provided) rates and our latest research in April 2014 shows that on average 84% of keyword visits are hidden. Whilst our research also shows that a very small number of small sites have low levels of (not provided), as you can see from the distribution on the graph, the overwhelming volume of sites have significant proportions of hidden keyword referrer data. 6 Page

7 3.0 Why You Need Your Keyword Data One of the sites in our research study had 30 day organic traffic of almost 5mn visits, 82% of which was showing up as (not provided). Not knowing which keywords are generating 3.3m visits each month is a pretty big headache to have! The bottom line is you need your keyword data back if you want to compete online and here s why. If you are using Owned and Earned media as part of your marketing mix then you need to know that your messages are reaching the right audience and are producing the intended results. Without a basic understanding of how well you are doing and how well your competitors are doing you cannot underpin your marketing strategy with a solid foundation of data that you can use to benchmark the results of your marketing efforts against. Here are just a few example real-life scenarios that illustrate how not having this valuable keyword data undermines your ability to market effectively. The Humble SEO Whether you are analysing a market, a new website or a bunch of competitors sites you need to understand which keywords represent the biggest opportunity for you to target and you need to have keyword visit data to understand whether the site you are analysing is already receiving visits from these primary keywords or not. To put together a well-considered and coherent SEO strategy you need to understand the market opportunity, the competitive gap and the resources at your disposal in the time frame you have to work on the site. Keyword ranks and keyword visit estimates are fundamental ingredients throughout this whole process and are extremely useful indicators of success and failure that clients comprehend. The Enthusiastic Content Marketer Let s say you have decided to build a new website or a new section of your site. Keyword trends and opportunity analysis will be an essential element in the planning process and you cannot work out whether this content is attracting the right audience and has been indexed and ranked relevantly by the search engines without an understanding 7 Page

8 of how, where and with which keywords potential customers are finding your site. The Connected Link Builder/PR Let s say you ve been given the glamorous task of pushing this lovely new content out to journalists and bloggers with a view to getting free (and for some paid) coverage and links. Surely, you want to be able to prove what a good job you have done to the person paying your invoice or your wages? Will the number and quality of links built really cut it? We have a lovely Link Referral module which integrates Google Analytics, Majestic SEO s backlink database and our link building tools to show you which sites you ve built relationships with are generating visits and sales. But it s only half the story. Having an understanding of whether this activity has contributed to overall site growth and in what areas at a keyword group or keyword level is invaluable. The Major brand/ecommerce site If online is crucial to your growth then you can utilise this data in areas such as; budget allocation (PPC vs Organic keyword performance); measuring the ROI you are getting from your in-house SEO and content marketing campaigns; measuring the work of your agency to determine whether you are getting value from your monthly spend with them; brand vs non-brand visits so you can understand whether your offline and online marketing efforts are having the desired impact or not. The Hard-working Agency keyword visit data is an essential element in; planning your client s online marketing strategy; competitive analysis; opportunity identification; winning pitches (you can give your clients access to data and insights that other agencies cannot), refining your campaigns and activities to give you the planned results and client retention (proving that you are delivering value). Until we search without using keywords, keyword level data will always be in demand by clients. Yes, page level data is much more important than ever and tracking which pages are performing well and contributing to sales is essential. However, to have some substantive data and credibility behind your plans and activities you need to have keyword visit data. 8 Page

9 4.0 How To Get This Data Back Yourself There have been quite a few posts from the SEO community since October 2011, providing suggestions as to how you can either get around this obstacle or replace this data using some educated guesswork. Whilst you can do this yourself, having a platform (like ours) to automate it for you is simply a nobrainer. Option 1: Using Web Analytics data (what data you were left with, that is) In the early days of Not Provided you could take your remaining keyword visit data in Google Analytics and extrapolate that out to fill the gap left by your (not provided) percentage. E.g. If your (not provided) percentage was only hovering around, say, the 20% or 30% mark you could effectively uplift each keyword visit by the relevant percentage. Reported Keyword Visits in Google Analytics 1 Not Provided% For example, if you had 70 visits in Google Analytics for example keyword and a 30% (not provided) rate, then a simple calculation gives you an estimate of 100 visits for example keyword. 70 = 100 estimated visits 1-(0.3) However, of course it s far from exact as Not Provided rates will vary by user and query and as the (not provided) percentage has continued to grow, it has become a bigger and more dangerous assumption to make. Some SEOs will also look at the split of branded and non-branded traffic as a whole and make a similar calculation. For example, if the keyword data that you can still see is 65% branded terms and 35% non-branded terms, you could simply apply the same ratios to the number of (not provided) visits. At least this would give you some idea of the breakdown of visits, but you d still be missing that useful keyword-specific visit data. 9 Page

10 However, today with average (not provided) rates around 84% using your web analytics data on its own has become fairly useless for calculating keyword visit data. In conjunction with other data sources it can still be useful in understanding total visits and conversions at a landing page level. Option 2: Using Keyword Ranking Data + CTR assumptions + Keyword Search Volume Estimates and Trends A widely used technique by SEOs involves using search volumes and assumptions about CTR% (click through rates) for different ranking positions in the SERPs in order to estimate the amount of visits you might have had for each keyword over a set period. Large scale use of this technique dates back to August 2006 when someone leaked some AOL data (AOL was powered by Google search at the time) showing CTR% for around 20 million searches. Since then there have been a couple of updated studies by Slingshot SEO in 2011 and Catalyst in October 2013 showing estimated CTR% data at different ranking positions. The main source of search volume estimates come via Google s Adwords interface or API and SEMrush s interface and API (which we currently use). For keyword ranking data you can of course take your pick of the myriad of keyword ranking tools out there. Then all you need to do is extract all this data daily into an Excel spreadsheet and repeat this time-consuming exercise every day of your working life! Clearly, if you have the skills then you can of course use APIs to automate this for you for each website. Of course, there are some potential issues with using solely this approach: It assumes CTR%s are consistent across millions of different SERPs It assumes CTR%s are consistent across different locations and that people behave the same way in different cultures and countries It might ignore the effect of different Universal Search results on click-thru patterns It might be based on outdated CTR% data The Search Volume data might not be accurate, up-to-date or reflect trends it also takes time for very new trending terms to become available It ignores the difference you should get between branded and non-branded CTR%s as you d probably expect higher CTR%s for the very top results for 10 Page

11 branded searches, since they indicate that users are actively searching for one of your web properties It ignores the personalisation of results as there is no way of knowing to what extent the search engines will personalise the results set for certain keywords You have to do some sort of trending analysis based on past year s or month s search volume estimates to get a picture of current or forecast search volumes It requires daily keyword rankings to see fluctuations in your keyword visits and the keyword needs to rank well (most CTR% curves stop after position 20 or 30) Option 3: Google Webmaster Tools (GWT) A lot of SEOs now rely solely on the data still available in Google Webmaster Tools as this allows you to see an estimate of the number of visits each keyword has generated, together with the ranking URL (if you painstakingly click on each URL to reveal the keyword!). Google Webmaster Tools only shows you the data for the last 90 days and you cannot do comparisons over time (other than the last 30 days). They have promised to store the data for a year but that has not happened at the time of writing. (We can store the data for you on a daily basis and over time will allow you to do weekly, monthly, quarterly and even yearly comparisons). Google Webmaster Tools Top Queries and Top Pages reports allow you to download your data in.csv format. But there is one crucial element missing from this report; it does not include the ranking URL for each keyword (even though the data is visible if you click on a URL in the Google Webmaster Tools user interface). The same is true if you switch on the connection between your Google Webmaster Tools and your Google Analytics account; you cannot see the URL for each keyword. (However, we can crawl this data for you and match it against our unique database of keyword rankings (in excess of 100 million rankings per month) to re-connect your data and show you which keywords are ranking for which URLs and how many visits they are making). Google Webmaster Tools only shows you the top 2,000 or so keywords (we have seen more but it is inconsistent) for your site. Clearly this changes over time and for news sites and social media sites with fast changing content the terms may fluctuate rapidly, meaning you may have data one month for a 11 Page

12 keyword but not the next. (By combining Google Webmaster Tools data with Google Analytics data and our proprietary keyword ranking databases we can piece together the missing data and give you insights on an unlimited number of keywords). Connecting your site to GWT is essential (and not just for getting keyword data back) and if you only want to use one source of data then this is probably the best source, however, relying on this approach alone has some potential pitfalls: It is limited to approximately 2,000 keywords (but this is inconsistent) The data is only visible for the previous two months You can only see impressions and click changes over the past 30 days Google Webmaster Tools keyword ranks are not reliable due to the way Google averages the top keyword rank results Google Webmaster Tools is normally at least 2 to 5 days behind, but we ve seen plenty of examples where it s taken much longer to update you d therefore need a methodology to predict traffic using rolling averages to have an up-to-date analysis The Google Webmaster Tools API does not include all the relevant search query data although Google has launched a Python script that allows you to download the Top Pages and Top Queries.csv downloads However, the Top Pages and Top Queries.csv downloads do not keep the crucial link between the search query and the ranking URL (and in fact even if you connect your Google Webmaster Tools account and your Google Analytics account then this relationship is hidden). Google has this data but they don t want you to have it! If you don t have the programming skills then you have to go in every day and manually download this data in.csv format So Google Webmaster Tools on its own, whilst useful, will only really give you a limited view of your keyword data. And if you are a news site or popular site then you will only get data on your top 2,000 or so keywords, and these keywords will change over time leaving you in the dark about keywords that GWT previously gave you data about. Whilst GWT is great in so many ways, you simply need a more reliable and less time consuming way of finding, grouping, organising and tracking your most important keywords and groups of keywords over time. 12 Page

13 Option 4: Landing Page Focus You could, of course, as a lot of SEOs point out, just look at your most popular landing pages and work out (based on what keywords those pages are targeting and ranking for) what keywords might have generated those organic visits for the page this month. Keeping on top of this data requires very regular downloads of large Excel files or extensive use of the Google Analytics API. It also still requires separate keyword ranking in multiple search engines to marry up the ranking URLs with the correct Google Analytics data at URL level. The obvious shortcoming of this approach is the amount of work involved and it s not an exact science as ranking URLs change, and there is not a simple 1 to 1 relationship between keyword and ranking URLs; very often keywords (brand keywords especially) will generate a SERP with several different ranking URLs which will all get organic clicks. Summary of Options Each potential solution provided SEOs with some insights, but each has its potential issues. If only you could combine all these data sources and let an algorithm automatically collect and store the data and work out the best option for each site and date range.. 13 Page

14 5.0 Our (Not Provided) Solution We looked at the Pros and Cons of all the different approaches and determined that using one single data source or one approach was not going to be enough. Google Analytics used to give us this data every day and we wanted to build a solution that produced the best estimated data daily using the best data we could find without lots of time consuming manual intervention. So, we now give our users the option of using one or more data sources (or all of them which is our default and preferred position) and our IntelliTraffic algorithm works out what s best in the circumstances for each website. It s even clever enough to check landing page visit numbers so that it doesn t overestimate the numbers. We ve looked at a number of competitive Not Provided solutions (see Annex A) and believe that (whether you choose to work with us or someone else) you really must choose a solution that incorporates Google Webmaster Tools into the mix to give you the best possible outcome and estimates every day. 14 Page

15 Data Sources Frequency Purpose Alternative Google Webmaster Tools Google Analytics Daily Search Volume estimates and trends CTR% analysis Keyword Rankings & Big SEO Data Combining them all together Daily Top 2,000 keywords, impressions, clicks and CTR% Monthly Updated regularly using anonymised GWT data Daily keyword rankings and a 25mn keyword database updated weekly Residual keyword data, visits and conversion information at landing page level Monthly estimates of consumer demand by country for the previous 2 years Branded vs Nonbranded split (can be customised by site) Used for modelling keywords not present in GWT or GA and finding new keyword opportunities Automated daily Gives you an easy way of tracking keyword visits for the keywords that you care about No API. Google s Python script or login daily and download a.csv Google s API or login and download a.csv daily Adwords API or SEMrush API or login and download a.csv monthly Use a CTR model or build your own using your GWT data (bear in mind this may skew results) Any keyword ranking tool with an API or download manually every day Do it yourself (seriously it will take hours per site per month) or use another platform (see Annex A) 15 Page

16 6.0 Benefits of an Integrated (Not Provided) Solution It s (almost completely) automated - just configure it and wait for it to update It s configurable - we know SEOs are always arguing over things like CTRs It can be used for competitive research - you can put any domain in and we can utilise ranking data, CTR%s and Search Volume to give you an idea of what keywords your competitors are attracting traffic for even if you don t have Google Analytics and Google Webmaster Tools data It saves time in carrying out manual calculations and removes a large amount of guesswork We will store the data for you - Google Webmaster Tools doesn t, so you don t have to remember to download it every month We can go back in time! We can backdate your Google Analytics imports, go back and re-analyse them all and, if we have ranking data for that period (or you send us some ranking histories we can import), we can then estimate your keyword visits for each keyword over this period No more guesswork no more finger in the air assumptions let the algorithm do the work for you in helping to show what Google isn t! You can even customise our IntelliTraffic algorithm to suit your needs and use all or just one or two of the possible data sources. The more search engines you track rankings on, the better your Not Provided coverage is and the better the estimates will be. 16 Page

17 7.0 Possible Shortcomings In Our Approach (Or anyone else s for that matter!) Whilst we rank track 25 million keywords every week automatically finding long-tail keywords that your brand ranks for, keyword coverage for very large sites can still be a problem for really long-tail keywords. Either there will be no search volume estimate data available at all or they will generate only a handful of visits. Google says that 15% of searches every day are unique. That s about 500 million queries that Google has seen for the first time which makes estimation for these keywords impossible but also unnecessary. We feel this is something we can all live with, after all the largest sites have been used to having sampled data in Google Analytics for a long time. We use the 80:20 guide as a rule of thumb; if we can identify the 20% of top keywords that are driving 80% of your traffic or that should be driving your traffic then we have given you something worthwhile to focus on. You cannot optimise for a raft of keywords that might only appear once and never be seen again. Our Big SEO Data will help give you the coverage you need for the keywords that really matter and you can always add an unlimited number of keywords to track ranks for and can add more at any time during your campaign (which may incur additional charges) Seasonal keywords are harder to estimate but our algorithm does take this into account and we will be using this for traffic forecasting, budgeting and business case development soon. New keywords that have never been seen before following major catastrophes or unexpected events are of course extremely hard to predict, but we have plans to use trending social media data in the future to improve this aspect of our algorithm. Erratic or extremely volatile keywords they are also hard to predict but we can use trend line and averages to give a good indication of whether a keyword is worth targeting and whether your traffic is increasing or decreasing. Our algorithm (as I m sure do all of our competitors ) ignores the personalisation of results as there is no way of knowing to what extent the search engines will personalise the results set for certain keywords 17 Page

18 8.0 What You Can Now See In Analytics SEO The main area the IntelliTraffic algorithm (once it has been enabled for your account) makes a difference is in the Keyword Visits module. Now you can track over time individual keywords or keyword groups to see how your SEO campaign is progressing. Beyond this module, you ll also see a difference in the Monitored Keywords module where the Organic visits number for each keyword will now be based on these IntelliTraffic calculations. 18 Page

19 It s also reflected in every other area where we use keyword specific visit data (and not just top line numbers from Google Analytics); for example, the Organic Visits module will now reflect more accurate brand and non-branded visit numbers over time. 9.0 Summary The measures we have taken to try and solve the (not provided) issue for marketers is testament to the lengths we will go to in order to give our customers the data they need to make better decisions and gain a competitive edge. An integrated approach to getting your ( not provided ) data back will save you and your teams hours and hours of time every month and give you a consistent and robust methodology that can underpin your organic search strategy. Unlock your (not provided) data Automate the collection of keyword data from multiple sources and produce estimates on a daily basis Develop a better SEO and content marketing strategy Track trends in keyword traffic on a keyword by keyword basis or by keyword group Measure the effectiveness of your online marketing campaigns Benchmark and measure international organic traffic and rankings Discover which pages bring the most organic traffic Get a much better understanding of which pages and keywords to invest time in optimising Customise our algorithm to suit your needs 19 Page

20 Annex A Comparison of Competitive Solutions We recognise that you have a choice of several different solutions for the Not Provided problem. Clearly we d love you to choose to work with us, but at the very least choose a solution that uses multiple data sources including GWT data integration. Analytics SEO Conductor Search Metrics Y Y Y Integrates Google Webmaster Data Updates the data daily Y Y Bright Edge Linkdex Stores the data to be viewed over Y Y Y time (GWT only stores data for up to 2 months) Calculates the data necessary to Y fill in the missing gaps (GWT is at least 2 to 5 days delayed) Data based on individual Y Y Y territories for greater accuracy Uses other data sources to Y Y Y recover keywords that are not covered by GWT Uses data from Google Analytics Y Y Y Y Y Y Finds the best keywords with the lowest (not provided) ratio Adjusts data if (not provided) ratio is too high Estimates data if no Analytics or Webmaster Tools data is available Uses keyword rank, search volume to calculate visits Allows click-through-rate adjustments for generic keywords Allows click-through-rate adjustments for brand keywords Can be used for competitive research Can calculate historical data Gshift Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Errors and omissions excluded - This data was, as far as we can tell from access to the competing solutions or their websites, correct at the time of going to press and we will update this with feedback as it is received. However, we urge you to check and compare the differences for yourself. 20 Page