Measuring the effectiveness of 3 rd -party demographic data Background One of the great benefits of digital advertising is that it provides advertisers with the ability to precisely target on an impression by impression basis. Using a variety of data sources, it is possible to serve ads only to those who are identified as falling within an advertiser s target audience. One challenge this creates, however, is in measuring the accuracy of these data segments and determining the best one for a particular audience. Using audience composition to inform buying decisions has been done for decades, but new ways of buying require a new approach to evaluating target audience composition. Tools like comscore provide planning data to inform buyers interested in understanding how many impressions purchased on a given site will reach their target audience. For example, the US audience composition as reported by comscore is shown below for the demographic M18-34: Composition unique visitors M18 34 Total visitors M18-34 ESPN 23.7% 7.9MM Yahoo 15.2% 22.9MM This data can help a media planner in assigning value to impressions on each site, forecast reach, and develop a media plan. When it comes to 3 rd -party data, planning tools are not as robust, but we can measure performance of live campaigns and use that data to inform future efforts. Setting up a test To help understand the composition of 3 rd -party data segments against standard demographics, we set up a test using eleven 3 rd -party data providers and common demographic segments. Where available, high confidence or measurement-optimized data segments were tested alongside standard audience segments. These high-confidence segments were expected to deliver greater accuracy but a smaller total audience size. The following demographic segments were used in the test for all eleven providers, separated out between the US and UK: Male 18-34 Male 18-49 Male 25-54 Female 18-34 Female 18-49 Female 25-54 1
% Lift over Control The test campaign went live on November 3,, with all providers tracked by Nielsen Online Campaign Ratings (OCR). US data providers were also tracked with comscore validated Campaign Essentials (vce). Both providers measure composition of impressions delivered against a target audience. A control campaign ran in both geographic regions with no data targeting in order to serve as a benchmark. Each segment was deactivated after exceeding 100,000 impressions. Reading the results Results were analyzed for all 6 data segments in each geographic region. Depending on the segment, the control campaigns delivered between 19% and 32% of impressions against the target demographic. Using the control as a baseline, Mediasmith evaluated the percentage lift over control that each provider delivered. Looking at results aggregated across all 6 segments, some providers significantly out-performed, while some delivered little to no lift compared to the control group. While the figures reported by comscore vce and Nielsen OCR vary somewhat, they are directionally similar, indicating the overall rankings are accurate. OCR US Lift over Control (aggregate) 183% vce 142% 118% 109% 116% 9 83% 98% 79% 92% 49% 34% 48% 36% 52% 42% 6 51% 4 34% 1 6% 5% 11% 2% 15% 3% -3% 2
% Lift over Control UK Lift over Control (aggregate OCR) 113% OCR 83% 85% 58% 19% 26% 33% 29% 9% 6% 8% 4% 2% 3
Looking at delivery by gender and region, there are significant differences in lift. For example: Vendor C appears strong in the US, especially when targeting women. In the UK, however, vendor C under-performs. Vendor B, tested for two available confidence levels, delivered lower performance in the UK for the high confidence segment than for the standard targeting segment. US Lift over Control by Gender (OCR) Average Male US Average Female US 25 20 15 10 5 UK Lift over Control by Gender (OCR) Average Male UK Average Female UK 25 20 15 10 5 4
Lift varies significantly depending on the targeted demographic. Below, the broadest age group of females (18-49) generally showed the least uplift, underscoring the benefit of 3 rd -party data in targeting smaller audience segments. 25 20 15 10 5-5 US Lift over Control by Age (OCR) Female 18-34 Female 18-49 Female 25-54 5
Measuring accuracy vs. scale The test was conducted using a small amount of impressions and Mediasmith expected the test campaign to deliver quickly. Most of the providers in the US hit the 100,000 impression cap within a few days. In the UK, however, some segments failed to meet our target minimum of 85,000 impressions over multiple weeks. The chart below shows the number of days it took each provider to deliver 85k impressions against a particular segment. Those with no data were unable to deliver the desired impressions over 25 days. Days to deliver in UK: Male 18-49 K (Max Comp) K J 2 2 I H (OCR Optimized) H G F E D C B (High Confidence) B A 3 4 7 15 Days to reach 85k Impressions 24 Conclusions When measured with independent tools from comscore and Nielsen, not all data providers deliver the same accuracy For brands seeking to identify the uplift of campaigns in-market, Mediasmith recommends conducting audience measurement tests at least occasionally Some data providers will also have insight into audience composition by segment if they conduct similar tests Gender and age are some of the most basic segments; if targeting narrowly defined audiences, expect audience compositions to decrease compared to broader demographic targeting It is important to consider scale alongside accuracy; ask data providers to share total size of cookie pools when considering specific target audience groups 6
Disclosure Quantcast (shown in this report as Vendor K) paid for the media used in this report. The campaign was set up and managed by Mediasmith staff. Data segments used were purchased off-the-shelf through a major DSP, except for segments from Quantcast, which are not available directly through the DSP and were deployed specifically for the purpose of this test. Appendix: Methodology details The campaign was set up with a $1 CPM bid for media, with a frequency cap of 1 per month. Brand-safe s were applied for all segments: one list was applied to US efforts and one to UK tests. Campaign changes were made to all live segments at specific intervals in an effort to fully deliver. The following actions were taking on the listed dates to see if easing restrictions as time progressed allowed any faster delivery of segments: Nov. 3, Nov. 4, Nov. 7, Nov. 11, Nov. 17, Nov. 19, Nov. 24, Nov. 26, - Test goes live - CPM $1 - Brand Safe - CPM $1 - Brand Safe - CPM $5 - Brand Safe - Remove some providers exhibiting slow delivery - CPM $1 3/month - CPM $0.50 1/Day - CPM $0.50 10/day - CPM $0.50 - Stop - Remove Brand Safe 7