Real-time Interactive Big Data Analysis Using In-Memory Computing

Transcription

1 Real-time Interactive Big Data Analysis Using In-Memory Computing Mike Joyce Manager So0ware Engineer, icrossing Shawn Nguyen Lead So0ware Engineer, icrossing

2 STRATEGY & PLANNING Market Research AnalyPcs Strategy & Planning PROGRAM DESIGN Media Planning & Buying + + CreaPve & Experience Design Content CreaPon & Management AUDIENCE ENGAGEMENT Search MarkePng Programs Social Media / Mobile Technology & App Development Measurement & OpPmizaPon CONNECTED MARKETING PLATFORM (TECHNOLOGY) Bid Management / Trading Desk Data Management PlaNorm (Core Audience)

3 DIGITAL AGENCY INSIDE A CONTENT EMPIRE Leveraging audience insights: 20+ brands 30+ TV networks 50+ newspapers 300+ magazines

4 Big Data - Cookies! Subscribers Visitors InternaPonal MulPple devices 300+ million unique cookies

5 DMP Audience Data A]ributes Geographic Demographic Behavioral Psychographic 11,000+ Unique Attributes

6 Cookies + Audience Attributes = Super Big Data! Male Age Sports Enthusiasts Average user 800+ attributes 90M+ Cookies Iowa High Income 72B+ Attribute User pairs ipad, iphone Drives Mini Van Foodie

7 Audiences Targeting vs Discovering Who you are targepng How do you connect with them? What describes them?

8 Data Scientists Discovering Audience A]ributes 1. Define an audience using a]ributes 2. IdenPfy all a]ributes of cookies in audience 3. Calculate highly indexing a]ributes

9 1) Define the Audience Population" 90M Cookies" Gender: Male" Age: 20-35" US > North Dakota" Audience" 300K Cookies"

10 2) Audience Attributes Attributes of" Audience" Cookies in Audience" 300K Cookies" Interest: Sports Enthusiast Interest: Moose HunPng Intent: Auto Purchase > Truck US > North Dakota > Fargo Pet Supplies > Dog Food

11 3) Index the Attributes Attributes of" Cookies in Audience" Audience PopulaDon A3ribute Interest: Frequency Sports Enthusiast Frequency Interest: Sports Enthusiast 24% 27% Interest: Moose HunPng Interest: Moose HunPng 40% 6% Intent: Auto Purchase > Truck Intent: Auto Purchase > Truck 17% 4% US > North Dakota > Fargo US > North Dakota > Fargo 30% 2% Pet Supplies > Dog Food Pet Supplies > Dog Food 6% 9%

12 Data Scientists Development Ask 1. Make it accessible to normals 2. Exportable visualizapons & calculapons 3. Reduce query Pme from 1 hr to 1 sec

13 Why is this Hard? 90M+ Cookies Male Age Sports Enthusiasts Iowa High Income Average user 800+ attributes 72B+ Attribute User pairs Algorithm 1. Check every cookie if it sapsfies audience criteria 2. Collect all a]ributes for every audience cookie 3. Calculate percentages & index Within 1 sec!!!!!! ipad, iphone Drives Mini Van Foodie

14 The Answer Audience Discovery Tool Audience discovery Cookie Attributes Frequency vs Population Built for non-technical users Strategy Sales / Account Anyone Flexible Research tool In-meeting, iterative discovery Approachable Real-time Results in seconds Simple, elegant interface Multiple export formats Making science accessible

15 Data Processing R& D

16 Traditional Relational Databases Long load Pme Complex queries resulpng in long query Pmes Rigid data model

17 Non Traditional Databases Lack of complex query feature Large memory footprint requirement AggregaPon query exceeded by many 10x of seconds

18 The Low Hanging Fruit In memory cache Customizable query using Java code RelaPvely low data loading Pme

19 The Vertical Problem

20 Distributed Computing Ecosystem Not producpon ready Data import fails without explanapon AggregaPon fails to impress

21 Back to Basics Pure Java code solupon Data and logic must exists in same memory space Capable of advanced filtering

22 Distributed compupng, low overhead Data locality Minimal code migrapon

23 The Distributed Solution

24 The Challenges Tedious manual data distribupon Gar building and deployment issues Development challenges

25 What We Learned Indexed data requiring minor calculapons - - databases (relaponal & nosql) great Large non- indexed data - - the data & processing need to live in the same (memory) space