Case Study: Real-time Analytics With Druid. Salil Kalia, Tech Lead, TO THE NEW Digital

Transcription

1 Case Study: Real-time Analytics With Druid Salil Kalia, Tech Lead, TO THE NEW Digital

2 Agenda Understanding the use-case Ad workflow Our use case Experiments with technologies Redis Cassandra Introduction to Druid Architecture Druid in production Demo

3 Understanding the use-case

4 What Is Analytics? Processing the HISTORICAL data to: Understand potential trends Analyze the effects of certain decisions or events Evaluate the performance of a system Make better business decisions

5 What Is Real-time Analytics?

6 Understanding The Ad Workflow Web Page Request PUBLISHER SERVER USER Ad Request Ad-Content AD EXCHANGE AD AGENCY-2 AD AGENCY-1 AD AGENCY-3

7 Examples From Our Use Case How many times a video has been viewed in a particular time-span? in a particular time-span at a particular site? in a particular time-span at a particular site in a particular country? in a particular time-span at a particular site in a particular country on a particular device?

8 Let s play a video ad

9 Video Events For The Analysis LOAD START PLAYING VIEW STOP / PAUSE FINISH

10 Event Data (Sample) TIMESTAMP Ad Site Advertiser Event Action T01:01:27 Z T01:01:33 Z T01:01:40 Z T01:01:45 Z T01:01:50 Z T01:01:51 Z 123 abc.com Brand X Player Load 234 abcd.com Brand Y Player Load 123 abc.com Brand X Player Start 123 abc.com Brand X Player Playing 123 abc.com Brand Y Player Playing 123 abc.com Brand X Player Stop

11 Why Real-time Analytics? Understand the real-time performance Control the velocity Control the targeting Avoid over serving Avoid under serving

12 Recap Things We Understood Our use-case How the ad-tech works (in general) Different video player events We are expecting a huge amount of data coming at a very high velocity.

13 Experiments with technologies

14 Experience From Redis There was a huge variety of keys all over the place Not a good fit to deal with time-series (big) data Persistence is another issue we can t afford loosing data. Not a right match for our use-case

15 Conclusion From Redis Never blame Redis It was too early decision Our misunderstanding with the real use-case Thanks to Redis to help us understanding our requirements, very soon.

16 Working With Cassandra Very good support for the time-series data Extremely good for writing the data at a very high speed Very easy to scale horizontally Supports aggregations through Counters

17 Writing into Cassandra AD PLAYER ANALYTICS SERVER CASSANDRA

18 Reading from Cassandra ANALYTICS SERVER CAMPAIGN MANAGER CASSANDRA

19 What didn t work with Cassandra Inconsistent results Unreliable counters No ad-hoc queries support Nodes were crashing out very frequently

20 Crossroads What next? Third party tools on the top of Cassandra for better consistency DataStax Enterprise edition Taking a deeper dive into Cassandra to reconfigure the whole architecture and setup Switching to different technology

21 Understanding druid

22 About Druid An open-source analytics data store Supports streaming - data ingestion Flexible filters for ad-hoc queries Fast aggregations sub second queries Distributed, shared-nothing architecture Highly scalable

23 Setting Up Druid In Production KAFKA (CLUSTER) AD PLAYER ANALYTICS SERVER DRUID CLUSTER CASSANDRA

24 Druid s Reliability Check KAFKA (CLUSTER) DRUID CLUSTER AD PLAYER ANALYTICS SERVER RAW FILES Job To Test Druid s Integrity RAW FILE CONSUME R RAW FILES RAW FILES

25 A Quick Demo

26 Druid Architecture Druid Nodes External Dependencies Steaming Data REAL TIME NODES MY SQL COORDINATO R NODES ZOOKEEPE R BROKER NODES Client Queries DEEP STORAGE HISTORICA L NODES Queries Data/Segments MetaData

27 Druid Data Ingestion Druid Nodes External Dependencies Steaming Data REAL TIME NODES MY SQL COORDINATO R NODES ZOOKEEPE R BROKER NODES Client Queries DEEP STORAGE HISTORICA L NODES Queries Data/Segments MetaData

28 Druid Data Ingestion KAFKA (CLUSTER) AD PLAYER ANALYTICS SERVER DRUID Realtime Node

29 Druid Data Retrieval Druid Nodes External Dependencies Steaming Data REAL TIME NODES MY SQL COORDINATO R NODES ZOOKEEPE R BROKER NODES Client Queries DEEP STORAGE HISTORICA L NODES Queries Data/Segments MetaData

30 Druid Data Coordination Druid Nodes External Dependencies Steaming Data REAL TIME NODES MY SQL COORDINATO R NODES ZOOKEEPE R DEEP STORAGE HISTORICA L NODES Queries Data/Segments MetaData

31 COMPANIES USING DRUID

32 Questions?

33