Big Data Router for Real-Time Analytics

Transcription

1

2 Real- &me Analy&cs How it Started

3 Ba:lefield 3 Player Sta&s&cs EA Collected 50TB/day Available Player Stats sites: h?p://ba?lelog.ba?lefield.com h?p://bf3stats.com Features per gun/vehicle/class leader boards etc. Geo- leader boards introduced when Ba?lefield 4 was released November Lacks interesong analysis!

4 Harvested Player Data from bf3stats.com Roughly 2 million player records Each player record has 1076 fields EffecOvely a spread sheet with 2 billion cells Details: Each player record has a field country. Each player record has fields for all assault rifles: AK- 74, M416, M16, AEK- 971, F2000, FAMAS, AUG- A3, KH- 2002, AN- 94, G3A3, SCAR- L, L85A2

5 Ques&on For each country & assault rifle: What percent of players have each assault rifle as favorite assault rifle? Bf3stats (MongoDB): >1h BioCAM RAW: 37 milliseconds 7,00 6,00 5,00 4,00 3,00 2,00 1,00 0,00 Log10(milliseconds) 6,56 1,57 Favorite Assault Rifle bf3stats (MongoDB) BioCAM RAW

6 Extract from the Analysis country_name AK- 74 M416 M16 AEK- 971 F2000 FAMAS AUG A3 KH 2002 AN- 94 G3A3 SCAR- L L85A2 Sweden 12,31% 20,98% 27,32% 19,13% 7,43% 3,65% 2,26% 1,87% 1,20% 2,11% 0,39% 1,34% United States 11,19% 23,68% 25,80% 16,53% 8,05% 4,26% 2,63% 1,71% 1,45% 2,26% 0,62% 1,83% Russian FederaOon 22,95% 12,96% 22,35% 26,44% 6,09% 1,85% 1,85% 1,76% 1,57% 1,18% 0,35% 0,66% France 11,72% 17,02% 33,34% 14,88% 8,79% 6,71% 2,15% 1,79% 0,90% 1,34% 0,35% 1,01% United Kingdom 13,34% 21,40% 26,52% 16,34% 7,68% 4,03% 2,45% 1,65% 1,05% 1,72% 0,43% 3,40% Conclusion: Player have a preference for weapons used by their country s armed forces!

7 Conclusion Sufficient reporong speed to handle high velocity data flows Fast enough to perform analysis in real- Ome on- the- fly BioCAM Web Service

8 BioCAM Web Service HTTP/JSON Core BioCAM AnalyOcs Engine Duda Web Services Framework (h?p://duda.io) Monkey Web Server (h?p://monkey- project.com) HTTP(S)/JSON Web Service Interface Create mulople BioCAM instances with different schemes Arbitrarily deep break downs for various kinds of analysis Each break down serves mulople aggregates Drill- downs naovely supported from the Web Service API Duda Monkey BioCAM

9 RTDS (Real- Time Data Storage) NoSQL graph database to persistently store generic interconnected objects in an applicaoon Linked directly into the applicaoon to store its state Designed for telecom requirements 24/7 always low latency (no maintenance windows!), 1+1 mirroring, fast switchover and failover, upgrades in runome Side- effect: low overhead and energy efficient HTTP/JSON Duda Monkey BioCAM RTDS

10 Real- Time Data Storage (RTDS) HTTP/JSON Persistent NoSQL graph database Stores generic interconnected objects in an applicaoon Linked directly into the applicaoon to store its state Low overhead Energy efficient Duda Monkey BioCAM RTDS

11 Real- Time Data Storage cont. HTTP/JSON Designed for telecom requirements 24/7 always low latency No maintenance windows 1+1 mirroring Fast switchover and failover Upgrades in runome Duda Monkey BioCAM RTDS

12 RTDS Internal Workings HTTP/JSON Data is stored as a transacoon log Proven method, provides atomic transacoons, audit history and correctly ordered updates in hot standby instance Robust in crash scenarios (corrupoon in end of log only) Self- rotaong transacoon log No checkpoinong (as it introduces latency and peaks in CPU/RAM resources) Background object traversal of all objects, writes latest state to log, when complete log is rotated ~1% of CPU, no latency peaks, no resource peaks, only last two logs required for restoring complete state Duda Monkey BioCAM RTDS

13 Real- Time Data Storage cont. Default operaoon: asynch without locks Lock- free algorithms to get and commit transacoon buffers Background threads for log flushing and mirroring Avoids latency and priority inversions Locks will be engaged in overload situaoons Overhead: one RAM copy per object For background traversal, verify state consistency etc HTTP/JSON Duda Monkey BioCAM RTDS

14 Three companies, one binary! Monkey Sooware Company Duda Monkey Oricane AB BioCAM Xarepo AB RTDS

15 BioCAM Internal Representa&on Records consists of value fields and class fields Value fields are typically numbers (price, quanoty, temperature etc.) Three types of class fields Explicit: color, brand, country etc. Implicit: Omestamp falling within hour, week, month etc. SyntheSc: favourite assault rifle Class field values are mapped to unsigned integers Master key built by packing class fields into a large unsigned integer Class field 4 Class field 1 Class field 5 Class field 2 Class field 3

16 Breakdown MulO- branch tree structure Each level corresponds to a unique class field Not all class fields need to be present Branches corresponds to class field values The branches (field values) traversed from root to leaf is called a path Records matching a path are recorded in the corresponding leaf

17 Breakdown Construc&on For each record a handle is created Each handle contain a reference to the record and a slave key The slave key is an integer representaoon of path where field values from higher levels are stored in more significant bits Array of handles is sorted by increasing slave keys Implicit tree structure is built bo?om up from the sorted array ComputaOonal complexity dominated by sorong!

18 Aggregates Zero or more aggregates are associated with each breakdown Aggregate values are associated with breakdown nodes and leaves Aggregate funcsons are associated with breakdown levels Leaf aggregate values are computed from value fields in the records using the leaf aggregate funcoon Node aggregate values are computed from childrens aggregate values using the node aggregate funcion Typically only one value field in records is considered Typically aggregate funcoons are idenocal between levels

19 Example Country: Sweden (S), Finland (F), Denmark (D), Norway (N) Brand: Audi (A), Ford (F), Volvo (V) Color: White (W), Red (R), Blue (B) Breakdown: Brand, Color, Country Aggregate: Sales

20 Example Brand A F V Color B R W B R W B R W Country D F N S D F N S D F N S D F N S D F N S D F N S D F N S D F N S D F N S Audi White Finland

21 Tradi&onal Analy&cs in Retail E- receipts sent to Data Warehouse 2. Analysis of new and historical data 3. Infrequent reports (once per week etc.) 3 Data not relevant to what s happening now involved in the analysis

22 Real- &me On- the- fly Analy&cs in Retail 2 1 BioCAM Web Service 3 1. E- receipts sent to Data Warehouse 2. E- receipts intercepted/sent in real- Ome to BioCAM WS 3. Analysis performed on- the- fly 4. ReporOng in real- Ome 4 Real- Ome monitoring, analysis and reporong with minimum stress on the data warewouse

23 Whatever Mart, Inc. The Mul& Tera Dollar Retail Corpora&on stores distributed across the globe open unique products when taking size, color etc. into account Customer purchases an average of 30 random products in each open store every second At peak rate customers purchase products per second thus surpassing USD per second net sales E- receipts are reported immediately to BioCAM Web Service Five different analyses are performed every ten seconds Reports are presented on a dashboard and updated in real- Ome

24 Whatever Mart, Inc. The Mul& Tera Dollar Retail Corpora&on Almost 1000 billion transacoons since launch whatever.oricane.com

25 Benchmarks ConfiguraOons: Web Service Access via Web Service front- end Direct access Test program linked with BioCAM, access via C API Stripped Direct access to BioCAM stripped from RTDS Four different data bases sizes (number of records) Six different transacoons loads (records updates per second)

26 Aggregate Value Re- calcula&on Time record transacoons per second Re- calculaoon speed not dependent on transacoons/second Measured in milliseconds Web Service Direct Access Stripped

27 Transac&on Time Web Service Direct Access Stripped Load (x/s) Time (us) Load (x/s) Time (us) Load (x/s) Time (us)

28 Direct Access

29 Stripped

30 Conclusion Aggregate value re- calculaoon cost linear in data base size is expected since the opomized re- calculaoon scheme is not yet implemented TransacOon cost completely dominated by Web Service front- end especially at higher load Would be interesong to bi- pass the web server and run JSON over IP TransacOon cost for Direct Access and stripped decreases with higher load most likely due to reduced context switching and higher cache locality

31 Key Applica&on Area: Gaming Counter Strike Global Offensive (CSGO) Real- Ome StaOsOcs Site to be launched Currently players on- line simultaneously Player base grows exponenoally Partnership with World #1 CSGO team Ninjas in Pyjamas ( Image source: h?p:// explains- how- csgo- became- the- second- most- played- game- on- steam/

32 Key Applica&on Area: Energy Oricane is involved in Cloudberry Datacenters (h?p:// datacenters.com) Focus is on energy savings in data centers - discussions are slow Oricane want to address: Energy producoon Energy trading Embedded applicaoons Looking for a fast paced key partner with lots of data to process Pilot project - value creaoon from ultra high analyocs performance