Real Time Analy:cs for Big Data Lessons Learned from Facebook

Transcription

1 SINGLE PLATFORM. COMPLETE SCALABILITY. Real Time Analy:cs for Big Data Lessons Learned from Head of Product GigaSpaces

2 About Me MTBK Junky A Proud Dad Technology addict Head of GigaSpaces

3 Real Time Analy:cs Use Cases Ecommerce Auc=on monitoring, addwards Search engines Real- =me Marke=ng Improving conversion rate Weather repor=ng Traffic analysis Call Center Management Supply- Chain Op=miza=on Quality Management in Manufacturing SLA Monitoring and Maintenance Global Shipment & Delivery Monitoring Fraud Detec=on in Financial Companies

4 TwiJer Counting How many request/day? What s the average latency? How many signups, sms, tweets? Correlating Desktop vs Mobile user? What devices fail at the same time? What features get user hooked? Research Duplicate detection Sentiment analysis Patterns and trends

5 Note the Time dimension Counting Real time (msec/sec) Correlating Near real time(min/hours) Research Batch (Days..)

6 The data resolu:on & processing models Counting Mostly Event Driven High resolution every tweet counts Correlating Ad-hoc queries Mid resolution - Aggregated counters Research Pre generated reports Cross grain resolution trends,..

7 Tradi:onal analy:cs applica:ons Scale- up Database Use tradi=onal SQL database Use stored procedure for event driven reports Use flash memory disks to reduce disk I/O Use read only replica to scale- out read queries Limita=ons Doesn t scale on write Extremely expensive (HW + SW) Copyright 2011 Gigaspaces Ltd. All Rights Reserved 7

8 CEP Complex Event Processing Process the data as it comes Maintain a window of the data in- memory Pros: Extremely low- latency Rela=vely low- cost Cons Hard to scale (Mostly limited to scale- up) Not agile - Queries must be pre- generated Fairly complex Copyright 2011 Gigaspaces Ltd. All Rights Reserved 8

9 In Memory Data Grid Distributed in- memory database Scale out Pros Scale on write/read Fits to event driven (CEP style), ad- hoc query model Cons - Cost of memory vs disk - Memory capacity is limited Copyright 2011 Gigaspaces Ltd. All Rights Reserved 9

10 NoSQL Use distributed database Hbase, Cassandra, MongoDB Pros Scale on write/read Elas=c Cons Read latency Consistency tradeoffs are hard Maturity fairly young technology Copyright 2011 Gigaspaces Ltd. All Rights Reserved 10

12 Hadoop Map/Reduce Reality check.. With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes. [I]t will never be true real-time.. (Yahoo CTO Raymie Stata) Hadoop/Hive..Not realtime. Many dependencies. Lots of points of failure. Complicated system. Not dependable enough to hit realtime goals ( Alex Himel, Engineering Manager at Facebook.) "MapReduce and other batch-processing systems cannot process small updates individually as they rely on creating large batches for efficiency, (Google senior director of engineering Eisar Lipkovitz) Copyright 2011 Gigaspaces Ltd. All Rights Reserved 12

15 Goals Show why plugins are valuable What value is your business deriving from it? Make the data more ac=onable Help users take ac=on to make their content more valuable. How many people see a plugin, how many people take ac=on on it, and how many are converted to traffic back on your site. Make the data more =mely Went from a 48- hour turn around to 30 seconds. Mul=ple points of failure were removed to make this goal. Handle massive load 20 billion events per day (200,000 events per second) Copyright 2011 Gigaspaces Ltd. All Rights Reserved 15

18 The solu:on.. Real Time Long Term FACEBOOK Scribe HDFS 10,000 write/sec per server Hbase Log FACEBOOK PTail Puma Log FACEBOOK Batch 1.5 Sec Log

19 Checking the assump:ons.. Memory is still core The NoSQL space is very dynamic.. (We) write extremely lean log lines. The more compact the log lines the more can be stored in memory.. (We) batch for 1.5 seconds on average. Would like to batch longer but they have so many URLs that they run out of memory when creating a hashtable When Facebook engineers started the project 6 months ago, Cassandra did not have distributed counters which is now committed in trunk.. (Eric Hauser Senior Software Engineer at ExactTarget) Copyright 2011 Gigaspaces Ltd. All Rights Reserved 19

20 Facebook Analy:cs.Next.. What if.. We can rely on memory as a reliable store? We can t decide on a particular NoSQL database? We need to package the solution as a product? Copyright 2011 Gigaspaces Ltd. All Rights Reserved 20

21 Step 1: Use memory.. We rely on memory anyway to get 10k msg/ sec.. Why not use memory to store the events Reliability is achieved through redundancy and replica=on FACEBOOK FACEBOOK FACEBOOK Events Memory Grid Data Grid Data Grid Data Grid Copyright 2011 Gigaspaces Ltd. All Rights Reserved 21

22 Step 1: Use memory.. We rely on memory anyway to get 10k msg/ sec.. Why not use memory to store the events Reliability is achieved through redundancy and replica=on FACEBOOK FACEBOOK FACEBOOK Events Any API Data Grid Copyright 2011 Gigaspaces Ltd. All Rights Reserved 22

23 Step 2 Collocate Pulng the code together with the data. FACEBOOK Events Processing Grid Data Grid FACEBOOK Data Grid FACEBOOK Data Grid

24 Step 2 Collocate Pulng the code together with the data. FACEBOOK Events Processing Grid Data public class SimpleListener { Data unprocesseddata() { Data template = new Data(); template.setprocessed(false); return template; } public Data eventlistener(data event) { //process Data here } Data Grid Data Grid

25 Step 3 Write behind to SQL/NoSQL FACEBOOK Events Processing Grid Open Long Term persistency Data Grid Write Behind MySQL FACEBOOK Data Grid Data Source Adaptor HBase FACEBOOK Cassandra Data Grid

26 Economic Data Scaling Combine memory and disk Memory is x10, x100 lower than disk for high data access rate (Stanford research) Disk is lower at cost for high capacity lower access rate. Solu=on: Memory - short- term data, Disk - long term. data Only ~16G required to store the log in memory ( 500b messages at 10k/h ) at a cost of ~32$ month per server. High Memory Memory Cores Clock speed Dell Price TB (~960GB)/ Month 192GB 12 cores 3.2 GHhz $367/month $1.9/GB 5x Blades = $1835/month Copyright 2011 Gigaspaces Ltd. All Rights Reserved 26

27 Economic Opera:ons Scaling Automa=on - reduce opera=onal cost Elas=c Scaling reduce over provisioning cost Cloud portability (JClouds) choose the right cloud for the job Cloud burs=ng scavenge extra capacity when needed Copyright 2011 Gigaspaces Ltd. All Rights Reserved 27

28 Pu_ng it all together Event Sources Write behind - In Memory Data Grid - RT Processing Grid Light Event Processing Map-reduce Event driven Execute code with data Transactional Secured Elastic Analytic Application Generate Patterns NoSQL DB Low cost storage Write/Read scalability Dynamic scaling Raw Data and aggregated Data 28

29 Pu_ng it all together Event Sources Analytic Application Write behind - In Memory Data Grid - RT Processing Grid Light Event Processing Map-reduce Event driven Execute code with data Transactional Secured Script script = new StaticScritpt( groovy, println Elastic hi; return 0 ) NoSQL DB Low cost storage Query q = em.createnativequery( execute Write/Read? ); scalability Generate Patterns q.setparamter(1, Dynamic script); scaling Raw Data and Integer result = aggregated Data query.getsingleresult(); 29

30 5x bejer performance per server! Event injector Up to 128 threads GigaSpaces/ (Other Msg Server) App Services Up to 128 threads Hardware Linux HP DL380 G6 servers - each has: 2 Intel quad- core Xeon X5560 processors (2.8 Ghz Nehalem) 32 Gb RAM (4GB per core) 6 * 146 Gb 15K RPM SAS disks 60,000 Red Hat ,000 50,000 write/sec per server 40,000 30,000 20,000 GS Giga WLS Other 10,000 0 Event injection throughput Event injection throughput with write multiple EJB/Remoting service invocation throughput

31 Pu_ng it all together Elas:c Big Data Plaborm The best of both worlds Support Real Time and Batch Fully managed stack Makes the development and deployment of Big Data applica=on significantly simpler Extremely cost effec=ve Best ra=o of Disk + Memory Run on any cloud TRUE Cloud burs=ng support Copyright 2011 Gigaspaces Ltd. All Rights Reserved 31

32 Other benefits Designed for real time event processing Built-in Pub/Sub Built-in CEP Open Standard Query Any database Reliable Transactional, consistent Survive complete database failure Simple Can be packaged into a single product Fully automated deployment End to end management and monitoring Copyright 2010 Gigaspaces Ltd. All Rights Reserved 32

33 Further reading.. natishalom.typepad.com Real Time Analytics for Big Data: An Alternative Approach GigaOM Big data in real time is no fantasy Highscalability.com Facebook's New Realtime Analytics System: HBase To Process 20 Billion Events Per Day GigaSpaces.com Copyright 2011 Gigaspaces Ltd. All Rights Reserved 33

34 THANK hjp://blog.gigaspaces.com 34

35 Economic Scaling Cloudify Application Cluster Console Controller er Cloudify Agent Cloudify Agent JClouds Cloud Driver Worker VM Instance Role VM Instance Scale-in Scale-out Load Balancer Storage Network Compute Services Copyright 2011 Gigaspaces Ltd. All Rights Reserved 35

36