Real Time Analytics for Big Data. NtiSh Nati

Transcription

1 Real Time Analytics for Big Data A Twitter Inspired Case Study NtiSh Nati

2 Big Data Predictions Overthe next few years we'll see the adoption of scalable frameworks and platforms for handling streaming, or near real time, analysis and processing. In the same way that Hadoop has been borne out of large scale web applications, these platforms will be driven by the needs of largescale location aware mobile, social and sensor use. Edd Dumbill, O REILLY 2

3 The Two Vs of Big Data g Velocity 3 Copyright 2011 Gigaspaces Ltd. All Rights Reserved Volume

4 We re Living in a Real Time World Social User Tracking & Engagement Homeland Security ecommerce Financial i Services Real Time Search 4

5 The Flavors of Big Data Analytics Counting Correlating Research 5

6 Twitter Counting g How many signups, How many signups, tweets, retweets for a topic? What ss the average What the average latency? Demographics Countries and cities Gender Age groups Device types 6 Copyright 2011 Gigaspaces Ltd. All Rights Reserved

7 Twitter Correlating What devices fail at the same time? What features get user hooked? What places on the globe are happening? 7

8 Twitter Research Sentiment analysis Obama is popular Trends People like to tweet after watching American Idol American Idol Spam patterns How can you tell when a user spams? 8 Copyright 2011 Gigaspaces Ltd. All Rights Reserved

9 It s All about Timing Real time (< few Seconds) Reasonably Quick (seconds minutes) Batch (hours/days) 9

10 It s All about Timing Event driven / stream processing High resolution every tweet gets counted Ad hoc querying Medium resolution (aggregations) This is what we re here to discuss Long running batch jobs (ETL, map/reduce) Low resolution (trends & patterns) 10

11 Challenge Word Count Tweets Count Word:Count Hottest topics URL mentions etc. 11

12 URL Mentions Here s One Use Case 12

13 Twitter in Numbers (March 2011) It takes a week for users to 1 billion send Tweets. Source: 13

14 Twitter in Numbers (March 2011) On average, 140 million tweets get sent every day. Source: 14

15 Twitter in Numbers (March 2011) The highest throughput to date is 6,939 tweets/sec. Source: 15

16 Twitter in Numbers (March 2011) 460,000 new accounts are created daily. Source: 16

17 Twitter in Numbers 5% of the users generate 75% of the content. Source: 17

18 Analyze the Problem (Tens of) thousands of tweets per second to process Assumption: Need to process in near real time Aggregate counters for each word A few 10s of thousands of words (or hundreds of thousands if we include URLs) System needs to linearly scale System needs to be fault tolerant 18

19 Key Elements in Real Time Big Data Analytics 19

20 Sharding (Partitioning) Tokenizer1 Filterer 1 Tokenizer2 Filterer 2 Counter Updater 1 Counter Updater 2 Tokenizer 3 Filterer 3 Counter Updater 3 Tokenizer n Filterer n Counter Updater n

21 Keep Things In Memory Facebook keeps 80% of its data in Memory (Stanford research) RAMis x faster RAM is x faster than Disk (Random seek) Disk: 5 10ms RAM: ~0.001msec

22 Use EDA (Event Driven Architecture) Raw Tokenizer Tokenized Filterer Filtered Counter 22

23 Know Your Toolset 23

24 References Learn and fork the code on github: analytics Detailed blog post bigdata analytics /gs bigdata anal tics Twitter in numbers: html Twitter Storm: storm Apache S4 24

25 25