Building Real-Time Analytics Into Big Data Applications

Transcription

1 Building Real-Time Analytics Into Big Data Applications Shawn Gandhi, Solutions 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

2 { } "payerid": "Joe", "productcode": "AmazonS3", "clientproductcode": "AmazonS3", "usagetype": "Bandwidth", "operation": "PUT", "value": "22490", "timestamp": " " Metering Record user-identifier frank [10/Oct/2000:13:55: ] "GET /apache_pb.gif HTTP/1.0" Common Log Entry <165> T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"][examplepriority@32473 class="high"] Syslog Entry SeattlePublicWater/Kinesis/123/Realtime MQTT Record <R,AMZN,T,G,R1> NASDAQ OMX Record

3 Data volume Generated data Available for analysis Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software Forecast and 2011 Vendor Shares

4 Abraham Wald ( )

5

6

7 Big data: best served fresh Big data Hourly server logs: were your systems misbehaving one hour ago Weekly / Monthly Bill: what you spent this billing cycle Daily customer-preferences report from your website s click stream: what deal or ad to try next time Daily fraud reports: was there fraud yesterday Real-time big data Amazon CloudWatch metrics: what went wrong now Real-time spending alerts/caps: prevent overspending now Real-time analysis: what to offer the current customer now Real-time detection: block fraudulent use now

8 Talk outline A tour of Amazon Kinesis concepts in the context of a Twitter Trends service Implementing the Twitter Trends service Amazon Kinesis in the broader context of a big data ecosystem Q&A

9 Sending and reading data from Amazon Kinesis streams Sending Reading HTTP Post Get* APIs AWS SDK LOG4J Kinesis Client Library + Connector Library Apache Storm Flume Fluentd Amazon Elastic MapReduce

10 A Tour of Amazon Kinesis Concepts in the Context of a Twitter Trends Service

11 Demo

12 twitter-trends.com website twitter-trends.com Elastic Beanstalk twitter-trends.com

13 Too big to handle on one box twitter-trends.com

14 The solution: streaming map/reduce My top-10 twitter-trends.com My top-10 Global top-10 My top-10

15 Core concepts Data record Stream Partition key twitter-trends.com My top-10 My top-10 Global top-10 My top-10 Data record Shard Worker Shard: Sequence number

16 How this relates to Amazon Kinesis twitter-trends.com Kinesis Kinesis application

17 Core concepts recapped Data record ~ tweet Stream ~ all tweets (the Twitter Firehose) Partition key ~ Twitter topic (every tweet record belongs to exactly one of these) Shard ~ all the data records belonging to a set of Twitter topics that will get grouped together Sequence number ~ each data record gets one assigned when first ingested Worker ~ processes the records of a shard in sequence number order

18 Using the Amazon Kinesis API directly twitter-trends.com K I N E S I S process(records): iterator = getsharditerator(shardid, { LATEST); while for (record (true) { in records) { [records, updatelocaltop10(record); iterator] = } getnextrecords(iterator, maxrecstoreturn); if process(records); (timetodooutput()) { } writelocaltop10toddb(); } } } while (true) { localtop10lists = scanddbtable(); updateglobaltop10list( localtop10lists); sleep(10);

19 Challenges with using the Amazon Kinesis API directly Manual creation How many of workers and assignment How per many EC2 to shards instance? EC2 instances? twitter-trends.com K I N E S I S Kinesis application

20 Using the Amazon Kinesis Client Library twitter-trends.com K I N E S I S Shard mgmt table Kinesis application

21 Elastic scale and load balancing Auto scaling Group Amazon CloudWatch twitter-trends.com K I N E S I S Shard mgmt table

22 Shard management Auto scaling Group Amazon CloudWatch Amazon SNS AWS Console twitter-trends.com K I N E S I S Shard mgmt table

23 The challenges of fault tolerance K I N E S I S X X X Shard mgmt table twitter-trends.com

24 Fault tolerance support in KCL K I N E S I S Availability Zone X1 Shard mgmt table twitter-trends.com Availability Zone 3

25 Checkpoint, replay design pattern Amazon Kinesis Host A 18X 0310 Shard ID Shard-i Local top-10 {#Obama: 10235, #Congress: #Seahawks: 9910, 9835, } Shard-i Host B Shard ID Lock Seq num Shard-i Host AB 10 0

26 Catching up from delayed processing Amazon Kinesis Host A How long until we ve caught up? Shard-i X2 Host B Shard throughput SLA: - 1 MBps in - 2 MBps out => Catch up time ~ down time Unless your application can t process 2 MBps on a single host => Provision more shards

27 Concurrent processing applications Amazon Kinesis twitter-trends.com Shard-i spot-the-revolution.org

28 Why not combine applications? Amazon Kinesis twitter-trends.com Output state and checkpoint every 10 seconds twitter-stats.com Output state and checkpoint every hour

29 Implementing twitter-trends.com

30 Creating an Amazon Kinesis stream

31 How many shards? Additional considerations: - How much processing is needed per data record/data byte? - How quickly do you need to catch up if one or more of your workers stalls or fails? You can always dynamically reshard to adjust to changing workloads

32 Twitter Trends shard processing code Class TwitterTrendsShardProcessor implements IRecordProcessor { public TwitterTrendsShardProcessor() { public void initialize(string shardid) { public void processrecords(list<record> records, IRecordProcessorCheckpointer checkpointer) { } public void shutdown(irecordprocessorcheckpointer checkpointer, ShutdownReason reason) { }

33 Twitter Trends shard processing code Class TwitterTrendsShardProcessor implements IRecordProcessor { private Map<String, AtomicInteger> hashcount = new HashMap<>(); private long tweetsprocessed = 0; public void processrecords(list<record> records, IRecordProcessorCheckpointer checkpointer) computelocaltop10(records); } } if ((tweetsprocessed++) >= 2000) { } emittodynamodb(checkpointer);

34 Twitter Trends shard processing code private void computelocaltop10(list<record> records) { for (Record r : records) { String tweet = new String(r.getData().array()); String[] words = tweet.split(" \t"); for (String word : words) { if (word.startswith("#")) { if (!hashcount.containskey(word)) hashcount.put(word, new AtomicInteger(0)); hashcount.get(word).incrementandget(); } } } private void emittodynamodb(irecordprocessorcheckpointer checkpointer) { persistmaptodynamodb(hashcount); try { checkpointer.checkpoint(); } catch (IOException KinesisClientLibDependencyException InvalidStateException ThrottlingException ShutdownException e) { // Error handling } hashcount = new HashMap<>(); tweetsprocessed = 0; }

35 Amazon Kinesis as a gateway into the big data ecosystem

36 Big data has a lifecycle Twitter trends anonymized archive Twitter trends statistics Twitter trends processing application twitter-trends.com

37 Connector framework public interface IKinesisConnectorPipeline<T> { IEmitter<T> getemitter(kinesisconnectorconfiguration configuration); IBuffer<T> getbuffer(kinesisconnectorconfiguration configuration); ITransformer<T> gettransformer( KinesisConnectorConfiguration configuration); } IFilter<T> getfilter(kinesisconnectorconfiguration configuration);

38 Transformer and Filter interfaces public interface ITransformer<T> { public T toclass(record record) throws IOException; } public byte[] fromclass(t record) throws IOException; public interface IFilter<T> { } public boolean keeprecord(t record);

39 Tweet transformer for S3 archival public class TweetTransformer implements ITransformer<TwitterRecord> { private final JsonFactory fact = public TwitterRecord toclass(record record) { try { return fact.createjsonparser( record.getdata().array()).readvalueas(twitterrecord.class); } catch (IOException e) { throw new RuntimeException(e); } }

40 Example Amazon Redshift connector Amazon Redshift K I N E S I S Amazon S3

41 Example Amazon Redshift connector v2 K I N E S I S K I N E S I S Amazon Redshift Amazon S3

42