Real-Time Analytics at Scale in the AWS Cloud

Transcription

1 Real-Time Analytics at Scale in the AWS Cloud Olivier Klein Solutions Architect, AWS 23 rd June 2015 Cloud & Big Data Analytics Summit 2015 Hong Kong 2015, Web Services, Inc. or its Affiliates. All rights reserved.

2 Three Types of Data Analytics Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart apps

3 Three Types of Data Analytics Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart apps

4 How Fast is Real-Time?

5 There s no such thing as real time. There s only near-real time. Typically when we talk about real-time, what we mean is architectures that allow you to respond to data without persisting it to a database first! John Akred CTO, Silicon Valley Data Science

6 So what is near real-time? Ability to process data as it arrives Roughly speaking, process data in the present rather than the future But what is the present? ecommerce Attention span of a potential customer Options Trader Milliseconds Guided Missile Microseconds

7 Solution: Stream Processing Stream storage which allows processing events as they come in and react accordingly

8 A high-throughput distributed messaging system.

9 What do we expect from a real-time data stream?

10 Real-Time Data Stream Expectations What do we expect from a real-time data stream? Highly Available Fully Scalable Fault Tolerant (Temporary) Durable How can we achieve this? Multiple Datacenter Facilities Auto-Scalable Server Infrastructure Global Load-Balancers etc.

11

12 AWS Global Infrastructure 11 Regions Oregon GovCloud Frankfurt Beijing Tokyo 29 Availability Zones 53 Edge Locations Northern California N. Virginia Ireland Sydney Continuous Expansion Singapore São Paulo

13 Web Services Applications Platform Services Core Services Compute Storage Database Networking Security Infrastructure Virtual Desktops Analytics App Deployment Mobile Access Control Availability Zones Collaboration & Sharing Auditing Monitoring Encryption Regions App Delivery Edge Locations A P I & S D K s

14 Web Services Applications Platform Services Core Services Security Infrastructure Virtual Desktops Analytics App Deployment Mobile Compute Storage Database Networking Access Control Availability Zones Collaboration & Sharing Auditing Monitoring Encryption Regions App Delivery Edge Locations A P I & S D K s

15 Let s simplify Big Data with AWS!

16 Simplified Big Data Pipeline Data Ingest Store Process Visualize Answers Time

17 Ingest Store Process Visualize EC2 RDS Kinesis EMR Machine Learning Mobile Analytics DynamoDB CloudSearch Redshift AWS Data Pipeline AWS Import/Export S3 Glacier Lambda EC2

18 Ingest Store Process Visualize EC2 RDS Kinesis EMR Machine Learning Mobile Analytics DynamoDB CloudSearch Redshift AWS Data Pipeline AWS Import/Export S3 Glacier Lambda EC2

19 Stream in Real Time: Kinesis Real-Time Data Processing over large distributed streams Elastic capacity that scales to millions of events per second React In real-time upon incoming stream events Reliable stream storage replicated across 3 facilities Kinesis

20 Kinesis for Real- Time

21 Kinesis: Produce and Consume App.1 HTTP Post [Aggregate & De-Duplicate] S3 AWS SDKs App.2 LOG4J [Metric Extraction] DynamoDB Flume App.3 Fluentd Kinesis [Decision Making Tree] Apache Storm Kinesis Producer Library (IoT) App.4 [Machine Learning] EMR

22 Ingest Store Process Visualize EC2 RDS Kinesis EMR Machine Learning Mobile Analytics DynamoDB CloudSearch Redshift AWS Data Pipeline AWS Import/Export S3 Glacier Lambda EC2

23 React in Real-Time: Lambda Run your code in the cloud, fully managed and highly-available Triggered through invocation or state changes in your setup Scales automatically to match the incoming event rate Can be connected to an Kinesis stream to react upon every incoming event Charged per 100ms execution time Kinesis Lambda

24 Ingest Store Process Visualize EC2 RDS Kinesis EMR Machine Learning Mobile Analytics DynamoDB CloudSearch Redshift AWS Data Pipeline AWS Import/Export S3 Glacier Lambda EC2

25 DynamoDB Fully Managed NoSQL Database Service table Schemaless Data Model Seamless scalability No storage or throughput limits Consistent low latency performance High durability and availability Replicated across 3 facilities items attributes DynamoDB

26

27 500,000 writes / second to their DynamoDB tables 200 additional servers during Superbowl 0 additional servers right after

28 1 instance x 100 hours = 100 instances x 1 hour

29 Let s put it all together: Demo Time!

30 Demo: Live Twitter Feed Analysis DynamoDB Visualization with D3.js Twitter Stream Kinesis Lambda SNS S3

31 Demo: Live Twitter Feed Analysis Cost of running this demo? Kinesis Shard: $0.15/h DynamoDB: $0.0065/h + $0.25/GB Lambda: $ /100ms S3: $0.03/GB Total: $ ~ $0.43 Highly available with virtually unlimited scalability.

32 What s next? Many AWS Services can help your Big Data Roadmap Talk to us at the AWS and Masterson booth to learn how to build a costeffective data analytics platform on us $50 US$50 AWS Credits to get you started

33 Thank you! Olivier Klein Solutions Architect, AWS 23 rd June 2015 Cloud & Big Data Analytics Summit 2015 Hong Kong