Big Data on AWS. Services Overview. Bernie Nallamotu Principle Solutions Architect

Transcription

1 on AWS Services Overview Bernie Nallamotu Principle Solutions Architect \

2 So what is it? When your data sets become so large that you have to start innovating around how to collect, store, organize, analyze and share it

3 100 GB Challenges start at relatively small volumes 1,000 PB

4 Unconstrained data growth EB ZB 95% of the 1.2 zettabytes of data in the digital universe is unstructured 70% of of this is usergenerated content GB TB PB Unstructured data growth explosive, with estimates of compound annual growth (CAGR) at 62% from Source: IDC

5 Where does it come from? Web sites Blogs/Reviews/ s/Pictures Social Graphs Facebook, Linked-in, Contacts Application server logs Web sites, games Sensor data Weather, water, smart grids Images/videos Traffic, security cameras Twitter 50m tweets/day 1,400% growth/year

6 Why AWS and big data? Storage Innovation Amazon Amazon DynamoDB RedShift Amazon S3 HPC Spot EMR

7 Services Amazon EMR (Elastic Map Reduce) AWS Data Pipeline Amazon Redshift Hosted Hadoop framework Move data among AWS services and onpremises data sources Petabyte-scale data warehouse service AWS Worldwide Public Sector Team

8 How do you get your slice of it? AWS Direct Connect AWS Import/Export Queuing Amazon Storage Gateway Dedicated low latency bandwidth Physical media shipping Highly scalable event buffering Sync local storage to the cloud

9 Where do you put your slice of it? AWS Relational Database Service Fully managed database (MySQL, Oracle, MS SQL Server, AWS SimpleDB NoSQL, Schema-less Smaller datasets AWS DynamoDB NoSQL, Schema-less, Provisioned throughput database Amazon S3 Object datastore up to 5TB per object % durability PostgreSQL)

10 Where do you put your slice of it? Amazon Glacier Long term cold storage From $0.01 per GB/Month % durability

11 How quick do you need to read it? Single digit ms 10s-100s ms <5 hours AWS DynamoDB Social scale applications Provisioned throughput performance Flexible consistency models AWS S3 Any object, any app % durability Objects up to 5TB in size Performance AWS Glacier Media & asset archives Extremely low cost S3 levels of durability Scale Price

12 Operate at any scale Unlimited data Performance Scale Price

13 Data has gravity App Data App

14 and inertia at volume Data

15 easier to move applications to the data Data

16 Bring compute capacity to the data Very large dataset seeks strong & consistent compute for short term relationship, possibly longer

17 Flexible compute resources, on demand Amazon Elastic Cloud (EC2) Basic unit of compute capacity Range of CPU, memory & local disk options 27 Instance types available, from micro through cluster compute to SSD backed Vertical Scaling From $0.02/hr Feature Flexible Scalable Machine Images Full control VM Import/Export Monitoring Inexpensive Secure Details Run Windows or Linux distributions Wide range of instance types from micro to cluster compute Configurations can be saved as machine images (AMIs) from which new instances can be created Full root or administrator rights Import and export VM images to transfer configurations in and out of EC2 Publishes metrics to Cloud Watch On-demand, Reserved and Spot instance types Full firewall control via Security Groups

18 Elastic capacity as you need it On and Off Fast Growth Variable peaks Predictable peaks

19 Elastic capacity as you need it WASTE On and Off Fast Growth Variable peaks CUSTOMER DISSATISFACTION Predictable peaks

20 Elastic capacity as you need it Capacity Traditional IT capacity Time Elastic cloud capacity Your IT needs

21 Elastic capacity as you need it On and Off Fast Growth Variable peaks Predictable peaks

22 From one instance

23 to thousands

24 Why AWS and big data? Storage Innovation DynamoDB S3 RedShift HPC Spot EMR

25 Why AWS and big data? Storage Innovation DynamoDB S3 RedShift HPC Spot EMR

26 AWS EMR Elastic MapReduce

27 Amazon Elastic MapReduce A key tool in the toolbox to help with challenges Makes possible analytics processes previously not feasible Cost effective when leveraged with EC2 spot market Broad ecosystem of tools to handle specific use cases AWS Worldwide Public Sector Team

28 Hadoop-as-a-service Map-Reduce engine Integrated with tools What is EMR? Massively parallel Integrated to AWS services Cost effective AWS wrapper

29 HDFS Reliable storage MapReduce Data analysis

30 EC2 instance Input file map reduce Output file

31 EC2 instance Input file map reduce Output file EC2 instance Input file map reduce Output file EC2 instance Input file map reduce Output file

32 Map? Reduce? Person Start End Bob 00:44:48 00:45:11 Charlie 02:16:02 02:16:18 Charlie 11:16:59 11:17:17 Charlie 11:17:24 11:17:38 Bob 11:23:10 11:23:25 Alice 16:26:46 16:26:54 David 17:20:28 17:20:45 Alice 18:16:53 18:17:00 Charlie 19:33:44 19:33:59 Bob 21:13:32 21:13:43 David 22:36:22 22:36:34 Alice 23:42:01 23:42:11 map Person Duration Bob 23 Charlie 16 Charlie 18 Charlie 14 Bob 15 Alice 8 David 17 Alice 7 Charlie 15 Bob 11 David 12 Alice 10 reduce Person Total Alice 25 Bob 49 Charlie 63 David 29

33 AWS Elastic MapReduce Architecture AWS Worldwide Public Sector Team

34 Pig HDFS Amazon EMR

35 HDFS Amazon EMR Amazon S3 Amazon DynamoDB

36 Data management HDFS Amazon EMR Amazon S3 Amazon DynamoDB

37 Data management Analytics languages Pig HDFS Amazon EMR Amazon S3 Amazon DynamoDB

38 Data management Analytics languages Pig HDFS Amazon EMR Amazon RDS Amazon S3 Amazon DynamoDB

39 Data management Analytics languages Pig HDFS Amazon EMR Amazon RDS Amazon RedShift AWS Data Pipeline Amazon S3 Amazon DynamoDB

40 Useful Resources & Links AWS : AWS HPC: Architecture Center: Documentation: Security Center: Whitepapers: Resources: Case Studies: Solution Providers: Calculator: TCO Calculator: AWS Blog: The Power of 60:

41 Thank you! Tim Bixler Manager, Federal Solutions Architecture