Building your Big Data Architecture on Amazon Web Services

Size: px

Start display at page:

Download "Building your Big Data Architecture on Amazon Web Services"

Victor Mathews
8 years ago
Views:

1 Building your Big Data Architecture on Amazon Web Services Abhishek

2 AWS Services Deployment & Administration Application Services Compute Storage Database Networking AWS Global Infrastructure

3 AWS Global Infrastructure 9 Regions 25 Availability Zones Continuous Expansion

4 $5.2B retail business 7,800 employees A whole lot of servers Every day, AWS adds enough server capacity to power that whole $5B enterprise

5 Powering the Most Popular Internet Businesses

6 We have partners and technologies ready to help

7 Solving Problems for Organizations Around the World

8 Value proposition of the AWS cloud No Upfront Investment Replace capital expenditure with variable expense Speed and agility Infrastructure in minutes not weeks Low ongoing cost Customers leverage our economies of scale 37 PRICE REDUCTIONS Focus on business Not undifferentiated heavy lifting Flexible capacity No need to guess capacity requirements and overprovision Global Reach Go global in minutes and reach a global audience

scale 37 PRICE REDUCTIONS Focus on business Not undifferentiated heavy lifting Flexible capacity No need to

9 Gartner Magic Quadrant for Cloud Infrastructure as a Service (August 19, 2013) Gartner Magic Quadrant for Cloud Infrastructure as a Service, Lydia Leong, Douglas Toombs, Bob Gill, Gregor Petri, Tiny Haynes, August 19, This Magic Quadrant graphic was published by Gartner, Inc. as part of a larger research note and should be evaluated in the context of the entire report.. The Gartner report is available upon request from Steven Armstrong (asteven@amazon.com). Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

. The Gartner report is available upon request from Steven Armstrong (asteven@amazon.com).

10 An engineer s definition When your data sets become so large that you have to start innovating how to collect, store, organize, analyze and share it

11 Generation Collection & storage Analytics & computation Collaboration & sharing

12 Lower cost, higher throughput Generation Collection & storage Analytics & computation Collaboration & sharing

13 Lower cost, higher throughput Generation Collection & storage Highly constrained Analytics & computation Collaboration & sharing

14 Data volume Generated data Available for analysis Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software Forecast and 2011 Vendor Shares

Center Infrastructure Through 2011 IDC: Worldwide Business

15 Amazon Web Services helps remove constraints

16 Elastic and highly scalable + No upfront capital expense + Only pay for what you use + Available on-demand = Remove constraints

17 More than 25 Million Streaming Members 50 Billion Events Per Day 30 Million plays every day 2 billion hours of video in 3 months 4 million ratings per day 3 million searches Device location, time, day, week etc. Social data

hours of video in 3 months 4 million ratings per day 3

18 10 TB of streaming data per day

21 Who buys video games?

22 Per day: 3.5 billion records 13 TB of click stream logs 71 million unique cookies

23 Today

26 Big Data tools Elastic MapReduce and Redshift

27 Big Data tools Elastic MapReduce and Redshift

28 How does EMR work? Choose: Hadoop distribution, # of nodes, types of nodes, custom configs, Hive/Pig/etc. Put the data into S3 EMR Cluster S3 EMR Launch the cluster using the EMR console, CLI, SDK, or APIs Get the output from S3 You can also store everything in HDFS

29 What can you run on EMR EMR Cluster S3 EMR

30 Resize Nodes EMR Cluster S3 EMR You can easily add and remove nodes

31 10 node cluster x 10 hours costs exactly the same as running 100 nodes cluster x 1 hours

32 Big Data tools Elastic MapReduce and Redshift

33 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud

34 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud MPP Load Query Resize Backup Restore Parallelize and Distribute Everything Dramatically Reduce I/O

35 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud MPP Load Query Resize Backup Restore Parallelize and Distribute Everything Dramatically Reduce I/O Direct-attached storage Large data block sizes Column data store Data compression Zone maps

36 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud Redshift data is encrypted Continuously backed up to S3 Automatic node recovery Transparent disk failure Protect Operations Simplify Provisioning

37 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud Protect Operations Simplify Provisioning Redshift data is encrypted Continuously backed up to S3 Automatic node recovery Transparent disk failure Create a cluster in minutes Automatic OS and software patching Scale up to 1.6PB with a few clicks and no downtime

38 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud Start Small and Grow Big Extra Large Node (XL) 3 spindles, 2TB, 15GiB RAM 2 virtual cores, 10GigE 1 node (2TB) 2-32 node cluster (64TB) 8 Extra Large Node (8XL) 24 spindles, 16TB, 120GiB RAM 16 virtual cores, 10GigE node cluster (1.6PB)

39 Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud Easy to provision and scale No upfront costs, pay as you go High performance at a low price Open and flexible with support for popular BI tools

40 Sydney Singapore Tokyo Price Per Hour for XL Node On-Demand $ Year Reservation $ Year Reservation $ 0.45 (US$)

41 So for example. 1 XL node reserved for 3 years: = 0.45c x number of hours in a month = $340 per month 1 XL node cluster gives you: 2 Cores, 16 GB RAM, 2 TB Disk Plus 2 TB storage in S3 for backups & snapshots

42 Big Data + Cloud = Awesome Combination Big data: Potentially massive datasets Iterative, experimental style of data manipulation and analysis Frequently not a steady-state workload; peaks and valleys Data is a combination of structured and unstructured data in many formats AWS Cloud: Massive, virtually unlimited capacity Iterative, experimental style of infrastructure deployment/usage At its most efficient with highly variable workloads Tools for managing structured and unstructured data

43 THANK YOU Please come visit us at the Solution Architects Corner at AWS

How to Leverage Cloud to Quickly Build Scalable Applications

How to Leverage Cloud to Quickly Build Scalable Applications Chris Keyser Principal Solution Architect David Polley Senior Director Cloud Product Management Cloud Growth Recent IDC cloud research shows