Analyzing Big Data with AWS

Transcription

1 Analyzing Big Data with AWS Peter Sirota, General Manager, Amazon Elastic

2 What is Big Data?

3 Computer generated data Application server logs (web sites, games) Sensor data (weather, water, smart grids) Images/videos (traffic, security cameras)

4 Human generated data Twitter Firehose (50 mil tweets/day 1,400% growth per year) Blogs/Reviews/ s/Pictures Social graphs Facebook, linked-in, contacts

5 Big Data is full of valuable, unanswered questions!

6 Why is Big Data Hard (and Getting Harder)?

7 Why is Big Data Hard (and Getting Harder)? Data Volume Unconstrained growth Current systems don t scale

8 Why is Big Data Hard (and Getting Harder)? Data Structure Need to consolidate data from multiple data sources in multiple formats across multiple businesses

9 Why is Big Data Hard (and Getting Harder)? Changing Data Requirements Faster response time of fresher data Sampling is not good enough Increasing complexity of analytics Users demand inexpensive experimentation

10 We need tools built specifically for Big Data!

11 Innovation #1: Apache Hadoop The MapReduce computational paradigm Open source, scalable, fault tolerant, distributed system Hadoop lowers the cost of developing a distributed system for data processing

12 Innovation #2: Amazon Elastic Compute Cloud (EC2) provides resizable compute capacity in the cloud. Amazon EC2 lowers the cost of operating a distributed system for data processing

13 Amazon Elastic MapReduce = Amazon EC2 + Hadoop

14 Elastic MapReduce applications Targeted advertising / Clickstream analysis Security: anti-virus, fraud detection, image recognition Pattern matching / Recommendations Data warehousing / BI Bio-informatics (Genome analysis) Financial simulation (Monte Carlo simulation) File processing (resize jpegs, video encoding) Web indexing

15 Clickstream Analysis Big Box Retailer came to Razorfish 3.5 billion records 71 million unique cookies 1.7 million targeted ads required per day Problem: Improve Return on Ad Spend (ROAS)

16 Clickstream Analysis User recently purchased a sports movie and is searching for video games Targeted Ad (1.7 Million per day)

17 Clickstream Analysis Lots of experimentation but final design: 100 node on-demand Elastic MapReduce cluster running Hadoop

18 Clickstream Analysis Processing time dropped from 2+ days to 8 hours (with lots more data)

19 Clickstream Analysis Increased Return On Ad Spend by 500%

20 World s largest handmade marketplace 8.9 million items 1 billion page view per month $320MM 2010 GMS

21 Job Job Web event logs ETL Step 1 ETL Step 2 Production DB snapshots Job Easy to backfill and run experiments just boot up a cluster with 100, 500, or 1000 nodes

22 Recommendations The Taste Test

23 Recommendations Gift Ideas for Facebook Friends etsy.com/gifts

24

25 Yelp s Business Generates a Lot of Data 400 GB of logs per day ~12 Terabytes per month

26 They Frequently Analyze this Data to Power Key Features of their Site

27 Autocomplete Search

28 Recommendations

29 Automatic spelling corrections

30 Automatic spelling corrections Let s take a Look at how this works

31 1) Load log file data for six months of user search history into Amazon S3 Amazon S3 Search ID Search Text Final Selection westen Westin wisten Westin westenn Westin

32 Hadoop Cluster 2) Spin up a 200 node cluster of virtual servers in the cloud Log Files Amazon S3 Amazon EMR

33 Hadoop Cluster 3) 200 nodes simultaneously analyze this data looking for common misspellings this takes a few hours Amazon S3 Amazon EMR

34 Hadoop Cluster 4) New common misspellings and suggestions loaded back into S3 Log Files Amazon S3 Amazon EMR

35 5) When the job is done, the cluster is shut down. Yelp only pays for the time they used. Log Files Amazon S3 Amazon EMR

36 Each of their 80 developers can do this whenever they have a big data problem to analyze Log file data 250 clusters spun up and down every week

37

38 Data size Global reach Native app for almost every smartphone, SMS, web, mobile-web 10M+ users, 15M+ venues, ~1B check-ins Terabytes of log data

39 Data Stack Application Stack Stack Scala/Liftweb API Machines WWW Machines Batch Jobs Scala Application code Mongo/Postgres/Flat Files Databases Logs mongoexport postgres dump Flume Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/Mahout Analytics Dashboard Map Reduce Jobs

40 Computing venue-to-venue similarity Spin up 40 node cluster Submit Ruby streaming job Invert User x Venue matrix Grab Co-occurrences Compute similarity Spin down cluster Load data to app server

41 Who is checking in? 0.6 Gender Age Female Male

42 What are people doing?

43 Where are our users?

44 When do people go to a place? Gorilla Coffee Gray's Papaya Amorino Thursday Friday Saturday Sunday

45 Why are people checking in? Explore their city, discover new places Find friends, meet up Save with local deals Get insider tips on venues Personal analytics, diary Follow brands and celebrities Earn points, badges, gamification of life The list grows

46 Over 1000 s customers using EMR

47 RDBMS vs. MapReduce/Hadoop RDBMS Predefined schema Strategic data placement for query tuning Exploit indexes for fast retrieving SQL only Doesn t scale linearly MapReduce/Hadoop No schema is required Random data placement Fast scan of the entire dataset Uniform query performance Linearly scales for reads and writes Support many languages including SQL Complementary technologies

48 AWS Data Warehousing Architecture

49 Elastic Data Warehouse Customize cluster size to support varying resource needs (e.g. query support during the day versus batch processing overnight) Reduce costs by increasing server utilization Improve performance during high usage periods Data Warehouse (Steady State) Data Warehouse (Batch Processing) Data Warehouse (Steady State) Expand to 25 instances Shrink to 9 instances

50 Reducing Costs with Spot Instances Mix Spot and On-Demand instances to reduce cost and accelerate computation while protecting against interruption Scenario #1 Job Flow Duration: 14 Hours Scenario #2 Job Flow Duration: 7 Hours #1: Cost without Spot 4 instances *14 hrs * $0.50 = $28 #2: Cost with Spot 4 instances *7 hrs * $0.50 = $ instances * 7 hrs * $0.25 = $8.75 Total = $21.75 Time Savings: 50% Cost Savings: ~22% Other EMR + Spot Use Cases Run entire cluster on Spot for biggest cost savings Reduce the cost of application testing

51 Big Data Ecosystem And Tools We have a rapidly growing ecosystem Business Intelligence MicroStrategy, Pentaho Analytics Datameer, Karmasphere, Quest Open source Ganglia, Squirrel SQL

52 Thank You!!