Big Data for everyone Democratizing big data with the cloud. Steffen Krause Technical

Transcription

1 Big Data for everyone Democratizing big data with the cloud Steffen Krause Technical

2 Does this Data make me look big?

3 Overview Designing big data solutions in the cloud Not the only way to do it (but one that we have seen)

4 Big Data withaws Storage Big Data Compute Challenges start at relatively small volumes 100 GB 1,000 PB

5 Big Data withaws Storage Big Data Compute When data sets and data analytics need to scale to the point that you have to start innovatingaround how tocollect, store, organize, analyze and share it

6 Invest in data centers?

7 Generation Collection & storage Analytics & computation Collaboration & sharing

8

9 Generation Collection & storage Analytics & computation Collaboration & sharing

10 Storage Big Data Data has gravity Compute App Data App

11 Storage Big Data and inertia at volume Compute Data

12 Storage Big Data easier to move applications to the data Compute Data

13 S3 as a single source of truth S3 Courtesy

14 Generation Collection & storage Analytics & computation Collaboration & sharing

15 Hadoop based Analysis Amazon SQS DynamoDB Any SQL or NO SQL Store Log Aggregation tools Amazon S3 Amazon EMR

16 Amazon Elastic MapReduce(EMR)? EMR is Hadoop in the Cloud

17 1 instance for 100 hours = 100 instances for 1 hour

18 Small instance = $6

19 1 instance for 1000hours = 1000instances for 1 hour

20 Small instance = $60

21 When you turn off your cloud resources, you actually stop paying for them

22 SQL based processing Amazon SQS DynamoDB Pre-processing framework Petabyte scale Columnar Data - warehouse Any SQL or NO SQL Store Log Aggregation tools Amazon S3 Amazon EMR Amazon Redshift

23 Generation Collection & storage Analytics & computation Collaboration & sharing

24 Sharing results and visualizations Amazon SQS Amazon Redshift Business Intelligence Tools DynamoDB Any SQL or NO SQL Store Log Aggregation tools Amazon S3 Amazon EMR Business Intelligence Tools

25 The complete architecture Amazon SQS DynamoDB Amazon Redshift Business Intelligence Tools Visualization tools Any SQL or NO SQL Store Log Aggregation tools Amazon S3 Amazon EMR Amazon data pipeline GIS tools Business Intelligence Tools GIS tools on hadoop

26 Use cases

27 28

28

29 Lesson 1: Don t leave your Amazon account logged in at home Lesson 2: Use the data you have to drive proactive processes

30 Analyzing Credit Risk Requires 5 million simulations On AWS the simulation time reduced from 23 hours to 20 minutes

31 3000 cores for Risk Analysis (Monte Carlo) CPU cores 300 cores during the weekend Wed Thu Fri Sat Sun Mon Tue

32 In 60 minutes, CHANNEL 4can analyze and model in-session data to deliver highly targeted ads to viewers before a program ends To get closer to growing video-on-demand (VOD) audiences and match them with advertisers, Channel 4 chose a cloud-based solution to help make sense of and monetize its unprecedented volumes of platform data.

33 Features powered by Amazon Elastic MapReduce: People Who Viewed this Also Viewed Review highlights Auto complete as you type on search Search spelling suggestions Top searches Ads 200 Elastic MapReduce jobs per day Processing 3TB of data

34

35

36

37

38 SkillPages Customer Use Case Everyone Needs Skilled People At Home At Work In Life Repeatedly

39

40 Data Architecture Join via Facebook Add a Skill Page Web Servers Raw Data Amazon S3 User Action Trace Events Invite Friends Get Data Amazon Redshift Amazon S3 Aggregated Data Raw Events Excel Data Analyst Tableau EMR Hive Scripts Process Content Process log files with regular expressions to parse out the info we need. Processes cookies into useful searchable data such as Session, UserId, API Security token. Filters surplus info like internal varnish logging. Internal Web

41 Foursquare Female Gender Male We found that Amazon Redshift offers the performance we needed while freeing us from the licensing costsof our previous solution With Amazon Redshift and Tableau, anyone in the company can set up any queries they like from how users are reacting to a feature, to growth by demographic or geography, to the impact sales efforts have had in different areas. It s very flexible Jon Hoffman, Software Engineer, Foursquare Age When do people go to a place? Gorilla Coffee Gray's Papaya Amorino

42 Stack analysis and sharing Application Stack Scala/Liftweb Scala Mongo/Postgres/Flat Files API Machines Databases mongoexport postgres dump WWW Machines Application code Batch Jobs Logs Flume Data Stack Amazon S3 Database Dumps Log Files Hadoop Elastic Map Reduce Hive/Ruby/Mahout Analytics Dashboard Map Reduce Jobs

43 Everything that was a limited resource is now a programmable resource

44 Resources Hadoop Technology and Use Cases: Start withthefree Tier: 25 US$ creditsfornewgerman customers: Facebook: Webinars: