/ Cloud Computing. Recitation 11 November 10 th and November 12 th, 2015

Transcription

1 / Cloud Computing Recitation 11 November 10 th and November 12 th, 2015

2 Overview Administrative issues Tagging, 15619Project, project code Last week s reflection Project 3.4 Quiz 9 This week s schedule Project 3.5 Unit 5 - Module Project Phase 2 Quiz 10 Demo Twitter Analytics: The 15619Project

3 Reminders Monitor AWS expenses regularly and tag all resources Check your bill (Cost Explorer > filter by tags). Piazza Guidelines Please tag your questions appropriately Search for an existing answer first Provide clean, modular and well documented code Large penalties for not doing so. Double check that your code is submitted!! (verify by downloading it from TPZ from the submissions page) Utilize Office Hours We are here to help (but not to give solutions) Use the team AWS account and tag the 15619Project resources carefully

4 Project 3.4 : FAQs Problem 1: DynamoDB import failure Make sure the data is processed correctly to meet DynamoDB s requirement. Be aware of escaping problems. Problem 2: Maven dependency error Be aware of what you include. Some dependencies might have conflicts.

5 This week: Project 3.5 P3.1: Files, SQL and NoSQL P3.2: Sharding and Replication P3.3: Consistency P3.4: Social network with heterogeneous backends P3.5: Data warehousing and OLAP

6 P3.5: Background Carnegie Eagle(CE), a supermarket chain wishes to expand their business further into other markets and wants a decision support system The CTO of CE wants you to analyze various data warehouses and pick the best one for the job The CTO subscribes to the Hottest Data Warehouses weekly and decides she wants you to analyze the following: Hive Impala Redshift

7 OLTP vs OLAP

8 Data warehousing and OLAP OLAP (Online Analytical Processing) queries deals with historical/archived data OLAP warehouses optimized for reads and aggregations Rarely perform updates Tables in OLAP are denormalized as compared to normalized tables in OLTP (Online Transaction Processing) Data warehouses tuned for high throughput since they process large amounts of data OLTP databases are more tuned for smaller updates and lower latency

9 Hive Impala Redshift OLAP data warehouses Built on Hadoop and provides a SQL-like interface to query the data Translates user SQL query to MapReduce jobs Based on Hadoop as well and provides a SQL-like interface similar to Hive Uses its own engine to translate user queries and directly access data on the cluster (hence lower latency) Datawarehouse-as-a-Service provided by Amazon Web Services Used for real time analytics

10 Module to Read UNIT 5: Distributed Programming and Analytics Engines for the Cloud Module 18: Intro to distributed programming for the Cloud Module 19: Distributed analytics engines: MapReduce Module 20: Distributed analytics engines: Spark Module 21: Distributed analytics engines: GraphLab

11 Distributed Programming Taxonomy of Programs: Sequential Concurrent Parallel Challenges in programming the cloud: Scalability Communication overhead Heterogeneity Synchronization Fault Tolerance Scheduling

12 Upcoming Deadlines Quiz 9 : Unit 5 - Module 18 Open: 11/13/ :01 AM Pittsburgh Due: 11/13/ :59 PM Pittsburgh Project 3.5 : Data warehousing and OLAP Due: 11/15/ :59 PM Pittsburgh 15619Project : Phase 2 Live-test due: 11/11/2015 4:59 PM Pittsburgh Code and report due: 11/12/ :59 PM Pittsburgh

13 Busy Weeks Coming Up! Wednesday Thursday Friday Sunday Wednesday 11/11/ :00:01 EST Phase 2 Live Test Thursday 11/12/ :59:59 EST Phase 2 Code & Report Due Friday 11/13/ :59:59 EST Quiz 10 Sunday 11/15/ :59:59 EST P3.5 Due Wednesday 12/2/ :00:01 EST Phase 3 Live Test Thursday 12/3/ :59:59 EST Phase 3 Code & Report Due Friday 12/4/ :59:59 EST Quiz 12 Sunday 12/6/ :59:59 EST P4.2 Due

14 Project 3.5 OLAP with Cloud Data Warehousing

15 Star Schema

16 Data Warehousing Benchmark

17 Data Warehousing Benchmark Hive No optimization required Follow the instructions and you are done Impala and Redshift Load data, execute unoptimized queries Optimize table schemas and/or queries The evaluation will be on both correctness and response time

18 Notes Hive May take a few hours, be patient. Impala Some unoptimized queries may throw exceptions. Redshift Be aware of the high expenditure! Think before you start.

19 Project 3.5 Demo

20 twitter DATA ANALYTICS: PROJECT

21 15619 Project Phase 2 Deadlines Project Phase Project Phase 2 Live Test Thursday 10/15/ :00:01 ET Thursday 10/29/ :59:59 ET Wednesday 11/11/ :00:01 ET Wednesday 11/11/ :59:59 ET Thursday 11/12/ :59:59 ET Project Phase 2 Q3 & Q4 Development Project Phase 2 Code & Report Due

22 Phase 2 Two more queries (Q3 and Q4) More ETL Multiple tables and queries Live Test!!! Warmup, Q1, Q2, Q3, Q4, MixedQ1-Q4 Each for 30 min Both HBase and MySQL Two DNS Must get over 50% score on both! No more pre-caching of known requests

23 Query 3: Spreading Happiness & Sadness Q. How to compute impact score for each tweet? score = sentiment_score * ( 1 + followers_count ) Q. What s followers_count and how do I find it? Read

24 Query 3: Spreading Happiness & Sadness Sample Query: GET /q3? start_date= &end_date= &userid= &n=10 n will be less or equal to 10

25 Query 3: Spreading Happiness & Sadness Sample Response: Positive Tweets Austin please follow me i love you much! x ,512, ,@ArianaGrande Ariana please follow me i love you much! x ,512, ,@ArianaGrande Ariana please follow me i love you much! x ,512, ,@ArianaGrande Ariana please follow me i love you much! x ,512, ,@ArianaGrande Ariana please follow me i love you much! x ,508, ,@ArianaGrande Ariana please follow me i love you much! x ,270, ,@AustinMahone please follow me. I know that it is "impossible" that you see this tweet but if you see it, please!! x ,270, ,@ArianaGrande please follow me. I know that it is "impossible" that you see this tweet but if you see it, please!! x7 Negative Tweets

26 Query 4: Hashtag Analysis Use the hashtag entity Sample Query: GET /q4?hashtag=01net&n=4 Sample Response: :2: , :#01net: Philips attaque Nintendo et veut faire bannir la Wii U des ventes aux Etats-Unis :1: :#01net: Apple pourrait lancer un MacBook Air Retina et un ipad 12'' au second semestre :1: :#01net: TV 4K : faut-il craquer tout de suite? Nos tests, les prix et ce qui arrive :1: :#01net: Le rapprochement entre Bouygues Telecom et Free se prépare en douceur

27 Phase 2 Clarifications Censorship? q3:, q4: X No required time filtering Tie breaker? q3: q4: Primary key - absolute value of score (descending) Secondary key - tweet id resolve ties by tweet i (ascending) Primary key - count, date with higher count com first Secondary key - date, earliest date comes first

28 Tips for Phase 2 Carefully design and check ETL process Watch your budget. $60 = phase + Live Test Preparing for the live test You are required to submit two URLs, one for the MySQL live test and one for the HBase for live test Budget limited to $1.25/hr for MySQL and HBase web service separately. Tag correctly. Do NOT run anything else. Caching known requests will not work Need to have all Q1-Q4 running at the same time Don t expect testing in sequence.

29 MySQL Understand the difference between different storage engines Understand the difference between different types of indices Remember, replication may not be the best strategy if the tables are big Avoid costly operations in front-end Make use of connection pools, caching configuration in MySQL Consider using a MySQL Cluster

30 HBase Make use of WebUI: Check which tables are getting requests the mosts Use split, move, merge functions to evenly distribute loads Make sure your front end is not the bottleneck Carefully design schema Consider building your own HBase- EMR is expensive and abstracts away details

31 Phase 2 Deadline What s due soon? Submission by 16:59 ET (Pittsburgh) Wed 11/11 HBase Live from 6 PM to 9 PM ET MySQL Live from 9 PM to midnight ET Fix Q1 and Q2 if your Phase 1 did not go well New queries Q3 and Q4. Targets 5000 and 8000 rps Get score higher than 50% on both MySQL and HBase Heads up: Phase 2 counts for 30% of 15619Project grade Report at the end of Phase 2 Make sure you highlight failures and learning If you didn t do well, explain why If you did, explain how Cannot begin to stress how critical this is!!!!