Big Data Analytics. Lucas Rego Drumond

Transcription

1 Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Big Data Analytics Big Data Analytics 1 / 1

2 Outline 2. Overview 3. Organizational Stuff Big Data Analytics 1 / 1

3 Outline 2. Overview 3. Organizational Stuff Big Data Analytics 1 / 1

4 What is Big Data? Big Data Analytics 1 / 1

5 What is Big Data? Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it. - Dan Ariely Big Data Analytics 2 / 1

6 What is Big Data? Some definitions: A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. data Big Data Analytics 3 / 1

7 What is Big Data? Some definitions: A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. data Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. Big Data Analytics 3 / 1

8 What is Big Data? Big Data is about: Storing and accessing large amounts of (unstructured) data Big Data Analytics 4 / 1

9 What is Big Data? Big Data is about: Storing and accessing large amounts of (unstructured) data Processing high volume data streams Big Data Analytics 4 / 1

10 What is Big Data? Big Data is about: Storing and accessing large amounts of (unstructured) data Processing high volume data streams Making sense of the data Big Data Analytics 4 / 1

11 What is Big Data? Big Data is about: Storing and accessing large amounts of (unstructured) data Processing high volume data streams Making sense of the data Predictive technologies Big Data Analytics 4 / 1

12 Where to find Big Data? 1.28 billion users (1.23 billion monthly active in January 2014) Size of user data sored by Facebook: 300 Petabytes Average amount of data that Facebook takes in daily: 600 terabytes Size of Facebook s Graph Search database: 700 Terabytes Source: b Big Data Analytics 5 / 1

13 Where to find Big Data? 3.3 billion searches per day (on average) 1 30 trillion unique URLs identified on the Web 1 20 billion sites crawled a day 1 In 2008 Google processed more than 20 Petabytes of data per day Jeffrey Dean and Sanjay Ghemawat MapReduce: simplified data processing on large clusters. Commun. ACM 51, 1 (January 2008), Big Data Analytics 6 / 1

14 Where to find Big Data? Average number of tweets per day: 58 million 1 Number of Twitter search engine queries every day: 2.1 billion 1 Total number of active registered Twitter users: 645,750, Big Data Analytics 7 / 1

15 Where to find Big Data? Ensembl database contains the genome of humans and 50 other species only 250 GB Big Data Analytics 8 / 1

16 Where to find Big Data? Large Hadron Collider has collected data from over 300 trillion proton-proton collisions Approx. 25 Petabytes per year Big Data Analytics 9 / 1

17 What to do with Big Data? We don t want to know things but to understand them! Big Data Analytics 10 / 1

18 What to do with Big Data? - Case Studies T-Mobile USA: integrated Big Data across multiple IT systems to combine customer transaction and interactions data in order to better predict customer defections By leveraging social media data along with transaction data from CRM and Billing systems, customer defections has been cut in half in a single quarter. Big Data Analytics 11 / 1

19 What to do with Big Data? - Case Studies T-Mobile USA: integrated Big Data across multiple IT systems to combine customer transaction and interactions data in order to better predict customer defections By leveraging social media data along with transaction data from CRM and Billing systems, customer defections has been cut in half in a single quarter. US Xpress: collects data elements ranging from fuel usage to tire condition to truck engine operations to GPS information Optimal fleet management Big Data Analytics 11 / 1

20 What to do with Big Data? - Case Studies T-Mobile USA: integrated Big Data across multiple IT systems to combine customer transaction and interactions data in order to better predict customer defections By leveraging social media data along with transaction data from CRM and Billing systems, customer defections has been cut in half in a single quarter. US Xpress: collects data elements ranging from fuel usage to tire condition to truck engine operations to GPS information Optimal fleet management McLaren s Formula One racing team: real-time car sensor data during car races Real time identification of issues with its racing cars Big Data Analytics 11 / 1

21 What to do with Big Data? - The BI Approach Data Warehouse Static databases Structured data Centralized approaches Big Data Analytics 12 / 1

22 What to do with Big Data? Massive Parallelism Heterogeneous data sources Unstructured data Data streams Big Data Analytics 13 / 1

23 What to do with Big Data? Application examples: Online personalized advertising Big Data Analytics 14 / 1

24 What to do with Big Data? Application examples: Online personalized advertising Sentiment analysis and behavior prediction Big Data Analytics 14 / 1

25 What to do with Big Data? Application examples: Online personalized advertising Sentiment analysis and behavior prediction Detecting adverse events and predicting their impact Big Data Analytics 14 / 1

26 What to do with Big Data? Application examples: Online personalized advertising Sentiment analysis and behavior prediction Detecting adverse events and predicting their impact Automatic Translation Big Data Analytics 14 / 1

27 What to do with Big Data? Application examples: Online personalized advertising Sentiment analysis and behavior prediction Detecting adverse events and predicting their impact Automatic Translation Image Classification and object recognition Big Data Analytics 14 / 1

28 What to do with Big Data? Application examples: Online personalized advertising Sentiment analysis and behavior prediction Detecting adverse events and predicting their impact Automatic Translation Image Classification and object recognition Inteligent public services Big Data Analytics 14 / 1

29 How? In order to deal with large volumes of data we need to address the following challenges: Big Data Analytics 15 / 1

30 How? In order to deal with large volumes of data we need to address the following challenges: Effectively store and large amounts of data in a distributed environment Big Data Analytics 15 / 1

31 How? In order to deal with large volumes of data we need to address the following challenges: Effectively store and large amounts of data in a distributed environment Query distributed databases Big Data Analytics 15 / 1

32 How? In order to deal with large volumes of data we need to address the following challenges: Effectively store and large amounts of data in a distributed environment Query distributed databases Parallel and distributed programing models Big Data Analytics 15 / 1

33 How? In order to deal with large volumes of data we need to address the following challenges: Effectively store and large amounts of data in a distributed environment Query distributed databases Parallel and distributed programing models Data Mining and machine learning techniques to make sense of the data Big Data Analytics 15 / 1

34 How? In order to deal with large volumes of data we need to address the following challenges: Effectively store and large amounts of data in a distributed environment Query distributed databases Parallel and distributed programing models Data Mining and machine learning techniques to make sense of the data Effective data visualisation techniques Big Data Analytics 15 / 1

35 Course Introduction (1 Lecture) Big Data Analytics 16 / 1

36 Course Introduction (1 Lecture) Large scale storage and retrieval mechanisms (3 Lectures) Big Data Analytics 16 / 1

37 Course Introduction (1 Lecture) Large scale storage and retrieval mechanisms (3 Lectures) Parallel and distributed programing models (4 Lectures) Big Data Analytics 16 / 1

38 Course Introduction (1 Lecture) Large scale storage and retrieval mechanisms (3 Lectures) Parallel and distributed programing models (4 Lectures) Machine Learning techniques for large scale and streamed data (4 Lectures) Big Data Analytics 16 / 1

39 2. Overview Outline 2. Overview 3. Organizational Stuff Big Data Analytics 17 / 1

40 2. Overview Overview Part III Machine Learning Algorithms Part II Large Scale Computational Models Part I Distributed Database Distributed File System Big Data Analytics 17 / 1

41 2. Overview Overview Part I Distributed Database Distributed File System Big Data Analytics 18 / 1

42 2. Overview Storing In a distributed environment the data storing mechanisms should address the follwoing issues Parallel Reading and Writing Data node Failures High Availability Big Data Analytics 19 / 1

43 2. Overview Distributed File Systems The Google File System Architecture Big Data Analytics 20 / 1

44 2. Overview Databases Databases are needed for Querying and indexing transaction procesing Big Data Analytics 21 / 1

45 2. Overview Databases Databases are needed for Querying and indexing transaction procesing State-of-the-art: Relational Databases Big Data Analytics 21 / 1

46 2. Overview Databases Databases are needed for Querying and indexing transaction procesing State-of-the-art: Relational Databases For processing big data one needs a database which: Supports high level of parallelism Supports analytical processing Has a flexible data model to deal with unstructured data sources Big Data Analytics 21 / 1

47 2. Overview Databases for Big Data - NoSQL NoSQL - Not only SQL Wide variety of database technologies Big Data Analytics 22 / 1

48 2. Overview Databases for Big Data - NoSQL NoSQL - Not only SQL Wide variety of database technologies Dynamic Schemas Big Data Analytics 22 / 1

49 2. Overview Databases for Big Data - NoSQL NoSQL - Not only SQL Wide variety of database technologies Dynamic Schemas sharded indexing Big Data Analytics 22 / 1

50 2. Overview Databases for Big Data - NoSQL NoSQL - Not only SQL Wide variety of database technologies Dynamic Schemas sharded indexing horizontal scaling Big Data Analytics 22 / 1

51 2. Overview Databases for Big Data - NoSQL NoSQL - Not only SQL Wide variety of database technologies Dynamic Schemas sharded indexing horizontal scaling support columnar storage Big Data Analytics 22 / 1

52 2. Overview NoSQL Databases Big Data Analytics 23 / 1

53 2. Overview Overview Part II Large Scale Computational Models Part I Distributed Database Distributed File System Big Data Analytics 24 / 1

54 2. Overview Accessing A computational model is needed to: Provide a set of useful computational primitives Hide the complexity of distributed and parallel programming Ensure Fault Tolerance Examples: MapReduce GraphLab Pregel Big Data Analytics 25 / 1

55 2. Overview MapReduce Big Data Analytics 26 / 1

56 2. Overview GraphLab Big Data Analytics 27 / 1

57 2. Overview Overview Part III Machine Learning Algorithms Part II Large Scale Computational Models Part I Distributed Database Distributed File System Big Data Analytics 28 / 1

58 2. Overview Making sense of the data Linear Models for classification and regression Non linear classifiers Factorization Models Models for Link Prediction and link analysis Big Data Analytics 29 / 1

59 2. Overview Recommender Systems Big Data Analytics 30 / 1

60 2. Overview Graph Analysis Obama o y C (o, c) = 1 y F (l, o) = 1 c Annexation of Crimea Lucas l y C (l, p) = 1 p Paco s Death y S (l, n) = 1 ys (n, l) = 1 Nico n Big Data Analytics 31 / 1

61 2. Overview Graph Analysis y F (n, o) =? Obama y F (o, l) =? o y C (o, c) = 1 y F (l, o) = 1 c Annexation of Crimea Lucas l y C (l, p) = 1 p Paco s Death y S (l, n) = 1 ys (n, l) = 1 Nico n y F (n, l) =? Big Data Analytics 31 / 1

62 2. Overview Graph Analysis y S (n, o) =? Obama y S (o, l) =? o y C (o, c) = 1 y F (l, o) = 1 c Annexation of Crimea Lucas l y C (l, p) = 1 p Paco s Death y S (l, n) = 1 ys (n, l) = 1 Nico n y S (n, l) =? Big Data Analytics 31 / 1

63 2. Overview Graph Analysis Obama o y C (o, c) = 1 c Annexation of Crimea y F (l, o) = 1y C (l, c) =? Lucas y C (o, p) =? l y C (l, p) = 1 p Paco s Death y S (l, n) = 1 ys (n, l) = 1 Nico n y C (n, o) =? Big Data Analytics 31 / 1

64 3. Organizational Stuff Outline 2. Overview 3. Organizational Stuff Big Data Analytics 32 / 1

65 3. Organizational Stuff Exercises and tutorials There wil be a weekly sheet with two exercises handed out each Friday in the tutorial. 1st sheet will be handed out Fri Solutions to the exercises can be submitted until next Monday before the lecture. 1st sheet is due Mon Exercises will be corrected Tutorials each Friday 10-12, 1st tutorial at Friday Successful participation in the tutorial gives up to 10% bonus points for the exam. Big Data Analytics 32 / 1

66 3. Organizational Stuff Exams and credit points There will be a written exam at the end of the term (2h, 4 problems). The course gives 6 ECTS The course can be used in IMIT MSc. / Informatik / Gebiet KI & ML Wirtschaftsinformatik MSc / Informatik / Gebiet KI & ML Big Data Analytics 33 / 1

67 3. Organizational Stuff Some books Anand Rajaraman, Jure Leskovec, and Jeffrey Ullman: Mining of massive datasets Available online: ullman/mmds.html Gautam Shroff: The Intelligent Web: Search, smart algorithms, and big data Big Data Analytics 34 / 1

68 3. Organizational Stuff Big Data Software Anand Rajaraman, Jure Leskovec, and Jeffrey Ullman: Mining of massive datasets Available online: ullman/mmds.html Gautam Shroff: The Intelligent Web: Search, smart algorithms, and big data Big Data Analytics 35 / 1