Hadoop and NoSQL Basics: Big Data Demystified. NYS Innovation Summit, 12/17/2013. Matt

Transcription

1 Hadoop and NoSQL Basics: Big Data Demystified NYS Innovation Summit, 12/17/2013 Matt

2 When I want people to think I m smart, I just say HADOOP really loud.

3

4 Hadoop! There it is. Big Data! Data Science! Algorithms!

5

6

7 ... why are we thinking about this at all?

8 ALL the data created until the year 2003 = ALL the data created every two days

9

10 Writes > 12 terabytes of data per day.

11 *the 451 group

12 ... how did we get here?

13 HIERARCHICAL DATABASE MODEL RELATIONAL DATABASE MODEL DOCUMENT DATABASE MODEL

14 HIERARCHICAL DATABASE MODEL Fruit Orange Apple Grape Granny Smith Honeycrisp Red Delicious Used in early mainframe computing! Stores data in one-to-many trees! Not very flexible

15 RELATIONAL DATABASE MODEL Fruit_Variety Granny Smith Honeycrisp Red Delicious Navel Fruit Apple Apple Apple Orange Invented in 1970 by Edgar F. Codd at IBM! Stores data in tuples which resemble rows of a table! Still the most widely used database model

16 RELATIONAL DATABASE MODEL Fruit_ID Fruit_Name 1 Orange 2 Apple 3 Grape Variety_ID Variety_Name Fruit_ID 1 Granny Smith 2 2 Honeycrisp 2 3 Red Delicious 2 4 Navel 1... can also store hierarchical data!

17 RELATIONAL DATABASE MODEL Fruit_ID Fruit_Name 1 Orange 2 Apple 3 Grape Variety_ID Variety_Name Fruit_ID 1 Granny Smith 2 2 Honeycrisp 2 3 Red Delicious 2 4 Navel 1 Has rigid structure or schema.

18 RELATIONAL DATABASE MODEL Fruit_ID Fruit_Name 1 Orange 2 Apple 3 Grape Variety_ID Variety_Name Fruit_ID 1 Granny Smith 2 2 Honeycrisp 2 3 Red Delicious 2 4 Navel 1 Uses unique keys for consistency across tables

19 DOCUMENT DATABASE MODEL Red Delicious Apple Honeycrisp Apple Navel Orange Granny Smith Apple Doesn t have a single structure or schema that each entry must follow! Developed in 1995 for use with Lotus Notes! SO TRENDY

20 DOCUMENT DATABASE MODEL {! Fruits : [! {! Type : Apple,! Variety : Red Delicious! },! {! Name : Granny Smith Apple! },! Navel Orange! ]! }!! CAN have structured elements, but structure doesn t need to be consistent across entries

21 HIERARCHICAL RIGID DATABASE MODEL RELATIONAL DATABASE MODEL DOCUMENT DATABASE MODEL FLEXIBLE

23 Relational Database is to Document Database As Excel Spreadsheet is to Word Document!

24 Relational Database is to Document Database As Excel Spreadsheet is to Word Document!... as SQL is to NoSQL

25 Relational Database is to Document Database As Excel Spreadsheet is to Word Document!... as SQL is to NoSQL* *... mostly / sorta. Stay tuned!

26 SQL, or Structured Query Language, is a language for getting data into and out of a relational database. SELECT Variety_Name FROM fruits WHERE fruit_id = 2! Variety_Name! ! Granny Smith! Honeycrisp! Red Delicious!

27 Depending on who you ask, NoSQL means NOT SQL or NOT ONLY SQL.

28 (in fact, some characterize NoSQL as a movement, not a particular technology or set of technologies.)

29 SQL Databases are highly standardized.! NoSQL Databases are highly fragmented.

30 SQL Databases are highly standardized.! NoSQL Databases are highly fragmented. Some are document model databases, some use a variation of a key-value store. Document Databases

31 So, what are the characteristics of NoSQL databases* that make them so trendy and exciting? * Generally

32 Relational databases have strict schemas dictating the structure of data. NoSQL databases are generally schemaless, even when they use key-value stores.

33 NoSQL databases are generally schemaless, even when they use key-value stores. More flexible Can start entering data before deciding on how that data will be formatted Less structured, consistent

34 NoSQL databases are generally schemaless, even when they use key-value stores. More flexible Can start entering data before deciding on how that data will be formatted Less structured, consistent

35 Relational databases can scale up (on one computer) but not easily out (across many computers). NoSQL databases are designed to scale out across many computers.

36 NoSQL databases are designed to scale out across many computers. Lots of machines == BIG data Can scale quickly if needed No single point of failure More complicated to set up

37 Relational databases read and write information directly to a disk drive. NoSQL databases store information in memory, and/ or include robust built-in caching in memory.

38 NoSQL databases store information in memory, and/ or include robust built-in caching in memory. Faster Memory more expensive than disk Potential reliability issues

39 Relational databases follow the ACID model: NoSQL databases do not follow the ACID model.

40 NoSQL databases do not follow the ACID model. More freedom to handle requests in a way that honors the uniqueness of things. Much greater room for (potentially serious) errors.

41 Relational databases represent data as rows and columns. NoSQL databases often represent data in formats such as JSON, which are native to many programming languages.

42 NoSQL databases often represent data in formats such as JSON, which are native to many programming languages. Easier, faster for programmers Harder for non-programmers

43 SO WAIT, THOUGH, how the f*** do you find anything in a NoSQL database????

44

45 HADOOP is an open source framework for doing MapReduce.! MapReduce is one way to make sense of a document database. (That s how GOOGLE does it.)!

46 MapReduce has two core steps:! Map! and! Reduce.!!!... both are pretty much what they sound like.

47 This is what it actually looks like: function map(string name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1) function reduce(string word, Iterator partialcounts): // word: a word // partialcounts: a list of aggregated partial counts sum = 0 for each pc in partialcounts: sum += ParseInt(pc) emit (word, sum)

48 MAP: For a given document, map each word phrase or item to the number of times that word phrase or item appears. function map(string name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1)

49 REDUCE: NOW, take all of those maps from every document, and reduce them to a single list of items and counts. function reduce(string word, Iterator partialcounts): // word: a word // partialcounts: a list of aggregated partial counts sum = 0 for each pc in partialcounts: sum += ParseInt(pc) emit (word, sum)

50 Honeycrisp Apple Granny Smith Apple Red Delicious Apple Navel Orange

51 Honeycrisp Apple Granny Smith Apple Red Delicious Apple Navel Orange MAP (Red, 1) (Delicious, 1) (Apple, 1) (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1) (Apple, 1)

52 Honeycrisp Apple Granny Smith Apple Red Delicious Apple Navel Orange MAP (Red, 1) (Delicious, 1) (Apple, 1) (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1) (Apple, 1) REDUCE (Red, 1) (Delicious, 1) (Apple, 3) (Honeycrisp, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1)

53 Honeycrisp Apple Granny Smith Apple Red Delicious Apple Navel Orange MAP (Red, 1) (Delicious, 1) (Apple, 1) (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1) (Apple, 1) REDUCE (Red, 1) (Delicious, 1) (Apple, 3) (Honeycrisp, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1)

54 The hard work is distributed

55 The hard work is distributed The easy work is centralized

56 COMP 1 Honeycrisp Apple COMP 2 Granny Smith Apple Red Delicious Apple Navel Orange... but what if we ve got our documents stored on multiple machines?

57 COMP 1 Honeycrisp Apple COMP 2 Granny Smith Apple Red Delicious Apple Navel Orange (Red, 1) (Delicious, 1) (Apple, 1) MAP (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) MAP (Granny, 1) (Smith, 1) (Apple, 1)

58 COMP 1 Honeycrisp Apple COMP 2 Granny Smith Apple Red Delicious Apple Navel Orange (Red, 1) (Delicious, 1) (Apple, 1) MAP (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) MAP (Granny, 1) (Smith, 1) (Apple, 1) (Red, 1) (Delicious, 1) (Apple, 2) (Honeycrisp, 1) REDUCE (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1) (Apple, 1) REDUCE

59 COMP 1 Honeycrisp Apple COMP 2 Granny Smith Apple Red Delicious Apple Navel Orange (Red, 1) (Delicious, 1) (Apple, 1) MAP (Honeycrisp, 1) (Apple, 1) (Navel, 1) (Orange, 1) MAP (Granny, 1) (Smith, 1) (Apple, 1) (Red, 1) (Delicious, 1) (Apple, 2) (Honeycrisp, 1) REDUCE (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1) (Apple, 1) REDUCE REDUCE (Red, 1) (Delicious, 1) (Apple, 3) (Honeycrisp, 1) (Navel, 1) (Orange, 1) (Granny, 1) (Smith, 1)

60 Is this the easiest way to count apples?

61 NOT

62

63 * relational database *

64

65 Tweet Text: I am so happy! Tweet Location: Albuquerque, NM User Home: New York, NY Tweet Text: #FML #FML #FML Tweet Location: Palo Alto, CA User Home: San Francisco, CA

66 Tweet Text: I am so happy! Tweet Location: Albuquerque, NM User Home: New York, NY Tweet Text: #FML #FML #FML Tweet Location: Palo Alto, CA User Home: San Francisco, CA MAP (WITH MATH + SENTIMENT) (1808, +.9) (Distance in Miles, Sentiment Score) (33, -.6)

67 Tweet Text: I am so happy! Tweet Location: Albuquerque, NM User Home: New York, NY Tweet Text: #FML #FML #FML Tweet Location: Palo Alto, CA User Home: San Francisco, CA MAP (WITH MATH + SENTIMENT) (1808, +.9) (Distance in Miles, Sentiment Score) (33, -.6) REDUCE (1808, +.9) (33, -.6)

68 Tweet Text: I am so happy! Tweet Location: Albuquerque, NM User Home: New York, NY Tweet Text: #FML #FML #FML Tweet Location: Palo Alto, CA User Home: San Francisco, CA MAP (WITH MATH + SENTIMENT) (1808, +.9) (Distance in Miles, Sentiment Score) (33, -.6) REDUCE (1808, +.9) (33, -.6) RINSE AND REPEAT LIKE A MILLION TIMES

69 ... none of this is magic.

70 ... in fact, the magic part is just a precursor to doing the actual hard work.

71

72 Danah Boyd s Six Provocations for Big Data: 1. Automating Research Changes the Definition of Knowledge.! 2. Claims to Objectivity and Accuracy are Misleading! 3. Bigger Data are Not Always Better Data! 4. Not All Data Are Equivalent! 5. Just Because it is Accessible Doesn t Make it Ethical! 6. Limited Access to Big Data Creates New Digital Divides

73 What about THE FUTURE?

75 HIERARCHICAL RIGID DATABASE MODEL? RELATIONAL DATABASE MODEL DOCUMENT DATABASE MODEL FLEXIBLE

76

77

78

79 Further Reading: Martin Fowler on NoSQL: Helpful Stack Overflow thread: Finding Friends with MapReduce: Choosing a Database That s Right for Your Business: Demystifying the Role of Big Data in Marketing: mar/12/big-data-marketing-demystified! The NoSQL Movement: Big Data Tools Cost Too Much, Do Too Little: hadoop_no_sql_dont_believe_the_hype/! Is Big Data an Economic Big Dud?: Six Provocations for Big Data: