Big Data Storage, Management and challenges. Ahmed Ali-Eldin

Transcription

1 Big Data Storage, Management and challenges Ahmed Ali-Eldin

2 (Ambitious) Plan What is Big Data? And Why talk about Big Data? How to store Big Data? BigTables (Google) Dynamo (Amazon) How to process Big Data Map-Reduce (Hadoop) Under the Hood (DS at work) Paxos for consensus Chubby (Google s implementation) ZooKeeper (Yahoo s implementation. Last Year) How to transfer Big Data Kafka (LinkedIn s solution)

3 What is Big Data? Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it. Edd Dumbill

4 What is Big Data? It is Big Brother s (Orwell s 1984) enabler to watch the people of Oceania Every day, humans create 2.5 quintillion (10 21 ) bytes of data (quote from IBM) 100 hours of video are uploaded to YouTube every minute. All needs to be stored. All needs to be searched 350 million new photos each day uploaded to FB 25 to 30 Petabytes of data needs to be stored by the LHC Much much much more is produced per day (at least 1 petabyte per day that is filtered data)

5 Big Data Storage

6 Quick Review on ACID Atomicity if one part of the transaction fails, the entire transaction fails, and the database state is left unchanged Consistency any transaction will bring the database from one valid state to another Isolation concurrent execution of transactions results in a system state that would be obtained if transactions were executed serially Durability once a transaction has been committed, it will remain so, even in the event of power loss, crashes, or errors.

7 CAP Theorem For a Distributed system (and database) you can mainain two of three properties: Consistency Availability Partition tolerance

8 BASE: A newer data storage paradigm Maintaining ACID is impossible for highly distributed data stores You never want to sacrifice availability Basically Available, Soft state, Eventual consistency

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35 Great Ideas in Computing Lecture 3: MapReduce

36 Example: Counting Words Count the most popular terms on the Internet: Sounds easy Why not this? CountWords(internet): hash_map<string, int> word_count; for each page in internet: for each word in page: word_count[word] += 1

37 Motivation: Large-Scale Data Processing Example: 20+ billion web pages x 20KB = 400+ ~1,000 hard drives just to store the web terabytes One computer can read MB/sec from disk ~four months to read the web Even more to do something with the data

38 Large-Scale Data Processing 1. Problems: 1. Time 2. Errors / Problems 2. Solution: 1. Distribute processing on many machines

39 Indexing Algorithm Input: Billions of documents (html) Distributed across many machines Indexing Pipeline Output: Index of terms: word1 -> document 1, document 2, etc

40 Link Reversal Input: Billions of documents (html) Distributed across many machines Link Reversal Output: Incoming links for each document: URL1 -> URL2, URL3, URL5,

41 Any Algorithm Input: Billions of? Distributed across many machines? Output:?

42 Before MapReduce Each team needed to understand distributed computing Extra code Extra complexity...

43 Algorithm Requirements: Has to be scalable (terabytes of data) Has to be fast Has to be fault-tolerant Has to be easy to implement and flexible The solution: MapReduce: a distributed algorithm

44 MapReduce: Introduction MapReduce is a programming model and an associated implementation for processing and generating large data sets. Can run on thousands of machines Can process terabytes of information (input / output) It is fault-tolerant It is easy to implement

45 MapReduce Model Programming model inspired by functional language primitives, e.g., LISP 1. Read a lot of data 2. Map: extract something you care about from each record 3. Shuffle and Sort 4. Reduce: aggregate, summarize, filter, or transform 5. Write the results Outline stays the same, map and reduce change

46 MapReduce: Logical Diagram INPUT Grouping MAP Intermediate data G REDU CE OUTPUT

47 Programming Model Input & Output: each a set of key/value pairs Programmer specifies two functions: map (in_key, in_value) -> list(out_key, intermediate_value) Processes input key/value pair Produces set of intermediate pairs GROUPING reduce (out_key, list(intermediate_value)) -> list (out_value) Combines all intermediate values for a particular key Produces a set of merged output values (usually just one)

48 MapReduce: Logical Diagram INPUT Grouping MAP Intermediate data G REDU CE OUTPUT

49 MapReduce: Map Phase First, the MAP function: Programmer supplies this function map (in_key, in_value) -> list(out_key, intermediate_value) Input = any pair (e.g., URL, document) Output = list of pairs (key/value)

50 MapReduce: Map Example Example: word counting <URL, document text> -> Output: list(<word, count>) E.g. for one document <URL, nile university is the best university > Output -> list(<nile, 1>, <university, 1>, <is, 1>, <the, 1>, <best, 1>, <university, 1> ) For another document <URL, "UNIX is simple and elegant"> Output -> list(<unix, 1>, <is, 1>, <simple, 1>...)

51 MapReduce: Map Example Example: word counting <URL, document text> Output: list(<word, count>) Code: Map(string input_key, string input_value): // key: document URL // value: document text for each word in input_value: OutputIntermediate(word, 1 );

52 MapReduce: Logical Diagram INPUT Grouping MAP Intermediate data G REDU CE OUTPUT

53 MapReduce: Grouping The GROUPING (or combining) This happens automatically (by the library) Put all values for same key together For our example: <nile, list(1)> <university, list(1, 1)> <the, list(1)> <is, list(1, 1)> <best, list(1)>...

54 MapReduce: Logical Diagram INPUT Grouping MAP Intermediate data G REDU CE OUTPUT

55 MapReduce: Reduce Last phase is REDUCE Takes the output from grouping: reduce (out_key, list(intermediate_value)) -> list (out_value)

56 MapReduce: Reduce Example Example (word-counting): <university, list(1, 1)> <university, 2> Reduce(string key, List values): // key: word, same for input and output // values: list of counts int sum = 0; for each v in values: sum += v; Output(sum);

57 MapReduce: Word Count Want to count how many times each word occurs MAP: Input: <DocumentId, Full Text> For time a word occurs: Output: <Word, 1> REDUCE: Input: <Word, list(1, 1, )> Sum over each word: Output <Word, Count>

58 MapReduce: Grep (search) Want to search terabytes of documents for where a word occurs MAP: Input: <DocumentId, Full Text> For each line where the word occurs: Output: <DocumentId, LineNumber> REDUCE: Input: <DocumentId, list(linenumber)> Do nothing (output == input)

59 MapReduce: Link Reversal Want to build a reverse-link index MAP: Input: <source_url, Full Text> For each outgoing link <a href= (destination_url ) >: Output: <destination_url, source_url> REDUCE: Input: < destination_url, list(source_url)> Join source_url Output: for each URL, a list of all other URLs that link to it.

60 MapReduce Usage Index construction Web access log stats, e.g., search trends (Google Zeitgeist) Distributed grep Distributed sort Most popular terms on the Internet Document clustering (news, products, etc.) Find all the pages that link to each document (link reversal) Machine learning Statistical machine translation

61 Why complicate things?

62 MapReduce: Parallelism Each Map call is independent (parallelize) Run M different Map tasks (each takes a chunk of input) Each Reduce call is independent (parallelize) Run R different Reduce tasks (each produces a chunk of output)

63 MapReduce: Parallelism

64 Framework User: Map function (used in Map Phase) Reduce function (used in Reduce Phase) Options (Map tasks, Reduce tasks, Workers, Input, Output) Framework: Execute map and reduce phase (distributedly) Collect/move data: input intermediate output data Handle faults Dynamic load-balancing and scheduling

65 MapReduce: Scheduling One master, many workers Input data split into M map tasks (16-64MB in size) Reduce phase partitioned into R reduce tasks Tasks are assigned to workers dynamically Often: M =200,000; R =4,000; workers=2,000 Master assigns each map task to a free worker Considers locality of data to worker when assigning task Worker reads task input (often from local disk!) Worker produces R local files containing intermediate k/v pairs Master assigns each reduce task to a free worker Worker reads intermediate k/v pairs from map workers Worker sorts & applies user s Reduce op to produce the output

66 MapReduce: The Master One machine is the Master Controls other machines (workers), start or stop MAP/REDUCE Keeps track of progress of each worker Stores location of input, output of each worker There are MAP workers and REDUCE workers

67 MapReduce: Parallelism

68 MapReduce: Map Parallelism MAP worker1 MAP output 1 MAP output 2 Input #1 Input #2 Input #3 MAP worker2 MAP output 1 MAP output 2 MAP worker3 MAP output 1 MAP output 2

69 MapReduce: Map Parallelism MAP output is split for REDUCERs by key Bins for keys E.g. keys starting (a m) output1, (n z) output2 (Usually use <hash(key) modulo (# of reducers)> for more random partition) MAP worker1 MAP output 1 MAP output 2

70 MapReduce: Reduce Parallelism MAP worker1 MAP output 1 MAP output 2 REDUCE worker1 MAP worker2 M1 M2 MAP worker3 M1 M2 REDUCE worker2

71 MapReduce: Reduce Parallelism Each REDUCE worker reads one section of input from all MAP workers Performs GROUPing Performs reducing Stores output REDUCE worker1 Output 1

72 MapReduce: Fault Tolerance More machines = more chance of failure Probability of one machine failing p (small) If each machine fails independently, chance of any one out of k machine failing 1 - (1 p)n (larger than p) If p = 0.01, probability of one failure: 1 machine machines machines machines machines 0.99

73 MapReduce: Fault Tolerance If one machine fails, the program should not fail If many machines fail, the program should not fail

74 MapReduce: Fault Tolerance Worker fails (happens often): Master checks workers If worker responds as failed, or does not respond: Master restarts on different machine Bad input: Master keeps track of bad input Asks workers to skip bad input Master fails (happens rarely): Can abort operation Or, can restart master on a different machine

75 MapReduce: Optimizations No REDUCE can start until all MAPs are complete (bottleneck) Sometimes one machine is really slow (bad RAM, corrupted disk, etc) Master notices slow machines Executes input on another machine Uses output of machine that finishes first Other optimizations Locality optimization Intermediate data compression

76 Task Granularity and Pipelining Fine granularity tasks: many more map tasks than machines e.g., 200,000 map/5000 reduce tasks w/ 2000 machines Minimizes time for fault recovery Can pipeline grouping with map execution Better dynamic load balancing

77 MapReduce: Performance Using 1,800 machines: Two 2GHz Intel Xeon 4GB RAM Two 160GB disks Gigabit Ethernet MR_Grep scanned 1 terabyte in 100 seconds MR_Sort sorted 1 terabyte of 100 byte records in 14 minutes

78 Conclusion The internet is composed of billions of documents To process that information, we need to use distributed algorithms and run processes in parallel MapReduce has proven to be a useful abstraction Greatly simplifies large-scale computations at Google Fun to use: focus on problem, let library deal w/ messy details

79 MapReduce: Beyond Google Hadoop: Open-source implementation of MapReduce MapReduce implementations in C++, Java, Python, Perl, Ruby, Erlang... Most companies that run distributed algorithms have an implementation of MapReduce (Facebook, Nokia, MySpace,...)

80 For Next Class: Read the Google Labs MapReduce paper (by Ghemawat and Dean)