Algorithm Design for MapReduce

Transcription

1 Algorithm Design for MapReduce CprE 419X, Spring 2015 Iowa State University Srikanta Tirthapura 1/29/15 CprE 419 X, Srikanta Tirthapura 1

2 Problem 0: Sum Find the sum and average of many input integers Write the map and reduce pseudocode What is the map cost, reduce cost, per reducer cost, and communication cost of your method? 1/29/15 CprE 419 X, Srikanta Tirthapura 2

3 Problem 1: Set Difference Input: Two sets A and B Output: Set difference (A B), i.e. all those elements that are present in A but not in B Ex: Find all IP addresses that appeared in one log but not the other What is the map cost, reduce cost, per reducer cost, and communication cost of your method? 1/29/15 CprE 419 X, Srikanta Tirthapura 3

4 Set Difference MR Algorithm map(key = set_name, value = elementid) emit (key = elementid, value = set_name) reduce (key = elementid, values) if (A in values) and (B not in values): emit(key = elementid, value = 1) 1/29/15 CprE 419 X, Srikanta Tirthapura 4

5 Observation Also works if A and B are multisets (i.e. the same value appears multiple times within A and B) 1/29/15 CprE 419 X, Srikanta Tirthapura 5

6 Analysis Is the set difference algorithm good? How much does it cost? Not an easy question to answer Consider: Per Mapper Cost, Total Map Cost Per Reducer Cost, Total Reduce Cost Total Bytes of Communication 1/29/15 CprE 419 X, Srikanta Tirthapura 6

7 Set Difference Let n = input size ( A + B ), M = number of mappers, R = number of reducers Total Map Cost = Theta(n) Per Mapper Cost = Theta(n/M) Total Reduce Cost = Theta(n) Per Reducer Cost = Theta(n/R) Total Communication Cost = Theta(n) 1/29/15 CprE 419 X, Srikanta Tirthapura 7

8 Problem 2: Matrix-Vector Input: n x n matrix M Multiplication M[i,j] provided as (i,j,m[i,j]) within a HDFS file rows numbered 1.. n, columns similar Element at row i and column j denoted M[i,j] n x 1 vector A A[j] provided as (j,a[j]) within a HDFS file Output: Vector B = M A B[i] = M[i,1] * A[1] + M[i,2] * A[2] +. 1/29/15 CprE 419 X, Srikanta Tirthapura 8

9 Matrix-Vector Using Mapreduce Suppose reduce key was i, i=1 n Reduce function for key i computes B[i] Needs values M[i,1]*A[1], M[i,2]*A[2], How do we compute these? Computing M[i,j] * A[j] Use j as the key in one MapReduce round 1/29/15 CprE 419 X, Srikanta Tirthapura 9

10 Matrix-Vector MR Round 1 map1(key, value = (i,j,m[i,j])) emit (key = j, value = (i,j,m[i,j])) map1(key, value = (j,a[j])) emit (key = j, value = A[j]) reduce1(key = j, values = [A[j], M[1,j], [2,j],..]) for (i = 1 to n): emit(i, M[i,j] * A[j]) 1/29/15 CprE 419 X, Srikanta Tirthapura 10

11 Matrix-Vector MR Round 2 map2(key = i, value) emit (key = i, value) reduce2(key = i, values) B[i] = 0 for (v in values): B[i] += v emit (key = i, value = B[i]) 1/29/15 CprE 419 X, Srikanta Tirthapura 11

12 Matrix-Vector Analysis First MR Round. Total Map cost = O(n 2 +n) = O(n 2 ) Per Mapper Cost = O(n 2 /M) Total Reduce Cost = O(n 2 ) Per Reducer Cost = O(n 2 /R) Communication = O(n 2 ) Second Round is Similar Q: One Round Algorithm for Matrix-Vector using MapReduce? 1/29/15 CprE 419 X, Srikanta Tirthapura 12

13 Problem 3: Graph Processing Input: A graph G presented as a sequence of edges e1 = (v11, v12) e2 = (v21, v22) e3 = (v31, v32). Output: Set of all triangles in G A triangle is a set of three vertices (u,v,w) such that all three edges (u,v) (v,w) and (u,w) exist in G 1/29/15 CprE 419 X, Srikanta Tirthapura 13

14 Enumerating Triangles: Solution Idea e1 = (a,b) a: (b,d) a: (b,d) b: (a,c,d) d: (b,a) e2 = (b,c) e3 = (b,d) MR Round 1 b: (a,c,d) c: (b) MR Round 2 b: (a,c,d) a: (b,d) d: (b,a) d: (b) e4 = (a,d) d: (b,a) 1/29/15 CprE 419 X, Srikanta Tirthapura 14

15 MapReduce: Triangles, Round 1 map1 (String key, String edge): v1, v2 = vertices in edge emit (v1, v2) emit (v2, v1) reduce1(string vertexid, Iterator values): for other_vertex in values: emit (vertexid, values) 1/29/15 CprE 419 X, Srikanta Tirthapura 15

16 MapReduce: Triangles, Round 2 map2 (String vertexid, String neighbor_list): for each vertex v in neighbor_list: emit (v, (vertexid, neighbor_list)) reduce2 (String vertexid, Iterator values): construct 2-neighborhood graph of vertexid enumerate all triangles in this graph 1/29/15 CprE 419 X, Srikanta Tirthapura 16

17 Problem 4: Length 2 Paths in a Graph Input: A graph G presented as a list of edges Output: All paths of length 2 in the graph Solution: Similar to the problem of enumerating triangles in a graph 1/29/15 CprE 419 X, Srikanta Tirthapura 17

18 Problem 5: Finding Pairs of Nearby Bit Strings Input: Set of bitstrings, each of length k Output: All pairs of bit strings such that the two strings in the pair differ at no more than two positions, i.e. at a Hamming distance of no more than 2. 1/29/15 CprE 419 X, Srikanta Tirthapura 18

19 Example: Nearby Bit Strings Input: 10010, 00010, 11000, 11110, Output: (10010, 00010), (10010, 11000) (10010, 11110), (10010, 00000) (00010, 00000), (11000, 11110) (11000, 00000) 1/29/15 CprE 419 X, Srikanta Tirthapura 19

20 Algorithm 1 Compare all pairs of strings and see if they are within a Hamming Distance of 2 Mapper: For each string b, Send a copy of b to each reducer Reducer i: Receives entire set S Computes a subset S i Examines all pairs in S i x S 1/29/15 CprE 419 X, Srikanta Tirthapura 20

21 Algorithm 1 Analysis Let n = total number of input strings. M = # of mappers, R = # of reducers Total Map Cost = O(nkR) Per Mapper Cost = O(nkR/M) Total Reducer Cost = O(n 2 k) Per Reducer Cost = O(n 2 k/r) Total Communication = O(nkR) 1/29/15 CprE 419 X, Srikanta Tirthapura 21

22 Algorithm 2 For each bitstring b, there are only (k choose 2) strings at Hamming distance 2 (k choose 1) strings at Hamming distance 1 Search all these possibilities 1/29/15 CprE 419 X, Srikanta Tirthapura 22

23 Algorithm 2 Map(key, value = bitstring b): emit (key = b, value = b) for each b formed by flipping 1 bit of b: emit (key = b, value = b) Reduce(key = bitstring b, values): for any two strings b1, b2 in values: emit(key = (b1, b2), value = 1) 1/29/15 CprE 419 X, Srikanta Tirthapura 23

24 Analysis of Algorithm 2 Let n = total number of input strings, M = # mappers, R = # reducers Total Map Cost = O(nk 2 ) Per Mapper Cost = O(nk 2 /M) Total Reduce Cost = O(kn + output size) Per Reducer Cost = O(kn/R + output size) Total Communication = O(nk 2 ) 1/29/15 CprE 419 X, Srikanta Tirthapura 24

25 Generalization List all pairs of bitstrings that are at a Hamming distance of t or lesser. 1/29/15 CprE 419 X, Srikanta Tirthapura 25

26 Problem 6: Sorting Given a large file of strings, one per line, sort them in alphabetical order 1/29/15 CprE 419 X, Srikanta Tirthapura 26

27 Sorting in Mapreduce The key, value paradigm does not help here Use the fact that in Mapreduce, the reducer is provided a list of keys in the sorted order Also use a splitter that sends data to reducers not according to hash, but using a different method 1/29/15 CprE 419 X, Srikanta Tirthapura 27

28 Problem 7: Graph Connectivity Input: A graph, presented as a list of edges Output: Yes if graph is connected, and No if it is not connected. 1/29/15 CprE 419 X, Srikanta Tirthapura 28