Fast Exact Shortest-Path Distance Queries on Large Networks by Pruned Landmark Labeling

Transcription

1 2013/06/26 SIGMOD 13 Fast Exact Shortest-Path Distance Queries on Large Networks by Pruned Landmark Labeling Takuya Akiba (U Tokyo) Yoichi Iwata (U Tokyo) Yuichi Yoshida (NII&PFI)

2 Problem Definition Given a graph G = V, E 1. construct an index to 2. answer distance d G s, t s t Goal: Good trade-off Scalability Indexing time Index size Query Performance Query time Precision 2

3 Distance on Graphs Context-Aware Web Search Socially-Sensitive Search Social Network Analysis Biological Analysis Route Navigation 3

4 Real-world Networks Road Networks Complex Networks Social Networks Web Graphs Computer Networks Biological Networks 4

5 Real-world Networks Road Networks Complex Networks Social Networks Web Graphs Computer Networks Biological Networks Structural differences Different methods 5

6 Real-world Networks Road Networks Complex Networks SIGMOD Research 24 A. D. Zhu, H. Ma, X. Xiao, et al. (tomorrow) Today s topic: methods for complex networks 6

7 Previous Methods Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] 7

8 Previous Methods: Problems Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] 2-Hop Cover [Cheng+,EDBT 09] Landmark-based methods [Potamias+, CIKM 09] Bad precision for close pairs Distance Sketch [Sarma+, WSDM 10] TD-based Cannot handle networks Highway 2-Hop with 10M edges [Wei, SIGMOD 10] [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Slow querying (ms) 8

9 Proposed Method: Pruned Landmark Labeling Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) 9

10 Proposed Method: Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based Cannot handle networks Highway 2-Hop with >10M edges [Wei, SIGMOD 10] [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, Can handle networks with >100M edges [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) 10

11 Proposed Method: Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] 2-Hop Cover [Cheng+,EDBT 09] Landmark-based methods [Potamias+, CIKM 09] Bad precision for close pairs Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] No error (exact method) Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) 11

12 Proposed Method: Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop Fast [Jin+, querying SIGMOD 12] (μs) Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) Slow querying (ms) 12

13 Proposed Method: Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, Simple and easy to implement [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) 13

14 Preliminaries 14

15 2-Hop Labeling: Index Data Structure Commonly used framework [Cohen+ 02], [Cheng+ 09], [Jin+ 12], [Abraham+ 12] and ours Index: label L v = l 1, δ 1, l 2, δ 2, v δ 1 δ 2 l 1 l 2 l i V, δ i = d G v, l i δ 3 l 3 Example L(1): L(2): L 3 : Vertex Distance Vertex Distance Vertex Distance d G 1,10 = 5 15

16 2-Hop Labeling: Query Algorithm Query: d G (s, t) = min l L s L(t) d G s, l + d G l, t Paths through common vertices L(1): Example Vertex Distance s t L 3 : Vertex Distance Distance between vertex 1 and 3: : = 7 Answer min{6, 7} = : = 6 16

17 2-Hop Labeling: Challenge Challenge: computing labels Correctness (Exactness) Sizes of labels (Index Size & Query Time) Efficiency (Scalability) Previous approach [Cohen+ 02], [Cheng+ 09], [Jin+ 12], [Abraham+ 12] Reduce to optimization problems Our approach Directly assign label entries by graph searches 17

18 Pruned Landmark Labeling 18

19 Our Approach 1. Naïve Landmark Labeling Conduct a BFS from every vertex 2. Pruned Landmark Labeling (proposed method) Pruning during BFSs 19

20 Naïve Landmark Labeling (w/o pruning) 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i O(nm) indexing & space, O(n) query 20

21 Naïve Landmark Labeling (w/o pruning) After a BFS from 1 L 1 (1): L 1 (2): L 1 (3): Vertex 1 Distance 0 Vertex 1 Distance 2 Vertex 1 Distance After a BFS from 2 L 2 (1): L 2 (2): L 2 (3): Vertex 1 2 Distance 0 2 Vertex 1 2 Distance 2 0 Vertex 1 2 Distance

22 Pruned Landmark Labeling 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a pruned BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i 22

23 Pruned BFS Pruned BFS from v i v i Distance δ u If QUERY v i, u, L i 1 We do not label u this time Current incomplete index We do not traverse edges from u δ Prune u 23

24 Example First BFS from vertex 1 L 1 (1): L 1 (2): Vertex 1 Distance 0 Vertex 1 Distance 2 L 1 (6): Vertex 1 Distance 1 Second BFS from vertex 2 QUERY 2, 6, L 1 = = 3 = d(2,6) Vertex 6 is pruned. 24

25 Example The search space gets smaller and smaller 25

26 Vertex Ordering Strategies Choosing the order of vertices to conduct BFSs from. To prune later BFSs as much as possible, Central vertices should come first Selection of landmarks for landmark-based approx. methods [Potamias+,CIKM 09][Gubichev+,CIKM 10][Sarma+,WSDM 10][Qiao+,ICDE 12][Tretyakov+,CIKM 12] Central vertices should be selected Well studied in [Potamias+ 09] Similar We conduct BFSs from vertices with higher degree 26

27 Pruning on Real-world Networks Number of vertices labeled in each pruned BFS. 27

28 Pruning on Real-world Networks 1% 99% Number of vertices labeled in each pruned BFS. 28

29 Theorems: Correctness Theorem 4.1 QUERY s, t, L i for any s, t and i = QUERY s, t, L i Corollary 4.1 (Correctness) QUERY s, t, L n = d G s, t for any s, t i.e., our method is exact. 29

30 Theorems: Minimality Theorem 4.2 (Minimality) L n (the constructed index) is minimal. i.e., we cannot remove any label entry from the index. 30

31 Theorems: Relations between other methods Landmark-based approx. methods [Potamias+ 09][Gubichev+ 10][Sarma+ 10][Qiao+ 12][Tretyakov+ 12] Attain high average precision by exploiting hubs Tree-decomposition-based methods [Wei 10], [Akiba+ 12] Attain good performance by exploiting tree-like fringes Theorem 4.3 If landmark-based methods attain good precision on a graph then PLL will have small label sizes PLL can also exploit hubs (highly central vertices) Fringe Core Theorem 4.4 PLL perform well on graphs with small tree-width PLL can also exploit tree-like fringes 31

32 Combination of Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) 32

33 Combination of Advantages Exact Methods Approx. Methods 2-Hop Cover [Cohen+,SODA 02] Landmark-based methods [Potamias+, CIKM 09] 2-Hop Cover [Cheng+,EDBT 09] Distance Sketch [Sarma+, WSDM 10] TD-based [Wei, SIGMOD 10] Highway 2-Hop [Jin+, SIGMOD 12] Path Sketch [Gubichev+, CIKM 10] TD-based [Akiba+, EDBT 12] Hierarchical HL [Abraham+,ESA 12] Low tree-width of tree-like fringes Landmark-based method + LCA, Shortcuts, [Qiao+, ICDE 12] [Tretyakov+, CIKM 12] Pruned Landmark Labeling (this work) Existence of highly central nodes 33

34 Bit-parallel Labeling 34

35 Two Labeling Schemes Further improve the performance of pruned labeling In the beginning, pruning does not work much. Therefore, we ignore pruning in the beginning. To skip the overhead of vain pruning testes We apply another labeling scheme here 1. Bit-parallel Labeling t times (typically <100) 2. Pruned Labeling For the rest (works only on unweighted graphs) 35

36 Naïve Landmark Labeling 1. L 0 an empty index 2. For each vertex v 1, v 2,, v n Conduct a BFS from v i Label all the visited vertices L i u = L i 1 u v i, d G u, v i 36

37 Key Insight: Distances of neighbors d v, u = δ n 1 ~n 8 are neighbors of v v δ + 1 δ n 1 n 2 n 3 v n 4 n 5 δ 1 n 6 n 7 n 8 u d n i, u = δ 1, δ, or δ + 1 u 37

38 Bit-parallel Labeling: 65 BFSs at once v Vertex v + at most 64 neighbors n 1 n 2 n64 v δ u u For each u, we compute: Distance δ u (BFS) 64bit bitset 2 (Dynamic Programming) Neighbors with distance δ u 1 Neighbors with distance δ u (Neighbors with distance δ u + 1) 38

39 Experiments 39

40 Indexing Time Dataset Network V E PLL (this work) HHL [Abraham+ 12] TD [Akiba+ 12] Gnutella Computer 63 K 148K 54 s 245 s 209 s Slashdot Social 82 K 948K 6.0 s 670 s 343 s Notredame Web 326 K 1.5M 4.5 s 10,256 s 243 s MetroSec Computer 2.3 M 22M 108 s DNF DNF Hollywood Social 1.1 M 114 M 15,164 s DNF DNF Indochina Web 7.4 M 194 M 6,068 s DNF DNF DNF: Did not finish in one day or ran out of memory Much faster & scalable! PLL, TD: Xeon X5670, 48GB, Linux, C++, g++, sequential HHL: Xeon X5680 2, 96GB, Windows, C++, VC++, 12 cores PLL: degree strategy, 16 BP-BFS for first three datasets, 64 BP-BFS for the others 40

41 10 6 Edges Scalability Largest networks used in experiments (indexing time: <1day) Two orders of magnitude larger networks TEDI HCL TD HHL PLL [SIGMOD 10] [SIGMOD 12] [EDBT 12] [ESA 12] (this work) 41

42 Index Size Dataset Network V E PLL (this work) HHL [Abraham+ 12] TD [Akiba+ 12] Gnutella Computer 63 K 148K 209 MB 380 MB 68 MB Slashdot Social 82 K 948K 48 MB 182 MB 83 MB Notredame Web 326 K 1.5M 138 MB 64 MB 120 MB MetroSec Computer 2.3 M 22M 2.5 GB - - Hollywood Social 1.1 M 114 M 12 GB - - Indochina Web 7.4 M 194 M 22 GB - - Comparable PLL, TD: Xeon X5670, 48GB, Linux, C++, g++ HHL: Xeon X5680 2, 96GB, Windows, C++, VC++ PLL: degree strategy, 16 BP-BFS for first three datasets, 64 BP-BFS for the others Distance: 8bit, Vertex ID: 32 bit 42

43 Query Time Dataset Network V E PLL (this work) HHL [Abraham+ 12] TD [Akiba+ 12] Gnutella Computer 63 K 148K 5.2 μs 11 μs 19 μs Slashdot Social 82 K 948K 0.8 μs 3.9 μs 12 μs Notredame Web 326 K 1.5M 0.5 μs 0.4 μs 39 μs MetroSec Computer 2.3 M 22M 0.7 μs - - Hollywood Social 1.1 M 114 M 15.6 μs - - Indochina Web 7.4 M 194 M 4.1 μs - - Also faster PLL, TD: Xeon X5670, 48GB, Linux, C++, g++, sequential HHL: Xeon X5680 2, 96GB, Windows, C++, VC++, sequential PLL: degree strategy, 16 BP-BFS for first three datasets, 64 BP-BFS for the others Average for 1,000,000 random queries 43

44 Conclusion Distance querying on large complex networks is important and challenging Proposed method: Pruned Landmark Labeling Based on 2-hop cover Conduct pruned BFSs from every vertex Further improved by bit-parallel BFSs Experiments 100x larger networks with comparable indexing time, index size and query time Software available: 44