Succinct Data Structures: From Theory to Practice

Transcription

1 Succinct Data Structures: From Theory to Practice Simon Gog Computing and Information Systems The University of Melbourne January 5th 24

2 Big data: Volume, Velocity,...

3 ...and Variety Picture source IBM

4 ...and Variety This talk addresses two Vs Variety: Index unstructured data Volume: Index larger data sets Velocity is not directly addressed Index: Construct once, query often Picture source IBM

5 Outline The Inverted Index: a success story 2 Suffix sorted based indexes: a tour de force 3 More virtues of the Wavelet Tree 4 Various Structures 5 SDSL: A toolbox of succinct structures 6 Conclusion

6 Index structures are the bases of information retrieval

7 Inverted Index Inverted Indexes for a collection of documents C used for web indexing practical in domains with well-defined vocabulary space is usually 5 % of C (without positions) space is about 5% if positions are included can not be used to reconstruct original text (parsing, stemming, stopping,...) original text has to be stored separately to allow snippet extraction

8 Inverted Index Documents (already normalized) d : is big data really big d : is it big in science d 2 : big data is big Inverted Lists big : {(,),(,4),(,2),(2,),(2,3)} data : {(,2),(2,)} in : {(,3)} is : {(,),(,),(2,2)} really : {(,3)} science : {(,4)}

9 Exact search using Inverted Index based systems () No document containing <line=break /> is returned.

10 Exact search using Inverted Index based systems (2) but it exists...

11 Summary: Inverted Indexes Advantages: Sequential locality in many operations (fast!) Index can be stored on disk Good compression for natural language text Disadvantages: Decision, what is searchable, made at parsing time No fast calculation of phrase queries No alphabet not applicable Adapt data to index structure Search quality seems to be enough for casual web search. Scientific tasks require high quality search.

13 Suffix sorted based indexes Suffix sorted based indexes for a text T sort all n suffixes of T works also in domains with no well-defined vocabulary used in Bioinformatics space uncompressed: 5 to 5 size of T compressed: n H k (T ) bzip d T index structure adapts to data can be used to reconstruct original text (Self-Index) Not cover in this talk: LZ compressed indexes (works well for highly repetitive data)

14 H k of Pizza&Chili corpus texts (2MB versions) H k (T ) in bits contexts/ T in percent k DBLP.XML DNA ENGLISH PROTEINS

15 Index size comparison (English text) size original text Suffix Tree Suffix Array Self-Index

16 Self-Index Operations Let P be a character pattern of length m = P. Operations: count(p) How many times does P occur in T? locate(p) List all count(p) positions where P occurs. extract(i,j) Extract text T [i..j] from the index. Operations are efficient (like O(m log σ), O(m log n))

17 Suffix Tree (ST) of T=umulmu 5 7 dumulmum$ $ 4 lmu $ u m u lmu m n = 6 Σ ={$,d,l,m,n,u} σ = 6 m$ 3 9m$ lmu m$ 6 m$ $ 8m$ ulmu 5 O(n) construction (Weiner,973) Problem: Structure size: 2 T

18 Suffix Array (SA) of T=umulmu i T umulmu mulmu ulmu lmu mu u dumulmum$ umulmum$ mulmum$ ulmum$ lmum$ mum$ um$ m$ $

19 Suffix Array (SA) of T=umulmu i SA T $ dumulmum$ lmum$ lmu m$ mulmum$ mulmu mum$ mu ulmum$ ulmu um$ umulmum$ umulmu u Size: 5 T Binary search to answer count( P ) Match pattern from left to right: e.g. mum

20 Burrows-Wheeler Transform (BWT) of T i SA T BWT T m n u u u u u l l u m m m d $ m $ dumulmum$ lmum$ lmu m$ mulmum$ mulmu mum$ mu ulmum$ ulmu um$ umulmum$ umulmu u Well-compressible New search algorithm (Ferragina & Manzini, 2): Backward search Does not require SA.

21 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ Array C contains start of each character interval: $ a b c d r r rank(i, c) = # occurences of c in T BWT [, i). Search backwards for bar.

22 Backward Search i T BWT T a $abracadabrabarbara r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = C[r]+rank(sp, r) ep = C[r]+rank(ep +, r)

23 Backward Search i T BWT T a $abracadabrabarbara r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+rank(, r) ep = 5+rank(9, r)

24 Backward Search i T BWT T a $abracadabrabarbara r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+ ep = 5+rank(9, r)

25 Backward Search i T BWT T a $abracadabrabarbara r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+ = 5 ep = 5+4 = 8

26 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = C[a]+rank(sp, a) ep 2 = C[a]+rank(ep +, a)

27 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +rank(5, a) ep 2 = +rank(ep, a)

28 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +rank(5, a) ep 2 = +rank(ep, a)

29 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +6 ep 2 = +rank(9, a)

30 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +6 = 7 ep 2 = +8 = 8

31 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = C[b]+rank(sp 2, b) ep 3 = C[b]+rank(ep 2 +, b)

32 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+rank(7, b) ep 3 = 9+rank(ep, b)

33 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+rank(7, b) ep 3 = 9+rank(ep, b)

34 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+ ep 3 = 9+rank(9, b)

35 Backward Search i T BWT T a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Now search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+ = 9 ep 3 = 9+2 =

36 Conclusion Backward Search String matching on T only requires a structure which can answer rank queries on T BWT the C array Text T itself is not needed. We search a data structure represents T BWT compressed answers rank quickly

37 The Wavelet Tree (WT) Let X be a sequence of length n over alphabet Σ of size σ. Efficient Wavelet Tree operations: access(i) Recover X[i]. rank(i,c) How many times does c occur in X[, i)? select(i,c) Location of the i-th count(p) positions where P occurs. Space: n log σ + o(n log σ) + σ log n

38 WT operation rank arrd$rcbbraaaaaabba a$bbaaaaaabba a$aaaaaaa $ bbbb dc rrdrcr rrrr Only the bitvectors (dark gray) is stored code(a) = Reduce WT rank on bitvector rank aaaaaaaa c d rank(, a, WT ) = rank(rank(rank(,, b ɛ ) = 5,, b ) = 3,, b ) = 2

42 Rank on bitvectors n + o(n) bit space and constant time solution Jacobson: Space-efficient Static Trees and Graphs, FOCS 989. Two level schema: Divide bitvector in superblocks of size log 2 n and blocks of log n/2. Store absolute prefix sums for superblocks. Relative perfix sums for blocks. Four Russian-Trick (Lookup-Table counting set bits) for blocks. Practice today Since 28 CPUs support popcount operation no need for Lookup-Tables.

43 Improving WT space: Huffman shape arrd$rcbbraaaaaabba $c d$c rrd$rcbbrbb d$cbbbb d bbbb rrrr aaaaaaaa Char c codeword(c) $ a b c d r $ c Avg. depth: H (BWT ). Total space: nh + 2σ log n

44 Other WT parametrizations Shape: balanced, Huffman-shape, Hu-Tucker-shape, Wavelet Matrix for large alphabets (decreases O(σ log n) to O(log σ log n)) run-length compressed versions parameterized bitvector

45 Experimental results: operation count 26 S. GOG AND M.PETRI Time per character in (µs) XZ K =27 instance = WEB-64MB Index GZIP FM-HF-R 3 K FM-RLMN FM-HF-V FM-HF-L XZ K =27 K =63 instance = WEB-64GB Feature GZIP TBL BLT+HP K =63 K = K = Index size in (%) Index size in (%) (G & Petri, 24) Figure. Count time and space of our index implementations on input instances of different size with

46 Experimental results: construction Resource diagram for the construction of a Self-Index for 2MB of English text.

48 More WT operations: Query point grids n σ Operations: count(x,x,y,y ) # points in rectangle [x, x ][y, y ]. report(x,x,y,y ) Report points in rectangle [x, x ][y, y ].

49 WT application: Top-k document retrieval T = b = ω 2 ω ω 3 ω 3 # ω ω ω 4 ω # ω ω 4 ω 3 ω # ω 5 ω 5 # ω CSA position= D =

50 WT application: Top-k document retrieval Represent document array D as wavelet tree First explore nodes, which contain the maximal interval Example: Search top-2 documents (frequency based ranking) Top documents containing ω :

53 WT application: Top-k document retrieval Represent document array D as wavelet tree First explore nodes, which contain the maximal interval Example: Search top-2 documents (frequency based ranking) Top documents containing ω : (3 times)

54 WT application: Top-k document retrieval Represent document array D as wavelet tree First explore nodes, which contain the maximal interval Example: Search top-2 documents (frequency based ranking) Top documents containing ω : (3 times), 2 (2 times)

55 WT application: Top-k document retrieval Other results: 2n + o(n) structure answers document frequency in constant time (Sadakane 27) Use RMQs for document listing (Sadakane 27) Prestore selected intervals (Hon et al. 29). Use skeleton Suffix Tree for mapping.

57 Compressed Bitvectors: SD-array (Elias-Fano coding) Applications Graphs (debruijn subgraph, web graph,...) Lists (Inverted Lists,...) monotone sequence X with x x... x m n. Lower l = log n m bits stored explicitly. Upper bits as sequence H of unary encoded gaps Uses 2 + log n m bits per element Constant time access X by addding o(m) extra bits to H.

58 Compressed Bitvectors: SD-array (Elias-Fano coding) X = δ = H = L = 2 3 X[3]=rank (H, select (H, 4)) L[3] = = 7

59 Compressed Bitvectors: SD-array (Elias-Fano coding) Interpret X as positions of set bits in a bitvector b. Constant time acccess, rank, select on b. Well suited for sparse bitvectors. b = H = L = select(b, 4) = rank (H, select (H, 4)) L[3] = = 7 rank(b, i) and access(b, i): use select on H to select right upper part + scan left for entry in L

60 Balanced Parentheses Sequences (BPS) Represent BPS as bitvector. (()()(()())(()((()())()()))()((()())(()(()()))())) find_open(i) Find matching closing paren of BPS[i]. find_close(p) Find matching opening paren of BPS[i]. enclose(i) Find tightest paren pair which encloses pair (i,find_close(i)) and more (double_enclose(i,j), rank(i), select(i),... ) can be answered in constant time by just adding o(n) space. (Munro 996, Geary et al. 24 & 26)

61 Balanced Parentheses Sequences (BPS) Application: Succinct representation of trees 5 7 $ u dumulmum$ lmu m lmu u m lmu m$ $ ulmu m$ m$ m$ m$ 4 $ tree uncompressed O(n log n) bits BPS dfs = (()()(()())(()((()())()()))()((()())(()(()()))())) compressed 4n bits

67 Balanced Parentheses Sequences (BPS) Application: Succinct representation of trees $ u 5 dumulmum$ lmu m 37 m$ 4 $ lmum 5 u lmu m$ $ ulmu m$ m$ m$ tree uncompressed O(n log n) bits BPS dfs = (()()(()())(()((()())()()))()((()())(()(()()))())) compressed 4n bits

68 Balanced Parentheses Sequences (BPS) Application: Succinct representation of trees $ u 5 dumulmum$ lmu m 37 m$ 4 $ lmum 5 u lmu m$ 6 $ ulmu m$ m$ m$ tree uncompressed O(n log n) bits BPS dfs = (()()(()())(()((()())()()))()((()())(()(()()))())) compressed 4n bits

72 Balanced Parentheses Sequences (BPS) Application: Succinct representation of trees $ u 5 dumulmum$ lmu m m$ 4 $ lmum 5 u lmu m$ 36 6 $ ulmu 5 m$ m$ m$ tree uncompressed O(n log n) bits BPS dfs = (()()(()())(()((()())()()))()((()())(()(()()))())) compressed 4n bits

73 Balanced Parentheses Sequences (BPS) Application: Range Minimum Queries Array X of n elements from a well-ordered set. rmq(i,j) returns position of minimum element in X[i, j]. 2n + o(n) bits and O() time solution (Fischer & Heun 2) X = rmq(2, 6)=3

74 Compressed Suffix Trees (CST) $ Full functionality root() 4 is leaf (v) parent(v) degree(v) child(v, c) 3 2 select child(v, i) 4 2 sibling(v) depth(v) lca(v, w) sl(v), wl(v, c) (()()(()())(()((()())()()))()((()())(()(()()))())) id(v), inv_id(i) Size: CSA + LCP + BPS, i.e. about original text size dumulmum$ 9m$ m$ $ $ lmu lmu u m m$ u m$ lmu m 8m$ ulmu

75 Compressed Suffix Trees (CST) Construction Resource diagram for the construction of a CST for 2MB of English text.

77 The Succinct Data Structure Library SDSL Input from about 4 research papers Fast and space-efficient construction Index structures available for byte and integer alphabets Well-optimized (e.g. hardware POPCOUNT, hugepages,... ) Easy configuration of myriads of CSAs, CSTs, WTs Fast prototyping Tests, benchmarks, tutorial, cheat sheet

79 Conclusion Limitations/Challenges of Succinct Structures: Often only work for a static setting Memory access pattern often non-local Implementation from scratch difficult Construction is often the bottleneck Algorithms have to be adapted to work fast with succinct structures

80 Conclusion Advantages of Succinct Structures: Allow indexation of larger data sets (about - larger compared to classical indexes). Adapt to the data: E.g. capture repetitions. Often provide more functionality. More functionality often produces conceptional easier solutions. Not limited to text indexing.

81 Conclusion Many applications heavily depend on Indexes. Self-Indexes made it possible to process Next Generation Sequencing in Genomics and Life Sciences. Why? Only the speed of self indexes matches the amazing throughput of sequencing technologies.

82 Thank you for your attention. Thank MASTODONS