Information Retrieval. Exercises Skip lists Positional indexing Permuterm index

Size: px

Start display at page:

Download "Information Retrieval. Exercises Skip lists Positional indexing Permuterm index"

Jeffery Richardson
7 years ago
Views:

1 Information Retrieval Exercises Skip lists Positional indexing Permuterm index

2 Faster merging using skip lists

3 Recall basic merge Walk through the two postings simultaneously, in time linear in the total number of postings entries Brutus Caesar If the list lengths are m and n, the merge takes O(m+n) operations. Can we do better? Yes, if index isn t changing too fast.

4 Augment postings with skip pointers (at indexing time) Why? To skip postings that will not figure in the search results. Where do we place skip pointers? No skip pointers for intermediate results

5 Recall: basic merge algorithm pointer version The algorithm is Fig. 1.6 at of Chapter 1 of Raghavan's book. Modify to use skip pointers

6 Merge algorithm with skip lists The algorithm is Fig of Chapter 2 of Raghavan's book.

7 Placing skips Simple heuristic: for postings of length L, use L evenly-spaced skip pointers. This ignores the distribution of query terms. Easy if the index is relatively static; harder if L keeps changing because of updates.

8 Placing skips Es. 2.6 on IIR. We have a two-word query. For one term the postings list consists of the following 16 entries: [4,6,10,12,14,16,18,20,22,32,47,81,120,122,157,180] and for the other it is the one entry postings list: [47]. Work out how many comparisons would be done to intersect the two postings lists with the following two strategies. Briefly justify your answers: a. Using standard postings lists b. Using postings lists stored with skip pointers, with a skip length of P

9 Other operators Adapted from Es. 2.5 on IIR. Are skip pointers useful for queries of the form x OR y? Motivate your answer Are skip pointers useful for queries of the form x AND NOT y? Motivate your answer

10 Other operators Adapted from Ex. 2.5 on IIR. Are skip pointers useful for queries of the form x OR y? Motivate your answer No, because you have to visit every docid of each postings list Are skip pointers useful for queries of the form x AND NOT y? Motivate your answer In principle you do less comparisons (consider ex. 2.6 in previous slide). In practice, you still have to take all the elements to return and put them into a result list, so no gain, at least asymptotically

11 Phrase queries and positional indexes

12 Phrase queries Want to answer queries such as stanford university as a phrase Thus the sentence I went to university at Stanford is not a match. The concept of phrase queries has proven easily understood by users; about 10% of web queries are phrase queries No longer suffices to store only <term : docs> entries

13 Solution 2: Positional indexes Store, for each term, entries of the form: <number of docs containing term; doc1: position1, position2 ; doc2: position1, position2 ; etc.>

14 Processing a phrase query Extract inverted index entries for each distinct term: to, be, or, not. Merge their doc:position lists to enumerate all positions with to be or not to be. to: 2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191;... be: 1:17,19; 4:17,191,291,430,434; 5:14,19,101;... Same general method for proximity searches

15 Exercise 2.10 in IIR Consider the following fragment of a positional index with the format: <term>: document: position, position,... ; document: position, Gates: 1: 3 ; 2: 6 ; 3: 2,17 ; 4: 1 ; IBM: 4: 3 ; 7: 14 ; Microsoft: 1: 1 ; 2: 1,21 ; 3: 3 ; 5: 16,22,51 ; The /k operator, word1 /k word2 finds occurrences of word1 within k words of word2 (on either side), where k is a positive integer argument. Thus k = 1 demands that word1 be adjacent to word2. a. Describe the set of documents that satisfy the query Gates /2 Microsoft. b. Describe each set of values for k for which the query Gates /k Microsoft returns a different set of documents as the answer.

16 Exercise 2.12 in IIR (revised) Consider the adaptation of the basic algorithm for intersection of two postings lists (Figure 1.6) to the one in Figure 2.12, which handles proximity queries. A naive algorithm for this operation could be O(PL max2 ), where P is the sum of the lengths of the postings lists (i.e., the sum of term frequencies) and L max is the maximum length of a document (in tokens) a. Prove the trivial O(PL max 2 ) bound b. Go through this algorithm carefully and explain how it works. c. What is the complexity of this algorithm? Justify your answer carefully. d. For certain queries and data distributions, would another algorithm be more efficient? What complexity does it have?

17 Exercise 2.12 in IIR (revised) Find term pairs in doc that are no more than k away

18 Positional index size Can compress position values/offsets. Nevertheless, this expands postings storage substantially

19 Positional index size Need an entry for each occurrence, not just once per document Index size depends on average document size Average web page has <1000 terms SEC filings, books, even some epic poems easily 100,000 terms Consider a term with frequency 0.1% Document size ,000 Postings 1 1 Positional postings 100 Why? 1

20 Resources for today s lecture IIR Chapters

EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling

EECS 395/495 Lecture 3 Scalable Indexing, Searching, and Crawling Doug Downey Based partially on slides by Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze Announcements Project proposals due