Text Analytics. Assignment 2: Full text search. Ulf Leser

Transcription

1 Text Analytics Assignment 2: Full text search Ulf Leser

2 Full text search On the website, you find a link to a corpus of approximately 100MB Excerpt from 1977 Medline, do not redistribute Most groups: Write a program which finds arbitrary words in this corpus as fast as possible finds arbitrary phrases as fast as possible In other words: Implement Boolean IR queries (AND, phrases) Remarks A word is series of characters delimited by a blank or a linefeed A phrase is a series of words, e.g. holy grail, duchenne s musclar dystrophy Ignore punctuation marks and XML markup: Treat them as nonblank character All searches must be case insensitive

3 Document Preparation The XML file contains citations Between <MedlineCitation > and </MedlineCitation> contain an abstract text, the rest is only citation Consider all tags as token separators, delete the tags, keep all the words Special treatment: <PMID> </PMID> This is the document number which must be returned Each of the citations must be treated as one document Which may match a query or not Identified by the PMID Searches must search across all words of a citation Except the XML tags Do not care about stop words or stemming

4 Program The search program must read the query from the command line Either a list of terms (AND)./mysearch disease obesity function Or a list of terms in (phrase search)./mysearch obesity is a disease No combination of these two search modes required I will call the program subsequently for different queries Yes, the OS will cache any file that is read during the first start which might make subsequent calls faster Exploiting this is allowed Benchmarking is difficult

5 Results For each query, return the following (on STDDEV) The PubMed ID of the abstract The start position (in words) and the word of all the matching words PMID= , disease 120, disease 83, obesity 6, function PMID= For phrase search, return only the position of the first match(es) PMID= If an abstract has more than one match, give one result for each match Finally, return the total number of documents that matched

6 Competition You can always check the correctness of your results using GREP Be ambitious: Implement a fast search Build indexes as much as you like The best group (of new queries!) wins a big bar of chocolate The program must be ready to run on AMSEL

7 Exception Two groups do it differently Perform the same task using Lucene Install package Index the corpus Perform searches Provide the same STDDEV search interface Give the same results Who wants to use Lucene? Group 10 Group 6

8 Deliverables By Wednesday, , 23:59 o clock Three weeks Send by mail as ASCII JAVA source code as JAR The query script Results and query times on AMSEL for the following phrase queries dicyclocarbodimide reduce the appeal of experiment protein complex was observed duchenne's muscular We will perform a live test of your code during the Praktikum at the Have login data ready and be prepared to answer some queries Install everything on AMSEL

9 How To: Inverted Files Very simple and effective index structure for terms K in a collection D of documents Bag of words approach We give up on order of terms in docs (reappears later) We cannot reconstruct docs based on index only Start from docs contain terms (~docs) and invert to terms appear in docs (=index) docs t1 t2 t3 D D D D D D D D Terms D1 D2 D3 D4 D5 D6 D7 t t t

10 Building an Inverted File [Andreas Nürnberger, IR-2007] Term Doc # now 1 is 1 the 1 time 1 for 1 all 1 Doc1: good 1 Now is the time men 1 to 1 for all good men come 1 to come to the aid to 1 the 1 of their country. aid 1 of 1 their 1 country 1 Sort it 2 Doc2: was 2 It was a dark and a 2 dark 2 stormy night in and 2 the country stormy 2 night 2 manor. The time in 2 was past midnight. the 2 country 2 manor 2 the 2 time 2 was 2 past 2 Ulf Leser: Text Analytics, Praktikum, midnight Sommersemester Term Doc # a 2 aid 1 all 1 and 2 come 1 country 1 country 2 dark 2 for 1 good 1 in 2 is 1 it 2 manor 2 men 1 midnight 2 night 2 now 1 of 1 past 2 stormy 2 the 1 the 1 the 2 the 2 their 1 time 1 time 2 to 1 to 1 was 2 was 2 Merge a 2 aid 1 all 1 and 2 come 1 country 1,2 dark 2 for 1 good 1 in 2 is 1 it 2 manor 2 men 1 midnight 2 night 2 now 1 of 1 past 2 stormy 2 the 1,2 their 1 time 1,2 to 1,2 was 1,2

11 Boolean Retrieval We can now efficiently implement Boolean queries For each query term k i, look-up doc-list D i containing k i Evaluate query in the usual order k i k j : D i D j k i k j : D i D j NOT k i : D\D i Example (time AND past AND the) OR (men) = (D time D past D the ) D men = ({1,2} {2} {1,2}) {1} = {1,2} a 2 aid 1 all 1 and 2 come 1 country 1,2 dark 2 for 1 good 1 in 2 is 1 it 2 manor 2 men 1 midnight 2 night 2 now 1 of 1 past 2 stormy 2 the 1,2 their 1 time 1,2 to 1,2 was 1,2

12 Challenges Process the XML to generate indexable documents Build the index efficiently Will not be evaluated Doc-list per term has no constant length Efficient computation of doc-lists per term Might be large searching for the Efficient implementation of AND operator Doc-list intersection Implementation of phrase search How?