Lucene in Action OTIS GOSPODNETIC ERIK HATCHER MANNING. Greenwich (74 w. long.)

Transcription

1 2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Lucene in Action OTIS GOSPODNETIC ERIK HATCHER i\ 11 MANNING Greenwich (74 w. long.)

2 contents foreword xvii preface xix acknowledgments xxii about this book xxv PART 1 CORE LLCENE, Meet Lucene Evolution of information organization and access Understanding Lucene 6 What Lucene is 7 What Lucene can do for you 7 History of Lucene 9 Who uses Lucene 10 Lucene ports: Perl, Python, C++,.NET, Ruby Indexing and searching 10 What is indexing, and why is it important? 10 What is searching? Lucene in action: a sample application 11 Creating an index 12 * Searching an index 15

3 1.5 Understanding the core indexing classes 18 indexwriter 19 Directory 19 * Analyzer 19 Document 20 Field Understanding the core searching classes 22 IndexSearcher 23 * Term 23 Query 23 TermQuery 24 Hits Review of alternate search products 24 IR libraries 24 Indexing and searching applications 26 Online resources Summary 27 Indexing Understanding the indexing process 29 Conversion to text 29 Analysis 30 Index writing Basic index operations 31 Adding documents to an index 31 * Removing Documents from an index 33 * Undeleting Documents 36 * Updating Documents in an index Boosting Documents and Fields Indexing dates Indexing numbers Indexing Fields used for sorting Controlling the indexing process 42 Tuning indexing performance 42 In-memory indexing: RAMDirectory 48 * Limiting Field sizes: maxfieldlength Optimizing an index Concurrency, thread-safety, and locking issues 59 Concurrency rules 59 * Thread-safety 60 Index locking 62 * Disabling index locking Debugging indexing Summary 67

4 Adding search to your application Implementing a simple search feature 69 Searching for a specific term 70 Parsing a user-entered query expression: QueryParser Using IndexSearcher 75 Working with Hits 76 Paging through Hits 77 Reading indexes into memory Understanding Lucene scoring 78 Lucene, you got a lot of 'splainin to do! Creating queries programmatically 81 Searching by term: TermQuery 82 Searching within a range: RangeQuery 83 Searching on a string: PrefixQuery 84 Combining queries: BooleanQuery 85 * Searching by phrase: PhraseQuery 87 * Searching by wildcard: WildcardQuery 90 Searching for similar terms: FuzzyQuery Parsing query expressions: QueryParser 93 Query.toString 94 Boolean operators 94 Grouping 95 Field selection 95 * Range searches 96 Phrase queries 98 Wildcard and prefix queries 99 Fuzzy queries 99 Boosting queries 99 To Query Parse or not to Query Parse? Summary 100 Analysis Using analyzers 104 Indexing analysis 105 QueryParser analysis 106 Parsing versus analysis: when an analyzer isn't appropriate Analyzing the analyzer 107 What's in a token? 108 * TokenStreams uncensored 109 Visualizing analyzers 112 * Filtering order can be important Using the built-in analyzers 119 Stop Analyzer 119 * StandardAnalyzer Dealing with keyword fields 121 Alternate keyword analyzer "Sounds like" querying 125

5 xii CONTENTS 4.6 Synonyms, aliases, and words that mean the same 128 Visualizing token positions Stemming analysis 136 Leaving holes 136 * Putting it together 137 Hole lot of trouble Language analysis issues 140 Unicode and encodings 140 Analyzing non-english languages 141 Analyzing Asian languages 142 Zaijian Nutch analysis Summary 147 Advanced search techniques Sorting search results 150 Using a sort 150 * Sorting by relevance 152 Sorting by index order 153 Sorting by afield 154 * Reversing sort order 154 Sorting by multiple fields 155 * Selecting a sorting field type 156 Using a nondefault locale for sorting 157 Performance effect of sorting Using PhrasePrefixQuery Querying on multiple fields at once Span queries: Lucene's new hidden gem 161 Building block of spanning, SpanTermQuery 163 * Finding spans at the beginning of a field 165 * Spans near one another 166 Excluding span overlap from matches 168 Spanning the globe 169 SpanQuery and QueryParser Filtering a search 171 Using DateFilter 171 Using QueryFilter 173 Security filters 174 * A QueryFilter alternative 176 Caching filter results 177 Beyond the built-in filters Searching across multiple Lucene indexes 178 Using MultiSearcher 178 * Multithreaded searching using ParallelMultiSearcher 180

6 xm 5.7 Leveraging term vectors 185 Books like this 186 «What category? Summary 193 Extending search Using a custom sort method 195 Accessing values used in custom sorting Developing a custom HitCollector 201 About BookLinkCollector 202 Using BookLinkCollector Extending QueryParser 203 Customizing QueryParser's behavior 203 Prohibiting fuzzy and wildcard queries 204 Handling numeric field-range queries 205 Allowing ordered phrase queries Using a custom filter 209 Using a filtered query Performance testing 213 Testing the speed of a search 213 * Load testing 217 QueryParser again! 218 * Morals of performance testing Summary 220 PART 2 AP < )«) Parsing common document formats Handling rich-text documents 224 Creating a common DocumentHandler interface Indexing XML 226 Parsing and indexing using SAX 227 * Parsing and indexing using Digester Indexing a PDF document 235 Extracting text and indexing using PDFBox 236 Built-in Lucene support Indexing an HTML document 241 Getting the HTML source data 242 Using JTidy 242 Using NekoHTML 245

7 7.5 Indexing a Microsoft Word document 248 Using POI249 «Using TextMining.orgs API Indexing an RTF document Indexing a plain-text document Creating a document-handling framework 254 FileHandler interface 255 * ExtensionFileHandler 257 Filelndexer application 260 * Using Filelndexer 262 Filelndexer drawbacks, and how to extend the framework Other text-extraction tools 264 Document-management systems and services Summary 265 Tools and extensions Playing in Lucene's Sandbox Interacting with an index 269 lucli: a command-line interface 269 Luke: the Lucene Index Toolbox 271 LIMO: Lucene Index Monitor Analyzers, tokenizers, and TokenFilters, oh my 282 SnowballAnalyzer 283 * Obtaining the Sandbox analyzers Java Development with Ant and Lucene 284 Using the <index> task 285 Creating a custom document handler 286 Installation JavaScript browser utilities 290 JavaScript query construction and validation 291 * Escaping special characters 292 Using JavaScript support Synonyms from WordNet 292 Building the synonym index 294 * Tying WordNet synonyms into an analyzer 296 * Calling on Lucene Highlighting query terms 300 Highlighting with CSS 301 Highlighting Hits Chaining filters Storing an index in Berkeley DB 307 Coding to DbDirectory 308 * Installing DbDirectory 309

8 8.10 Building the Sandbox 309 Check it out 310 Ant in the Sandbox Summary 311 Lucene ports Ports' relation to Lucene CLucene314 Supported platforms 314 API compatibility 314 Unicode support 316 Performance 317 Users dotlucene3l7 API compatibility 317 Index compatibility 318 Performance 318 Users Plucene318 API compatibility 319 Index compatibility 320 Performance 320 Ifam 32(9 9.5 Lupy 320 API compatibility 320 Index compatibility 322 Performance 322 «Users PyLucene 322 API compatibility 323 Index compatibility 323 Performance 323 Users Summary 324 Case studies Nutch: "The NPR of search engines" 326 More in depth 327 Other Nutch features Using Lucene at jguru 329 Topic lexicons and document categorization 330 * Search database structure 331 * Index fields 332 Indexing and content preparation 333 * Queries 335 JGuruMultiSearcher 339 Miscellaneous Using Lucene in SearchBlox 341 Why choose Lucene? 341 SearchBlox architecture 342 Search results 343 Language support 343 Reporting Engine 344 * Summary 344

9 XVI CONTENTS 10.4 Competitive intelligence with Lucene in XtraMind's XM- InformationMinder 344 The system architecture 347 How Lucene has helped us Alias-i: orthographic variation with Lucene 351 Alias-i application architecture 352 Orthographic variation 354 The noisy channel model of spelling correction 355 * The vector comparison model of spelling variation 356 A subword Lucene analyzer 357 * Accuracy, efficiency, and other applications 360 Mixing in context 360 * References Artful searching at Michaels.com 361 Indexing content 362 Searching content 367 Search statistics 370 Summary I love Lucene: TheServerSide 371 Building better search capability 371 High-level infrastructure 373 * Building the index 374 Searching the index 377 * Configuration: one place to rule them all 379 Web tier: TheSeeeeeeeeeeeerverSide? 383 Summary Conclusion 385 appendix A: Installing Lucene 387 appendix B: Lucene index format 393 appendix C: Resources 408 index 415