A SRW/U Compliant Search Service for Fedora

Transcription

1 A SRW/U Compliant Search Service for Fedora Dr. Michael Hoppe Matthias Razum FIZ Karlsruhe Open Repositories 2008 Southampton

2 The escidoc Project Joint project of the Max Planck Society and FIZ Karlsruhe, funded by the German Federal Ministry of Education and Research Aims to build a next-generation e-research platform Set of loosely coupled services (service-oriented architecture) including: object storage search and indexing statistics and reporting persistent identification workflows validation and transformation

3 The escidoc Indexing and Search Service Standards based indexing and search service Integrates existing open source software Apache Lucene (Apache) Fedora s GSearch (Technical University of Denmark) SRW/U server (Online Computer Library Center) SRWLucene (OCKHAM Initiative of the Oregon State University) Supports arbitrary metadata profiles Configurable search hit format

4 Indexing OCLC SRW/U Search SRWLucene XML Stylesheet GSearch Index Information Document Analyzer Adaptor escidoc Object Manager Tokenizer Stemmer other filters API-A Fedora Lucene Index

5 Indexing with GSearch GSearch was originally designed and implemented by Gert Schmeltz-Pedersen (Technical University of Denmark) and is now part of the Fedora distribution Accesses Fedora via API-A to retrieve the FOXML for a specific PID Transforms FOXML objects via XSLT into an intermediate document containing all the relevant information that should later on be searchable index information document The index information document is then further processed by the Lucene indexer to create or update the Lucene index Whenever an object gets publicly available, GSsearch is triggered via a messaging system

6 GSearch Customizations for escidoc No direct access from GSearch to Fedora Each escidoc object consists of several Fedora objects GSearch should retrieve whole object representations a lightweight adapter translates GSearch s API-A calls into escidoc ObjectManager calls The adapter returns escidoc Item XML instead of FOXML We have integrated Apache POI libraries to enable text extraction from MS Word documents Additionally, we have created customized analyzers for GSearch

7 GSearch Configuration Stylesheet transforms escidoc Item XML into an index information document request to GSearch can include name of the stylesheet to use Stylesheet may be stored outside of GSearch Analyzers Have to extend lucene.analysis.analyzer Have to be included into gsearch.war

8 Searching with SRW/SRU OCLC s SRW/U open source project implements the SRW/U protocol, including CQL provides a Java-based server The SRWLucene addon for OCLC s SRW/U implementation allows for an integration with Lucene created by the OCKHAM Initiative of the Oregon State University translates cql into lucene-query-language queries lucene-index and generates search-hits

9 Customizations of SRW/U for escidoc performance issues We switched off caching of the search result set for performance reasons method does not scale well for large result sets repeated searches for paging are more resource-efficient compared to result set caching however, built in sorting functionality doesn t work anymore

10 Customizations of SRWLucene for escidoc generating hit-display SRW/U returns a hit list as an XML structure conforming to the SRW/U schema Each hit is represented as an XML node with a self-defined XML structure, which contains the metadata needed to display the hit appropriately Nodes are stored within the index for faster retrieval hit lists are generated out of the index time-consuming access to the repository for each hit can be omitted

11 Customizations of SRWLucene for escidoc extentions Sorter define custom sorter to use in configuration for SRW-Database Analyzers define Analyzer to use in configuration for SRW database. Analyzer has to extend lucene.analysis.analyzer Analyzer has to be included into srw.war Use the same analyzers as for indexing! Highlighter define custom highlighter to use in configuration for SRW- Database. Highlighter has to extend SrwHighlighter interface Highlighters have to be included into srw.war

12 Re-using the Indexing Service With Fedora and FOXML objects Add Fedora URL to the configuration file Adapt the stylesheet to create an appropriate index information document With other system and proprietary XML objects Implement an adapter that implements some API-A methods (describerepository, getdatastreamdissemination, export) and returns any XML structure Create your own stylesheet that transforms the returned XML object into an appropriate index information document

13 Re-using the Search Service Search service is standalone, does not require fedora Needs existing Lucene-index Configure existing, stored index-field as data-field for searchresult display

14 Conclusion We have created a standards-based indexing and search service by re-using existing open-source software The service is re-usable and configurable It may be used independently from the escidoc Infrastructure The software is available as open source, see