Open Source Techniques push Enterprise Search & Search Driven Applications and especially foster the application of Text Analytics

Transcription

1 Open Source Techniques push Enterprise Search & Search Driven Applications and especially foster the application of Text Analytics Exploring the Future of Enterprise Search, IPTS, Seville, October 2011 Dr. Christoph Goller, Director Research, IntraFind AG

2 Emergence of Lucene and Solr Lucene / Solr Built in late 90 s by Doug Cutting. Apache release 2001 State of the art Java library for indexing and ranking, many ports Wide acceptance by 2005, mostly by technology organizations and inside products Solr (2006): Lucene as Web Service, Converters, Connectors, UIMA and Carrot2 integration Open Source: developers mainly from the US and Europe 4,000+ sites Apple, Cisco, EMC, HP, IBM, LinkedIn, MySpace, CNET, Netflix, Salesforce, Twitter, Ebay, Immoscout, Gov, Wikipedia Exploring the Future of Enterprise Search, Seville,

3 Lucene/Solr Strengths Strengths: Best practice segmented index (like Google, Fast) Best practice, flexible ranking (term/field/doc boosts, function queries, custom scoring ) Best overall query performance and complete query capabilities (Boolean Operations, Wildcards, Similarity Search, Efficient Range Queries, Geo Search, Synonyms, Spell-Check, Hit-Snippets, Hit- Highlighting, Proximity Operators ) Multilingual Stemmers, Filters, Memory Mapped Indexes, Near Real- Time Search, Cache Management, Replication, Faceting, Grouping, Autocomplete Rapid Innovation, Extensible Architecture, complete control (open source) CORE TECHNOLOGY AS GOOD OR BETTER THAN ANY OTHER AND OPEN SOURCE Exploring the Future of Enterprise Search, Seville,

4 Lucene/Solr Weaknesses Weaknesses: No formal support, limited access to training & consulting Lack of stringent integrated QA Certain features for Enterprise Search missing: Secure Search Graphical user- and administration interfaces Integrated search of structured and un-structured content Connectors and stable Converters Viewer Component (preview or view docs in their original layout) Search Quality (intent, semantics, NLP, Linguistics)? Text Analytics? Exploring the Future of Enterprise Search, Seville,

5 Competitive Situation Fulltext Search has become a commodity Market strength and features of established Enterprise Search Providers will keep them going a while but Very hard to justify high prices, especially for large applications Very hard to justify closed and proprietary technology Good Enterprise Search Solutions have to offer more than just full-text search Companies like Intrafind can concentrate on solving the real problems in Enterprise Search Exploring the Future of Enterprise Search, Seville,

6 IntraFind Software AG Exploring the Future of Enterprise Search, Seville,

7 IntraFind Business Model Founding of the company: October 2000 More than 700 customers mainly in Germany, Austria, and Switzerland Partner Network (> 30 VAR & embedding partners) Employees: 20 Lucene Committers: B. Messer, C. Goller Our Open Source Search Business: Product Company: ifinder, Topic Finder, Knowledge Map, Tagging Service, Products are a combination of Open Source Components and in-house Development Support (up to 7x24), Services, Training, Stable API Relevancy, automatic generation of meta information Linguistic Analyzers for most European Languages Named Entity Recognition Text Classification Semantic Search Clustering Exploring the Future of Enterprise Search, Seville,

8 Morphological Analyzer vs. Stemming Morphological Analyzer: Lemmatizer: maps words to their base forms English going -> go (Verb) bought -> buy (Verb) bags -> bag (Noun) bacteria -> bacterium (Noun) German lief -> laufen (Verb) rannte -> rennen (Verb) Bücher -> Buch (Noun) Taschen -> Tasche (Noun) Decomposer: decomposes words into their compounds Kinderbuch (children s book) -> Kind (Noun) Buch (Noun) Versicherungsvertrag (contract of insurance) -> Versicherung (Noun) Vertrag (Noun) Holztisch (wooden table), Glastisch (table made of glass) Stemmer: usually simple algorithm going -> go king -> k??????????? Messer -> mess?????? Exploring the Future of Enterprise Search, Seville,

9 Advantages of Morphological Analysis Combines high Recall with high Precision for Search Applications Improves subsequent statistical methods Better suited as descriptions for semantic search / faceting / clustering than artificial stems, categories used for phrase detection Reliable Lookup in Lexicon Resources Thesaurus / Ontologies Cross-lingual search Available Languages: German, English, Spanish, French, Italian, Dutch, Russian, Polish, Serbo-Croatian, Greek, Chinese, Arabian, Pasthu Exploring the Future of Enterprise Search, Seville,

10 Named Entity Recognition (NER) Automated extraction of information from unstructured data People names Company names Brands from product lists Technical key figures from technical data (raw materials, product types, order IDs, process numbers, eclass categories) Names of streets and locations Currency and accounting values Dates Phone numbers, addresses, hyperlinks Search query: Which people / companies strongly correlate with each other? Exploring the Future of Enterprise Search, Seville,

11 Named Entities: Applications in Search Applications: Facets Search for Experts Additional Query Types Index Structure: Additional Tokens on the same position: N_PersonName N_Peter Müller N_Peter N_Müller Search for a person named Brown Search for a company near founded and Bill Gates Question Answering / Natural Language Queries (Demo) Exploring the Future of Enterprise Search, Seville,

12 Text Classification Automatic Assignment of Documents to Topics based on their content Learning Phase Example Documents Definition of Topics 1 N Topic Learner Classifier Rules (Parameters) Classification Phase Topic Association New Document Topic Classifier Exploring the Future of Enterprise Search, Seville,

13 Applications of Text Classification News: Newsletter-Management System Spam-Filtering Mail / Classification Product Classification (Online Shops), ECLASS /UNSPSC Subject Area Assignment Libraries & Publishing Companies Opinion Mining / Sentiment Detection Part of our Tagging Services Exploring the Future of Enterprise Search, Seville,

14 Knowledge Map Knowledge Map = Graphic way of visualizing enterprise knowledge Quick access to existing enterprise knowlege via setting filters or entering search terms Browse (navigate and explore) through your enterprise content Enables a 360 degree view on the existing enterprise knowledge Consideration of individual user authorizations Combine available meta-data (SAP invoice number, order number) and entities from unstructured data (e.g. minutes of SAP Team Meeting in PDF) Exploring the Future of Enterprise Search, Seville,

15 Enterprise Search via the Knowledge Map Tab browsing Full-text search combined with Knowledge Maps Search-while-you-type based on filters (in this example: filter author ) Content sources provide filtering capabilities within the Knowledge Map Exploring the Future of Enterprise Search, Seville,

16 Semantic Search Compute (on the fly) strongly correlated words and phases Presented as Contextor / Facet Used to expand or restrict search OntologyNet results after a search for ifinder Exploring the Future of Enterprise Search, Seville,

17 Questions? Dr. Christoph Goller Director Research Phone: Fax: Web: IntraFind Software AG Fraunhofer Strasse Planegg Germany Interested in becoming an IntraFind Partner? Just contact us! Exploring the Future of Enterprise Search, Seville,