Information Retrieval Elasticsearch

Transcription

1 Information Retrieval Elasticsearch

2 IR Information retrieval (IR) is the activity of obtaining information resources relevant to an information need from a collection of information resources. Searches can be based on metadata or on full-text (or other content-based) indexing. Wikipedia

3 Concepts term t : a noun or compound word used in a specific context tf (t in d) : term frequency in a document; measure the number of times term t appears in the currently scored document d idf (t) : inverse document frequency measures how often the term appears across the index: common or rare obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.

4 Concepts tf idf, (term frequency inverse document frequency) is a numerical statistic that reflects how important a word is to a document in a collection or corpus. It is used as a weighting factor in information retrieval and text mining. The tf-idf value increases proportionally to the number of times a word appears in the document, adjusted for the fact that some words appear more frequently in general.

5 Concepts Then tf idf is calculated as and

6 Apache Lucene Fast, high performance, scalable search/ir library Open source Indexing and Searching Inverted Index of documents Provides advanced Search options like synonyms, stopwords, based on similarity, proximity.

7 Lucene Internal Representation Credit:

8 Lucene Based on documents model Index contains documents Each document consist of fields Each Field has attributes field data type (FieldType) way to handle the content (Analyzers, Filters) Stored or indexed field (stored="true") or (indexed="true")

9 Indexing Pipeline Analyzer : creates tokens using a Tokenizer and/or applying Filters (Token Filter) Each field can define an Analyzer at index time/query time or both at same time

10 Elasticsearch - Introduction Enterprise Search platform for Apache Lucene Open source Highly reliable, scalable, fault tolerant Support distributed Indexing, Replication, and load balanced querying

11 Elasticsearch - Features Distributed (multiple nodes) RESTful search server (GET, PUT, POST, DELETE) Document oriented, full text (JSON format) Schema free Easy to scale horizontally Real time analytics Multi tenancy (multiple indexes)

12 APIs HTTP RESTful Api Java Api Clients perl, python, php, ruby,.net and more All APIs perform automatic node operation rerouting

13 Cluster Architecture Source:

14 Index Request Source:

15 Search Request Source:

16 Logstash - Architecture Source:

17 Logstash life of an event Input Filters Output Filters are processed in order of config file Outputs are processed in order of config file Input: Input stream File input (tail) Log4j Redis Syslog and many more

18 Kibana

19 Source:

20 Source:

21 Analytics Analytics source : Kibana.org based on ElasticSearch and Logstash Image Source :