Elasticsearch for Lua Developers. Pablo Musa

Transcription

1 Elasticsearch for Lua Developers Pablo Musa

2 + +

3 Me Pablo Musa Educational Elastic

4 Which student? 5 interested students 3 very good proposals Key Points: - Background (Lua, Elasticsearch, Open Source) - Availability (how many hours per week) - Future (development likelihood after GSoC) 4

5 Active participant in CTFs. Also, a 3rd year computer science student at IIT Roorkee developer and challenge setter of Backdoor CTF. ( Mostly self-taught himself by reading books and online articles. Dhaval Kapil Passionate about networking, security and development

6 Agenda Elasticsearch Elasticsearch & Lua elasticsearch-lua WERpapers Elastic 6

7 Elasticsearch is...

8 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful full text search engine with soft real-time search and analytics capabilities 8

9 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Apache 2.0 License 9

10 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities 10

11 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities 11

12 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities 12

13 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities { } "name" : "Webinar" "geo" : { "city" : "Amsterdam", "lat" : 4.85, "lon" : } Source: 13

14 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Source: 14

15 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities 15

16 Elasticsearch is... an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities 16

17 Getting up and running... is easy $ wget $ tar -zxf elasticsearch tar.gz $ cd elasticsearch $./bin/elasticsearch 17

18 Elasticsearch & Lua

19 About the Project 3-month project A "complete" Lua client to access the Elasticsearch REST API Influence of official clients Respect the student 19

20 Elasticsearch Clients Design Principles "Nobody should have a reason not to use the client" We have no opinions Respect the language

21 Much more than just a "JSON and HTTP wrapper" Distributed systems - sniffer - selector Add Features - reindex - common syntax 21

22 High Level Architecture Helper Client Parser cjson lua- yajl Transport Sniffer Connection Pool Connection luasocket other http lib Selector RoundRobin Random... 22

23 elasticsearch-lua

24 Lua vs JSON { } title = "A Requirements Elicitation Approach Based in Templates and Patterns", author = "A. Durán Toro", add_authors = { "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" }, key_words = { "requirements engineering", "requirements elicitation" }, language = "en", conference = "WER99", { } "title" : "A Requirements Elicitation Approach Based in Templates and Patterns", "author" : "A. Durán Toro", "add_authors" : [ "B. Bernárdez Jiménez", "A. Ruiz Cortés", "M. Toro Bonilla" ], "key_words" : [ "requirements engineering", "requirements elicitation" ], "language" : "en", "conference" : "WER99", 24

25 CRUD local paper = { title = "A Requirements Elicitation Approach Based in Templates and Patterns",... } local elasticsearch = require"elasticsearch" local client = elasticsearch.client() -- use default configs local r, e = client:index({ index = "papers", type = "wer", id = 1, body = paper }) local r, e = client:get( { index = "papers", type = "wer", id = 1 } ) local r, e = client:update({ index = "papers", type = "wer", id = 1, body = { doc = { title = "Updated Title" } } }) local r, e = client:delete( { index = "papers", type = "wer", id = 1 } ) 25

26 Searching local r, e = client:search( { index = "papers", type = "wer", q = "requirements" } ) { local took r, e = = 1.0, client:search( timed_out = { false, index = "papers", type = "wer", q = "title:requirements"}) _shards = { total = 5.0, successful = 5.0, failed = 0.0 }, local hits r, e = = { client:search{ index total = "papers", = 1.0, max_score = , type = hits "wer", = [ { body = { _index = "papers", "_type": "wer", "_id": "1", query = _score { = , filtered _source = { = { query title = { = "A Requirements Elicitation Approach Based in Templates and Patterns", multi_match key_words = { "requirements engineering", "requirements elicitation" }, query year = "2013" "requirements", fields = { "key_words^10", "title^5", "abstract" } } }, } filter } = { range = { year = { gte = "now-3y" } } } } } }] } } } 26

27 Why Elasticsearch?

28 Workshop on Requirements Engineering Ibero-american community of researchers on Requirements Engineering 28

29 WERpapers Dedicated site to the WER publications Apache + CGILua + MySQL + Google Search 29

30 Not for full text search Slow No text analysis No scoring No auto-completion...

31

32 Elasticsearch Implement your own search result page Relevancy based on downloads, title, abstract, etc. Auto-complete, Highlighting, Geo, etc. Everything in Lua 32

33 Elasticsearch is not Alone

34 Elastic Elasticsearch Store, search, analyze Logstash Collect, parse and enrich data Kibana Visualize and explore data Found Elasticsearch as a Service 34 Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

35

36 Community, community, community Forums for all products Issue tracker User groups & Meetups 36 Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

37 Community, community, community Forums for all products Issue tracker User groups & Meetups 37 Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

38 Community, community, community Forums for all products Issue tracker User groups & Meetups 38 Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

39 Next Steps +Use cases +Real applications +Tests Automate Integration with Elasticsearch Make it an official client 39

40 Thanks for listening! Questions? Pablo Musa 40

41 References