Data Discovery and Systems Diagnostics with the ELK stack. Rittman Mead - BI Forum 2015, Brighton. Robin Moffatt, Principal Consultant Rittman Mead

Transcription

1 Data Discovery and Systems Diagnostics with the ELK stack Rittman Mead - BI Forum 2015, Brighton Robin Moffatt, Principal Consultant Rittman Mead T : +44 (0) (UK)

2 About Me Principal Consultant with Rittman Mead OBIEE & ODI SysAdmin Performance Previously OBIEE/DW developer at large UK retailer SQL Server DBA, Business Objects, DB2, COBOL. Oracle ACE Frequent blogger for Rittman Mead : IRC: rmoff / #obihackers / freenode

3 ELK

4 ELK Elasticsearch - schema-free, document-orientated, distributed data store Logstash - centralised data processing Kibana - analytics and visualisation

5 Kibana T : +44 (0) (UK)

6 Kibana T : +44 (0) (UK)

7 Kibana T : +44 (0) (UK)

8 Kibana

9 Data Discovery

10 Information Management and Big Data - A Reference Architecture

11 DEMO!

12 Elasticsearch

13 Elasticsearch The core component of the ELK stack Distributed for scalability & resilience Based on Apache Lucene (same as Cloudera s Solr) Near-realtime document indexing

14 Elasticsearch Uses - Search and Analytics Search Soundcloud GitHub Analytics The Guardian s Ophan application

15 Elasticsearch Stores data as JSON documents within an index An index is made up of shards Shards are distributed around a cluster automatically Resilience and scale-out are simple

16 Elasticsearch Administration

17 Elasticsearch REST API $ curl -XPOST ' -d '{"name":"finbarr saunders"}'

18 Elasticsearch REST API $ curl -XPOST ' -d '{"name":"roger mellie, "notes":"the man on the tele"}'

19 Elasticsearch REST API $ curl -XGET ' [ ] "hits" : { "total" : 1, "max_score" : , "hits" : [ { "_index" : "viz", "_type" : "characters", "_id" : "AUyyNUrTI0Rm5Pb-t8_l", "_score" : , "_source":{"name":"roger mellie","notes":"the man on the tele"}

20 Elasticsearch REST API $ curl -XDELETE '

21 Systems Diagnostics with ELK

22 T : +44 (0) (UK)

23 System Diagnostics

24 System Monitoring

25 Logstash

26 Logstash T : +44 (0) (UK)

27 Grok Time to get your RegEx on! Input data Grok pattern Key/Value output

28 DEMO!

29 MOAR DATA?

30 Elasticsearch-Hadoop Two-way connector between Hadoop and Elasticsearch Read/Write with Elasticsearch from Hive, Pig, Spark, etc Tweets Website logs Blog post metadata Datasift Flume CSV MongoDB HDFS mongo-hadoop Hive elasticsearch-hadoop Elasticsearch Kibana T : +44 (0) (UK)

31 Other Elasticsearch Input Methods JDBC River or Feeder method Pull data from any Oracle, mysql, etc with schema intact Native libraries for common languages: Perl / Python / Ruby / PHP / Groovy / Scala /.NET / R / etc etc T : +44 (0) (UK)

32 Summary Summary

33 Summary Data Data Discovery With ELK

34

35 Data Discovery

36 Summary System Diagnostics With ELK

37

38

39

40

41

42 Interested? Data Discovery web twitter System Diagnostics & Monitoring irc #obihackers T : +44 (0) (UK)

43 Data Discovery and Systems Diagnostics with the ELK stack Rittman Mead - BI Forum 2015, Brighton Robin Moffatt, Principal Consultant Rittman Mead T : +44 (0) (UK)