WORKSHOP Log Management with NetEye 3.5

Transcription

1 WORKSHOP Log Management with NetEye 3.5 Program 2015 by Thomas Forrer

2 LogAnalysis with Logstash & Kibana Log Management in NetEye Configuration of Log sources / Agents Log acquisition and configuration of filters Query definition and storage Definition of Visualizations and Graphs Kibana Dashboard definition Integrating a Log Index database 15/04/2015 2

3 NetEye as a centralized Syslog server Log collection and archiving Internet WAN Signed log files per day per host Centralized Syslog Server Syslog protocol data transmission 15/04/2015

4 LogAnalysis: The Indexing Architecture The log server extension towards log analysis LogStash Incoming Log Parser Kibana Visualization dashboard of aggregated Log Data Rsyslog The well known RFC 5424 Protocol acceptor daemon Elasticsearch Database of Log Indexes 15/04/2015

5 LogStash: Setup /Troubleshooting Accept incoming SysLog Stream and integrate LogStash Precondition from previous Lession: Setup the Log Archivation in Neteye LogManager Are log archives created by rsyslog in the /var/log/rsyslog/ folders? Are log archives readable by logstash user? # sudo -u logstash ls -la /var/log/rsyslog/* Is logstash running? # /etc/init.d/logstash status Is logstash reading files? # lsof grep /var/log/rsyslog 15/04/2015

6 Kibana4: Access & Setup procedures Initial global settings definition Access Define default index structure: Index files are date and time oriented Default Index Archive Structure 15/04/2015

7 Kibana4: Perform the first search Search on indexes Change time range Search Query Avail. Fields Matches

8 Kibana4: Perform the first search Perform custom query and save query Visualization Filters Save / Open Query Add include/exclude Filters

9 Kibana4: Define a Dashboard Graphical elements to hold aggregated data 1. Define a Query to extract logs of interest: In our example we extract all logs, tagged* as FTP Server logs 2. Save the Query for re-use 3. Define a Visualization: Choose among various types of Charts, Tables, Lines or Maps We choose a Pie Chart, to visualize the FTP connection codes 4. Define a Dashboard, where varius Visualizations are joined Choose the previous visualizations and define some useful views * Specific LogStash Patterns, allow to recognize log contents and categorize the content 15/04/2015

10 Kibana4: Define a Visualization Ver. Bar Chart: Store a query and create a new visualization 4/29/

11 Kibana4: Define a Visualization Number of FTP connection codes on the timeline 4/29/

12 Kibana4: Define a Visualization Pie Chart: Number of FTP connection codes on the timeline 4/29/

13 Kibana4: Define a Kibana Dashboard Bringing together the Visualizations 4/29/

14 Identify available search attributes Discover: Click settings next to Fields Settings > Indices: Index attributes list incl. field type 15/04/

15 Bind new Index Database Define new Index Pattern Go to: Settings > Indexes 15/04/

16 Interacting with new Indexes On Discover > Settings: Choose new index 15/04/

17 Visualization of Webserver request origin 4/29/

18 THANK YOU for your attention S.r.l. All rights reserved. The text, images and graphics as well as their arrangement on the Log Management with NetEye 3.5 Workshop slides are all subject to Copyright and other intellectual property protection. These objects may not be copied for commercial use or distribution, nor may these objects be modified or reposted on any platform. Some slides also contain images that are subject to the copyright rights of their providers. 4/29/