WORKSHOP Log Management with NetEye 3.5 Program 2015 by Thomas Forrer
LogAnalysis with Logstash & Kibana Log Management in NetEye Configuration of Log sources / Agents Log acquisition and configuration of filters Query definition and storage Definition of Visualizations and Graphs Kibana Dashboard definition Integrating a Log Index database 15/04/2015 2
NetEye as a centralized Syslog server Log collection and archiving Internet WAN Signed log files per day per host Centralized Syslog Server Syslog protocol data transmission 15/04/2015
LogAnalysis: The Indexing Architecture The log server extension towards log analysis LogStash Incoming Log Parser Kibana Visualization dashboard of aggregated Log Data Rsyslog The well known RFC 5424 Protocol acceptor daemon Elasticsearch Database of Log Indexes 15/04/2015
LogStash: Setup /Troubleshooting Accept incoming SysLog Stream and integrate LogStash Precondition from previous Lession: Setup the Log Archivation in Neteye LogManager Are log archives created by rsyslog in the /var/log/rsyslog/ folders? Are log archives readable by logstash user? # sudo -u logstash ls -la /var/log/rsyslog/* Is logstash running? # /etc/init.d/logstash status Is logstash reading files? # lsof grep /var/log/rsyslog 15/04/2015
Kibana4: Access & Setup procedures Initial global settings definition Access http://neteyexx.neteye.lab/kibana4/ Define default index structure: Index files are date and time oriented Default Index Archive Structure 15/04/2015
Kibana4: Perform the first search Search on indexes Change time range Search Query Avail. Fields Matches
Kibana4: Perform the first search Perform custom query and save query Visualization Filters Save / Open Query Add include/exclude Filters
Kibana4: Define a Dashboard Graphical elements to hold aggregated data 1. Define a Query to extract logs of interest: In our example we extract all logs, tagged* as FTP Server logs 2. Save the Query for re-use 3. Define a Visualization: Choose among various types of Charts, Tables, Lines or Maps We choose a Pie Chart, to visualize the FTP connection codes 4. Define a Dashboard, where varius Visualizations are joined Choose the previous visualizations and define some useful views * Specific LogStash Patterns, allow to recognize log contents and categorize the content 15/04/2015
Kibana4: Define a Visualization Ver. Bar Chart: Store a query and create a new visualization 4/29/2015 10
Kibana4: Define a Visualization Number of FTP connection codes on the timeline 4/29/2015 11
Kibana4: Define a Visualization Pie Chart: Number of FTP connection codes on the timeline 4/29/2015 12
Kibana4: Define a Kibana Dashboard Bringing together the Visualizations 4/29/2015 13
Identify available search attributes Discover: Click settings next to Fields Settings > Indices: Index attributes list incl. field type 15/04/2015 14
Bind new Index Database Define new Index Pattern Go to: Settings > Indexes 15/04/2015 15
Interacting with new Indexes On Discover > Settings: Choose new index 15/04/2015 16
Visualization of Webserver request origin 4/29/2015 17
THANK YOU for your attention S.r.l. All rights reserved. The text, images and graphics as well as their arrangement on the Log Management with NetEye 3.5 Workshop slides are all subject to Copyright and other intellectual property protection. These objects may not be copied for commercial use or distribution, nor may these objects be modified or reposted on any platform. Some slides also contain images that are subject to the copyright rights of their providers. 4/29/2015 18