Andrew Moore Amsterdam 2015
Agenda Why log How to log Audit plugins Log analysis Demos
Logs [timestamp]: [some useful data]
Why log? Error Log Binary Log Slow Log General Log
Why log?
Why log?
Why log?
How [not] to audit log General log Slow log Binary log Sniff network In schema init_connect
MySQL Pluggable Audit Interface Available since 5.5.3 Audit interface notifies plugin of General log messages Error log messages Query results sent to client * https://dev.mysql.com/doc/refman/5.6/en/audit-plugins.html
MySQL Pluggable Audit Interface Custom plugin Most popular open source plugins McAfee Percona MariaDB
CLI: Installing Plugins INSTALL PLUGIN plugin_name SONAME='shared_lib_name.so' my.cnf: (RECOMMENDED) plugin-load=plugin_name=shared_lib_name.so Startup: mysqld plugin-load='plugin_name'='shared_lib_name.so'
Force: Installing Plugins plugin_name=force_plus_permanent Configuration Options: (vary from plugin to plugin) Filtering Sync/performance Output
Available for 5.1+ Binary hooking McAfee Audit Plugin Great community support Most filtering options JSON output format Socket and file options
McAfee Audit Plugin Install may require generation of offsets (requires GDB)./offset-extract.sh \ /path/to/mysqld \ /path/to/mysqld.debug
Filtering McAfee Audit Plugin audit_record_cmds audit_record_objects audit_whitelist_users audit_whitelist_cmds
McAfee Audit Plugin {"msg-type":"activity", "date":"1425967153721", "thread-id":"2", "query-id":"17", "user":"root", "priv_user":"root", "host":"localhost", "ip":"", "cmd":"select", "objects":[{"db":"test","name":"people","obj_type":"table"}], "query":"select * from people"}
McAfee Audit Plugin {"msg-type":"activity", {"msg-type":"activity", "date":"1425968812525", "date":"1425967153721", "thread-id":"3", "thread-id":"2", "query-id":"29", "query-id":"17", "user":"root", "user":"root", "priv_user":"root", "priv_user":"root", "host":"localhost", "host":"localhost", "ip":"", "ip":"", "cmd":"select", "cmd":"select", "objects": "objects":[{"db":"test","name":"people","obj_type":"table"}], [{"db":"test","name":"people_vw","obj_type":"view"}, "query":"select {"db":"test","name":"people","obj_type":"table"}], * from people"} "query":"select * from people_vw"}
Percona Audit Plugin Available for 5.5.3+ Ships with Percona Server Drop in replacement for Oracle's plugin Limited filtering JSON, XML, CSV output
Percona Audit Plugin audit_log_strategy ASYNCHRONOUS PERFORMANCE SEMISYNCHRONOUS SYNCHRONOUS
Percona Audit Plugin Support for syslog audit_log_handler = FILE SYSLOG
Percona Audit Plugin <AUDIT_RECORD NAME="Query" RECORD="20_2015-03-10T06:49:49" TIMESTAMP="2015-03-10T06:53:55 UTC" COMMAND_CLASS="select" CONNECTION_ID="3" STATUS="0" SQLTEXT="select * from people" USER="root[root] @ localhost []" HOST="localhost" OS_USER="" IP="" />
Percona Audit Plugin <AUDIT_RECORD NAME="Query" RECORD="32_2015-03-10T06:49:49" TIMESTAMP="2015-03-10T06:55:35 UTC" COMMAND_CLASS="select" CONNECTION_ID="4" STATUS="0" SQLTEXT="select * from people_vw" USER="root[root] @ localhost []" HOST="localhost" OS_USER="" IP="" />
MariaDB Audit Plugin Available for 5.5+ Expanded Audit API Included by default
MariaDB Audit Plugin Table level events CSV output Syslog Plain text passwords Fixed 5.5.42 (1.2.0)
MariaDB Audit Plugin Filtering server_audit_events server_audit_excl_users server_audit_incl_users
MariaDB Audit Plugin 20150310 03:07:25,localhost.localdomain,root,localhost,3,10,QUERY,test,' select * from people',0
MariaDB Audit Plugin 20150310 03:11:18,localhost.localdomain,root,localhost,3,9,READ,test,pe ople, 20150310 03:11:18,localhost.localdomain,root,localhost,3,9,QUERY,test,'s elect * from people_vw',0
Percona and MariaDB Replication McAfee Slaves log replication events by default *Whitelist blank user {} to prevent
Best practices Secure data OS level not logged Utilize log rotation
Best practices Sequential IO keep away from random access IO (data files) Use FS with journal to be crash safe(r), EXT4 Synchronizing writes kills performance
Performance
Audit off Audit on
Performance Audit off Audit on
Audit off Audit on
Log File Storage Secure storage (encryption) Sign logs to ensure not altered Set permissions Store offsite (encrypted of course) Store on read only media
Log Aggregation Commercial Oracle Audit Vault McAfee DAM Splunk Open source ELK Stack??? Oracle Audit grep tool
ELK
Elasticsearch Full text and analytics Apache Lucene RESTful web interface 'Schema-free' JSON documents
Elasticsearch Index = database Type = table Document = row
Logstash Extract [input] Read files, syslog, socket Transform [filter] Filter, mutate, add & drop Load [output] Write to a destination
Logstash Output plugins available Nagios and Nagios_nsca XMPP (hipchat, slack, etc.) Pager duty
logstash.conf input { file { path => "/var/log/mysql/audit.log" type => "mysql-audit" }
logstash.conf filter { if type == audit_log { grok { source => {%GREEDYDATA} target => new_field } } }
logstash.conf output { elasticsearch { cluster => "logstash" host => elasticsearch1 } }
Kibana Javascript & node.js browser based dashboards Real-time search and analytics Seamless integration with Elasticsearch No auth, use something else!
Typical ELK pipeline
Alerts Output workflow for Pager duty nagios_nsca hipchat
Fin! Useful links The definitive guide to elasticsearch http://goo.gl/rc1qjh The logstash book http://goo.gl/h0z72d http://logstash.net/docs/1.4.2/