ACHIEVING REGULATORY COMPLIANCE WITH MYSQL AUDIT PLUGINS. Jeremy Glick. Andrew Moore. April 2014

Transcription

1 ACHIEVING REGULATORY COMPLIANCE WITH MYSQL AUDIT PLUGINS Jeremy Glick Andrew Moore April 2014

2 Your Speakers Jeremy Glick Lead icrossing 13 years working with MySQL Chicago MySQL Meetup Jeremy d glick at yahoo.com jd.glick

3 Your Speakers icrossing Digital Marketing Agency unit of Hearst Corporation 40+ Internationally recognized clients 17 Global offices LA, New York, Munich, London, Madrid... Multiple data centers CentOS, MySQL 5.5/5.6, MariaDB PostgreSQL, Oracle, SQL Server, Netezza

4 Your Speakers Andrew Moore UK based rdba for Percona 6+ years Production DBA experience Previous roles with Nokia and Pythian Bristol MySQL Meetup Interests in football, comedy, ethics & philosophy, eroomydna

5 Your Speakers Percona Remote DBA Remote DBAs, 24/7 coverage Unique customer portal providing support metrics Key Contributer to mydumper project Maintainers of Percona Monitoring Plugins Best in business backup service Tiered support models

6 Agenda Audit Logging 101 Regulatory Compliance SOX,HIPAA,PCI DSS Audit Logging with MySQL Non-plugin options McAfee, MariaDB, Oracle Performance considerations Summary

7 Auditing What is Database Auditing? Database auditing involves observing a database so as to be aware of the actions of database users. Database administrators and consultants often set up auditing for security purposes, for example, to ensure that those without the permission to access information do not access it. Source:

8 Audit Logging What Audit Logging does not do Audit logging is post-activity, meaning that you have likely already granted permission Logging will assist detection, it does not prohibit

9 Auditing Why Audit Security Regulatory Compliance y Data Integrity Accountabilit Debugging

10 Security Track data access Alert to suspicious activity Identify weakness Identify security breach REMEBER: Audit logging does NOT prevent access!

11 Regulatory Compliance

12 Regulatory Compliance Common Compliance standards SOX HIPAA PCI DSS

13 Regulatory Compliance SOX The Sarbanes-Oxley Act of 2002 (often shortened to SOX) is legislation enacted in response to the high-profile Enron and WorldCom financial scandals to protect shareholders and the general public from accounting errors and fraudulent practices in the enterprise. l

14 SOX Goals of Sox Strengthen confidence in public accounting. Harden against invalid transactions and data modification that could invalidate data integrity Strict access control on target systems

15 Regulatory Compliance PCI DSS The Payment Card Industry Data Security Standard (PCI DSS) is a set of requirements designed to ensure that ALL companies that process, store or transmit credit card information maintain a secure environment. l

16 PCI DSS Goals of PCI DSS Prevent credit card fraud Harden data integrity by ensuring only privileged users have access to only the resources they need to do their job.

17 Regulatory Compliance HIPAA The HIPAA Privacy Rule addresses the saving, accessing and sharing of medical and personal information of any individual, while the HIPAA Security Rule more specifically outlines national security standards to protect health data created, received, maintained or transmitted electronically, also known as electronic protected health information (ephi). l

18 HIPAA Goals of HIPAA Protect individuals health care information Track all access to (even seen) the data/information

19 Regulatory Compliance Database Auditing Requirements of Regulations Audit Requirement SOX Access to sensitive data (SELECT) PCI DSS HIPAA X X Modification of sensitive data (INSERT,UPDATE,DELETE) X Schema Changes (CREATE, ALTER, DROP) X X X Security Authorisations (GRANT, REVOKE) X X X Security Exceptions (eg. Failed logins) X X X Source: Database Administration 2nd Edition, Craig Mullins

20 Data Integrity Do we trust our data? Has anyone changed data outside of the business rules? Audit logging can assist the analysis of the data's integrity based on any extra-programatic changes (e.g. manual updates). Who/what performed writes on the slave?

21 Benefits of Logging Debugging Targeting database objects to see data transition and access such as stored routines/triggers Logging error codes Forensic investigation of data changes

22 Benefits of Logging Legal Logs generated after an incident risk being rejected by the court as hearsay Logs generated during the regular course of business will likely be accepted by a court.

23 What To Log Compliance Driven: Determine what laws & regulations you must be compliant with Access (Select) Modification (Insert, Update, Delete) Schema Change (DDL) Security Authorizations (Create User, Grant, Revoke) Security Exceptions (Failed logins, attack)

24 Logging Without Plugin init_connect MySQL logs (general/error/slow) Transaction logs (binary logs) Proxy (software/hardware) Connectors (jdbc, PDO_MYSQL...) In-schema ON UPDATE fields Network Sniffing

25 MySQL's Pluggable Audit Interface Available as of MySQL Audit interface notifies plugin of these operations: l Message written to general log l Message written to error log l Query results sent to client l *

26 MariaDB's Pluggable Audit Interface MariaDB has expanded the API Added query_id and database fields (as of MariaDB ) The only way (thru API) to log which tables have been accessed thru views, stored functions, and triggers Change doesn't prevent MariaDB plugin from being used with MySQL or Percona Server.

27 Write Your Own Plugin If you're an experience programmer, using the plugin API isn't incredibly difficult. Should you write your own? Need features? Better performance? Contribute to an existing project? Start a new project?

28 Off-the-shelf Audit Plugins MariaDB McAfee MySQL Enterprise

29 Installing An Audit Plugin mysql> show global variables like 'plugin_dir'; Installing a Plugin Variable_name Value Move the plugin to plugin_dir (if it's not already there) plugin_dir /usr/lib64/mysql/plugin/ l row in set (0.00 sec) Install with one of two methods: INSTALL PLUGIN --plugin-load (preferred method for production systems) mysql> INSTALL PLUGIN plugin_name SONAME 'shared_library_name' shell> mysqld plugin-load="myplug1=myplug1.so

30 Installing An Audit Plugin mysql> show plugins\g *************************** 43. row *************************** Verify the Plugin is Installed Name: audit_log Status: ACTIVE Type: AUDIT Library: audit_log.so License: PROPRIETARY SHOW PLUGINS \G or 43 rows in set (0.01 sec) SELECT * FROM *************************** 43. row *************************** PLUGIN_NAME: audit_log information_schema.plugins PLUGIN_VERSION: 0.1 PLUGIN_STATUS: ACTIVE PLUGIN_TYPE: AUDIT PLUGIN_TYPE_VERSION: 3.1 PLUGIN_LIBRARY: audit_log.so PLUGIN_LIBRARY_VERSION: 1.4 Also ensure to check for messages within the MySQL error log to ensure no issues PLUGIN_AUTHOR: Oracle PLUGIN_DESCRIPTION: Auditing events logger PLUGIN_LICENSE: PROPRIETARY LOAD_OPTION: FORCE_PLUS_PERMANENT experienced.

31 Installing An Audit Plugin Further Considerations Check the mysql error log for evidence of issues starting the plugin. In some cases, you may have to make changes to allow SELinux or AppArmor. chcon Audit2allow, Setsebool etc

32 Off The Shelf solutions

33 MySQL Enterprise Audit Plugin MySQL Enterprise Audit Plugin

34 MySQL Enterprise Audit Plugin MySQL 5.5+ Commercial extension Uses MySQL Audit API XML formatted logs Compatible with Oracle Audit Vault

35 MySQL Enterprise Audit Plugin - Options mysql> show global variables like 'audit%'; Variable_name Value audit_log_buffer_size audit_log_file audit_log_flush audit.log OFF audit_log_format audit_log_policy OLD ALL audit_log_rotate_on_size 0 audit_log_strategy ASYNCHRONOUS rows in set (0.03 sec)

36 MySQL Enterprise Audit Plugin - Options audit_log_buffer_size (static variable) Only used when asynchronous logging is used. Default 1M

37 MySQL Enterprise Audit Plugin - Options audit_log_policy (dynamic variable) ALL = Logs everything NONE = Logs nothing LOGINS = Logs logins QUERIES = Logs queries

38 MySQL Enterprise Audit Plugin - Options audit_log_format (read only variable) NEW format developed for 5.7 and backported to 5.6 NEW format has better compatibility with Oracle Audit Vault NEW formatted differently and contains more values (equals larger log)

39 MySQL Enterprise Audit Plugin - Options audit_log_rotate_on_size (dynamic variable) Controls log rotation When size is exceeded, file is renamed with timestamp and new file is created. Storage/Compression of files to be handled outside of MySQL.

40 MySQL Enterprise Audit Plugin - Options audit_log_strategy (read only variable) ASYNCHRONOUS = async logging, uses buffer PERFORMANCE = ascync, (drops requests if buffer is full) SEMISYNCHRONOUS = sync logging, uses OS caching SYNCHRONOUS = sync() each request

41 MySQL Enterprise Audit Plugin - Replication Other Notes Replicated events ARE NOT logged Stored Procedures not logged at statement level.

42 MySQL Enterprise Audit Plugin XML Log <AUDIT_RECORD TIMESTAMP=" T04:00:30 UTC" RECORD_ID="28479_ T03:56:35" NAME="Connect" CONNECTION_ID="3" STATUS="0"... STATUS_CODE="0" USER="jdoe" OS_LOGIN="" HOST="localhost" IP="" Activity: Connect COMMAND_CLASS="connect" PRIV_USER="jdoe" PROXY_USER="" DB="" />

43 MySQL Enterprise Audit Plugin XML Log <AUDIT_RECORD TIMESTAMP=" T04:01:10 UTC"... RECORD_ID="28489_ T03:56:35" STATUS_CODE="0" localhost []" NAME="Query" OS_LOGIN="" CONNECTION_ID="3" HOST="localhost" STATUS="0" IP=""... Activity: DELETE COMMAND_CLASS="delete" SQLTEXT="delete from t1" />

44 MySQL Enterprise Audit Plugin XML Log <AUDIT_RECORD Statement: CREATE USER TIMESTAMP=" T23:45:28 UTC" RECORD_ID=" _ T23:11:06" NAME="Query" CONNECTION_ID="763" STATUS="0" STATUS_CODE="0" localhost []" OS_LOGIN="" HOST="localhost" IP="" COMMAND_CLASS="create_user" Logs the password hash SQLTEXT="CREATE USER 'moore'@'localhost' IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49 EE7'"/>

45 McAfee Audit Plugin McAfee Audit Plugin

46 McAfee Audit Plugin Available for MySQL 5.1 as Daemon plugin Audit plugin type for newer version json log format Heavy use of binary hooking Supports MySQL 5.1+ Percona Server 5.1+ MariaDB not officially supported

47 McAfee Audit Plugin Plugin Type Installation on MySQL 5.5: mysql> select * from information_schema.plugins where plugin_name = 'audit' \G; *************************** 1. row *************************** PLUGIN_NAME: AUDIT PLUGIN_VERSION: 1.0 PLUGIN_STATUS: ACTIVE PLUGIN_TYPE: DAEMON PLUGIN_TYPE_VERSION: PLUGIN_LIBRARY: libaudit_plugin.so PLUGIN_LIBRARY_VERSION: 1.3 PLUGIN_AUTHOR: McAfee Inc PLUGIN_DESCRIPTION: AUDIT plugin, creates a file mysql-audit.log to log activity PLUGIN_LICENSE: GPL LOAD_OPTION: ON 1 row in set (0.00 sec)

48 McAfee Audit Plugin Plugin Type Installation on MySQL 5.6: mysql> select * from information_schema.plugins where plugin_name = 'audit' \G; *************************** 1. row *************************** PLUGIN_NAME: AUDIT PLUGIN_VERSION: 1.0 PLUGIN_STATUS: ACTIVE PLUGIN_TYPE: AUDIT PLUGIN_TYPE_VERSION: 3.1 PLUGIN_LIBRARY: libaudit_plugin.so PLUGIN_LIBRARY_VERSION: 1.4 PLUGIN_AUTHOR: McAfee Inc PLUGIN_DESCRIPTION: AUDIT plugin, creates a file mysql-audit.log to log activity PLUGIN_LICENSE: GPL LOAD_OPTION: ON 1 row in set (0.02 sec)

49 McAfee Audit Plugin Binary Interception/Hooking Modifying MySQL function code at runtime to get what they need Required prior to Audit API Stayed with hooking after Audit API Some table info was still missing (views based on more than one table) Security exploit to by-pass audit log existed, has since been patched

50 McAfee Audit Plugin Offsets offsets allow access to internal structures Required to have single binary that supports all versions Debug symbols are required to generate offsets Some are not shipped with debug symbols Percona ships without (has separate debug rpm package) Debian ships without (must compile from source to extract offsets)

51 McAfee Audit Plugin Offsets If plugin fails to load, check the error logs for: Audit Plugin: Couldn't find proper THD offsets for: MYSQL_VERSION You can disable checksum verification: [mysqld] audit_validate_checksum=off plugin-load=audit=libaudit_plugin.so

52 McAfee Audit Plugin Offsets If disabling checksum verification doesn't work, you may need to extract checksums. ~]#./offset-extract.sh /usr/sbin/mysqld /usr/lib/debug/usr/sbin/mysqld.debug //offsets for: /usr/sbin/mysqld ( ) {" ","687a95a2cd61388d5ccbae33f2ef7b73", 8408, 8456, 4312, 4848, 88, 2720, 96, 0, 32, 104} *This extract example was for Percona Server and required the installation of the debug package (Percona-Server-56-debuginfo.x86_64)

53 McAfee Audit Plugin Offsets my.cnf [mysqld] audit_offsets=8408, 8456, 4312, 4848, 88, 2720, 96, 0, 32, 104 plugin-load=audit=libaudit_plugin.so

54 McAfee Audit Plugin Options AUDIT_CHECKSUM AUDIT_OFFSETS AUDIT_DELAY_CMDS AUDIT_OFFSETS_BY_VERSION AUDIT_DELAY_MS AUDIT_RECORD_CMDS AUDIT_JSON_FILE AUDIT_RECORD_OBJS AUDIT_JSON_FILE_FLUSH AUDIT_UNINSTALL_PLUGIN AUDIT_JSON_FILE_SYNC AUDIT_VALIDATE_CHECKSUM AUDIT_JSON_LOG_FILE AUDIT_VALIDATE_OFFSETS_EXTENDED AUDIT_JSON_SOCKE AUDIT_WHITELIST_USERS AUDIT_JSON_SOCKET_NAME

55 McAfee Audit Plugin Options audit_json_socket (dynamic variable) syslog-ng McAfee Database Activity Monitor (DAM) Custom Solution

56 McAfee Audit Plugin Options audit_record_cmds (dynamic variable) SELECT INSERT UPDATE etc

57 McAfee Audit Plugin Options audit_record_objs (dynamic variable) List of objects to log (i.e. world.city) Wildcards allowed

58 McAfee Audit Plugin Options audit_whitelist_users (dynamic variable) List of users whose queries will not be logged Could use for non-privileged users Use {} to specify blank user / SQL_THREAD *REPLICATION WARNING: A slave cannot see the user account used on the master. You can specify to ignore blank users which will include the sql_thread.

59 McAfee Audit Plugin Options audit_json_file_sync (dynamic variable) If 0, will use built in buffering provided by libc If > 0, will sync after audit_json_file_sync events

60 McAfee Audit Plugin Options audit_uninstall_plugin (read_only variable) Enable Disable Enable: Users can uninstall plugin using UNINSTALL PLUGIN Disable: Users cannot uninstall plugin

61 McAfee Audit Plugin Replication Replicated events ARE logged Whitelist can be setup to disable logging of replicated events

62 MariaDB Audit Plugin Passwords logged in plain text :( SQL mysql> create user identified by 'secret_password'; Query OK, 0 rows affected (0.01 sec) logfile {"msg-type":"activity","date":" ","thread-id":"2","queryid":"12","user":"root","priv_user":"root","host":"localhost","ip":"","cmd":"create _user","query":"create user 'jdoe'@'%' identified by 'secret_password'"}

63 McAfee Audit Plugin json Log User Connection { "msg-type":"activity", "date":" ", "thread-id":"4", "query-id":"0", "user":"jdoe", "priv_user":"jdoe", "host":"localhost", "ip":"", "cmd":"connect", "query":"connect" }

64 McAfee Audit Plugin json Log Delete { "msg-type":"activity", "date":" ", "thread-id":"4", "query-id":"28", "user":"jdoe", "priv_user":"jdoe", "host":"localhost", "ip":"", "cmd":"delete","objects": [{"db":"test","name":"t1","obj_type":"table"}], "query":"delete from t1"

65 MariaDB Audit Plugin MariaDB Audit Plugin

66 MariaDB Audit Plugin - Options SERVER_AUDIT_EVENTS SERVER_AUDIT_SYSLOG_FACILITY SERVER_AUDIT_EXCL_USERS SERVER_AUDIT_SYSLOG_IDENT SERVER_AUDIT_FILE_PATH SERVER_AUDIT_SYSLOG_INFO SERVER_AUDIT_FILE_ROTATE_NOW SERVER_AUDIT_SYSLOG_PRIORITY SERVER_AUDIT_FILE_ROTATE_SIZE SERVER_AUDIT_FILE_ROTATIONS SERVER_AUDIT_INCL_USERS SERVER_AUDIT_LOGGING SERVER_AUDIT_MODE SERVER_AUDIT_OUTPUT_TYPE

67 MariaDB Audit Plugin - Options server_audit_logging (dynamic variable) Enables/disables logging

68 MariaDB Audit Plugin - Options server_audit_output_type (dynamic variable) FILE SYSLOG

69 MariaDB Audit Plugin - Options server_audit_file_rotate_size (dynamic variable) Once this size is reached, the log file is renamed and a new file is created. *Used for FILE output (not SYSLOG)

70 MariaDB Audit Plugin - Options server_audit_events CONNECT QUERY TABLE (dynamic variable) Can include one or more events type as comma separated list

71 MariaDB Audit Plugin - Options server_audit_excl_users (dynamic variable) server_audit_incl_users (dynamic variable) Comma separated lists of users who will will be included/excluded from logs. server_audit_incl_users has higher priority than server_audit_excl_users. Filters work for query and table level events. Not for connect/disconnect.

72 MariaDB Audit Plugin Error Codes Error codes are available in the log Can use for debugging

73 MariaDB Audit Plugin CSV Log Activity: Connection Db host User,host :29:50,localhost.localdomain,jdoe,localhost,5,0,CONNECT,,,0 timestamps Return code

74 MariaDB Audit Plugin CSV Log Activity: DELETE :30:03,localhost.localdomain,jdoe,localhost,5,24, QUERY,test,'delete from t1',0 SQL statement

75 MariaDB Audit Plugin CSV Log server_audit_events='table' :21:07,localhost.localdomain,John,localhost,3,31,READ,db1,services, :21:07,localhost.localdomain,John,localhost,3,31,READ,db1,services_types, :21:07,localhost.localdomain,John,localhost,3,31,QUERY,db1,'SELECT * from myview',0

76 MariaDB Audit Plugin Passwords logged in plain text :( SQL mysql> create user identified by 'secret_password'; Query OK, 0 rows affected (0.01 sec) logfile :10:10,localhost.localdomain,root,localhost,179, ,QUERY,mysql,'cre ate user \'jdoe\'@\'%\' identified by \'secret_password\'',0

77 Log Files The Audit Log Files

78 Log Files Analyzing Log Rotation Storage Monitoring

79 Log Files Analyzing Access Patterns Honey Tokens Third Party Tools Times Data Aggregation McAfee DAM Oracle Audit Vault logstash

80 Log Files Log Rotation Log Rotation File Syslog Syslog-ng

81 Log Files How To Store Secure storage Do they contain PII? Sign logs to ensure they have not been altered Set permissions correctly Store offsite Store on read only media

82 Log Files Monitoring Custom alerting Ensure the logging is ongoing Identifying particular events based on rules

83 Performance Performance

84 Performance Benchmark using Sysbench (Synthetic) YMMV...YMWV

85 Performance: MySQL 5.6 Enterprise

86 Performance: MySQL 5.6 Enterprise

87 Performance: MySQL 5.6 Enterprise

88 Performance: MySQL 5.6 Enterprise

89 Performance: MySQL 5.6 Enterprise sudo pt-pmp 7 lll_lock_wait(libpthread.so.0),_l_lock_995(libpthread.so.0),pthread_mutex_lock(libpthrea d.so.0),inline_mysql_mutex_lock(mysql_thread.h:688),alog_block_alloc( mysql_thread.h:688),audit_log_notify(audit_log.cc:311),event_class_dispatc h(sql_audit.cc:454),general_class_handler(sql_audit.cc:454),mysql_audit_notify(sql_audit.cc: 215),mysql_audit_general(sql_audit.h:177),dispatch_command(sql_audit.h:177),do_comman d(sql_parse.cc:1036),do_handle_one_connection(sql_connect.cc:982),handle_one_connection (sql_connect.cc:898),pfs_spawn_thread(pfs.cc:1858),start_thread(libpthread.so.0),clone(libc.s o.6)

90 Performance: MariaDB 5.5

91 Performance: MariaDB 5.5

92 Performance: MariaDB 5.5

93 Performance: MariaDB 5.5

94 Do we have time?

95 Performance: Summary Universal Truths The more you want to log the larger the log file and and file admin overhead Synchronizing writes out to the disk hurts a lot. Not suitable for a busy system.

96 Audit Pugins of Tomorrow Feature Wishlist Object level filtering Ability to select log format (json,csv,xml,etc) syslog & syslog-ng support for all Better community effort Rapid development Compliance driven Better consideration for sensitive input i.e. passwords

97 Summary: Feature Comparison FEATURE Oracle McAfee MariaDB Source code Closed Open Open Distributions Enterprise MySQL, PS *MySQL, PS, MariaDB , , 5.5, , 10.0 Event access method API binary Hooking API (expanded) Log file format XML JSON CSV Manual Syslog-ng (Socket) syslog Audit/filter based on objects No Yes No Audit/filter based on command type No Yes No Table level logging (ie. logs tables within views) N/A Yes Yes Versions Log management *When installed in MySQL or Percona Server, MariaDB will not provide table level logging (because of API differences)

98 Summary: Feature Comparison FEATURE Logging strategy Oracle McAfee MariaDB Sync, semi-sync, auit_json_file_sync= N N/A No Yes async, performance Error codes available No

99 Questions Thank you for attending!