Using an Open Source Framework to Catch the Bad Guy. Norman Mark St. Laurent Senior Solutions Architect, Red Hat 06.28.12

Using an Open Source Framework to Catch the Bad Guy Norman Mark St. Laurent Senior Solutions Architect, Red Hat 06.28.12

Agenda Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management RHEL 6 Log Storage and Rotation RHEL 6 Remote Host Storage Specific RHEL 6 Log Generation Settings Red Hat Enterprise Linux 6 Log Management Operational Process RHEL 6 Forensics and Incident Response Log Analysis

White Paper Available: http://www.redhat.com/resourcelibrary/whitepapers/osframework-to-catch-the-bad-guy

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management In order to run on US Government systems, RHEL has met a stringent set of technical security requirements for: Confidentiality Integrity Availability

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management Confidentiality: defined as ensuring that information is accessible only to those authorized to have access. Integrity: defined as quality of an IT system reflecting the logical correctness and reliability of the operating system. Completeness of the hardware and software implementing the protection mechanisms. Availability: defined as timely, reliable access to data and information services for authorized users.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management RHEL has received Common Criteria certification at: Enterprise Assurance Level 4 (EAL4+) Controlled Access Protection Profile (CAPP) Label Security Protection Profile (LSPP) Role-Based Access Protection Profile (RBACPP) Note: RHEL has passed the Common Criteria process 15 times.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management Common Criteria security event auditing requirements are covered in both CAPP and LSPP protection profiles. CAPP (Orange Book C2 criteria) defines audit to provide comprehensive logging of security events that are reliable and robust. LSPP (Orange Book B1 criteria) extends audit requiring enhanced security event auditing to include MAC labeling and decision information.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management Audit must be: Non-bypassable. The right to add records to the audit trail must be controlled. Setting/viewing the audit configuration must be controlled. Audit review must be controlled and assignable Must have the ability to fail-stop the system.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management Security Policies State: Information Systems should be monitored in order to detect, isolate, and react to intrusions, disruption of services, or other incidents that threaten the security. There should be a way to collect and retain audit data to support forensics and incident response relating to misuse, penetration, reconstruction, or other investigations.

White Paper Available: http://www.redhat.com/resourcelibrary/whitepapers/gov-wpcommoncriteria-2012

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management Audit logs are a source of evidence. Proof that a malicious person has not altered logs. Logs must be creditable. Logs produces by a computer are not admissible as evidence unless it can be shown that there is no reasonable ground for believing them to be inaccurate.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management RHEL 6 Auditing Sub System provides: Kernel-resident logging of system calls User space tools collect and view the logs The capability to monitor real-time auditable events that may indicate an imminent violation of the security policy. Provides a means for both a detailed and granular forensics investigation and incident response.

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management RHEL 6 Auditing Sub System has the ability to monitor tty logging (keystrokes) via the pam_tty_audit PAM Module: File /etc/pam.d/system-auth pam_tty_audit addition Session required pam_tty_audit.so disable=* enable=root Reviewing keystroke captures with aureport #aureport tty ts today

Audit Log Management Infrastructure Establishing Policies and Procedures for Log Management When tty auditing is enabled via PAM, it is inherited by all processes started by that user. It is recommended to use the disable=* as the first option so that it turns off tty audit by default, then you add which user(s) after.

Audit Tool Definition auditd The daemon auditd is the user space component. It is responsible for writing audit records to the disk. /etc/audit/audit.rules The audit.rules file contains audit rules that will be loaded by the audit daemon s init script when the daemon is started. /etc/audit/auditd.conf The auditd.conf file is the configuration file for the audit daemon. auditctl The auditctl command is used to assist controlling the kernel s audit system. ausearch The ausearch command is used to query the audit daemon logs events based on search criteria. aureport The aureport command will produce a summary report of the audit system logs. aulast The aulast command will print out a listing of the last logged in and logged out users. autrace The autrace audit tool is a program that will add th audit rules to trace a process similar to strace. Very useful to see what a program maybe doing. audispd The audispd daemon is an audit event multiplexer It takes audit events and distributes them to child programs that want to analyze events in real time. /etc/audisp/audispd.conf The audispd.conf file control the configuration of the audit event dispatcher.

Audit Log Management Infrastructure Log Storage and Rotation The Audit Sub System allows for log storage from the system level and infrastructure level. Retain logs on the system. Transmit logs to a the log management infrastructure host.

Audit Log Management Infrastructure Log Storage and Rotation If either the system or infrastructure logging host fails to log, the other retains the log data. During an incident on a system, logs might be altered or destroyed by attackers. Incident response can use the data from the infrastructure logs to help with the forensics. Comparing infrastructure logs to system logs also helps what data was changed or removed indicating what the attacker wanted to conceal.

Audit Log Management Infrastructure Log Storage and Rotation System log files are stored in the /var/log directory. The Audit Sub System logs are stored in /var/log/audit. It is recommended that partitioning requirements should match the United States Governments Configuration Baseline (USGCB). http://usgcb.nist.gov

Audit Log Management Infrastructure Log Storage and Rotation The audit trail is so important in a CAPP environment that access to the system resources must be denied if an audit trail cannot be created. Log File Directory partition size Formula: MAX_SIZE_OF_LOG_FILE X NUMBER_OF_LOG_FILES

Audit Log Management Infrastructure Log Storage and Rotation The default setting with auditd rotates 4 logs by size (5MB), retaining a maximum of 20MB of data. The below script: Sets compression (compress). Rotates the log files for 90 days (rotate 90). Keeps logs on a daily basis (daily). Does not rotate if log file is empty (notifempty). If log is mission do not issue an error message (missingok).

Audit Log Management Infrastructure Log Storage and Rotation cat <<LOGROT1 > /etc/logrotate.d/audit compress /var/log/audit/audit.log { rotate 90 daily notifempty missingok postrotate /sbin/service auditd restart 2> /dev/null > /dev/null true endscript } LOGROT1

Audit Log Management Infrastructure Log Storage and Rotation Compressing the log files will make the audit tools aureport and ausearch unable to read the log files. Use the zcat or bzcat commands to decompress the files to stdout for the audit tools to then read stdin. #bzcat /var/log/audit/audit.log.1.bz aureport

Audit Log Management Infrastructure Log Storage and Rotation It is important to set the rotation time as close to midnight as possible (Daily rotations / 24 hour clock). Edit the file: /etc/audit/auditd.conf Set: max_log_file_action=ignore Next copy the script auditd.cron that was shipped with the audit RPM (located in /usr/share/doc/audit version directory) to the /etc/cron.daily directory. Change the permissions to 0770. Make the ownership root.root.

Audit Log Management Infrastructure Log Storage and Rotation RHEL 6 includes the cronie package as a replacement for vixie cron. The main difference between these packages is how the regular jobs (daily, weekly, and monthly) are done. Cronie uses the /etc/anacrontab file to start its daily cron jobs. To ensure that the daily rotation is close to the 24 hour clock, edit the /etc/anacrontab file with the following changes:

Audit Log Management Infrastructure Log Storage and Rotation # /etc/anacrontab: configuration file for anacron # See anacron(8) and anacrontab(5) for details. SHELL=/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # the maximal random delay added to the base delay of the jobs RANDOM_DELAY=0 #No Random Delay # the jobs will be started during the following hours only START_HOURS_RANGE=0 #Defines the Midnight Interval when scheduled jobs can run. #period in days delay in minutes job identifier command 1 0 cron.daily nice run parts /etc/cron.daily 7 25 cron.weekly nice run parts /etc/cron.weekly @monthly 45 cron.monthly nice run parts /etc/cron.monthly #Set the delay in minutes for cron.daily to 0, so that it anacron will not delay #and do cron.daily as close to midnight as possible.

Audit Log Management Infrastructure Remote Host Storage RHEL 6 should be configured to send its log files to a remote host log server. Virtual Lans (VLANs) should be considered: having the log files be distributed on a different network. Increases network performance. Segments the data. In addition to confidentiality and integrity of archived log files, organizations need to protect the availability of log files.

Audit Log Management Infrastructure Remote Host Storage Table 12: The /etc/audit/auditd.conf log server setup configuration describes the configuration settings for the audit daemon for a server that is being configured to aggregate and collect log files from numerous hosts. log_file = /var/log/audit/audit.log log_group = isso num_logs = 90 dispatcher = /sbin/audispd #audispd servers and clients to receive and send log files. space_left = 500 space_left_action = email disk_full_action = {ignore, syslog, exec:/path/to/a/script, suspend, single, halt} tcp_listen_port = 60 tcp_max_per_addr use_libwrap = yes

Audit Log Management Infrastructure Remote Host Storage RHEL 6 currently does not support Kerberos encryption with auditd. SSH Port Forwarding Allows a port from one host to appear on another, using a connection through ssh. Allows traffic from that is being forwarded to be encrypted. Uses TCP Transport Method. Meets NSA Suite B encryption Standards for encryption with AES. Has a realm of supported ciphers.

Audit Log Management Infrastructure Remote Host Storage The client host must specify a non-standard port to connect. Specifying ports 1 1023 makes sure that clients send from a privileged port to help prevent log injection attacks by untrustworthy users. Table 13 provides a script for setting up a SSH tunnel between an aggregate log host and its clients that will be sending their log files. Red Hat Network Satellite (Configuration Management of scripts and system settings).

Audit Log Management Infrastructure Remote Host Storage The /etc/audisp/audispd.conf file controls the configuration of the audit event dispatcher. This file should be setup on the clients as well as the log server. Controls the remote server setup and actions needed by audispd. SEE TABLE 14

Audit Log Management Infrastructure Remote Host Storage The /etc/audisp/audisp remote file describes the needed settings to allow a RHEL 6 host to become a log client. remote_server = 192.168.1.22 port = 60 local_port = 61 mode = forward network_failure_action = suspend disk_low_action = suspend disk_full_action = suspend SEE TABLE 16

Audit Log Management Infrastructure Log Generation Settings By default the audit daemon auditd only logs SELinux denials. Helpful in discovering intrusion attempts, modifications to user accounts, and calls to sudo. #chkconfig auditd on ; service auditd start Add audit=1 to the /etc/grub.conf kernel line.

Audit Log Management Infrastructure Log Generation Settings The /etc/audit/audit.rules file allows for the configuration of specific policy. The /etc/audit/audit.rules file goes hand in hand with the auditctl command. The auditctl command is a utility that controls the kernel s audit subsystem. Controls behavior Get status Add or delete rules

Audit Log Management Infrastructure Log Generation Settings Rules for maintainability should be kept in the /etc/audit/audit.rules file. The auditctl command is not needed it is implied. The audit subsystem supports an ample collection of events, to include: Tracing of system calls (system call name or system call number) Filter by PID, UID, system call success, and system call argument. Ability to monitor specific files for modifications (content and metadata).

Audit Log Management Infrastructure Log Generation Settings The audit rules and settings in the white paper follow the recommended audit rules setting in /usr/share/doc/audit version/stig.rules. Red Hat provides a number of audit.rules templates that meet a number of standards and regulations. Successful and unsuccessful attempts to read audit records Modifications to audit configuration Things that could affect time Things that could affect system locale Things that could affect MAC and DAC policy Successful and unsuccessful logins and logoffs Successful and unsuccessful accesses to security relevant objects Use of privileged commands Export to media Specific watches with keys

Audit Log Management Infrastructure Log Management Operational Process There must be an operational process in place to review the log files. Must be routine Flexible Made mandatory within the security policy Organized *If not, the configuration and guidance we applied in the previous slides are irrelevant and there is no frame work to catch the Bad Guy.

Audit Log Management Infrastructure Log Management Operational Process Centralizing log files helps the ISSO get the big picture. Red Hat audit tools can be used to tunnel down and find specific notes of interest. Use the input file flag with the tools to view data in a rotated log file. Compression and zcat.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis When performing log analysis or even working a postmortem investigation, start up front with the main aureport output. Gives you an idea what is happening on the system. Reports events that are hard coded by the audit sub system. Login and logout Authentication uses System anomalies

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis [root@mstlaure audit-2.1.3]# aureport Summary Report ====================== Range of time in logs: 05/29/2012 16:20:01.046-06/08/2012 08:50:03.027 Selected time for report: 05/29/2012 16:20:01-06/08/2012 08:50:03.027 Number of changes in configuration: 10 Number of changes to accounts, groups, or roles: 0 Number of logins: 10 Number of failed logins: 0 Number of authentications: 22 Number of failed authentications: 1 Number of users: 3 Number of terminals: 10 Number of host names: 2 Number of executables: 16 Number of files: 2 Number of AVC's: 36 Number of MAC events: 28 Number of failed syscalls: 0 Number of anomaly events: 17 Number of responses to anomaly events: 0 Number of crypto events: 20 Number of keys: 0 Number of process IDs: 1551 Number of events: 7881

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis Once a point of interest has been found, event details can be found at a more granular level with the ausearch a event number output. All reports have an event number. Specify start and stop times to help narrow specifics. Messages always start like: msg=audit(1116360555.329.2401771). The event ID is the number after the :.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis The white paper details review and an example of forensics and incident response log analysis by shift. A starting point and a routine. Find anomalies that do not comply with security policy.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis aureport summary start yesterday Running this report will allow for an overview of the current audit statistics for the previous day. aureport failed start yesterday Running this report will get statistics of failed events.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis aureport k KeyName start yesterday This command provides a high level report on all the keys set in the audit.rules file. Specific keys we set are: ISSO-audit, ISSO-cfg-audit, ISSOtime-change, ISSO-system-local, ISSO-MAC-policy, and ISSOmedia-export ausearch event audit event id

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis auditctl a exit,always S all F PID=1005 To see all syscalls made by a specific program. auditctl a exit,always S open F auid=510 To see all files opened by a specific user.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis auditctl a exit,always S open F success=0 To see all unsuccessful open calls. ausearch word IPADDRESS String based matches {Hostname, IPADDR, Filename, SELinux Context}.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis ausearch loginuid 11549 Search for an event with the given login user ID.

Audit Log Management Infrastructure RHEL 6 Forensics and Incident Response Log Analysis Thank You!