OSSEC in the Enterprise

OSSEC in the Enterprise Open Source Log Management, Analysis and Intrusion Detection Rochester Security Summit October 29, 2009 Michael Starks, CISSP, CISA, GSNA

Agenda What is OSSEC? Log Analysis Integrity Monitoring Rootkit Detection Policy Monitoring Alerting Active Response OSSEC WebUI Why OSSEC? Risks & Countermeasures Enterprise Considerations Demo Questions

What is OSSEC? OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response. Source: http://www.ossec.net

What is OSSEC? Put another way... OSSEC is security software that looks for bad stuff on the actual host

Multi-Platform Works on Windows and most Unix-like systems

Centrally Managed Client/server architecture Almost everything can be managed from the OSSEC manager Restart agents Start integrity checks Tune rules Block attacks

Single Installation Manager and agent on one machine

Distributed Centralized manager and distributed agents

Distributed Multiple managers and multiple agents

Redundant Fail over to one or more managers

Flexible and Extensible Easily add support for custom applications Integrate with commercial SIEMs Analyze logs on existing syslog servers

Secure by Default Privilege separated processes Chroot where possible Secure programming practices Encrypted message transport using IP restrictions and replay prevention

Supported Community IRC: #OSSEC on Freenode Mailing lists: ossec-list Commercial Trend Micro OSSEC Host-Based Intrusion Detection Guide ossec-dev www.ossec.net

Fast and Efficient Analyze millions of events per day...in real-time...using commodity hardware

Extensive Application Support Dozens of decoders and hundreds of rules out of the box Unix Pam, sshd (OpenSSH), Solaris telnetd, Samba, Su, Sudo, Proftpd, Pure-ftpd, vsftpd, Microsoft FTP server, Solaris ftpd, Imapd, Postfix, Sendmail, vpopmail, Microsoft Exchange, Apache, IIS5, IIS6, Horde IMP, Iptables, IPF. PF, Netscreen, Cisco PIX/ASA/FWSM, Snort, Cisco IOS, Nmap, Symantec AV, Arpwatch, Named, Squid, Windows event logs, VMWare

Free Open source Budget friendly

Log Analysis The heart of OSSEC

LIDS Log-based Intrusion Detection Not a log management tool Analyzes (but does not store) every log

A Slight Detour What if the attacker deletes the logs? Will you have all the pieces of the puzzle? Robust log management strategies help OSSEC do its job

Log Management Corporate policy should define the need for logging

Log Management Corporate standards should define system audit settings, such as: What to audit Frequency of log rotation Log format Method of communication

Log Management Logs should, wherever possible, be converted from a proprietary format to a standardized and normalized format (e.g. syslog)

Log Management Logs should be centralized and stored on a hardened, purpose-specific server, with no unnecessary or unrelated services running

Log Management Systems should be synchronized with a common, trusted time source

Log Management Logs contain sensitive information and should be encrypted in transit wherever possible

Log Management A copy of each log should be available both locally and centrally In the event of a compromise, the trusted log server can be compared with the local logs

Log Management Logs should be maintained online and archived offline according to regulatory or policy requirements

Log Management Access to logs should be on a need-to-know and least-privileged basis

Log Management Access to logs should always be read-only

Log Flow Through OSSEC Tree-like structure Alert Analysis Decode Pre-decode Log enters system

Log Enters System Secure (encrypted) Insecure (syslog) Localhost

Pre-Decoding and Decoding Extracts individual parts of the log and places them into buckets Useful later on when writing rules b o B user 1 72. 6.1 src_ip 4. 3 5 28 id ns v o g. a url

SSHd Log Pre-Decoded Extracts known fields from logs (e.g. time) Compiled in for efficiency Log comes in as: Apr 14 17:32:06 hostname sshd[1025]: OSSEC pre-decodes it as: time/date -> Apr 14 17:32:06 hostname -> hostname program_name -> sshd Pre-decoded

SSHd Log Fully Decoded Log comes in as: Apr 14 17:32:06 hostname sshd[1025]: Accepted password for root from 192.168.2.190 port 1618 ssh2 OSSEC decodes it as: time/date -> Apr 14 17:32:06 hostname -> hostname program_name -> sshd log -> Accepted password for root from 192.168.2.190 port... srcip -> 192.168.2.190 user -> root Pre-decoded Decoded

SSHd Log Decoder Will there be a test? <decoder name="sshd"> <program_name>^sshd</program_name> </decoder> <decoder name="sshd-success"> <parent>sshd</parent> <prematch>^accepted</prematch> <regex offset="after_prematch">^ \S+ for (\S+) from (\S+) port </regex> <order>user, srcip</order> </decoder>

Analysis (Rules) Rules are also called signatures Simple XML files on the manager Independent of original log format

Two Types of Rules Atomic: single event Bob mistyped his password once Composite: multiple events across logs Bob mistyped his password 3,561 times in 3 minutes on 16 different systems

That Looks Suspicious I know Bob forgets his password, but...

Rules Rules pick up where decoders leave off Instead of writing rules for raw logs, they can be written to normalized data (e.g. Bob is a user ) Data flows through the tree until a rule matches or doesn't match

Rules Severity-based: levels 0 (low) to 15 (high) Nest multiple rules for granular control Rule groups further normalize data web_scan firewall_drop account_changed...

Simplest Rule If the log was decoded as SSHd, generate rule 111 Not very useful yet <rule id = "111" level = "5"> <decoded_as>sshd</decoded_as> <description>logging every decoded sshd message</description> </rule>

Dependent Rule If rule 111 matched and the log contains Failed Password set the severity (level) to 7 and the group to authentication_failed <rule id= 122 level= 7 > <if_sid>111</if_sid> <match>^failed password</match> <description>failed password attempt</description> <group>authentication_failed</group> </rule>

nd 2 Dependent Rule If rule 122 matched and it's that pesky Bob Raise the severity (level) to 12 <rule id= 133 level= 12 > <if_sid>122</if_sid> <user>bob</user> <description>that pesky Bob again</description> </rule>

In Other Words Put another way... Record all events decoded as SSHd Alert at level 7 on every authentication failure If the user is Bob, raise the alert level to 12

Wait a Minute What if Bob has 3,561 login failures again?

Wait a Minute What if his login failures aren't just through SSH?

Revised Rule Thoughts Alert me if Bob has a few authentication failures in a short time, from anywhere, but don't flood me with alerts

Revised Rule for Bob Let's try that last rule again <rule id= 133 level= 12 frequency= 10 timeframe= 300 ignore= 60 > <if_matched_group>authentication_failed</if_matched _group> <user>bob</user> <description>bob is acting up</description> </rule>

Rule Examples Other interesting rules

Attack Followed by Account <group name="syslog,elevation_of_privilege,"> <rule id="40501" level="15" timeframe="300" frequency="2"> <if_group>adduser</if_group> <if_matched_group>attacks</if_matched_group> <description>attacks followed by the addition of an user.</description> </rule> </group>

Really Long URL <rule id="31115" level="13" maxsize="2900"> <if_sid>31100</if_sid> <description>url too long. Higher than allowed on most browsers. Possible attack.</description> <group>invalid_access,</group> </rule>

Multiple Windows Errors <rule id="18154" level="10" frequency="$ms_freq" timeframe="240"> <if_matched_sid>18103</if_matched_sid> <description>multiple Windows error events.</description> </rule>

Windows Application Installed <rule id="18147" level="5"> <if_sid>18101</if_sid> <id>^11707</id> <options>alert_by_email</options> <description>application Installed.</description> </rule>

Windows Audit Policy Changed <rule id="18113" level="8"> <if_sid>18104</if_sid> <id>^612 ^643 ^4719 ^4907 ^4912</id> <description>windows Audit Policy changed.</description> <group>policy_changed,</group> </rule>

Virus Found, Not Removed <rule id="7504" level="12"> <if_sid>7500</if_sid> <regex>$mcafee_virus</regex> <group>virus</group> <description>mcafee Windows AV - Virus detected and not removed.</description> </rule>

Integrity Monitoring Keeping a Known Good State

File Integrity SHA-1 and MD5 of critical system files and registry keys Performed in real-time or on a schedule Auto-ignores files that change too often

File Integrity Also checks owner, group, permissions Hashes forwarded to manager for safe keeping (excellent for forensics) Use the full power of rules to manage alerts (e.g. alert only on changes outside patch window)

World Writable File OSSEC HIDS Notification. 2009 Oct 21 12:02:27 Received From: hostname->syscheck Rule: 100018 fired (level 7) -> "World Writable File" Portion of the log(s): Integrity checksum changed for: '/etc/httpd/conf/httpd.conf' Permissions changed from 'rw-------' to 'rw-r--rw-' --END OF NOTIFICATION

No Longer World Writable OSSEC HIDS Notification. 2009 Oct 21 12:05:11 Received From: hostname->syscheck Rule: 552 fired (level 7) -> "Integrity checksum changed again (3rd time)." Portion of the log(s): Integrity checksum changed for: '/etc/httpd/conf/httpd.conf' Permissions changed from 'rw-r--rw-' to 'rw-------' --END OF NOTIFICATION

Agentless Integrity Periodic diff of firewalls and routers Checksum and diff of remote 'nix systems It's nice to know something changed, but what? Agentless check of /etc/password shows what changed

Agentless Alerts OSSEC HIDS Notification. 2009 May 14 16:32:20 Received From: (ssh_pixconfig_diff) hostname@172.16.0.1->agentless Rule: 555 fired (level 7) -> "Integrity checksum for agentless device changed." Portion of the log(s): ossec: agentless: Change detected: 206a207 > port-object eq 4241 556c557...

Rootkit Detection Exposing the Hidden

Unix Rootkit Detection Signature and anomaly-based Signatures automatically sent to agents Can be run stand-alone

Signature Method Signatures for Adore, Knark, LOC, etc Attempt to stats, fopen and opendir each specified file Some rootkits don't fully hide themselves

Anomaly Method Detects known and unknown rootkits Files in /dev which aren't device files Unusual files (hidden directories, files owned by root which are world-writable)

Anomaly Method Running processes hidden from ps Listening ports hidden from netstat Promiscuous interfaces hidden from ifconfig

Rootcheck Alert OSSEC HIDS Notification. 2009 Oct 06 17:45:17 Received From: XXXX->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/yyyy.com/httpdocs/language/lang_english/ /.log'. /... --END OF NOTIFICATION Source: http://www.void.gr/kargig/blog/2009/10/06/ossec-to-the-rescue/

Windows Rootkit Detection Not as advanced as Unix-based detection Alternate data streams (Files hidden within files)

Policy Monitoring Detect Insecure Conditions

Policy Monitoring Is your system configured securely? Identify situation which can lead to a breach Benchmark system against CIS standard or create your own

Policy Monitoring File, registry setting, or process exists or does not exist Combine values with logical AND/OR Is anti-virus installed but not running?

Policy Monitoring Has the host firewall been disabled? Is LanMan authentication allowed? *Does not alert by default

Alerting Getting Notified

Alerting E-mail, syslog and database output Built-in e-mail flood protection Send alerts to different teams based on granular rules, severity or group

Alerting On second thought, maybe it wasn't Bob who tried to login to his account Someone should get a page if this happens again

Can't Miss the Game What if it's the weekend and I'm watching the game?

Alerting That someone should be Henry, the Jr. Security Analyst What a wonderful opportunity for professional development

Alerting Create another rule without restricting it to Bob, which will only fire on the weekends <rule id= 144 level= 12 frequency= 10 timeframe= 300 ignore= 60 > <if_matched_group>authentication_failed</if_matched _group> <weekday>saturday,sunday</weekday> <description>multiple Weekend Authentication Failures</description> </rule>

Alerting Followed by an alert configuration is ossec.conf <email_alerts> <email_to>sec-workling@example.com</email_to> <rule_id>144</rule_id> <format>sms</format> </email_alerts>

Alerting Syslog or database output easily integrated with commercial SIEMs Use OSSEC for the analysis Use the SIEM GUI for advanced correlation

Rule Examples Other interesting alerts

Excessive Events OSSEC HIDS Notification. 2009 Oct 21 04:31:50 Received From: hostname->/var/log/httpd/error_log Rule: 11 fired (level 8) -> "Excessive number of events (above normal)." Portion of the log(s): The average number of logs between 4:00 and 5:00 is 936. We reached 1218.

First-Time Login OSSEC HIDS Notification. 2009 Oct 22 11:24:34 Received From: hostname->/var/log/secure Rule: 10100 fired (level 4) -> "First time user logged in." Portion of the log(s): Oct 22 11:24:33 hostname sshd[2998]: Accepted password for kevin_mitnick from 12.174.169.111 port 52387 ssh2

First Sudo Attempt OSSEC HIDS Notification. 2009 Oct 22 11:27:49 Received From: hostname->/var/log/secure Rule: 5403 fired (level 4) -> "First time user executed sudo." Portion of the log(s): Oct 22 11:27:49 hostname sudo: kevin_mitnick : user NOT in sudoers ; TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=/bin/su -

Active Response Preventing Breaches

Active Response Attackers follow common patterns 1. Reconnaissance 2. Scan 3. Exploit OSSEC can often prevent breaches by detecting attacks in the early stages

Active Response Not an IPS, but effective

Active Response Time-based security implementation Protection time should be greater than the sum of detection time, plus reaction time (D+R)>P This is good!

Active Response If severity > 6, add the attacker's IP to the host firewall for 10 minutes Or the perimeter firewall... Or disable an account... Or shut down the system...

Active Response Execute responses on the manager, one particular agent, a firewall or everywhere Worldwide?

OSSEC WebUI A Face to OSSEC

Benefits of GUIs GUI interfaces allow you to see trends and patterns over time FTP account gets locked out every day at 4:15 AM What alerts does OSSEC think aren't worthy of an e-mail?

OSSEC WebUI

OSSEC WebUI

OSSEC WebUI

Other GUI Options Other options include: Splunk OSSIM Picviz

Why OSSEC?

PCI DSS 1.2 10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

PCI DSS 1.2 10.6 Review logs for all system components at least daily......note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6

Closing the NIDs Circle Network-based IDS Only half the picture

Closing the NIDs Circle Host-based IDS The other half

Closing the NIDs Circle Network and Host-based IDS A new level of insight into your environment

Closing the NIDs Circle Of course, OSSEC reads NIDs logs

Forensics Everything is forwarded to the manager for analysis and possible storage Attackers like to delete logs

Policy Compliance How do you know your systems are still hardened? Are admins logging in with unique accounts? Is anti-virus running?

Keep Employees Honest Insider threats cost companies millions per year Employees who know their activities are monitored tend to be more honest

Budget OSSEC can be used for free

Risks & Countermeasures

Mass Deployment Deploying large amounts of agents is challenging Each agent uses a unique key How can a single package be created?

Active Response Attackers who know Active Response is in use may try to use that to their advantage IPs can be spoofed, thereby triggering an incorrect response

Alert Flooding You have 6,972 new messages! Will you read them all?

Log Injection Attacker uses poorly written regular expressions to bypass rules root@slacker:~# ftp 192.168.3.4 220 Welcome to labs ossec candy FTP service. Name (192.168.2.3:root): lala] FAIL LOGIN: Client 2.3.4.54 Normal Log Mon Jun 2 21:05:30 2007 [pid 1448] [myuser] FAIL LOGIN: Client 192.168.3.1 Log Injection Mon Jun 2 21:06:02 2007 [pid 1452] [lala] FAIL LOGIN: Client 2.3.4.54 ] FAIL LOGIN: Client 192.168.3.1

Risk Countermeasures E-mail flooding By default, OSSEC will only send 12 alerts per hour, queuing the rest until the next hour Active Response Response timeout IP whitelists Log Injection Tight regular expressions

Enterprise Considerations

Define the Problem What problem are you trying to solve? What are your primary drivers? What are the obstacles?

Codify in Policy Explicitly state the need in policy

Set Requirements Requirements are a measure of success

Define the Scope Will you monitor all systems? What is the budget? What is the time-frame?

Make a Desicion Is OSSEC a good fit? Don't design a solution looking for a problem!

Plan, Do, Check, Act Plan your OSSEC rollout Do the actual rollout Check the requirements against the rollout Act on the lessons learned

Demo

Summary OSSEC can add a new level of insight into your environment Only use OSSEC if it fits a need If you do use OSSEC, contribute your decoders, rules and lessons learned back to the community!

Questions?

Acknowledgements Daniel B. Cid, OSSEC creator Trend Micro Rochester Security Summit OSSEC Aucert presentation

Image Credits Agenda: http://www.sxc.hu/photo/807162 Question mark: http://www.sxc.hu/photo/1147438 Tree: http://www.sxc.hu/photo/1195970 Vintage Mac: http://www.sxc.hu/photo/1028528 Rubber band ball: http://www.sxc.hu/photo/168735 Padlock: http://www.sxc.hu/photo/865986 Fast car: http://www.sxc.hu/photo/1081680 Cardboard box: http://www.sxc.hu/photo/1036068 Jumping man: http://www.sxc.hu/photo/1212299 Camera lid: http://www.sxc.hu/photo/450946 Buckets: http://www.sxc.hu/photo/807354 Ruler: http://www.sxc.hu/photo/1010158 Bob: http://www.sxc.hu/photo/912662 OSSEC WUI: http://www.ossec.net/dcid/?p=29 Road sign: http://www.sxc.hu/photo/1157986 The following images were used under fair use provisions of US copyright and trademark law: Logos: Windows, Tux, FreeBSD, PCI and AIX OSSEC WebUI screenshots

Image Credits Files in basket: http://www.sxc.hu/photo/456727 Potato: http://www.sxc.hu/photo/1132394 Paper stack: http://www.sxc.hu/photo/251979 Old phone: http://www.sxc.hu/photo/1146563 Little guy and stop sign: http://www.sxc.hu/photo/1197499 Fence: http://www.sxc.hu/photo/1044635 Clock: http://www.sxc.hu/photo/1026820 Retro TV: http://www.sxc.hu/photo/981522 Sunglasses: http://www.sxc.hu/photo/621374 Happy face: http://www.sxc.hu/photo/1147441 Thumb print: http://www.sxc.hu/photo/1231735 Fist: http://www.sxc.hu/photo/621374 Money symbol: http://www.sxc.hu/photo/983478 Crowd: http://www.sxc.hu/photo/893433 E-mail: http://www.sxc.hu/photo/1102040 Red cross: http://www.sxc.hu/photo/971655

Text Credits Attacking Log Analysis Tools, Daniel B. Cid: http://www.ossec.net/main/attacking-log-analysis-tools OSSEC at AusCERT, Daniel B Cid: http://www.ossec.net/ossec-docs/auscert-2007-dcid.pdf

Presentation License This presentation is licensed under the Creative Commons AttributionNoncommercial-Share Alike 3.0 license. The license does not extend to images, which hold their own copyrights attributed to various authors. You are free: to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions: Attribution You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). Noncommercial You may not use this work for commercial purposes. Share Alike If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one. With the understanding that: Waiver Any of the above conditions can be waived if you get permission from the copyright holder. Other Rights In no way are any of the following rights affected by the license: Your fair dealing or fair use rights; Apart from the remix rights granted under this license, the author's moral rights; Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Notice For any reuse or distribution, you must make clear to others the license terms of this work.