OSSEC Intrusion detection and response System and log analysis of Drupal sites and servers
Looked at your logs recently? Client site, November 2012 33.44.55.66 - - [04/Nov/2012:05:48:59 +1100] "POST http://www.example.com/?q=fckeditor%2fxss HTTP/1.1" 404 32956 "-" "-" 33.44.55.66 - - [04/Nov/2012:05:49:01 +1100] "POST http://www.example.com/?q=ckeditor%2fxss HTTP/1.1" 200 0 "-" "-" 33.44.55.66 - - [04/Nov/2012:05:49:04 +1100] "GET http://www.example.com/sites/default/files/wtm5439n.php HTTP/1.1" 200 109 "-" "-" 33.44.55.66 - - [04/Nov/2012:06:27:25 +1100] "POST http://www.example.com/sites/default/files/wtm5439n.php? cookies=1&showimg=1&truecss=1&t2122n=1 HTTP/1.1" C99 (R57) shell (PHP-based Backdoor) https://www.drupal.org/node/1482528 https://www.drupal.org/sa-core-2013-003
Looked at your logs recently? /var/log/apache2 crawlers hunting for holes brute-forcing /user/password, /user/register error 500, 504 (gateway timeouts, slow PHP?) worse (see last slide)
Looked at your logs recently? /var/log/syslog (Drupal) brute forcing (in more detail) exceptions, permissions problems crashes, panics, timeouts external service drama: Mollom, Payment GW
Looked at your logs recently? /var/log/auth.log SSH, user/group modifications Use of sudo Whoa, someone ran sudo nano /var/www/ drupal/includes/bootstrap.inc on prod???
Attacks: just one risk Bad practice Human error Dependant services (third parties) Packages installed or removed (/var/log/apt/history.log) all has impact, all in the logs
What s not in logs? Bad practice and human error :) insecure permissions on files Or bigger problems: unexpected modifications (Apache modules?) http://blog.unmaskparasites.com/2012/09/10/malicious-apache-module-injects-iframes/
What s security? Security is not just about intrusions Security is anything that could compromise availability, integrity, reliability, trust, money
What to do about it? Enter
What is OSSEC? http://www.ossec.net Free and open source Host-based intrusion detection system (HIDS) Acquired by Trend Micro in 2009 2.8 just released Active development / community
Installation Download and run install.sh Try my wrapper script (does checksums etc) http://is.gd/ossec_install Not in Debian yet - PPA exists
OSSEC model Server->agent mode (central config, active response propagates) Local mode (standalone) Hybrid mode (multi-tier, complex topography)
The 4 main features Log analysis (What s happening now that s being logged?) Syscheck (integrity checking - what happened that left traces?) Rootcheck (rootkit detection) Active Response (what to do about it?)
Log Analysis what s happening?
Log Analysis How does it work? Decoders How to interpret logs (regexes to split up timestamps, IPs, messages) Rules Analysis of the decoded parts Grading them by level/threshold
Log Analysis Out of the box: SSH (bruteforcing, first time user logged in ) First time user executed sudo SMTP (spam relay attempts, SASL bruteforcing) Apache/Nginx issues (40Xs, 50Xs)
Log Analysis Write your own Decoders (custom logs) such as Drupal watchdog (Syslog module) Thank you madirish : http://www.madirish.net/428
Log Analysis OSSEC HIDS Notification. 2014 Jun 23 18:11:38 Received From: (example) 11.22.33.44->/var/log/messages Rule: 104130 fired (level 10) -> "Possible Drupal brute force attack (high number of logins)." Portion of the log(s): Jun 23 18:11:38 example drupal: http://www.example.com 1403511098 user 185.17.27.182 http://www.example.com/index.php?q=user/login http://www.example.com/ index.php?q=user/login 0 Login attempt failed for wembleylman10. Jun 23 18:11:36 example drupal: http://www.example.com 1403511096 user 185.17.27.182 http://www.example.com/index.php?q=user/login http://www.example.com/ index.php?q=user/login 0 Login attempt failed for wembleylman10. Jun 23 18:09:12 example drupal: http://www.example.com 1403510952 user 68.64.172.234 http://www.example.com/content/welcome?destination=node/4 http:// www.example.com/node/add/submission 0 Login attempt failed for arrevemof. Jun 23 18:09:12 example drupal: http://www.example.com 1403510952 user 68.64.172.234 http://www.example.com/content/welcome?destination=node/4 http:// www.example.com/node/add/submission 0 Login attempt failed for arrevemof. Jun 23 18:09:09 example drupal: http://www.example.com 1403510949 user 68.64.172.234 http://www.example.com/content/welcome?destination=node/4 http:// www.example.com/node/add/submission 0 Login attempt failed for abralfultifug. Jun 23 18:09:09 example drupal: http://www.example.com 1403510949 user 68.64.172.234 http://www.example.com/content/welcome?destination=node/4 http:// www.example.com/node/add/submission 0 Login attempt failed for abralfultifug. --END OF NOTIFICATION
Log Analysis Resource problems? (bottleneck/memory leak?) OSSEC HIDS Notification. 2014 May 07 14:49:44 Received From: (example) 11.22.33.44->/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): May 7 14:49:43 example drupal: http://www.example.com 1399470583 php 55.66.77.88 http://www.example.com/user/68/edit http://www.example.com/user/68/edit 25 PDOException: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: DELETE FROM {XXXXXXXXX} #012WHERE (uid = :db_condition_placeholder_0) AND (subid = :db_condition_placeholder_1) ; Array#012(#012 [:db_condition_placeholder_0] => 68148#012 [:db_condition_placeholder_1] => 77217#012)#012 in XXXXXXX_update::delete() (line 652 of /var/www/drupal/www/sites/all/modules/custom/xxxxxx/xxxxx.inc). --END OF NOTIFICATION OSSEC HIDS Notification. 2014 Jun 14 15:17:02 Received From: (example) 11.22.33.44->/var/log/messages Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): - Jun 14 15:17:02 example ool www: PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/www/sites/ all/modules/contrib/views/modules/field/views_handler_field_field.inc on line 674 --END OF NOTIFICATION
Log Analysis Everything is a DNS problem OSSEC HIDS Notification. 2014 Jul 03 17:35:27 Received From: (example) 11.22.33.44->/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): Jul 1 17:35:26 example drupal: http://www.example.com1404405326 php 11.22.33.44 http://www.example.com/node/44983 http://www.example.com.com/44983 0 User error: Failed to connect to memcache server: mem01.example.com:11211 in dmemcache_object() (line 415 of /var/www/drupal/www/sites/all/modules/ memcache/dmemcache.inc). --END OF NOTIFICATION OSSEC HIDS Notification. 2014 Jun 29 10:01:53 Received From: (example) 11.22.33.44->/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): Jun 29 10:01:51 example drupal: http://www.example.com 1404032511 brightcove 55.66.77.88 http://www.example.com/content/summit-2014-presentation 0 Loading Brightcove video failed. --END OF NOTIFICATION
Syscheck what s changed?
Syscheck Scheduled scans Detects when files have changed (checksums) lots of false positives due to software patching, find the balance between useful/fatiguing New services start or stop (netstat: port watching)
Syscheck OSSEC HIDS Notification. 2014 Jul 01 04:01:03 Received From: (example) 11.22.33.44->syscheck Rule: 550 fired (level 7) -> "Integrity checksum changed." Portion of the log(s): Integrity checksum changed for: '/usr/bin/ssh'" " " " " " << hopefully that s legit because you recently patched OpenSSH.. Size changed from '434024' to '641640' Old md5sum was: '50226273f654d7a2d7b38a0b0c09def4' New md5sum is : 'a8bf35316eb4f46e377a957ecb6cfdca' Old sha1sum was: '976af6f53338a7e9d4eb71617a2a8471aeb6937b' New sha1sum is : 'e871e0a907cdfb76c6e0722a6196b0c9f8edb1fd' --END OF NOTIFICATION
Syscheck OSSEC HIDS Notification. 2014 Jul 01 17:43:20 Received From: (example) 11.22.33.44->netstat -tan grep LISTEN grep -v 127.0.0.1 sort Rule: 533 fired (level 7) -> "Listened ports status (netstat) changed (new port opened or closed)." Portion of the log(s): ossec: output: 'netstat -tan grep LISTEN grep -v 127.0.0.1 sort': tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:49279 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN " " " New process listening on TCP port 5666?" tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp6 0 0 :::111 :::* LISTEN tcp6 0 0 :::25 :::* LISTEN tcp6 0 0 :::4949 :::* LISTEN tcp6 0 0 :::58285 :::* LISTEN Previous output:" ossec: output: 'netstat -tan grep LISTEN grep -v 127.0.0.1 sort': tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN " " " ^^^ SSH no longer running above? Did you push a bad config via Puppet?" tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:49279 0.0.0.0:* LISTEN
Rootcheck who left the door open?
Rootcheck I like rkhunter too, but get a 2nd opinion Hopefully more false positives than not OSSEC HIDS Notification. 2012 Nov 20 23:37:22 Received From: (example) 11.22.33.44->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Anomaly detected in file '/tmp/#sql_1020_0.myi'. Hidden from stats, but showing up on readdir. Possible kernel level rootkit." --END OF NOTIFICATION - OSSEC HIDS Notification. 2014 Jul 07 10:24:18 Received From: Miguels-MacBook-Pro->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Files hidden inside directory '/var/tmp/launchd'. Link count does not match number of files (2,3)." --END OF NOTIFICATION
Rootcheck Gah OSSEC HIDS Notification. 2012 Nov 12 09:36:16 Received From: example->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): File /var/www/sites/default/settings.php is owned by root and has written permissions to anyone." --END OF NOTIFICATION
Active Response now what?
Active Response OSSEC HIDS Notification. 2014 Jun 28 23:52:41 Received From: (example) 11.22.33.44->/var/log/apache2/other_vhosts_access.log Rule: 31151 fired (level 10) -> "Multiple web server 400 error codes from same source ip." Portion of the log(s): 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/id_rsa HTTP/1.1" 404 - "-" "-" 63 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/id_rsa.old HTTP/1.1" 404 - "-" "-" 63 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/key.priv HTTP/1.1" 404 - "-" "-" 61 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/key HTTP/1.1" 404 - "-" "-" 56 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/dsa HTTP/1.1" 404 - "-" "-" 56 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/rsa HTTP/1.1" 404 - "-" "-" 56 143 162.243.13.133 - - [28/Jun/2014:23:52:39 +1000] "HEAD /.ssh/id_dsa HTTP/1.1" 404 - "-" "-" 59 143 --END OF NOTIFICATION OSSEC HIDS Notification. 2014 Jun 28 21:36:54 Received From: (example) 11.22.33.44->/var/log/nginx/access.log Rule: 31151 fired (level 10) -> "Multiple web server 400 error codes from same source ip." Portion of the log(s): 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.11.1-all-languages/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.11.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.2.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.1.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.0.2/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //php/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" 89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //cpphpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-" --END OF NOTIFICATION
Active Response Most crawlers are harmless / absurd But you re human You don t need them hunting for holes You don t need them using up active HTTP connections
Active Response And what if they find another ckeditor/drupal Core hole? (before you)
Active Response firewall-drop.sh most common response (but can be anything you want) Blocks the IP for a defined period (like fail2ban)
Active Response firewall-drop.sh Can employ repeat offender punishment When using server->agent model: One agent detects : every agent blocks
Active Response Drupal behind loadbalancers/varnish? Make sure you have IPs logging correctly Nginx/Apache to log X-Forwarded-For as client IP $conf[ reverse_proxy ] $conf[ reverse_proxy_addresses ]
Configuration Rules XML based ( ) <rule id="100112" level="10"> < level 10 - bad enough to take action > <if_sid>31151</if_sid> < if the rule that was hit was 31151 ( many 40X errors ) > <program_name>^drupal</program_name> < if the program that logged the message was drupal > <match>node/add</match> < if the log contained this message (hitting /node/add) > <options>no_email_alert</options> < don t e-mail me these, but still take action (level 10) > <description>spambots trying to add content</description> </rule>
Configuration Rules Dreaded rule 1003 Message too large (Mollom verbosity = a classic) <rule id="100113" level="0"> < Shush don t e-mail me and don t take action (harmless) > <if_sid>1003</if_sid> < if the rule that was hit was 1003 ( Message too large ) > <program_name>^drupal</program_name> < if the program that logged the message was drupal > <match>retrieved new CAPTCHA</match> < if the log contained this message > <description>large Mollom messages</description> </rule>
Configuration Frequency <rule id="104120" level="6"> < sometimes an accident (wrong password), level 10 a bit too harsh? > <if_sid>104110,1002</if_sid> <match>login attempt failed</match> <description>drupal failed login</description> </rule> - <rule id="104130" level="10" frequency="8" timeframe= 120"> < but too many, too quickly > <if_matched_sid>104120</if_matched_sid> < matching our rule above > <description>possible Drupal brute force attack</description> <description>(high number of logins).</description> </rule>
Tiered Notifications Alert devs to PHP errors, warnings, or specific servers Alert sysadmins to OS stuff (file integrity, kernel panics, software/network changes etc) CISO / manager / Redmine for whatever the ISO27001 incident register needs to include
Reporting ossec-reportd Who tried to login to the site as admin last March & from where? Top 10 users to SSH in during June? (Bob s on holiday - why/how did he login? Compromised SSH key?)
What to take from this? No mysteries You are human: you don t watch logs all day You make mistakes: save time debugging/finding the logs, let machines tell you Instant awareness: coverage over entire infra
What to take from this? Knowledge Is Power Not everything is a risk, but it can matter (impact) The element of surprise is an adversary s weapon It s harder to surprise someone with their finger on the pulse
What to take from this? OSSEC is not (meant to be) perfect False positives Docs/rule syntax can be a learning curve Software-based (VPS? What about the host?) HIDS, not NIDS (look at SecurityOnion, Bro, Splunk)
mig5 says: Filter out the noise to avoid monitoring fatigue But don t ignore rule 1002, tune it Whitelist your IPs: don t lock yourself out Defense in depth: Run other layers of integrity checking too (NIDs, WAFs, rkhunter, ClamAV scans)
Resources These slides Website Mailing List Monitoring Drupal with OSSEC OSSEC book Different syslog identity per site My quick-start install script: https://mig5.net/files/ossec.pdf http://www.ossec.net http://groups.google.com/groups/ossec-list http://www.madirish.net/428 http://is.gd/ossec_book http://is.gd/ossec_drupal_identities http://is.gd/ossec_install