OSSEC. Intrusion detection and response System and log analysis of Drupal sites and servers

Transcription

1 OSSEC Intrusion detection and response System and log analysis of Drupal sites and servers

2 Looked at your logs recently? Client site, November [04/Nov/2012:05:48: ] "POST HTTP/1.1" "-" "-" [04/Nov/2012:05:49: ] "POST HTTP/1.1" "-" "-" [04/Nov/2012:05:49: ] "GET HTTP/1.1" "-" "-" [04/Nov/2012:06:27: ] "POST cookies=1&showimg=1&truecss=1&t2122n=1 HTTP/1.1" C99 (R57) shell (PHP-based Backdoor)

3 Looked at your logs recently? /var/log/apache2 crawlers hunting for holes brute-forcing /user/password, /user/register error 500, 504 (gateway timeouts, slow PHP?) worse (see last slide)

4 Looked at your logs recently? /var/log/syslog (Drupal) brute forcing (in more detail) exceptions, permissions problems crashes, panics, timeouts external service drama: Mollom, Payment GW

5 Looked at your logs recently? /var/log/auth.log SSH, user/group modifications Use of sudo Whoa, someone ran sudo nano /var/www/ drupal/includes/bootstrap.inc on prod???

6 Attacks: just one risk Bad practice Human error Dependant services (third parties) Packages installed or removed (/var/log/apt/history.log) all has impact, all in the logs

7 What s not in logs? Bad practice and human error :) insecure permissions on files Or bigger problems: unexpected modifications (Apache modules?)

8 What s security? Security is not just about intrusions Security is anything that could compromise availability, integrity, reliability, trust, money

9 What to do about it? Enter

10 What is OSSEC? Free and open source Host-based intrusion detection system (HIDS) Acquired by Trend Micro in just released Active development / community

11 Installation Download and run install.sh Try my wrapper script (does checksums etc) Not in Debian yet - PPA exists

12 OSSEC model Server->agent mode (central config, active response propagates) Local mode (standalone) Hybrid mode (multi-tier, complex topography)

13 The 4 main features Log analysis (What s happening now that s being logged?) Syscheck (integrity checking - what happened that left traces?) Rootcheck (rootkit detection) Active Response (what to do about it?)

14 Log Analysis what s happening?

15 Log Analysis How does it work? Decoders How to interpret logs (regexes to split up timestamps, IPs, messages) Rules Analysis of the decoded parts Grading them by level/threshold

16 Log Analysis Out of the box: SSH (bruteforcing, first time user logged in ) First time user executed sudo SMTP (spam relay attempts, SASL bruteforcing) Apache/Nginx issues (40Xs, 50Xs)

17 Log Analysis Write your own Decoders (custom logs) such as Drupal watchdog (Syslog module) Thank you madirish :

18 Log Analysis OSSEC HIDS Notification Jun 23 18:11:38 Received From: (example) >/var/log/messages Rule: fired (level 10) -> "Possible Drupal brute force attack (high number of logins)." Portion of the log(s): Jun 23 18:11:38 example drupal: user index.php?q=user/login 0 Login attempt failed for wembleylman10. Jun 23 18:11:36 example drupal: user index.php?q=user/login 0 Login attempt failed for wembleylman10. Jun 23 18:09:12 example drupal: user Login attempt failed for arrevemof. Jun 23 18:09:12 example drupal: user Login attempt failed for arrevemof. Jun 23 18:09:09 example drupal: user Login attempt failed for abralfultifug. Jun 23 18:09:09 example drupal: user Login attempt failed for abralfultifug. --END OF NOTIFICATION

19 Log Analysis Resource problems? (bottleneck/memory leak?) OSSEC HIDS Notification May 07 14:49:44 Received From: (example) >/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): May 7 14:49:43 example drupal: php PDOException: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: DELETE FROM {XXXXXXXXX} #012WHERE (uid = :db_condition_placeholder_0) AND (subid = :db_condition_placeholder_1) ; Array#012(#012 [:db_condition_placeholder_0] => 68148#012 [:db_condition_placeholder_1] => 77217#012)#012 in XXXXXXX_update::delete() (line 652 of /var/www/drupal/www/sites/all/modules/custom/xxxxxx/xxxxx.inc). --END OF NOTIFICATION OSSEC HIDS Notification Jun 14 15:17:02 Received From: (example) >/var/log/messages Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): - Jun 14 15:17:02 example ool www: PHP Fatal error: Allowed memory size of bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/www/sites/ all/modules/contrib/views/modules/field/views_handler_field_field.inc on line END OF NOTIFICATION

20 Log Analysis Everything is a DNS problem OSSEC HIDS Notification Jul 03 17:35:27 Received From: (example) >/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): Jul 1 17:35:26 example drupal: php User error: Failed to connect to memcache server: mem01.example.com:11211 in dmemcache_object() (line 415 of /var/www/drupal/www/sites/all/modules/ memcache/dmemcache.inc). --END OF NOTIFICATION OSSEC HIDS Notification Jun 29 10:01:53 Received From: (example) >/var/log/syslog Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): Jun 29 10:01:51 example drupal: brightcove Loading Brightcove video failed. --END OF NOTIFICATION

21 Syscheck what s changed?

22 Syscheck Scheduled scans Detects when files have changed (checksums) lots of false positives due to software patching, find the balance between useful/fatiguing New services start or stop (netstat: port watching)

23 Syscheck OSSEC HIDS Notification Jul 01 04:01:03 Received From: (example) >syscheck Rule: 550 fired (level 7) -> "Integrity checksum changed." Portion of the log(s): Integrity checksum changed for: '/usr/bin/ssh'" " " " " " << hopefully that s legit because you recently patched OpenSSH.. Size changed from '434024' to '641640' Old md5sum was: ' f654d7a2d7b38a0b0c09def4' New md5sum is : 'a8bf35316eb4f46e377a957ecb6cfdca' Old sha1sum was: '976af6f53338a7e9d4eb71617a2a8471aeb6937b' New sha1sum is : 'e871e0a907cdfb76c6e0722a6196b0c9f8edb1fd' --END OF NOTIFICATION

24 Syscheck OSSEC HIDS Notification Jul 01 17:43:20 Received From: (example) >netstat -tan grep LISTEN grep -v sort Rule: 533 fired (level 7) -> "Listened ports status (netstat) changed (new port opened or closed)." Portion of the log(s): ossec: output: 'netstat -tan grep LISTEN grep -v sort': tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN " " " New process listening on TCP port 5666?" tcp : :* LISTEN tcp6 0 0 :::111 :::* LISTEN tcp6 0 0 :::25 :::* LISTEN tcp6 0 0 :::4949 :::* LISTEN tcp6 0 0 :::58285 :::* LISTEN Previous output:" ossec: output: 'netstat -tan grep LISTEN grep -v sort': tcp : :* LISTEN tcp : :* LISTEN " " " ^^^ SSH no longer running above? Did you push a bad config via Puppet?" tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN tcp : :* LISTEN

25 Rootcheck who left the door open?

26 Rootcheck I like rkhunter too, but get a 2nd opinion Hopefully more false positives than not OSSEC HIDS Notification Nov 20 23:37:22 Received From: (example) >rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Anomaly detected in file '/tmp/#sql_1020_0.myi'. Hidden from stats, but showing up on readdir. Possible kernel level rootkit." --END OF NOTIFICATION - OSSEC HIDS Notification Jul 07 10:24:18 Received From: Miguels-MacBook-Pro->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Files hidden inside directory '/var/tmp/launchd'. Link count does not match number of files (2,3)." --END OF NOTIFICATION

27 Rootcheck Gah OSSEC HIDS Notification Nov 12 09:36:16 Received From: example->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): File /var/www/sites/default/settings.php is owned by root and has written permissions to anyone." --END OF NOTIFICATION

28 Active Response now what?

29 Active Response OSSEC HIDS Notification Jun 28 23:52:41 Received From: (example) >/var/log/apache2/other_vhosts_access.log Rule: fired (level 10) -> "Multiple web server 400 error codes from same source ip." Portion of the log(s): [28/Jun/2014:23:52: ] "HEAD /.ssh/id_rsa HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/id_rsa.old HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/key.priv HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/key HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/dsa HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/rsa HTTP/1.1" "-" "-" [28/Jun/2014:23:52: ] "HEAD /.ssh/id_dsa HTTP/1.1" "-" "-" END OF NOTIFICATION OSSEC HIDS Notification Jun 28 21:36:54 Received From: (example) >/var/log/nginx/access.log Rule: fired (level 10) -> "Multiple web server 400 error codes from same source ip." Portion of the log(s): [28/Jun/2014:21:34: ] "GET //phpmyadmin all-languages/scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin /scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //php/phpmyadmin/scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //forum/phpmyadmin/scripts/setup.php HTTP/1.1" "-" "-" [28/Jun/2014:21:34: ] "GET //cpphpmyadmin/scripts/setup.php HTTP/1.1" "-" "-" --END OF NOTIFICATION

30 Active Response Most crawlers are harmless / absurd But you re human You don t need them hunting for holes You don t need them using up active HTTP connections

31 Active Response And what if they find another ckeditor/drupal Core hole? (before you)

32 Active Response firewall-drop.sh most common response (but can be anything you want) Blocks the IP for a defined period (like fail2ban)

33 Active Response firewall-drop.sh Can employ repeat offender punishment When using server->agent model: One agent detects : every agent blocks

34 Active Response Drupal behind loadbalancers/varnish? Make sure you have IPs logging correctly Nginx/Apache to log X-Forwarded-For as client IP $conf[ reverse_proxy ] $conf[ reverse_proxy_addresses ]

35 Configuration Rules XML based ( ) <rule id="100112" level="10"> < level 10 - bad enough to take action > <if_sid>31151</if_sid> < if the rule that was hit was ( many 40X errors ) > <program_name>^drupal</program_name> < if the program that logged the message was drupal > <match>node/add</match> < if the log contained this message (hitting /node/add) > <options>no_ _alert</options> < don t me these, but still take action (level 10) > <description>spambots trying to add content</description> </rule>

36 Configuration Rules Dreaded rule 1003 Message too large (Mollom verbosity = a classic) <rule id="100113" level="0"> < Shush don t me and don t take action (harmless) > <if_sid>1003</if_sid> < if the rule that was hit was 1003 ( Message too large ) > <program_name>^drupal</program_name> < if the program that logged the message was drupal > <match>retrieved new CAPTCHA</match> < if the log contained this message > <description>large Mollom messages</description> </rule>

37 Configuration Frequency <rule id="104120" level="6"> < sometimes an accident (wrong password), level 10 a bit too harsh? > <if_sid>104110,1002</if_sid> <match>login attempt failed</match> <description>drupal failed login</description> </rule> - <rule id="104130" level="10" frequency="8" timeframe= 120"> < but too many, too quickly > <if_matched_sid>104120</if_matched_sid> < matching our rule above > <description>possible Drupal brute force attack</description> <description>(high number of logins).</description> </rule>

38 Tiered Notifications Alert devs to PHP errors, warnings, or specific servers Alert sysadmins to OS stuff (file integrity, kernel panics, software/network changes etc) CISO / manager / Redmine for whatever the ISO27001 incident register needs to include

39 Reporting ossec-reportd Who tried to login to the site as admin last March & from where? Top 10 users to SSH in during June? (Bob s on holiday - why/how did he login? Compromised SSH key?)

40 What to take from this? No mysteries You are human: you don t watch logs all day You make mistakes: save time debugging/finding the logs, let machines tell you Instant awareness: coverage over entire infra

41 What to take from this? Knowledge Is Power Not everything is a risk, but it can matter (impact) The element of surprise is an adversary s weapon It s harder to surprise someone with their finger on the pulse

42 What to take from this? OSSEC is not (meant to be) perfect False positives Docs/rule syntax can be a learning curve Software-based (VPS? What about the host?) HIDS, not NIDS (look at SecurityOnion, Bro, Splunk)

43 mig5 says: Filter out the noise to avoid monitoring fatigue But don t ignore rule 1002, tune it Whitelist your IPs: don t lock yourself out Defense in depth: Run other layers of integrity checking too (NIDs, WAFs, rkhunter, ClamAV scans)

44 Resources These slides Website Mailing List Monitoring Drupal with OSSEC OSSEC book Different syslog identity per site My quick-start install script: