SB34: Event Logs Don t Lie: Step-by-Step Security Rick Simonds, Sage Data Security
AGENDA 1. Learn best practices for event and audit log review. 2. Learn which devices to track and monitor. 3. Learn how to determine what information you need to review, and how often to review it. 4. Learn the benefits of collecting, aggregating, and correlating event data to help identify breaches and attacks as well as create a baseline for normal activity. 5. Learn how to balance security compliance objectives and staff/resource limitations. 6. Learn what tools are available to ease the management of log data.
The Threats It s no secret that information security is critical to business success. Even the best networks are at risk. Malicious attacks from unknown/unauthorized sources. Unauthorized access to or against your systems from either internal or external locations. These are not nuisance attacks. They are bonafide criminal activity. Malicious attacks from known/authorized sources. A significant number of attacks are generated by insiders authorized users, business partners, and third-party service providers. Unfortunately, not all of these individuals are trustworthy.
The Threats Continued Proxy attack scenarios. It is very common for an attacker to use computers distributed throughout the world as weapons. This process is transparent to the system owner. No one wants to have their computer systems used this way. Having your computer systems used as part of a larger threat certainly flies in the face of good corporate citizenry and can cause major reputational damage. Unintended breaches created from human error. Not all threatening activity is malicious sometimes, people just make mistakes or are fooled into taking action. Privacy and regulatory compliance violations. Many organizations have a legal and a fiduciary obligation to safeguard protected information. Violations however unintentional can have serious ramifications.
Logs Don t Lie Mining and monitoring the information generated by the logs of your network and technology devices offers a wealth of information to help protect your organization. Each log offers clues about hacking attempts or attacks as well as on innocent activities that have unexpected - and possibly harmful - consequences. Factual. Event and audit logs created by network devices are accurate and unbiased. Reliable. Logs don t take holidays or sick days. Standard. Logs report events and activity in a consistent manner. Timely. Logs document activity as they happen. When properly implemented and analyzed, event and audit logs provide the information and insight needed for proactive risk management.
Prioritization An organization should define its requirements and goals for performing logging and monitoring logs to include applicable laws, regulations, and existing organizational policies. Determining which devices are critical, and which information is significant, is not a one-size-fits-all proposition. Note: organizations should conduct an impact assessment of its network prior to establishing a log-capture and -review program. Other considerations include: type of information to be logged, storage collection and archiving storage requirements, analysis technique, and oversight responsibilities. Minimum Security Log Device Category Recommendations: Border Devices such as firewalls, routers, IDS Authentication Servers such as Windows Active Directory Domain Controller, Novell NDS Servers, Radius Servers Web Servers such as IIS, Apache
What to Look for? Clues, Hints, and Observations Firewalls Unusual pattern or volume of internal and external Common traffic Unexpected types of traffic Firewall administrator logons Firewall rule set changes Firewall bandwidth and utilization Authentication Server User Activity: Invalid passwords, password changes, account lockouts, activity outside of normal times User Management: New accounts, changes to system rights and privileges Group Management: Creation or deletion of groups, addition of users to high security groups Computer Management: Policy changes (inc. audit policies), clearing audit logs, adding computer accounts, service resets, reboots Web Servers Entries that result in errors: i.e. 404 Page not Found, 403 Forbidden, 500 internal server error Hacking tools Directory traversals SQL injection attempts Site mirroring A common mistake is to focus only on denied activity.
Log Output Firewall 2007-07-24 07:00:15 Daemon.Notice 192.168.162.2 firewall.domain.com kernel [1304] 20166: Sending ICMP unreachable, IP Code=Unreachable (port), Source IP=71.19.166.11, Destination IP=64.69.119.75, IP Code=UDP, Source Port=32857, Destination Port=53, Interface=eth1 2007-07-24 07:00:15 Daemon.Error 192.168.162.2 firewall.domain.com Authentication [1447] 40463: An invalid user tried to login to the system using the SSH service. Original syslog message: [Firewall sshd[29132]: Failed password for invalid user admin from 222.68.195.14 port 36243 ssh2]2007-07- 24 07:00:15 Daemon.Notice 192.168.162.2 firewall.domain.com kernel [1304] 20166: Sending ICMP unreachable, IP Code=Unreachable (host prohibited), Source IP=192.168.162.3, Destination IP=192.168.162.1, IP Code=UDP, Source Port=123, Destination Port=123, Interface=eth2 Windows Server 2007-09-19,2007-0914:31:27,SERVER06,529,16,"serviceacct01 4 Advapi MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 SERVER- 06"2007-09-19,2007-09-19 14:57:25,SERVER-15,632,8,"CN=Sam Horn,OU=Staff,DC=THISDOMAIN,DC=internal %{S-1-5-21-997095950- 1628968691-619646970-1081} Domain Admins THISDOMAIN %{S-1-5-21-997095950-1628968691-619646970- 512} adminacct07 THISDOMAIN (0x0-0x15A15028) -"2007-09-19,2007-09-19 15:12:08,SERVER 09,529,16,"adminacct03 THISDOMAIN 10 User32 Negotiate SERVER-09" 2007-09-19,2007-09-19 15:53:43,SERVER-23,576,8,"- - (0x0-0xB95337) SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege SeAuditPrivilege"2007-09-19,2007-09-19 15:54:58,SERVER- 16,529,16,"tcruise OurCompany.com 8 Advapi Negotiate SERVER-44"2007-09-19,2007-09-19 16:02:56,SERVER- 16,529,16,"tcruise OurCompany.com 8 Advapi Negotiate SERVER-44"2007-09-19,20 Web Server 2007-05-30 01:27:38 192.168.154.5 HEAD /personal-banking/images/visagiftcardweb1206_000.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406 2007-05-30 01:27:39 192.168.154.5 GET /business-banking/index.asp - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 200 2007-05-30 01:27:39 192.168.154.5 HEAD /business-banking/images/nophishingwhitetag.gif - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 404 2007-05-30 01:27:39 192.168.154.5 GET /a1b2c3d4e5f6g7h8i9/nonexistentfile.php - 80-219.149.232.60 2007-05-30 01:27:39 192.168.154.5 GET /adxmlrpc.php - 80-219.149.232.60-404 0 64 468 2007-05-30 01:27:40 192.168.154.5 GET /adserver/adxmlrpc.php - 80-219.149.232.60-404 0 64 468 2007-05-30 01:27:41 192.168.154.5 GET /phpadsnew/adxmlrpc.php - 80-219.149.232.60-404 0 64 484 2007-05-30 01:27:41 192.168.154.5 HEAD /images-headers/header_business.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406 2007-05-30 01:27:41 192.168.154.5 HEAD /business-banking/images/businessbankingweb.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406
Raw Log Manipulation Log parsing is extracting data from a log so that the parsed values can be used as input for another logging process. A simple example of parsing is reading a text-based log file that contains 10 comma-separated values per line and extracting the 10 values from each line. Event filtering is the suppression of log entries from analysis because their characteristics indicate that they are unlikely to contain information of interest. For example, duplicate entries and standard informational entries might be filtered because they do not provide useful information to log analysts. In event aggregation, similar entries are consolidated into a single entry containing a count of the number of occurrences of the event. For example, a thousand entries that each record part of a scan could be aggregated into a single entry that indicates how many hosts were scanned. Log conversion is parsing a log in one format and storing its entries in a second format. For example, conversion could take data from a log stored in a database and save it in an XML format in a text file. In log normalization, each log data field is converted to a particular data representation and categorized consistently. One of the most common uses of normalization is storing dates and times in a single format. For example, one log generator might store the event time in a twelve-hour format (2:34:56 P.M. EDT) categorized as Timestamp, while another log generator might store it in twenty-four (14:34) format categorized as Event Time, with the time zone stored in different notation (-0400) in a different field categorized as Time Zone.
Log Analysis The meaning of an entry often depends upon the context surrounding it. Correlation ties individual log entries together based on related information. Sequencing examines activity based on patterns. Trend analysis identifies activity over time that in isolation may appear normal. Insight While tools and scripts can be used in the process of preparing, correlating, sequencing, and trending data, the final step in event and audit log management requires the human touch. Attention Even the best report that synthesizes the most valuable information into a concise format is worthless unless someone pays attention on a regular, consistent basis.
Actionable Intelligence While event log management is time-consuming, intricate, and challenging, the rewards are great for those that mine the data and turn analysis into actionable intelligence. From the 5/30/07 Web Server Log -scripted php scan A device at 222.68.195.14, on a " ChinaNet Shanghai Province Network " network in China, generated errors scanning the domain, domain2, and domain3 web sites. This traffic appears to be a scan for php-based vulnerabilities performed between 01:37:39 and 01:37:42 GMT on 05/30/2007 2007-05-30 01:27:38 192.168.154.5 HEAD /personal-banking/images/visagiftcardweb1206_000.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406 2007-05-30 01:27:39 192.168.154.5 GET /business-banking/index.asp - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 200 2007-05-30 01:27:39 192.168.154.5 HEAD /business-banking/images/nophishingwhitetag.gif - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 404 2007-05-30 01:27:39 192.168.154.5 GET /a1b2c3d4e5f6g7h8i9/nonexistentfile.php - 80-222.68.195.14 2007-05-30 01:27:39 192.168.154.5 GET /adxmlrpc.php - 80 22.68.195.14-404 0 64 468 2007-05-30 01:27:40 192.168.154.5 GET /adserver/adxmlrpc.php - 80-222.68.195.14-404 0 64 468 2007-05-30 01:27:41 192.168.154.5 GET /phpadsnew/adxmlrpc.php - 80-222.68.195.14.60-404 0 64 484 2007-05-30 01:27:41 192.168.154.5 HEAD /images-headers/header_business.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406 2007-05-30 01:27:41 192.168.154.5 HEAD /business-banking/images/businessbankingweb.jpg - 80-66.131.70.243 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0) - - 406
Actionable Intelligence From the 7/23/07 Windows Log - user & group management activity Administrative account adminacct07 created account Sam Horn and added account to Security Enabled Global Group "Domain Admins" on 07/23/2007 2007-07-23,2007-09723:31:27,SERVER06,529,16,"serviceacct01 4 Advapi MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 SERV ER-06, 2007-07-23, 14:57:25,SERVER-15,632,8,"CN=Sam Horn,OU=Staff,DC=THISDOMAIN,DC=internal %{S-1-5-21-997095950-1628968691-619646970-1081} Domain Admins THISDOMAIN %{S-1-5-21-997095950-1628968691-619646970-512} adminacct07 THISDOMAIN (0x0-0x15A15028) -"2007-09-19,2007-09-19 15:12:08,SERVER 09,529,16,"adminacct03 THISDOMAIN 10 User32 Negotiate SERVER-09" 2007-07-23, 9 15:53:43,SERVER-23,576,8,"- - (0x0-0xB95337) SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege SeAuditPrivilege 29, 2007-07-23 15:54:58,SERVER- 16,529,16,"tcruise OurCompany.com 8 Advapi Negotiate SERVER-44"2007-07-23 16:02:56,SERVER- 16,529,16,"tcruise OurCompany.com 8 Advapi Negotiate SERVER-44"2007-07-23 From the 7/24/07 Firewall Log - brute force attack on SSH A device at 222.68.195.14, on a ChinaNet Shanghai Province Network in China, attempted 1000 SSH login using the credentials SamHorn against the firewall on 7/24/07. All login attempts failed. 2007-07-24 07:00:15 Daemon.Notice 192.168.162.2 firewall.domain.com kernel [1304] 20166: Sending ICMP unreachable, IP Code=Unreachable (port), Source IP=71.19.166.11, Destination IP=64.69.119.75, IP Code=UDP, Source Port=32857, Destination Port=53, Interface=eth1 2007-07-24 07:00:15 Daemon.Error 192.168.162.2 firewall.domain.com Authentication [1447] 40463: An invalid user tried to login to the system using the SSH service. Original syslog message: [Firewall sshd[29132]: Failed password for invalid user SamHornfrom 222.68.195.14 port 36243 ssh2] 2007-07-24 07:00:15 Daemon.Notice 192.168.162.2 firewall.domain.com kernel [1304] 20166: Sending ICMP unreachable, IP Code=Unreachable (host prohibited), Source IP=192.168.162.3, Destination IP=192.168.162.1, IP Code=UDP, Source Port=123, Destination Port=123, Interface=eth2
Compliance Requirements Monitoring and reviewing activity is a core component of every information security regulation and law. Gramm Leach Bliley (GLBA) Health Insurance Portability and Accountability Act (HIPPA) Sarbanes-Oxley (SOX) Federal Information Security Management Act (FISMA) Payment Card Industry Data Security Standard (PCI DDS) State Security Breach Laws (39 States)
The Challenge Time & Resources Consistency Complexity Knowledge base Customization Independence
Demo - Culling Information from Raw Logs 1. The Raw Log 2. Parsing the Logs 3. Filtering the Events 4. Event Aggregation 5. Log Conversion 6. Log Normalization 7. Reviewing the Logs in a Readable Format
Tools and Methods to Make Log Review Manageable 1. Free resource kit tools 2. Third-party vendor products 3. In-house programming 4. Outsourcing