4/7/16. Logging for IR. Planning Central Logging

Transcription

1 Logging for IR Planning Central Logging 1

2 Objectives Failures of distributed logging Pros and cons of centralized logging Centralized logging considerations Possible solutions A Sample solution Distributed logging Logging on each system is great An administrator can pull the logs and search for troubles Can research back in time to see what occurred on a system Logging into 10, 100, 1000, 10,000 or 100,000 different machines is not practical Miss system wide events 2

3 Centralized Logging Advantages Security Logs reside apart from their originals Protection from loss Comparison to find changes Complicates attackers activities More Difficult to cover tracks Needs access to more than one machine Major source of events New Users/groups/membership New software Scanning and password attacks Centralized Logging Advantages Visibility Logs in one place No longer necessary to visit each machine Can search for patterns across many machines Single log patterns makes for easy searching Move logs off of systems quickly If a system is malfunctioning it can be difficult to retrieve logs Logs can be lost altogether if a malfunction is large enough Access by non IT people Not all logs are for IT Sales, Accounting, and HR may want to see logs of certain systems 3

4 Centralized Logging Advantages Proactive Actions Resource exhaustion Errors that show future failure Poor network performance Spare local resources Takes space to store logs centralizing means space is centralized Processing power logs must be filtered and rotated, keeping them small which limits cpu time Considerations: Human Resources Why collect logs if no one will look at them? There will be thousands, possibly millions of events It will be someone s job to sort and draw conclusions from the data Some can be automated Will require significant configuration Someone with knowledge of the system Will require ongoing time and effort if it is to be relevant Not normally considered when implementing a logging solution Main Cause of failure Management must plan for this! 4

5 Considerations: Physical Resources Log storage takes up space Amount will be massively dependent on What is logged Size of logs Number of events per second (EPS) Averages Each log is about 34 bytes (based on cisco logging) If each machine produced 1000 logs per day and you had 100 machines that is about a year s worth of logs per gig of storage Considerations: Storage Formats Text files (syslog) Easily compressible Easy to read files Difficult to aggregate EVT files (Windows uses compressed xml) Space efficient Need Windows event viewer to view Database SQL, MySQL, PostGreSQL Great for search ability Cost and complexity Other formats Non-schema database (ElasticSearch) Great search ability Complexity 5

6 Considerations: Retention needs Related closely to space needs Retained logs do not need to be easily searchable (at first) Longer retention allows for more diagnosis Attacks could last weeks or months If logs are destroyed the original breach may be lost Court cases may last years! Company council may wish to see records deleted Industry standards may dictate storage length Considerations: Log Security Server Treasure trove of information about network Logs reveal usernames (sometimes passwords) Attacker MUST have access to cover tracks Therefore -- log servers are huge targets Log server lockdown Single purpose Limit open ports Limit network access to and from them (most communication is in) Logs Themselves Encrypted drives may be necessary Encryption for removable media may also be indicated 6

7 Considerations: Event Transport Reliable transport of events to central log server TCP guarantees delivery (slower and can be overwhelmed) UDP faster but based on best effort (can be ok if program takes care of reliability issues) Prevent unauthorized messages Can cause message loss by DOS Make finding legitimate messages more difficult Prevent unauthorized message viewing Some messages contain important system data Attackers that can intercept log messages may gain knowledge about network Considerations: Log Auditing Are all machines participating in logging? Keep lists of machine names Make sure every machine sends at least one event per day (even if you have to create it yourself) Check any machine that fails to report Scan periodically for new devices when possible 7

8 Considerations: Timing An event is basically a message and a time stamp The time stamp is generated by the system who created the event When combining events from many different machines it is best if they AGREE as to what time it is Log correlation is nearly impossible when the clocks are wrong Network Time Protocol (NTP) best option for maintaining clocks Setup beyond scope of this class Linux tool Microsoft maintains a similar tool (necessary for Kerberos) Considerations: Multiple Formats Syslog Oct 17 08:59:24 peradam.cs.colorado.edu sendmail[21601]: e9hexow21601: SYSERR(root): Can't create transcript file./xfe9hexow21601: Permission denied Apache log [28/Jul/2006:10:22: ]"GET / HTTP/1.0" PFSense Log Jan 11 07:28: pf: rule 141/0(match): block in on bge0: (tos 0x0, ttl 128, id 58078, offset 0, flags [none], proto UDP (17), length 1052) > : UDP, length 1024 Each system has its own unique way of logging if there is to be a common search, each log must be parsed The logging solution will need to provide a mechinism for it 8

9 Central Logging Solutions: Windows Windows comes with a solution for logging Windows Event collector services Data stored as event logs on specified Windows server Sent via web services management Installed in Windows 8 and 2k8 and above Can be added to xp and older (no longer supported) Same service Microsoft Systems Operations Manager uses MOM will scan for patterns (some given but can be user generated) Will generate alerts based on findings Windows Log Subscription 9

10 Central Logging Solutions: Windows MOM costs some thousands Designed for windows only Can receive syslog entries too They are accepted by port 514 udp No control (within software) for what servers can send events No guaranteed delivery Central Logging Solutions: Third party Splunk Widely considered best Considered by many to be expensive Micros installs (<500 MB free) From.5 10 GB $4500 per gig From GB $2500 per gig Estimation difficult but ~3.5 gig per 1000 users 4500*4= $18K Get what you pay for Easy install Easy Configuration Many alert rules 10

11 Central Logging Solutions: Third party Others Arcsite Full system for looking at logs Allows for searching and alerting on logs Processes many different types of logs including event log Licensed LogRythm Log searches Alerting Processes many different types of logs including event log Licensed ELK Stack (Elastic Search Logstash and Kibana) Log retrieval Robust search capability Limited alerting Open Source Central Logging Solutions: Third party Convert all logs to syslog and log the results KIWI Syslog server ( Snare ( Adicson WinSyslog ( OSSEC Open source software using agents to pull log data from Windows machines (more later) 11

12 Central Logging Solutions: DIY Sometimes necessary to pull logs manually Linux is easy -- simply ssh in and copy off the logs Windows takes a little more Powershell Get-winevent can query local and remote logs Export-csv will export any input to text files I.E. get-winevent application export-csv output.csv Get-wmiobject (wmi objects include a wide array of items) Central Logging Solutions: DIY Wmi scripting Find examples online on msdn Free script by Dumpeventlogs.vbs Will dump clear and sort logs 12

13 An Open Source Solution What we want to do Preparation We need a tool that aggregates logs to one spot It needs to translate the different type of logs into a format that can be searched The data needs to be transmitted in such a way that they can t be intercepted The system needs to audit which systems can transmit data to prevent intentional overload Discovery The logs must be examined for malicious patterns or other oddities The system should then alert the administrator of anything suspicious 13

14 What we want to do Containment It should then allow the administrator to search the logs for further evidence of the incident Discovering the extent of the breach If the evidence is convincing enough it should take preventative actions Eradication Deep inspection of the logs will tell investigators what occurred on the system This will allow them to clean up the mess What we want to do Recovery The ability to add rules to closely monitor servers put back on the network Lessons learned The ability to produce graphs and data to show how the attack progressed and was stopped Allow Investigators to see what went well and what did not 14

15 One Solution Using Open Source Tools Combine 4 pieces of software by two different companies Trend Micro supports Open Source SECurity or OSSEC Elastic Supports ELK ElasticSearch Logstash Kibana OSSEC More than a log aggregator full host based intrusion detection system Rootkit checker File integrity checker Registry auditing Active response 15

16 OSSEC Agents are installed on as many systems as possible Agents will collect data and send it to the OSSEC server for analysis Transmissions are guaranteed and encrypted Endpoints are authenticated to prevent unauthorized messages OSSEC Alerts are generated and placed into the alerts.log file Alerts can also be output in json format (latest build only) All logs can be archived (if activated) in the archive folder 16

17 OSSEC Alerts based on rules Several hundred rules are included New rules added regularly Rules can be generated by administrators Administrator rules can override package rules OSSEC Alerts have a severity rating 1-3 information 4-10 Possible bad action High chance of foul play Alerts can trigger automated response Lock down firewall Kill connections Alerts can also generate e- mails or texts 17

18 OSSEC The OSSEC project does include a web interface It is poor.3 is the current stable beta.8 is newer and fixes many issues ELK ELK is three products that work together to provide a robust log search tool Logstash is the front end It takes log information from various systems It sorts it and splits it up into index able fields It then stores the data in ElasticSearch ElasticSearch is a schema-less database engine It stores data in a free form manner The data can then be indexed and searched Kibana is a web interface It interfaces with ElasticSearch Allows the data to be searched, sorted, and displayed All kinds of abilities to create dashboards 18

19 OS Solution Pros/Cons Pros This solution works Meets all criterion Free to implement Requires only human resources and servers Server requirements relatively low Cons Setup not Trivial Nearly all Configuration via text files Different software requires different types of syntax Will require significant tweaking (true of all solutions) OS Solution Pros/Cons Note the full implementation is beyond the scope of this course A quick write up provided in the appendix of the labs Even if you follow the instructions it will require some modification to integrate into your environment All the pieces of software are documented on the internet and have paid support if you would like to purchase it 19

20 Final lab Exploring the Open source solution Conclusion Windows logging We discussed what could be logged Showed how to log deep details Linux logging Spoke about the various logs automatically generated Added a few logs that could help Central logging Spoke about the advantages and disadvantages Discussed some of the products Showed an open source solution 20