DETECTING ADVANCED THREATS WITH SYSMON, WEF AND ELASTICSEARCH

DETECTING ADVANCED THREATS WITH SYSMON, WEF AND ELASTICSEARCH JOSH LEWIS ROOT9B 05.25.15

DETECTING ADVANCED THREATS WITH SYSMON, WEF, AND ELASTICSEARCH WHY EVENT LOGS? From an advanced threat detection perspective, most analysts are relatively blind at the host level until they receive network telemetry or a security agent alert (Anti-Virus/HIPS). Based on my experience, network telemetry data is typically collected at network egress points and Anti-Virus/ HIPS is poor at detecting pivot and memory based attacks. One approach to overcome these obstacles is to increase visibility at the host level and create indicators of compromise that can trigger forensic investigation by alerting on specific event logs. Event logs are built natively into most operating systems and can immediately send valuable artifacts to a log collector prior to the attacker having the ability to modify the integrity of the system. Event logs are not a silver bullet. However, event logs can provide a tremendous amount of host telemetry data that can aid in the detection of an advanced adversary. Very few organizations collect the right event logs from relevant devices, and even fewer organizations are able to action these event logs. This article is designed to showcase a proof of concept architecture for detecting indicators of compromise through event logs. WHAT CAN WE DETECT? Examples of advanced threat detection that this architecture can detect include, but are not limited to: Outbound web connections not originating from Internet Explorer/Firefox/Chrome The presence of a hash across other machines on the network Lateral movement (file shares, remote desktop) within and between subnets For a detailed listing of what can be detected, read the section on overview of what can be detected HOW IS THIS ARCHITECTURE DIFFERENT FROM TRADITIONAL SIEM S OR OTHER SECURITY TOOLS? Based on my experience, most SIEM s are: Very expensive

Have overwhelming alerts that are not actionable Due to excessive cost, have limited data inputs and a fractional view of the environment This architecture provides: Scalable, elastic, parallel processing architecture Open source (free), not limited by events-per-second licensing Dynamic collection of indicators that can be turned off/on based on threat conditions Targeted, hyper-actionable indicators Additional host telemetry data for networks, running processes/binaries ARCHITECTURE Three design principles were taken into consideration during the development of this architecture: 1. Most Security Information and Event Management (SIEM) analysts are already overwhelmed with a mountain of non-actionable logs. Log analysis architectures strive to collect relevant/actionable information that can be used as an indicator of compromise. 2. Most SIEM s are at capacity from a storage and bandwidth perspective; it is not be feasible for an organization to turn on client side logging and forward all logs without a significant infrastructure upgrade. This architecture is designed for quick, actionable and scalable deployments using open source (FREE) tools. 3. The log collection and analysis tools should be easy to use and require little training/ develop to gain instant return on investment and innovation.

ARCHITECTURE OVERVIEW DETECTING INDICATORS OF COMPROMISE THROUGH EVENT LOGS CONCEPTUAL ARCHITECTURE

WINDOWS EVENT FORWARDING (WEF) WEF BENEFITS The WEF architecture forwards events in the native event log format. This is helpful because the event log format uses XML to cleanly structure data into different fields that will be helpful when querying in ElasticSearch. Alternatively, using the syslog protocol will flatten all data structure, requiring parsers to be built for each event type. WEF utilizes group policy, so all clients that join a network or change OUs will automatically begin to participate in the log forwarding architecture. This is helpful to ensure completeness of coverage from a logging perspective. WEF does not require an agent to be installed. The lack of an agent requirement is helpful as some organizations will be hesitant to install another agent on each box (e.g. event-to-sys) and ensuring completeness of agent deployment can be challenging as clients are constantly changing WEF encrypts all data between the log collector and the clients by default If using HTTP: WEF will use the Microsoft Negotiate security support provider (SSP) in workgroup environments or the Microsoft Kerberos SSP in domain environments If using HTTPS: WEF uses a Secure Sockets Layer (SSL) certificate which can be specified during setup WEF SETUP 1. Stand up a dedicated Windows Server 2008 R2 or later server to be used as an event collector. A dedicated server is suggested to handle the event log load from each of the clients. 2. On the log collector server, run the following commands a. winrm qc b. wecutil qc 3. Open the event viewer and create new event subscriptions that the clients will use to conditionally forward important security events a. Click Create New from the actions pane b. **Change the destination log to Forwarded Events c. Click the source computer initiated subscription option and add YourDomain\ Domain Computers group d. Click on configure events to collect and define the query parameters. Note that you can copy and paste the XML from some of the sample queries that I have already created. Reference appendix A.

FIGURE 1: BUILDING NEW SUBSCRIPTIONS IN EVENT VIEWER

WHAT DATA TO COLLECT THE FOLLOWING TABLE PROVIDES A LIST OF ACTIONABLE EVENT LOG IDS TO COLLECT. THIS TABLE WAS DERIVED FROM THE NSA ARTICLE REFERENCED IN THE APPENDIX AND MODIFIED BASED ON OPERATIONAL EXPERIENCE. LEVERAGE THIS DATA TO CREATE EVENT LOG SUBSCRIPTIONS. APPENDIX A ALSO PROVIDES A SCRIPT WITH PRE-BUILT XML QUERIES TO CREATE EVENT SUBSCRIPTIONS. CATEGORY EVENT ID Application whitelisting AppLocker Block 8003, 8004 Application whitelisting AppLocker Warning 8006, 8007 Application whitelisting SRP Block 865, 866, 867, 868, 882 Application crashes App Hang 1002 Application crashes Appcrash 1001 Application crashes EMET 2 System or Service Failures Windows Service Fails or Crashes 7022, 7023, 7024, 7026, 7031, 7032, 7034 Windows firewall Firewall Rule Add 2004 Windows firewall Firewall Rule Change 2005 Windows firewall Firewall Rules Deleted 2006, 2033 Anti-forensics Event Log was Cleared 104 Anti-forensics Audit Log was Cleared 1102 Anti-forensics Event log service stopped 6005 Software and Service Installation New Kernel Filter Driver 6 Kernel Driver Signing Detect an invalid image hash of 5038 a file Kernel Driver Signing Detected an invalid page hash of 6281 an image file Kernel Driver Signing Code integrity check 3001, 3002, 3003, 3004, 3010, 3023 Kernel Driver Signing Failed Kernel Driver Loading 219 Windows Defender Detected Malware 1006 Windows Defender Action on Malware failed 1008 Persistence Scheduled task created 4698 Powershell Use of Powershell 400 New Process New process create, include 1 image path, hash, parent process GUID, etc. New Process New Process Create with 4688 Command Line Options File Time/Data Change Change of a file create or last 2 modified date Remote Desktop Tracking use of RDP 21 Network Connection Network connection src & dst, requesting process 3

4. Note that subscriptions can be disabled/enabled (See screenshot below). The ability to enable/disable subscriptions is a great way to provide flexibility in advanced threat identification. The majority of windows events that are identified in the what data to collect section are hyper actionable, low signal-to-noise ratio (meaning, if you are receiving events, there is likely an investigation that should take place). However, sysmon generates quite a bit of data and may not be sustainable across an entire enterprise network. In this case, sysmon could be selectively enabled based on suspected intrusion events or other triggers. FIGURE 2: EXAMPLE SUBSCRIPTIONS; SCREENSHOT SHOW SYSMON SUBSCRIPTION ENABLED 5. Configure group policy to define the log collector server, enable windows remote management, start the remote management service, allow Network Service to read security event logs, and create firewall rules to allow WEF traffic on client computers a. Create a new group policy named Windows Event Forwarding b. Define the log collector server: Expand computer configuration>policies>administrative templates>control panel>windows components>event forwarding i. Edit the Configure the server address, refresh interval and issuer certificate ii. Select enabled iii. Select show iv. Enter Server=YourCollecterHostName.FQDNofYourDomain

FIGURE 3: GROUP POLICY CONFIGURATION FOR CLIENTS THAT SPECIFIES THE COLLECTOR SERVER c. Configure windows remote management: Expand computer configuration>policies>administrative templates>control panel>windows components>windows remote management(winrm) i. Select Enabled ii. Under IPV4 enter * iii. Under IPV6 enter * FIGURE 5: GROUP POLICY SETTING THAT CONFIGURES THE CLIENT WINRM SETTINGS (NSA, 2013)

d. Start the WEF service: Expand Computer configuration > policies> Windows Settings > Security Settings >System Services > Windows Remote Management. See image below (NSA, 2013) i. Double click on Windows Remote Management ii. Check Define this setting iii. Change the startup type to automatic FIGURE 6: ENABLING WINDOWS REMOTE MANAGEMENT FIGURE 7: SETTING SERVICE STARTUP TYPE e. Allow Network Service to read security logs (without this step, WEF will not be able to pull any data from the security logs): Computer configuration > policies > windows settings >security settings > restricted i. Right click restricted groups ii. Select add group iii. Type Event Log Readers and select check names, then OK

iv. Right click event log readers and select properties v. Click add and type Network Service vi. Select check names, then ok f. Add a firewall rule to allow WEF traffic: Computer configuration > Policies > Windows Settings >Security Settings > Windows Firewall with Advanced >Inbound rules i. Allow TCP 5985 g. Link the Windows Event Forwarding group policy to applicable Active Directory Organizational Units (OUs)

SYSMON SYSMON BENEFITS Per Microsoft, Sysmon provides the following capabilities: Logs process creation with full command line for both current and parent processes. Records the hash of process image files using SHA1 (the default), MD5 or SHA256. Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs. Optionally logs network connections, including each connection s source process, IP addresses, port numbers, hostnames and port names. Detects changes in file creation time to understand when a file was really created. Modification of file create timestamps is a technique commonly used by malware to cover its tracks. Generates events from early in the boot process to capture activity made by even sophisticated kernel-mode malware. SYSMON SETUP (WITHOUT GPO) Sysmon.exe i h SHA256 n FIGURE 8: SYSMON SETUP

SYSMON SETUP (WITH GPO) LOGSTASH During the setup of logstash 1.4.2, two problems occurred. First, logstash would not pull data from the Forwarded Event log file. Although the subscriptions can be configured to save data to another file this complicated the setup and analysis. Second, logstash had a memory leak and would stop running under a heavy load. To solve these issues, the architecture was tweaked slightly. NX Log CE was installed on the Windows Server event log collector and used to forward data to a logstash instance running on a Ubuntu instance. The logstash instance would listen for connections data from NX log and forward to the ElasticSearch cluster.

INSTALL NX LOG Download and install the binary from the link in the references. Create a configuration file and place in the configuration directory, similar to the screenshot below. FIGURE 9: SAMPLE COPY OF THE NX LOG CONFIGURATION LOGSTASH CONFIGURATION FILE SETUP Install java apt-get install default-jre update-alternatives --config java Capture location (e.g. /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java)

Vi /etc/environment Add new line JAVA_HOME= /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java source /etc/environment echo $JAVA_HOME Create a configuration file similar to the screenshot below and place in the /usr/share/ logstash/bin/ folder. RUNNING LOGSTASH C:\logstash\bin\logstash.bat agent f logstash.conf ELASTICSEARCH ELASTICSEARCH BENEFITS Scalable horizontal compute and storage (similar to Hadoop) ElasticSearch can accommodate node failure and automatically re-distribute shards without data loss ElasticSearch leverages parallel processing to execute queries on massive data sets very quickly Open source (free) Great for long tail analysis and querying

FIGURE 10: ELASTICSEARCH CONSUMING MASSIVE AMOUNTS OF DATA ELASTICSEARCH SETUP ElasticSearch will run out of the box with one command. However, it is highly encouraged that you become familiar with the different options within the elasticsearch/config/elasticsearch.yml configuration file. Note that the additional configuration items are not covered within the scope of this document. Prior to starting ElasticSearch, open the elasticsearch.yml file Remove the # on the line that contains #cluster.name: elasticsearch Change the elasticsearch to a different name that describes your cluster Note that new ElasticSearch instances will auto-join your cluster if they have the same cluster name Note that you may want to setup iptables or put the ElasticSearch cluster behind a firewall. ElasticSearch does not authenticate any connections by default. ElasticSearch listens on port 9200 for RESTful HTTP connections. FIGURE 11

Install java apt-get install default-jre update-alternatives --config java Capture location (e.g. /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java) Vi /etc/environment Add new line JAVA_HOME= /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java source /etc/environment echo $JAVA_HOME Start ElasticSearch./elasticsearch FIGURE 12: STARTING ELASTICSEARCH ON UBUNTU KIBANA BENEFITS OF KIBANA Kibana provides a beautiful interface that allows you to easily query ElasticSearch Using Kibana, you can quickly filter to perform long tail analysis and anomaly detection KIBANA SETUP Edit the config.js file and point Kibana to the ElasticSearch cluster Cange the IP address to the host IP address on line elastic search: http://127.0.0.1:9200

FIGURE 13: KIBANA CONFIG.JS CONFIGURATION FILE vi /etc/init/kibana.conf # auto start kibana start on filesystem exec /usr/share/kibana/bin/kibana chmod -R 755 /usr/share/kibana/ ln -s /etc/init/kibana.conf /etc/init.d/kibanae service kibana start Netstat -l grep 5601 REFERENCES Spotting the Adversary Through Windows Event Log Monitoring https://www.nsa.gov/ia/_files/app/spotting_the_adversary_with_windows_event_log_ monitoring.pdf

WEF Setting up a source initiated subscription http://msdn.microsoft.com/en-us/library/ windows/desktop/bb870973(v=vs.85).aspx Quick and dirty large scale eventing for windows http://blogs.technet.com/b/wincat/ archive/2008/08/11/quick-and-dirty-large-scale-eventing-for-windows.aspx Logstash Getting started http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash Eventlog input http://logstash.net/docs/1.4.2/inputs/eventlog NX Log CE http://nxlog-ce.sourceforge.net/ Sysmon Sysmon download http://technet.microsoft.com/en-us/sysinternals/dn798348 APPENDIX A WEF SUBSCRIPTIONS WEF Subscriptions can be created in event viewer then exported for backup or for quickly importing into other machines. export a subscription: wecutil gs Powershell Invocation /f:xml >Powershell.xml Note that Powershell Invocation is the name of the subscription in event viewer import a subscription: wecutil cs <InsertSubscriptionName>.xml SAMPLE APPLICATION CRASH SUBSCRIPTION <?xml version= 1.0 encoding= UTF-8?> <Subscription xmlns= http://schemas.microsoft.com/2006/03/windows/events/subscription > <SubscriptionId>Application Crashes</SubscriptionId> <SubscriptionType>SourceInitiated</SubscriptionType> <Description></Description> <Enabled>true</Enabled> <Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri> <ConfigurationMode>Custom</ConfigurationMode> <Delivery Mode= Push > <Batching> <MaxLatencyTime>60</MaxLatencyTime> </Batching> <PushSettings> <Heartbeat Interval= 120 />

</PushSettings> </Delivery> <Query> <![CDATA[ <QueryList><Query Id= 0 ><Select Path= Application >*[System[(EventID=1000 or EventID=1001 or EventID=1002)]]</Select><Select Path= System >*[System[(EventID=1000 or EventID=1001 or EventID=1002)]]</Select></Query></QueryList> ]]> </Query> <ReadExistingEvents>false</ReadExistingEvents> <TransportName>HTTP</TransportName> <ContentFormat>RenderedText</ContentFormat> <Locale Language= en-us /> <LogFile>ForwardedEvents</LogFile> <PublisherName>Microsoft-Windows-EventCollector</PublisherName> <AllowedSourceNonDomainComputers> <AllowedIssuerCAList> </AllowedIssuerCAList> </AllowedSourceNonDomainComputers> <AllowedSourceDomainComputers>O:NSG:BAD:P(A;;GA;;;DC)S:</ AllowedSourceDomainComputers> </Subscription> Sample New Process Create Subscription <?xml version= 1.0 encoding= UTF-8?> <Subscription xmlns= http://schemas.microsoft.com/2006/03/windows/events/subscription > <SubscriptionId>New Process Create With Command Line Options</SubscriptionId> <SubscriptionType>SourceInitiated</SubscriptionType> <Description>Native Eventlog Source, Moderate Traffic</Description> <Enabled>true</Enabled> <Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri> <ConfigurationMode>Custom</ConfigurationMode> <Delivery Mode= Push > <Batching> <MaxLatencyTime>60</MaxLatencyTime> </Batching> <PushSettings> <Heartbeat Interval= 120 /> </PushSettings> </Delivery> <Query> <![CDATA[ <QueryList><Query Id= 0 Path= Security ><Select Path= Security >*[System[(EventID=4688)]] and *[EventData[Data[@Name= NewProcessName ]!= C:\Windows\System32\VSSVC.exe ]] and *[EventData[Data[@Name= NewProcessName ]!= C:\Windows\System32\SearchFilterHost. exe ]] and *[EventData[Data[@Name= NewProcessName ]!= C:\Windows\System32\ SearchProtocolHost.exe ]] and *[EventData[Data[@Name= NewProcessName ]!= C:\Windows\ System32\Defrag.exe ]] and *[EventData[Data[@Name= CommandLine ]!= taskhost.exe network ]] and *[EventData[Data[@Name= CommandLine ]!= taskhost.exe ]] and *[EventData[Data[@ Name= CommandLine ]!= taskhostex.exe Regular ]]</Select></Query></QueryList> ]]> </Query>

<ReadExistingEvents>true</ReadExistingEvents> <TransportName>HTTP</TransportName> <ContentFormat>RenderedText</ContentFormat> <Locale Language= en-us /> <LogFile>ForwardedEvents</LogFile> <PublisherName>Microsoft-Windows-EventCollector</PublisherName> <AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers> <AllowedSourceDomainComputers>O:NSG:BAD:P(A;;GA;;;DC)S:</ AllowedSourceDomainComputers> </Subscription> For blog inquiries contact blog@root9b.com For general inquiries contact info@root9b.com