ThreatSpike Dome: A New Approach To Security Monitoring 2015 ThreatSpike Labs Limited
The problem with SIEM Hacking, insider and advanced persistent threats can be difficult to detect with existing product sets. Many of their comprising actions are often in themselves not that unusual or dissimilar to regular network activity. For example, once inside the network, a threat agent might copy documents from a file share and begin uploading them to the Internet. A file upload to the Internet can and often does look the same at the protocol layer regardless of whether it was performed by a user sitting in front of their browser or some malware residing on their device. In order to determine whether the file upload is actually a cause for concern additional context is useful such as what data the file holds, where the file came from, where it is being uploaded to and the rate at which similar files are being uploaded. This is the point at which organisations in the past have turned to Security Information and Event Management (SIEM) as a way of achieving contextual analysis across user and network activity. The promise of contextual analysis is an exciting one and has lured many companies to embrace these technologies with varying levels of success. Many organisations have unfortunately seen SIEM fail to deliver on its promise due largely to the following reasons: 1. Lack of quality input data - The expression garbage in, garbage out applies here. High quality data is required to accurately and confidently detect threats. Most SIEM solutions come with adapters to extract and normalise data from other security products (IDS, firewalls, Anti- Virus, etc) as well as infrastructure products (DNS, OS events, web proxies, etc). In order to get the data required for detection you must first configure these products to log this information - this is difficult when there are literally hundreds of these managed by different teams. The pain is more evident when a new security threat emerges requiring changes to the log configuration. Keeping the configuration in sync across them is key otherwise rules will begin to generate erroneous output. Getting vendor products to log the information required for threat detection can be difficult and relies on the vendor having provided a flexible logging mechanism. In our example of a file upload, we might wish to extract the Content Disposition header of HTTP POST requests in order to determine the names of any files uploaded - this is a very specific requirement and most products will not be able to readily provide this data. It can be even harder to detect a SQL injection in a HTTP POST request because the form data is rarely logged and hence not available for interrogation by the SIEM toolset. 2. Inability to perform complex analysis - SIEM products allow rules to be applied to incoming events in order to detect patterns of interesting behaviour. For a skilled security analyst this is an exciting and powerful prospect. However, when implementing proactive detection of security threats across a broad threat landscape this suddenly becomes an ominous task and one which most organisations will be unable to undertake due to lack of resources. The result is that most SIEM solutions end up running without any significant customisations, particularly around correlation of activities thus missing out on their greatest potential. The rule sets provided by SIEM vendors are typically orientated towards basic correlation of events originated from security products which have already done the analysis. Ambitious organisations attempting to extract meaningful insight from non-security events such as web logs will find the prepackaged rules very lacking. The tools made available by the vendor for developing new rules are typically a stripped down set of functions not too dissimilar from those found in spreadsheet applications. They lack key programming constructs such as loops, meaning any sort of complex analysis is impossible. In the file upload example, due to the 2015 ThreatSpike Labs Limited 2
limited rule customisation functionality provided by SIEM it would not be possible to recursively search backwards through historic file movements in order to identify all the persons who had handled a file before it was uploaded or to compare all the email addresses found in the file against a client list to determine its sensitivity. 3. Strict event ordering - SIEM rules mandate strict in-order event processing. For example, if a rule is created to detect a file being copied from a file share and then uploaded to the Internet then the extraction event must be processed before the upload event. If the upload event happens before the extraction event then no incident will be recorded. In reality guaranteeing the event order is near on impossible because different products will take different amounts of time to feed the log data to the SIEM solution. Network latency, log file flushing, push/pull mechanisms and raw CPU and I/O constraints will lead to different event ordering. 4. Significant investigation effort - When an incident is generated by a SIEM product, there is typically significant effort required from a security analyst in order to determine the details of the incident and to understand the impact. This is in part due to missing linkage between events which occurs when multiple rules are combined through some custom state. The second reason is the lack of forensic data to backup the investigation since SIEM relies solely on metadata. In our file upload example there may be two rules, one for building a list of hashes of files copied from file shares and then the second for comparing uploaded file hashes against those in the list - when a match is found by the second rule an incident is generated. Unfortunately when the incident is generated all the analyst has is a hash of the file which appears in the list so they must firstly find the original event that put the entry in the list in order to determine where the file was copied from, who it was copied by and to identify any further context around the copy operation. Once the original event is found, the analyst may not be able to view the file due to access rights or because the file may have been deleted or modified since that time. The analyst also has very little information upon which to identify the intent of the file upload without visiting the URL itself and even then the URL may be part of the dark web, locked away behind a login screen. It s easy to see how an analyst might spend a number of hours looking at such a case which is untenable given the number of incidents analysts must deal with. 5. No opportunity to correct mistakes - No detection solution is going to be 100% correct, free of false positives or false negatives. Data will be parsed incorrectly, misclassifications will occur, events will sometimes be incorrectly correlated and subsequent incidents generated. When this happens there is no way to fix the mistake because events are never updated and they are analysed only once ever. When a rule is tweaked, changing its logic to correct an edge case the incidents that it previously generated remain. Ultimately this means that lists will need to be cleared and rules retrained or worse still left as they are, compromising the correctness of future detections. These issues often filter through to management reports, prompting questions about their correctness. A new approach to security monitoring ThreatSpike Dome is based on a new approach to security monitoring: Network Traffic + Algorithms = Threat Detection Unlike SIEM which works with processed metadata received from other products, ThreatSpike Dome works with raw network traffic. Network traffic is collected, stored and processed by in-house developed algorithms to provide full end-to-end analysis including stateful session 2015 ThreatSpike Labs Limited 3
reconstruction, traffic decryption, signature matching, behavioural analysis, correlation and automated investigations. When new algorithms are developed in response to customer requirements or emerging threats, they are run back across the historical network traffic to detect previous incidents and train behavioural rules so that they are immediately ready to detect threats. ThreatSpike Dome elegantly solves the aforementioned problems associated with SIEM: 1. High quality input data - Network traffic is captured via a variety of methods including a device agent, TAP, SPAN port or VPN server. Captured network traffic is sent to the ThreatSpike servers over an encrypted, mutually authenticated connection and stored, becoming the basis for all analysis. Since network traffic is incredibly data rich, there is never a need to rely on or reconfigure any other products. During analysis the network traffic is examined to identify useful information which is then extracted to support security detection algorithms. When a new algorithm is developed which requires additional information it can simply extract it from the stored network data. In the file upload example, ThreatSpike Dome statefully reconstructs the HTTP protocol sessions, extracts each of the HTTP requests and responses, examines the content payload, identifies the file types and extracts it along with any other useful information such as the content disposition header. Figure 1: An example of ThreatSpike Dome data collection using a TAP or SPAN port. 2. Ability to perform complex analysis - As a result of using raw network traffic, one customer network looks very similar to another customer network and hence there is significant opportunity for common algorithms to be developed. Similar to Network IDS, ThreatSpike Dome comes with these algorithms built in which are constantly updated as part of a subscription meaning customers are able to install the product and take advantage of a vast collection of security knowledge and expertise without having to develop it from scratch themselves. ThreatSpike Dome includes algorithms for parsing protocols, decrypting network traffic, parsing files, identifying security attacks and correlating activities to reveal dangerous insider threat, hacking and APT behaviour. In the file upload example, algorithms are provided which take care of identifying the HTTP protocol, decrypting any SSL/TLS wrapper, identifying the presence of a file upload and deconstructing the file content. Key information items such as 2015 ThreatSpike Labs Limited 4
credit card numbers and email addresses is then extracted and compared to previous or future transactions on the network to determine context. 3. Flexible event ordering - In the case of a large network or mobile workforce, network traffic will be forwarded from many different locations simultaneously. Depending on latency and the uplink bandwidth available at each location, network traffic is likely to arrive to the ThreatSpike Dome servers in an unpredictable order. If devices have a LAN but not an Internet connection, network traffic will be significantly delayed further complicating matters. This problem is solved through the use of constant investigations - once network traffic has been analysed, interesting events are extracted and placed onto a timeline which is then constantly reviewed. This means that regardless of the order in which events turn up, they will at some point be in the correct order during an investigation and hence trigger the same incidents as would have been the case if the traffic had not been reordered. 4. Simple investigations - ThreatSpike Dome processes network traffic using a series of analysis layers, culminating in a final investigation which automates the manual tasks which would typically be performed by a security analyst. The aim of the investigations is to correlate individual events and build a story which can then be reviewed for dangerous behaviours and presented to the security analyst in a manner in which they can instantly assess the impact and next steps. Investigations retrieve forensic evidence to provide context to the events that make up the story - this may include extracts of files and screenshots from the user s perspective. In the file upload example an investigation will begin with the upload event and progress from there, reviewing the content of the file, the history and flow of the file around the network as well as the history and content of the website where the file is being uploaded to. During the course of the investigation additional helpful resources will be attached to the incident report such as a screenshot of the website where the user uploaded the file to and a copy of the raw file contents. When the analyst receives the incident they can scan through the content and make a decision without having to scramble around trying to understand the context. 5. Ability to correct mistakes - The only permanent data is the captured network traffic. If mistakes are found and corrected in an analysis algorithm or investigation routine, ThreatSpike Dome will automatically pick them up and re-run the investigations correcting the incident and report output. In the file upload example if a gap was found whereby file uploads were not detected due to use of an unsupported protocol, additional protocol support could be added and on rerunning the analysis engine all historical file uploads would appear in the reports and security incidents would be generated accordingly. By integrating data collection with end-to-end security analysis, ThreatSpike Dome demonstrates a new approach to security monitoring which increases visibility into complex network security threats, reduces errors and provides agility when operating in an evolving threat landscape. 2015 ThreatSpike Labs Limited 5