Real-World Security Investigations with Network Forensics

Transcription

1 WHITE PAPER IT security threats aren t going away, but in many organizations, the ability to analyze and resolve threats is on the wane. Why? Traditional network analysis tools have trouble keeping up with today s high-speed (10G+) networks. To get by, IT organizations end up relying on high-level flow metrics, which lack sufficient details for characterizing attacks, or trusting traditional 1G tools that drop packets and skew metrics. Fortunately, a solution is at hand. Network forensics the recording, storage, and analysis of traffic gives IT organization and security experts the comprehensive data they need for finding proof of attacks. Read this white paper to learn how forensics helps solve real-world security attacks. WildPackets, Inc Treat Blvd, Suite 500 Walnut Creek, CA

2 Network Analysis vs. Stealthy, Costly Security Attacks...3 Network Forensics Workflow...4 The Need for Special-purpose Network Forensics Solutions...5 Real-World Security Investigations and Best Practices...6 Investigation #1: Tracing the Course of a Server Attack...6 Summary... 8 Investigation #2 Ensuring Compliance with Security Regulations and Catching Leaked Data...9 Summary Investigation #3 - Transaction Verification for an Online Gaming Company...11 Summary Investigation #4 - Transaction Verification for a Merchant Services Company...12 Summary Security Best Practice #1 Capture Traffic at Every Location...13 Security Best Practice #2 Capture Traffic 24/ Security Best Practice #3 Set Filters to Detect Anomalous Behavior...13 Summary WildPackets Network Forensics Solutions...15 The Omnipliance Difference...18 Conclusion...18 About WildPackets, Inc...18 More Resources about Network Forensics WHITE PAPER 2

3 Network Analysis vs. Stealthy, Costly Security Attacks IT security attacks are increasing in frequency, sophistication, and cost. The 2013 Cost of Cyber Crime Study by HP and the Ponemon Institute found that cybercrime cost large enterprises $11.56 million on average, up 78% from Just as troubling: the time it takes IT organization to resolve attacks rose even more over the same period, increasing 130%. As HP noted, Recovery and detection are the most costly internal activities. For the past year, recovery and detection combined accounted for 49 percent of the total internal activity cost, with cash outlays and labor representing the majority of these costs. The bottom line: a single attack now costs over $1 million on average and takes more time than ever before to resolve. 1 Organizations have continued to invest heavily in security tools and training, so why is it taking them longer to recover from security attacks? There are several reasons. New attacks are increasingly subtle and sophisticated. They rarely arrive through something as obvious as the spam deluges that were common a decade ago. Rather, they use zeroday techniques not yet catalogued in any firewall or IDS blacklists. Rather than transmitting large files of stolen data over FTP or SMTP, they might exfiltrate data at a low trickle that most network monitoring tools will overlook. Another reason for delayed recovery times has nothing to do with the features of the security attacks themselves; rather it has to do with data volumes and network infrastructure. Over the past few years, all kinds of organizations especially large enterprises have upgraded network equipment to new 10G and 40G ports, boosting network speeds by 10x or more. 2 The performance of these high-speed networks helps applications run faster than ever before, but it also outstrips the data collection and monitoring capabilities of many network analysis tools. Flow-based monitoring tools can continue to provide approximations of bandwidth usage and other metrics, but detailed packet-level analysis the type of analysis that s essential for characterizing new stealthy attacks is practically impossible for organizations with fast networks and old analysis gear. As a result, networks are doing more, but IT organizations are seeing less. And at network speeds of 10G or higher, a security attack can accomplish a lot in the blink of an eye. The solution for analyzing security attacks and potential security attacks on 10G and 40G networks is the same solution for analyzing other aspects of traffic on these faster networks: network forensics. Network forensics is the recording, storage, and analysis of network events. A network forensic solution records network traffic, stores it in a searchable repository, and provides IT engineers with powerful search tools and filters for mining stored data to discover and analyze network anomalies. Using network forensics, IT engineers can discover both the cause of an anomaly and its effects on IT services and systems. In fact, for organizations that have deployed 10G and 40G networks, network forensics provides the only practical way to analyze network traffic systematically. Traffic is flying by far too quickly on 10G and 40G networks for IT engineers to monitor and analyze in detail through real-time dashboards. Only by analyzing captured traffic can IT engineers really understand what has taken place on a high-speed network, which problems, if any, are occurring, and how they might be solved Investment in 10G and 40G networks has been growing steadily for several years, and market research firm Infonetics predicts that, between 2013 and 2017, sales of 10G and faster networks will grow ten-fold. WHITE PAPER 3

4 Network Forensics Workflow Because the traffic is flowing so quickly on 10G and 40G networks, the only way to analyze it in detail is to capture it first with a network forensics solution and replay it for inspection. To do this, IT organizations must already have identified key points on the network and deployed 24/7 monitoring solutions (specialized hardware, not generic NICs in PCs) that continuously capture traffic for analysis. Once a network forensics solution has been deployed at key locations, an IT organization can benefit from two types of traffic captures: Continuous, comprehensive captures The first type is a continuous capture of all traffic, providing IT engineers and security experts with a comprehensive record of everything that has taken place on the network. This comprehensive recording allows IT engineers to investigate any type of problem, including security attacks. Ad hoc captures The second type is a more focused, ad hoc capture that records traffic relevant to a specific issue (e.g., FTP traffic on a segment where FTP traffic has recently spiked.) An ad hoc capture can be initiated manually by an IT engineer investigating an issue (perhaps in response to alerts or alarms from other IT systems). It can also be initiated automatically when network conditions trigger a pre-defined filter. The advantage of ad hoc captures is their specificity. File sizes are smaller, and searches are faster. The diagram below depicts the general workflow for recording network traffic. Figure 1: Network Analysis Workflow for 10G and 40G networks. WHITE PAPER 4

5 The Need for Special-purpose Network Forensics Solutions To create loss-less recordings of traffic and to be able to replay and search these recording for analysis requires specialized software and hardware. Network monitoring and security attack analysis now require capabilities far beyond those of even the fastest laptops. Special-purpose appliances configured with multi-terabyte disk arrays are the only practical solution for implementing network forensics on today s networks. Once traffic has been recorded it had be replayed repeatedly for analysis. IT engineers can apply different filters and tools to inspect network activities that merit attention. WHITE PAPER 5

6 Real-World Security Investigations and Best Practices The following sections describe four different security investigations that are based on real events, and one security best practice that is used by many network forensics customers. In all but one of the investigations, the names and IP addresses have been changed to protect the privacy of the organizations involved. The screenshots come from WildPackets OmniPeek, a network analyzer that works with WildPackets Omnipliance network recorders to provide network forensics solutions for SMBs and enterprises. The WildPackets solution for network forensics is described later in this paper. Investigation #1: Tracing the Course of a Server Attack A security tool on an enterprise network raised an alert about unusual activity on a server. (In the screenshots below, identified by the address ) When the IT team investigated, they discovered that the server had been compromised by a security attack. Unfortunately, the security tool provided no further information about the attack, such as who the culprit was and which other systems, if any, had also been compromised. To answer these questions, the team turned to their network forensics system. Using a dashboard (in this case, WildPackets Compass), they were able to see that the compromised system had initiated a spike in Common Internet File System (CIFS) traffic shortly after the attack had begun. The screenshot below shows an example of such a CIFS spike. Figure 2. The Compass dashboard provided a clear view of the spike in CIFS traffic. WHITE PAPER 6

7 Because the network forensics appliance had recorded all network traffic around the time of the spike, the team was able to examine network activity in detail to explore this burst of traffic and its consequences. To learn more about the systems involved in the CIFS spike, the team opened a Peer Map, showing all IP communications during the period in question. The Peer Map confirmed that the compromised server had communicated with several other systems. Figure 3. A Peer Map illustrates all network conversations during a selected period of time. Next the team filtered traffic to show communications only from the compromised server. This made it easy to identify the three other systems that the compromised server had communicated with after the attack. Figure 4. Filtering on the Peer Map made it easy to identify the addresses of the systems with which the compromised server had been communicating. The forensics system s Nodes view provided another look at the communication among these systems. Figure 5. The Nodes view provided more information about the communication among these systems during the critical time of the attack. WHITE PAPER 7

8 Now the IT team knew which servers to focus their attention on in their efforts to contain the attack and reverse its effects. In addition to quarantining and repairing , the IT team would also focus on , , and Summary Working from a vague security alert, the team was able to use network forensics to identify specific systems to quarantine and where to focus attention on cleaning up the attack. Network forensics enabled the team to find proof of the attack and trace its effects. WHITE PAPER 8

9 Investigation #2 Ensuring Compliance with Security Regulations and Catching Leaked Data In an audit, examiners look for evidence of compliance with security regulations. Many enterprise IT teams now use network forensics to ensure that traffic complies with regulation and to demonstrate that compliance to auditors. Using tools like the Peer Map shown in the previous section, IT engineers can monitor and record traffic patterns, demonstrating to auditors which users have access to which resources, and which devices are talking to which other devices. They can also configure filters based on regular expressions (Regex expressions) to look for traffic that may include personal information. The filters they use look for any packet that looks to include any number that looks like a SSID, a phone number, credit card numbers (strings of 16 digits), etc., that are sent in clear text. Since these filters only look for the specific packets with the personal data, they expect to never capture a packet. If the filters do find matches, the network forensics solution alerts the IT team through syslog and SNMP traps, so IT engineers can review the data immediately to prevent additional loss of data. The screenshot below shows packet decodes from traffic that includes an HTTP POST command containing data that seems to include hacked Social Security IDs. Figure 6. Packet-level capture enables IT engineers and security experts to examine decoded traffic and discover exactly how a security breach is occurring. WHITE PAPER 9

10 The hex decode below shows another view of this problematic traffic. Figure 7. A hex decode of the HTTP traffic including the suspicious POST operation. Summary Network forensics provides IT teams and security experts with evidence of data breaches and details that are invaluable for tracking down the particulars of specific security attacks. WHITE PAPER 10

11 Investigation #3 - Transaction Verification for an Online Gaming Company bet365 is one of the world s leading online gambling groups with over ten million customers in 200 different countries. The Group employs over 2,000 people and is one of the UK s largest private companies. bet365 uses the WildPackets network forensics solution when it needs to verify business transactions, such as bets, that have been called into question. Because network forensics captures all aspects of network traffic, including the IP addresses of senders and receivers and all data transmitted between them, it provides a comprehensive record of orders, payments, and other financial transactions. In the case of bet365, these transactions include online bets. A customer who had lost quite a bit of money after a late night of gambling called bet365 and complained that he was not the person who had placed the losing bets. He claimed that someone else must have used his account from another location and run up the losses. Using network forensics, the IT team at bet365 was able to verify that the IP address and other characteristics of the traffic on the night in question matched his other activity with the bet365, including previous sessions in which he had gambled and never complained. By verifying that the same address had been used for all his transactions, they were able to refute his claim that the losses were someone else s responsibility. Summary Network forensics enables ecommerce and service organizations to verify transactions, including source, recipients, and data transmitted. This analysis can be used not only for troubleshooting, but also for customer service. WHITE PAPER 11

12 Investigation #4 - Transaction Verification for a Merchant Services Company Here s another example of using network forensics to verify online transactions. The merchant services division of a major bank is using a WildPackets Omnipliance network analysis and recorder appliance to capture and store traffic containing credit card authorizations. When a bank customer, such as an online retailer, contacts the bank with questions about a specific transaction, the bank s data center team can use the WildPackets network recorder to find and analyze the relevant transaction. The bank can then easily determine whether the authorization or denial was transmitted correctly. For example, a consumer ordered a product from a major online retailer, charging the purchase to her credit card. To the consumer s surprise, the charge was declined. The consumer called the retailer to complain. As part of investigating the decision to decline the charge, the bank reviewed the network traffic that contained the authorization request and the bank s subsequent decline of that request. Having verified that the transaction complied with the bank s credit guidelines and that its servers had handled the request and response correctly, the bank was able to close the service ticket with the retailer. Summary Network forensics enables financial services organizations to verify transactions, including source, recipients, and data transmitted. Because it captures all the packets that constitute a transaction, network forensics provides comprehensive evidence of what has been transacted between two or more parties. WHITE PAPER 12

13 Security Best Practice #1 Capture Traffic at Every Location As a best practice for network security, IT organizations should capture traffic at every location, not just as the network core. Consider the case of a large enterprise that suffered a security attack at a branch office. The breach spread from the branch office to headquarters. Without a detailed analysis of the traffic in the branch, the IT organization would have been unable to identify the source of the attack and apply the appropriate controls to prevent its spread. Security Best Practice #2 Capture Traffic 24/7 In addition to capturing traffic at every location, IT organizations should ensure that they capture traffic around the clock, so that even anomalies that occur outside of business hours can be investigated. Security Best Practice #3 Set Filters to Detect Anomalous Behavior In addition to maintaining a continuous, week-long capture of all network traffic, it s often helpful to define a secondary capture consisting only of network anomalies that may signal a security violation. If no anomalies occur, then no secondary capture is initiated and no alerts are raised. But if anomalies occur, IT engineers and security experts can take advantage of the evidence in a small capture file containing just the relevant data. To configure a capture like this, IT engineers simply define a file that starts recording data when any of the following conditions occur: Mail traffic (SMTP traffic) is not going to mail servers, possibly indicating the presence of a worm on the network. DHCP offers are coming from a source other than the DHCP servers, possibly indicating the presence of a rogue DHCP server. Offnet traffic is not destined for the MAC address of a router, possibly indicating the presence of a Man-in-the- Middle attack. Any user other than a member of the Finance team tries to connect to the Finance department s servers, possibly indicating a hacker and a probably Sarbanes-Oxley violation, as well. Any server in the DMZ tries to initiate an outbound connect other than to known backend servers, possibly indicating that a server has been compromised. Each organization can identify its own list of anomalies relevant for the infrastructure and services being maintained. If a secondary capture begins, IT engineers can open the capture files (which will be small) and know immediately where to begin their investigation. WHITE PAPER 13

14 The screenshot below shows a series of NOT conditions that define a filter to capture anomalies on the network, such as SMTP traffic that does not involve the organization s mail server and DNS traffic that does not include the organization s DNS server. Figure 8. Setting a filter on a WildPackets Omnipliance to automatically start capturing traffic when anomalies are detected. Summary IT teams can accelerate troubleshooting by configuring network forensics solutions to automatically capture evidence of anomalous behavior. Then, instead of poring through terabytes of live traffic, they can simply examine small data recordings that include suspicious traffic associated with a specific anomaly. To learn about other best practices for network forensics, see the WildPackets white paper, Best Practices for 10G and 40G Network Forensics. WHITE PAPER 14

15 WildPackets Network Forensics Solutions WildPackets provides network forensics solutions that enables organizations of all sizes to monitor, analyze, and troubleshoot 1G,10G, and 40G networks. WildPackets network forensics solutions feature OmniPeek network analyzers and consoles and the Omnipliance family of network analysis and recorder appliances. Each Omnipliance continuously captures, analyzes, and stores data at remote locations on the network, and gives real-time and postevent visibility into every aspect of a network, including Ethernet, 1/10/40 Gigabit, , and voice and video over IP. Omnipliances are engineered to meet the technical demands of monitoring and analyzing high-speed networks. They provide loss-less data capture at speeds up to 25 Gbps and rapid analysis through highly flexible filtering and powerful search tools. The diagram below shows how Omnipliances can be deployed on an enterprise network. 40G Figure 9. WildPackets Network Forensics Solutions The Omnipliance family includes three models of network forensics appliances: Omnipliance CX is WildPackets most affordable network analysis and recorder appliance. It is ideal for smallto medium-sized businesses and remote offices of larger enterprises. Omnipliance MX is a workhorse for data centers that constantly monitors the health of the network with its unique network traffic capture, recording and deep packet inspection technology. Omnipliance TL is a high-performance network analysis and recorder appliance that offers continuous network traffic capture, allowing for analysis of historical network traffic and quick data retrieval for troubleshooting. OmniStorage disk arrays can double the capacity of an Omnipliance TL appliance, thus supporting the capture and analysis of high-speed network traffic for longer periods of time. OmniStorage arrays are available in configurations of 32 TB, 48 TB, and 64 TB, enabling IT organizations to store up to 128 TB of traffic in a single, high-performance appliance. WHITE PAPER 15

16 With WildPackets network forensics solutions, data is always available for reconstruction and easy analysis of intermittent issues, cyberattacks, and network security or data breaches. Because it captures all packets, OmniPeek can reconstruct network traffic such as messages, which can be important in HR and security investigations. All recorded traffic is collected in a single location for rapid access and analysis. The screenshot below shows an example of OmniPeek s features, such as Select Related, that help IT engineers zero in on suspicious traffic. Figure 10. Using WildPackets OmniPeek, IT engineers and security experts can quickly drill down through capture files to find the data relevant to a specific user, device, application, or incident. WHITE PAPER 16

17 IT engineers can use Peer Maps and other tools to focus on specific conversations, such as the FTP conversation shown below. Figure 11. A Peer Map includes details conversation data about specific flows. WildPackets Network Forensic solutions offer the following capabilities: Comprehensive data collection: Hours or even days of network traffic anything that crosses the network, whether , IM, VoIP, FTP, HTML, or some other application or protocol is collected by a single system and stored in a common, searchable format. Omnipliances record tens of terabytes of data and make that data searchable through a single, easy-to-use interface. Precise data recording: Omnipliances capture packets without data loss at speeds up to 25 Gbps. Rich data analysis: WildPackets award-winning Expert Analysis, graphical reports, and application performance scoring eliminate the need for time-consuming, brute force analysis of network data. With WildPackets network forensics solutions in place, IT engineers can do the following: Network performance benchmarking for detailed reporting on network performance, bottlenecks, activities, etc. Network troubleshooting for handling any type of network problem, especially those that happen intermittently. Transactional analysis for providing the ultimate audit trail for any transactions where server logs and other server-based evidence doesn t provide a thorough picture of a transaction. Security attack analysis for enabling security officers and IT staff to characterize and mitigate an attack that slipped past network defenses. To analyze highly utilized 40G networks, IT organizations can add a network tap such as those from VSS Monitoring to capture 40G traffic and split it into streams of 20 Gbps or less for recording and analysis. Traffic can be divided by subnet, protocol, or whatever other metric makes the most sense for a particular network. WHITE PAPER 17

18 The Omnipliance Difference WildPackets Omnipliances mark an evolutionary step forward in network analysis, recording, and forensics. Compared to other network forensics solutions, they deliver: More power in a smaller footprint. The high-performance architecture of Omnipliances enables them to capture 1G, 10G, and faster line-rate data to disk with no data loss while consuming half the rack space of competitive solutions. Omnipliances deliver more comprehensive traffic capture and more analytical power while consuming less storage, less cooling, and less electrical power. Greater precision. Instead of simply collecting network statistics and flow data, Omnipliances capture complete network traffic for real-time monitoring and forensic analysis. Having access to every bit in every packet can be invaluable when investigating security attacks, troubleshooting voice or video over IP traffic, or verifying online transactions. Better price/performance. Omnipliances provide superior power and precision at a price significantly lower than other network forensics products, especially those that require significant external storage to keep up with today s high-speed networks. Conclusion The network analysis tools that organizations have invested in over the past decade or so are simply not able to keep up with today s high-speed networks. New tools and IT practices are necessary if IT organizations are going to keep new networks running as well and as securely as old ones. Network forensics enables organizations to realize the full benefits of 10G and 40G networks: high performance with the control and security IT organizations take for granted on 1G networks. By investing in network forensics solutions and following the best practices listed in this paper, IT organizations can ensure that speed does not come at the expense of visibility, control, or security. About WildPackets, Inc. WildPackets develops hardware and software solutions that drive network performance, enabling organizations of all sizes to analyze, troubleshoot, optimize, and secure their wired and wireless networks. WildPackets products are sold in over 60 countries and deployed in all industrial sectors. Customers include Boeing, Chrysler, Motorola, Nationwide, and over 80 percent of the Fortune WildPackets is a Cisco Technical Development Partner (CTDP). To learn more about WildPackets solutions, please visit or contact WildPackets Sales: sales@wildpackets.com or (925) More Resources about Network Forensics You ll find white papers and other resources about Network Forensics here: security.wildpackets.com WHITE PAPER 18