High-Performance Network Data Capture: Easier Said than Done

Transcription

1 Introduction Network data capture is an essential tool for all IT disciplines. It has proven to be the best way to find and fix the most difficult performance issues and network outages, because it is the only means of going back in time to investigate performance degradations or other service problems. Captured data is a reliable technique for tracing problems from the first symptoms to the application component or network segment also called the fault domain that gave rise to degradations or problems. Once the fault domain is identified, captured data is critical for domain experts to perform root cause analysis. As the saying goes, It s easier said than done. High-performance monitoring is key for applications that generate large amounts of data for multiple clients. For example, voice over IP (VoIP) monitoring, video distribution, video conferencing and virtual desktop infrastructure deployments all generate volumes of traffic on behalf of large communities of users. Data must be recorded between application tiers, if it is to be used for problem diagnostics. For example, in Figure 1, data must be captured from network locations A, B and C. These taps, when fed to a data capture appliance, record the network messages between the tiers of the application. Figure 1 - Example of Multi-tiered Application Recorded data must be absolutely complete to provide accurate evidence of network activity. That is, all network packets must be recorded and each packet must be recorded in its entirety. Likewise, recorded data must be timestamped with high resolution in order to coordinate cross-tier traces. Missed or truncated packets can easily mislead an investigation and result in a misdiagnosis. A missed packet could be in the microsecond that the error or intrusion occurred. Simply stated, data recording mechanisms must be able to record every packet on multiple 10Gb Ethernet (10GbE) networks without loss and with minimal delay. 10GbE networking is currently the technology of choice for new data center build-outs, and 40GbE is close behind. 1 High-Performance Network Data Capture:

2 Recorded data is retrieved for real-time or post-event analysis. It must be quickly and easily searched by a number of parameters, including capture time, IP addresses, flows and protocols. This requires that the recording mechanism analyze and index packets from multiple 10GbE networks in real time. Retrieval should be fast enough to investigate events in real time, while maintaining full capture capabilities. Write-to-Disk Speed With a write-to-disk speed for a single, modern hard disk drive on the order of 1 Gb per second (1 Gbps), careful system engineering is required to record data from multiple 10GbE links. Figure 2 is a block diagram of the major components of a data capture appliance designed to meet these stringent requirements. Figure 2 - Data Capture Alliance The engineering issues associated with high-speed data recording can be broken down into three areas: Input: receiving packets from data networks Processing: receiving data from the input block, analysis and indexing of data Output: recording packet and indices to permanent storage Emulex, an Avago Technologies Company, Network Visibility Products division has more than a decade of experience in the design, manufacture and support of data capture platforms. It has distinguished itself by offering the highest recording rates and capacities for existing technologies. EndaceProbe TM Intelligent Network Recorders (INRs) are available in a range of network types and speeds, processing power, and disk size, types and performance. EndaceProbe INRs are part of a family of products that include EndaceVision TM Network Visibility Software, which accesses and presents captured data with sophisticated displays and management tools, and manages and coordinates the data from multiple EndaceProbe INRs. 2 High-Performance Network Data Capture:

3 By controlling all aspects of design, EndaceProbe INRs offer: 100% data capture from input through to disk storage Sustained write-to-disk speeds of 20 Gbps 1 Storage for up to 2.5 days of typical network traffic Indexing and metadata storage for fast retrieval of recorded data, using multiple filters Data retrieval without affecting write performance A platform that supports other network related activity Figure 3 - EndaceProbe 9004 Intelligent Network Recorder Figure 4 - EndaceDAG Card Input Handling Emulex designs and manufactures its own Ethernet and other network interface cards, specifically tailored for capture and on-board analysis. A 4x 1/10GbE EndaceDAG TM Data Capture Card is shown in Figure 4. 2 EndaceDAG Cards are an integral part of EndaceProbe INRs. Network traffic is received on an EndaceDAG Card port directly from a network tap, from a router/switch SPAN port or from a network packet broker. Each packet is timestamped using a precision time source either 1 pulse per second (1PPS) or precision time protocol (PTP) signaling. Timestamping is critical for accurate synchronization of messages, with less than 100 nanoseconds between the shortest 10GbE packets. Lower resolution timestamps on 10GbE networks can result in thousands of records with the SAME timestamp. Building a timeline of events is critical in a forensic investigation, be it real world or cyber; if packets cannot be ordered due to poor timestamps, a timeline cannot be established. When combined with multi-tier analysis, low resolution timestamps can render captured data unusable. 1 Experience with data center networks has shown that they use only a small percentage of their maximum bandwidth. A 20 Gbps recording rate is well equipped to handle four or more 10GbE links. 2 An EndaceAccess TM Network Visibility Head-End device is also available. It handles 40 and 100GbE traffic, breaking it down into multiple 10GbE streams. 3 High-Performance Network Data Capture:

4 EndaceDAG Cards generate a hash of header information that will make it easy to later build an indexing database, which will in turn facilitate quick data lookup. Direct memory access (DMA) is used to transfer captured packets and other data directly to the processor s main memory. Significant buffering is available both on the card and in main memory to ensure no packet loss. EndaceDAG Cards support PCI Express 3.0 that enable twice the data transfer rates of PCIe 2.0 cards, making them capable of transferring 40 Gbps plus additional metadata to the host memory. The Emulex next generation EndaceDAG Cards provide support for the capture of a variety of encapsulated telecom protocols, such as GTP and GRE, including load balancing and classification filtering across these protocols. EndaceDAG Cards operate using field programmable gate arrays (FPGAs) that offer high performance and flexibility. FPGAs allow EndaceDAG Cards to perform more processing than standard network interface cards that merely transfer received data to host memory. Additional features and updates can be easily installed by EndaceProbe owners. Processing EndaceProbes use state-of-the-art Intel processors and motherboards specifically chosen for high performance. The EndaceProbe 9004 INR, for example, has 20 cores (40 hyper-threaded cores). Six of those cores are used by Emulex software to receive, process and output packet data. The remaining cores are used for visualization and network-related processing. Captured packets are directly deposited in memory by the EndaceDAG Cards. Multiple streams, one from each port of one or more EndaceDAG Cards, are merged to a single stream according to their timestamps. Figure 5 shows some of the operations occurring within the EndaceProbe INR. Figure 5 - EndaceProbe Processing 4 High-Performance Network Data Capture:

5 Significant processing of the merged stream yields numerous performance benefits. One copy of the stream is sent to the Endace Application Dock, which hosts third party modules (discussed later). Another is handled by the packet processing module to perform protocol analysis through packet decodes and deep packet inspection (DPI). Flows are identified and a traffic index is built. This information is saved in a database that is used by the EndaceVision for displays of captured data and derived statistics. The packet stream is written to disk storage, as discussed in the next section. Extreme traffic, processing and disk access conditions can result in some packet loss. Module interfaces are fully instrumented to determine the location of packet loss and the conditions. This information provides guidance for how the user can adjust EndaceDAG Card and EndaceProbe INR filtering and/or the third-party applications to ensure 100 percent packet capture. Output Writing large, continuous volumes of data to disk is the most challenging part of a data capture solution, requiring simultaneous writes to multiple disks. There are a number of factors that affect the write-to-disk speed: Technology: rotating hard disk or solid state disk (SSD) Interface technology: SATA or SAS Rotational speed: 7.2k or 10k RPM Number of drives used Choice of RAID scenario Emulex has chosen RAID 50 for its good balance between storage performance, storage capacity and data integrity that s not necessarily found in other RAID levels. RAID 50 operates by striping (RAID 0) data across multiple RAID 5 sets. For example, the EndaceProbe 9004 INR uses three RAID 5 sets that span a total of 24 disks. Each RAID 5 set has eight disks, with one disk s worth of capacity, dedicated to parity. The 24 near-line SAS drives total of 96TB of storage. 3 The combined write-to-disk speed for the the EndaceProbe 9004 INR s 21 available disks is 20 Gbps. If a disk fails and needs to be rebuilt, the reconstruction time can range from 12 to 48 hours, based on how busy the system is. An important objective in the implementation of the disk recording sub-system is that reading should not compromise write performance. Nor should read performance be dramatically affected by data recording. This is where the careful indexing of data on the EndaceDAG Cards and on the processor, coordinated with placement of the index on the disk s file system, come into play. Together, they make it straightforward to find data to be retrieved and then to access the data from a nearby location. EndaceProbe INRs optimize disk usage in several ways, including pre-allocation of blocks and optional edge platter protection. Preallocation of file system blocks minimizes the amount of head movement, resulting in a smooth, stepped write from inner to outer disk tracks. The pre-allocation also avoids file system fragmentation. Edge platter protection avoids use of the outer edge of the disk, whose write-to-disk rate is lower than inner tracks. That part of the disk is partitioned away and does not participate in saving of captured data. Data security is important, considering that recorded data will likely have sensitive information, e.g. file transfers or s. To address this, drives used in many EndaceProbes incorporate self-encryption. Data is encrypted as it is stored to disk, and decrypted on retrieval. The encryption/decryption key is held in hardware on the EndaceProbe INR rather than on the hard drive, making it useless to physically remove disks to get at their data. 3 This corresponds to about 2.5 days of storage for average usage. 5 High-Performance Network Data Capture:

6 Emulex is the first to offer SSDs in a recording appliance. The EndaceProbe 4104 INR offers 7.6TB of storage from eight 960GB SSD drives. SSDs change the rules with minimal data access latency, the ability to conduct writes and reads simultaneously, and reliability that is so high that it exceeds the expected lifetime of the appliance. The EndaceProbe 4104 is an ultra-reliable platform for harsh or hard to access environments. Although the system is initially configured for RAID 5 redundancy, it can also be used with RAID 0, without redundancy. With a high transfer rate, the only limit on SSD write-to-disk speed is that imposed by the controller and PC bandwidth. The EndaceProbe 4104 offers a slightly higher rate than other capture platforms at 22 Mbps. Other features As mentioned earlier, the EndaceProbe INRs offer extra processing power for use by other network-related activity. EndaceVision is the EndaceProbe web-based diagnostic data visualization application. Analysts access EndaceVision from any web browser and utilize its simple, but powerful, workflow to quickly isolate and inspect the exact packet-level diagnostic data they require to complete their investigation. EndaceVision runs on the EndaceProbe INR itself, where the captured data and index are stored. This makes it an extremely efficient tool for efficient viewing and analysis of captured data. EndaceProbe INRs provide a simple, fast and open REST Application Programming Interface (API), by which third-party analysis tools can find and extract complete traffic flows. Several common network tools have already integrated with EndaceVision, including Splunk and Cisco Sourcefire. When examining packets captured from 10GbE links, copying the data associated with even a short time period to a tool resident on a separate system, can be prohibitive in terms of download time and the network traffic generated. It is sometimes easier to move the application to the data rather than the data to the application. This is where the Endace Application Dock comes in. Every EndaceProbe INR offers a powerful, fully integrated virtual machine hypervisor that creates a secure and highly optimized environment for hosting custom and third party traffic monitoring, and analysis applications. Analysis software can be quickly and easily added or moved onto the EndaceProbe INR without the need for dedicated servers, which saves time and money, while reducing the operational complexities associated with hardware deployment. The fidelity of data from EndaceDAG Cards offers a unique dock environment, in which capture and accurate timestamping is guaranteed. For example, an intrusion detection system (IDS) can be embedded within EndaceProbe INRs, thereby receiving all the benefits of locally stored 100 percent data capture, nanosecond accurate timestamps, and detailed packet analysis and indexing. 6 High-Performance Network Data Capture:

7 Conclusion Capturing, recording and accessing data from multiple 10GbE data links is a difficult task. Careful design and system engineering is required to perfect input, processing and output systems. Since data capture is critical for fault domain isolation and root cause analysis, special care must be taken to process, buffer and save the data. Key metrics should be examined when comparing network data capture products: Long term data capture rate contrasting 24 hour performance rather than burst performance Actual data capacity not just raw disk size Full duplex operation do reads impact writes at the maximum expected capture rate? Data fidelity ensuring that 100% of desired packets are captured High resolution timestamping ensuring that a capture timeline can be uniquely built Data security ensuring that no important information can be gleaned from a stolen disk Scalability able to handle the fastest and largest networks Emulex is the only vendor of network capture appliances that controls their product from end to end. Every aspect of input offload, multi-core processing and write-to-disk is engineered to maintain a high throughput rate. The flexibility offered by its in-house EndaceDAG Cards is especially important, handling reliable capture and packet pre-processing. All other components of EndaceProbe INRs are selected and matched to achieve write-to-disk performance in excess of 20 Gbps. The Emulex commitment to excellence has been continued with the release of the first SSD-based appliance, offering even more reliability and speed. EndaceProbes are engineered to be the best performing and most cost-effective data capture appliances in the industry. Avago, Avago Technologies, Emulex, the Emulex logo, Endace, EndaceProbe, EndaceVision, EndaceDAG and EndaceAccess are trademarks of Avago Technologies in the United States and other countries. All other brand and product names are the trademarks of their respective owners. Copyright 2015 Avago Technologies. All rights reserved. END /15