Emulex and SevOne Provide Unparalleled Clarity for Enterprise Network Performance Management

DEPLOYMENT GUIDE Emulex and SevOne Provide Unparalleled Clarity for Enterprise Network Performance Management How to increase visibility by delivering un-sampled 10Gb/s NetFlow to SevOne NPM with EndaceFlow 3040 appliances EndaceFlow 3040 Appliances

Abstract This document describes the why businesses should deploy Emulex EndaceFlow 3040s to provide flow visibility in high throughput Gigabit Ethernet networks or to augment flow visibility that may be provided by traditional NetFlow enabled Routers and switches. The advantages of using Endace NetFlow Generators with the SevOne s Performance Appliance Solution (PAS) and Dedicated NetFlow Collector (DNC) are also detailed and a reference deployment scenario is provided. Contents Abstract...2 Background...3 EndaceFlow 3040 appliances...3 SevOne NPM...3 Use case Large Enterprise Deployment...3 Requirements...3 Implementation...4 Configuration commands to setup EndaceFlow 3040...4 SevOne DNC configuration...7 SevOne vpas virtual machine creation...8 SevOne vpas virtual machine configuration:... 9 SevOne vpas Endace certification files upload...10 Configuring peering with the DNC...10 Monitoring the DNC health...13 Configure monitoring with EndaceFlow 3040...14 Configuring peering with the EndaceFlow 3040 NetFlow Generator...15 Verifying the SevOne DNC server is receiving the flows...16 Conclusion...18 2 Emulex Deployment Guide EndaceFlow 3040 Appliances

Background NetFlow is one of the primary inputs into a wide range of network performance monitoring, management and planning tools, such as the SevOne Network Performance Management solution. Traditionally the network elements themselves generate NetFlow records, but this is not their prime function and in order to manage the resource consumption of this secondary activity, most devices resort to sampling only a very small fraction of the packets. This can still impose a significant performance overhead and under extreme conditions, where visibility is most needed, NetFlow is often disabled to preserve the switching and routing function of the device. Two factors conspire against the continued usefulness of sampled NetFlow; the migration to 10Gb/s, and the adoption of mobile devices (which result in more, but much shorter, transient flows). The combination of these result in sampled NetFlow seeing less and less of what is actually occurring on the network, compromising the usefulness of the upstream tools the collect and analyze these records for the business critical functions they serve. EndaceFlow 3040 appliances Emulex, a world leader in Intelligent Network Recorders, has applied its proven DAG technology to the NetFlow generation domain. The EndaceFlow 3040 significantly improves the speed of core data center routers and switches, by offloading the requirement for these critical network elements to process high-speed data flows and publish millions of NetFlow records. By pushing this high-overhead task to a purpose-built 1RU Emulex appliance instead, an unprecedented level of visibility is provided to organizations by unlocking the full potential of their modern NPM deployments. Each EndaceFlow 3040 is capable of ingesting 30Gb/s of traffic, tracking 16 million concurrent flows and providing up to 600,000 records per second egress without resorting to indiscriminate packet sampling. Only the most modern and scalable NetFlow analysis solutions can cope with this volume of NetFlow data and this is why Endace has partnered with SevOne to bring the highest fidelity, most scalable NetFlow visibility solution to the market. SevOne NPM SevOne provides the most powerful integration of metric and flow reporting in the market today. When troubleshooting an alert such as a spike in utilization, SevOne provides one-click navigation to flow reports that instantly show the associated traffic for the exact period of time for that specific interface. This integration of raw flows into the troubleshooting workflow speeds problem resolution, and saves valuable time and effort when trying to understand and answer the who, what, when and where questions regarding traffic on your network. The SevOne performance management solution is delivered as a distributed network of turnkey appliances, where each system is a collector, a reporter and a distributed storage system. The robust peer-to-peer software architecture with multi-threaded collection and distributed reporting algorithms scales from a single interface to tens of thousands of flow interfaces. SevOne s reporting algorithms generate network-wide reports that utilize the computing resources of all SevOne appliances in parallel, enabling reporting capacity to be scaled in step with collection capacity. This innovative peer-to-peer appliance architecture reduces the complexity and footprint of the solution, and therefore the Total Cost of Ownership (TCO). Use Case Large Enterprise Deployment This section describes an actual large enterprise customer deployment of a joint Emulex-SevOne Solution. Due to the rapid bandwidth growth in the customer network, traditional network element sampled NetFlow was proving to be less and less accurate, and the NetOps team efficiency was degrading as their visibility decreased. By supplementing the existing low fidelity NetFlow, with 100% unsampled NetFlow from the EndaceFlow 3040, the utility of the customer s existing installed SevOne NPM was unlocked and network performance and stability increased across the board. Requirements The customer needed Emulex to provide high throughput, real-time NetFlow generators to help provide visibility and security insight across all their large global datacenters and border gateway routers. Many of these routers were processing 100Gbps and could not provide raw (1 to 1) reliable NetFlow data at these rates. To compound the issue, as bandwidth rates increased, the rate of sampling (which needs to scale proportionally to keep CPU utilization down) was decreased due to the volume of core processing the device needs to do to perform its primary function. The increase in sampling ratio became as high as 1 out of every 10,000 packets to keep the router CPU within normal operating tolerance. This large ratio resulted in inaccurate flow reporting for the customer. Micro spikes which were being caught on a regular basis disappeared due to the sampling rate making it harder to understand the true behavior of the network. Missing these micro spikes could lead to erroneous application performance analysis and troubleshooting. The customer needed fast accurate NetFlow reporting to provide visibility across each ecosystem to help determine if SLA s are being met and integration with SevOne was a must for reporting and analysis. 3 Emulex Deployment Guide EndaceFlow 3040 Appliances

Implementation Configuration commands to setup EndaceFlow 3040 Each EndaceFlow 3040 comes preconfigured from the factory in a high performance configuration designed to support 30Gb/s ingress, 16 million concurrent flows and 600k NetFlow record exports per second. This pre-configuration reduces set up time and after a few simple steps, the 3040 will begin delivering NetFlow records to the target flow collectors. Note that the EndaceFlow functionality is contained within a VM that runs on top of OSm. Step A) below discusses the OSm configuration, with the remaining occurring inside the EndaceFlow VM hosted on the physical 3040 appliance. Step A: Each EndaceFlow is preconfigured with two metering instances, and each of these has four process ID s allocated. When more than one process ID is allocated to a metering instance, this causes a flow safe Hash Load Balancing (HLB) function to be applied to the ingress traffic, which evenly distributes the packets across the process IDs which optimally distributes the computational load across threads. To deliver packets to the metering instances, create Data Pipes in OSm such that each DAG card is associated with each of the metering instances. Directionality is important to consider when sending NetFlow records to a SevOne DNC as it requires the directionality of the flows to be identified with the ingress and egress interface fields. To best achieve this each direction of the bi-directional link should be fed into separate DAG ports and from there through separate Data Pipes to the vdags associated with each metering instance inside the EndaceFlow VM. In the event that you are collecting from a SPAN port, two Data Pipes will be needed to read from the single incoming traffic stream and converse IP filters applied to only supply a single direction upstream to the metering instances. Note that the packets are snapped to 128 bytes as they leave the Data Pipe and are passed in the vdags (which is the interface between the Data Pipe and the metering instances). Since the NGA does not need the packet payload, this is removed to preserve bandwidth and improve the performance of the system overall. It is important that consistent time synchronization is applied across all components of this solution. To ensure that both OSm and the EndaceFlow VM are synchronized, NTP must be configured at a minimum. If external time synchronization devices are attached, then the DAG cards will need further configuration as detailed in the EndaceFlow User Guide that is supplied with each Emulex appliance. ntp server 0.pool.ntp.org ntp server 1.pool.ntp.org ntp server 2.pool.ntp.org ntp server 3.pool.ntp.org 4 Emulex Deployment Guide EndaceFlow 3040 Appliances

Step B: Once packets are being supplied to the metering instances, the NetFlow templates are now created inside the NGA VM. Each deployment is unique in this respect and the combination of attributes supported across NetFlow v5, v9 and IPFIX provides a wide range of flexibility in the type of information the NGA can deliver to upstream collectors. ngalancope (config) #NetFlow template sevone_v10 ngalancope (config) #NetFlow template sevone_v10 id 256 ngalancope (config) #NetFlow template sevone_v10 version 10 In this situation IPFIX is desired and is configured as above. The EndaceFlow is unique from typical NetFlow devices in that the packets are collected passively from the network link, rather than being generated by a router or switch as the packets are distributed. Accordingly the EndaceFlow has no information about what interface the packets egressed or ingressed, only what link they were captured on. The ingress and egress interfaces can be manually defined for collectors that require these fields when defining the metering input in the GUI or CLI. Note that this will require the separation of each direction of the traffic so each can be handled by a separate metering instance and have the relevant ingress and egress fields populated. As such it is required that the ingressinterface and egressinterface fields are present in the template in order for the metering instance to have a container into which to insert this information. The other mandatory fields required by the SevOne DNC are listed below; you can add others at your discretion. NetFlow template sevone_v10 definition 1 field sourceipv4address NetFlow template sevone_v10 definition 2 field sourcetransportport NetFlow template sevone_v10 definition 3 field destinationipv4address NetFlow template sevone_v10 definition 4 field destinationtransportport NetFlow template sevone_v10 definition 5 field protocolidentifier NetFlow template sevone_v10 definition 6 field octetdeltacount NetFlow template sevone_v10 definition 7 field packetdeltacount NetFlow template sevone_v10 definition 8 field ipclassofservice NetFlow template sevone_v10 definition 9 field tcpcontrolbits NetFlow template sevone_v10 definition 10 field ingressinterface NetFlow template sevone_v10 definition 11 field egressinterface NetFlow template sevone_v10 definition 12 field flowendsysuptime NetFlow template sevone_v10 definition 10 field flowstartsysuptime NetFlow template sevone_v10 definition 11 field flowactivetimeout NetFlow template sevone_v10 definition 12 field flowidletimeout Step C: In situations where the egress NetFlow record rate is higher than the SevOne DNC can collect (250k records per second), filtering can be used to distribute the records across multiple DNC s. Filters can be defined based on the standard IP tuple fields, or for maximum convenience, a flow safe load balancing scheme can be implemented. Below is an example of how HLB can be configured, although this was not needed in this deployment as there was only one SevOne DNC connected to each EndaceFlow NGA. NetFlow filter name hlb_0-25 set hlb match 0-25 NetFlow filter name hlb_25-50 set hlb match 25-50 NetFlow filter name hlb_50-75 set hlb match 50-75 NetFlow filter name hlb_75-100 set hlb match 75-100 NetFlow filter enable 5 Emulex Deployment Guide EndaceFlow 3040 Appliances

Step D: Now that the template is defined, these are now associated with each of the predefined collectors. NetFlow export collector outgoing NetFlow export collector outgoing address 172.18.13.205 NetFlow export collector outgoing filter NetFlow export collector outgoing input endace1 NetFlow export collector outgoing port 9996 NetFlow export collector outgoing protocol UDP NetFlow export collector outgoing type 4 NetFlow export collector incoming NetFlow export collector incoming address 172.18.13.205 NetFlow export collector incoming filter NetFlow export collector incoming input endace2 NetFlow export collector incoming port 9996 NetFlow export collector incoming protocol UDP NetFlow export collector incoming type 4 Note that no filter is specified here as this was not required in this deployment. When configuring the collectors, the address of the associated SevOne DNC is supplied and the default port is set (this is port 9666, but this can be changed if desired, provided the NGA and SevOne DNC are identically configured). Finally each collector is enabled allowing NetFlow records to be delivered to the SevOne DNCs as can be seen in the figure below. 6 Emulex Deployment Guide EndaceFlow 3040 Appliances

SevOne DNC configuration Now that the EndaceFlow 3040s are configured and delivering NetFlow records, the DNCs themselves require configuration. This is a simple process with a few logical steps covering the default password change and then configuring the network settings. In order to ensure correct operation of the overall solution, it is important that each DNC has the NTP system correctly set, as unsynchronized clocks will cause time resolution issues between elements of the solution. 7 Emulex Deployment Guide EndaceFlow 3040 Appliances

Save the configuration (this will restart the network interface briefly): SevOne vpas virtual machine creation The SevOne Performance Appliance Solution (PAS) provides a scalable and comprehensive data collection, monitoring, reporting and analysis solution and is the primary dashboard for this solution. The PAS is also available as virtual image (vpas) and in order to leverage the consolidation benefits offered by EndaceProbes, this will be installed on the EndaceProbe Central Management Server (CMS) which is responsible for monitoring and maintenance of the EndaceFlow 3040s deployed at this site. The SevOne vpas image file, as provided by SevOne, should be stored on a server that has SCP or SFTP access available from the EndaceProbe CMS. From here it can be downloaded onto the CMS for install: virt volume fetch url scp://user@server/path/to/sevone-vpas-5.2.4.0-kvm.img Create the vpas vm: virt vm vpas-vm arch x86_64 virt vm vpas-vm memory 8192 virt vm vpas-vm storage device drive-number 1 source file SevOne-vPAS-5.2.4.0-KVM.img virt vm vpas-vm interface 1 model e1000 virt vm vpas-vm interface 1 bridge br0 Assign two CPU s to the vpas vm: virt vm vpas-vm vcpus count 2 virt vm vpas-vm vcpus vcpu 0 pin 8 virt vm vpas-vm vcpus vcpu 1 pin 20 Save the configuration: wr mem 8 Emulex Deployment Guide EndaceFlow 3040 Appliances

SevOne vpas virtual machine configuration Now that the virtual machine has been defined, it s a simple matter of booting it and running through a few installation steps. Power on the vpas vm: virt vm vpas-vm power on connect-console text Log into the probe using password: changeme and follow the usual network configuration steps. As noted earlier, it s important to ensure all elements of the solution have their clocks synchronized to ensure consistent timing across all devices. Save the configuration (this will restart the network interface briefly): 9 Emulex Deployment Guide EndaceFlow 3040 Appliances

SevOne vpas Endace certification files upload An important component of the SevOne PAS is the ability to poll devices for system health and performance, as well as ingest NetFlow data from the NGAs via the DNCs. As well as collecting this information from the switches and routers in the network, the EndaceFlow NGAs can also be monitored. The following steps should be followed to upload the Endace Certification files to the SevOne vpas server to allow it to pull performance metrics from the Endace probe. Contact the SevOne Support team at 302-319-5400 option 2 or at support@sevone.com and request the Endace Probe certification files. Once the files are received, upload the files (SFTP or FTP) to the /usr/local/scripts directory on the vpas virtual machine. Then issue the following commands to install the files: php /usr/local/scripts/import.operatingsystems.php -Endace_os.s1o --force php /usr/local/scripts/import.snmp.objects.php -Endace_objs.s1o force Configuring peering with the DNC This section details the procedure on configuring peering between the SevOne vpas virtual machine and each of the downstream devices that we ll be collecting information from. To complete the peer to peer configuration where multiple Dedicated NetFlow Collectors exist, the following steps must be taken. From the command line on the vpas run the following command: SevOne-peer-add Then answer the questions on the screen: 10 Emulex Deployment Guide EndaceFlow 3040 Appliances

11 Emulex Deployment Guide EndaceFlow 3040 Appliances MONITOR

12 Emulex Deployment Guide EndaceFlow 3040 Appliances MONITOR

Monitoring the DNC health Starting with the DNC, after logging into the PAS go to the Devices tab > Device Manager section and click on the Properties Icon and Select Add New Device : Enter the DNC hostname and IP address and SNMP settings. The read and write strings are both sevone. 13 Emulex Deployment Guide EndaceFlow 3040 Appliances

Configure monitoring with EndaceFlow 3040 Now that the DNCs are connected, each of the EndaceFlow appliances needs to be connected in a similar fashion. Go back to Devices > Device Manager and select Add New Device as before and define the host name and IP address: After defining the connection to each EndaceFlow 3040, configure the SNMPv3 connection information that allows the PAS to query each device (the EndaceProbe also supports SNMP v1 and v2c as well). Be default the login details are username: admin password: injab0xn key: injab0xn. Click Save As New when finished: 14 Emulex Deployment Guide EndaceFlow 3040 Appliances

Configuring peering with the EndaceFlow 3040 NetFlow Generator The NetFlow Generation software runs inside a VM on the EndaceFlow Generator, and appears as a simplified version of the appliance OS running on the host. As such, SNMP can be used to query the health and status of the VM OS, much the same as for the physical host, so we follow the same setup steps as we did previously. Go back to Devices > Device Manager and select Add New Device and define the hostname and IP address of the VM running on the EndaceFlow 3040: Set the SNMPv3 information. The default setting is username: admin password: injab0xn key: injab0xn. Click Save As New when finished: 15 Emulex Deployment Guide EndaceFlow 3040 Appliances

Once finished, the Device Manager list should update with all of the devices the vpas is monitoring. Verifying the SevOne DNC Server is receiving the Flows Now that all of the EndaceFlow 3040s and DNCs are connected, it s time to check to ensure that the DNCs are delivering NetFlow metadata to the PAS. Go to Administration > Flow Configuration > Flow Interface Manage, to open the flow statistics page on each interface. 16 Emulex Deployment Guide EndaceFlow 3040 Appliances

Make note of the flows per second that each interface is seeing: Log into the NetFlow VM and go to NetFlow > Statistics to view the outgoing flow rate: Confirm that the Per Second output flow rate on the Endace NetFlow generator is consistent with the incoming flow rate on the SevOne vpas: If the two flow rates are identical then the EndaceFlow 3040 and the DNC pair are correctly configured. 17 Emulex Deployment Guide EndaceFlow 3040 Appliances

Conclusion This deployment guide documents how to deploy a joint Emulex EndaceFlow and SevOne DNC/PAS solution to bring 100% visibility to 10GbE enterprise networks through a combination of high performance, un-sampled NetFlow, and the deep insight and analytics provided by SevOne. By further parallelizing the EndaceFlow and SevOne DNC appliances, this deployment guide can be used to scale up to 100Gb/s+ Data Center implementations, providing the most comprehensive NetFlow monitoring capability on the market for Enterprise 10Gb/s networks. When compared with analysis provided by sampled NetFlow from networks elements, the depth and fidelity of the data provided by the EndaceFlow 3040 restores visibility into network behavior that has been eroded by the combination of fixed sampling rates and the transition to 10Gb/s networks. With the capacity of each EndaceFlow being 30Gb/s of ingress packet data, network operators have new opportunities to collect un-sampled NetFlow from multiple points in the network, further extending visibility down from the core, through the distribution and into the access layer providing a pervasive, high fidelity view of the overall network behavior. www.emulex.com Endace USA 2291 Wood Oak Drive, Suite 150 Herndon, VA 20171, USA Phone +1 408 220 9051 Endace Limited (UK) Davidson House, Forbury Square Reading, Berkshire, RG1 3EU United Kingdom Phone +44 118 900 1436 Fax +44 118 900 1426 Endace Australia Pty. Ltd. Level 32, 101 Miller Street North Sydney, NSW 2060 Australia Phone +1 800 196 594 Phone +61 2 8912 2157 Emulex Corporate Office 3333 Susan Street Costa Mesa, CA 92626, USA Phone +1 714 662 5600 end14-0140 1/14