Essential Reference for Monitoring Your IT Infrastructure

Transcription

1 Essential Reference for Monitoring Your IT Infrastructure An ebook by How do you choose a network monitoring solution SNMP monitoring, flow-based monitoring, or packet analysis that is appropriate for your IT infrastructure? This ebook is essential for your crash course in network monitoring for analysis and learn to use multi-segment analysis, a post-capture technique, to diagnose performance problems with distributed application architectures.

2 Introduction...3 Crash Course in Network Monitoring for Analysis...3 Network Monitoring Simple Network Management Protocol (SNMP)...3 Flow-Based Monitoring...4 Packet-Based monitoring...4 The Difference between Packet Analysis and Protocol Analysis...5 Deep Packet Inspection (DPI) Explained...5 Background...5 Network Monitoring...6 Network Analysis...6 Network Security...7 Lawful Intercept...7 Traffic Shaping and QoS...7 Data Leak Prevention (DLP)...8 Four Factors that Affect Your Network Performance...8 Latency...8 Throughput...8 Packet Loss...9 Retransmission...9 Flow-Based Monitoring Solutions...10 What is Flow Analysis?...10 The Data Source...10 The Data Collector...10 The Difference between Flow-Based Solutions Is a Flow-Based Solution a Whole-Based Solution? Case Study 1: End-User Frustration Case Study 2: Too Much Traffic on Your Network...12 Case Study 3: Flow-Based Technologies on a Heavily Utilized Network...12 What Is a NetFlow Analyzer?...13 Special Section: Focus on Critical Infrastructure Components...15 Network Bandwidth...15 Three Tips for Determining Whether Latency is Caused by the Network or Application...15 Six Steps for Getting and Keeping Control of Your Network Bandwidth...16 Application Performance Monitoring...18 The Basics of Application Performance Monitoring...18 The Basics of Multi-Segment Analysis...19 How Does Multi-Segment Analysis Work?.20 Best Practices for Network Management in the Era of Distributed Applications...20 Virtual Networks...22 Monitoring Virtual Networks in Real Time and Analyzing Post Capture...22 Real-Time Analysis...22 Post-Capture Analysis...22 Real-Time versus Post-Capture...23 Packet Analysis in a Virtual World...23 Learning More...25 Conclusion

3 Introduction Choosing a solution for monitoring your IT infrastructure should not be scary. There are major flow-based network monitoring solutions on the market today, not to mention flow analyzers, protocol analyzers, and packet analyzers. How do you determine which solution - or combination of solutions - is right to monitor your network environment? With this Essential Reference in hand, you will be an IT Hero instead of an IT Zero. You will learn how to: Choose a monitoring solution SNMP monitoring, flow-based monitoring, protocol analysis, and/or packet analysis appropriate for your needs Measure both network latency and application latency Determine where to monitor to catch developing problems in complex network topologies, such as distributed application architectures and virtual networks Discover the benefit of using multi-segment analysis, a post-capture analysis technique across multiple segments, to diagnose performance problems with distributed application architectures Crash Course in Network Monitoring Network Monitoring 101 Network monitoring is far more complex than its name implies. Technically speaking, network monitoring is a systematic checking of key performance metrics to assure that the quality of service and the network capacity are within predetermined boundaries. Network monitoring examines an internal network for problems or irregularities with the end goal of ensuring network health. To complete the task of network monitoring, network engineers are ideally equipped with tools that provide them with an overall as well as a granular view of the network. There are three main technologies that are primarily used for network monitoring: SNMP, flow-based monitoring, and packet-based monitoring. Each of these technologies has benefits and downsides. With that in mind let us look at each of the technologies used in network monitoring and determine which one(s) might be the best option for your business. Simple Network Management Protocol (SNMP) SNMP is one of the oldest network monitoring techniques on the market, and its main purpose is to manage devices on IP networks. These devices typically include routers, switches, servers, workstations, printers, etc. SNMP data provides network engineers with a high-level view of the condition of networked devices. With SNMP you can see, for example, the core temperature of a device, what hardware is installed, overall throughput (for network connections), errors and dropped packets per interface, etc. 3

4 This device-centric view is one of the major reasons why SNMP is still frequently used. However, one of the drawbacks to SNMP is that it is based on polling, so configuration for each device is required before meaningful data can be obtained, and a specific polling interval must be specified, typically every minute, or longer. As the number of devices being monitored grows, SNMP polling can create a significant amount of network traffic, further taxing the network you are trying to monitor. In addition, detailed troubleshooting and root-cause analysis of network issues is not possible with the level of data available via SNMP, so even if you know that a device has a problem, you cannot typically determine the exact nature of the problem in order to fix it. SNMP is a bit old fashioned as a network monitoring solution, but it still provides one of the best ways to see device metrics and summary-level activity on your network just be aware of the network overhead attached with SNMP solutions and the limited ability to perform root-cause analysis. Flow-Based Monitoring Flow-based monitoring solutions are by far the most popular solutions on the market today. Flow-based solutions use existing resources like network switches and/or routers to obtain data that is already being processed by these devices. It can be very cost-effective because it eliminates the need for additional hardware and software to obtain network data for analysis. Flow-based technologies are intended to provide network engineers with an overview of network performance, including information like application performance and overall bandwidth utilization. Flow-based systems analyze seven distinct characteristics of each packet on the network and group the overall data into network conversations, or flows. All network statistics must be compiled on the basis of these seven characteristics and the resulting network flow data. With all the advantages that flow-based solutions can provide a network engineer, they lack the ability to zero in on specific problems that require deeper packet information and decodes. In addition, flow-based systems can impact the devices being used to run your network your switches and routers when networks get busy. In this case, network devices will default to their primary objective, routing IP packets, and loss of flow-based data and analysis can result. For a deeper dive into how flow-based systems work, as well as the various vendors and how their products differ, see the Flow-Based Monitoring Solutions section on page 10. Packet-Based monitoring Packet-based analysis was historically reserved for deep dive troubleshooting. However, packet-based systems have evolved into complete network monitoring, reporting, and troubleshooting solutions that can deliver the same statistical data as flow-based and SNMP systems while also providing the most detailed network analysis possible. Packet-based monitoring analyzes all of the details of every IP packet on the network, including the packet payloads, providing a complete view of network activity and allowing for true root-cause analysis of the most complex network problems. Packet-based solutions typically require additional hardware to capture network 4

5 data, but this extra cost is offset by faster resolution times and root-cause analysis. Additionally packet-based solutions allows your network analysis solution to be truly passive a significant advantage as network speeds move from 10G to 40G and beyond. From a business perspective, packet-based solutions are the only way to solve issues quickly and effectively without impacting the performance of the network itself. The Difference between Packet Analysis and Protocol Analysis Packet analysis, protocol analysis, six of one, half a dozen of another, right? You might think so. Google either term and you will find them used interchangeably by just about everyone out there, including the experts. But, packet analysis is quite different from protocol analysis, and far more complete. Protocol analysis is a subset of packet analysis. Protocol analyzers interrogate packet headers to first of all determine which protocol is being used for communication, like HTTP (always a well-understood example), and then to ensure that the rules of the protocol are being adhered to. Protocol analysis is commonly found in network security equipment like Intrusion Detection and Next-Generation Firewalls. While this is valuable analysis, it is strictly at the communication layer. But what about when the protocol is absolutely correct, yet users are still complaining about poor network performance? That is when we need to get to deeper layers of analysis, or true packet analysis. Packet headers, which contain the information about the protocol, are not the only sources of information for network analysis. Packet payloads also contain critical information regarding the workings of your network, and when you include payload analysis with protocol analysis you get packet analysis the complete solution. Packet analyzers can now address more complex network issues, like figuring out if it is the network or a specific application that is causing a problem. The answers lie in the packet payloads, and in packet, not just protocol, analysis. Deep Packet Inspection (DPI) Explained In principle, DPI is very straightforward. As the name implies, it involves the inspection of every packet traversing a specific point on the network, and analyzing the packet deeper than the Layer 3 or Layer 4 headers, depending on the application. The key phrase here is depending on the application. At first used strictly for protocol analysis and development, DPI has become the foundation for many different network management and control functions, making it a sometimes overloaded term. A common misconception is that because Company A and Company B both claim to do Deep Packet Inspection, they must be competitors. This is most often a false assumption, leading to significant confusion. Let us try to eliminate some of the confusion by outlining the key technologies where DPI is in use. Background A little more background is helpful before diving into the applications. The purpose of a network packet is 5

6 simply to move information from one host to another. The information can take any form, from overall network management to data exchanges between clients and servers. Each packet can only contain a certain amount of data, so most interactions between hosts involve the delivery of multiple packets to complete a particular transaction. Each packet is self contained, meaning that each packet has all the information needed to correctly route it from one host to another. The basic elements of the packet are the header and the payload. The header contains all of the routing information as well as metadata about the payload, while the payload is the actual data being transmitted. It gets a whole lot more complicated than that, but for the purposes of describing how DPI is used this should suffice. One other bit of background that is worth covering is how the packets are actually intercepted for inspection. At any given moment in a transmission, a packet is either being conducted down a cable or being processed by a network device, like a switch or router. To inspect the packet, you must be in this path. Because the only devices in this path are typically routers or switches (networking gear), either the router or switch itself must be capable of doing the inspection or a network connection needs to be tapped. Tapping involves disconnecting an existing network connection and adding a network device inline which will perform a particular function, and in our case, one based on DPI. Network Monitoring One of the primary uses of DPI is to perform network monitoring keeping track of everything that is happening on the network. Given that each packet is self-contained and provides detailed information, including the data itself, the depth of data reported for network monitoring can vary greatly based on DPI. The most common approach used in network monitoring today is to employ flow-based monitoring, which only looks at the packet header (the routing information) to determine overall statistics like top network users and top applications. The advantage is that most network devices (switches and routers) supply flow-based information, eliminating the need to tap into the network. The drawback is that the level of detail available for monitoring is limited. Network Analysis Network analysis carries network monitoring much further. Network analysis uses the full packet, both header and payload, to perform detailed analysis of everything happening on the network, from Layer 2 to Layer 7 events. Network analysis can provide all of the information typically found in a network monitoring solution, but adds the ability to perform detailed troubleshooting of any network problem. A corollary to network analysis is network recording, where all network packets are stored for a period of time so analysts can go back and replay exactly what happened on the network hours or even days ago. In order to obtain this increased detail, the network must be tapped with a device capable of capturing and analyzing the network packets. Though an added expense, most enterprises decide to employ network analysis solutions as they provide the only way to truly achieve root cause analysis of network issues. 6

7 Network Security Though not the only technology used for network security, DPI does provide the basis for many key network security technologies, from firewall security to dedicated Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS). Most security applications require a dedicated appliance to be inline, and unlike network monitoring and analysis solutions, which are typically passive, network security solutions often provide active controls of network behavior based on the DPI results. Network security solutions require far less storage than network monitoring and analysis solutions, because once a packet is inspected, only metadata for any detected security anomalies need to be retained. Lawful Intercept Lawful intercept is a very specific application of DPI, typically employed by governments under strict regulation. It involves the capture and analysis of network data pertaining to a particular user, typically someone who is under investigation, and performed within the guidelines of a specific court order. DPI allows the law enforcement agency to capture all of the packets for a particular user, and either analyze the data in real time or save the packets for post-capture analysis. Lawful intercept typically involves close cooperation between law enforcement agencies and service providers. Traffic Shaping and QoS With the huge amount of data traversing enterprise networks today, traffic shaping has become very common on corporate networks. Traffic shaping uses DPI to inspect packet headers to determine the type of traffic contained in that particular packet (HTTP, VoIP, FTP, etc.). Then, based on user-defined rules, it alters the delivery priority for the packet. For example, FTP transfers may be given a lower priority as compared to HTTP traffic if most of the applications used by the company are web-based, so FTP packets may be delayed (or dropped) if the current volume of HTTP traffic is high. Traffic shaping systems may look even more deeply into packet headers, differentiating between specific types of HTTP traffic. This is very useful for prioritizing corporate web-based applications over YouTube videos, for example. Network quality of service (QoS) is a specific form of traffic shaping that can be done at the network level (i.e. by switches and routers). Certain data types, like VoIP, are real time in nature and much more sensitive to network delays. VoIP packets can be (and should be) assigned a higher QoS which will give these packets priority routing over other network packets. While traffic shaping refers to the general practice of adjusting priorities and bandwidth of traffic, the term is often used in marketing materials to indicate that the processing is done independently on each device. QoS, on the other hand, usually refers to network-wide traffic shaping, in which the packet gets a specific priority tag when it enters the network often using the Differential Services Code Point (DSCP) field in the IP header which is then processed by the rest of the routers and switches along the traffic path until the packet reaches either its local destination or the edge of the network. The advantage of QoS is that DPI only needs to happen once, reducing the load on the rest of the networking equipment. 7

8 Data Leak Prevention (DLP) This is a relatively new technology that allows users to inspect network traffic for the purpose of determining if sensitive data is being transmitted in ways that violate corporate policy. It uses DPI and requires both an inline appliance and analysis of both the header and payload information. Often based on keyword rules and/or file fingerprints, DLP systems analyze each packet for rules violations, only saving metadata when violations are detected. DLP systems are far from foolproof, and require detailed knowledge of typical network traffic patterns and constant adjustment to reduce false alarms while keeping data secure. Networks have become increasingly more sophisticated, with the daily complexity of issues and outside threats growing proportionally. DPI enables a wide range of network technologies that provide essential services from overall network monitoring to detailed network analysis to security systems that prevent attacks and data leakage. It is no wonder that DPI has become such a buzzword. Hopefully when you come across a product that uses DPI now, the first question you will ask is HOW is it using DPI?! Four Factors that Affect Your Network Performance Solving issues related to poor Application Response Time (ART) is a key task that network engineers tackle all the time. Is it the application itself or a slow network that is frustrating users? Maybe it is the server that is simply stalling too much? Finding the cause of your users frustration the application or the network is important because knowing exactly where to look is the first step to solving slow response times. Here are four factors that affect network performance you might want to check when facing with network issues. Latency Think of latency as the speed limit on a highway. Traffic speed on a motorway is affected by many variables such as weather, number of other cars on the road, and highway signs. Likewise, data packets traversing a network are affected by many variables as well. The first step in mitigating latency is to break down the overall latency into latency due to the network and latency due to the application and its associated servers. With that determination made, visually graph both the application and network latency to help identify patterns and anomalies that deserve closer attention so that you can later drill down and figure out exactly what is causing the bottleneck. Throughput Throughput is the amount of traffic a network can carry at any point. Like the analogy of traffic used to explain latency above, think of throughput as analogous to the number of lanes on a highway. The more lanes, the more traffic a highway can accommodate. When thinking of networks, the higher the bit rate, the faster files transfer. Slow response times might be an issue with your network not having enough throughput. 8

9 Packet Loss Glitches, errors, or network overloading might result in the loss of data packets. Sometimes routers or switches might shed traffic intentionally to maintain overall network performance or to enforce a particular service level. In a well-tuned network, intentional packet loss is hopefully a rare occurrence, though packet loss is still something that happens regularly due to a host of other reasons and must be monitored closely to ensure overall network performance. Retransmission When packet loss does occur, those lost packets are retransmitted. This retransmission process can cause two delays; one from re-sending the data and the second delay resulting from waiting until the data is received in the correct order before forwarding it up the protocol stack. These factors are not exclusive, but they do help paint a picture of the many things that can contribute to a slow network. Hopefully, armed with this information, you can start accurately diagnosing your network before performance issues arise. 9

10 Flow-Based Monitoring Solutions What is Flow Analysis? When it comes to enterprise network monitoring, flow-based solutions are by far the most popular, with major flow-based network monitoring solutions on the market today. With that many solutions, how do they differentiate from one another, and which one will be best for your network? To determine this, let us start with the basics. How does a flow-based solution work? The Data Source The primary source of flow data is your switches and routers. As they forward packets, they keep track of each flow. Because the packets are traversing the device, it is relatively easy for the router or switch to extract key data, but it does require some extra processing. The flow data is temporarily stored in available RAM. Periodically, the device packages up the data into a stream of UDP packets following a predefined format (like NetFlow or sflow) and transmits these packets to a user-configured IP address, known as the Collector. The frequency of data pushed from the switch to the Collector is something that is configured on the switch, and is typically set to one minute, though you may find a different interval works best in your specific environment. All flow-based reporting protocols categorize packets into a flow based on the following seven characteristics: source IP address, destination IP address, source port, destination port, layer 3 protocol type, TOS byte, and input logical interface. Depending on the protocol being used to analyze the packets, and the current load on the router or switch, sampling may be employed. Meaning that the device does not collect or report information about every flow. While sampling gives a reasonable approximation of the top talkers on the network, the lack of full visibility generally lowers the accuracy of the data being reported. The Data Collector Once the UDP data stream with the flow-based information leaves the switch or router, it is purged and forgotten. It is now the responsibility of the Collector to receive, process, and store the flow-based information. Since the delivery from the switch or router to the Collector is over UDP, which does not guarantee reliable transport, some of the flow data may be lost while it is being reported. (A network analyzer like the OmniPeek Network Analyzer can help you identify if this is an issue on your network). Each packet typically contains information on only five to ten flows, so a busy network segment can generate a significant number of flow reporting packets. If your network is already nearing capacity, keep in mind that the flow data reporting will add to your traffic, and that a busy network is more likely to experience dropped packets. The Collector becomes the central repository for all data from that switch or router, and from many others, because a single Collector is designed to support multiple data sources. A Collector employs either a proprietary data structure or database to store the large volume of data that accumulates from the flow-based sources, and retains the data for long periods of times (months, at least) for reporting. A flow-based monitoring solution 10

11 is a combination of a Collector, or set of Collectors, and a central server which processes user requests, communicates with Collectors, and returns the desired results to the user. The Difference between Flow-Based Solutions Differences between network monitoring solutions based on flow data come in two forms. The first is the type of flow data. Different network device vendors support different flow-based protocols. The most common protocols are NetFlow (Cisco), J-Flow (Juniper), sflow (Brocade, HP, and others), and IPFIX (proposed IETF standard). Each protocol deals with the generation of flow records just a bit differently, with the major difference centered on whether or not sampling is used and how aggressively it is used. The other difference in flow-based network monitoring solutions is in how the flow analyzer presents (displays) the data, and any unique ways each vendor finds to process the data to provide unique results. Unique data processing and presentation is really the only way for vendors to differentiate themselves, because the source and format of the data from the routers and switches is essentially the same regardless of the underlying flow-based protocol. What solution would you find most helpful for your company and why? We always suggest that enterprises have something more comprehensive than just a flow-based solution. Flows are okay for tracking top talkers and protocols, but they lack the detailed information you will need for troubleshooting and root-cause determination. Given the inherent uncertainly in flow collection and reporting, which comes from the potential sampling and lack of reliable delivery, flows are not a dependable way to track whether a given router or switch forwarded or dropped a packet. There is also no way to get timing information or payload from flow data. If you are interested in learning more about these issues, check out the next section. Is a Flow-Based Solution a Whole-Based Solution? NetFlow and other flow-based technologies, like sflow, J-Flow, and IPFIX, have become increasingly popular given their leverage of existing resources network switches and/or routers to obtain data that is already being processed by these devices. Flow-based solutions lack the information to solve specific problems experienced by the end-user, and can lose key data when your network is most heavily utilized. Case Study 1: End-User Frustration Problem: You receive a call from an end-user who is experiencing significant application performance problems. Of course the immediate blame goes to the network. How do you quickly fix this problem using only flow-based data? Answer: Without access to the packets and payloads, the network engineer has no way to determine if there is a problem in the network or if it is in the application. If a flow record was reported by a switch or router, it is an indication that the packet made it that far, but the reverse is not true: the lack of a record does not mean the packet was not forwarded. Sampling and UDP-based delivery can cause loss of flow information before reaching the Collector, so flow-based analysis is not reliable for detecting problems. 11

12 The network engineer would have to resort to a lot of trial-and-error with the user: try pinging it, try a traceroute, or just try it again. When a network outage occurs, the results are clear, but when the network is just slow, it could be a misconfiguration, some bad hardware, or a number of other causes or the server itself could be slow. The network engineer may need help from the application or server engineer to figure out the root cause. Now you have three players that are experiencing frustration: end-user, network engineer, and application engineer. Remedy: Get a solution that is able to troubleshoot and provide you with access to both the payload and packet information. Watch Troubleshoot the End User Experience with Payloads to see Jay Botelho, Director of Product Management of WildPackets, walk through the process of determining the issues centered around a particular user and a particular application, as discussed in the above scenario, and display how simple and short this process can be when using a solution that provides visibility into your packets and payloads. Case Study 2: Too Much Traffic on Your Network Flow-based analysis generates additional network traffic, with the volume of traffic proportional to the amount of traffic on the network segment being monitored. These packets come in spurts ranging from tens of Kilobytes, up to several Megabytes, for each reporting interval, depending on how many flows are monitored by the switch or the router for that given interval. On highly utilized network segments, the forwarding queues on switches and routers can start to fill, which means that they are more likely to drop packets. At the same time, the flow data they are tracking creates additional traffic to be sent to a Collector. The burst of several Megabytes once every minute to report flow data is added to the queues of upstream routers and switches on the way to the Collector. The flow records sent from the switch or the router to the flow-based processor are based on UDP packets, an unreliable transport mechanism. There are no acknowledgments with UDP, so dropped packets result in missing and inaccurate flow-based data. Remember, each NetFlow packet reports on five to ten flows, so for each dropped packet, many flows are ignored. And, this is most likely to happen when the network is busy, compounding your ability to get an accurate picture of the current state of the network. When a network is overloaded enough to drop packets, there are two problems with flow reporting. It generates its own traffic that makes the problem worse, and it is more likely that the flow data itself will be lost before it reaches the Collector. If you are relying on flow data to resolve the problem, it is least likely to be available when you need it the most. Case Study 3: Flow-Based Technologies on a Heavily Utilized Network All flow-based solutions share resources with the prime directive of your router or switch forwarding packets. In most modern devices, packet forwarding happens on dedicated chipsets, and flow collection happens in software. On a switch, that means it is usually on the supervisor module, which is often busy doing Layer 3 tasks like routing between subnets and VLANs. On a router, it is competing for resources with routing table updates. If your router or switch is heavily utilized, it will hopefully focus first on its prime directive, compromising flow-based 12

13 reporting. This can create intermittent inaccuracies in your monitoring and reporting that are very difficult to detect, affecting your ability to collect essential information from your network when your equipment is busy. In practice, the device is likely to throttle down the flow collection and start sampling. The default configuration for NetFlow is to monitor and develop flow records for 100% of the packets no sampling. But it can be configured to 1 out of k static sampling, or the network device itself can switch to a sampling mode when network traffic gets heavy. Sampling leads to inaccuracies in reporting, and these inaccuracies can vary substantially because it all depends which flows are being ignored through the sampling. What Is a NetFlow Analyzer? Before we address this question, we must address an even more basic question What is NetFlow? NetFlow, and other flow-based technologies like sflow, JFlow, and IPFIX, are simply specifications for collecting certain types of network data for monitoring and reporting. The data sources are network devices themselves, like switches and routers, the idea being to leverage existing resources in the network to provide data that is for the most part already being processed by these devices. To that end, flow-based systems provide an economical source for network monitoring data. All flow-based systems start with flows as their basic element. A flow is a sequence of packets that has the following seven identical characteristics: source IP address, destination IP address, source port, destination port, layer 3 protocol type, TOS byte, and input logical interface. By definition, a flow is unidirectional, so a typical client-server connection will be reported as two flows, one in each direction. Flows are processed and stored by supported network devices as flow records, and it is these flow records that vary from specification to specification - e.g. a NetFlow flow record does not take quite the same form as a sflow flow record. This requires different parsing and processing techniques for each flow-based specification. It is at this step where flow records are consumed and the term NetFlow analyzer is introduced. Basic flow analysis is a multistep process, requiring several different elements to be present. Packets enter a switch or router, just as they would as part of normal network operation. If the network device is flow-enabled and the feature is active, additional processing will take place to identify individual flows in the packet stream per the seven characteristics mentioned above. Depending on the configuration of the network device and how busy the network is at any given time, this processing may be done on every packet, or just a sampling of the packets. As flows are identified, flow records are created per the specification supported by the network device, and the records are stored locally in RAM. Usually configured to be once every minute, the records associated with those flows are exported to an external NetFlow collector, where they are archived for further analysis and reporting. Once the flow record leaves the network device, it is deleted from memory to make room for other flow records. Though efficient, because the packets already must be processed by the network device, NetFlow does put an additional strain on the network device because it requires additional processing beyond that required for only switching or routing, it requires additional storage on the switch for the flow records being processed and exported, and it generates additional traffic on upstream devices. 13

14 A NetFlow analyzer includes the NetFlow collector, which accepts and stores the completed flow records; a storage system to allow for long-term storage of large volumes of flow-based data; and analysis software to mine, aggregate, and report on the collected data per user requests through a customized UI, often web-based but sometimes client-server. The NetFlow analyzer can be software-only or appliance-based, but most systems are appliance-based, and the system often includes multiple appliances. So what are the advantages? NetFlow data comes for free from NetFlow-enabled network devices, eliminating the need for additional network probes to collect the flow-based data. But remember, it is not entirely free because it requires processing and storage resources on the network device thereby competing with the prime directive of the device forward packets. Given the seven characteristics of a flow, NetFlow analyzers can provide a relatively detailed set of network performance data, and given enough storage this data can be archived for quite a long time providing a long-term record of network behavior. NetFlow analyzers may not always be 100% accurate because the source of the flow data can be from sampling and not an analysis of each and every packet. NetFlow analyzers also create additional network traffic moving flow records from the network device to the NetFlow collector, possibly impacting performance on an already busy network. And NetFlow analyzers can report on nothing more than the information they can interpolate from the seven flow characteristics, making them excellent network monitors, but poor network analysis solutions because they often lack the data to perform root-cause analysis once a network anomaly is detected. Network analysis systems that derive data from independent Interrogation of each and every packet, like the Omni Distributed Analysis Platform, provide all the data necessary not only for detailed network reporting, but also for advanced, root-cause analysis. No sampling, no need to move data across the network for storage and analysis. All analysis is done at the source, by tapping into a network device and processing all the data locally. Each system has its place, but when the time comes for root-cause analysis, and it always does, a packet-based analysis solution like the Omni Distributed Analysis Platform is what you need. 14

15 Special Section: Focus on Critical Infrastructure Components Managing today s IT infrastructure need not be scary. With the right network monitoring for analysis solution in place, you will be able to see potential problems as they are developing. In this section, we dive deeper into three critical infrastructure areas: managing and controlling network bandwidth, monitoring performance of local and distributed applications, and monitoring and analyzing virtual networks. Network Bandwidth Three Tips for Determining Whether Latency is Caused by the Network or Application Networks typically run various applications, from single-tier, locally-hosted applications like , to multi-tier web-based applications, or even time-sensitive, multi-hop applications like VoIP. While application traffic typically resides within the data center, Software-as-a-Service (SaaS) and cloud computing are driving application traffic outside of the traditional enterprise network, making network latency even more of an issue. Pinpointing and correcting slowdowns is therefore a necessity, and can be a real challenge. So what is responsible for latency that creates poor application performance the network or the application itself? First, we must distinguish between the two basic types of latency network and application latency. Network latency is how long it takes the network to deliver a packet. Since that is tough to measure without multiple synchronized packet captures, it can be estimated as the amount of time between a client request packet and the server TCP acknowledgement packet. Application latency is how long it takes for the server to respond to the client s request. It can be measured as the amount of time between a client request packet and the server payload response packet. Most network-monitoring products provide some sort of latency-monitoring features, but usually it is either one or the other, not both. Here are three tips to determine whether latency is being caused by the network or the applications (or both) in your environment. 1. Clearly Determine Network versus Application Latency. Every application issue is blamed on the network until proven otherwise guilty until proven innocent. Clearly measuring network latency vs. application latency is the proof the network engineer needs for acquittal. Packetlevel monitoring is ideal for accumulating evidence. By visually inspecting a packet-level conversation between a client and a poor performing application, a network engineer can see whether the network (or a network device) is the source of the delay or if the application is the bottleneck. This is done by comparing the responsiveness of the TCP ACK to a client request versus the application response, which includes actionable payload data. Quite often the network acknowledges the client request quickly (within milliseconds) while the application may take tenths of seconds or even multiple seconds to respond with payload data. When you see this, you know it is the application causing the problem. 15

16 2. Periodically Test and Monitor Key Applications and Network Connections. Periodic, active monitoring can provide insight into network performance on key interfaces, and can alert you when conditions begin to degrade. While this technique only addresses network latency (not application latency), it can still provide important data when determining whether the issue is network or application related. For example, let us say the organization s CRM application is via a web-hosted service. Running periodic traffic in the background (even just pings) to the CRM application host can provide an ongoing baseline of the performance of the network between users and the host. If the baseline increases, alarms are used to notify that the network latency is becoming an issue. User complaints about CRM performance without a marked change in the network latency baseline almost always indicate the application host or the application itself is at fault. 3. Graph Latency over Time. Graphing latency over time helps to identify patterns and anomalies that deserve closer attention. Latency monitoring can help correlate areas of latency with other relevant statistics, as well as the actual network traffic occurring at that time. This type of high-resolution forensics analysis can help to detect latency problems at the highest level and drill down quickly for closer inspection. Ideally, network latency and application latency measurements can be graphed together over time, making clear whether the problem lies with the network or the application. Comparing the measurements of the two types of latency over time and seeing the differences can provide information that might have otherwise been overlooked. Latency monitors can include a feature that sets thresholds on latency, so alarms will go off when normal conditions are exceeded. You can be made aware of excessive latency before application performance becomes a widespread issue, allowing you to make necessary adjustments to the network proactively. This type of proactive latency monitoring allows you to detect and correct problems in the network and applications before users even notice a slowdown. Six Steps for Getting and Keeping Control of Your Network Bandwidth Garbled and choppy VoIP calls? Check. Slow Internet connection? Check. Loss of detail in video image? Check. All these signs point to a case of bandwidth overload creating unacceptable network latency. With protocols like n expanding the need for core bandwidth with much faster edge data rates, it is more important than ever to have a thorough understanding of your network to prevent strain and overuse. Detailed below are six steps to prevent latency issues associated with bandwidth overload. 1. Create a Baseline. An important first step in improving bandwidth management is to know the number of users and their bandwidth needs, as well as application bandwidth needs. Know who is using what, when, where, and why in regards to network segments. This will determine the overall demand on the network and will help you allocate bandwidth appropriately. Networks have a rhythm, so be sure to assess these needs over a period of time, focusing on 16

17 both daily and weekly rhythms. Once this baseline is established, you will have something to measure success against. You can start this process by looking at your Internet connections, WAN links, WLAN environments, and data centers. A network analyzer is a great tool for creating baselines for both wired and wireless networks as it provides critical statistics in an easy-to-read PDF or web report. These analyzers allow you to identify problems in the network and validate performance and bandwidth utilization. 2. Prioritize Critical Business Applications and Tie Baseline Protocols and Usage to Those Applications. Every organization will have different priorities. In fact, each network segment may have different protocol priorities because of the specific applications that traverse those segments. Certainly, the top application (based on business importance) on the sales segment will be different from the top application on the marketing segment. Those application protocols need to be handled in terms of importance for the segment they are individually on. But, when those protocols get to the same wire at the core or elsewhere, it is important that they still respect other segments needs. It is important to understand which applications are specific to your organization and their use of protocols. And remember, there is usually more than one. Any protocol that is not performing well can affect the overall application performance (the weakest link per se). This is another area where a network analyzer can help break down and show individual flows and their performance. Network analyzers also give you a view into your network to see the weakest link as well as options to sort application flows with various criteria choices. 3. Pay Attention to Fluctuations in Network Resources. New application introductions can tip the balance of bandwidth usage and have serious impacts on network performance. Visibility into network resource usage is essential to help network managers accurately meet user needs, particularly when bandwidth-intensive applications are in use. 4. Use Dynamically-Adjusting WLANs WLANs have the ability to dynamically adjust to changing conditions and to configure themselves to make the greatest use of available bandwidth. These capabilities work best when the problems they address are kept within limits. To do this, you must understand the limits of the RF environment in the areas where wireless is to be deployed. Assess the overall area over space and time to get a quantifiable baseline of your environment. Also, with the increased bandwidth of n, you will likely be considering applications like VoIP over wireless (VoFi), which will require additional measurement techniques like wireless roaming to ensure proper operation of your network and ensure wireless quality. (Watch the Network Analysis for VoIP and Video: Just One More Call and You are Over the Cliff video to see how one minute your VoIP applications can be performing optimally, and then one more call drastically reduces performance for all.) Be sure to plan wireless management upfront. 17

18 5. Prune Your Protocols/Traffic. Most networks have unnecessary traffic. Often, WLAN traffic has not been pruned and this can cause a clog in bandwidth. Check protocols that help manage the network like routing protocols, SNMP, etc. and determine if they have a purpose. 6. Constantly manage. Networks and users are dynamic. They will not always do the same thing twice and it is critical that you consistently and constantly review your network activity. It is important to see new trends approaching and make changes to your network to account for the behavioral changes in your organization s user community. Application Performance Monitoring The Basics of Application Performance Monitoring Ensuring that your network and applications are performing optimally is essential for your business. Application performance relates to the time it takes for an application to respond to a specific user request, measured from the user s perspective, through either the network and/or the web services infrastructure. ART is often broken down into two key components the network response time (NRT), which addresses just the network latency, or the time it takes data to get from one end of the network to the other, and the transaction response time (TRT), which addresses the processing time required by all application processes. In order to make sure your applications are performing properly, you need to first determine what optimal performance is, and have tools in place that can perform 24/7 monitoring on your applications. Application performance depends heavily on network performance, so if your network is not performing correctly, then your users will experience problems with their applications. If an application is not performing properly, users typically blame the network as the culprit for the issue. If a problem arises and users become frustrated with the performance of an application, which is really the only perspective that an end-user has, the first step is figuring out what is causing the problem: the network or the application. 1. Prove If It Is the Network or Application. To prove unequivocally if it is the application (TRT) or the network causing the problem, you need an analysis solution based on deep-packet inspection. With packet-based analysis, you can inspect and even visualize the conversation between a client and a poorly performing application, packet by packet, to determine what is causing the delay network or application. A user request followed by a quick network acknowledgement (ACK) but a delayed data response is indicative of an application issue, while delayed or even missing ACKs indicate a network issue. Often times when the responsiveness of application data is poor, the application data payloads themselves contain detailed clues as to why for example, database error codes embedded in the data packet payload. 18

19 Quick Tip: Network issues are shown through slow acknowledgements, TCP slow segment recovery, slow and frequent retransmissions, and low throughput. Application problems manifest themselves in slow HTTP response times (for web-based applications), slow database response times, and inefficient client errors. 2. Catch the Problem before It Becomes an Issue. Before you can tell if network and transaction latency are snowballing, you must have an understanding of what normal means for your network. Benchmarking your ART provides you with details of how your applications regularly perform. When you are establishing an application benchmark, pay close attention to both network and transaction latencies and assess whether or not they appear across a wide range of users (especially in different locations) and/or a wide range of applications. 3. Find the Right Technology to Do This. Again, a key factor in solving application performance issues is having the ability to analyze down to the packet level. However, having only a packet-level view is not enough. Be sure to look at solutions that provide both high-level monitoring capabilities that can keep you aware of how your applications are functioning from a business perspective, as well as perform deep packet analysis when a problem does occur. Keeping up the performance of your applications is essential to your business, and understanding how to monitor network and transaction performance, as well as their interaction, will help you keep your users happy and your network healthy. The Basics of Multi-Segment Analysis In today s digital age, where companies are increasingly relying on applications for business-critical tasks, application performance has become a key issue. Network disruptions are now business disruptions, and the worst disruptions can sometimes have financial or even legal consequences. Network engineers tasked with keeping this essential system of applications, networks, clients, and servers up and running need to have the right tools and processes available to help them ensure the availability of these services. Most applications today are no longer centralized in a single data center. Applications and the data they access are now widely distributed, whether it is distributed data architecture within an enterprise or increased usage of SaaS or cloud-based computing. The nature of these multi-tiered, distributed applications requires that they traverse both LAN and WAN links, often with multiple hops, making it increasingly complicated for the user to diagnose performance issues. This increased complexity in the data path puts a strain on traditional application performance analysis where a single data path could be easily dissected to determine if poor ART was due to the network or the application itself. Contemporary, distributed application architectures require a new technique multi-segment analysis in order to pinpoint the cause of latency or other application performance issues. 19

20 How Does Multi-Segment Analysis Work? Traditional performance analysis of centralized applications lended itself to real-time network analysis as all relevant data could be collected from a single network link. Application metrics like latency (both network and transaction), number of turns, overall network bandwidth, payload sizes, and even the packet payloads themselves (for detailed application-level troubleshooting) were readily available on that single link. With distributed application architectures the same data is required, but multiple network links, or hops, must be analyzed to get the full picture and to isolate not only the issue but what network link it is occurring on. Multi-segment analysis is a post-capture method that automates and simplifies the process of gathering network data from multiple network segments and/or multi-tiered applications. Multi-segment analysis correlates this data across the various network segments, finding common elements so the individual application transactions can be reassembled from a network perspective, visualized, and analyzed to indicate potential problem areas. It provides a clear view of the application flow, including network and transaction latency and application turn times. With this information in hand, you can easily pinpoint where the anomalies are occurring with applications at each point on the network between the client and server. Many network monitoring dashboard and reporting tools have a multi-segment analysis feature, but troubleshooting application performance problems requires more than pretty graphs. Be sure that any solution you choose also has the ability to drill down into each and every packet that comprises the application transaction. Important clues, including application error messages, are often buried within application packet payloads, providing you with the unequivocal proof you need to approach the application designer when the issue is NOT the network. Best Practices for Network Management in the Era of Distributed Applications A network error just occurred in your environment and you (the network engineer) have about two hours to fix it before your entire company is breathing down your neck. About 10 years ago, when centralized computing was all the rage, the ability to fix this problem was very simple: you physically went down the hall to the wiring closet, and connected a network analyzer to the appropriate port to troubleshoot the issue. However, this is no longer the scene. Whether you are an organization of 50 or 50,000 employees, most likely your network environment is highly distributed. Imagine sending your network admin to China to fix a problem in that data center. It is simply not realistic as part of a daily workflow. Today s distributed application architecture takes many forms, from locally-hosted applications to web-based applications to multi-tier, third-party hosted applications. While application traffic has historically resided in the data center, like the scenario above, SaaS and cloud computing are driving application traffic outside the traditional enterprise network making the ability to determine network and performance issues far more challenging. No two networks are the same, with topologies depending on many factors, but most networks can be characterized using similar metrics. These metrics can be used to help you plan for a holistic solution that will best monitor and analyze your entire environment. Below are some key tips to getting started, or for reevaluating your environment to monitor and analyze distributed applications. 20