IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER Statistical Analysis of Network Traffic for Adaptive Faults Detection


 Cecil Walton
 1 years ago
 Views:
Transcription
1 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER Statistical Analysis of Network Traffic for Adaptive Faults Detection Hassan Hajji, Member, IEEE Abstract This paper addresses the problem of normal operation baselining for automatic detection of network anomalies. A model of network traffic is presented in which studied variables are viewed as sampled from a finite mixture model. Based on the stochastic approximation of the maximum likelihood function, we propose baselining network normal operation, using the asymptotic distribution of the difference between successive estimates of model parameters. The baseline random variable is shown to be stationary, with mean zero under normal operation. Anomalous events are shown to induce an abrupt jump in the mean. Detection is formulated as an online change point problem, where the task is to process the baseline random variable realizations, sequentially, and raise alarms as soon as anomalies occur. An analytical expression of false alarm rate allows us to choose the design threshold, automatically. Extensive experimental results on a real network showed that our monitoring agent is able to detect unusual changes in the characteristics of network traffic, adapt to diurnal traffic patterns, while maintaining a low alarm rate. Despite large fluctuations in network traffic, this work proves that tailoring traffic modeling to specific goals can be efficiently achieved. Index Terms Normal operation baselining, online anomaly detection, online change detection, remote monitoring (RMON) management information base (MIB), traffic model. I. INTRODUCTION NETWORKS and distributed processing systems have become an important substrate of modern information technology. The rapid growth of these systems throughout the workplace has given rise to a discontinuity in expertise of human operators to manage them. There is a need for automating the management functions to reduce network operations and management cost. Detection of network problems is a crucial step in automating network management. It has a direct impact on the accuracy of fault, performance and security management functions. Well designed anomaly detection algorithms enhance the network control capability, by providing timely indication of network incipient problems. Early warnings from monitoring agents can trigger preventive actions, and serious and expensive outages can be avoided. A large amount of work has gone into developing mechanisms and protocols for collecting traffic statistics. Indeed, currently most of the work in the simple network management architecture focuses on defining detailed system and network traffic objects. Comparatively, little work is done to support analysis of collected statistics. Most of the interpretation is Manuscript received October 31, 2003; revised April 29, The author is with the IBM Business Consulting, Marunochi Chiyodaku, Tokyo, Japan ( Digital Object Identifier /TNN left to the commonsense of network operators. Unless control mechanisms are driven by objective measures using welltested network traffic models, the benefits and the results of network traffic analysis will remain biased by the common sense of human operators. On the other hand, existing work on fault and performance management assumed that the alarm generating mechanism is accurate, and network problems are given a priori [14], [15], [31]. Current practice in network management rely on userdefined thresholds for detection. Alarms are generated when some variable of interest crosses a predefined threshold. Generally, the value of the threshold is no more than an estimation of the normal range within which the measured feature is believed to operate. Not only there is little objective insights on how to choose these thresholds, but also there is a risk of missing subtle changes in the network state [13]. In addition, the complexity and size of current network systems makes them vulnerable to novel, previously unseen problems. A natural approach to anomaly detection in network systems can proceed by generating residuals, obtained from the discrepancy between the observed and the predicted behavior. Change in the properties of theses signals is a symptom of an abnormal behavior of the system. The main difficulty in network anomaly detection is the lack of a generally accepted definition of what constitutes normal behavior [19]. The dynamics of normal operations need to be identified from routine operation data. Earlier work reported in [21] characterizes the normal behavior by different templates, obtained by taking the standard deviations of observations (typically Ethernet load and packets count), at different operating times. An observation is declared abnormal if it exceeds the upper bound of the envelope. Given the nonstationary nature of network traffic, the standard deviation estimates are likely to be distorted, making subtle changes in the network state go undetected. To mitigate the effect of the nonstationary nature of network traffic, [13] considered the model formed by segmenting time series obtained form management information base (MIB) objects. Observations are declared abnormal if they do not fit an autoregressive model of the traffic inside segments. In [27], observations are declared abnormal after a statistical test with the mean of 24h period sample. In these approaches, alarms are raised too frequently, and ad hoc correlations schemes are introduced to reduce the number of generated alarms. In this paper, we address the problem of change in characteristics of network traffic, and its relation with anomalies in local area networks. Here anomaly is used to mean changes from a normal baseline. It should be understood as encompassing faults, performance, and security problems, as far as they induce changes in the characteristic of network traffic. The emphasis /$ IEEE
2 1054 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 is on fast detection, an important requirement for reducing potential impact of problems on network services users. Knowledge of anomalies to be detected is not required. We parameterize network traffic variables using finite Gaussian mixtures. Based on this parametric model, we propose a baseline of network normal operation as the asymptotic distribution of the difference between successive estimates of model parameters. This difference is shown to be approximately multivariate Normal, with mean zero under normal operations, and sudden jumps in this mean are characteristics of abnormal conditions. The detection problem is formulated as a change point problem. A realtime online change detection algorithm is designed to processes, sequentially, the baseline random variable realizations, and raise an alarm as soon as the anomaly occurs. We motivate this formulation through a real scenario. The proposed approach requires neither the set of anomalies to be detected, nor the thresholds to be supplied by the user. Experimental results on a real network showed the effectiveness of the method. A low alarm rate and a high detection accuracy has been demonstrated. Our earlier work [11], [12] reported some of the experimental results obtained. We provided some motivation of the model used but limited explanation of the experimental results in relation with objectives of fast detection. In this paper, we build on our earlier work, and motivate our approach of using a mixture model for network traffic modeling through a realworld example of broadcast packets. We explain in more details the experimental results obtained in relation with our objectives. We rigourously establish the formulas for residual generation, and a mathematical simplification to make calculation of the threshold for statistic detection easier is introduced. In addition, we report on some new results related to the behavior of alarms, and the CPU overhead consumed by the monitoring. This paper is arranged as follows. Section II introduces our proposed parametric model of the network traffic, and shows how network normal operation baseline is built. Section III shows the characteristics of the baseline model, under abnormal condition, and introduces the formulation of the network anomaly detection. In Section IV, we present results of our experiments in a real network. We conclude in Section V. II. NORMAL OPERATIONS BASELINING The goal of this section is to characterize network normal behavior. We first present a parametric model of traffic variables, and then show how this model can be used to build a baseline of normal operations. The paper assumes that we have collected a sample large enough to account for all possible normal behaviors of the network. There is no training set due to anomalies, and it is assumed that the sample used for training contains only normal operation data. A. Parametric Model of Traffic Variables Our approach to network model parameterization is to view each variable as switching between different regimes, where each regime is a Gaussian distribution. This is a form of what Fig. 1. Traffic fluctuations of broadcast packets. Time is relative to the start of data collection and not time of the day. referred to in the literature as finite mixture model [24]. The observations are generated by one of the Gaussian distribution, as shown in the following equations: The errors are assumed to be Gaussian, with mean 0 and variance. The constant is the mean of the distribution. The integer denotes the number of regimes. Each regime has a mixing probability, denoted by. The parametric finite mixture model has an attractive interpretation. It is useful in the analysis of data that are believed to come from a finite number of distinct subpopulations, indicated by a latent variable. Given the known fact that network traffic changes as a function of the time of the day, the latent variable in this case has the natural interpretation of time of the day. For instance, Fig. 1 shows how the number of broadcast packets, that arrived within a 10s sampling interval, changes as a function of time. It can be seen clearly that broadcast packet decrease at night hours, to pick up again in working days. It is important to note, however, that time and traffic dependency does not hold always. It is possible that traffic decreases to a level similar to late night hours, even during the busiest day of the week. If the time of the day and traffic dependency is explicitly modeled, alarms are raised mainly because of structural breaks in this dependency model. With finite mixture model, however, each observation is assigned to the closest regime, and not necessarily to the regime a given time of the day may imply. Given the obvious observation that if the random variable is lognormal, then follows a Normal distribution, the random variable whose realization are the logarithm of the original data can be modeled by the finite mixture model of (2). This model provides, then, a good parametric characterization for many important traffic variables, such call holding time in telephony [3], [4], Telnet originator responder bytes, data transmitted in a given FTP connection, FTP session bytes [18], and the distribution of message length in public access mobile radio (PAMR) [1], since these variables are shown to follow a lognormal, or a mixture of lognormal distributions. Note that in general, finite Gaussian mixture models are general enough to approximate any continuous function with a finite number of (1) (2)
3 HAJJI: STATISTICAL ANALYSIS OF NETWORK TRAFFIC FOR ADAPTIVE FAULTS DETECTION 1055 discontinuities [29], providing a general first approximation to other network traffic variables. More importantly, we note that the ultimate goal of modeling in this work is anomaly detection. This has a subtle but important difference with current work on traffic analysis. Most of the work reported recently on selfsimilar traffic [16], [17], and its impact on performance analysis is done with the goal of using estimated parameters of hypothesized models in traffic engineering applications. For example, buffer sizing applications are interested in estimating the probability of the queue size exceeding a certain value. This quantity does depend on the true parameters of the used model (Hurst parameter in the case of the formula derived in [8], for example). Under these conditions, an appropriate traffic model, and an accurate estimation procedure are needed (a requirement hardly met in current models of selfsimilar traffic models, however). In our case, the only requirement on the model is the capability to discriminate between the normal/faulty states. The key idea we stress throughout this work is to rely minimally on the model assumption, and tackle the anomaly detection directly. As we shall see in the remaining part of this paper, it is effectively possible to detect network problems, without even using estimated parameters of the model. It should be noted, however, that it may well happen that this model is not useful for other traffic engineering application. B. Normal Operation Baselining Our approach to operation baselining starts by recognizing that any parametric model of network traffic is, at best, an approximation to the reality. Approximation errors and model misspecification become very pronounced in any inference that use the estimated parameters, as if they are the true ones. Given that our ultimate goal is anomaly detection, formulated as a change detection in baseline model parameters, we claim that this task can be achieved without using these true parameters. The goal is to avoid solving the more general problem of accurate parameter estimation, as an intermediate step to change detection. Our approach to realize this idea is pictorially shown in Fig. 2. Observation are passed through the learning algorithm to produce the parameter estimate. As a new data point is sequentially added, the learning algorithm outputs a refined new estimate. The idea is to characterize network normal operations using the difference. There are two major advantages of the baseline built this way. First, the difference does not depend on the true value of the parameter. This is very important since, in practice, we do not know this true value, and the only available information is the value, estimated from the data. Approximating the true parameter with, and studying the difference is possible, but our experiments showed that this approach is inefficient, as shown later. Second, the learning algorithm can be designed to adaptively track local changes in model parameters. It is unrealistic to assume that the model parameters will remain exactly the same over all the operating times of the network. Intuitively, under normal conditions, the difference between the successive values and is expected to fluctuate Fig. 2. Normal operation baselining based on repeated estimation of model parameters. around zero. This difference should not drift constantly in a fixed direction. On the other hand, if this difference drifts systematically over long duration, then the new observations are generated by a different model, induced by a pattern not present in the training data. The learning algorithm will alter the parameter to its new value. The idea, then, is to characterize normal behavior based on the random variable. The mean value of this difference is a good indicator of the health of the network. To realize this principle, two issues need to be addressed. First, the issue of how to design an adaptive learning algorithm. Since the detection is required to be online, the learning algorithm of Fig. 2 should be as fast as possible. Second, we have to work out the distribution of the difference, under normal conditions. The remainder of this section addresses these two issues. C. Learning Algorithm and Baselining Normal Behavior A wellknown algorithm for parameter identification in finite mixture models is the expectation maximization (EM) algorithm [7], [24]. Given a sample of size, and an initial parameter, the EM algorithm estimate the unknown parameter by iteratively maximizing the expected log likelihood, given by The EM is a batchoriented algorithm. It requires the whole data to be available in memory, before a new refined estimates is produced. If we have to run this algorithm online (i.e., each time a new observation is made available to inspection), too much time and memory will be consumed. Worse yet, time and memory consumed keeps growing as monitoring goes on. We do not follow this approach. Instead, a stochastic approximation of the problem of maximizing the likelihood function, is used to turn the EM to an online algorithm. Let the vectors be a sequence of observations, whose joint probability distributions depends on the unknown parameter. The goal is to derive an online algorithm for (3) (4) (5)
4 1056 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 estimating the parameter.wedefine the recursive likelihood function as follows: (6) Where denotes the expectation with respect to the true parameter, y is the latent, unobservable variable. This is basically the same recursive likelihood as in [28], except that the expectation is taken, conditionally on the whole set of observations. It can be shown, given appropriate regularity conditions, that (7) (8) Where are defined as given in the differential in equation (8). denotes Fisher information matrix for the complete data, where the separation variable is given. Similar results are obtained in [30] by minimizing the Kullback Liebler divergence, instead of maximizing the recursive likelihood. The proof of the previous result is omitted, because even taking the expectation conditionally on the whole set of observation, the proof given in [28] still holds. Working out the scores, and the matrix, leads to the following recursive formulas for updating the model parameters: Fig. 3. Comparison of the drifts (m 0 m ) and the (m 0 ^m ) for a duration of 5 h, under the same network conditions. (9) (10) Now let us verify our design goal. First, note that parameter updating is fast enough, to be implemented online. Sequentially acquired data points are merged with existing processed data, and do not require the recomputation of all collected data. Time and memory requirements are kept minimal. To allow the learning algorithm to track slight change in model parameters, we introduce an exponential forgetting factor 0, that reduces the effect of old observations, much in the same way as proposed in [30]. Evaluating the sum is then replaced by. This way the learning algorithm adapts to small changes in model parameters. Also, note that our implementation assigns observation to the regime, with highest probability, and not to all regimes, but the weight of the regime with highest probability is kept in the computation. As we are not interested in inferring the latent variable, the weights are not updated. In the sequel, we shall be interested only in changes in the dimensional mean. As stated earlier, approximating with, and studying the difference is possible, but our experiments showed that this approach is inefficient. Fig. 3 compares both differences for a duration of 5 h, under the same network conditions. Here we show results takes from the number of outbound broadcast packets from a network file server, because this variable in our network is very smooth, and likely to give good results for. As it can be concluded from this figure, however, is not symmetric around zero, while the difference is both symmetric and its mean is very close to zero under normal conditions. Fig. 4. Behavior the random variable e under the same conditions as Fig. 3. For the distribution of the baseline random variable, we showed empirically [10] that the variate random variable given by (11) (12) is approximately Normal, with mean zero under network normal conditions. Note that given in (11) is simply the difference, scaled such that its variancecovariance matrix becomes Identity. Fig. 4 shows the behavior of under the same network conditions as Fig. 3. It can be seen that the baseline random variable is stable, and its mean is very close to 0. It should be said that in this experiment, the very smooth nature of is also a result of the data used in this experiment. In general, we have always been able to obtain a mean very close to zero, but a less smoother behavior of the baseline variable. To summarize results of this section, we showed how the learning algorithm transforms the raw data, to stationary multivariate variable. The baseline random variable have the desirable property of being Normal with mean zero and variance Identity matrix, under normal network operations. The mean of serves as the baseline of normal operation. The next section
5 HAJJI: STATISTICAL ANALYSIS OF NETWORK TRAFFIC FOR ADAPTIVE FAULTS DETECTION 1057 Fig. 5. Behavior of the baseline variable under abnormal network conditions. (a) Abnormal conditions induce a sudden change in the mean. (b) Slight change in the mean preceded the sudden jump. shows the behavior of these baseline random variable under abnormal conditions, and how we formulate and solve the detection problem. III. ONLINE ANOMALY DETECTION Given that we know the behavior of the baseline under normal conditions, an important point to investigate is how the variable behaves under abnormal conditions. Fig. 5 shows the behavior of the baseline random variable under a real abnormal condition. The problem is caused by a streaming network interface card sending excessively badly formatted packets. As shown in Fig. 5(a), this abnormal condition causes a sudden jump in the mean of the baseline variable. Fig. 5(b) shows the behavior of the baseline random variable just before the sudden jump in the mean. Interestingly, we notice that the sudden jump is preceded by a slight change in the mean of baseline. If the detection approach is designed to be sensitive to slight changes in the operating characteristics of the network, we could have predicted the problem of Fig. 5 before it became serious. The problem could have been avoided, or at least addressed immediately after its occurrence. Generally, one may not assume that all problems presents signs to allow their prediction. In this case, we require our detection method to raise alarm as soon as change in the mean occurs. A. Online Anomaly Detection Consider the realizations obtained by observing sequentially the baseline random variable from time point to. Under normal operation of the network, the sample of follows a variate Normal distribution with mean 0 and Identity covariance matrix (Section II). At some unknown time point, a change happens in the model, and the new generated realizations shift to a new distribution. The goal is to find a decision function and a stopping rule that detect this change and raise an alarm as soon as possible, under a controlled false alarm rate. This formulation is known in sequential analysis literature as the disruption problem. The main difference with classical hypothesis testing is that the sample size is a function of the observations made so far (i.e., not fixed a priori), and the distribution of the baseline random variable is known, under normal operation. The goal is to achieve fast change detection, by using the minimum sample size possible to decide whether an alarm is to be raised or not. It is wellknown that for known probability distribution after change, Page Lorden cumulative sum (CUSUM) [2], [20], [23] test is optimal, in the sense that it minimizes the delay to detection, among all tests with a given false alarm rate. The idea is to inspect sequentially, for each of the possible change points, the logarithm of the likelihood ratio, and to stop the first time this ratio is bigger than a predefined threshold. The resulting decision function, and the stopping rule are (13) (14) In the present case of network anomaly detection, we do not have a priori knowledge about the distribution probability after change, and the change point. The common extension of Page Lorden CUSUM test consists of estimating the postchange distribution mean, and the change point from the data. This approach is known as the generalized likelihood ratio (GLR) test [2]. That is, for the unknown parameter of the distribution of after change, and the change point are estimated from data, using the maximum likelihood estimator. The resulting decision function is given by (15) (16) In our case, where prechange and postchange distributions are Normal, the maximization problem of (15) can be worked out explicitly. It has a simple form, given by (17) (18)
6 1058 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 The equation assumes that after change, the distribution of the baseline realizations is still Normal, but with a different mean. For the abnormal case, it is not possible to obtain an unbiased fit of the postchange distribution. Fortunately, such accurate estimation is not crucial. What is needed is that, when the anomaly occurs, the closest Normal distribution, obtained by maximum likelihood estimation, has a mean significantly different from zero. B. Tuning the Threshold So far we have introduced the decision function and the stopping rule used for online detection of network anomalous behavior. The remainder of our problem setup concerns the choice of the design threshold. It can be shown that the expectation of the stopping rule under no change, denoted by, is given in [25] as (19) (20) Where denotes the Normal distribution function. For computational purposes, there is some theoretical justification to use the following small approximation [26]: (21) (22) Not surprisingly, (19) turns out to be the mean time between false alarms. It follows that, given a desired false alarm rate, we can recover the design threshold, by solving (19). Unfortunately, (18) cannot be written recursively. Consequently, the number of realizations to be inspected can grow large. To circumvent this difficulty, we use a moving horizon of fixed length, where the starting point of the horizon moves one step forward as new observation are made available to inspection. IV. EVALUATION AND RESULTS The network monitoring algorithms described earlier has been implemented in a real network. This section discusses how the data is collected, and the results that validate the agent capabilities. A. Data Collection The implementation of the monitoring software consists of two modules: statistics collection module, and the monitoring module introduced in this paper. Statistic collection module interfaces the network for protocols operation statistics. The monitoring module monitors management objects, for online anomaly detection. Statistics collection is implemented as a remote monitoring (RMON) agent, running as userlevel process. This solution is particularly appealing in the sense that dependency on the operating system kernel is minimally reduced to the interface to access the data link layer. This way, we have full control of all aspects of network statistics, as opposed to depending on whether TABLE I DEFINITION OF VARIABLES USED IN EXPERIMENTAL RESULTS operating system kernel keeps track of traffic statistics. Our earlier implementation experience of an SNMP agent [10], revealed that some kernels do not have entries for all management objects, as defined in MIBII. The network monitoring module operates on top of RMON. It accesses raw measurements through RMON MIB. All monitoring computation is done locally. In contrast to the network management station (NMS) based polling of network statistics, the bandwidth and computation time consumed during transfer and processing of raw measurement is kept minimal. Distributed management organizational models [9], [22], addressed these issues, but failed to address the critical issue of how to use collected statistics. In this sense, our approach complements distributed management by providing the details of monitoring tasks to be carried out by distributed agents. B. Experimental Setup To illustrate each of the capabilities of our proposed monitoring approach, Table I lists the variables studied. Some of these variables are not part of the standard RMON MIB definitions, but can be easily obtained using appropriate filters. All these variables are modeled using a finite Gaussian mixture model. Slightly better results are obtained by modeling the TCP connection requests as a mixture of lognormal distributions, but all of the experiments reported here use Gaussian mixtures. For each of these variables, Table III shows its experimental configuration. Normal network model is obtained by training the model using a sample of normal traffic, and a predefined number of components as shown in Table III. The number of components is determined in an ad hoc manner, based on observed fluctuation of traffic in one week training data. We are now studying how to choose the number of components automatically from the data. As of the current implementation, the forgetting factor was chosen after testing a number of choices, and the value of 0.8 provided satisfactory results in our case. Arguably, this is a subjective measure, and a more robust method is needed. A potential extension of our work might be to adopt methods developed in [6] to our model to set the forgetting factor automatically. The variables used in this study can detect a number of anomalies as listed in Table II. This is in no way an exhaustive list of all possible anomalies, but it should be enough to prove the concepts introduced in this paper. There are a large number of anomalies like ICMP echo requests, fragmentation anomalies that affect other variables that can be easily obtained from the large set of MIB variable RMON agent collects, and our method should be readily extensible to bear on these problems.
7 HAJJI: STATISTICAL ANALYSIS OF NETWORK TRAFFIC FOR ADAPTIVE FAULTS DETECTION 1059 TABLE II EXAMPLES OF ANOMALIES THAT CAN BE DETECTED BY STUDIED VARIABLES TABLE III EXPERIMENTAL CONFIGURATION OF EACH OF THE VARIABLES STUDIED The rest of this section evaluates the following capabilities of our approach. 1) Detection Accuracy: the detection accuracy is shown through four anomalous events, induced by injecting appropriately assembled packets: total number of broadcast packet on the segment, total number of TCP connection requests on a web server, ARP packet operations, and the total number of unknown protocols. Packet assembly is implemented by assembling, and injecting packets in the network. The packet injection program is written using the data link provider interface (DLPI), on Solaris operating system. 2) Adaptability to Normal Traffic Fluctuations: the goal here is to show that traffic fluctuations that are part of the normal operation of the network, are not interpreted as anomalies. That is, the monitoring agent adapts to diurnal changes in network traffic, and no alarm is raised. 3) Subtle Changes Detection: as stated in the introduction, thresholds applied directly to traffic variables may miss subtle changes in network traffic. In this experiment, the monitoring agent capability to detect subtle changes in the network state is investigated, through two real cases. 4) Alarm Rate: gives the average alarm rate of monitored variables, evaluated for an overall duration of one week, then one month. This allows us to both assess the validity of the monitoring agent over an extended amount of time, and give a general sense of how frequent alarms are generated. 5) Monitoring Overhead: evaluates the overhead associated with continuously monitoring variables. The goal here is to provide an idea about the scalability of our detection algorithm. In the sequel, details of these experiments are presented. The term test statistic will be used to denote the statistic in (18) used for detection. C. Detection Accuracy Fig. 6(a) shows how the test statistic reacts to excessive broadcasts, created by injecting additional two broadcast packets, every second. As shown in the figure, the test statistic shows a sharp increase, crossing the threshold after a delay of 16 min, approximately. Fig. 6(b) shows how the test statistic reacts to a sustained rate of TCP connection requests, created by injecting 10 additional packets every second. It can be shown that the problem is detected, with a delay of 17 min, approximately. Our RMON implementation allows us to study perprotocol statistics. Here we focus on address resolution protocol (ARP), as an illustrative example, given both the lack of counters for ARP packet operations, and the range of problems that manifest themselves as changes in the statistic characteristics of this protocol traffic. Fig. 6(c) shows results of monitoring ARP operation for 24 h, and then perturbing network operations by injecting an additional two ARP request packets per second. It can be clearly seen that the anomaly is detected. The delay to detection is 16 min, approximately. Fig. 6(d) shows how the agent reacts to excessive unknown protocols. Most of this traffic is caused by packets with protocol type less than 1500, which is typical to other Ethernet formats, other than EthernetII (that is IEEE 802.3/802.2, and Novell Raw 802.3). In this case, it took 50 s for this anomaly to be detected. As shown in this section, the detection speed can oscillate between very short delay to detection (50 s), to relatively long delay to detection (17 min). It should be stressed that the theoretical formulation of our algorithm attempts to minimize the delay among all tests with a given false alarm rate. However, the observed delay to detection is a function of the degree of abnormality. If the abnormal behavior induces a sample of observations with a mean very close to an existing regime, the test statistic requires a sample large enough to cross the predefined threshold, resulting in a relatively long delay to detection. While we have yet to derive the mathematical proof that our test is the fastest, it is reasonable to expect that, on average, another test statistic, with no optimality considerations and with the same false alarm rate examining the same sample, would take even longer. On the other hand, if the anomaly induces a mean significantly apart from existing regimes, detection takes a short time. Again here, on average another nonoptimal test statistic examining the same sample would take even longer. The point here, is that given a false alarm rate, our mathematical formulation does try to minimize the delay to detection among all test with a given false alarm rate, and the observed delays (17 min, or 50 s) are to be interpreted probabilistically as realizations of the delay to detection random variable. In summary, we note that in all cases tested the detection is accurate. Recall that we are not modeling particular anomalies. The agent contrasts the baseline normal behavior with observed traffic, making it possible to detect novel network problems. It follows that, in principle, our approach can be easily deployed across different networks. It is also important to notice that in all of the previous cases, analyzing the captured packets, after an alarm is raised, immediately reveals the problem. This is to be contrasted with methods as in [31], that take alarms as given, and yet have to match faults signatures with prestored patterns. This approach is both knowledgeextensive, and do not have the capability to learn new, unseen faults.
8 1060 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 Fig. 6. Behavior of the test statistic corresponding to (a) excessive broadcasts, (b) sustained rate of TCP connection requests, (c) excessive ARP packets, and (d) excessive unknown packets. Fig. 7. Adaptability to diurnal traffic patterns. (a) Normal increase in ARP traffic volume. (b) Behavior of the corresponding test statistic. (c) Normal decrease in ARP traffic volume. (d) Behavior of the corresponding test statistic. D. Adaptability to Normal Traffic Fluctuations Network traffic exhibits clear diurnal patterns. Night hours and less busy days of the week show a decrease in network traffic. Traffic picks up again during working days. The purpose of this section is to show that the monitoring agent learns these patterns, and does not take them for anomalies. Fig. 7(a) shows an increase in ARP volume, that is part of normal operation of the network. ARP packets count increases, as transition is made from night hours to day hours. Fig. 7(b) shows that the test statistic remains within the normal range, for this pattern. Similarly, Fig. 7(c) shows a decrease in ARP traffic volume, as network becomes less busy, in late evening hours. Fig. 7(d) plots the reaction of the test statistic, around the same time this transition took place. Here also, the test statistic remains within its normal range. What actually happens, is that ARP traffic model is a mixture, with three complements each modeling a given level of ARP traffic. Depending on the time of the day (latent variable), observations are assigned to the corresponding component, making it possible to adapt to these traffic fluctuation. One can conclude, then, that given a training data large enough to contain all possible regimes of operation, the monitoring can adapt to these patterns, and will not be taken for anomalies. E. Subtle Changes Detection As stated in the introduction, thresholds applied directly to traffic variables risk missing subtle changes. Here, we test the capability of our approach to detect subtle changes in network traffic. Detection is shown through two real scenarios, obtained by studying the number of connection requests on a web server. Fig. 8(a) shows the row statistics of tcpinsyn. As pointed by the arrow in the figure, the inbound TCP connection requests is sustained to almost a fixed level, for a duration of 6 h, approximately. Closer inspection of the captured packets revealed that this pattern was created by two Web robots, downloading the
9 HAJJI: STATISTICAL ANALYSIS OF NETWORK TRAFFIC FOR ADAPTIVE FAULTS DETECTION 1061 Fig. 8. Reaction of the detection algorithm to subtle changes in network traffic. (a) Subtle change in incoming TCP connection requests. (b) Reaction of the test statistic. (c) More clear change in incoming TCP connections requests. (d) Reaction of the test statistic. whole documents hosted by our web server. Fig. 8(b) shows that the test statistic reacts with a sharp increase, eventually crossing the threshold. Another result is shown in Fig. 8(c), in which the change in TCP behavior is obvious. Fig. 8(c) shows the reaction of the test statistic. The experiment was running unattended, but this behavior is likely to be explained by web robots, as the traffic pattern shown here is similar to the previous problem. Given that the training data did not contain such pattern, and apparently the web robot did not respect robots.txt file instructions supposed to exclude all robots, the anomaly was detected. It is important to note that these subtle changes would have been missed with static thresholds, applied to raw traffic. F. Alarm Rate Ideally, we would like to estimate the false alarm rate, given that we know for sure that the network is operating normally. Unfortunately, it is difficult to gain perfect knowledge about all the subtle changes in the network behavior. Instead, Tables IV, and V show the average alarm rate per hour, evaluated, after the monitoring agent is set to run for one week, then one month, respectively. The goal is to give a sense of how frequent the alarm are generated. It should be noted that most of the alarms generated by the variables tcpinsyn in Table V are caused by one particular anomalous activity. The problem shown earlier in Fig. 8(c) generated 29 alarms out of 38 total alarms raised, during one TABLE IV AVERAGEALARM RATE PER HOUR FOR A DURATION OF ONE WEEK (168 H) TABLE V AVERAGE ALARM RATE PER HOUR FOR A DURATION OF ONE MONTH (720 H) month duration of the experiment. In the remaining 29 days, only 9 alarms are generated. The same also applies to broadcast packets. This variable test statistic generated 12 alarms, 11 of them are shown in the Fig. 9, and caused by a failure of the file server in the neighboring subnet. Only one alarm is generated for the remaining 29 days. The result reported here proves that, first, our method can run without alarms for a very long time, and once anomalies are injected, the test statistic reacts with crossing the threshold. Since
10 1062 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 Fig. 9. Distribution of alarms generated by the variable counting broadcast packets. Fig. 10. Percentage of CPU consumed by monitoring 10, 20, and 30 variables. we are not modeling the anomalies themselves, it is safe to affirm that our method has a good detection capability across all possible anomalies that induce change in statistical characteristics of monitored variables. Second, our algorithm running for an extended amount of time without raising alarms proves that our algorithm can adapt to normal changes in traffic. These results are to be contrasted with research reported in [13], [27] where their algorithm produced a large number of alarms, and had to introduce ad hoc combination techniques to limit the number of alarms. For example in [27], variables produced as much as 2.23 alarms per hour. In our approach, for an extended amount of testing (720 h), the highest rate was as small as 0.12 alarms per hour. A comparison with methods that use data to train a model for a set of anomalies is probably in order. Methods that start with a model of known anomalies face the problems of how to characterize monitored variables given the anomaly. Apart from the fact that these methods cannot detect new, previously unseen problems, anomalies induce disparate behavior in the monitored variables. These approaches have to make a choice: take the characteristics induced by the anomaly literally, probably maximizing the fit with a predefined model, but at the risk of higher false alarm rate. Or build a model that only loosely captures the statistical essence of the anomaly and, hence, affect negatively the detection capability of the algorithm. There is an inherent tradeoff between detection capability and false alarm rate. Most of the current work does not spell out this tradeoff clearly in the formulation of the problem. The key question is to achieve good detection capability, within a reasonable false alarm rate. Compared to these methods, our approach mathematic formulation captures clearly the inherent tradeoff between detection capability and false alarm rate. As shown in (20), by fixing a false alarm rate, we derive the threshold to be associated with the test static. Compared to methods that may have modeled anomalies by overfitting observed data, our method has the advantage of minimizing the impact of model assumptions. As shown in Fig. 3, if estimated parameters have been used as true parameters of the model, approximation and model misspecification errors will build up to cause a high false alarm rate. Instead, our test statistic avoids using the true parameters, and is based on the difference between successive estimates of the model parameters. The result is a very stable test statistic under normal conditions where drifts due to estimation inaccuracies are minimized. rate. G. Monitoring Overhead As Section IVA explained, statistics collection is implemented as an RMON agent, running as userlevel process. Packets are intercepted at the kernel level in raw format and sent directly to our RMON agent for statistics logging. Network traffic is sampled every 10 s. Fig. 10 shows the amount of CPU time it took our online algorithm to calculate the test statistic for a number of variables: 10, then 20, then 30 variables. In this experiment, CPU time usage is sampled every 10 s. The agent runs on SunOS Release 5.8, with SPARC CPU (333 MHz). Our method uses (10) to calculate residuals. The computation involves only a linear combination of the new observation with a function of the old value of the parameter. This generates very minimal CPU load. Most of the CPU time is consumed by the detection statistic (18), as involves a computation over the entire moving window. Overall, our method is scalable consuming a little over 2% of CPU to monitor 30 variables. Note that, the CPU results reported in this section has been based on artificially created variables to simplify the implementation. In addition, we report only on CPU time used for detection. We do not include the overhead for capturing and updating different RMON variables statistics. This instrumentation overhead is occurred by any other method interested in studying network traffic, and we are not claiming that our implementation is optimal. V. CONCLUSION In this paper, we developed an online technique for realtime detection of network anomalies. We introduced a parametric model of network traffic, in which studied variables are viewed as sampled from a finite mixture model. A new method for baselining network operation, based on successive parameter identification, is introduced. The baseline random variable is shown to be approximately Normal, with mean zero under
11 HAJJI: STATISTICAL ANALYSIS OF NETWORK TRAFFIC FOR ADAPTIVE FAULTS DETECTION 1063 normal operations, and sudden jumps in this mean are characteristics of abnormal conditions. A realtime online change detection algorithm is designed to processes, sequentially, the baseline random variable realizations and raise an alarm as soon as the anomaly occurs. The proposed approach requires neither the set of anomalies to be detected, nor the thresholds to be supplied by the user. Experimental results showed the effectiveness of the method on real data. Low alarm rate and high detection accuracy has been demonstrated. The key point that allowed efficient detection of network problems was to avoid solving the more general problem of accurate parameter estimation, as in intermediate between traffic modeling and change detection. REFERENCES [1] F. Barceló and J. Jordán, Channel holding time distribution in public telephony system (PAMR and PCS), IEEE Trans. Veh. Technol., vol. 49, no. 5, pp , Sep [2] M. Basseville and I. V. Nikiforov, Detection of Abrupt Changes: Theory and Application. Englewood Cliffs, NJ: PrenticeHall, [3] V. A. Bolotin, Telephone circuit holding time distributions, in Proc. 14th Int. Conf. Fundamanetal Role of Teletraffic in the Evolution of Telecommunications Networks, vol. 1a, Amsterdam, The Netherlands, 1994, pp [4] V. A. Bolotin, Modeling call holding time distributions for CCS network design and performance analysis, IEEE J. Sel. Areas Commun., vol. 12, pp , Apr [5] V. A. Bolotin, Y. Levi, and D. Liu, Characterizing data connection and messages by mixtures of distributions on logarithmic scale, in Proc. 16th Int. Conf. Teletraffic Engineering in a Competitive World, Amsterdam, The Netherlands, [6] T. J. Brailsford, J. H. W. Penm, and R. D. Terrell, Selecting the forgetting factor in subset autoregressive modeling, J. TimeSeries Anal., vol. 23, no. 6, pp. 1 21, [7] A. Dempster, N. Laird, and D. Rubin, Maximum likelihood from incomplete data via the EM algorithm, J. Roy. Statist. Soc. B, vol. 39, pp. 1 38, [8] A. Erramilli, O. Narayan, and W. Willinger, Experimental queueing analysis with longrange dependent packet traffic, IEEE/ACM Trans. Netw., vol. 4, no. 2, pp , [9] G. Goldzsmidt and Y. Yemini, Distributed management by delegation, in Proc. 15th Int. Conf. Distributed Computing Systems, [10] H. Hajji and B. H. Far, Continuous network monitoring for fast detection of performance problems, in Proc Int. Symp. Performance Evaluation of Computer and Telecommunication Systems, [11] H. Hajji, B. H. Far, and J. Cheng, Detection of network faults and performance problems, in Proc. Internet Conf., Japan, pp [12] H. Hajji, Baselining network traffic and online faults detection, in Proc. IEEE Int. Conf. Communications, vol. 26, May 2003, pp [13] S. C. Hood and C. Ji, Proactive network fault detection, in Proc. IN FOCOM 97, 1997, pp [14] G. Jakobson and M. D. Weissman, Alarm correlation, IEEE Network, pp , [15] I. Katzela and M. Schwarz, Schemes for fault identification is communication networks, IEEE/ACM Trans. Netw., vol. 3, pp , [16] W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson, On the selfsimilar nature of ethernet traffic, IEEE/ACM Trans. Netw., vol. 2, no. 1, pp. 1 15, Feb [17] V. Paxson and S. Floyd, Widearea traffic: The failure of poisson modeling, IEEE/ACM Trans. Netw., vol. 3, no. 3, pp , Jun [18] V. Paxson, Empiricallyderived analytic models of widearea TCP connections, IEEE/ACM Trans. Netw., vol. 2, no. 4, Aug [19] L. LaBarre, Management by exception: OSI event generation, reporting, and logging, in Proc. 2nd Int. Symp. Integrated Network Management, [20] G. Lorden, Procedures for reacting to a change in distribution, Ann. Math. Statist., vol. 42, pp , [21] R. A. Maxion and F. E. Feather, A case study of ethernet anomalies in a distributed computing environments, IEEE Trans. Reliab., vol. 39, no. 4, pp , [22] J. P. MartinFlatin, S. Znaty, and J. P. Hubaux, A survey of distributed enterprise network and systems management paradigms, J. Netw. Syst. Manage., vol. 7, no. 1, pp. 9 26, [23] E. S. Page, Continuous inspection schemes, Biometrika, vol. 41, pp , [24] R. A. Redner and H. F. Walker, Mixture densities, maximum likelihood and the EM algorithm, SIAM Rev., vol. 26, no. 2, pp , [25] D. Seigmund and E. S. Venkatraman, Using the generalized likelihood ratio statistic for sequential detection of a change points, Ann. Statist., vol. 23, no. 1, pp , [26] D. Siegmund, Sequential Analysis: Tests and Confidence Intervals. New York: SpringerVerlag, [27] M. Thottan and C. Ji, Statistical detection of enterprise network problems, J. Netw. Syst. Manage., vol. 7, no. 1, [28] D. M. Titterington, Recursive parameter estimation using incomplete data, J. Roy. Statist. Soc., ser. B, vol. 46, no. 2, pp , [29] N. A. Vlassis, A kurtosisbased dynamic approach to Gaussian mixture modeling, IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 29, no. 4, pp , [30] E. Weinstein, M. Feder, and A. V. Oppenheim, Sequential algorithms for parameter estimation based on Kullback Leibler information measure, IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 9, pp , Sep [31] S. Yemini, S. Kliger, E. Mozes, Y. Yemini, and D. Ohsie, High speed and robust event correlation, IEEE Commun. Mag., pp , Hassan Hajji (M 05) received the B.Sc. degree from Mohamed Premier University, Morocco, in 1995 and the M.Sc. and Ph.D. degrees from Saitama University, Japan, in 1999 and 2002, respectively. He is currently with IBM Business Consulting, Tokyo, Japan, where he works on gon Demad Innovation Services service line. Prior to that, he was with IBM Research, Tokyo Research Laboratory. His research interests include enterprise risk management, security and privacy, and network and system management.
The CUSUM algorithm a small review. Pierre Granjon
The CUSUM algorithm a small review Pierre Granjon June, 1 Contents 1 The CUSUM algorithm 1.1 Algorithm............................... 1.1.1 The problem......................... 1.1. The different steps......................
More informationInternet Traffic Variability (Long Range Dependency Effects) Dheeraj Reddy CS8803 Fall 2003
Internet Traffic Variability (Long Range Dependency Effects) Dheeraj Reddy CS8803 Fall 2003 Selfsimilarity and its evolution in Computer Network Measurements Prior models used Poissonlike models Origins
More informationObservingtheeffectof TCP congestion controlon networktraffic
Observingtheeffectof TCP congestion controlon networktraffic YongminChoi 1 andjohna.silvester ElectricalEngineeringSystemsDept. UniversityofSouthernCalifornia LosAngeles,CA900892565 {yongminc,silvester}@usc.edu
More informationIP NETWORKS carrying voice traffic are beginning to
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 16, NO. 5, SEPTEMBER 2005 1019 Load Characterization and Anomaly Detection for Voice Over IP Traffic Michel Mandjes, Iraj Saniee, Member, IEEE, and Alexander
More informationCoMPACTMonitor: ChangeofMeasure based Passive/Active Monitoring Weighted Active Sampling Scheme to Infer QoS
CoMPACTMonitor: ChangeofMeasure based Passive/Active Monitoring Weighted Active Sampling Scheme to Infer QoS Masaki Aida, Keisuke Ishibashi, and Toshiyuki Kanazawa NTT Information Sharing Platform Laboratories,
More informationA HYBRID RULE BASED FUZZYNEURAL EXPERT SYSTEM FOR PASSIVE NETWORK MONITORING
A HYBRID RULE BASED FUZZYNEURAL EXPERT SYSTEM FOR PASSIVE NETWORK MONITORING AZRUDDIN AHMAD, GOBITHASAN RUDRUSAMY, RAHMAT BUDIARTO, AZMAN SAMSUDIN, SURESRAWAN RAMADASS. Network Research Group School of
More informationA Hybrid Approach to Efficient Detection of Distributed DenialofService Attacks
Technical Report, June 2008 A Hybrid Approach to Efficient Detection of Distributed DenialofService Attacks Christos Papadopoulos Department of Computer Science Colorado State University 1873 Campus
More informationOn Correlating Performance Metrics
On Correlating Performance Metrics Yiping Ding and Chris Thornley BMC Software, Inc. Kenneth Newman BMC Software, Inc. University of Massachusetts, Boston Performance metrics and their measurements are
More informationLoad Balancing and Switch Scheduling
EE384Y Project Final Report Load Balancing and Switch Scheduling Xiangheng Liu Department of Electrical Engineering Stanford University, Stanford CA 94305 Email: liuxh@systems.stanford.edu Abstract Load
More informationAnomaly Detection in IP Networks
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 8, AUGUST 2003 2191 Anomaly Detection in IP Networks Marina Thottan and Chuanyi Ji Abstract Network anomaly detection is a vibrant research area. Researchers
More informationNetwork Monitoring. RMONBased vs. Localized Analysis. White paper. w w w. n i k s u n. c o m
Network Monitoring RMONBased vs. Localized Analysis White paper w w w. n i k s u n. c o m Copyrights and Trademarks NetVCR and NIKSUN are registered trademarks of NIKSUN, Inc. NetReporter, NetDetector,
More informationTime Series Analysis of Network Traffic
Time Series Analysis of Network Traffic Cyriac James IIT MADRAS February 9, 211 Cyriac James (IIT MADRAS) February 9, 211 1 / 31 Outline of the presentation Background Motivation for the Work Network Trace
More informationInterpreting KullbackLeibler Divergence with the NeymanPearson Lemma
Interpreting KullbackLeibler Divergence with the NeymanPearson Lemma Shinto Eguchi a, and John Copas b a Institute of Statistical Mathematics and Graduate University of Advanced Studies, Minamiazabu
More informationCHAPTER 1 INTRODUCTION
21 CHAPTER 1 INTRODUCTION 1.1 PREAMBLE Wireless adhoc network is an autonomous system of wireless nodes connected by wireless links. Wireless adhoc network provides a communication over the shared wireless
More informationSelfCompressive Approach for Distributed System Monitoring
SelfCompressive Approach for Distributed System Monitoring Akshada T Bhondave Dr D.Y Patil COE Computer Department, Pune University, India Santoshkumar Biradar Assistant Prof. Dr D.Y Patil COE, Computer
More informationIntroduction. Chapter 1
Chapter 1 Introduction The area of fault detection and diagnosis is one of the most important aspects in process engineering. This area has received considerable attention from industry and academia because
More informationVoIP Network Dimensioning using Delay and Loss Bounds for Voice and Data Applications
VoIP Network Dimensioning using Delay and Loss Bounds for Voice and Data Applications Veselin Rakocevic School of Engineering and Mathematical Sciences City University, London, UK V.Rakocevic@city.ac.uk
More informationTraffic Behavior Analysis with Poisson Sampling on Highspeed Network 1
Traffic Behavior Analysis with Poisson Sampling on Highspeed etwork Guang Cheng Jian Gong (Computer Department of Southeast University anjing 0096, P.R.China) Abstract: With the subsequent increasing
More informationDetecting Anomalies in Network Traffic Using Maximum Entropy Estimation
Detecting Anomalies in Network Traffic Using Maximum Entropy Estimation Yu Gu, Andrew McCallum, Don Towsley Department of Computer Science, University of Massachusetts, Amherst, MA 01003 Abstract We develop
More informationG.Vijaya kumar et al, Int. J. Comp. Tech. Appl., Vol 2 (5), 14131418
An Analytical Model to evaluate the Approaches of Mobility Management 1 G.Vijaya Kumar, *2 A.Lakshman Rao *1 M.Tech (CSE Student), Pragati Engineering College, Kakinada, India. Vijay9908914010@gmail.com
More information522015 RMON, the New SNMP Remote Monitoring Standard Nathan J. Muller
522015 RMON, the New SNMP Remote Monitoring Standard Nathan J. Muller Payoff The Remote Monitoring (RMON) Management Information Base (MIB) is a set of object definitions that extend the capabilities
More informationAdaptive Tolerance Algorithm for Distributed TopK Monitoring with Bandwidth Constraints
Adaptive Tolerance Algorithm for Distributed TopK Monitoring with Bandwidth Constraints Michael Bauer, Srinivasan Ravichandran University of WisconsinMadison Department of Computer Sciences {bauer, srini}@cs.wisc.edu
More informationA Passive Method for Estimating EndtoEnd TCP Packet Loss
A Passive Method for Estimating EndtoEnd TCP Packet Loss Peter Benko and Andras Veres Traffic Analysis and Network Performance Laboratory, Ericsson Research, Budapest, Hungary {Peter.Benko, Andras.Veres}@eth.ericsson.se
More informationDetecting Flooding Attacks Using Power Divergence
Detecting Flooding Attacks Using Power Divergence Jean Tajer IT Security for the Next Generation European Cup, Prague 1719 February, 2012 PAGE 1 Agenda 1 Introduction 2 Kary Sktech 3 Detection Threshold
More informationDenial of Service and Anomaly Detection
Denial of Service and Anomaly Detection Vasilios A. Siris Institute of Computer Science (ICS) FORTH, Crete, Greece vsiris@ics.forth.gr SCAMPI BoF, Zagreb, May 21 2002 Overview! What the problem is and
More informationImpact of Feature Selection on the Performance of Wireless Intrusion Detection Systems
2009 International Conference on Computer Engineering and Applications IPCSIT vol.2 (2011) (2011) IACSIT Press, Singapore Impact of Feature Selection on the Performance of ireless Intrusion Detection Systems
More informationLowrate TCPtargeted Denial of Service Attack Defense
Lowrate TCPtargeted Denial of Service Attack Defense Johnny Tsao Petros Efstathopoulos University of California, Los Angeles, Computer Science Department Los Angeles, CA Email: {johnny5t, pefstath}@cs.ucla.edu
More informationIntroduction to Performance Measurements and Monitoring. Vinayak Hegde vinayakh@gmail.com @vinayakh
Introduction to Performance Measurements and Monitoring Vinayak Hegde vinayakh@gmail.com @vinayakh 1 Structure Part 1 Basics of Measurement (40 min) Part 2 Basic Statistics primer (40 min) Part 3 Measurement
More information400B.2.1 CH28274/90/00000350 $1.OO 0 1990 IEEE
Performance Characterizations of Traffic Monitoring, and Associated Control, Mechanisms for Broadband "Packet" Networks A.W. Berger A.E. Eckberg Room 3R601 Room 35611 Holmde1,NJ 07733 USA Holmde1,NJ
More informationEnterprise Application Performance Management: An EndtoEnd Perspective
SETLabs Briefings VOL 4 NO 2 Oct  Dec 2006 Enterprise Application Performance Management: An EndtoEnd Perspective By Vishy Narayan With rapidly evolving technology, continued improvements in performance
More informationPenalized regression: Introduction
Penalized regression: Introduction Patrick Breheny August 30 Patrick Breheny BST 764: Applied Statistical Modeling 1/19 Maximum likelihood Much of 20thcentury statistics dealt with maximum likelihood
More informationHow Useful Is Old Information?
6 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 1, JANUARY 2000 How Useful Is Old Information? Michael Mitzenmacher AbstractÐWe consider the problem of load balancing in dynamic distributed
More informationA NOVEL RESOURCE EFFICIENT DMMS APPROACH
A NOVEL RESOURCE EFFICIENT DMMS APPROACH FOR NETWORK MONITORING AND CONTROLLING FUNCTIONS Golam R. Khan 1, Sharmistha Khan 2, Dhadesugoor R. Vaman 3, and Suxia Cui 4 Department of Electrical and Computer
More informationChangePoint Analysis: A Powerful New Tool For Detecting Changes
ChangePoint Analysis: A Powerful New Tool For Detecting Changes WAYNE A. TAYLOR Baxter Healthcare Corporation, Round Lake, IL 60073 Changepoint analysis is a powerful new tool for determining whether
More informationTRAFFIC control and bandwidth management in ATM
134 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 1, FEBRUARY 1997 A Framework for Bandwidth Management in ATM Networks Aggregate Equivalent Bandwidth Estimation Approach Zbigniew Dziong, Marek Juda,
More informationSTATISTICA Formula Guide: Logistic Regression. Table of Contents
: Table of Contents... 1 Overview of Model... 1 Dispersion... 2 Parameterization... 3 SigmaRestricted Model... 3 Overparameterized Model... 4 Reference Coding... 4 Model Summary (Summary Tab)... 5 Summary
More informationSynchronization of sampling in distributed signal processing systems
Synchronization of sampling in distributed signal processing systems Károly Molnár, László Sujbert, Gábor Péceli Department of Measurement and Information Systems, Budapest University of Technology and
More informationOverview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written
More informationQuantifying the Performance Degradation of IPv6 for TCP in Windows and Linux Networking
Quantifying the Performance Degradation of IPv6 for TCP in Windows and Linux Networking Burjiz Soorty School of Computing and Mathematical Sciences Auckland University of Technology Auckland, New Zealand
More informationMeasurement and Modelling of Internet Traffic at Access Networks
Measurement and Modelling of Internet Traffic at Access Networks Johannes Färber, Stefan Bodamer, Joachim Charzinski 2 University of Stuttgart, Institute of Communication Networks and Computer Engineering,
More informationAN EFFICIENT DISTRIBUTED CONTROL LAW FOR LOAD BALANCING IN CONTENT DELIVERY NETWORKS
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 3, Issue. 9, September 2014,
More informationPublished in: Proceedings of 6th International Computer Engineering Conference (ICENCO)
Aalborg Universitet Characterization and Modeling of Network Shawky, Ahmed Sherif Mahmoud; Bergheim, Hans ; Ragnarsson, Olafur ; Wranty, Andrzej ; Pedersen, Jens Myrup Published in: Proceedings of 6th
More informationON THE FRACTAL CHARACTERISTICS OF NETWORK TRAFFIC AND ITS UTILIZATION IN COVERT COMMUNICATIONS
ON THE FRACTAL CHARACTERISTICS OF NETWORK TRAFFIC AND ITS UTILIZATION IN COVERT COMMUNICATIONS Rashiq R. Marie Department of Computer Science email: R.R.Marie@lboro.ac.uk Helmut E. Bez Department of Computer
More informationDrop Call Probability in Established Cellular Networks: from data Analysis to Modelling
Drop Call Probability in Established Cellular Networks: from data Analysis to Modelling G. Boggia, P. Camarda, A. D Alconzo, A. De Biasi and M. Siviero DEE  Politecnico di Bari, Via E. Orabona, 47125
More informationA Dynamic Polling Scheme for the Network Monitoring Problem
A Dynamic Polling Scheme for the Network Monitoring Problem Feng Gao, Jairo Gutierrez* Dept. of Computer Science *Dept. of Management Science and Information Systems University of Auckland, New Zealand
More informationTaxonomy of Intrusion Detection System
Taxonomy of Intrusion Detection System Monika Sharma, Sumit Sharma Abstract During the past years, security of computer networks has become main stream in most of everyone's lives. Nowadays as the use
More informationNetwork Intrusion Detection Systems
Network Intrusion Detection Systems False Positive Reduction Through Anomaly Detection Joint research by Emmanuele Zambon & Damiano Bolzoni 7/1/06 NIDS  False Positive reduction through Anomaly Detection
More informationTechnical Bulletin. Arista LANZ Overview. Overview
Technical Bulletin Arista LANZ Overview Overview Highlights: LANZ provides unparalleled visibility into congestion hotspots LANZ time stamping provides for precision historical trending for congestion
More informationAn Efficient Way of Denial of Service Attack Detection Based on Triangle Map Generation
An Efficient Way of Denial of Service Attack Detection Based on Triangle Map Generation Shanofer. S Master of Engineering, Department of Computer Science and Engineering, Veerammal Engineering College,
More informationReal Time Network Server Monitoring using Smartphone with Dynamic Load Balancing
www.ijcsi.org 227 Real Time Network Server Monitoring using Smartphone with Dynamic Load Balancing Dhuha Basheer Abdullah 1, Zeena Abdulgafar Thanoon 2, 1 Computer Science Department, Mosul University,
More informationA Spectral Clustering Approach to Validating Sensors via Their Peers in Distributed Sensor Networks
A Spectral Clustering Approach to Validating Sensors via Their Peers in Distributed Sensor Networks H. T. Kung Dario Vlah {htk, dario}@eecs.harvard.edu Harvard School of Engineering and Applied Sciences
More informationMachine Learning in Statistical Arbitrage
Machine Learning in Statistical Arbitrage Xing Fu, Avinash Patra December 11, 2009 Abstract We apply machine learning methods to obtain an index arbitrage strategy. In particular, we employ linear regression
More informationApplication of Netflow logs in Analysis and Detection of DDoS Attacks
International Journal of Computer and Internet Security. ISSN 09742247 Volume 8, Number 1 (2016), pp. 18 International Research Publication House http://www.irphouse.com Application of Netflow logs in
More informationSAS Software to Fit the Generalized Linear Model
SAS Software to Fit the Generalized Linear Model Gordon Johnston, SAS Institute Inc., Cary, NC Abstract In recent years, the class of generalized linear models has gained popularity as a statistical modeling
More informationCorrected Diffusion Approximations for the Maximum of HeavyTailed Random Walk
Corrected Diffusion Approximations for the Maximum of HeavyTailed Random Walk Jose Blanchet and Peter Glynn December, 2003. Let (X n : n 1) be a sequence of independent and identically distributed random
More informationSupplement to Call Centers with Delay Information: Models and Insights
Supplement to Call Centers with Delay Information: Models and Insights Oualid Jouini 1 Zeynep Akşin 2 Yves Dallery 1 1 Laboratoire Genie Industriel, Ecole Centrale Paris, Grande Voie des Vignes, 92290
More informationOn the Use of Traffic Monitoring and Measurements for Improving Networking
On the Use of Traffic Monitoring and Measurements for Improving Networking Sílvia Farraposo 1, Philippe Owezarski 2, Edmundo Monteiro 3 1 Escola Superior de Tecnologia e Gestão de Leiria, Morro do Lena
More informationNetwork Performance Monitoring at Small Time Scales
Network Performance Monitoring at Small Time Scales Konstantina Papagiannaki, Rene Cruz, Christophe Diot Sprint ATL Burlingame, CA dina@sprintlabs.com Electrical and Computer Engineering Department University
More informationComponent Ordering in Independent Component Analysis Based on Data Power
Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals
More informationWhite Paper. The Ten Features Your Web Application Monitoring Software Must Have. Executive Summary
White Paper The Ten Features Your Web Application Monitoring Software Must Have Executive Summary It s hard to find an important business application that doesn t have a webbased version available and
More informationNetwork performance and capacity planning: Techniques for an ebusiness world
IBM Global Services Network performance and capacity planning: Techniques for an ebusiness world ebusiness is about transforming key business processes with Internet technologies. In an ebusiness world,
More informationDefending Against Traffic Analysis Attacks with Link Padding for Bursty Traffics
Proceedings of the 4 IEEE United States Military Academy, West Point, NY  June Defending Against Traffic Analysis Attacks with Link Padding for Bursty Traffics Wei Yan, Student Member, IEEE, and Edwin
More informationHybrid processing of SCADA and synchronized phasor measurements for tracking network state
IEEE PES General Meeting, Denver, USA, July 2015 1 Hybrid processing of SCADA and synchronized phasor measurements for tracking network state Boris AlcaideMoreno Claudio FuerteEsquivel Universidad Michoacana
More informationStatistics in Retail Finance. Chapter 7: Fraud Detection in Retail Credit
Statistics in Retail Finance Chapter 7: Fraud Detection in Retail Credit 1 Overview > Detection of fraud remains an important issue in retail credit. Methods similar to scorecard development may be employed,
More informationData Cleansing for Remote Battery System Monitoring
Data Cleansing for Remote Battery System Monitoring Gregory W. Ratcliff Randall Wald Taghi M. Khoshgoftaar Director, Life Cycle Management Senior Research Associate Director, Data Mining and Emerson Network
More informationPerformance Evaluation of AODV, OLSR Routing Protocol in VOIP Over Ad Hoc
(International Journal of Computer Science & Management Studies) Vol. 17, Issue 01 Performance Evaluation of AODV, OLSR Routing Protocol in VOIP Over Ad Hoc Dr. Khalid Hamid Bilal Khartoum, Sudan dr.khalidbilal@hotmail.com
More informationCisco Performance Visibility Manager 1.0.1
Cisco Performance Visibility Manager 1.0.1 Cisco Performance Visibility Manager (PVM) is a proactive network and applicationperformance monitoring, reporting, and troubleshooting system for maximizing
More informationANALYZING NETWORK TRAFFIC FOR MALICIOUS ACTIVITY
CANADIAN APPLIED MATHEMATICS QUARTERLY Volume 12, Number 4, Winter 2004 ANALYZING NETWORK TRAFFIC FOR MALICIOUS ACTIVITY SURREY KIM, 1 SONG LI, 2 HONGWEI LONG 3 AND RANDALL PYKE Based on work carried out
More informationCommunication Protocol
Analysis of the NXT Bluetooth Communication Protocol By Sivan Toledo September 2006 The NXT supports Bluetooth communication between a program running on the NXT and a program running on some other Bluetooth
More informationAvaya ExpertNet Lite Assessment Tool
IP Telephony Contact Centers Mobility Services WHITE PAPER Avaya ExpertNet Lite Assessment Tool April 2005 avaya.com Table of Contents Overview... 1 Network Impact... 2 Network Paths... 2 Path Generation...
More informationRole of Anomaly IDS in Network
Role of Anomaly IDS in Network SumathyMurugan 1, Dr.M.Sundara Rajan 2 1 Asst. Prof, Department of Computer Science, Thiruthangal Nadar College, Chennai 51. 2 Asst. Prof, Department of Computer Science,
More informationOn the Traffic Capacity of Cellular Data Networks. 1 Introduction. T. Bonald 1,2, A. Proutière 1,2
On the Traffic Capacity of Cellular Data Networks T. Bonald 1,2, A. Proutière 1,2 1 France Telecom Division R&D, 3840 rue du Général Leclerc, 92794 IssylesMoulineaux, France {thomas.bonald, alexandre.proutiere}@francetelecom.com
More informationOBJECTIVE ASSESSMENT OF FORECASTING ASSIGNMENTS USING SOME FUNCTION OF PREDICTION ERRORS
OBJECTIVE ASSESSMENT OF FORECASTING ASSIGNMENTS USING SOME FUNCTION OF PREDICTION ERRORS CLARKE, Stephen R. Swinburne University of Technology Australia One way of examining forecasting methods via assignments
More informationStatistics in Retail Finance. Chapter 6: Behavioural models
Statistics in Retail Finance 1 Overview > So far we have focussed mainly on application scorecards. In this chapter we shall look at behavioural models. We shall cover the following topics: Behavioural
More informationAn Active Packet can be classified as
Mobile Agents for Active Network Management By Rumeel Kazi and Patricia Morreale Stevens Institute of Technology Contact: rkazi,pat@ati.stevenstech.edu AbstractTraditionally, network management systems
More informationDesign of an FX trading system using Adaptive Reinforcement Learning
University Finance Seminar 17 March 2006 Design of an FX trading system using Adaptive Reinforcement Learning M A H Dempster Centre for Financial Research Judge Institute of Management University of &
More informationVOIP TRAFFIC SHAPING ANALYSES IN METROPOLITAN AREA NETWORKS. Rossitza Goleva, Mariya Goleva, Dimitar Atamian, Tashko Nikolov, Kostadin Golev
International Journal "Information Technologies and Knowledge" Vol.2 / 28 181 VOIP TRAFFIC SHAPING ANALYSES IN METROPOLITAN AREA NETWORKS Rossitza Goleva, Mariya Goleva, Dimitar Atamian, Tashko Nikolov,
More informationA Secure Online Reputation Defense System from Unfair Ratings using Anomaly Detections
A Secure Online Reputation Defense System from Unfair Ratings using Anomaly Detections Asha baby PG Scholar,Department of CSE A. Kumaresan Professor, Department of CSE K. Vijayakumar Professor, Department
More informationA Power Efficient QoS Provisioning Architecture for Wireless Ad Hoc Networks
A Power Efficient QoS Provisioning Architecture for Wireless Ad Hoc Networks Didem Gozupek 1,Symeon Papavassiliou 2, Nirwan Ansari 1, and Jie Yang 1 1 Department of Electrical and Computer Engineering
More informationAnalysis of Load Frequency Control Performance Assessment Criteria
520 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 16, NO. 3, AUGUST 2001 Analysis of Load Frequency Control Performance Assessment Criteria George Gross, Fellow, IEEE and Jeong Woo Lee Abstract This paper presents
More informationA probabilistic multitenant model for virtual machine mapping in cloud systems
A probabilistic multitenant model for virtual machine mapping in cloud systems Zhuoyao Wang, Majeed M. Hayat, Nasir Ghani, and Khaled B. Shaban Department of Electrical and Computer Engineering, University
More informationNetwork Monitoring. ChuSing Yang. Department of Electrical Engineering National Cheng Kung University
Network Monitoring ChuSing Yang Department of Electrical Engineering National Cheng Kung University Outline Introduction Network monitoring architecture Performance monitoring Fault monitoring Accounting
More informationDetecting Network Anomalies. Anant Shah
Detecting Network Anomalies using Traffic Modeling Anant Shah Anomaly Detection Anomalies are deviations from established behavior In most cases anomalies are indications of problems The science of extracting
More informationTraffic Monitoring in a Switched Environment
Traffic Monitoring in a Switched Environment InMon Corp. 1404 Irving St., San Francisco, CA 94122 www.inmon.com 1. SUMMARY This document provides a brief overview of some of the issues involved in monitoring
More informationAgentbased Simulation Study of Behavioral Anticipation: Anticipatory Fault Management in Computer Networks
Agentbased Simulation Study of Behavioral Anticipation: Anticipatory Fault Management in Computer Networs Avdhoot Saple M&SNet: AMSL (The Auburn Modeling and Simulation Laboratory) Computer Science and
More informationEstablishing How Many VoIP Calls a Wireless LAN Can Support Without Performance Degradation
Establishing How Many VoIP Calls a Wireless LAN Can Support Without Performance Degradation ABSTRACT Ángel Cuevas Rumín Universidad Carlos III de Madrid Department of Telematic Engineering Ph.D Student
More informationKalman Filter Applied to a Active Queue Management Problem
IOSR Journal of Electrical and Electronics Engineering (IOSRJEEE) eissn: 22781676,pISSN: 23203331, Volume 9, Issue 4 Ver. III (Jul Aug. 2014), PP 2327 Jyoti Pandey 1 and Prof. Aashih Hiradhar 2 Department
More informationAn AnomalyBased Method for DDoS Attacks Detection using RBF Neural Networks
2011 International Conference on Network and Electronics Engineering IPCSIT vol.11 (2011) (2011) IACSIT Press, Singapore An AnomalyBased Method for DDoS Attacks Detection using RBF Neural Networks Reyhaneh
More informationA RegimeSwitching Model for Electricity Spot Prices. Gero Schindlmayr EnBW Trading GmbH g.schindlmayr@enbw.com
A RegimeSwitching Model for Electricity Spot Prices Gero Schindlmayr EnBW Trading GmbH g.schindlmayr@enbw.com May 31, 25 A RegimeSwitching Model for Electricity Spot Prices Abstract Electricity markets
More informationIP Forwarding Anomalies and Improving their Detection using Multiple Data Sources
IP Forwarding Anomalies and Improving their Detection using Multiple Data Sources Matthew Roughan (Univ. of Adelaide) Tim Griffin (Intel Research Labs) Z. Morley Mao (Univ. of Michigan) Albert Greenberg,
More informationComputer Network. Interconnected collection of autonomous computers that are able to exchange information
Introduction Computer Network. Interconnected collection of autonomous computers that are able to exchange information No master/slave relationship between the computers in the network Data Communications.
More informationWindows Server Performance Monitoring
Spot server problems before they are noticed The system s really slow today! How often have you heard that? Finding the solution isn t so easy. The obvious questions to ask are why is it running slowly
More informationSystem Identification for Acoustic Comms.:
System Identification for Acoustic Comms.: New Insights and Approaches for Tracking Sparse and Rapidly Fluctuating Channels Weichang Li and James Preisig Woods Hole Oceanographic Institution The demodulation
More informationGaining Operational Efficiencies with the Enterasys SSeries
Gaining Operational Efficiencies with the Enterasys SSeries HiFidelity NetFlow There is nothing more important than our customers. Gaining Operational Efficiencies with the Enterasys SSeries Introduction
More informationProbabilistic properties and statistical analysis of network traffic models: research project
Probabilistic properties and statistical analysis of network traffic models: research project The problem: It is commonly accepted that teletraffic data exhibits selfsimilarity over a certain range of
More informationTraffic Analyzer Based on Data Flow Patterns
AUTOMATYKA 2011 Tom 15 Zeszyt 3 Artur Sierszeñ*, ukasz Sturgulewski* Traffic Analyzer Based on Data Flow Patterns 1. Introduction Nowadays, there are many systems of Network Intrusion Detection System
More informationChapter 6: The Information Function 129. CHAPTER 7 Test Calibration
Chapter 6: The Information Function 129 CHAPTER 7 Test Calibration 130 Chapter 7: Test Calibration CHAPTER 7 Test Calibration For didactic purposes, all of the preceding chapters have assumed that the
More information5 Steps to Avoid Network Alert Overload
5 Steps to Avoid Network Alert Overload By Avril Salter 1. 8 0 0. 8 1 3. 6 4 1 5 w w w. s c r i p t l o g i c. c o m / s m b I T 2011 ScriptLogic Corporation ALL RIGHTS RESERVED. ScriptLogic, the ScriptLogic
More informationPerformance of Cisco IPS 4500 and 4300 Series Sensors
White Paper Performance of Cisco IPS 4500 and 4300 Series Sensors White Paper September 2012 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of
More information171:290 Model Selection Lecture II: The Akaike Information Criterion
171:290 Model Selection Lecture II: The Akaike Information Criterion Department of Biostatistics Department of Statistics and Actuarial Science August 28, 2012 Introduction AIC, the Akaike Information
More information