Real-time Network Monitoring Supporting Percentile Error Objectives

14 th HP Software University Association Workshop, July 2007, Munich, Germany Real-time Network Monitoring Supporting Percentile Error Objectives Alberto Gonzalez Prieto and Rolf Stadler School of Electrical Engineering KTH Royal Institute of Technology Stockholm, Sweden {gonzalez, stadler}@ee.kth.se Abstract. We report on the versatility of A-GAP for supporting different types of accuracy objectives. Previously, we considered accuracy objectives expressed in terms of the average error. In this paper, we focus on percentile error objectives. A-GAP is a protocol for continuous monitoring of network state variables. Network state variables are computed from device counters using aggregation functions, such as SUM, AVERAGE and MAX. A-GAP is designed to achieve a given monitoring accuracy with minimal overhead. A- GAP is decentralized and asynchronous to achieve robustness and scalability. It executes on an overlay that interconnects management processes on the devices. On this overlay, the protocol maintains a spanning tree and updates the network state variables through incremental aggregation. Based on a stochastic model, it dynamically configures local filters that control whether an update is sent towards the root of the tree. We evaluate A-GAP through simulation using real traces for an ISP topology (Abovenet). The results prove the versatility of A- GAP for supporting different types of accuracy objectives. The results also show that we can effectively control the trade-off between accuracy and protocol overhead, and that the overhead can be reduced significantly by allowing small errors. Keywords: Distributed management, real-time monitoring, large-scale distributed systems, adaptive systems 1 Introduction The ability to provide continuous estimates of management variables is vital for many management tasks, including network supervision, quality assurance, and proactive fault management. Often, management variables that are monitored in these tasks are aggregates that are computed from device variables across the network using functions such as SUM, AVERAGE, MIN, and MAX. Sample aggregates are the total number of VoIP flows, the maximum link utilization, or a histogram of the current load across routers in a network domain. While it is often crucial to know how accurate such estimates are, network management solutions deployed today usually provide only qualitative control of the accuracy and do not support the setting of an accuracy objective. 1

Management Station Global Aggregate Partial Aggregate Local variable 12 25 3 Root 10 Physical Node Aggregating Node Leaf Node 4 1 3 5 2 7 3 5 2 7 Fig. 1. Example of an aggregation tree with aggregation function SUM. Engineering continuous monitoring solutions for network management involves addressing the fundamental trade-off between accurate estimation of a variable and the management overhead in terms of traffic and processing load [6]. We focus on the problem of continuous monitoring with accuracy objectives for large-scale network environments. Our goal is to achieve an efficient solution that allows us to control the accuracy of the estimation. Specifically, in this paper, we show the versatility of our protocol, A-GAP [13] for supporting different accuracy objectives. A-GAP is a generic aggregation protocol with controllable accuracy. A-GAP allows for continuously computing aggregates of local variables by (i) creating and maintaining a self-stabilizing spanning tree and (ii) incrementally aggregating the variables along the tree (figure 1). It is push-based in the sense that changes in monitored variables are sent towards the management station along the aggregation tree. The protocol controls the management overhead by filtering updates that are sent from monitoring nodes to the management station. The filters periodically adapt to the dynamics of the monitored variables and the network environment. All operations in A-GAP, including computing the aggregation function and filter configuration, are executed in a decentralized and asynchronous fashion to ensure robustness and achieve scalability. We developed a stochastic model of the monitoring process, which allows us to compute the filter widths as the solution to the problem of minimizing the management overhead for a given estimation error. A distributed heuristic solution to this problem is implemented in A-GAP. A distinctive feature of this approach is that it provides us with an estimation of the error distribution at the management station in real-time. This allows us to choose from a variety of control objectives, including average error and percentiles, which we have chosen for this paper. 2

A-GAP assumes access to local variables. This can be realized through a variety of mechanisms including reading MIB variables and CLI commands. The specific mechanism used is transparent to A-GAP. This paper complements [13] by showing the versatility of A-GAP for supporting different types of accuracy objectives. In [13], we considered accuracy objectives expressed in terms of the average error. In this paper, we focus on percentile error objectives. Percentile-based accuracy is particularly relevant in the context of quality assurance. The paper is organized as follows. Section II defines the problem of real-time monitoring with accuracy objectives. Section III describes our solution for continuous monitoring with accuracy objectives, A-GAP. Section IV evaluates A-GAP through simulation. Section V discusses related work. Section VI concludes the paper. 2 Problem Statement: Real-time Monitoring with Accuracy 2.1 System Architecture This work assumes a distributed management architecture, whereby each network device participates in the computation by running management processes, either internally or on an external, associated device. These management processes communicate via a management overlay network for the purpose of monitoring. We also refer to this overlay as the network graph. A node in this graph represents a management process. The aggregation tree shown in figure 1 spans the management overlay. Inside each management process runs a leaf node and an aggregating node of this aggregation tree. The topology of this overlay can be chosen independently from the topology of the underlying physical network. 2.2 Problem Statement We consider a dynamically changing network graph G(t) = (V(t), E(t)) in which nodes n V(t) and edges/links e E(t) V(t) x V(t) may appear and disappear over time. Each leaf n has an associated local variable w n (t), which is an integer valued quantity. The term local variable is used to represent a local state variable or device counter that is being subjected to monitoring. Local variables are updated asynchronously with a given sampling rate. The objective is to engineer a protocol on this network graph that provides a management station with a continuous estimate of Σ n w n (t) for a given accuracy. The protocol should execute with minimal overhead in the sense that it minimizes the processing load on the nodes. We consider different types of accuracy objectives, such as the average error or percentiles (i.e., the administrator sets an interval that contains the estimation error 3

Local variable or partial aggregate Last update value Filter width Filter Exceeded: 1) Triggers an update to parent 2) Filter is shifted time Fig. 2. Each node has a local filter. It sends an update to its parent whenever the partial aggregate (or the local variable) exceeds the filter width. with probability p). The overhead is the maximum number of updates per second a node has to process. Throughout the paper we use SUM as aggregation function. Other functions can be supported as well, as discussed in [13]. 3 A-GAP: A Distributed Solution Our solution to the problem of real-time monitoring with accuracy is the A-GAP protocol [13]. A-GAP is based on GAP (Generic Aggregation Protocol), which we developed in our earlier work [1]. GAP is an asynchronous distributed protocol that builds and maintains a BFS (Breadth First Search) spanning tree on an overlay network. The tree is maintained in a similar way as the algorithm that underlies the 802.1d Spanning Tree Protocol (STP) [8]. In GAP, each node holds information about its children in the BFS tree, in order to compute the partial aggregate, i.e., the aggregate value of the local management variables from all nodes of the subtree where this node is the root. GAP is event-driven in the sense that messages are exchanged as results of events, such as the detection of a new neighbor on the overlay, the failure of a neighbor, an aggregate update or a change in the local management variable. A drawback of such an approach is that it can cause a high load on the root node or on nodes close to the root, specifically in large networks. In order to reduce this overhead, we introduce filters in the nodes. When the partial aggregate (or the local variable in the case of a leaf node) of a node n changes, then n sends an update to its parent if the difference between the value reported in its last update and the current value exceeds the local filter width F n. Figure 2 illustrates the behavior of a local filter. The solid line represents the partial aggregate (or the local variable), the dashed line gives the value of the last update from n to its parent, and the dotted line shows the filter width. The vertical line indicates the time at which n sends an update. 4

3.1 The Optimization Problem Estimating the network variable at the root node with minimal overhead for a given accuracy can be formalized as an optimization problem. Let n be a node in the network graph, ω n the rate of updates received by node n from its children, F n the filter width of node n, E root the distribution of the estimation error at the root node, and g() an arbitrary function of a probability distribution. We formulate the problem as Minimize { n } Max ω s.t. g( E root ) θ, (1) n whereby ω n and E root depend on the filter widths (F n ) n, which are the decision variables. We have developed a stochastic model for the monitoring process. The model is based on discrete-time Markov chains and describes individual nodes in their steady state. For each node n, the model relates the error of the partial aggregate of n, the step sizes that indicate changes in the partial aggregate, the rate of updates n sends and the width of the local filter. The model is described in detail in [13]. The model permits us to compute the distribution E root of the estimation error at the root node and the rate of updates ω n processed by each node. This allows us to support different types of accuracy objectives, i.e., different instantiations of the function g( E root ). In [13], we considered the average error as accuracy objective. In this paper, we show the versatility of our solution by considering a different type of accuracy objectives: percentile errors. Specifically, we show the capability of A-GAP for solving the network-wide problem Minimize { ω π } Max s.t. p( E root >γ) θ, (2) π where γ is the interval that contains the estimation error at the root with probability 1-θ. 3.2 The Realization of A-GAP An optimal solution to (eq. 1) can be computed using a (centralized) grid search algorithm, a well-known optimization technique, where the model variables for all nodes in the aggregation tree are computed bottom-up. Such an approach, however, is not feasible for large networks, since the computational cost of this algorithm grows exponentially with the number of nodes. A-GAP realizes a distributed heuristic, which attempts to minimize the maximum processing load on all nodes by minimizing the load within each node s neighborhood. A-GAP maps the network-wide problem (eq. 1) onto a local problem for each node n. For the particular case of (eq. 2), the mapping is as follows: Minimize { ω π } Max s.t. p( E n out >γn ) θ, (3) π 5

where π is the set composed by the node n and its children. This means that node n attempts to minimize the maximum load in a neighborhood for a given percentile error objective (γ n, θ) of its partial aggregate. (When the accuracy objective is expressed in terms of the average error a similar mapping is performed.) The node attempts to solve the local problem (eq. 3) by periodically re-computing the filters and accuracy objectives of its children, based on our model. Re-computing the filters (F c ) c allows node n to influence its own load ω n, while re-computing the accuracy objective (γ c, θ) of a child c allows the node to influence the load ω c on c. A-GAP computes the local filters and accuracy objectives in a decentralized and asynchronous fashion, as described in detail in [13]. The two key configuration parameters of A-GAP are (i) the maximum number of children whose filters and accuracy objectives are recomputed during a control cycle Ω, and (ii) the period of the control cycle τ. As discussed in [13], both parameters influence the adaptability and computational cost of A-GAP. 3.3 The Two Planes of A-GAP A-GAP can be understood as operating in two different planes. On the data plane, updates of the local variables are propagated towards the root and filtering occurs. In this plane, information flows bottom-up, from the leaves towards the root. In the control plane the filters are computed. In this plane, information flows in both directions, bottom-up and top-down. The model variables (step sizes and errors) flow from the leaves towards the root and are incrementally aggregated. The filters are distributed top-down, from the parents to their respective children. 4 Evaluation Through Simulation 4.1 Simulation setup and scenario. We have evaluated the capability of A-GAP for supporting percentile error objectives through extensive simulations using the SIMPSON simulator [4]. The results presented in the paper are based on simulations for two different types of overlay topologies. First, we consider an overlay that follows the physical topology of Abovenet [5], an ISP, which has 654 nodes and 1332 links. Second, we use a grid overlay topology with 25 nodes. The grid topology is built in such a way that each node has 4 neighbors, except for the nodes at the edges of a grid, which have 2 neighbors, and the four corner nodes of a grid, which have 1 neighbor. All evaluation scenarios share the following settings. Link speeds in the overlay are 100 Mbps. The communication delay is 4 ms, and the time to process a message at a node is 1 ms. We present results for θ=0.05 and different intervals γ (i.e., the estimation error is 6

600 500 Updates/sec 400 300 200 100 0 0 5 10 15 20 25 30 35 40 45 Measured Error c Fig. 3. Management overhead incurred by A-GAP as a function of the estimation error c, computed using (eq. 4). (Abovenet topology, θ=0.05) within the given interval γ with 0.95 probability). The local control cycle in A-GAP is set to τ=1 sec, and the (maximum) number of children whose filters are recomputed during a control cycle is Ω =6 for the Abovenet overlay topology and Ω =4 for the grid overlay topology. The local management variable in the simulation experiments represents the number of HTTP flows traversing a given node. The monitored aggregate is the number of HTTP flows in the network (the aggregate is in the order of 20.000 flows for the Abovenet scenarios). The local variables are updated asynchronously, once every second, based on packet traces captured on two 1 Gbit/s links that connect University of Twente to a research network [9]. All simulation runs start with an initialization phase of A-GAP, which takes some 30 seconds simulation time, followed by the setting of the accuracy objective (γ, θ) and a transient period of about 25 seconds, followed by the measurement period of 200 seconds. 4.2 The trade-off between estimation accuracy and protocol overhead. We have measured the protocol overhead (i.e., the maximum number of processed updates across all nodes) in function of the experienced error. Figure 3 shows the measurement results for the Abovenet overlay topology. Every point in figure 3 corresponds to a simulation run. As can be seen, the overhead decreases monotonically, as the estimation error increases. For small errors, the load decreases faster than for larger errors. Consequently, the overhead can be reduced by allowing a larger estimation error. For example, compared to an error objective of 0 (which results in an experienced error of 13, see below), allowing an error of 20 flows (experienced error 23) reduces the load by 77%. An error of 40 flows (experienced error 41) reduces the load by 88%. 7

25 20 Updates /sec 15 10 5 0 0 5 10 15 20 Measured Error c Fig. 4. Management overhead incurred by A-GAP as a function of the estimation error c, computed using (eq. 4). (Grid topology, θ=0.05) Figure 4 shows the measurement results for the grid overlay. We can observe that the trend is similar for both topologies. In this case, allowing an error of 20 flows (experienced error 20) reduces the load in one order of magnitude. Qualitatively, these results are identical to those we presented in [13], where the accuracy objective was expressed in terms of the average error. A-GAP behaves similarly for both types of accuracy objectives Recall in this context that the overhead incurred by A-GAP does not depend on the value of the aggregate, but on the absolute changes to the aggregate. This means that two scenarios with the same absolute changes result in the same overhead, independent of the value of the aggregate. This is why, in this paper, we consider the absolute error as control parameter rather than the relative error. That said, A-GAP can easily be extended for using the relative error as control parameter. 4.3 Meeting the accuracy objective For the simulations described in IV.B, we compute the difference between the accuracy objective (γ, θ) and the (experienced) estimation error (c, θ), as a measure of how well the accuracy objective is achieved. It is computed as: t o +T 1 c = min {} k min diff ( t, d, k) t θ, (4) T d to 1 A( t) A ~ ( t + d) k diff ( t, d, k) =, (5) 0 else where A stands for the actual aggregate and à for the estimation provided by A- GAP at the root node during the measurement period T. For our experiments, d has an upper bound of 50 msec. 8

45 40 Estimation Error c 35 30 25 20 15 10 5 A-GAP Ideal 0 0 5 10 15 20 25 30 35 40 Error Objective Fig. 5. Estimation error c (eq. 4) as a function of the accuracy objective γ (eq. 2) set by the manager at invocation of A-GAP. The figure shows how well the objective γ is achieved. (Abovenet topology, θ=0.05) Figure 5 shows the results for the Abovenet topology and Figure 6 those corresponding to the grid topology. We observe that the difference between the estimation error and the error objective is small. The difference between γ and c in figures 5 and 6 has two main causes. First, updates from different nodes in the network experience different delays in reaching the root, which distorts the evolution of the estimate at the root node. (This distortion is not captured by our stochastic model, since, for reasons of simplicity, it does not consider networking and processing delays). A second cause is the inaccuracy in the stochastic model variables used for filter computation, for instance, as a result of errors in the estimation of the evolution of the local variables. We note again that the difference between γ and c is small, despite these potential sources of error. For a given scenario, the minimum estimation error c min is achieved when no updates are filtered. Note that in that case, the difference between γ and c is only caused by the different delays updates experience in reaching the root. When no updates are filtered, the estimation of the model variables has no impact. For the Abovenet topology, c min = 13; and for the grid topology c min = 2. The minimum estimation error c min is larger for the larger topology (the Abovenet). We explain this by the fact that as the size of the network increases, the range of delays experienced by updates (from different nodes) increases. It is feasible to give a bound for the difference between γ and c in real-time, based on the evolution of local variables and the processing/communication delays, for the purpose of tuning the protocol at run-time. Comparing these results with those in [13], we see the same qualitative behavior again. 9

Estimation Error c 22 20 18 16 14 12 10 8 6 4 2 0 A-GAP Ideal 0 2 4 6 8 10 12 14 16 18 20 22 Error Objective Fig. 6. Estimation error c (eq. 4) as a function of the accuracy objective γ (eq. 2) set by the manager at invocation of A-GAP. The figure shows how well the objective γ is achieved. (Grid topology, θ=0.05) 5 Related Work A significant amount of research effort has been put on controlling the fundamental trade off between overhead and accuracy in the context of computer networks. To the best of our knowledge, all the solutions proposed to date only support one type of accuracy objectives, the maximum error [6][7][10][12]. These works focus on guaranteeing that the difference between the estimation and the actual aggregate is always within a configurable range. In contrast, our solution supports different types of accuracy objectives. A-GAP enables us to use any metric that can be computed from an error distribution. The works in [10] and [6] show that, while guaranteeing a maximum error is relatively simple, this is a loose upper bound on the accuracy achieved. This is in line with our results, and speaks for using other types of accuracy objectives. Note that our protocol also permits to provide information about the distribution of experienced error (including the maximum error), while the other schemes cannot provide any estimation of the experienced error. In [6], the authors propose a centralized design, where all nodes report directly to the management station, which aggregates the updates. In contrast, we use incremental aggregation, which (i) distributes the cost of computing the aggregate among all the nodes, and (ii) permits positive and negative errors to cancel each other out, which permits reducing the overhead. The work presented in [7] differs from our work in two key aspects. First, it uses synchronized rounds of operation. In contrast, A-GAP is asynchronous. The motivation for this is the achievement of robustness and scalability. Second, the algorithm in [7] estimates statistics from all the partial aggregates, which depend on the filters. A-GAP only estimates the local variables, which are not affected by filters. All other variables are continuously computed by A-GAP based on these estimates. This gives A-GAP more flexibility in selecting the duration of the control cycle. 10

The overhead vs QoE performance achieved by [7], [10], and [12] resembles a negative exponential function for the errors considered, as is the case for A-GAP. 6 Discussion In this paper, we have demonstrated the versatility of our protocol A-GAP for supporting different types of accuracy objectives. Specifically, we have shown that the protocol supports objectives expressed in terms of percentile errors. Previously [13], we had shown its capability for supporting accuracy objectives in terms of the average error. The design of A-GAP allows for supporting any accuracy objective that can be defined as a function of the error distribution. This versatility of A-GAP contrasts with related works, which can only support maximum error objectives, which are easier to control than percentiles and the average error. Based on our own experiences and those reported in [10] and [11], we argue that the average error and percentiles are more significant control parameters for practical scenarios. The protocol dynamically adapts to changes in network topology and to node failures [13], as well as to changes in the statistics that are computed from the local management variables. We consider management protocols that exhibit this kind of autonomic behavior essential for future communication systems. A-GAP continuously estimates the evolution of the management variables that the protocol aggregates. Based on these estimates, all others variables such as the error distributions and incurred overhead are dynamically computed. This approach makes A-GAP adaptive in the sense that changes in the model variables, such as the evolution of the partial aggregate, are computed instead of estimated and, therefore, adapt much faster. While we use SUM as aggregation function throughout this paper, additional aggregation functions, including AVERAGE, MIN, and MAX can be supported by the protocol with straightforward modifications, as discussed in [13]. In general, aggregation functions which are composed of functions that are both commutative and associative can be supported in A-GAP. Currently, we are investigating the quantitative benefits of decentralizing network monitoring. As for future work, we plan to investigate more efficient solutions to the local problem in (eq. 3). We also plan to include support for percentile error objectives in our A-GAP prototype [14]. (Currently, it only supports average error objectives.). This requires only minor modifications to the prototype. Acknowledgments. The authors would like to thank György Dán and Mads Dam at KTH for fruitful discussions around the stochastic model and the design of A-GAP. This work uses network traces collected by the DACS group of the University of Twente for the protocol evaluation. This paper was supported in part by the EC IST- EMANICS Network of Excellence (#26854). It describes work undertaken in the context of the Ambient Networks IST project, which is partially funded by the Commission of the European Union. 11

References 1. M. Dam, R. Stadler, A Generic Protocol for Network State Aggregation, Radiovetenskap och Kommunication (RVK), Linkoping, Sweden, June 2005. 2. S. Dolev, A. Israeli, and S. Moran, Self-Stabilization of Dynamic Systems Assuming only read/write Atomicity, Distributed Computing, 7:3-16, 1993. 3. K. Lim, R. Stadler, A Navigation Pattern for Scalable Internet Management, IEEE/IFIP IM 2001, Seattle, USA, May 2001. 4. K. Lim and R. Stadler. SIMPSON a SIMple Pattern Simulator for Networks. http://www.s3.kth.se/lcn/software/simpson.shtml, July 2006. 5. N. Spring, R. Mahajan, and D. Wetherall, Measuring ISP topologies with Rocketfuel, ACM/SIGCOMM, Pittsburgh, USA, August 2002. 6. C. Olston and J. Widom, Efficient Monitoring and Querying of Distributed, Dynamic Data via Approximate Replication, IEEE Data Engineering Bulletin, March 2005. 7. Antonios Deligiannakis, Yannis Kotidis, and Nick Roussopoulos, Hierarchical In- Network Data Aggregation with Quality Guarantees, EDBT 2004, Crete, Greece, March 2004. 8. IEEE. ANSI/IEEE Std 802.1D, 1998 Edition. IEEE, 1998. 9. R. van de Meent, A. Pras, Traffic Measurement Data Repository, University of Twente, http://traffic-repository.ewi.utwente.nl/ 10. M. A. Sharaf et al, Balancing energy efficiency and quality of aggregate data in sensor networks, ACM International Journal on Very Large Data Bases, 13(4):384 403, December 2004 11. A. Boulis et al., Aggregation in sensor networks: an energy - accuracy tradeoff, Elsevier Ad-hoc Networks Journal (special issue on sensor network protocols and applications), pages 317 331, 2003. 12. C. Intanagonwiwat et al., Impact of network density on data aggregation in wireless sensor networks, 22nd International Conference on Distributed Computing Systems, Vienna, Austria, July 2002. 13. A. Gonzalez Prieto, R.Stadler, A-GAP: An Adaptive Protocol for Continuous Network Monitoring with Accuracy Objectives, to appear in IEEE Transactions on Network and Service Management. Available at: http://www.ee.kth.se/~gonzalez 14. A. Gonzalez Prieto and R.Stadler "Implementation and Evaluation of A-GAP: Adaptive Monitoring with Controllable Accuracy", submitted to the ACM SIGCOMM workshop on Internet Network Management (INM 07), Kyoto, Japan, August 31, 2007. Available at: http://www.ee.kth.se/~gonzalez 12