Syslog Performance: Data Modeling and Transport

Transcription

1 Syslog Performance: Data Modeling and Transport Mohammad Rajiullah, Reine Lundin, Anna Brunstrom, and Stefan Lindskog Department of Computer Science, Karlstad University SE Karlstad, Sweden {mohammad.rajiullah reine.lundin anna.brunstrom Abstract Syslog is one of the basic methods for event logging in computer networks. Log messages that are generated by syslog can be used for a number of purposes, including optimizing system performance, system auditing, and investigating malicious activities in a computer network. Considering all these attractive uses, both timeliness and reliability is needed when syslog messages are transported over a network. The unreliable transport protocol UDP was specified in the original syslog specification; later a reliable transport service based on TCP was also proposed. However, TCP is a costly alternative in terms of delay. In our previous work, we introduced the partially reliable extension of, PR-, as a transport service for syslog, trading reliability against timeliness by prioritizing syslog messages. In this work, we first model syslog data using real syslog traces from an operational network. The model is then used as input in the performance evaluation of PR-. In the experiments, real congestion is introduced in the network by running several competing flows. Although PR- clearly outperformed TCP and in our previous work, our present evaluations show that PR- performance is largely influenced by the syslog data size characteristics. I. INTRODUCTION We are now probably observing the highest growth of computers attached to networks both in numbers and sizes since the inception of the Internet. There is a continuous deployment of network servers, workstations, routers or other networking devices. As a result, the need for security and reliability in a computer network is more important than ever. Log messages, generated in a computer network, can be a significant element for providing increased security and reliability. An event log is a record of a specific event that occurred in the system. Logs are generated from various sources, such as firewalls, switches, routers, wireless access points, operating systems or applications including intrusion detection systems and anti-malware programs. A system administrator can use these logs for trouble shooting purposes. Nowadays, logs have become vital elements for many other purposes, e.g., optimizing system performance, recording user actions, supporting system wide forensic analysis of malicious activities, etc. Taking all these attractive uses into account, reliable, secure, and timely delivery of log messages over the network becomes very important. The syslog protocol [] has been proposed and used for basic log management in computer networks since the early days of Internet. Simply speaking, syslog allows a machine or device to send event notification messages over networks to any predefined collector. This collector is called the syslog server. The User Datagram Protocol (UDP) [2] was specified as a transport service for syslog messages in the earliest specification of the syslog protocol [] when a centralized log management approach was used. Since UDP is an unreliable transport protocol, log messages containing critical alerts may be lost without notice. RFC 395 [3] proposed the connection oriented protocol, Transmission Control Protocol (TCP) [5] as a transport service to provide reliability in message delivery. The widely used syslog-ng system [4] uses TCP as the default underlying transport service. However, TCP causes high delays during loss recovery, which may not always be affordable due to the need of a timely reporting of critical events. A detailed description of potential drawbacks of UDP and TCP as transport services for syslog messages can be found in our previous work [8]. In addition to TCP, secure syslog transport over the Transport Layer Security (TLS) has been standardized in RFC 5425 [6], and a Datagram TLS (DTLS) secure transport has recently been specified in RFC 62 [7]. Log messages are inherently prioritized. During loss recovery, some messages are more important than others. Additionally, some messages have certain lifetimes after which they are no longer useful. For example, various applications may generate periodic status messages which are normally useful only for a short time period. The concept of prioritization becomes important during congestion periods allowing network resources to be prioritized for transporting the most crucial and fresh messages. In our previous work [8], we proposed and evaluated the use of the existing and standardized partially reliable extension of the Stream Control Transmission Protocol () [9], PR- [], as an underlying transport service for timely and reliable delivery of syslog messages over a network. PR- uses all the advanced transport related features in and uniquely supports (re)transmission policies on a per message basis. Since each syslog message is characteristically prioritized, according to the source that generates it and its importance level, PR- was used to provide different reliability policies for different syslog messages according to their priority levels. By using PR-, we in [8] saw a significant improvement of the average transfer delay for syslog messages as compared to TCP. Since PR- prioritizes more important messages over the rest, some messages that are lost and not important may never be recovered. The loss of a message is, however, notified to the receiver. Basically, by trading reliability against timeliness during loss recovery, time

2 and network capacity can be saved for important log messages. The performance of PR- was evaluated in an artificial loss based simplistic network scenario. However, in the present Internet, packet losses mainly occur due to congestion when a number of flows compete for resources. Moreover, in the referenced work, a synthetic work load of fixed message sizes was used. In this work, we first analyze a syslog trace file from a real operational network. Based on the trace, we determine several important parameters, such as message size distribution and arrival pattern. Besides, we introduce a shared network in the evaluation where PR- must compete with other flows for network resources. In the experiment results, we can see a large influence of message size distribution on the PR- performance. The remainder of this paper is organized as follows. Background and related work are described in Section II. In Section III, the characteristics of syslog data using real syslog traces are modeled. In Section IV, the performance of PR- in a shared network for syslog transport is evaluated. Finally, some concluding remarks are given in Section V. II. BACKGROUND In this section, we briefly describe the syslog protocol as well as PR- as a transport service for syslog messages. A. Syslog In UNIX like operating systems, kernels and/or applications generate several messages and alerts that are either saved locally or sent over the network as syslog messages. According to the BSD syslog specification [], an originator that creates syslog messages may forward them to a collector or syslog server. In addition, one or more devices may be used as relays between an originator and a collector. Various communication devices, such as routers, switches, and VPN connectors as well as operating systems and applications may work as originators for generating syslog messages for system related information or alerts. For instance, an alert may be generated when a router finds one of its interfaces is down. Besides, firewalls can generate syslog messages when it blocks a TCP or connection. All of these devices may be configured in such a way that they send specified syslog messages to external devices that works as a centralized syslog server and collects all the received messages for further analysis. This allows an administrator simply log in to the syslog server and monitor the whole system under surveillance rather than separately to go through a large number of devices. The most current specification of the syslog protocol was standardized in March 29 in RFC 5424 []. This specification uses a layered architecture where the specification of the message structure is separated from the transport of the message. A syslog message that can be seen on the wire as a payload normally has three distinguishable parts. The first part is called the PRI. This part specifies the priority of a syslog message according to the associated facility and severity values of the corresponding event that generates this message. Common examples of facilities are operating system daemons or processes that generate log messages. Each different type of facility is assigned a decimal number. For example, a facility value of 4 is given to all log messages generated from security or authorization related sources. Each facility that generates a log message also specifies a severity level for it. For example, a severity level of signifies an emergency condition when a system is unstable, whereas a severity level of 7 simply signifies a debug level message. To calculate the priority value of a message, its facility value is first multiplied by 8 and then added to the severity value. For example, a facility value of 5 (clock daemon) and a severity value of 5 (notice) will produce a priority value of 25. We use the severity value from PRI to prioritize a syslog message. Messages with severity values from to 3, which denote emergency, alert, critical, and error level, are marked as important messages and the rest as normal messages. The second part of the message contains a hostname or the IP address from which the message is transmitted. This part also includes a timestamp when the message is generated. The third part, finally, is the content of the syslog message which often contains information about the process that generates the message. One of the fundamental ideas with the syslog protocol is the simplicity of implementation. Any device that is configured to send syslog messages does so even without any explicit knowledge about the receiver. A syslog server, on the other hand, cannot request a specific device to send syslog messages. This operational simplicity has greatly helped for the wide deployment of the syslog protocol. Additionally, syslog is a simplex protocol. It does not give any acknowledgement from the application layer for the message deliveries. A separate, reliable transport may provide this functionality at the transport layer. B. PR- as a Syslog Transport PR- is an extension of and accommodates all of its features. We include a brief description of some of these features below. is a connection oriented protocol like TCP. is message oriented whereas TCP is byte oriented. If an application sends messages that are sufficiently smaller than the path maximum transmission unit (MTU), a number of these small messages can be bundled into a single packet. A message or control information is referred to as a chunk when it is put inside a packet as payload. Besides, inside a packet, control information can be bundled together with user messages. This message based abstraction is particularly suitable for the syslog protocol, since syslog messages are semantically independent. can be configured to multiplex segments over multiple independent streams to reduce TCP s Head of Line (HOL) blocking problem [2]. Moreover, if unordered delivery is used, a receiver can immediately deliver a received message to the upper layer even if the arrival of the message is unordered. This is also useful for independent syslog messages.

3 Syslog clients may generate an unlimited amount of messages. Using UDP as a transport protocol for these syslog messages may not be appropriate if the network is not properly managed, since UDP does not have any congestion control. TCP or plain, on the other hand, may increase the message transfer delay during congestion periods as they are fully reliable. According to RFC 5424 [], message-prioritization based on the severity values should be considered when syslog messages need to be discarded; further sending will otherwise be blocked. PR- may thus be an appropriate transport service to implement such an idea. PR- uses a (re)transmission policy on a per message basis. The idea is that PR- considers a particular application specific reliability policy for every message. Timed reliability is an example of an application specific reliability policy. In such a policy, a certain lifetime is given for every application message. Upon expiration of this lifetime, PR- does not consider any (re)transmission of the message. For syslog messages, a lifetime can be determined according to the severity level of each message. If messages are lost due to congestion, only those messages for which the lifetime has not yet expired will be retransmitted. The advantages are that overall message transfer time will be improved and a timely delivery of important messages is possible. In this way, PR- can improve the system performance. PR- supports prioritization by using a special control chunk named FORWARD CUMULATIVE TSN or forward tsn. The idea is as follows. When a sender abandons a message to (re)transmit, it sends a forward tsn chunk to the receiver. This chunk simply tells the receiver to forward its cumulative ACK point, which means that the receiver must not expect the abandoned message anymore. Besides, PR- adjusts all the congestion related variables even though it may not retransmit any lost messages. Note that a PR- receiver is always unaware of any particular prioritization policy taken at the sender. Several existing works [3 8] have used PR- for various applications such as real time multimedia streaming, IPTV transmission, SIP signaling, etc. III. SYSLOG DATA In this section, the distribution of message length, interarrival time and important messages from a syslog database sample are investigated and modeled. The sample was captured at the syslog server at the Computer Science Department network at Karlstad University, Sweden between August 28 and December 2, and consists of more than 3.5 million syslog messages. A. Message Length Distribution A syslog message normally consists of the priority, header and content. However, the format of the stored syslog records in the database do not reveal if headers were included or not in the sent syslog messages. By using Wireshark to capture syslog messages on the network for a one day session, it was observed that the header was included in some messages, but not in all. The average length of the priority and header part was calculated from the Wireshark session to be 3.5 bytes Content length [bytes] Fig. : The content length density distribution. The content length distribution from the syslog database sample is plotted in Fig.. From the pattern in the figure it seems reasonable to model the behavior of the content length distribution with a normal distribution. The normal distribution is given by the bell-shaped probability density function f(x) = (x µ)2 e 2σ 2 () 2πσ 2 where µ is the mean and σ is the standard deviation of the distribution. Using maximum likelihood estimation the mean and standard deviation of () can be estimated with the mean and standard deviation of the content length distribution. This gives µ = 7.5 bytes and σ = 3.2 bytes. After adding the priority and header part of 3.5 bytes, the total mean becomes 85 bytes for the message length. To show how the normal distribution fits to the content length distribution, the corresponding two cumulative distributions are plotted in Fig. 2. The maximum error between the two cumulative distributions is 23.4 % and the mean error is. %. B. Inter-arrival Time Distribution The inter-arrival time distribution for the syslog database sample is plotted in Fig. 3. However, since the timestamp granularity in the syslog database is of the time scale of seconds, a smaller plot of the Wireshark sample, having a time scale of microseconds, is also given in the figure. In Fig. 3, the leftmost spike for the syslog database sample reaches up to 76.9 % and the corresponding spike for the Wireshark sample reaches up to 4.2 %. The other spikes in the graphs are probably generated by the applications that send syslog messages on a regular basis. From the pattern of the plots in Fig. 3, it seems reasonable to model the behavior of the inter-arrival time distribution with an

4 Content length [bytes] Fig. 2: The normal and content length cumulative distributions Time [s] Time [s] Fig. 3: The inter-arrival time density distribution for the whole sample (big graph) and the Wireshark sample (small graph). exponential distribution. The exponential distribution is given by the probability density function f(x) = λe λx (2) where λ > is the rate parameter of the distribution. Using maximum likelihood estimation, the rate parameter of (2) can be estimated with the inverse of the mean of the interarrival time distribution. Thus, for the Wireshark sample, having the finest time granularity, the mean of the inter-arrival time distribution is.73 s, giving λ =.36 s. To show how the exponential distribution fits to the inter-arrival time distribution, the corresponding two cumulative distributions are plotted in Fig. 4. The maximum error between the two cumulative distributions is 9.7 % and the mean error is. % Time [s] Fig. 4: The exponential and interarrival time cumulative distributions. C. Important Message Distribution The number of important and normal messages per day for our syslog database sample is plotted in Fig. 5. From this sample, the fraction of important messages is 7.6 %. In [2], the authors used two syslog database samples. The first sample was captured during heavy loads and contained % of important messages, while the second sample was captured during light loads and contained.4 %. Hence, the authors used in their experiments the assumption that during heavy loads the fraction of important messages increases. We examine the spikes in our database sample for similar trends. During heavy loads, the largest load spike in Fig. 5 has a fraction of 5.3 %, at day 255, while the second largest load spike has a fraction of 2.9 %, at day 87. Thus, in our system we cannot draw the conclusion that heavier loads increase the fraction of important messages, but consistent with [2] we can see that the fraction of important messages may vary greatly. IV. EXPERIMENTAL EVALUATION In this section, we first describe the experiment setup that we use to evaluate the performance of PR- for transporting syslog messages, followed by our experiment results. A. Experiment Setup We adopt a single bottleneck, emulation based experiment for the performance evaluations. We use three computers. All of these three machines have the same hardware configuration of 4 GB RAM and an Intel Core 2 duo processor (2.6 GHz). The Dummynet traffic shaper [9] is set up in the middle machine to introduce physical delays and bandwidth limitations in the network. Both end machines are configured with FreeBSD 8.. Based on the findings in Section III, we have created a syslog message generator application. The lengths of the generated messages are drawn from a normal

5 865 normal messages. Additionally, we perform each experiment with 3 repetitions to allow for 95 % confidence intervals. The network related parameters are summarized in Table. 855 TABLE I: Network related parameters. Number of messages Normal messages Important messages Parameter Value(s) One way Delay: ms Queue size: 38 KB Bandwidth: Mbps (Up and Down) # of background flows: 5 and Time [day] Fig. 5: Fraction of important and normal messages per day. distribution of mean 85 and standard deviation 3. Besides, since we see an exponential distribution of the inter-arrival times of syslog messages in the traces, our application creates messages according to a Poisson arrival process. We do not, however, use the obtained mean for inter-arrival times in the experiment, since we are in this case interested to see how the evaluated protocols behave during congestion. To avoid having to use an excessive amount of background flows or a very small bandwidth to create congestion, we instead use a shorter mean inter-arrival time of 2 ms. Our application generates two main types of messages: important and normal. According to our analysis in section III, we use several distributions of important messages starting from % to 25 %. In this experiment, we use a timed reliability based PR- policy. In such a policy, a certain lifetime is given for every application message. Upon expiration of this lifetime, PR- does not consider any (re)transmission of this message. Our expectation is that even under heavy congestion in the network, important messages should reach the receiver whereas normal messages have a smaller or no chance. We use a time-to-live (TTL) value of 5, ms for important messages and a TTL of 2 ms for normal messages. We use the same application settings on top of various transport services such as TCP,, and PR-. Both and PR- use unordered delivery, since syslog generates semantically independent messages. Additionally, in each run, syslog messages are sent from the server to the client. In the experiment, we have several competing background flows in the network. These are greedy TCP flows. In consequence, a foreground flow being any of the evaluated protocols, TCP,, or PR-, must compete with the background flows for network resources. We vary the number of background flows to differ the congestion level in the network. We measure average message delay for both important and B. Experiment Results Fig. 6 shows the performance of PR- along with and TCP for different number of background flows. Since the PR- graphs for important and normal messages visually overlap, we only put the graphs for important messages. In both cases, although PR- performs better than TCP for smaller fractions of important messages, it surprisingly performs worse than. PR- is expected to outperform the fully reliable, since it can ignore retransmission of normal messages if they are lost. Two factors have been changed from our previous work; the network settings and the distribution of message sizes. Hence, we perform another experiment using the same synthetic workload that we used before in our present network settings to isolate the phenomena that is influencing the PR- results most. We use a fixed message size of 25 bytes and keep the same average send rate as we used in the first experiment. In addition, we keep the number of background flows to 2. The results from this experiment are given in Fig. 7. In this figure, a subgraph is included to distinguish the results of PR- from. Here we see that PR- outperforms both TCP and by prioritizing important messages over normal messages during (re)transmission. This shows that the message size distribution has a major influence on the PR- results. Based on a detailed analysis of the trace files from our experiments, we have identified that during loss recovery, the existing forward tsn mechanism becomes inefficient. For our scenario where the message size distribution is normal and the mean is not very large, we identify two cases. Firstly, as each message can be of different size and can be placed alone in a packet, the messages have different loss probabilities when the byte based queue at the network is full; in consequence, this can produce a non consecutive loss pattern. In this case, the forward tsn mechanism cannot tell a receiver to forward the cumulative TSN across several TSNs or messages. Instead, it does it separately for individual lost messages, since the delivered message(s) must be cumulatively acknowledged before the forward tsn for the next lost message can be transmitted. On the contrary, fully reliable can retransmit multiple messages together, and becomes faster. Secondly, if there is a backlog created in the send buffer at the transport layer, many messages can be bundled into a single packet, since the mean of the message size distribution

6 Average delay [s] TCP Average delay [s].5.5 TCP Average delay [sec] (a) 5 background flows Fig. 7: Average delay performance using a fixed message size of 25 bytes with 2 background flows. Average delay [s] TCP (b) 2 background flows Fig. 6: Average delay performance for different important message distributions. is quite small. As a result, when even a single packet is lost, many consecutive messages are lost. In such a case, when a single packet is lost, and if many messages are bundled into that packet, we loose many messages with several importance levels. Thus, if all the messages are normal, then a forward tsn chunk can tell a receiver to forward the cumulative ACK point across all these messages. As a result, PR- can be faster than by ignoring to retransmit messages. However, when we have a mixture of reliable and unreliable messages, the forward tsn mechanism becomes slow. This is due to the fact that it only sends a forward tsn chunk to the receiver for an unreliable lost message when the preceding reliable message is confirmed to be delivered. Hence, PR- suffers from a considerable delay. This is also true when we have a fixed message size of 25 bytes; however, the resulting delay is limited, since at most five messages can be bundled into a single packet. V. CONCLUSION In this paper, we first model the characteristics of syslog data using real traces from an operational network. Then, we investigate the performance of PR- for transporting syslog messages using the derived model in a network scenario where multiple flows compete for network resources. However, unlike what we have shown in our previous work [8], PR- noticeably performs worse than. After a detailed analysis of the traces from our experiments, we have identified that the existing forward tsn mechanism in PR- becomes inefficient if an application generates message sizes with small mean according to a normal distribution. We are currently working on improving the forward tsn mechanism and aim to implement and evaluate our solution. Besides, we are planning to obtain syslog traces from a larger network to further understand the characteristics of syslog data. ACKNOWLEDGMENT The work was carried out within the Compare Business Innovation Centre phase 2 project, funded partly by the European Regional Development Fund. REFERENCES [] J. Postel, RFC 364: The BSD Syslog protocol, August, 2. [2] J. Postel, RFC 768: User Datagram Protocol (UDP), August, 98. [3] D. New and M. Rose, RFC 395:Reliable Delivery for syslog, November, 2. [4] Syslog New Generation (Syslog-ng). network-security/syslog-ng/, visited February 27, 2. [5] J. Postel, RFC 793: Transmission control protocol, September, 98. [6] F. Miao et al., RFC 5425: Transport Layer Security (TLS) Transport Mapping for Syslog, March 29. [7] J. Salowey et al., RFC 62: Datagram Transport Layer Security (DTLS) Transport Mapping for Syslog, October, 2.

7 [8] M. Rajiullah, A. Brunstrom, and S. Lindskog, Priority Based Delivery of PR- Messages in a Syslog Context, in International Workshop on Autonomic Networking and Self-Management in the Access Networks (SELFMAGICNETS 2), 2. [9] R. Stewart, Q. Xie, and K. Morneault, RFC 496: Stream control transmission protocol, September, 27. [] R. Stewart, M. Ramalho, Q. Xie, and M. Tuexen, RFC 3758: Stream Control Transmission Protocol () Partial Reliability Extension, may, 24. [] R. Gerhards et al., RFC 5424: The syslog Protocol, March, 29. [2] G. D. Marco et al., as a transport for SIP: a case study, In 7th World Multiconference on Systemics, Cybernetics and Informatics (SCI), pp , Orlando, FL, USA, July, 23. [3] T. Maeda, M. Kozuka, and Y. Okabe, Reliable Streaming Transmission Using PR-, in Ninth Annual International Symposium on Applications and the Internet, SAINT 9, pp , IEEE, 29. [4] H. Sanson, A. Neira, L. Loyola, and M. Matsumoto, PR- for real time H. 264/AVC video streaming, in The 2th International Conference on Advanced Communication Technology (ICACT), vol., pp , IEEE, 2. [5] H. Wang, Y. Jin, W. Wang, J. Ma, and D. Zhang, The performance comparison of PR-, TCP and UDP for MPEG-4 multimedia traffic in mobile network, in International Conference on Communication Technology Proceedings, ICCT, vol., pp , IEEE, 23. [6] M. Molteni and M. Villari, Using with partial reliability for MPEG-4 multimedia streaming, in European BSD Conference, 22. [7] S. Kim, S. Koh, and Y. Kim, Performance of for IPTV Applications, in The 9th International Conference on Advanced Communication Technology, vol. 3, pp , IEEE, 27. [8] X. Wang and V. Leung, Applying PR- to transport SIP traffic, in Global Telecommunications Conference, GLOBECOM 5, vol. 2, pp. 5 78, IEEE, 26. [9] L. Rizzo, Dummynet: a simple approach to the evaluation of network protocols, ACM SIGCOMM Computer Communication Review, vol. 27, no., pp. 3 4, 997. [2] H. Tsunoda et el., A Prioritized Retransmission Mechanism for Reliable and Efficient Delivery of Syslog Messages, in Proceedings of Seventh Annual Communication and Services Research Conference, pp , Washington, DC, USA, 29.