NetScaler Logging Facilities www.citrix.com
Table of Contents Overview...3 SNMP Traps...3 SNMP Polling...3 Syslog and Audit Server...3 NetScaler Web Logging...4 Historical Reporting...5 Performance Record Logging...5 Page 2
Overview To integrate well within most infrastructures, it is necessary for load balancing systems to not only provide the traffic management functionality to control traffic in flexible ways, but it is also important to be able to log what occurs for problem resolution and debugging. The Citrix NetScaler system provides a rich set of logging interfaces that can be used under various situations to meet the needs of a variety of customers, and under different traffic conditions. These logging interfaces include: SNMP Traps SNMP Polling Syslog Audit Server NetScaler Web Logging Historical Reporting Performance record logging SNMP Traps Through the SNMP protocol, alerts can be configured to be pushed to external management systems for a wide variety of events that occur on the system, some of which report operationally varying information such as the CPU usage going too high, and others for platform issues, such as high temperature. This information can be logged on standard SNMP management stations and used for post-mortum analysis. Commonly configured and monitored traps include CPU load going above 90 percent, or bandwidth utilization. As a general rule, SNMP traps are designed to trigger on events that may require immediate action, and can be configured based on customer needs. For more details on the available SNMP alerts, see the NetScaler documentation. SNMP Polling As with most vendors, NetScaler provides the ability to gather statistics for monitoring with third party management applications using SNMP polling. As part of this ability, the NetScaler provides a diverse set of SNMP Object Identifiers (OID s) that cover a wide range of functions that may need monitoring. For a complete list of SNMP OID s available for each release of code, see the NetScaler documentation provided with each software build. Syslog and Audit Server Syslog is another commonly used protocol for providing log information to external sources, and is supported by the NetScaler. In terms of the information it provides, Syslog can be thought of as a superset of what SNMP alerts can provide because Syslog logs all SNMP alerts that are generated and other information that does not fit well with SNMP alerts, including: TCP connection logging SSL VPN User logging Configuration Command Auditing Kernel messages and errors For a complete list of NetScaler internally generated syslog messages, see the NetScaler syslog message documentation, provided with each build. While Syslog is a generic protocol that provides for the ability to log generic messages, the NetScaler Audit server is designed to provide an alternative higher performance interface for the same information. The Audit Server operates as a separate process residing on an external machine or set of machines, and communicates with one or more Page 3
NetScalers to receive the information that otherwise goes to Syslog. Audit server not only allows for a more optimized channel of communication between the NetScaler and the log server, but also offloads the work of formatting the log data from the NetScaler itself. As a result, when logging large volumes of data, such as with TCP connection logging, the Audit server can provide vast improvements in performance. As an example, under heavy TCP connection logging, with Audit Server, the NetScaler has been tested to handle as much as 2.5 times the number of TCP connections per second versus with direct Syslog logging. In most environments, the logging level does not impacting performance to such a degree that this becomes a problem, but if needed, Audit server provides for a scalable option. When needed, the Audit server is available for operating systems including Windows, Linux, FreeBSD, OS X, Solaris (SPARC), and AIX. NetScaler Web Logging The NetScaler Web Logging functionality is similar to the Audit server, where the NetScaler communicates with one or more external NetScaler Web Logging agents, which then format and write logs in standard W3C or NCSA formats. There are several benefits to this approach to logging: 1) When using a farm of servers, the Web logging function provides for a centralized aggregated view of the logs, which allows easier real time analysis of the data. 2) When caching is involved, the NetScaler can report the transactions that never make it to the back-end server, providing a complete view of the data, not just the non-cacheable content. 3) With compression active, the server side and client side view of the object size is available, so that actual compression ratios can be computed. 4) By having logs off of the server, if a server is compromised, it is much more difficult for someone to cover their tracks if they gain access over HTTP. Like the Audit Server, the Web logging client is available for most common server operating systems. To assist with reliability, the NetScaler Web logging agent can run on multiple systems, and to make aggregation of logs easier. You can also insert transaction IDs into each log entry, which is an internal ID the NetScaler uses for tracking individual HTTP requests and responses. This allows for easy and reliable log aggregation. Page 4
Historical Reporting To assist users that might not have a centralized SNMP monitoring station or for debugging issues that require tracking of data that normally does not require tracking, the NetScaler provides a mechanism for creating historical reports on the system itself that allows for reports to be created with various variables, and then tracked over time. As an example, a chart can be created that shows the CPU utilization versus memory usage, and HTTP requests per second: To build the reports, a wide variety of information is available to select from, and once a report is built, the data is gathered in real time for the user, or to view as needed. Performance Record Logging In addition to the logging that is accessible externally, a final logging mechanism is performed on the NetScaler; the logging of performance records. Every seven seconds, a snapshot of internal variables is made called a performance record. This record is logged on the NetScaler itself and stored in a file called newnslog. The information in this log provides a comprehensive view of the performance of the NetScaler, including variables that were not anticipated to be needed, and so is an excellent source of information for post-mortum analysis after an unexpected event. To provide an example of the number of variables and depth of information provided, the next few pages contain a sample of the output to show the variables tracked just for interface counters. The counters cover a wide range of functions including load balancing, memory allocation, compression, SSL behaviors, TCP retransmits, and so on. In all, thousands of counters are tracked even on an unconfigured system, and hundreds more for every object that is added to the system. No other vendor on the market tracks this type of data to this depth by default on this type of device. Page 5
root@test# nsconmsg -g nic_ -d stats Displaying current counter value information Performance Data Record Version 2.0 reltime:mili second between two records Tue Dec 1 05:20:10 2009 Index reltime counter-value symbol-name&device-no 0 0 490643 nic_tot_rx_packets interface(lo/1) 1 0 1097728302 nic_tot_rx_packets interface(0/1) 2 0 55643421 nic_tot_rx_bytes interface(lo/1) 3 0 370945769548 nic_tot_rx_bytes interface(0/1) 4 0 13213367 nic_tot_tx_packets interface(lo/1) 5 0 14435093 nic_tot_tx_packets interface(0/1) 6 0 1778458280 nic_tot_tx_bytes interface(lo/1) 7 0 1842469654 nic_tot_tx_bytes interface(0/1) 8 0 539 nic_tot_rx_mbits interface(lo/1) 9 0 3178329 nic_tot_rx_mbits interface(0/1) 10 0 16764 nic_tot_tx_mbits interface(lo/1) 11 0 17511 nic_tot_tx_mbits interface(0/1) 12 0 0 nic_err_rx_nobufs 13 0 0 nic_err_rx_length 14 0 0 nic_err_rx_fifo 15 0 0 nic_err_rx_long_frame 16 0 0 nic_err_rx_crc 17 0 0 nic_err_rx_alignment 18 0 0 nic_err_rx_fcs 19 0 0 nic_err_rx_missed 20 0 0 nic_err_tx_excess_collisions 21 0 0 nic_err_tx_late_collisions 22 0 0 nic_err_tx_carrier 23 0 0 nic_err_tx_fifo 24 0 0 nic_err_tx_collisions 25 0 0 nic_err_tx_1_collision_frame 26 0 0 nic_err_tx_multi_collision_frame 27 0 0 nic_err_tx_deferred 28 0 0 nic_err_tx_heartbeat 29 0 0 nic_err_tx_overflow 30 0 219589374 nic_err_dropped_pkts interface(0/1) 31 0 806094124 nic_tot_broadcast_pkts interface(0/1) 32 0 274148713 nic_tot_multicast_pkts interface(0/1) 33 0 490643 nic_tot_netscaler_pkts interface(lo/1) 34 0 12010288 nic_tot_netscaler_pkts interface(0/1) 35 0 0 nic_tot_bsd_packets 36 0 0 nic_err_rx_macattn 37 0 0 nic_err_rx 38 0 1581 nic_err_tx interface(lo/1) 39 0 0 nic_err_tx_nonucpkt 40 0 0 nic_err_tx_dropped Page 6
41 0 0 nic_err_rx_norxbds 42 0 00:0a:5e:7a:85:d8 nic_cur_mac_addr interface(lo/1) 43 0 00:0a:5e:64:83:7f nic_cur_mac_addr interface(1/3) 44 0 00:0a:5e:7a:8c:19 nic_cur_mac_addr interface(1/4) 45 0 00:0a:5e:7a:85:d8 nic_cur_mac_addr interface(1/1) 46 0 00:0a:5e:7a:85:0d nic_cur_mac_addr interface(1/2) 47 0 00:30:48:5a:de:07 nic_cur_mac_addr interface(0/1) 48 0 0 nic_cur_txqlen 49 0 0 nic_cur_txpkts_active 50 0 UP nic_cur_link_state interface(lo/1) 51 0 UP nic_cur_link_state interface(0/1) 52 0 129 ( COPPER FULL_DUPLEX 100Mbit NO_FLOWCONTROL ) nic_cur_link_media interface(0/1) 53 0 1384 nic_err_ifindiscards interface(0/1) 54 0 1581 nic_err_ifoutdiscards interface(lo/1) 55 0 0 nic_tot_xonpauseframesreceived 56 0 0 nic_tot_xoffpauseframesreceived 57 0 0 nic_tot_xoffstateentered 58 0 0 nic_tot_flowcontroldone 59 0 0 nic_tot_outxonsent 60 0 0 nic_tot_outxoffsent 61 0 0 nic_err_congested_pkts_dropped 62 0 0 nic_err_congestionlimit_pkts_dropped 63 0 0 nic_err_link_hangs 64 0 3 nic_err_link_reinits interface(1/3) 65 0 3 nic_err_link_reinits interface(1/4) 66 0 3 nic_err_link_reinits interface(1/1) 67 0 3 nic_err_link_reinits interface(1/2) 68 0 2 nic_err_link_reinits interface(0/1) 69 0 0 nic_err_link_sts_stalls 70 0 0 nic_err_link_tx_stalls 71 0 0 nic_err_rx_nonsb 72 0 0 nic_err_duplex_mismatch 73 0 200 ( RX_FLOWCONTROL ) nic_cur_link_media_req interface(0/1) 74 0 4308583 nic_cur_link_uptime interface(lo/1) 75 0 510994 nic_cur_link_uptime interface(0/1) 76 0 4308585 nic_cur_link_downtime interface(1/3) 77 0 4308585 nic_cur_link_downtime interface(1/4) 78 0 4308585 nic_cur_link_downtime interface(1/1) 79 0 4308585 nic_cur_link_downtime interface(1/2) 80 0 32 nic_cur_link_downtime interface(0/1) 81 0 0 nic_err_txpkt_size_fixed 82 0 0 nic_cur_ptxqlen 83 0 0 nic_cur_lptxqlen 84 0 0 nic_err_link_rx_stalls 85 0 0 nic_err_link_errdisables Page 7
86 0 0 nic_tot_rx_lacpdus 87 0 0 nic_tot_tx_lacpdus 88 0 0 nic_tot_bdg_mac_moved 89 0 0 nic_err_bdg_muted 90 0 0 nic_tot_tx_stall_on_xoff_state 91 0 18183 nic_tot_tx_multicast_pkts interface(lo/1) 92 0 112115 nic_tot_tx_multicast_pkts interface(0/1) 93 0 35946 nic_tot_tx_broadcast_pkts interface(lo/1) 94 0 55560 nic_tot_tx_broadcast_pkts interface(0/1) 95 0 0 nic_err_la_frame_collect_drops 96 0 0 nic_err_la_tagged_bpdu_drops 97 0 0 nic_err_vlan_promisc_tag_drops 98 0 0 nic_err_la_untagged_pkt_drops 99 0 0 net_err_disablednic_txpkts 100 0 3178329 allnic_tot_rx_mbits 101 0 17511 allnic_tot_tx_mbits Page 8