Diagnostics and Troubleshooting Using Event Policies and Actions Brocade Network Advisor logs events and alerts generated by managed devices and the management server and presents them through the master log and various other views. Brocade Network Advisor offers a variety of tools and techniques to control which events the management application monitors, on which products events are monitored, how often they are monitored, and what to do when the monitored events are generated. This paper describes the procedure to define and use event action policies in Brocade Network Advisor.
CONTENTS Introduction...3 Use Cases...3 Conditional logging... 3 Preventive actions and remediation... 3 Run diagnostics based on events... 3 Conditional suppression... 3 Threshold monitoring and notification... 3 Configuration...3 Example...9 Problem... 9 Solution... 9 Event Correlation And Event Actions... 10 Summary... 11 Diagnostics and Troubleshooting Using Event Policies and Actions 2 of 12
INTRODUCTION Event action policies allow you to create and enable policies that can be applied to control types of events being logged in the management server, and to define what needs to be done when an event is received in the management server. This tool mainly helps: Control which events are logged Generate smart alerts based on event correlation Trigger various actions when specific policy conditions are met Event policies and other monitoring tools in Brocade Network Advisor provide a set of powerful tools for monitoring and diagnostics. USE CASES Some of the common use cases where event actions can be applied are listed below. Conditional logging You can create an event action definition if you want the management application to monitor link up and link down traps only, and only on products that belong to specific product groups. Furthermore, you might want these traps to be logged in the management application database only if they occur x number of times within a certain interval of time. You might also want an e-mail message sent to a network administrator when these traps are generated. Preventive actions and remediation Brocade Network Advisor allows you to disable a device port if an event that resembles an attack on the network occurs at a certain frequency. Run diagnostics based on events Brocade Network Advisor allows you to run a set of diagnostics checks and reports the findings when a specific event is received in the management server from one or more managed devices. Conditional suppression If you expect certain events to be generated from managed devices during a certain period, automatically acknowledge those events so that they do not flood the logs. Similarly, enable troubleshooting (maintenance) mode on a device for a certain period to suppress the events and control false alarms. Threshold monitoring and notification Brocade Network Advisor allows you to monitor health status, error counters, and performance measures of Brocade switches and routers and notifies you when a specified performance threshold is crossed. For example, an event action might be to monitor CPU utilization for one or more products and send an e-mail notification to the network administrator when utilization crosses a predefined threshold value. Diagnostics and Troubleshooting Using Event Policies and Actions 3 of 12
CONFIGURATION You can configure and enable event policies in Brocade Network Advisor by launching the Event Actions dialogue box. Select Monitor > Event Processing > Event Actions. Figure 1. Event Actions launch menu. Selecting this menu launches the event actions main dialogue box, which lists all available event action policies. The list of policies includes the predefined default policies, as well as policies created by all users. Figure 2. Event Actions main dialogue box. Diagnostics and Troubleshooting Using Event Policies and Actions 4 of 12
Create a new event policy by pressing the Add button. As shown in Figure 3, this opens up an event policy creation wizard. Figure 3. Event Policy configuration wizard. After entering a name for the policy, press Next to configure the events to be monitored, as shown in Figure 4. This page allows you to select one or more events to monitor, from the following categories: SNMP traps generated from managed devices Application events generated by the management server Pseudo-events: Smart alerts generated by the management server Custom events: Any event logged in the server, based on dynamic selection criteria Snort message You can further filter SNMP traps here by specifying conditions based on Varbinds available in the trap. A Varbind or Variable Binding is a sequence of two specific fields. The first field is an object identifier (OID), and the second contains the value of the specified object. Diagnostics and Troubleshooting Using Event Policies and Actions 5 of 12
Figure 4. Event Policy configuration wizard: Event selection. After you choose the events to monitor, you can select the source devices that need to be monitored in the next page, as shown in Figure 5. Figure 5. Event Policy configuration wizard: Source selection. You can select one or more managed devices from the SAN, IP or Hosts tabs in this page and move them to the right panel. You can also select fabrics, system product groups, and user-created product groups as event sources. Diagnostics and Troubleshooting Using Event Policies and Actions 6 of 12
If you want to preprovision a policy for a device before starting to manage it, you can specify the address of the device by selecting the option Provide the IP Address / WWN / Name of the source on this page. After you select the source devices, you can specify the policy criteria on the next page, as shown in Figure 6. Figure 6. Event Policy configuration wizard: Policy Criteria. The policy criteria control when an action needs to be triggered. Actions can be triggered based on one of the following: Immediately after the event occurs When a frequency-based or time-based condition is met You can specify these conditions on this page. You can also specify a message and severity for an application event that will be generated when an event policy action is triggered. Diagnostics and Troubleshooting Using Event Policies and Actions 7 of 12
After you define policy criteria, you can choose what actions are taken when the selected events occur. As shown in Figure 7, this can be done from the next page, where various types of supported actions are listed. Figure 7. Event Policy configuration wizard: Action configuration. The following actions are currently supported: Apply Logging Policy: This option determines whether the event should be logged in the management server and displayed in the client master log. Auto Acknowledge: This is a useful option to hide events without actually dropping them, in order to reduce event noise for the administrator. It may be used for certain types of events that are expected periodically and need not appear in the master log. Alert by E-mail: Allows e-mail notifications to be sent to the Administrator or any other user when this action is triggered. Run Policy Monitor: This option allows you to run policy monitor health checks against devices that are involved when certain incidents are observed. This action helps you do proactive health checks and diagnostics. Launch a Script: You can use this action to run any scripts that you have created and stored in the management server. Broadcast to Client: This option notifies all active clients about an incident that has occurred. A customized message, along with the event description, can be broadcasted to all active client machines. Mark as Special Event: This option allows you to mark an event as a special event. When the management server receives and processes special events, the following indication is shown in the client status bar: You can view all special events by pressing this icon and launching the special events view. Diagnostics and Troubleshooting Using Event Policies and Actions 8 of 12
Collect Support Save: You can configure this action to collect, support, and save data from the device that generated the event, for troubleshooting purposes. Deploy CLI Configuration: This powerful tool allows you to take remedial action upon receiving an event. A CLI Configuration template is a CLI template that contains one or more CLI commands and that can be deployed on a device. This action allows you to choose a predefined CLI template to be deployed on the source device when the event occurs. This action is applicable only for IP devices. Deploy Product Configuration: This action is used to deploy a specific product configuration on the source device when a specified event occurs. This action is applicable only for IP devices. EXAMPLE The following is an example of how event action policies can be leveraged for diagnostics and troubleshooting. Problem High CPU utilization on the switch can lead to several issues, such as slow performance, high buffer failure, and so forth. Monitoring CPU utilization of the switch in real time and receiving an alert when utilization crosses a desired threshold can help you troubleshoot at an early stage and avoid such issues. Solution Within Brocade Network Advisor, the Administrator can set a high value threshold for CPU utilization. When the threshold is crossed, an event is generated and appears in the Brocade Network Advisor master log. You can adjust the threshold settings from the data collection configuration page. The Administrator selects the appropriate CPU utilization collector from the historical data collector page and sets the threshold by editing the configuration. Once the threshold setting is set, the Administrator creates a new event policy using the event actions wizard, as follows: Select the Traps event and the bnarisingthresholdcrossed event from BNA-MIB under Available Traps. Then, select the required Varbinds and set the filtering criteria. Figure 8 shows the process of setting a filter based on the IP address of the device that the Administrator wants to monitor. Figure 8. Sample Event Policy configuration: Selection of source event. Diagnostics and Troubleshooting Using Event Policies and Actions 9 of 12
The Administrator selects the device that needs to be monitored and configures appropriate actions (such as e-mail notification, technical support data collection, and so forth) to take when a threshold violation occurs. Figure 9. Sample Event Policy configuration: Summary. Next, the Administrator saves and enables the event action policy, as shown in Figure 9. Whenever CPU utilization exceeds the threshold specified, a master log alert is generated and the event actions that were configured are triggered. EVENT CORRELATION AND EVENT ACTIONS You can generate intelligent alerts by using event correlation rules in conjunction with event action policies in Brocade Network Advisor. You can use this powerful tool to detect conditions in the network that are usually not easy to find using the normal events that are generated by devices alone. The Pseudo Events feature available in Brocade Network Advisor allows the Administrator to define correlation rules such as escalation, flapping, and resolve to correlate selected events over a period of time. You can define event actions around the pseudo-events that occur when the defined condition is observed. You can create pseudo-events in Brocade Network Advisor using the path Monitor > Event Processing > Diagnostics and Troubleshooting Using Event Policies and Actions 10 of 12
Pseudo Events. Figure 10 illustrates a sample screen after defining a flapping rule for link up and link down traps. Figure 10. Defining event correlation rules through pseudo-events. This pseudo-event can be associated to devices to which the Administrator wants to apply the rule and actions by creating an event action policy, as shown in Figure 11. Figure 11. Defining an event action policy based on pseudo-events. Note: Refer to the Brocade Network Advisor user manual or online help for more details on configuration of pseudo-events. SUMMARY Event action policies and actions, combined with other monitoring tools in Brocade Network Advisor, can help you provide proactive monitoring and effective diagnostics for the managed network. Diagnostics and Troubleshooting Using Event Policies and Actions 11 of 12
NETWORK MANAGEMENT 2014 Brocade Communications Systems, Inc. All Rights Reserved. 01/14 GA-TB-493-00 ADX, AnyIO, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, and Vyatta are registered trademarks, and The Effortless Network and The On-Demand Data Center are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of others. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.