Using SolarWinds Log and Event Manager (LEM) Filters and Alerts Introduction... 1 Definitions... 1 LEM Components and Architecture... 2 LEM Alerts A Peek Under the Hood... 3 Troubleshooting Agents and Connectors... 6 Keeping your Connectors and Agents up to Date... 6 LEM Filters A Peek Under the Hood... 7 Important Filter Properties... 8 Filter Use Cases... 8 Additional Resources... 9 This paper covers how to create and use Filters and Alerts within the SolarWinds Log and Event Manager (LEM) Product.
Copyright 1995-2012 SolarWinds Worldwide, LLC. All rights reserved worldwide. No part of this document may be reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or other means without the written consent of SolarWinds. All right, title, and interest in and to the software and documentation are and shall remain the exclusive property of SolarWinds and its respective licensors. SOLARWINDS DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL SOLARWINDS, ITS SUPPLIERS, NOR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHER LEGAL THEORY EVEN IF SOLARWINDS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. The SolarWinds, the SolarWinds & Design, ipmonitor, LANsurveyor, Orion, and other SolarWinds marks, identified on the SolarWinds website, as updated from SolarWinds from time to time and incorporated herein, are registered with the U.S. Patent and Trademark Office and may be registered or pending registration in other countries. All other SolarWinds trademarks may be common law marks or registered or pending registration in the United States or in other countries. All other trademarks or registered trademarks contained and/or mentioned herein are used for identification purposes only and may be trademarks or registered trademarks of their respective companies. Microsoft, Windows, and SQL Server are registered trademarks of Microsoft Corporation in the United States and/or other countries. Document Revised: 05/22/2012
Using SolarWinds Log and Event Manager (LEM) Filters and Alerts 1 Introduction This paper is focused on how to properly use LEM filters and alerts. SolarWinds Log & Event Manager (LEM) collects, displays, and responds to network device events. LEM managed devices send messages to the LEM virtual appliance where LEM processes the events. Alerts and filters are two key components of LEM. A thorough understanding of LEM alerts and filters allows you to more effectively use LEM in your environment. To begin, we will define some key LEM elements. Definitions Agent Alert A software component installed on LEM-managed devices that allow third-party agents. Containers LEM uses to display events/messages from LEM monitored devices. Connector (Formerly Tools) Filters Event Rules A software component that converts raw events into normalized events. Connectors can reside on device agents or on the LEM appliance. A component in the LEM console that groups alerts by specific values, such as IP address, device type, or alert name. An unaltered message from a LEM-managed device. A component on the LEM appliance that allows for automated actions based on specific alert correlations.
2 Using SolarWinds Log and Event Manager (LEM) Filters and Alerts LEM Components and Architecture The following diagram shows the functional relationship of LEM components. Beginning from the upper left corner of the above diagram here is the flow of event data through LEM: 1. LEM managed devices send events to the LEM appliance either as raw log messages or LEM agent normalized alerts. 2. The LEM appliance connectors process raw messages for devices that do not allow a LEM agent. Appliance connectors normalize the events and forward LEM alerts to the alert distribution manager. 3. The manager service receives the normalized messages, matches them up with alerts definitions, and sends the alerts to the alert distribution policy. 4. The alert distribution policy distributes the alerts to storage, any connected consoles, and to the alert correlation engine. The following steps are independent of each other. 5. The alert correlation engine examines the alert for any defined actions and executes applicable actions. 6. The LEM console applies filters to the alerts for display purposes. 7. The LEM database stores normalized alerts for reporting and on-demand search.
Using SolarWinds Log and Event Manager (LEM) Filters and Alerts 3 LEM Alerts A Peek Under the Hood Alerts are containers LEM uses to display events/messages from LEM monitored devices. These events can originate from a variety of devices including: Microsoft products. Network switches and routers. Unix, Linux and similar operating systems. Firewalls and other security devices. NetFlow exporters. Antivirus software. Although there are similarities, the raw event messages sent by these devices can vary greatly. Because of the lack of event message standards, interpreting a rapid flow of raw messages from multiple devices is not possible. LEM uses a software component called a connector Connectors reside on agents where available and on the appliance for devices logging directly to LEM. The following figure illustrates this process: Before After Normalization makes alerts Human-readable Consistent Column-oriented Field-based Agents are specific to operating systems. For example, a Windows desktop PC will use the LEM agent for Windows. A default set of connectors is included in the agent installation package. You can add or remove connectors using the LEM console once the agent is connected to the LEM manager. Agents normalize the log data and then send the normalized data to the LEM manager. Network infrastructure systems such as routers, firewall and switches do not allow for the installation of third party agents. These systems, called non-agent nodes, send their log data in raw form to the LEM manager where local connectors are used to parse and normalize the log information. Connectors serve the same purpose whether they are agent based or locally installed on the LEM manager. The only difference is that agent-based connectors are able to normalize messages before the messages are sent to the LEM manager. Manager-based connectors receive raw event information and normalize the information to create alerts. As mentioned previously, normalized messages are human readable. Consistent, defined fields allow for relational database message storage.
4 Using SolarWinds Log and Event Manager (LEM) Filters and Alerts Agents-based connectors minimize the impact of message traffic on the network by discarding unnecessary message data at the device. Agents send normalized data in an encrypted and compressed format to maintain data integrity and optimize bandwidth use. SolarWinds recommends you use agent connectors wherever possible. Use a LEM appliance connector only when you cannot install an agent on the device. To access agent-based connectors and assign them to an agent complete the following: 1. Open the LEM console and connect to your virtual appliance. 2. Click Manage > Nodes, then click the on the left of the node name. 3. Choose Tools. This brings you to a screen similar to this one: This view is unfiltered, meaning that all LEM connectors are selectable. This view is useful for searching for a connector when you are not sure which category it may be in. It is best to use the category name filtered view if you know which category the connector is in. This saves time troubleshooting nonapplicable connectors. Remember, connectors were called tools in earlier versions of LEM. Some of the old nomenclature may still exist in the interface.
Using SolarWinds Log and Event Manager (LEM) Filters and Alerts 5 The Category menu is useful for viewing only connectors that apply to the node you have selected. In this screenshot, note that Operating Systems, Physical Infrastructure and Proxy Servers category connectors are displayed. The Node in this case is a Microsoft Windows 7 computer. Using the categories filter makes it easier to find the connectors that apply only to that node type. The following screenshot shows the Tools view with the Operating Systems category selected. This view shows several operating systems (OSs), so care should be taken to not apply a connector for the wrong OS. Once a connector has been applied to a node s agent, you click the gear menu in the first column and choose Start. If you want to determine which connectors are assigned to a node, select the Configured check box beneath the Status menu. The resulting screen looks like this:
6 Using SolarWinds Log and Event Manager (LEM) Filters and Alerts For each configured connector you will see two rows. The top row shows the connector chosen and the row below that shows the status of the connector and its alias. If you see one or more connecters are grey, and you believe that it should be running, try starting it first using the next to the Status column. If all the connectors are grey ensure the agent shows as connected in the Manage > Nodes view, and try starting the connectors again. Troubleshooting Agents and Connectors Starting your troubleshooting at the agent level and proceeding to the connector level provides a topdown method of troubleshooting. Start at the screen shown above and check on the connectors status. If all of the connectors on an agent are gray, this most probably an agent issue. Knowledge Base article 3611 provides detailed instructions for troubleshooting agent issues. http://knowledgebase.solarwinds.com/kb/questions/3611/troubleshooting+lem+agent+connections Knowledge base article 3679 provides detailed instructions for troubleshooting connector issues for nonagent devices. http://knowledgebase.solarwinds.com/kb/questions/3697/troubleshooting+network+devices+logging+to +LEM If a particular LEM connector will not start, and the connector is running on an agent with other connectors that are functioning correctly, see the Connectors category of the SolarWinds Knowledge Base at http://knowledgebase.solarwinds.com/kb/categories/log+and+event+manager/connectors/. If your connector is not listed, contact Support. Keeping your Connectors and Agents up to Date When you see connectors or agents fail, a possible cause is the equipment vendor changing the way the device logs or the type of information logged. When this happens SolarWinds creates new connectors to comply with the logging changes. From time to time SolarWinds updates the available agents and connectors. Customers with active maintenance can locate updated agents and connectors in the SolarWinds Customer Portal. The updated agents and connectors are in the Additional Resources area of the portal. After the LEM agents and connectors are working properly, you can apply filters to further define what LEM will do with the incoming data.
Using SolarWinds Log and Event Manager (LEM) Filters and Alerts 7 LEM Filters A Peek Under the Hood Filters organize your alerts into views that you define. Filters are stored in the LEM console, and they allow you to view all of your alerts in real time. For an unfiltered view, use the default All Alerts filter. For a narrower view, select another default filter, or create your own. Filters are based on standard logical operators, and you can pinpoint alerts using any field-value combination. Take the following Virus Attack alert as an example: To filter for alerts like this one, look for values in the Information column that differentiate the alerts you are looking for from all the others. If you want to filter for a partial value, you can use wildcard characters. For example, create a filter like this to see only viruses that your AV quarantined: VirusAttack.EventInfo = *quarantined* Use the Filter Creation dialog as shown below to make these types of filters.
8 Using SolarWinds Log and Event Manager (LEM) Filters and Alerts Important Filter Properties It is impossible to list all of the possible filters you could create. The basic rule is, "If you can see the alert, you can create a filter for it." However, keep the following in mind as you explore and create filters: Filters are user-specific. Whether you create filters using the web or desktop console, filters are always related to the user who created them. When you use the web console, filters are related to the LEM user who created them. For example, if you log into the web console as the admin user and create a filter, you do not see that filter when you log in as a different LEM user. When you use the desktop console, filters are related to the Windows user who created them. For example, if you log into a Windows computer as DOMAIN\Administrator and create a filter, the filter is only available on that computer. Furthermore, it is only available for the DOMAIN\Administrator user. It does not matter what LEM user account you use. To share filters across your enterprise, use the export and import options in the LEM console. Filters display real-time data. When you view alerts in your filters, you only see real-time data. When you close and reopen the console, all of your filters start fresh. Furthermore, the LEM console limits the number of alerts a filter can display. The default limit is 1,000 alerts per filter, but you can increase that limit to a maximum of 2,000 alerts when you create or edit a filter. To view alerts no longer in your filters, use ndepth or LEM Reports. LEM stores the alerts on its database as soon as it displays them. Filters generate local notifications. When you create a filter, you have the option to specify one or more of the following local notifications: Display Popup Message Display New Alerts as Unread Play Sound Enable Blinking Filter Name These notifications only work if you have the LEM console open. If you want a notification outside of the console, create a rule to send a popup or email message. Filter Use Cases In addition to allowing you to monitor your log data in real time, filters address the following use cases: Monitor specific servers. To monitor all logons, logon failures, and network changes made on your domain controllers, create a filter for that group of servers. A filter like this requires a LEM agent on each of these servers. However, if you want to monitor web traffic from these servers, you can do that without an agent. Monitor your firewalls and other network devices, and then create a filter for that traffic, specifying your critical servers.
Using SolarWinds Log and Event Manager (LEM) Filters and Alerts 9 Power Ops Center widgets. To get a graphical overview of your real-time alert data, use widgets. All user-defined widgets are powered by filters. So, if you want a graph to show you all logon failures, you'll need a filter for that data first. After the filter is in place, create a widget to point to that filter. Widgets display data in pie chart, bar graph, line graph, or table format. Create test scenarios for LEM rules. Since rules execute real-time actions on your network, you might want to test them out before you set them loose. Filters and rules use a similar configuration interface, so you can use them to test your rules. If you see something you want to create a rule for, create a filter for it first and watch your console for the filter to catch the event. After you verify the filter works the way you expected, create the rule using the same logic. Remember, while filters only provide local notifications, LEM rules can execute real-time actions, such as sending you an email, logging off a user, or restarting a service. Find what you may have missed. If you want to see if you missed any alerts that meet a filter's conditions, send the filter to ndepth. ndepth queries your LEM database on demand, so you always have access to that data, even if the data does not show up in your filters. Additional Resources SolarWinds LEM Knowledge Base http://knowledgebase.solarwinds.com/kb/categories/log+and+event+manager/ SolarWinds LEM Support Documentation Page http://www.solarwinds.com/documentation/lem/lemdoc.aspx SolarWinds thwack Community Product, betas and release candidates. http://thwack.solarwinds.com/community/log-and-event_tht/log-and-event-manager