SolarWinds SAM RELEASE CANDIDATE DRAFT - NOT FINAL VERSION MAY 4, Server & Application Monitor. Getting Started Guide. Version 6.2.

Transcription

1 SolarWinds Server & Application Monitor Version Getting Started Guide Part 3 of 3: Scale and Manage Last Updated: April 14, 2016

2 Table of Contents Scale and manage SolarWinds SAM 4 Related Guides 4 Scale 5 Plan to scale monitoring 5 Advanced discovery 5 Add a single node for monitoring 7 Agents 10 How agents work 10 Agent resource consumption 12 Agent settings 12 Agent-initiated communication 13 Orion Server-initiated communication 13 Deploy agent software 14 Deploy the agent on the network 14 Connect to a previously installed agent 15 Troubleshooting deployment 15 Credential test for deployment fails 15 If the agent deployment fails 16 Quality of Experience 17 How Quality of Experience works 17 QoE views 18 Install a network sensor 19 Before you begin 19 Deploy a network sensor 19 Install the network sensor 20 Monitor website traffic based on domains 20 2

3 Install a server sensor 21 Manage your environment 22 Database Maintenance 22 Check the database size 22 Specify the time to run database maintenance 23 Adjust retention periods 24 Use additional polling engines to balance polling 24 Install an Additional Polling Engine 25 Configure polling settings to optimize polling performance 25 High availability and disaster recovery options 26 Options for Orion database failover 27 Options for Orion application server failover 27 Advantages of using disaster recovery site licenses 27 Reduce alerting noise 29 Complex trigger conditions 29 How condition blocks are evaluated 29 Aggregate alerts 30 Alerts with multi-element triggers 30 Use instance names 31 Use duration in the alert trigger 32 The 'Do Not Alert' alert 33 Send alerts to specific contacts 35 Alerts with device-specific thresholds 35 Single alert when multiple devices go down 36 Use groups 37 Define object dependencies 37 Event correlation alerts (Y after X) 39 3

4 Scale and manage SolarWinds SAM Welcome back, and congratulations on customizing SolarWinds SAM. Now that you are monitoring a few mission-critical applications, it is time to deploy SolarWinds SAM to other parts of your IT environment. In this part of getting started, you will: Identify other areas of your environment you want to monitor Schedule a series of jobs that discover other applications you can monitor Deploy agents Learn about Quality of Experience Learn about features that help you manage SolarWinds SAM for long-term success, including adding polling engines, and maintaining the database. Depending on your workload, this part of the process should take two or fewer weeks. If you are a customer and need assistance, contact our Support Geeks. If you are evaluating the product and need assistance, contact sales@solarwinds.com. Related Guides Server & Application Monitor Getting Started Guide: Get Started (Part 1 of 3) Server & Application Monitor Getting Started Guide: Customize (Part 2 of 3) 4

5 Scale This section includes guidelines for scaling your SolarWinds deployment to your entire IT environment: Plan to scale monitoring Advanced discovery Add a single node for monitoring Plan to scale monitoring You installed and configured your Orion product, discovered part of your IT environment, and have monitoring statistics displayed in Orion Web Console views. As you continue the deployment, consider the following questions: Are there any gaps in your monitoring coverage? Is there an essential device whose failure could affect your environment? Are there less important devices or applications that you want to monitor? Are there other groups or locations that you might want to monitor? As you deploy monitoring across your environment, you can: Add discoveries to include other segments of your IT environment. Add individual nodes for monitoring. This is the recommended approach when you have a node with high latency. Advanced discovery At this point you have completed an initial discovery. Now you can add discoveries to include other segments of your IT environment. Discovery jobs do not impact polling. Polling is prioritized first. Multiple jobs. SolarWinds recommends building as many discovery jobs as needed to scan your network. Initially, run the jobs immediately so you can see everything on your network, and then schedule the jobs to run periodically. Dividing the discovery into multiple jobs makes it easier to be selective about what to monitor, and decreases the amount of time each job runs. When you have a large environment, consider dividing discovery jobs by: Credentials - the more credentials you have, the longer it takes for the discovery job to complete. Place the most common credentials at the top of the list. IP address range - use a range that consists of fewer than 2,000 IP addresses. In a range, unresponsive IP addresses slow down discovery. Latency - run discoveries for remote offices separately so that you can adjust the timeout threshold. 5

6 Polling engine - if you have multiple polling engines, configure a discovery for a specific polling engine. Discovery ranges. Although you can discover specific nodes, SolarWinds recommends using a range of IP addresses or subnets for a more complete picture of your network. None of the discovered elements count toward your license total or affect system performance until you begin monitoring. You can add multiple IP ranges or subnets to the same scan, but you cannot include IP ranges and subnets in the same scan. Discovery thresholds. If you run a discovery and nodes you expect to see are not found, you may need to adjust the timeout and retry thresholds. In an environment with high latency, the default values may not be high enough. Only adjust these values after an initial scan. The higher the threshold value, the longer the discovery job takes to complete. Polling engine. If you have multiple polling engines, you will see an option to select a polling engine. The polling engine you select runs the discovery job and monitors your network. SolarWinds recommends that you limit a poller to 12,000 elements, so be careful not to overload one polling engine. If you have a large environment with significant differences in latency, position the polling engine close to the monitored objects. 6

7 Schedule intervals. Schedule your discovery jobs to run periodically to identify new devices that were added to your network. Unless you work in a dynamic environment where new devices are frequently added to your network, SolarWinds recommends scheduling discovery daily. You can also select Advanced from the Frequency drop-down menu to create a custom frequency. The schedule interval you choose depends on how often you want to scan for changes to the network and the size and performance of your deployment. Scheduled results. A scheduled discovery only finds network elements, but does not automatically start monitoring. You must select what you want the system to monitor. If the scheduled job locates nodes that you do not want to monitor, select those nodes and click Add to Ignore List. Ignoring hides elements from the results list the next time the discovery job runs. Scheduled discovery profiles should not use IP address ranges that include nodes using DHCP. Add a single node for monitoring As an alternative to using the Network Sonar Discovery wizard, you can add individual nodes for monitoring. Adding a single node offers more detail in monitoring and is the recommended approach when you have a node with high latency. Do not include nodes with high latency in a discovery job. 7

8 As you add a single node for monitoring, you can: Select the statistics and resources to monitor. Identify how often the node status, monitored statistics, or topology details are updated. Add custom properties. Edit alert thresholds. To add a single node for monitoring: 1. Log in to the Orion Web Console as an administrator. 2. Click Settings > Manage Nodes, and then click Add a Node. 3. Specify the node, and click Next. a. Provide the host name or IP address. b. Select the polling method, and provide credentials. 8

9 4. Select the statistics and resources (volumes or interfaces) to monitor on the node, and click Next. 5. Review and adjust the device properties. a. To edit the SNMP settings, change the values, and click Test. b. To edit how often the node status, monitored statistics, or topology details are updated, change the values in the Polling area. For critical nodes, you may need to poll status information or collect statistics more frequently than the default polling intervals. Change the polling intervals if polling the nodes takes too long. c. Enter values for custom properties for the node. The Custom Properties area will be empty if you have not defined any custom properties for the monitored nodes. See "Add custom properties to nodes" in the SolarWinds Getting Started Guide - Customize. d. To adjust when the status of the node changes to Warning or Critical, edit alerting thresholds for the metric. Select the Override box and set thresholds specific for the node. 6. Click OK, Add Node. The node will be monitored according to the options you set. 9

10 Agents This section includes information on deploying agents: How agents work Agent resource consumption Agent-initiated communication Orion Server-initiated communication Deploy agent software How agents work An agent is a software application that provides a communication channel between the Orion server and a Windows computer. Agents are used as an alternative to WMI or SNMP to provide information about your selected key devices and applications. These agents provide the following advantages: Polls host and applications behind firewall Network Address Translation (NAT) or proxies. Polls nodes and applications across multiple discreet networks with overlapping IP address space. Provides secure encrypted polling over a single port. Supports low bandwidth, high latency connections. Polls nodes across domains where no domain trusts are established. Provides full encryption between the monitored host and the Orion poller. Monitors the server and installed applications during a network outage, regardless of whether the agent can communicate with the poller. When the poller connection is restored, the agent forwards the results of its monitoring data collected during the outage to the poller for processing. All data gaps are filled with the data collected by the agent. The agent allows you to monitor servers hosted by services, such as Amazon EC2, Rackspace, Microsoft Azure, or virtually any other Infrastructure as a Service (IaaS). All communications between the Orion server and the agent occur over a single fixed port. The communications are fully encrypted using 2048-bit Transport Layer Security (TLS) encryption. The agent protocol supports NAT traversal and passage through proxy servers that require authentication. If an agent is used, all SAM application data is collected by the agent. You can choose to override this behavior. For example, if you assign an application template that uses a User Experience Monitor and you do not want to measure response time locally from the server where the application is installed, you can switch the application to poll without using an agent. This can be configured at both the application and template level. 10

11 Agent Communication can be deployed as either Agent Initiated or Orion Server initiated: Agent initiated communication: The agent initiates communication with the server on the default port of This port must be opened on the server firewall so the agent can connect. No change to the agent firewall is required. Orion Server initiated communication: The agent waits for requests from the server on the default port of This port must be opened on the firewall of the agent computer so the server can connect. No change to the server firewall is required. Reports installed for use with the agent: Agent Inventory Agent Plugin Version Tip: Use Orion Server initiated communication in DMZ environments or cloud scenarios such as Azure. Use agent initiated communication with a proxy to poll multiple computers within a single Azure cloud service. 11

12 Agent resource consumption A single polling engine can support up to 1,000 agents. The following table details agent resource consumption. CPU Memory 0.24% on average under normal operating condition Between 10 and 100 MB depending upon the number and types of jobs Bandwidth Storage Agent settings The agent consumes roughly 20% (on average) of the bandwidth consumed by the WMI protocol for transmission of the same information For example, Agent: 1.3 KBPS versus WMI at 5.3 KBP 100 MB The Agent Settings page provides access to all of the settings and tools needed to install and manage agents. Additional agent settings can be found in the Control Panel. On the Web Console, click Settings > All Settings > Agent Settings. Manage Agents: Opens the Manage Agents page so you can add a new agent, edit, update, or reboot an existing agent. Download Agent Software: Opens the Agent Downloads page so you can mass deploy or manually install an agent. 12

13 Define Global Agent Settings: Opens the Global Agent Settings page so you can allow automatic agent registration or allow automatic agent updates. Agent-initiated communication All communication between your Orion server or additional poller and the agent is initiated by the agent. Your Orion server does not initiate communication with your agent. A direct route from the server with the agent installed to your Orion server or additional poller is not necessary. To use this communication method, open port on the Orion server or additional poller to receive information from the agent. This communication method is most useful when the agent is installed on a network separated from your Orion server by one or more NAT devices, and you have no easy way to connect the two. Orion Server-initiated communication With passive communication, the agent waits for requests from the server on the default port of (This can be changed if needed.) This port must be opened on the firewall of the agent computer so the server can connect. No change to the server firewall is required. 13

14 Deploy agent software The Orion server push deployment method allows you to perform a network-wide deployment from Orion and does not require you to download additional files. In order for this deployment method to succeed, the Orion server must be able to communicate with the client computers. You may need to log in with an administrator account to perform this action. 1. On the Web Console, click Settings > All Settings > Manage Agents. 2. Click Add Agent, select the method to use to add the agent, and click Next. 3. Select one of the following procedures, based on your selection, and click Next: Deploy the agent on the network Connect to a previously installed agent Deploy the agent on the network Deploying the agent on the network allows you to install the agent on multiple client computers. 1. On the Deploy Agent on Network page, enter the IP address or host name of the Windows computer where you want the agent to be installed, or select nodes from the list, and click Next. This field does not accept ranges. It is used to add computers that are currently not nodes in the system. 2. On the Agent Settings page: a. Select the computer you selected in the previous step, and click Assign Credentials. b. Choose a credential from the list or enter new credentials, and click Submit. 3. Click Deploy Agent. Orion now installs the agent software. 14

15 Connect to a previously installed agent This action allows you to connect to agents that were configured with Orion Server Initiated Communication or if Allow Automatic Agent Registration is not enabled. When you connect to an agent, select the communication mode that was chosen when the agent was installed. If the communication mode is server-initiated (passive), an Agent Shared Secret was required during installation. This Agent Shared Secret needs to be re-entered. 1. On the Add Agent page, name the Agent. 2. Select the agent communication mode. For Agent-Initiated Communication, go to Step 7. For Server Initiated Communication, continue with Step For Server-Initiated Communication, perform the following steps: a. Enter the IP address or host name where the agent is located. b. Enter the Agent Shared Secret. c. You can optionally expand Advanced and adjust the following as needed: i. Change the agent port number. (This is the port the agent uses for listening.) ii. Use a proxy by checking its box and selecting a Proxy and entering its Proxy URL in the fields provided. iii. Select proxy authentication, and enter a user name and password. 4. For Agent-Initiated Communication, select the agent. 5. Select Allow Automatic Agent Updates to have the agent automatically upgraded when upgrading to new versions of Orion modules that support the agent. Disabling this option will require you to manually upgrade the agents after upgrading your Orion products and modules. 6. Click Submit. When the connection is successful, the agent will appear in the agent list on the Manage Agents page. Orion supports three methods of deploying an agent to a client computer running Windows: Enable the Orion server to push the agent software to one or more computers. Mass deploy the agent software to multiple computers using a mass-deployment technology, such as Group Policy. Manually install the agent on a client computer. Troubleshooting deployment Possible errors and their resolutions: Credential test for deployment fails Ensure the user account can access the following folder: \\<hostname_or_ip>\admin$\temp. Also ensure that a folder can be created at that location. 15

16 Ensure the Remote Procedure Call (RPC) Windows service is running and the required ports are open. If you are using a domain account, use the complete name when entering credentials. For example: Domain\Username If the agent deployment fails Ensure there are no other installations in progress. For example, Windows updates and installations prevent other installations from finishing successfully. Try reinstalling the agent when other installations complete. On the target machine, make sure the SolarWinds Agent service is installed and running. If it is, the agent may have connectivity issues with the Orion server. Ping the Orion server from the client machine and ensure that port is open on the Orion server. Also, verify the client machine can connect to the Orion server web interface. Manually install the agent on the target machine, ensuring that permissions are configured correctly. Agent deployment can fail if a previous installation or upgrade is waiting for a reboot. To resolve this issue, you must reboot the server before the agent installation can proceed on that computer. 16

17 Quality of Experience This section includes information on enabling the Quality of Experience dashboard: How Quality of Experience works Install a network sensor Install a server sensor How Quality of Experience works The Quality of Experience (QoE) dashboard helps you answer the question: Is it the application or the network? QoE discovers and monitors applications running on your network. By analyzing packets for information such as network and application response time, traffic volume, and number of transactions, the QoE dashboard provides a high-level view of whether a slowdown is on the physical network or on the application side. It also allows you to see which applications are running on which nodes, and how much traffic is being generated by each application. QoE can also help you locate risky applications that may be running on your network. QoE uses Packet Analysis Sensors (PAS), which install an agent on a Windows machine, to collect packetlevel traffic statistics. You can deploy a PAS in two ways: On a Windows server that is running applications you want to monitor. For example, you might install a server sensor on your Exchange servers, Windows IIS server, or domain controllers. If you selected the recommended QoE option when you installed SolarWinds Orion, you installed a PAS on your SolarWinds server. In this case, you will see packet-level traffic information about your server on the QoE dashboard. This option is quick and easy to set up, and no additional hardware is needed. However, a server PAS will only monitor the traffic going to and from the server it is installed on. On a Windows server that is monitoring a switch or router s SPAN or mirror port. This option requires a dedicated Windows server monitoring the SPAN or mirror port. However, it collects statistics about all the traffic going through the network device, regardless of the type of device the traffic originated from. Before you install a PAS, review the Packet Analysis Sensor system requirements. QoE charts and graphs enable you to monitor network spikes, or to drill down to the network or application level. QoE provides observed network response time (NRT) and application response time (ART), and can classify and categorize traffic for over 1000 different applications by associated purpose and risk level. QoE auto-detects applications and URLs. 17

18 QoE views Click My Dashboards > Home > Quality of Experience. QoE data are displayed in different views of the Orion Web Console. The QoE Summary View contains overall information about all monitored QoE nodes. The QoE Node Details View displays information related to a specific node. 18

19 The QoE Application Detail View displays information related to a specific QoE application. With QoE, you can: Compare statistics, such as network response time (TCP Handshake) and application response time (Time to First Byte) to determine if a bottleneck is on the application or the network. Use data volume trends to pinpoint traffic anomalies and investigate the cause. Monitor risky types of traffic, for example, traffic that might bypass firewalls or lead to data leaks. For detailed information about deployment, see Common Packet Analysis Sensor Deployment Scenarios. Install a network sensor Before you begin Data from sensors is directed to the polling engine assigned to the node when the sensor was deployed. A high number of applications or nodes can cause performance issues with the sensors. Deploy a network sensor Network sensors analyze packet data flowing through a monitored switch or router. The network sensor must be installed on a Windows computer that is monitoring the switch's SPAN or mirror port. 19

20 Install the network sensor 1. Click Settings > All Settings in the menu bar. 2. Click QoE Settings > Manage QoE Packet Analysis Sensors, and click Add Packet Analysis Sensor. 3. Select Network, and click Add Nodes. 4. Move the node that monitors your switch to the Selected Nodes panel, and click Add Selected Nodes. Make sure you select the Windows machine that is monitoring the SPAN or mirror port of the switch. 5. Assign and test the credentials for the node, and click Submit. 6. Click Add Nodes and Deploy Agents. When the sensors are successfully deployed, a message is displayed in Notifications. Deploying the sensor and receiving the first set of data can take several minutes. When the deployment is finished, select the sensor on the Manage Quality of Experience (QoE) Packet Analysis Sensors page, click Edit Sensor, and verify the selected NIC. Monitor website traffic based on domains After you deploy a network sensor, you can filter application traffic based on domain names instead of all http traffic. 1. Click Settings > All Settings in the menu bar. 2. Click QoE Settings > Manage Global QoE Settings. 20

21 3. Set the HTTP application domain detection level. 4. Verify the Auto-detect QoE applications option is set to Active, and click Submit. QoE can automatically detect the first 50 applications, or you can add specific applications. Discovered applications have the "No Risk" Risk Level and the "Both Business and Social" Productivity Rating associated with them. To modify the Risk Level and Productivity Rating, click QoE Settings > Manage (QoE) Applications, and edit the application. Use the Global QoE Settings page to disable monitoring or discovery of multiple applications. Select the applications, and click Disable Monitoring or Disable Discovery. Install a server sensor Server sensors analyze all the packet-level data flowing to and from the Windows server that the sensor is installed on. After you deploy a server sensor to the application node, the sensor monitors packets to and from the node, identifies the application or the URL, and analyzes the packets for QoE metrics, such as response time or traffic volume. 1. Click Settings > All Settings in the menu bar. 2. Click QoE Settings > Manage QoE Packet Analysis Sensors, and click Add Packet Analysis Sensor. 3. Select Server, and click Add Nodes. 4. Move the Windows nodes that will host the server sensors to the Selected Nodes panel, and click Add Selected Node. 5. Assign and test credentials for each node, and click Submit. 6. Click Add Nodes and Deploy Agents. QoE auto-discovers the applications on the servers. 21

22 Manage your environment This section provides guidelines in managing your SolarWinds environment: Database Maintenance Use additional polling engines to balance polling Configure polling settings to optimize polling performance High availability and disaster recovery options Database Maintenance Database maintenance is an automatic process that optimizes the size of your SolarWinds Orion database. During maintenance, the data collected for a certain period are aggregated and new statistical values, based on the aggregated data, are calculated. The data are discarded, and only the aggregated statistics are retained. Database maintenance is run every day at a specified time. Depending on the data granularity and retention period, you may need to configure your database differently. Keep in mind that the more granularity and the longer the retention period, the larger the database. Check the database size 1. Start the Database Manager in the SolarWinds Orion > Advanced Features program folder. 2. Add your database server and expand it. 22

23 3. Right-click your SolarWinds Orion database, and select Database Details. The database size is displayed in the Properties tab. Specify the time to run database maintenance Make sure database maintenance runs after business hours. 1. Log in to the Orion Web Console using an administrator account. 2. Click Settings > All Settings. 3. Click Polling Settings in the Thresholds & Polling grouping. 4. Scroll down to Database Settings, and enter an Archive Time. 5. Click Submit. 23

24 Adjust retention periods Data for collected statistics are retained for a specified time. Shorten the retention periods to save storage space in your database. 1. Log in to the Orion Web Console using an administrator account. 2. Click Settings > All Settings. 3. Click Polling Settings in the Thresholds & Polling grouping. 4. Scroll down to Database Settings, and adjust the retention periods. The detailed data are retained for the specified period and summarized into hourly data. Hourly data are then summarized into daily statistics, and daily statistics are discarded after the specified time. The shorter the data interval, the greater effect the setting has on the database size. You can also adjust retention periods for other statistics. Changing the shortest retention period has the greatest effect on the database size. Assuming a 10-minute polling interval, one new entry is added to the database for each monitored object every 10 minutes. That means 144 new entries a day are added for each monitored object during the Detailed Statistics Retention period. In the Daily Statistics Retention period, only one entry a day is added to the database for each object. 5. Click Submit to commit the changes. Use additional polling engines to balance polling Before you begin, review the Scalability Engine Guidelines for your product for information about the limits and options associated with additional polling engines. Installing and configuring an Additional Polling Engine is identical to installing a primary SolarWinds polling engine, with the following considerations: The most recent installer is available in your SolarWinds Customer Portal under My Downloads > View downloads for: Orion Additional Polling Engine. Up to three total polling engines can be installed on a single server. In this case, individual licenses must be activated for each polling engine. If you configured an alert with a Send action to trigger on a node monitored by an additional polling engine, confirm that the additional polling engine can access your SMTP server. Otherwise, the s will not be sent. 24

25 Install an Additional Polling Engine 1. Extract the.zip file you downloaded, and run the executable file. The extracted folder contains Additional Polling Engine installers for all Orion products that support Additional Polling Engines. Launch the installer that corresponds to the SolarWinds product installed on your primary SolarWinds Orion server. If you have multiple Orion products installed on your primary SolarWinds Orion server, install the additional polling engine for each product. 2. On the Welcome window of the Compatibility Check, provide the following information: The host name or IP address of your primary SolarWinds Orion server. The user name and password of a user with administrative privileges to the Orion Web Console on your Main Polling Engine. 3. Click Next to complete the installation the same way as on a primary SolarWinds Orion server. Configure polling settings to optimize polling performance To optimize your polling engines for best performance, SolarWinds recommends tuning them regularly. If you use more than one polling engine, you must balance the load so each engine performs optimally. The Polling Engines view provides information about the performance of all polling engines in your Orion Platform product installation. To access the Polling Engine view, click Settings > All Settings, and then Polling Engines in the Details group. Click Settings > All Settings, and in the Thresholds & Polling group, click Polling Settings to configure your poller. 25

26 On the Polling Settings screen, modify the appropriate settings, click Re-Apply, and click Submit. On the Polling Settings screen, you can configure: Polling intervals Polling statistic intervals Dynamic IP address and hostname resolution Database settings Network settings Calculations and thresholds High availability and disaster recovery options SolarWinds supports options that ensure your system provides high availability and continues to operate despite disruptive events. The options you select depend on: Your IT environment. Your business requirements. The expertise you have on-site to maintain the system. The cost of implementing a high availability or disaster recovery solution. 26

27 The Orion database and application server require separate high availability or disaster recovery solutions. SolarWinds recommends planning your implementation before you begin. Options for Orion database failover Database mirroring increases database availability by transferring transaction log records directly from one server to another, and can quickly fail over to the standby server. Failover clustering is a process where the operating system and the SQL Server database server work together to provide availability in the event of an application failure, hardware failure, or operating-system error. Failover clustering provides hardware redundancy through a configuration in which critical resources are transferred automatically from a failing machine to an equally configured server to ensure continuity of service. Backup log shipping increases the availability of SQL Server by automatically copying and restoring the database transaction logs to another database on a standby database server. Because the standby database receives all changes to the original database, it is a duplicate of the original database, and out-of-date only by the delay in the copy-and-load process. You then can make the standby server a new primary server if the original primary server becomes unavailable. When the original primary server is available, you can reconfigure it as a standby server, reversing the server roles. NeverFail for SQL is a clustered solution that replicates data from your active server to your passive server. See Options for SQL failover with SolarWinds Orion for more information. Options for Orion application server failover Disaster recovery site license is a discounted license that is essentially the same as a production site license. You can deploy the disaster recovery licenses in a disaster recovery scenario or a high availability scenario. In the disaster recovery scenario, by running the Configuration Wizard and pointing nodes to new EngineIDs, one site is manually made active when another site goes down. In a high availability scenario, two Orion sites poll different sites at the same time. You can also use vmotion for your Orion server, however, SolarWinds does not support vmotion. Advantages of using disaster recovery site licenses There are several advantages to using disaster recovery site licenses: Extra training is not required for Orion Server Administrators. The disaster recovery site polls using the same software, except from an alternative site. Orion pollers can be installed on any hardware that meets the Orion software requirements. The production and disaster recovery sites each have their own hardware, including local on-site Orion Servers and SQL Server. The production and disaster recovery sites can run different versions of the Orion software. In a high availability scenario, disaster recovery site licenses offer these additional advantages: 27

28 Since it has its own database, the disaster recovery site is always live and polling. If one Orion site implementation goes down, the other site is up and running. Orion administrators can enable and disable alert actions on the disaster recovery and production sites. 28

29 Reduce alerting noise This section includes techniques for reducing alert noise: Complex trigger conditions Alerts with multi-element triggers Use duration in the alert trigger The 'Do Not Alert' alert Send alerts to specific contacts Alerts with device-specific thresholds Single alert when multiple devices go down Define object dependencies Event correlation alerts (Y after X) Complex trigger conditions Use the Advanced options in the trigger conditions to create complex conditions, such as alerting when an IIS application goes into a warning or critical state. Complex conditions are generally enabled by users who are comfortable with building normal trigger conditions, or who have trialed alerts using the normal trigger conditions and require more control over the trigger conditions to better refine the environmental conditions that trigger an alert. Do not use complex conditions until you have tested the trigger conditions individually. Creating an alert with complex conditions without testing it may prevent you from receiving important alerts. How condition blocks are evaluated The condition blocks are evaluated at the same time. If they are all true based on the conditions, the alert triggers. For example, condition A, B, and C must be true in order for the alert to trigger. (Condition A) & (Condition B) & (Condition C) Condition blocks are evaluated using variations of AND, so the trigger condition in each section must be met. A condition block can be evaluated at a different time than other condition blocks. For example, if you want to be alerted if the backup system is active for more than an hour, you can choose to wait an hour after the primary condition block, where the application going down is the trigger condition, before evaluating whether the backup system is still active. 29

30 Aggregate alerts With complex conditions enabled, you can choose to trigger alerts only when multiple objects meet the trigger condition. After you have enabled complex conditions, the following option is available in your trigger condition: This setting then combines all alerts that would be sent for each object into a single alert. Do not use this setting until you are confident that the trigger condition is correct. This setting can prevent important alerts from triggering. Alerts with multi-element triggers You can use complex conditions to trigger alerts when objects meet separate trigger conditions at the same time. For example, if you have two applications on two separate servers, you can still manage if you lose one application. However, if you lose both applications, you need the system to send a critical alert. Complex conditions should be used with caution. SolarWinds recommends testing each trigger condition individually before combining them. On the Trigger Conditions panel, expand Advanced options, and select Enable complex conditions. With a standard alert, you can create an alert for each of the two applications. With a complex condition, you can create a single alert where the application would have to fail on both Server A and Server B in order to generate an alert. In this example, the primary trigger condition is written for Server A. 30

31 Click Add Section and write a trigger condition for Server B. In this example, both conditions must resolve to TRUE in order for the alert to fire, but do not have to relate to each. The condition blocks are evaluated at the same time. If they are all TRUE based on the conditions, the alert triggers. You can add additional conditions to configure granular and complex alerts. Using condition blocks is a great way to reduce alerts. Instead of receiving an alert for each element, you can receive a single alert. Use instance names In the example above, a specific node name is selected in the condition. This works well if the alert has a very limited scope. Another approach is to use instance names in the condition. 31

32 Use duration in the alert trigger Using durations for alerts is a great way to reduce the total number of alerts you receive. There are times when you want an alert to trigger as soon as the condition is detected. Other times, you may only want the alert to trigger when a condition exists for more than a certain amount of time. For example, your Exchange service usually runs at 80% RAM and 75% CPU. Occasionally, the RAM and CPU utilization spikes to over 90%, and after 5 minutes, returns to a normal level. In this scenario you want to be notified of prolonged spikes, so you configure an alert to trigger when CPU and RAM utilization is greater than 90% for more than 15 minutes. This setting ensures that you are notified when the service requires attention, and allows enough time for a technician to intervene before the service goes down. The following example illustrates an alert that includes duration. Ensure that you select the Condition must exist check box and enter a duration. 32

33 The 'Do Not Alert' alert You can use custom properties to suppress alerts from being triggered when an object goes down. In this case, include a custom property in the alert condition that stops the alert from being fired. 1. Click Settings > All Settings. 2. In the Node & Group Management group, click Manage Custom Properties. 3. Click Add Custom Property. 4. Create a Boolean (Yes/No) custom property called AlertOnThis. 5. Click Select Nodes. 6. Select all of the nodes on the system. 7. Click the green arrow to move all selected nodes to the right. 33

34 8. Set the AlertOnThis property to Yes. 9. For devices for which you do not want to receive an alert, set the custom property to False. 34

35 10. Define the alert to trigger when the AlertOnThis custom property is equal to Yes and the Node Status is equal to Down. If you use the AlertOnThis custom property, every alert must include the AlertOnThis check. Otherwise, all nodes will trigger alerts regardless of how this custom property is set. Send alerts to specific contacts When you have people or groups that are responsible for different monitored objects, you might only want them to receive alerts for objects for which they are responsible. 1. Create a custom property called Contact . 2. Enter the address for each monitored object. 3. Create an alert that sends an when an object goes down. 4. On the Trigger Action page in the Alert builder wizard, add an action. Because you want the recipient of the to be variable depending on who the Contact is, you need to add a variable to the Recipients section. The variable ensures that your alert is only delivered to people who are listed in the Contact custom property. The To: field does not have an Insert Variable button, so if you have labeled your custom property something other than Contact , open the Message section, click Insert Variable to find the custom property, and cut and paste the variable into the To field. Alerts with device-specific thresholds You can set different alert thresholds per device, interface, or other monitored element. For example, it is acceptable to you if most of your nodes reach a 90% threshold for CPU load, but you want a few nodes to alert when they reach a 70% threshold. You can set CPU thresholds locally on a node, and then create a generic alert that notifies you when a threshold has been reached. 35

36 1. Begin by creating the actual trigger condition with a double value comparison. a. Click the plus sign. b. Select Add Double Value Comparison. The trigger condition fields expand, and you can enter a double value comparison trigger. 2. Include the threshold in the trigger condition. An alert will be triggered when the threshold value is reached. Single alert when multiple devices go down There are situations when multiple devices go down, and instead of getting an alert for each device, you only want to receive one alert that notifies you that something is wrong. A multiple device alert is closely related to a dependency alert, where Orion detects a down status for a network element because an object in front of the network element is down. For example, if a router goes down, Orion detects that all servers behind the router are also down because Orion cannot communicate with the servers. But, the servers behind the router are still physically up, they are just reported as down. In this case, the alerts triggered can be numerous and inaccurate. You can create a dependency between the router and servers so that when the router goes down, only a single alert is triggered. With a multiple devices go down alert, the devices reported as down are actually down. For example, you have an application monitor for your Custom Corporate System that monitors 15 services. Each service may go down briefly for various valid reasons. However, when more than 10 services are down, you know something is wrong and you need to take action. The following alert checks the up and down status of all components, but only alerts when 10 or more components are down at the same time. 36

37 Use groups You can also use groups to alert you when multiple devices go down. In this approach, create a group and assign related devices to it. In this scenario, an alert is only triggered when all devices in the group are down, and not just a subset of devices. Using groups saves you time when configuring alerts, and is also useful when you have load balanced devices and you only care when all the devices are down, not just when some are down. Define object dependencies On your network, there are objects that are dependent on other devices to function. For example, all monitored objects behind an edge router are dependent on that router for connectivity. If the router has problems or goes down, any dependent devices will also appear to have problems, and you will receive status alerts about those devices as well. However, if you define those nodes as being dependent on the edge router, you will not get false alerts about the status of dependent nodes when the parent object has issues. Another example is when several IIS servers rely on the same SQL server for their database backends. If you define those nodes as being dependent on the edge router, or the IIS servers as being dependent on the SQL server, you will not get false alerts about the status of dependent nodes when the parent object has issues. When you use dependencies, consider the following: Dependencies themselves do not prevent alerts from being triggered. Instead, dependencies prevent the child objects from going down, and as a result, down node alerts are not triggered. By 37

38 using dependencies, status alerts are prevented from triggering, but all other alerts are triggered. A child object can have several parents, and the child's status only turns to Unreachable if all parents in all defined dependencies go down. In this example, you will create a dependency between an IIS server and its SQL database backend. Creating this dependency means that if the SQL database goes down, any IIS server alerts will be suppressed, although the IIS application will continue to be monitored. 1. Click Settings > All Settings in the menu bar. 2. In the Node & Group Management grouping, click Manage Dependencies. 3. Click Add New Dependency. 4. Select the parent object or group, and click Next. 5. Type a Dependency Name, select the child entities, and click Next. 6. Review the settings for the dependency. If there are active alerts on child objects, they are listed on this view. 7. Click Submit. The dependency appears on the Manage Dependencies page. 38

39 You can also display the dependency on custom views in the Orion Web Console. Event correlation alerts (Y after X) You may have some conditions you want to alert on if Y happens after X. For example, you might want to receive an alert if there is a problem with your failover. In this example, if node A goes down, the alert waits 5 minutes, and then checks that node B is up. If node B is not up, the alert is triggered. Make sure to select a And then after value, which is located between the alert sections. Then, in the secondary section, configure the actual trigger condition so that Node Name is equal to Node B and Node Status is equal to Down. 39