Live Health Catching Business Service Failures Before They Catch You

Transcription

1 Focus on Value Live Health Catching Business Service Failures Before They Catch You When your business is an e-business, your systems and applications are your storefront, your sales force. When they break, your business comes to a full stop. The Live Health Suite, part of the ehealth family of business service management solutions, lets you keep close watch on your missioncritical infrastructure. Live Health provides instantaneous feedback on trouble spots, telling you where the problems are, when they started, and their severity. It also points out growing problems before they become failures, allowing you to take proactive action, and keep your business running smoothly. Live Health includes three powerful tools: Live Exceptions a detailed list of alarms for the systems and elements you need to monitor. The Live Exceptions Notifier sends alarm notifications to IT personnel via or pager. Live Status a visualization tool that provides an immediate, at-a-glance picture of each business service element and its status. Live Trend a real-time trend monitoring tool for affected elements. Together, these applications help you identify problems proactively, and take you from identification through notification and resolution, getting your business back on track. Live Health enables you to track performance in real time, identifying and addressing problems before they effect your business. Real-time vs. Downtime Live Health allows you to react immediately to service problems, identifying them by location and severity. This allows you to zero in on the business-critical problems first, monitoring them in real-time to gauge your resolution measures. It also highlights impending problems and danger points, allowing you to shift or augment resources before the need becomes critical. Live Health can help you: Quickly identify real problems by receiving a single alarm for persistent or recurring outages and slowdowns (via Live Exceptions). Send alarm notifications, via or pager, to appropriate IT personnel worldwide (via Live Exceptions Notifier). See service slowdowns and failures in real time, including their location, potential interdependencies, and severity (via Live Status). Drill down to real-time trend reports to investigate problems with a specific element (via Live Trend). Drill down to other ehealth reports for a historical context on current problems. September 2005

2 Live Exceptions Overview Live Exceptions is the engine that powers the Live Health Suite. Live Exceptions identifies performance problems and service degradations and raises alarms if problems occur. Out of the box, Live Exceptions contains a wide assortment of default technology-based monitoring profiles, rules, and thresholds that you can tailor to your own specific infrastructure. Live exceptions includes: Intelligent Alarm Processing The Notifier The Live Exceptions Browser Filtering and Collapsing Alarms Alarm Profiles Acknowledgments and Annotations Drilldowns Fault Manager Intelligent Alarm Processing Instead of generating multiple redundant alarms for a single incident, Live Exceptions raises one alarm that remains active while the problem is present. This reduction in alarm noise lets you find, diagnose, and resolve problems without being distracted by repetitive alerts to the same problem. When a problem occurs for an IT resource, that resource sends continuous traps for that problem. Most network management systems treat these repeating traps as individual events, resulting in huge scrolling screens and endless log files. When Live Exceptions receives a trap, however, it processes the trap information and raises one alarm for that problem. If Live Exceptions receives repetitive traps for that same problem, it simply keeps that alarm raised and filters out the noise of repetitive messages. Likewise, when a typical management system monitors for service violations, it will raise alarms when a performance variable for an element exceeds a threshold. Often, however, one instance of a threshold violation is not a reliable indicator of a performance problem it may be a random spike. Live Exceptions solves this dilemma by raising an alarm only when the problem persists for a certain duration. This reduces the noise and distraction of alarms for transient events that occur then disappear immediately. Not all transient events are inherently benign, however, so Live Exceptions can be configured to raise alarms immediately for situations that concern you. This duration-control feature is also useful for reducing the distraction and noise of flapping alarms instances when a problem appears, disappears, and reappears, over and over, raising a new alarm at each problem occurrence. The Notifier The Notifier is another key component of Live Health. It takes action when alarms are raised, cleared, or acknowledged immediately notifying anyone, anywhere, and letting you control notification frequency. The Notifier can send notification to the appropriate IT personnel; it can also run scripts, such as pager and cellphone programs, or other commands that take immediate action to help manage expanding and escalating situations. Live Exceptions also has an auto-acknowledge script that allows it to automatically acknowledge an alarm when it is raised; this feature lets you save time when you have an established procedure for addressing certain alarm types. This auto-acknowledge script can also be the basis for more customized Notifier actions or used to change default values. For example, you could change the default message in the Live Exceptions Browser Notes annotation field so you can identify the IT team member who typically responds to that alarm type. The Live Exceptions Browser The Live Exceptions Browser (Figure 1) is the user interface to Live Exceptions alarm information. The Browser is a tabular readout of all problems for the elements managed by an ehealth system, along with the type of error and the severity level. It provides a realtime view of the performance of your IT infrastructure. You can customize the information presented in the Live Exceptions Browser to tailor the information you want to see. Most columns sort alphabetically; you can sort severity by ascending or descending levels. You can also rearrange, resize, and hide columns. By default, the Browser displays active alarms. You can also view historical alarm information by specifying the time period you want to view. The Live Exceptions Browser lets you prioritize a problem by its level of criticality, fixing the serious matters first and getting your resources up and running as quickly as possible. Because the Live Exceptions Browser can display alarms for the entire organization based on any criteria you specify (group, system, application), it can help you 2

3 identify situations in your environment that must be resolved. Figure 1. The Live Exceptions Browser Filtering and Collapsing Alarms To effectively interpret the alarms you may see, Live Exceptions offers a versatile set of filtering capabilities. You can filter and collapse alarm information to show only the alarms you care about. By reducing the number of alarm rows in the Live Exceptions Browser, filtering and collapsing let you concentrate on the alarms that matter most to you. Filtering. Filtering removes any alarms you don t want to see from the Live Exceptions Browser. If you are responsible for all elements in the Boston area, you can filter the alarms to show only those for Boston elements. Under the Filter tab (Figure 2), select Group Name and type in the group name Boston. The browser displays only alarms for elements in your Boston groups you now have a comprehensive picture of what is happening in Boston. Figure 2. Filtering Menu 3

4 Collapsing. When you collapse alarms, the Browser combines alarm rows containing common information. The number shown in the Alarm Count column now indicates how many alarms are active for that collapsed element. For each alarm row, you can see that the alarm represents one or more occurrences and drill down for more details if needed. Condensing the alarms allows you to easily see how many instances of this problem exist while providing you the information to help you keep an eye on the rest of your infrastructure resources. For example, if you want to see one alarm row for each unique element name, collapse on everything except Element name. The Browser now displays one row for each element name and combines all the alarms that element may hvae. The Alarm Count field now shows how many alarms exist for that element. Alarm Profiles Live Exceptions uses rules and profiles to trigger alarms. Rules define the conditions to monitor, their thresholds, their duration, and their severity they control when Live Exceptions raises an alarm. Rules. There are four types of rules: Availability Generates an alarm when an element is unavailable. Reachability Generates an alarm when an element does not respond to a ping or an SNMP Get request. Variable Generates an alarm when data for a specific performance variable exceeds service thresholds. These alarms are based on three condition types: Time over Threshold; Deviation From Normal; and Time Over Dynamic Threshold. Event (trap) Generates an alarm when Live Exceptions receives traps. Availability, reachability, and variable rules are processed for alarm conditions once during each polling interval (default is five minutes). Event rules are processed when traps are received. Live Exceptions keeps an alarm active until the condition specified by the rule no longer exists or is manually cleared. Profiles. A Live Health profile is a set of alarm rules that ehealth applies to groups or group lists of elements. Each default profile detects different problems; it is important to apply profiles that detect problems you need to see. Service level agreements (SLAs) are common between IT departments and the users they serve. A structured service-level management approach is necessary to meet your SLAs. Service providers and enterprise users can use the Live Exceptions Browser as part of real-time service monitoring alarm profiles can be set relative to your service levels so you know when you re nearing an SLA violation and can take steps to restore performance before users complain or you incur contractual penalties. Thresholds are important even in less formal settings. For example, you may want management informed of serious problems as they occur, but not notified of every minor alarm that IT addresses. You can create a separate profile for management with a higher threshold than IT uses, so that only the more serious issues are escalated to management. Profile Customization. As a best practice, begin monitoring your groups using the default profiles. Based on the alarms you see, you may decide you need to create your own alarm rules and profiles to better suit your specific infrastructure. Live Health administrators can create their own profiles and rules to reflect their specific environment. For example, you may be responsible for only a certain set of resources. You have no need to know about alarms generated, through a default profile, for other resources. To eliminate unneeded alarms, you might want to copy and modify the default profile by deleting the alarm rule for conditions you don t care about. You may also want to lower thresholds in order to give yourself more time to rectify a problem that may endanger your SLAs. In fact, if you have established meaningful thresholds for performance problems and defined them in your ehealth report service profiles, you can create customized Live Health profiles based on that data. For more information on Profiles, refer to the Live Health Profile Descriptions, on the Device and Technology Certification page of the Concord Support Web site ( 4

5 Acknowledgments and Annotations Live Exceptions can show you and other users that a problem is being addressed and how. You can view the Ack (acknowledgement) and Assigned columns of the Live Exceptions Browser to see status or information about problem resolution. This information can help reduce the number of phone calls and s you receive requesting problem status. Acknowledgments and annotations are visible to all Live Exceptions users and on the Business Services Console. Acknowledging an Alarm. Acknowledging an alarm shows that a problem has been discovered, noted, and is being addressed. When an alarm is acknowledged, the following actions occur: A check mark appears in the Ack field; your Web user account name appears in the Ack User field; and the day and time you acknowledged the alarm appears in the Ack Time field. Annotating an Alarm. Live Exceptions provides fields that can be used to annotate an alarm; these annotations appear as columns in the Live Exceptions Browser. The default labels for these columns are: Ticket #; Status; Assigned; and Notes. These are freeform text fields that you can use to provide additional information about the problem or resolution state. You can also change the column names if you prefer alternate column headings. Annotation provides you with additional detail on an alarm. This detail is a snapshot of problem status, possible causes, and possibly an estimated time to repair. Drilldowns You can select an alarm and drill down to a variety of reports to obtain more information about that resource or problem. The following standard drilldowns are available from most alarms: At-a-Glance Reports allow you to compare key performance indicators for an element, helping you to investigate root causes. Trend Reports show a history of performance for an element or a user-defined group over a period of days, weeks, or months showing you how today s problem relates to history. Live Trend provide a real-time charting tool for monitoring elements polled by ehealth. From an alarm, you can drill down to a Live Trend chart for a real-time analysis of the element. Groups Live Health monitors resources and displays alarm information based on groups and group lists. A group contains all the elements for something you want to monitor as a unit, such as a department, an office, or an application. Group lists are collections of these groups. Organizing your resources into logical groups helps you manage your resources most effectively. You can group elements by location, business function, or technology, whatever makes the most sense for your situation. For example, you may want to create a group for all the elements in your engineering department, and another for all the elements in the Boston office, regardless of department. Creating logical groups makes it easy to organize and visualize specific business applications or IT resources, allowing you to see where trouble is starting and how severely specific services are impaired. Security Considerations Live Health has two types of users: administrators and general users. An administrator can perform tasks such as creating and modifying rules, associating profiles to subjects, and creating notifications. General users can view alarms but have minimal management capabilities. These levels of permissions allow you to establish access policies and SLA policies controlling how much control a user has to change Live Health management policies. You can also enhance security by using Web user account permissions to filter the types of elements and groups that a user can see. For example, if you limit a user to see only LAN/WAN elements in the group named Sales, the user s Live Exceptions Browser will show only the alarms for that technology and the elements in that group. In the case of an ISP using Live Health, Web user account permissions can bet set such that the ISP s administrator for Smith Corporation can only see and monitor information for Smith Corporation, ensuring the privacy of the ISP s other customers and allowing the administrator to focus solely on the Smith Corporation. Fault Manager Fault Manager is a key component of Live Health that can receive traps sent to the ehealth system. ehealth is certified to receive traps from a number of sources; you can also add your own trap sources. For a list of certified trap sources, refer to the Certified Devices support page on the Concord Web site: 5

6 Live Status and Live Trend Live Status Live Status is a graphical visualization tool that provides the real-time status of all of your business resources. Using Live Status, you can identify problems as they occur, locate where the trouble is occurring, and see the severity of the problem. The Live Status diagram is a fishbone-style visualization (Figure 3). The horizontal spine in the center is the highest-order view of resources (a group list). Each rib off the spine is a set of related resources (a group). Each rib has one or more icons that represent an element. The icon s color represents the operational status of the element. Live Trend With Live Trend you can seamlessly launch realtime monitoring on affected elements to view performance patterns of your IT infrastructure. Live Trend s real-time diagnostic trending helps you determine the nature of a failure or slowdown by monitoring multiple variables in real time. It also allows you to monitor continuous real-time performance to determine whether your corrective action is working. For a majority of alarms (except traps), you can select the alarm and drill down to Live Trend. Live Trend creates charts that monitor the elements you are polling using ehealth (Figure 4). You can use Live Trend charts to plot: Element trends A single element with multiple variables, which can indicate element problems and their possible causes Variable trends A single variable for multiple elements, which can highlight problems, such as response time, for multiple elements. Live Trend updates the charts each time ehealth polls the elements. You can display as-polled data, with charts updated every five minutes, and fast-sampled data, with charts updated every 30 seconds. You can also preload the chart with up to 48 hours of history. Live Trend can be launched at any time, not just as a drilldown from an alarm. Figure 3. Sample Live Status Diagram Live Status can show you the business infrastructure for the company, a location, an organization, or a business service anything you want to monitor. Visualization is the key to business service management. With Live Status, managers have an immediate, at-a-glance picture of current business performance in a graphical display. They can identify problems as they begin, understand severity, prioritize troubleshooting efforts, and determine the impact to the business. As a live window to your business activity, Live Status can help you correlate all infrastructure and application problems in one comprehensive diagram, allowing you to quickly evaluate the performance of your key business services and resources. To learn more about Live Status, see the focus topic Visualizing Your Business Services Using Live Status. Figure 4. Sample Live Trend Chart 6

7 Case Study: Keeping Customers by Maximizing Service Uptime A major international telco s end users were complaining that the network is slow, saying they could not access critical applications in a reasonable amount of time. Since customer satisfaction was critical to this company s continued growth, no instance of substandard service was considered acceptable. To address this problem, the company needed to be proactive, finding response time problems before users were effected, and addressing them immediately any time of the day or night. The company installed Live Health. Almost immediately, the Live Exceptions Browser identified several critical system problems. Drilling down At-a-Glance and Trend reports on the most critical items told the company that server disk utilization was dangerously high, and several WAN links were overutilized the major cause of the poor performance. A few days later, a critical application server began experiencing problems late on a Saturday night. Live Exceptions identified the problem, and using the Notifier, immediately paged the on-call IT personnel. When IT personnel checked in, Live Status revealed that the server was approaching maximum capacity, and critical applications were running slow enough to draw complaints from users. Using Live Exceptions, they drilled down into the heart of the problem with an At-A-Glance report, which illustrated the out-of-bounds conditions, how they related to other system functions, and how long they had been out of bounds. A Trend Report (Figure 6) showed them how severe the CPU s condition was. Figure 5. At-A-Glance Report Showing LAN Links Approaching Critical Usage Levels. Figure 6. Severely Erratic CPU Performance. Live Trend provided up-to-the-minute views of system performance to contrast with recent historical data; the Live Trend data reinforced the problem diagnoses. By adding hardware and reducing the number of processes and applications running on that particular server, along with minor operating system and application tuning, they were able to bring application response times back down into their acceptable range. Running new Trend Reports and monitoring Live Exceptions for recurring alarms assured them that peak performance had been restored to their end-users and a possible critical failure had been avoided. 7

8 Case Study: Reducing Fire Drills Through Proactive Monitoring A state agency found themselves handling too many IT crises. The IT department was in reactive mode, running from failure to failure and barely keeping ahead. With limited staff and budget, they needed to be able to spot problem areas and eliminate them before they became business-interrupting events. The IT department installed Live Health to help address their problems. Immediately, their problematic systems began generating a high number of alarms. Because of this alarm volume, they raised the default Live Exceptions alarm thresholds so they could identify and correct the worst, most business-critical issues first. Live Exceptions real-time monitoring features showed their IT personnel that systems were the root of their most serious problems. Live Exceptions brought multiple problems to light: overutilized CPUs, disk space problems, and memory shortages. System CPU utilization was high and constant CPU spikes were occurring in vital systems critical problems that they had never known about before. These problems are often the root cause of widespread slowdowns and failures; until Live Exceptions, the agency was only treating the symptoms, not the illness (Figure 7). Drilling down to At-a-Glance reports showed that the CPUs on their critical end-user-facing systems were being overwhelmed by the amount of processing they were being asked to perform. Large, memory-intensive applications were starving out other business-critical applications, resulting in unacceptable access times and slowdowns for end users. By upgrading CPU capacity, moving some memory-intensive applications off of the user-facing servers, and cleaning up unused files, the IT team brought performance levels back to specified levels. Continued monitoring of the upgraded systems showed a significant drop in number and severity of alarms for their business-critical user-facing systems. Live Exceptions enabled the agency to track down and correct the root causes of their problems and correct them before they became serious enough to cause business-halting failures. The IT team now keeps Live Exceptions running continuously, allowing them to monitor system elements and address potential issues before they cause a failure or a crippling application slowdown. The number of fires they have to deal with has been substantially reduced. The key priority for IT management is to find problems and resolve them before users report them. They want to catch network issues early, before services slow down and executives reach for the phone. Because of Live Health, they are now meeting this goal. Figure 7. Live Exceptions Browser 8