Focus on Value Monitoring Microsoft Exchange to Improve Performance and Availability With increasing growth in email traffic, the number and size of attachments, spam, and other factors, organizations need to provide high levels of service and availability in their e-mail systems. Any shortcomings in e-mail response time can impact business. Losing e-mail access for even an hour can significantly affect an organization s bottom line. Concord s ehealth Application Insight Module (AIM) for Microsoft Exchange enables IT operators to monitor the performance and availability of their Microsoft Exchange servers and their underlying systems. The Exchange AIM extends Concord s SystemEDGE agent to actively manage and monitor the Microsoft Exchange Server. It provides standard, real-time snapshots of Exchange performance, availability, configuration, and virus scanning activity. Accessible anytime and anywhere from a Web browser, these status snapshots are ideal for troubleshooting your systems and tracking the behavior of critical Exchange servers. The AIM also verifies that all the components of the Exchange application service are working and responding within service thresholds. ehealth AIM for Microsoft Exchange integrates with Concord s ehealth Suite to deliver end-to-end, real-time content with historical context to meet growing business needs. Additionally, users can integrate with ehealth Live Health for real-time detection of performance problems, potential outages, and delay for the Exchange application as well as the underlying system and network infrastructure. The Exchange AIM: Monitoring Your Critical E-Mail Services If your e-mail is slow, your business is slow. The ehealth AIM for Microsoft Exchange continuously monitors Microsoft Exchange services and system resources, allowing you to optimize Exchange application and server performance. This results in the following benefits: Deliver real-time problem detection. Provide self-healing, corrective action. Identify potential viruses. Achieve intelligent capacity planning.
Deliver Real-Time Problem Detection Exchange problems often start off small. They may appear, at first, as unexpected peaks in traffic at the same time each day. Over time they may begin to affect the amount of disk space available on your server and the size and length of the email queues. They may even impact system memory and gradually affect overall capacity. Before long these problems have compromised server performance and limited your ability to meet the daily requirements of the business. Taking control of your network requires an automated monitoring solution with continuous, around-the-clock, 24x7 coverage. You can set thresholds unique to your environment and generate traps and alarms notifying you via e-mail or pager when those thresholds are violated. Concord s solution provides monitoring capabilities that allow you to see peaks in CPU, disk, and memory usage as well as various Exchange metrics before they impact the performance of your mission-critical Exchange servers. Controlling events ensures that your network is properly aligned with and fully capable of supporting your business needs. 100GB capacity. Of the 63GB used, the Private Store is using 54GB and the Public Store is using 9GB. Figure 1. Disk Usage The Footprint report also provides Exchange CPU and memory utilization over the last SystemEDGE measurement interval. This snapshot report can be used as a means of troubleshooting. In this case, Exchange is currently using 60% of the CPU. Of the available physical memory (1GB), Exchange is currently using 72%. The Exchange Footprint The Footprint and FootPrint Details Reports (shown in Figure 1 and Figure 2) are both available through AdvantEDGE View and provide a current snapshot of system resources consumed by the Exchange application. Footprint information is presented at varying levels of detail, ranging from summary values to those for individual folders in the Exchange store. The Exchange AIM monitors and reports back on Exchange CPU, memory, and disk resource consumption, and helps you to detect and correct resource exhaustion due to viruses, security incidents, and hardware failures. In Disk Usage, you can see that Exchange is consuming 63GB of the overall disk s Figure 2. Footprint Details 2
Analysis of footprint information is useful for anticipating and avoiding bottlenecks in CPU, memory, or disk usage that may impact Exchange performance. To see the total system CPU percentage in use for a query period and the percentage of time spent in idle, user, system, and wait modes, run an AdvantEDGE View CPU Statistics Query (shown in Figure 3). This report shows the overall performance statistics for each CPU in a server. Figure 3 shows that the CPU is idle 48% of the time, with users consuming 50% of available resources, and system processes consuming 2%. Figure 3. CPU Statistics Query Monitoring Variables in Real Time Proactively monitoring your business extends to e-mail services where users require continuous availability and high performance. AdvantEDGE View provides templates that you can use and customize to configure performance monitoring for your Exchange servers. The Exchange AIM, in association with SystemEDGE, can monitor, alarm, and report on the following variables: CPU Utilization, Memory and Disk Space. The AIM can identify CPU, disk, or memory shortages that can affect application performance. Information Store. This store is comprised of public folder messages (public store) and user mailboxes (private store). These stores can grow up to 16 GB for the Exchange 2000/2003 Standard edition or up to 16 TB for the Enterprise edition. Performance problems, including mail service stoppages, occur when the when these stores exceed their maximum size or when they fill the disk capacity where they reside. Message Queues. These queues contain messages waiting to be processed or delivered. You can configure the AIM to monitor these queues to watch for increasing sizes that exceed specified thresholds. Unusual changes in the size of a queue can indicate a problem with sending or receiving of e-mail, as well as potential malicious activity such as spamming or viruses. Threads. Every remote procedure call (RPC) process that is initiated in Exchange creates a thread. If the number of threads exceeds the configured limit, Exchange performance can suffer. SMTP. The Simple Mail Transfer Protocol server is an inter-server mail transport service. The Exchange AIM provides over 75 variables associated with SMTP server metrics that can be monitored using SystemEDGE. These metrics include badmail (improperly formatted mail), message counts, connection counts, categorizerrelated information, LDAP information, and delivery status notification (DSN), indicating that messages cannot be delivered. Log File Monitoring. SystemEDGE can monitor any ASCII-based log files for user-specified patterns such as error or warning messages. When a match occurs, SystemEDGE can send a trap and/or run a script or program to take corrective action or notify IT personnel. Windows Events. SystemEDGE can monitor the NT System, Application, and Security event logs for a user-specified string or a specific event ID. When a match occurs, SystemEDGE can send traps and/or notifications. In Exchange, examples of these dedicated NT events could include data write errors or data read errors, service failures, excessive virtual memory fragmentation, and space management events which indicate that a store has reached a maximum size. Directory Services Cache, DS Proxy. These variables help determine how effectively Exchange is caching information that is frequently queried from Active Directory. If the number of cache hits is 3
low relative to the number of LDAP searches performed, the server may be imposing an excessive load on the Active Directory server. ehealth Live Exceptions Profiles ehealth offers several Live Health profiles that you can use to monitor your Exchange resources for real-time performance degradations relating to delay, failure, and unusual workload problems. Live Health can raise alarms and notify IT personnel whenever a problem begins with your Exchange resources. Provide Self-Healing, Corrective Action To automate your Exchange environment, the Exchange AIM works with SystemEDGE to identify system and application problems in real-time and activate scripts to perform corrective actions. SystemEDGE s self-healing capability ensures that your systems and applications will continue to provide business services without fail. SystemEDGE runs specific commands or scripts when real-time problems are detected, such as: Restarting failed processes and services Sending SNMP traps to third party tools, allowing you to preserve your investment in your existing solutions Identify Potential Viruses New viruses emerge constantly. Many of these can defeat your existing virus detection software and overwhelm your Exchange server. Most Internet worms and viruses exhibit a pattern of behavior that typically results in substantial increases in the amounts of e-mail and Internet traffic. Rapid increases in the message queues and message traffic are good early indicators of a virus outbreak. The AIM provides information that allows these conditions to be detected. With System-EDGE s monitoring and action script capabilities, you can configure various ways to notify IT staff immediately before a virus spreads across your organization. To identify potential virus activity, you can set up monitoring for the following three variables: 1) SMTP connections. Many viruses and worms move through the Internet and corporate networks using SMTP, the protocol used for sending and Rebooting a system upon failure Removing core files and cleaning up full file systems Stopping CPU-intensive processes when CPU utilization reaches peaks levels Logging out unauthorized users SystemEDGE includes a pair of out-of-the-box applications (restartproc.exe and restartsvc.exe) that can restart a Windows process or service. SystemEDGE also allows you to leverage scripts and applications that you have developed internally in support of your Exchange processes. receiving e-mail in data networks, as their transport mechanism. The AIM can monitor and report the total number of active SMTP connections. A substantial and rapid increase in the number of new SMTP connections could indicate virus or worm attack and warrants further investigation. 2) SMTP queues. During message categorization and delivery, the advanced queuing engine sends all outbound mail through the SMTP queues of an SMTP virtual server. Viruses, spammers, DNS problems and the like can cause hundreds, sometimes thousands of e-mails to fill the SMTP queues. The AIM can monitor queue lengths and report on unusual message accumulation occurring in the SMTP queues. If there is a sudden and unusual increase in queue sizes, a worm or virus may be to blame. 4
3) Non-delivery reports (NDRs). A substantial growth in the number of undeliverable email to addresses that are not local often indicates that external, unknown clients are attempting to use email to gain entry to your network. If the recipient address cannot be resolved, Exchange sends an NDR to the originator of the message. If the originator is a fake address, the SMTP retry queues could fill with undeliverable NDRs, impacting the performance of your Exchange server. The AIM reports on the number of NDRs on your network, which can help detect attacks or intrusion attempts. Error conditions that may cause NDRs include: Local address could not be found Forwarding loop detected Ambiguous address Illegal address At-a-Glance Reports for Virus Monitoring ehealth Release 5.7 has an At-a-Glance Report for Exchange Virus information that shows the virus scanning activity for your server. It provides information about the number of messages and email folders that have been scanned, as well as details about the messages that have been cleaned and quarantined due to viruses. Figure 4 shows the Messages Processed and Messages charts of the Ata-Glance report. These samples indicate that over the report period, the Exchange server processed anywhere from 20 to 200 or more messages every five minutes, and at times, a very small number (0.035 messages) were cleaned of viruses. This report helps to baseline the typical virus processing activity of the Exchange server and identify changes or increases in virus cleaning or quarantines. If you observe a large increase in the ratio of cleaned and/or quarantined messages and the total messages processed, you may be encountering malicious attacks on the network. Other charts on a virus At-a-Glance report include: Files Scanned the total number of files (usually attachments) scanned for viruses. Files the total number of files cleaned or quarantined as a result of suspected viruses. Figure 4. Exchange Virus Info At-a-Glance Report Scanned Background the total number of messages and folders re-scanned in the background, usually following a virus update. Queue Length the length of the virus scanning queue, which holds requests for virus scanning. Queues lengths are typically zero or 1 because requests are processed very quickly. Queue lengths greater than 1 usually indicate a performance problem for the scanning software. Figure 5 shows a sample Queue Length chart where most of the activity is normal; a significant spike on March 9 that may indicate a problem for the virus software. Figure 5. Virus Scan Queue Length 5
Achieve Intelligent Capacity Planning In association with ehealth, the Exchange AIM and SystemEDGE deliver intelligent capacity planning data for your Exchange servers. By understanding how resources are being used and by whom, you can forecast your resource needs and identify when you may run out of capacity. With this information, you can make informed decisions about capacity requirements based on current trends as well as anticipated changes. Starting with a System Health report, you can leverage the Health Index charts and Exceptions reports to identify where the server hardware may be having capacity problems now. These charts can show you whether your servers are encountering CPU, memory, disk, and bandwidth issues. Schedule System Health reports to run daily and investigate problems reported on these charts. The Health report also includes optional supplemental reports called Capacity Projection and Capacity Provisioning which you can use to forecast growth based on the historical activity of your servers. These reports project memory, CPU, disk, and partition utilization to help you plan for additional system resources and future capital purchases. You should schedule these reports to run at the beginning of the month to review the previous month s activity. You can also use Trend Reports to examine Exchange capacity performance in detail. The AIM monitors many key variables that are critical for Exchange capacity planning such as: CPU Usage, Disk Space, Memory Number of users SMTP messages You could also run reports to review queue sizes, CPU utilization, or virus scanning activity over the past few months. Trend reports help you to visualize the activity so that you can identify when peaks in activity occur and baseline the overall performance history. For example, you might want to review a Trend report for the total disk space consumed by Exchange to determine how disk space consumption is changing over time. This can help you to plan for capital purchases such as larger disk drives for the Exchange servers. Figure 6 shows a Trend report for disk usage for an Exchange server over the past six months. (Although not shown in the report, the server has 200GB of total disk space.) The report shows that disk space has grown steadily from 100 GB to almost 120 GB, or 20% overall growth. For the 200 GB server, this indicates that the disks are 60% full. If the growth rate stays on track, in six months the disk usage would be 144 GB, or 72% full. In a year, the disk usage would be 173 GB, or 87% full. To avoid possible disk space problems in the near future, the Exchange administrator should plan to add about 50 to 100 GB more disk space in 6 to 9 months, before the disk usage reaches 90%. Figure 6. Trend Report of Exchange Total Disk Space 6
Supported Exchange Configurations Exchange AIM 2.0, released with ehealth 5.7, supports Exchange 5.5, 2000, and 2003 server platforms. The AIM also supports data collection in Mixed-Mode Exchange environments where 5.5, 2000, and 2003 Exchange servers coexist on the same network. Table 1 lists metrics that the Exchange AIM provides for enhanced performance and fault monitoring. Table 1. Performance Metrics Included in Exchange AIM Feature Exchange Server CPU, memory, and disk space Core server processes including information store, Message Transfer Agent (MTA), system attendant, and IIS. Administrators can use these statistics as an indication of overall MTA performance (how well messages move between Exchange servers, either to the local Information Store, connector, or remote MTA). Connector elements (Internet Mail Service, Lotus Notes cc:mail) that enable Exchange to communicate with other e-mail systems Mailbox stores disk footprint information including sizes for Exchange public and private mailbox stores Web service connections, logons, and uptime RPC Remote procedure call connections SMTP Server elements, including total bytes sent and received, total DSN failures and lookups SRS Site Replication Services for Exchange 2000/2003 servers in mixed-mode environments. SRS enables Exchange 2000 to emulate Exchange 5.5 directory services, replicating information between them. The AIM monitors SRS services that include Replication Updates, Replication Syncs Pending, Replication In Messages, and Replication Out Messages. IMAP Internet Message Access Protocol 4 connections POP3 Post Office Protocol 3 connections DS Directory Services cache and DSProxy, including Name Service Provider Interface (NSPI). These variables help determine how effectively Exchange is caching information that is frequently queried from Active Directory. ehealth AIM for Microsoft Exchange in a Clustered Environment By clustering Exchange, downtime due to hardware failures and maintenance activities can be reduced. A clustered Exchange environment involves one or more Exchange virtual servers distributed among two or more physical nodes. In a two-node failover cluster, two types of clustering are supported: active/passive and active/ active. Both types permit increased system availability for Exchange services. In active/ passive clustering, the cluster consists of one primary node and one secondary node. In this configuration, the primary node supports all clients while the secondary node is a backup server that is ready to take over whenever a failure occurs on the primary node. If the primary node fails, the secondary node picks up all operations and continues to service clients. In active/active failover clustering, both the primary as well as secondary node support Exchange services If one of the nodes fails, the other takes the full load with performance possibly reduced. With Exchange 2003 and the appropriate version of Windows Server, it is possible to build clusters consisting of up to eight nodes. In clusters with more than two nodes, only an active/passive configuration is supported. Running two Exchange virtual servers on the same node is not supported in clusters with three or more nodes. The ehealth AIM for Microsoft Exchange offers limited support for the active/passive type of cluster configuration up to the limit of nodes and virtual servers supported by the underlying platform (Exchange and Windows versions). Active/active clustering is not supported. In a cluster, the AIM provides basic information about the cluster itself, as well as the name and state of the current virtual server, if any, running on the node that is the target of a query. Information equivalent to that displayed in the Microsoft Cluster Administrator is also provided for all Exchange groups and resources in the cluster. Copyright 2005 Concord Communications, Inc. ehealth, the Concord Logo, Live Health, Live Status, SystemEDGE, AdvantEDGE and/or other Concord marks or products referenced herein are either registered trademarks or trademarks of Concord Communications, Inc. Other trademarks are the property of their respective owners. 7