Fifty Critical Alerts for Monitoring Windows Servers Best Practices

Fifty Critical Alerts for Monitoring Windows Servers Best Practices The importance of consolidation, correlation and detection Enterprise Security Series White Paper 8815 Centre Park Drive Publication Date: Jan 31, 2009 Columbia MD 21045 877.333.1433

Abstract How important is it for your organization to stop an intrusion immediately? How important is it for your organization to keep critical applications up at all times? This document identifies and describes the most important events generated by your Windows servers so they can be addressed and corrected by IT personnel in the most efficient manner. The strategic benefit of monitoring these critical events combined with a robust resolution strategy is significant reduction of IT costs while ensuring increased service availability and enhanced security of your enterprise. The information contained in this document represents the current view of Prism Microsystems Inc. on the issues discussed as of the date of publication. Because Prism Microsystems must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Prism Microsystems, and Prism Microsystems cannot guarantee the accuracy of any information presented after the date of publication. This document is for informational purposes only. Prism Microsystems MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, this paper may be freely distributed without permission from Prism, as long as its content is unaltered, nothing is added to the content and credit to Prism is provided. Prism Microsystems may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Prism Microsystems, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred. 2009 Prism Microsystems Corporation. All rights reserved. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Prism Microsystems 2

Overview IT Managers today are responsible not only for optimizing IT operational performance but also ensuring the security of critical systems, and doing both effectively can often be a difficult and time consuming challenge. Event log data, which provides an overview of all activity on a company s systems, can be leveraged to significantly reduce the headache of maintaining server uptime and detecting and preventing security intrusions in a timely manner. When a service-impacting event occurs on a critical server, quick detection and notification can enable faster issue resolution, and in many cases quick action can prevent a service disruption. But how do you get instant event notification to the right person at the right time? Manually collecting and analyzing event log data for discrepancies is not only impossible in a large environment given the sheer volume of information, but is also not a timely or practical solution you would essentially be searching for the proverbial needle in a haystack. The key is to identify events that are most relevant to detecting potential operational or security issues and then automating the creation of alerts on these events to notify appropriate personnel in real-time. These alarms dramatically shorten service outage times and lower costs by reducing the time it takes IT Operations to respond to, and resolve, the issue. EventTracker from Prism Microsystems is a powerful and reliable Security Information and Event Log Management solution that automates the real-time collection and consolidation of all enterprise logs and enables centralized monitoring, analysis, correlation and reporting. EventTracker provides hundreds of preconfigured alerts for critical issues to enable immediate resolution. Custom alert rules, specific to your environment, can also be easily and quickly created to minimize time spent chasing false alarms and routine events. These alerts notify the right person at the right time through multiple notification methods including RSS, email, pager, audible alarm or an SNMP trap notification to an enterprise console such as HP OpenView or Tivoli. This White Paper details the top 50 conditions and alerts that Prism Microsystems believes are absolutely critical to monitor on critical servers. Automating the monitoring of your critical systems produces the best of all worlds. It is less expensive and resource intensive than manual processes and it frees resources to work on other priorities, while ensuring that problems are detected faster and addressed sooner. Prism Microsystems 3

50 Critical Conditions EventTracker includes these fifty alerts that are critical for optimizing IT performance and ensuring the security of your enterprise. Alert Name Description 1 Disk Space is critically low This alert is generated when the system is running low on logical disk space. By default, 80% full is considered a warning point; the threshold is however a configurable parameter. 2 Critical service is not running Monitoring the availability of critical services is vital for remote server diagnosis and problem resolution. Critical services being stopped during unusual hours of operation can also be a warning sign of an intrusion. 3 Critical service could not be started This alert indicates that a critical service configured in EventTracker for an automatic restart has failed to restart. An alert of this nature needs immediate action from system administrators. 4 Detected high memory usage This event is generated when the memory usage exceeds a defined threshold and indicates that a system administrator should examine what processes are consuming excessive memory. 5 Detected software <Some S/W> has been installed on this system Monitoring unauthorized software changes aids in early intrusion detection. 6 EventTracker agent service failed This alert notifies that the EventTracker agent service has failed and could not be restarted. Events from the system during this downtime could be lost unless guaranteed event delivery mode has been configured. 7 Domain policy changed This alert indicates a successful change to the Windows Active Directory security policies. This alert is also triggered when Group Policies are applied. 8 Active Directory: Group policy changed 9 Runaway CPU process A process is consuming high CPU 10 Runaway Memory Process A process is consuming too much memory This alert indicates that a group policy or an OU policy has changed. It may change the behavior of active directory user permissions. A CPU-intensive process can adversely affect server performance, slowing other processes to a crawl and even bringing the server to a halt. These alerts are critical for continued reliable performance and minimizing downtime. This alert suggests that the process running may have a memory leak. It s important to monitor such a process closely. Prism Microsystems 4

Alert Name Description 11 Software uninstalled from a system Installation of unauthorized software packages can increase system vulnerability resulting in virus attacks. 12 Excessive logon (Event ID 529) failures This event indicates an attempt to logon using an unknown user account or a valid user account but with an incorrect password. Concurrent occurrences of these events represent an attack on the enterprise. 13 Excessive audit failures on a system Excessive audit failures on a system or a particular resource on a system is an indication of a potential intrusion or violation of a security policy 14 Excessive access failures by a user Logon failures using accounts that have been locked can result in this intrusion alert. 15 Excessive access failures on a specific computer 16 Excessive access failures across your enterprise Sophisticated scripts run by hackers use a variety of user name and password combinations to get past windows security. Logon failures on each system should be monitored closely. Enterprise-wide repeated logon failures in a short interval of time are a sure sign of an attempted intrusion. 17 Excessive file deletes on a computer This alert indicates that data on a critical server has been compromised. 18 Excessive VPN connection failures This alert indicates that someone may be persistently trying to access your VPN server to come into your network 19 Too many concurrent requests to your website 20 Excessive logon attempts from a particular IP address 21 Excessive Ping failures Several systems are not reachable 22 Excessive remote connections established on a local network service (port) 23 Excessive User lockouts in your enterprise (ID=539) This alert indicates that too many users are accessing your company website at this time. Performance may be impacted. A number of successive logon attempts from a single remote IP address are an indication of hacking activity. The source of attack should be identified and blocked to prevent further attack. Monitoring responses to ICMP packet requests and receipt time of ICMP packets from each destination is essential for network performance tuning. Numerous unknown processes attached to local ports are sure signs of intrusion. This event indicates a logon attempt for a locked account. This event can indicate that a password attack was launched unsuccessfully resulting in the account being locked out. 24 High CPU utilization Continuous high CPU usage is an indication of potential problems or an overloaded system. Prism Microsystems 5

Alert Name Description 25 IIS: Logging Shutdown IIS logging shuts down when a disk-full error is encountered. Administrators can either free some disk space on the logged drive or move log files to another location. 26 IIS: Server Stopped When users access an application from an ASP page, the underlying COM+ application fails if there is no user logged on to the IIS console. Administrators can quickly resolve this issue by specifying appropriate user accounts. 27 IIS: World Wide Service Terminated This problem can occur if the Microsoft Distributed Transaction Coordinator (MSDTC) has been configured to use a certain range of ports for incoming requests, but the range that has been specified is not large enough. 28 ISA Server: All Port Port Scan detected 29 ISA Server: Excessive Win Sock Applications open This alert notifies that an attempt was made to access more than the pre-configured number of ports. One can specify a permissible threshold, indicating the number of ports that can be accessed. This alert is generated when the network system has run out of socket handles. WinSock applications that open and close sockets often without closing them properly can cause this error. 30 ISA Server: Failed to start service This alert indicates that an ISA server service failed to start. Analysis of associated windows events can help identify the cause. 31 ISA Server: Land attack This alert notifies that a TCP SYN packet was sent with a spoofed source IP address and port number that matches that of the destination IP address and port. This attack can cause some TCP implementations to go into a loop that crashes the computer. If this alert occurs, server policy rules or packet filters should be configured to inhibit traffic from the source of the scans. 32 ISA Server: Network communication device may be down 33 ISA Server: Out of band attack detected This event refers to a problem that has occurred at the datalink level, or if the link connection has been cleared. One should check for errors logged for data link or data communication hardware devices. This alert is triggered by an out-of-band denial-of-service attack attempted against a computer protected by ISA Server. If mounted successfully, this attack causes the computer to crash or causes a loss of network connectivity on vulnerable computers. Prism Microsystems 6

Alert Name Description 34 ISA Server: Ping Attack This event occurs if a large amount of information has been appended to an ICMP echo request packet. If the attack is successful it will cause a kernel buffer overflow and system crash. If this alert is received, one should create a protocol rule that specifically denies incoming ICMP echo request packets from the Internet. 35 ISA Server: Port scan detected on a well known port This alert indicates that an attempt was made to scan wellknown ports on a computer to detect services running on those ports. If this alert occurs, one should identify the source of the port scan and check the access logs for indications of unauthorized access. If indications of unauthorized access are present, the system should be considered as compromised and appropriate action should be taken. 36 ISA Server: Spoof Attack A spoof attack occurs when packets are received on an IP address that is not reachable via the interface. If logging for dropped packets is set, one can view details in the packet filter log 37 ISA Server: UDP attack This alert occurs when there is an attempt to send an illegal UDP packet. A UDP packet that is constructed with illegal values in certain fields will cause some older operating systems to crash when the packet is received. If the target machine does crash, it is often difficult to determine the cause. Steps against this intruder activity include setting up a packet filter or policy rules to inhibit traffic from the source of the intrusion. 38 MSExchange: ADC service stopped This can be merely an informational event or it could imply a service shutdown due to unexpected errors. If the service fails to start manually, administrators should analyze related errors and warning messages in order to resolve the issue. 39 MSExchange: Database maximum size reached 40 MSExchange: IS Service cannot be started Normally logged after database has shutdown for reaching its capacity. This message generally means the server requires an upgrade to Enterprise Server or you should run utilities to free up space. A fix from Microsoft enables database extension by 1 GB. A critical error indicating that Microsoft Exchange Information Store service failed to initialize. 41 MSExchange: Log disk is full This issue can occur with insufficient disk space on the drive that contains the databases that are being mounted. 42 MSExchange: Server cannot handle influx of mail This error alert is generated when another MTA service is attempting to send to an address that does not exist on the local server. It might be required to cleanup AD with ADSI and rebuild the server. Prism Microsystems 7

Alert Name 43 MSExchange: Unable to start exchange server Description This error can result from a variety of faulty applications such as iexplore, dns, mmc, winlogon etc. Requires application updates. 44 SQL Server: SQL server stopped Untimely service shutdown events of SQL server and SQL server agent can be warning signs of intrusions. 45 SQL Server: Transaction log full These messages indicate that SQL Server cannot allocate additional free space, needed for expanding the database 46 SQL Server: Backup failed Failing to perform backups within the given time frame exposes the server to the risk of data loss. 47 System is not reachable, it may be down Monitoring unreachable destinations is vital for network management. 48 System Resource exhausted This is a critical audit event indicating loss of audit records due to overwriting of earlier records or due to cessation of auditing, depending on the audit policy established; or by internal event queues exceeding their maximum length. 49 Backup failed This alert indicates that a backup operation has failed for some reason and immediate attention may be required. 50 Critical Web URL is not reachable This alert indicates that certain critical Web URLs may not be accessible. It may indicate that your website is down. Prism Microsystems 8

Summary The complexity of IT infrastructures today makes it difficult for overworked IT departments to respond to critical problems in time. With EventTracker, IT managers are able to configure real-time alerts on the most important events in their IT organization to proactively prevent an intrusion, slow-down, or outage, while freeing up valuable resources to attend to other responsibilities. Prism Microsystems 9

The EventTracker Solution The EventTracker solution is a scalable, enterprise-class Security Information and Event Management (SIEM) solution for Windows systems, Syslog/Syslog NG (UNIX and many networking devices), SNMP V1/2, legacy systems, applications and databases. EventTracker enables defense in depth, where log data is automatically collected, correlated and analyzed from the perimeter security devices down to the applications and databases. To prevent security breaches, Event Log data becomes most useful when interpreted in near real time and in context. Context is vitally important because often the critical indications of impending problems and security violations can only be learned by watching patterns of events across multiple systems. Complex rules can be run on the event stream to detect signs of such a breach. EventTracker also provides real-time alerting capability in the form of an email, page or SNMP message to proactively alert security personnel to an impending security breach. The original Event Log data is also securely stored in a highly compressed event repository for compliance purposes and later, forensic analysis. For compliance, EventTracker provides a powerful reporting interface, scheduled or on-demand report generation, automated compliance workflows that prove to auditors that reports are being reviewed and many other features. With pre-built auditor grade reports included for most of the compliance standards (FISMA, HIPAA, SOX, GLBA, and NISPOM); EventTracker represents a compliance solution that is second to none. EventTracker also provides advanced forensic capability where all the stored logs can be quickly searched through a powerful Google-like search interface to perform quick problem determination. EventTracker lets users completely meet the logging requirements specified in NIST SP 800-92 Guide To Computer Security Log Management, and additionally provides Host Based Intrusion Detection, Change monitoring and USB activity tracking on Windows systems, all in an off the shelf, affordable, software solution. EventTracker provides the following benefits A highly scalable, component-based architecture that consolidates all Windows, SNMP V1/V2, legacy platforms, Syslog received from routers, switches, firewalls, critical UNIX servers (Red Hat Linux, Solaris, AIX etc), Solaris BSM, workstations and various other SYSLOG generating devices. Automated archival mechanism that stores activities over an extended period to meet auditing requirements. The complete log is stored in a highly compressed (>90%), secured archive that is limited only by the amount of disk storage. Real-time monitoring and parsing of all logs to analyze user activities such as logon failures and failed attempts to access restricted information. Alerting interface that generates custom alert actions via email, pager, beep, console message, etc. Event correlation modules to constantly monitor for malicious hacking activity. In conjunction with alerts, this is used to inform network security officers and security administrators in real time. This helps minimize the impact of breaches. Various types of network activity reports, which can be scheduled or generated as required for any investigation or meeting audit compliances. Host-based Intrusion Detection (HIDS). Role-based, secure event and reporting console for data analysis. Prism Microsystems 10

Change Monitoring on Windows machines USB Tracking, including restricted use, insert/removal recording, and a complete audit trail of all files copied to the removable device. Built-in compliance workflows to allow inspection and annotation of the generated reports. Prism Microsystems 11

About Prism Microsystems Prism Microsystems delivers business-critical solutions to consolidate, correlate and detect changes that impact the performance, availability and security of your IT infrastructure. With a proven history of innovation and leadership, Prism provides easy-to-deploy products and solutions for integrated Security Management, Change Management and Intrusion Detection. EventTracker, Prism s market leading Security Information and Event Log Management solution, enables commercial enterprises, educational institutions and government organizations to increase the security of their environments and reduce risk to their enterprise. Customers span multiple sectors including financial, communications, scientific, healthcare, banking and consulting. Prism Microsystems was formed in 1999 and is a privately held corporation with corporate headquarters in the Baltimore-Washington high tech corridor. Research and development facilities are located in both Maryland and India. These facilities have been independently appraised in accordance with the Software Engineering Institute s Appraisal Framework, and were deemed to meet the goals of SEI Level 3 for CMM. For additional information, please visit http://www.prismmicrosys.com/. Prism Microsystems 12