BMC Performance Manager Active Directory Best Practices White Paper

BMC Performance Manager Active Directory Best Practices White Paper Problem The IT department delivers user authentication services to their internal and external customers. Users complain that they can t login to their systems or that the systems login response time is not acceptable. The IT department wants to maintain control over their own users and network resources while simplifying Active Directory administration. Solution The technology features that will be implemented in Active Directory monitoring will be based on the priorities that the business has expressed. The members of the IT department need clearly defined key availability indicators so that they know when and why the users were not able to login with their provided user credentials. The members of the IT department need clearly defined key performance indicators so that they know when and why the users were experiencing delays during the login process with their provided user credentials. The members of the IT department need to verify the functionality of the provided logon services from a client perspective. The members of the IT department need the objective data to prove to themselves and management that their time and budget is allocated to provide user authentication services with the highest ROI. Primary Message The BMC Performance Manager for Servers based Active Directory monitoring solution provides clearly defined key performance and availability indicators to measure the quality of the user authentication services. Description BMC Performance Manager for Servers gathers key performance indicators based on performance metric data, events and synthetic transaction in order to measure, verify the core services of Active Directory and alarm if the status exceeds or is less than a configured threshold. Monitoring the distributed Active Directory service and the services that it depends on helps to ensure consistent directory data and a consistent level of service throughout the organization. Product Highlights Gather, analyze and correlate key performance indicators based on Windows performance monitor metrics on each individual domain controller. Gather, analyze and correlate key performance indicators based on WMI objects on each individual domain controller. Gather, analyze and correlate key performance indicators based on Lightweight Directory Access Protocol (LDAP) objects on each individual domain controller. Gather, analyze and correlate key performance indicators based on the Active Directory topology. Gather, analyze and correlate key performance indicators based on the Domain Name Service. Verify and determine Active Directory s logical function from a client perspective by means of synthetic transactions. BMC PM for Servers - Active Directory Version 2.3 Page 1 of 30

Products involved BMC Performance Manger for Servers P for Windows 3.2.20 PKM for Microsoft Windows 3.9.20 PKM for Microsoft Windows Domain Services 1.5.02 PKM for Microsoft Windows Active Directory 1.6.00 P Wizard for MS PM and WMI 2.0.05 Contents Monitoring of Components... 3 Six Sigma Values... 3 Controlling the cost of your Active Directory Infrastructure & Monitoring Solution... 5 Device and Application Probing... 6 Event Correlation Module... 6 Notification Module... 6 Types of Monitoring and Monitoring Systems... 7 Methods of Monitoring the Active Directory... 8 Simple Network Monitoring Protocol (SNMP)... 8 LDAP Probing... 8 DNS Probing... 8 Operating System Specific Probes... 8 Indirect Monitoring... 8 Log File Analysis... 8 Using the Event Log as a Data Source... 23 Application Log... 23 System Log... 23 Security Log... 23 Directory Service Log... 23 File Replication Service Log... 23 Domain Name Systems Server Log... 23 Monitoring replication within the configuration naming context... 27 PATROL defaultaccount required permissions... 28 To configure PATROL KM for Microsoft Windows Active... 29 Microsoft Windows Server 2003 SP1 / R2... 29 BMC PM for Servers - Active Directory Version 2.3 Page 2 of 30

Monitoring of Components Why Monitor? Monitoring is the only indication as to the health and well being of your deployed directory solution. Experience proves-out that companies who do not initiate proactive monitoring, always fall prey to crisis situations, disasters that could have been foreseen and avoided, and quickly fall into the unenviable position of constantly having to respond to situations where customers are impacted by failures or service interruption. How many times have we seen a service interruption first reported by a customer or end-user (usually a call to the service desk indicating a problem connecting to or accessing a system or service). With a little thought and a little more work, you can implement a proactive monitoring scheme that ultimately saves you time, money and vastly improves customer satisfaction. Six Sigma Values As stated in related documentations, Six Sigma values tell us: You don't know what you don't know You can't do what you don't know You won't know until you measure (or monitor) The more sophisticated, elaborate and distributed your directory service, the more you need to monitor, so that you fully understand what is going on at all times. The following text is provided within the context of these values, as these are the keys that frame your proactive monitoring paradigm. How does Six Sigma works? Follow these steps closely to successfully monitor and manage your Active Directory: 1. Define 2. Measure 3. Analyze 4. Improve 5. Control See the next graphic for additional explanations. BMC PM for Servers - Active Directory Version 2.3 Page 3 of 30

BMC PM for Servers - Active Directory Version 2.3 Page 4 of 30

Controlling the cost of your AD Infrastructure & Monitoring Solution Ask yourself the following questions to identify and determine the quality and depth of your monitoring solution for the Active Directory: Are Systems personnel divided into silos where they specialize on a specific platform? How many servers/subsystems does each person manage? How fast are existing servers growing? What happens if key applications or technologies are not available? How much of your day is taken up with repetitive tasks? Are you experiencing costly hardware and software upgrades to meet computer resource demands? What are your goals this year in terms of service quality, new application deployment, and capacity growth? Do your have a proactive methodology in place to ensure that your mission critical business applications perform as required today and in the future? Do inconsistencies between different tools cause staff to duplicate work? Does the data center consistently meet deadlines and SLAs? So far, we have established that your directory is the heart and souls of your computing environment, used by customers to logon to the network, authenticate to services and application, and look-up other users and resources network-wide. An interruption to these core directory services results in downtime for users and business applications, which directly translates into lost productivity and money. By monitoring your directory, you can learn of outages as soon as they occur, and in some cases, even before they occur. With more sophisticated monitoring tools, you can further anticipate failures, understand where performance degradation exists, and use the captured information for the purpose of system tuning. A monitoring system consists of three elements: The monitored devices and services. The monitoring solution or system. The alert, notification, and escalation processes that determine how you will respond to events. BMC PM for Servers - Active Directory Version 2.3 Page 5 of 30

Device and Application Probing This element is the function or process responsible for periodically checking the status of a monitored service, device, host, application, or other system. When a device fails to respond to a specified probe, an alert is generated that indicates the failed device and nature of the failure. BMC Performance Manager Agent based Probe Event Correlation Module This element receives input from the probing module and correlates the inputs to determine the root cause. It then suppresses any events that might have occurred as a result of other events. After suppressing indirect events, the module constructs one or more alerts and forwards them to the notification module. Notification Module This module receives alerts from the correlation module and generates notifications to the appropriate (pre-programmed) respondent or responsible party. Also, this module might generate a notification to an automated response system pre-programmed to restart a service, or some other remedial action designed to address a failure. The monitoring system shown in the figure above is a conceptual model. Humans or a software application could perform any of the models elements. The goal is to automate these elements as much as possible into a cohesive, predictable solution that addresses your specific monitoring needs. BMC PM for Servers - Active Directory Version 2.3 Page 6 of 30

Types of Monitoring and Monitoring Systems There are essentially three types of monitors - hard-error monitors, soft-error monitors, and performance monitors. Hard errors occur as a direct result of a hardware or network failure. Soft errors are typically caused by programming or data problems, resulting in incorrect or inconsistent data in the directory proper. Performance monitors provide valuable feedback on the system's performance, identifying bottlenecks, points of contention, and performance degradation. Performance monitoring can also provide baseline information, allowing you to capture trend information useful in understanding when you will need to perform capacity planning or execute an upgrade to the directory infrastructure. The main goal of the monitoring solution should help to control the cost of your IT Infrastructure. Some Best practices are listed in order to turn from chronic instability to stability : Be current Test everything Prevent all repeat problems Avoid all known problems Optimize risk profile of all changes Manage proactively BMC PM for Servers - Active Directory Version 2.3 Page 7 of 30

Methods of Monitoring the Active Directory Simple Network Monitoring Protocol (SNMP) Although SNMP [Security is Not My Problem] has found its widest application in the management of networking hardware such as switches, hubs and routers, it is also possible to use SNMP to monitor and manage applications and process running on servers and other support devices. SNMP allows a management application to monitor the status of an entity on a network. It is also possible for a management application to be asynchronously notified via the SNMP trap mechanism when an event or error occurs. LDAP Probing One of the most straightforward and useful ways to monitor your directory is to probe it by connecting from a client and issuing LDAP commands and/or requests. For example, a simple probe tool might connect to a directory and search for a pre-determined entry. If the response is within a pre-specified response window, the directory is considered to be functioning. If not, the probe tool can generate an error. DNS Probing Another way to monitor your directory is to probe it by connecting from a client and issuing DNS query requests. For example, a simple probe tool might connect to a domain name server and search for a pre-determined entry. If the response is within a prespecified response window, the directory is considered to be functioning. If not, the probe tool can generate an error. Operating System Specific Probes Most modern operating systems come with tools that provide for monitoring their respective services, including their native directory services. This type of information can assist you in determining when your directory is experiencing a problem as a result of the operating system. Indirect Monitoring Monitoring the applications that directly touch your directory provides more of an enduser view of the responsiveness and reliability of the system. Log File Analysis You can automatically scan your directory's log files for events that indicate an error condition. Additionally, you can monitor for conditions that indicate performance problems. BMC PM for Servers - Active Directory Version 2.3 Page 8 of 30

BMC Performance Manager for Servers As a distributed service, Active Directory depends on many interdependent services that are distributed across many devices and in many remote locations. Correlation of key performance indicators, object state and logical transaction based data becomes more important. To monitor the key performance indicators of a simple server configuration, the members of the IT department need to collect three different types of performance data over a specified period of time: 1. general performance data 2. baseline performance data 3. data for service level reports General performance data is information that can help the members of the IT department to identify short-term trends such as memory leaks. After a month or two of data collection, the members of the IT department can average the results and save them in a more compact format. Baseline performance data is information that can help the members of the IT department to discover changes that occur slowly, over time. By comparing the current state of your system with historical data, you can troubleshoot and tune your system. Data for service level reports is information that can help you to ensure that your system meets a certain service or performance level, and that you will likely present to decision makers who are not performance analysts. How often you collect and maintain this data depends on your specific needs. Use Case Scenarios This use case will address the following issue: The members of the IT department need clearly defined key indicators so that they know when and why the users were not able to login with their provided user credentials. In order to find out why the user was not able to login with the provided user credentials a member of the IT department will perform the following tasks: 1. Collect and analyze key availability indicator metric based: general performance data baseline performance data 2. Collect and analyze key availability indicator event based: general availability data baseline availability data 3. Collect and analyze key performance indicator metric based: general performance data baseline performance data 4. Collect and analyze key performance indicator event based: general availability data baseline availability data BMC PM for Servers - Active Directory Version 2.3 Page 9 of 30

Implementation Scenarios General Performance Monitoring Adjust Polling Cycle based on CPU Collection interval Step 1: Select the properties of the application class of the collector you want to change Step 2: Select the Parameter / Collector and click on Customize Step 3: Adjust the polling interval according to your requirements Step 4: Press OK BMC recommends the use of PATROL Configuration Manager (PCM) in a production environment to configure BMC Performance Manager for Servers, PATROL Agents, Thresholds, Alerts and the configuration of Knowledge Modules. BMC PM for Servers - Active Directory Version 2.3 Page 10 of 30

Implementation Scenarios General Performance Monitoring Adjust Threshold based on CPU performance metric Step 1: Select the metric you want to change Step 2: Select the Task and click on Customize Step 3: Adjust the thresholds according to your requirements Step 4: Press OK BMC PM for Servers - Active Directory Version 2.3 Page 11 of 30

Implementation Scenarios General availability monitoring Service monitoring The member of the IT Department can monitor all windows services. All available services are being monitored out of the box. If the startup type is set to automatic, you can activate a corrective action in order to restart the service. In the case of Active Directory monitoring BMC recommends selecting the core services to monitor: Certificate Service Distributed File System DNS Server DNS Client Event Log Intersite Messaging Kerberos Key Distribution Center Server Workstation Net Logon File Replication Service IPSEC Services Remote Procedure Call (RPC) Remote Procedure Call (RPC) Locator Windows Time The current status of the service is displayed and historical data is kept in a database. A menu command allows the member of the IT Department to change the services being monitored and related properties. See online help for further details Configuring service monitoring. BMC PM for Servers - Active Directory Version 2.3 Page 12 of 30

Implementation Scenarios General availability monitoring Process Performance Monitoring In addition to the service monitoring, the monitoring of Active Directory core processes, corresponding to the services mentioned earlier and recommended by BMC, are essential to detect possible bottlenecks or inconsistent system behavior. Processes can be grouped to get the overall picture of the system performance and workload. Clearly defined Key performance indicators help to monitor the performance and availability of Active Directory. Formatted data as well as raw data is provided in order to satisfy the need of having objective data. Menu commands provide the flexibility to add, delete or modify process monitoring and manage process groups. BMC PM for Servers - Active Directory Version 2.3 Page 13 of 30

Implementation Scenarios Domain Name Services AD from client perspective The process for monitoring DNS to support Active Directory varies according to whether your organization already has an existing DNS service or whether you are deploying a new DNS service. To verify Active Directory s Domain Name Server system and appropriate name resolution from a client perspective, it s recommended to incorporate BMC Performance Manager for Internet Servers into your monitoring design. The member of the IT Department can configure lookup requests to search for SRV records in the DNS database. Additional value is given by means of the content check. It enables the member of the IT Department to check the logical function of Active Directory in connection with DNS. A subset of recommended resource records to look for is listed below. BMC PM for Servers - Active Directory Version 2.3 Page 14 of 30

SRV Resource Records and Dynamic Updates DNS exists independently of Active Directory, whereas Active Directory is designed specifically to work with DNS. For Active Directory to function properly, DNS servers must support Service Location (SRV) resource records. SRV resource records map the name of a service to the name of a server offering that service. Active Directory clients and domain controllers use SRV resource records to determine the IP addresses of domain controllers. Mnemonic Type DNS Record Requirements PDC SRV _ldap._tcp.pdc._msdcs.<dnsdomainname> One per domain GC SRV _ldap._tcp.gc._msdcs.<dnsforestname> At least one per forest GcIpAddress A _gc._msdcs.<dnsforestname> At least one per forest DSACName CNAME <DsaGuide>._msdcs.<DnsForestName> One per domain controller KDC SRV _kerberos._tcp.dc._msdcs.<dnsdomainname> At least one per domain DC SRV _ldap._tcp.dc._msdcs.<dnsdomainname> At least one per domain A <DomainControllerFQDN> One per domain controller (domain controllers that have multiple IP addresses can have more than one A resource record) BMC PM for Servers - Active Directory Version 2.3 Page 15 of 30

Implementation Scenarios Domain Name Services AD from server perspective Active Directory uses the name resolution services provided by DNS to enable clients to locate domain controllers and to enable the domain controllers hosting the directory service to communicate with each other. Active Directory uses DNS as its locator service to support the various types of services that AD offers, for example: Global Catalog (GC) Kerberos Lightweight Directory Access Protocol (LDAP) Sometimes clients might need to contact a Microsoft-hosted service. For that reason, each domain in DNS has a _msdcs sub domain that hosts only DNS SRV records that are registered by Microsoft-based services. The Netlogon process dynami cally creates these records on each domain controller (DC). The _msdcs sub domain also includes the globally unique identifier (GUID) for all domains in the forest and a list of GC servers. BMC Performance Manager for Servers verifies the proper DNS registration of a domain controller and reports failures of a domain controller to register DNS records dynamically that advertise its availability as a domain controller. The error Event IDs 5774, 5775 and 5783 and the warning Event ID 5781, for example, are being tracked and a member of the IT Department is being informed of the situation. Besides the monitoring of the Win32 service DNS.EXE, BMC Performance Manager for Servers monitors the ability of the server to initiate a remote procedure call (RPC). A selection of DNS based performance indicators help to identify possible bottleneck in the logical operations of the DNS. Process monitoring in turn, will provide performance data information that can help the members of the IT Department to discover changes that occur slowly, over time. DNS performance testing, by means of DNS lookup queries are supported in order to measure the general response time of a DNS server. It will help the members of the IT department to evaluate the current workload of the DNS system. BMC PM for Servers - Active Directory Version 2.3 Page 16 of 30

Implementation Scenarios AD related Services AD from client perspective The member of the IT Department can configure the Lightweight Directory Access Protocol (LDAP) monitor to perform an anonymous bind to a specific server. It will help the members of the IT Department to evaluate the current workload of the LDAP system. The member of the IT Department can configure the SMTP monitor to perform an SMTP login to a specific server. It will help the members of the IT department to evaluate the current workload of the SMTP system. There is a menu command with BMC Performance Manager for Internet Servers provides the Interface to configure the server monitoring. Standard TCP/IP based internet services, like LDAP, SMTP and HTTP/S that can have a crucial part in the overall availability of the services provided by Active Directory, should be monitored from a performance, workload and logical perspective. BMC PM for Servers - Active Directory Version 2.3 Page 17 of 30

Implementation Scenarios FSMO & GC related Services AD from server perspective Domain controllers must be able to locate and establish an LDAP connection with Flexible Single Master Operation (FSMO) role holders. BMC Performance Manager for Servers monitors the connectivity status of each of the five FSMO role holders from the current domain controller. Each FSMO role has one instance, named to reflect its FSMO role: Schema Master Domain Naming Master Relative ID Master PDC Emulator Infrastructure Master BMC Performance Manager for Servers enables the member of the IT Department to: Report whether the domain controller that holds the operations master role is allowing LDAP connections. Detect and report when a master operations FSMO role is moved to or from the current DC and when the current DC acquires the role. Verify that the domain naming master owner hosts a global catalog (GC). Verify that the infrastructure naming master is not a global catalog. Exceptions include o o a single domain forest a multi-domain forest where every domain controller also hosts a global catalog Verify that the Domain Naming Master and Schema Master reside on the same domain controller. Sample Report on an Active Directory implementation based on the InfoBox of the BMC Performance Manager for Servers Active Directory monitoring solution. BMC PM for Servers - Active Directory Version 2.3 Page 18 of 30

BMC Performance Manager for Servers enables the member of the IT Department to verify the health of the Server holding the global catalogue by: Reporting the connectivity/availability of LDAP on a global catalog server using the global catalog port number, 3268. Reporting the current number of threads in use by the LDAP subsystem of the local directory service. Reporting the amount of time required to issue an LDAP bind operation and perform a small search on a global catalog server using the global catalog port, 3268. Reporting the amount of time required to issue an LDAP bind operation. The bind operation is performed locally on the domain controller to eliminate network latency. Verifying that a global catalog is correctly advertising itself as a global catalog. If a global catalog is not advertising correctly, clients will not be able to locate it. Verifying that a domain controller is correctly advertising itself correctly as a domain controller. If a domain controller is not advertising correctly, clients cannot locate it. Operations Master Role Consequences if Role is Unavailable Risk of Improper Restoration Recommendation for Returning to Service After Seizure Schema master You cannot make changes to the schema. Conflicting changes can be introduced to the schema if both schema masters attempt to modify the schema at the same time. This can result in a fragmented schema. Not recommended. Can lead to a corrupted forest and require rebuilding the entire forest. Domain naming master You cannot add or remove domains from the forest. You cannot add or remove domains or clean-up metadata. Domains might appear as though they are still in the forest even though they are not. Not recommended. Can require rebuilding domains. PDC emulator You cannot change passwords on pre-active Directory clients. No replication to Windows NT 4.0 backup domain controllers. Password validation can randomly pass or fail. Password changes take much longer to replicate throughout the domain. Allowed. User authentication can be erratic for a time, but no permanent damage occurs. Infrastructure master Delays displaying updated group membership lists in the user interface when you move users from one group to another. Displays incorrect user names in group membership lists in the user interface after you move users from one group to another. Allowed. May impact the performance of the domain controller hosting the role, but no damage occurs to the directory. RID master Eventually, domain controllers cannot create new directory objects as each of their individual RID pools is depleted. Duplicate RID pools can be allocated to domain controllers, resulting in data corruption in the directory. This can lead to security risks and unauthorized access. Not recommended. Can lead to data corruption that can require rebuilding the domain. BMC PM for Servers - Active Directory Version 2.3 Page 19 of 30

Implementation Scenarios Replication related Services AD from server perspective In addition to Win32 service and process monitoring BMC Performance Manager for Servers enables the member of the IT Department to verify the health of Active Directory replication by: Monitoring directory replication to report duplicate object errors. Monitoring the warning Event ID 1265 in the directory service event log. Reporting the number of unsuccessful synchronization requests processed. Monitoring for the occurrence of lingering objects. Monitoring the error Event ID 1388 in the directory service event log. Reporting the number of directory synchronization requests that are queued for this server but have not yet been processed. Reporting errors that occur when the NT Directory Service (NTDS) received a failure while trying to perform an authenticated RPC call to another Domain Controller due to a service principal name (SPN) mismatch. Reporting a topology mismatch, this occurs when replication configuration information in Active Directory Sites and Services does not reflect the physical topology of the network. Monitoring the File Replication service (FRS) for occurrences of duplicate connections. Detecting a journal wrap. This parameter issues an alert if the NT File System (NTFS) change journal exceeds its maximum size limit and wraps to restore its maximum size by deleting the oldest records. Detecting when the FRS service has been unable to complete the RPC connection to a specific replication partner. Monitoring the resolution of user security IDs (SIDs) for File Replication System (FRS) and issues an alarm if the SID cannot be determined from the distinguished name. Monitoring the available space in the staging area and issues an alert if this area becomes full. Verifying that the domain controller has an enabled inbound connection from a SYSVOL replication partner and that an enabled outbound connection from the domain controller resides on a SYSVOL replication partner. Reporting the existence of replication collisions for the types of objects that were selected for monitoring. Reporting whether replication between the site/domain, site/forest, or both for a domain controller is occurring properly. Reporting whether replication within the site/domain, site/forest, or both for a domain controller is occurring properly. Reporting whether or not SYSVOL is shared. BMC PM for Servers - Active Directory Version 2.3 Page 20 of 30

Implementation Scenarios Performance Monitor AD from server perspective BMC Performance Manager for Servers enables the member of the IT Department to verify the health of Active Directory by collecting and analyzing key performance indicators based on the Windows Performance monitor. The WMI and Performance Monitor Wizard let you integrate all available performance counters and custom WQL based quires. To highlight just a few possibilities: BMC PM for Servers - Active Directory Version 2.3 Page 21 of 30

Implementation Scenarios Eventlog Monitor AD from server perspective BMC Performance Manager for Servers enables the member of the IT Department to verify the health of Active Directory by collecting and analyzing key performance indicators based on the Windows event log. All major Active Directory events are being taken care of by means of the AD monitoring solution. To enhance the monitoring capabilities, BMC Performance Manager for Servers enables the member of the IT Department to: Monitor additional events and can do so by incorporating appropriate event log filtering. To reduce the amount of events being forwarded by the infrastructure monitoring by allowing to alter the respective configuration. To create filters that monitor Windows events based on event properties. Whenever an event occurs that matches the filter criteria that the member of the IT Department specifies BMC Performance Manager for Servers generates reports or alerts, depending upon the notification settings the member of the IT Department selected for the filter. BMC PM for Servers - Active Directory Version 2.3 Page 22 of 30

Using the Event Log as a Data Source You can use the Event Log service and BMC Performance Manager for Servers Windows Event monitoring solution to gather and filter information about hardware, software, and system problems, and to monitor Windows security events. Windows 200x records events in six types of logs: Application Log This log contains events logged by applications or programs. System Log This log records events such as valid and invalid logon attempts, as well as events related to resource use such as creating, opening, or deleting files or other objects. Security Log This log contains events logged by Windows system components. The event types logged by system components are predetermined by the server. Directory Service Log This log contains events logged by the Microsoft Windows Active Directory directory service. File Replication Service Log This log contains events logged by the Windows File Replication service. Domain Name Systems Server Log This log contains events logged by the DNS service. BMC PM for Servers - Active Directory Version 2.3 Page 23 of 30

Events raised by replication issues Common events that might indicate a problem with Active Directory replication, together with root cause and solution information. Event Root Cause Solution Net Logon Event ID 5805 NTDS Event ID 1083 NTDS Event ID 1265 NTDS Event ID 1311 NTDS Event ID 1388 NTDS Event ID 1645 A machine account failed to authenticate, which is usually caused by either multiple instances of the same computer name, or the computer name has not replicated to every domain controller. A duplicate object is present in the Active Directory of the replication partner of the local domain controller, so updating it is impossible. Replication failed for the reason stated in the message text. This error occurs when the replication configuration information in Active Directory Sites and Services does not accurately reflect the physical topology of the network. This error is usually generated by a lingering object which resulted from disconnecting a domain controller for too long. This error occurs over an existing replication link when the GUID of the NTDS Settings object of a replication partner does not match the GUID defined in the Service Principal Name (SPN) attributes of the computer object of this replication partner. If you do not find multiple instances of the computer name, verify that replication is functioning for the domain that contains the computer account. See Troubleshooting Directory Data Problems. Use Repadmin.exe to further identify the problem, and use Table x.x to determine the appropriate action to take for the message generated by Repadmin.exe. If the event message indicates a DNS lookup failure or the RPC server is unavailable, see Troubleshooting Active Directory Related DNS Problems. If the event message indicates that the target account name is incorrect, troubleshoot GUID discrepancies. If the event message indicates a time difference between the client and server, synchronize replication from the PDC emulator. Troubleshoot NTDS event ID 1311. If the domain controller does not also function as a global catalog server, see Remove Lingering Objects from an Outdated Writable Domain Controller. If the domain controller also functions as a global catalog server, see Remove Lingering Objects from a Global Catalog Server. Troubleshoot GUID discrepancies. BMC PM for Servers - Active Directory Version 2.3 Page 24 of 30

Event Root Cause Solution SceCli event ID 1202 A user account in one or more Group Policy objects (GPOs) cannot be resolved to a security identifier (SID). This error is possibly caused by a mistyped or deleted user account Troubleshoot SceCli event ID 1202. referenced in either the User Rights Assignment or Restricted Groups branch of a GPO. Events raised by Active Directory Event Log Source Event Why Event is Important Application SCECLI Severity = error 1058 Critical NETLOGON service errors Application USERENV Severity = error Responsible for the application of group policy User = System and profiles on domain controllers Directory The primary error events for the Active Directory All Sources Severity = error Service service. FRS All Sources Severity = error FRS is used to synchronize policy between all Domain Controllers in the Forest. System DNSAPI 11150 1162 DNS server timed out 11151 System DNSAPI 11155 11163 A resource record for the domain controller is not registered in DNS 11167 11152 System DNSAPI 11153 11164 The zone or the currently connected DNS server does not support dynamic update 11165 System DNSAPI 11154 11166 Domain controller does not have sufficient rights to perform a secure dynamic update System KDC Severity = error System LSASS Severity = error System NETLOGON 5773 System NETLOGON 5774 Critical Kerberos Distribution Center service error messages Local Security Authority is the core security subsystem for Active Directory One or more DC locator records are not registered because the primary DNS server does not support dynamic update One or more DC locator records are not registered in DNS BMC PM for Servers - Active Directory Version 2.3 Page 25 of 30

Event Log Source Event Why Event is Important System NETLOGON Severity = error 5705 Critical NETLOGON service errors 5723 System W32TIME Severity = error Severity = warning BMC PM for Servers - Active Directory Version 2.3 Page 26 of 30

Security Hardening Synthetic Transactions AD from server perspective Monitoring replication within the configuration naming context BMC Performance Manager for Servers Knowledge Module for Active Directory version 1.6 monitors Active Directory intrasite and intersite replication for both errors and latency issues within the domain naming context. This release of BMC Performance Manager for Servers Knowledge Module for Active Directory provides the ability to monitor replication within the configuration naming context. Replication monitoring within the configuration naming context is not enabled by default. To enable replication monitoring within the configuration naming context, create and set the /ActiveDirectory/Configuration/ReplMonConfigNC configuration (pconfig) variable as shown in Table 1. Simultaneous replication monitoring of both the configuration and domain naming context is supported, but not required. To disable replication monitoring of the domain naming context, create and set the /ActiveDirectory/Configuration/ReplMonDomainNC configuration (pconfig) variable as shown in Table 1. For inter operability with previous releases of the KM, replication monitoring of the domain naming context must be enabled (the default). Variable name Default Values /ActiveDirectory/Configuration/ReplMonConfigNC 0 (Off) 0=Configuration naming context monitoring is off 1=Configuration naming context monitoring is on /ActiveDirectory/Configuration/ReplMonDomainNC Table 1 1 (On) 0=Domain naming context monitoring is off 1=Domain naming context monitoring is on BMC Performance Manager for Servers uses the same parameters to monitor configuration naming context replication as it uses to monitor domain naming context replication. The alarm annotations report the following: Replication context Names of the domain controllers that failed to replicate or that did not replicate in a timely manner BMC PM for Servers - Active Directory Version 2.3 Page 27 of 30

Insertions into the Active Directory Interdomain o PatrolReplication container under the Configuration Container Intradomain o PatrolReplication container under the Domain NC Default Configuration for CN=PatrolReplication based on Active Directory PATROL defaultaccount required permissions Monitoring replication within the configuration naming context requires that the PATROL Agent defaultaccount have sufficient Active Directory permissions to create a container object and child container objects in the configuration naming context of the forest in which the domain controller resides. The account must have full control of the created objects. The PATROL Agent defaultaccount must be granted permission to Create Container Objects in the Configuration NC and to give Full Control to the created container object and its children. Monitoring replication within the domain naming context requires that the PATROL Agent defaultaccount have sufficient Active Directory permissions to create a container object and child container objects in the domain naming context of the domain in which the domain controller resides. The account must have full control of the created objects. The BMC PM for Servers - Active Directory Version 2.3 Page 28 of 30

PATROL Agent defaultaccount must be granted permission to Create Container Objects in each Domain NC and to give Full Control to the created container object and its children. To configure PATROL KM for Microsoft Windows Active Directory for configuration naming context replication In Active Directory, grant PATROL Agent defaultaccount the following permissions: o Create Container Objects in the configuration naming context o Full Control to the created container object and its children Set the pconfig variable /ActiveDirectory/Configuration/ReplMonConfigNC to 1. (Optional) Set the pconfig variable /ActiveDirectory/Configuration/ReplMonDomainNC to 0. Microsoft Windows Server 2003 SP1 / R2 Additional requirements: Microsoft Windows Server 2003 requires Performance Monitor Users Group and the PATROL Agent needs to be able to access the registry via the internal command. Also, the PATROL Agent defaultaccount needs to be able to query our PatrolReplication container on the remote boxes. BMC PM for Servers - Active Directory Version 2.3 Page 29 of 30

Reference http://www.bmc.com http://www.bmc.com/supportu/hou_support_prodversion/0,3648,19097_19695_103 926_0,00.html BMC Software Technical Bulletin *53535* PATROL KM for AD 1.5.2.11 Microsoft MSDN http://msdn.microsoft.com http://en.wikipedia.org/wiki/six_sigma Six Sigma is a methodology to manage process variations that cause defects, defined as unacceptable deviation from the mean or target; and to systematically work towards managing variation to eliminate those defects. The objective of Six Sigma is to deliver high performance, reliability, and value to the end customer. It was pioneered by Bill Smith at Motorola in 1986 and was originally defined as a metric for measuring defects and improving quality; and a methodology to reduce defect levels below 3.4 Defects Per (one) Million Opportunities (DPMO). Six Sigma has now grown beyond defect control. Six Sigma is a registered service mark and trademark of Motorola, Inc. Feedback & Comments BMC Software, Inc. Product Management BMC Performance Manager e-mail: Volker_Scheithauer@bmc.com Copyright August 31, 2006 BMC Software, Inc., as an unpublished work. All rights reserved. BMC Software, the BMC Software logos, and all other BMC Software product or service names are registered trademarks or trademarks of BMC Software, Inc. All other trademarks belong to their respective companies. BMC Software considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable End User License Agreement for the product and the proprietary and restricted rights notices included in this documentation. Disclaimer; Limitation of Liability; Indemnity http://www.bmc.com/legal/ BMC PM for Servers - Active Directory Version 2.3 Page 30 of 30