IBM Tivoli IBM Tivoli Composite Application Manager for Microsoft Active Directory Agent KB Notes and HOW TOs
CONTENTS 1 Overview... 3 1.1 Introduction... 3 1.2 Terms and abbreviations... 3 1.3 User Guides... 3 2 HOW TOs... 4 2.1 How to configure the agent startup to run as non-administrator?... 4 2.2 How to turn caching ON/OFF?... 5 2.3 How to start\stop sysvol replication test?... 6 2.4 Setting RAS trace parameters... 6 2.5 How to enable / reload perfmon counters?... 8 2.6 How to bundle logs for L3 team?... 9 3 Problems, Symptoms, Solutions... 10 3.1 If ADO agent is crashing on window server 2008.... 10 3.2 ADO agent taking too much time in response? Slow response of ADO agent might result in triggering of situation related to DNS response time, trust fail.... 10 3.3 When You turn the caching on (i.e 240 recommended) for AD agent the CPU usage increases a lot.... 10 3.4 Many times agent log shows below error for specific counters-... 10 3.5 Many times agent log shows below error for some counters -... 10 3.6 Failures are indicated in the logs as - "Ping failed, check results above"... 10 3.7 Sysvol replication might cause high load on the network.... 11 3.8 Data for perfmon attribute groups not displayed on the TEP... 11 4 Microsoft tools for AD troubleshooting:... 12 4.1 Nslookup:... 12 4.2 Dcdiag tool (Domain Controller Diagnostic Tool)... 13 4.3 Netdiag tool... 13 4.4 Nltest tool:... 13 4.5 NETDOM... 14
1 Overview 1.1 Introduction This document lists and provides solutions to common problems faced by customers of IBM Tivoli MS Active Directory Monitoring Agent during setting up the product in their Exchange environments. The intended audience of the document is Level 2 personnel for IBM Tivoli MS Exchange Monitoring Agent and the customers of the product. Various sections of the document refer to a user account / user name to be used. 1.2 Terms and abbreviations Term / Abbreviation IBM ITM MS L3 PDF TEP Description International Business Machines IBM Tivoli Monitoring Microsoft Level three support Adobe Postscript Data Format Tivoli Enterprise Portal 1.3 User Guides activediragent623_user.pdf
2 HOW TOs 2.1 How to configure the agent startup to run as non-administrator? You can run the monitoring agent for Active Directory as a non-administrator Domain user however, Trust Topology attributes and Sysvol Replication attributes might not be available. Note: Attributes that are related to the Trust Topology workspace and the Sysvol Replication workspace are displayed on the Tivoli Enterprise Portal Server after you configure the agent service to an administrator account. For Trust Topology attributes, complete the following steps to create a non-administrator user and provide access to required registry paths: 1. Create a new Domain User with the authority - Domain Users or Users. 2. Grant full access in the registry to the HKEY_LOCAL_MACHINE\SOFTWARE\Candle*. 3. Grant read access in the registry to the HKEY_LOCAL_MACHINE\SOFTWARE\ Microsoft\Windows NT\CurrentVersion\Perflib. For Sysvol Replication attributes, complete the following steps to create a non-administrator user and provide access to the Sysvol folder: 1. Create a new User ID with the authority - Domain Users or Users. 2. Grant full access to the Sysvol folder on all domain controllers in a domain. Follow the steps mentioned below to configure the agent startup. Step 1) Launch the IBM Tivoli Monitoring application Step 2) Right click the agent service / agent name and click Change Startup
Step 3) Select the options as in the image below and provide the username and the password of the user created with the steps above Once done, click OK. 2.2 How to turn caching ON/OFF? Turning ON/OFF caching during run time is done through Manage Tivoli Enterprise Monitoring Services (kinconfig.exe). The steps are as follows: 1. In Manage Tivoli Enterprise Monitoring Services (kinconfig.exe), select the Monitoring Agent for Active Directory. 2. Right-click and select Advanced options. 3. Select Edit ENV File from the options. This opens the K3ZENV file for editing. The ADO_PING variable exists in the K3ZENV file. 4. Edit the following variables, a. ADO_PING_COUNT=1 (Number of time the agent tries to ping a resource) b. ADO_PING_TIMEOUT=2000 (Timeout for ping request) c. ADO_PING_SIZE=32 (Ping Packet Size) 5. After editing the K3ZENV file, save and close the file to implement the new ping behavior. 6. A message box appears asking if the agent needs to be recycled to include the changes in agent functionality. Clicking Yes recycles the agent with the new ping values. Clicking No lets the agent continue to run without the changes to the ping values. When the agent is restarted, the changes are implemented. The following attribute groups have an option for caching the data they collect for some configurable period: Domain Controller Availability DNS AdIntegrated Domain Controller Performance Replication Replication Latency Replication Partner Trusts
2.3 How to start\stop sysvol replication test? To start\stop sysvol replication test follow the following steps: 1. In Manage Tivoli Enterprise Monitoring Services (kinconfig.exe), select the Monitoring Agent for Active Directory. 2. Right-click and select Advanced options. 3. Select Edit ENV File from the options. This opens the K3ZENV file for editing. The ADO_SYSVOL_REPLICATION_TEST_INTERVAL and ADO_SYSVOL_REPLICATION_TEST_VERIFICATION_INTERVAL variables exist in the K3ZENV file. Set these value according to the interval in which the sysvol replication is needed to be tested. Recommendations are ADO_SYSVOL_REPLICATION_TEST_INTERVAL = 1440 ADO_SYSVOL_REPLICATION_TEST_VERIFICATION_INTERVAL = 30 If you do not want the sysvol replication test to be performed then set the above environment variables to 0. 4. After editing the K3ZENV file, save and close the file to implement the new ping behavior. 5. A message box appears asking if the agent needs to be recycled to include the changes in agent functionality. Clicking Yes recycles the agent with the new ping values. Clicking No lets the agent continue to run without the changes to the sysvol replication values. When the agent is restarted, the changes are implemented. 2.4 Setting RAS trace parameters Specify RAS1 trace options in the K3ZENV file. You can manually edit the configuration file to set trace logging: 1. Open the trace options file: install_dir\tmaitm6\k3zenv. 2. Edit the line that begins with KBB_RAS1= to set trace logging preferences, a. For Maximum Tracing option: KBB_RAS1=ERROR (UNIT:k3z ALL) (UNIT:kra ALL) 3. Edit the line that begins with KBB_RAS1_LOG= to manage the generation of log files: Edit the following parameters to adjust the number of rolling log files and their size. MAXFILES: the total number of files that are to be kept for all startups of a given program. Once this value is exceeded, the oldest log files are discarded. Default value is 9. LIMIT: the maximum size, in megabytes (MB) of a RAS1 log file. Default value is 5. IBM Software Support might guide you to modify the following parameters: COUNT: the number of log files to keep in the rolling cycle of one program startup. Default value is 3. PRESERVE: the number of files that are not to be reused in the rolling cycle of one program startup. Default value is 1. Note: The KBB_RAS1_LOG parameter contains multiple specification for logging e.g. file directory, log file name etc. Only modify the values mentioned above. 4. Restart the monitoring agent so that your changes take effect. (Windows only) Alternate method to edit trace logging parameters: 1. Open the Manage Tivoli Enterprise Monitoring Services window. 2. Right-click the icon of the monitoring agent whose logging you want to modify.
3. Select Advanced > Edit Trace Parms. The Tivoli Enterprise Monitoring Server Trace Parameters window is displayed. 4. Select a new trace setting in the pull-down menu in the Enter RAS1 Filters field or type a valid string. The selections are as follows: No error tracing. KBB_RAS1=-none- General error tracing. KBB_RAS1=ERROR Intensive error tracing. KBB_RAS1=ERROR (UNIT:k3z ALL) Maximum error tracing. KBB_RAS1=ERROR (UNIT:k3z ALL) (UNIT:kra ALL) Note: As this example shows, you can set multiple RAS tracing options in a single statement.
5. Modify the value for Maximum Log Size Per File (MB) to change the log file size (changes LIMIT value). 6. Modify the value for Maximum Number of Log Files Per Session to change the number of logs files per startup of a program (changes COUNT value). 7. Modify the value for Maximum Number of Log Files Total to change the number of logs files for all startups of a program (changes MAXFILES value). 8. (Optional) Click Y (Yes) in the KDC_DEBUG Setting menu to log information that can help you diagnose communications and connectivity problems between the monitoring agent and the monitoring server. Note: The KDC_DEBUG setting and the Maximum error tracing setting can generate a large amount of trace logging. Use them only temporarily, while you are troubleshooting problems. Otherwise, the logs can occupy excessive amounts of hard disk space. 9. Click OK. You see a message reporting a restart of the monitoring agent so that your changes take effect. 2.5 How to enable / reload perfmon counters? The agent collects most of the data from perfmon objects provided by Microsoft. In certain cases, the agent doesn t display data on the TEP for certain counters. The reason behind this could be the counters haven t been loaded / enabled on the machine where the agent is running. Enabling / reloading the concerned counters could solve the problem of data not being displayed on the TEP. Follow the steps mentioned below to enable the perfmon counters. 1. Start Command Prompt on the machine where the agent is running. 2. Run the following command on the Command Prompt, lodctr <counter INI file name> e.g. loading counter for ntds object: lodctr ntdsctrs.ini 3. Run perfmon by typing perfmon (without the quotes) in the Run Program window
4. Verify if the data for the newly enabled counters is being displayed in perfmon as in the image below. For additional information read the following link. http://technet.microsoft.com/hi-in/library/cc961614(en-us).aspx 2.6 How to bundle logs for L3 team? The logs generated by the agent are trace marks about the working of the agent. These logs contain vital information about the steps the agent executed. The logs generated by the agent are located at the path C:\IBM\ITM\TMAITM6\logs(for 32 bit) or C:\IBM\ITM\TMAITM6_x64\logs(for 64 bit) (in case the agent is installed on another drive, the path would start with that drive letter). ZIP the logs folder entirely to be sent to the L3 team.
3 Problems, Symptoms, Solutions 3.1 If ADO agent is crashing on window server 2008. Sol: Turn caching ON. 3.2 ADO agent taking too much time in response? Slow response of ADO agent might result in triggering of situation related to DNS response time, trust fail. Sol: Turn caching ON. 3.3 When You turn the caching on (i.e 240 recommended) for AD agent the CPU usage increases a lot. Reason: The AD server has a lot of load on it for e.g. Larger number of trust, Organizational units, GPOs etc. and because the collection is done after every four minutes so the average CPU utilization increase. Sol: Give some higher value while turning on the caching, this will not cause any data loss as the attributes collected by caching is not very frequently changing ones. 3.4 Many times agent log shows below error for specific counters- "PDH error message text <The specified counter could not be found>." Reason: The counters on the AD box are not enabled. Sol: Enable AD counters 3.5 Many times agent log shows below error for some counters - "PDH error message text < PGFCV failed for some counter >." Reason: The counters existed when the agent started but has been deleted\removed after starting the agent. Sol: Restart the agent. 3.6 Failures are indicated in the logs as - "Ping failed, check results above" It has been observed that AD issues spawn out of ping failures to the concerned machines. These failures are indicated in the logs as - "Ping failed, check results above" In such cases, do the following steps on the machine where AD agent is installed to examine the ping response by the target machine -
1. Start the Command Prompt. 2. Run the command "ping -n 1 -w 2000 -l 32 <hostname>" Whereas, hostname:- concerned machine name. If the above command fails with the hostname, try the same using IP of the concerned machine. If this PING fails with the hostname but succeeds with the IP then, probably the problem is with his WINS or DNS configuration, or possibly with the LMHosts file. Please contact system administrator to resolve the configuration issue. 3.7 Sysvol replication might cause high load on the network. The agent forcefully initiates the sysvol replication, this might cause some high load on the network. Sol1: Provide ADO_SYSVOL_REPLICATION_TEST_INTERVAL and ADO_SYSVOL_REPLICATION_TEST_VERIFICATION_INTERVAL variable a higher value in agent env i.e k3zenv file (recommended value 1440 and 30 respectively), so that the test is not performed after very small interval to avoid load on the network. Sol2: Assign 0 to ADO_SYSVOL_REPLICATION_TEST_INTERVAL and ADO_SYSVOL_REPLICATION_TEST_VERIFICATION_INTERVAL variable in agent env i.e k3zenv file, this will disable the sysvol replication test feature and the test will not be performed. 3.8 Data for perfmon attribute groups not displayed on the TEP Symptom(s): - Perfmon attribute group(s) return zero rows on the TEP. - Log statement like "AddData") Passing row 0 to InsertRow() missing after statement such as "AddData") Entry in the k3zcma log(s). Reason(s): The related perfmon counters are disabled on the machine. Sol: Enable the appropriate perfmon counters on the agent machine. Refer to: How to enable / reload perfmon counters?
4 Microsoft tools for AD troubleshooting: Some of the utilities that Microsoft recommend are nslookup, dcdiag, netdiag should help to locate and troubleshoot the problem: 4.1 Nslookup: nslookup is used in Windows and Unix to query Domain Name System (DNS) servers to find DNS details, including IP addresses of a particular computer, MX records for a domain and the NS servers of a domain. The name nslookup means "name server lookup". When you need more information about a DNS problem than what PING provides you with, you can always turn to the NSLOOKUP command. Nslookup.exe is a command-line administrative tool for testing and troubleshooting DNS servers. This tool is installed along with the TCP/IP protocol through Control Panel. Nslookup.exe can be run in two modes: interactive and non-interactive. Non-interactive mode is useful when only a single piece of data needs to be returned. The syntax for non-interactive mode is: To use Nslookup follow the following steps: 1. Click Start, click Run, type cmd in the Open box, and then click OK. 2. Type Nslookup, and then press ENTER. This will open the Nslookup shell. > set type=srv >_ldap._tcp.dc._msdcs.yourdomin.com you should see something like this: Server: dnsserver.yourdomain.com Address: 192.168.100.2 _ldap._tcp.dc._msdcs.yourdomain.com SRV service location: priority = 0 weight = 100 port = 389 svr hostname = server1.yourdomain.com _ldap._tcp.dc._msdcs.yourdomain.com SRV service location: priority = 0 weight = 100 port = 389 svr hostname = server2.yourdomain.com server1.yourdomain.com internet address = 1.1.1.2 server2.yourdomain.com nternet address = 1.1.1.1 If it does not, definitely there is a DNS problem. In this case the DNS configuration needs to be checked by the system administrator. For more details please check the site: http://support.microsoft.com/kb/200525#top
4.2 Dcdiag tool (Domain Controller Diagnostic Tool) The Dcdiag command-line tool is included when you install Windows Server 2003 Support Tools from the Media DIsk or from the Microsoft Download Center (http://go.microsoft.com/fwlink/?linkid=100114). For more information about how to install Windows Support Tools, see Install Windows Support Tools (http://go.microsoft.com/fwlink/?linkid=62270). This command-line tool analyzes the state of domain controllers in a forest or enterprise and reports any problems to assist in troubleshooting. As an end-user reporting program, Dcdiag encapsulates detailed knowledge of how to identify abnormal behavior in the system. Dcdiag displays command output at the command line. If RPC Endpoint Mapper problems exist, the Dcdiag tool may respond with error messages. To do this, follow these steps: 1. Click Start, click Run, type cmd in the Open box, and then click OK. 2. Type dcdiag, and then press ENTER. For more details please check the site: http://technet.microsoft.com/en-us/library/cc731968.aspx 4.3 Netdiag tool This command-line diagnostic tool helps to isolate networking and connectivity problems by performing a series of tests to determine the state of your network client. These tests and the key network status information they expose give network administrators and support personnel a more direct means of identifying and isolating network problems. Moreover, because this tool does not require parameters or switches to be specified, support personnel and network administrators can focus on analyzing the output rather than on training users how to use the tool. You can use the Netdiag tool to help isolate networking and connectivity problems. You can use the Netdiag tool to help troubleshoot RPC Endpoint Mapper problems. To do this, follow these steps: 1. Click Start, click Run, type cmd in the Open box, and then click OK. 2. Type netdiag, and then press ENTER. For more details please check the site: http://technet.microsoft.com/en-us/library/cc783438.aspx 4.4 Nltest tool: Nltest tool is used to determine trust relationship configurations. NLTEST is a Windows Resource Kit utility you can use to display the current list of trusted domains known by a given server. For each domain listed, you can view the following data: Trust Index (specific to each DC as the trusts are enumerated) NetBIOS Domain Name of the Trusted Domain DNS Domain Name of the Trusted Domain
Trust Type (NT 4, NT 5, MIT, or DCE) Any of the following flags: o Direct Outbound: There is a direct trust relationship between the domain for the server queried and this domain. o Native: This domain is currently in native mode. o Primary Domain: This domain is the domain for the server that was used in the query. o Forest Tree Root: This domain represents the root of a tree in the forest. o Forest: index number: For this trusted domain, where index number is the index number of its parent domain in the same NLTEST list. To do this, follow these steps: 1. Click Start, click Run, type cmd in the Open box, and then click OK. 2. Type nltest /domain_trusts /forest / and then press ENTER. For more details please check the site: Window server 2000 http://support.microsoft.com/kb/228477 Window server 2003 http://technet.microsoft.com/en-us/library/cc784211.aspx Window server 2008 http://technet.microsoft.com/en-us/library/cc731935.aspx 4.5 NETDOM Use NETDOM utility to verify trust relationship. This command-line tool enables administrators to manage Windows Server 2003 and Windows 2000 domains and trust relationships from the command line. For more details please check the site: http://technet.microsoft.com/en-us/library/cc737599.aspx