[Title of paper] [Month Year] Best Practices for Oracle Monitoring with Oracle Extended Agent. Document version [0.3] [Byline for author] [Credential line or job title]
Page 2 of 30
CONTENTS List of Figures...v List of Tables... vi Revision History... vii 1.1 Planning your enterprise monitoring infrastructure...1 1.2 Reference sample Oracle database environment...1 2.1 Local Monitoring or Remote Monitoring...2 2.2 Oracle Instant Client or Oracle Home...2 2.3 Database Connection Number per Agent Instance...3 3.1 Remote installation...3 3.2 Apply latest Interim Fix...3 3.3 Oracle Instant Client...4 3.4 Mount of Alert Log Directory...4 3.5 Configure of TNSNAMES.ORA file in agent site...5 3.6 Disable of sqlnet.log file in agent site...6 3.7 Agent configuration from command line...6 3.8 ASM database monitoring...8 iii
3.9 Dataguard database monitoring...8 3.10 Use non-root user to startup agent to monitor Oracle database...10 3.11 Check the status for connections...10 3.12 Summary...11 4.1 Extend agent timeout...12 4.2 Listener Monitoring...12 4.2.1 Stop the Listener/Net Service Monitoring...12 4.2.2 Avoid the SQLNET.LOG file generated in the ITM directory...12 4.2.3 Selected Listener/Net Services to be monitored...13 4.3 Alert Log Monitoring...13 4.4 Customized SQL...14 4.5 Product Predefined SQL Customization...14 4.6 Cluster Support...14 Appendix... vi References...8 iv
LIST OF FIGURES Figure 1: Blank Database Connection and Agent Event... 10 Figure 2: Dataguard Database Connections... 11 Figure 3: ASM Database Connection... 11 Figure 4: History Collection Configuration for TOPSQL... 16 Figure 5: Selection of Oracle Instant Client Downloads...vi v
LIST OF TABLES Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener... 1 Table 2: Compare of Local monitoring and Remote monitoring... 2 Table 3: Environment variable for agent timeout... 12 Table 4: Environment variables for alert log monitoring... 13 Table 5: Agent Affinities... 14 vi
REVISION HISTORY Date Version Revised By Comments 2011/4/27 0.1 liujingq@cn.ibm.com initials 2011/10/17 0.2 chengliu@cn.ibm.com Add more best practices items 2012-4-30 0.3 chengliu@cn.ibm.com Add ASM/Dataguard vii
1 Overview Oracle Real Application Cluster Database (RAC) support users to access the same data from multiple instances on different servers. In RAC environment, the instance level information includes multiple instances information; the database level information has only one copy. For example, you can get following information for each instance: Instance Name, SGA, Session, and Process. The database level information includes Tablespace, Table, Index and Datafile, such kind of database level information is consistent and shared by all instances in a RAC environment. This document will introduce the best practices in using the Oracle Extended Agent to monitor RAC database, ASM database or Dataguard database. 1.1 Planning your enterprise monitoring infrastructure The Oracle Extended Agent v6.3.1 requires IBM Tivoli Monitoring (ITM) Infrastructure version 622FP2 or higher. 1.2 Reference sample Oracle database environment The following Oracle Database environment will be used as sample environment in this document. The RAC database is installed into two Solaris Sparc servers as two nodes (RAC1 and RAC2). The listener is running on port 1521 in both two nodes. The Oracle cluster ware (including ASM) is installed at /opt/oracle/grid, the Oracle database software is installed at /opt/oracle/db. The Oracle database server s alert log location is /opt/oracle/base/diag/rdbms/rac1/trace/alert_rac1.log. The Oracle software version is 11.2.0.1. The Oracle RAC database myrac is created with two RAC instances, one instance name is rac1 in the RAC1 node, and the other instance name is rac2 in the RAC2 node. The Oracle ASM database +ASM is created and there are also two instance, one instance is +asm1 in the RAC1 node, the other is +asm2 in the RAC2 node. The Oracle Dataguard database is installed into two Solaris Sparc Sun OS 5.10 servers as Primary Node in the machine sunpri and standby node in the machine sunstd. Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener Oracle Node Type Oracle Instance Name Server Name Net Service Name in Listener RAC1 rac1 sol_rac1 myrac RAC2 rac2 sol_rac2 myrac ASM1 +ASM1 sol_rac1 +ASM 1
ASM2 +ASM2 sol_rac2 +ASM Dataguard Primary oradb sol_dg1 dbpri Dataguard Standby oradb sol_dg2 dbstd 2 Strategy of Monitoring This chapter introduces the monitoring strategy with the Oracle Extended Agent about the agent capacity, limitation and scalability. 2.1 Local Monitoring or Remote Monitoring Oracle extended agent support local monitoring and remote monitoring. Following table compares these two different monitoring ways. Table 2: Compare of Local monitoring and Remote monitoring Category Local Remote Installation in Oracle Yes No Server machine Oracle physical disk Yes Not support usage monitoring Oracle alert log monitoring Yes Yes, need mount the remote machine s log directory to local, and need to specify mount path at agent configuration Other monitoring Yes Yes features Maintenance effort Big, need to apply agent patch in each Oracle server machines. Little, only need to apply patch at agent installation machine once. Note: For a RAC environment, we recommend installing the extended agent into one node of the RAC environment, and then the agent performs the remote monitoring. The disk usage for local node can still be monitored; the alert log for remote nodes can be monitored after mount the alert log directory at the remote node. 2.2 Oracle Instant Client or Oracle Home The Oracle extended agent is built with Oracle Call Interface (OCI) technology, the Oracle extended agent can use the dynamic library provided as Oracle Instant Client or Oracle server installation (as Oracle home). We recommend using the Oracle instant client, because the user can always use the latest version of Oracle Instant Client package which contains the latest Oracle client dynamic library code fixes. By using the latest Oracle Instant Client, the user does not need to upgrade the Oracle server in case there are some known bugs at Oracle Client dynamic library. 2
Oracle Instant Client can be downloaded from the oracle web site http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html 2.3 Database Connection Number per Agent Instance We suggest each agent instance monitor no more then 10 database connections in case there are no more then more then 5 situations or historical collections in one agent. This will ensure the total request number in an agent instance is less then 50(10 connection * 5 situations/historical collections). If there are more situations or historical collections, then we suggest reducing the db connection number for each agent instance and adding more agent instances. The agent is affected by ITM limitation also: http://www-01.ibm.com/support/docview.wss?uid=swg21263550 Note: Note: Technotes link: Assuming no OS agent, a maximum of 15 extended agent instances in ITM installation directory. This is a common ITM limitation and it can be extended by following technotes: http://www-01.ibm.com/support/docview.wss?uid=swg21312708 http://www-01.ibm.com/support/docview.wss?uid=swg21253327 3 Agent installation and configuration 3.1 Remote installation Oracle Extended Agent V6.3.1 support remote installation and you can install agent into a separated remote machine with Oracle servers. The agent installation directory is /opt/ibm/tivoli. 3.2 Apply latest Interim Fix There are two options to download and apply latest Interim Fix. Optipon1: Search with key word of 6.3.1-TIV-ITM_KRZ from the Fix Central below, then download the package. 3
Search for downloads and fixes Option 2: Download 6.3.1 IF0002 package from link below: http://www-01.ibm.com/support/docview.wss?uid=swg24031214 You can get all agents download link for latest fix pack or interim fix from below wiki. Latest Fixpacks and IFixes of Tivoli Composite Application Manager for Applications 3.3 Oracle Instant Client If there is no Oracle Home or Oracle Client installed in the agent machine by the Oracle OUI installer, then you need to download the Oracle Instant Client package and extract to one readable directory in the agent machine, for example, /opt/ibm/tivoli/instantclient_11_2. Note: The LITE version of Oracle Instant Client is not supported. You need to create a soft link from libclntsh.so.11.1 to file libclntsh.so in the Oracle Instant Client directory. #ln -s /opt/ibm/tivoli/instantclient_11_2/libclntsh.so.11.1 /opt/ibm/tivoli/instantclient_11_2/libclntsh.so Note: There is a full step-by-step guide for the Oracle Instant Client downloading in the Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent. 3.4 Mount of Oracle alert log directory As the agent is installed in the remote server, so oracle alert log directory (such as /opt/oracle/base/diag) in the Oracle server need to be mounted to agent machine. Following is sample steps: Step 1) Check if the NFS server is started in the Oracle server, or run the following command as user root. # /etc/init.d/nfs.server start Step 2) Check alert log directory permission in the Oracle server and add the rx permission for directory recursively to ensure the mount client can access the mounted alert log files, run the following command as user root. # chmod -R +rx /opt/oracle/base/diag Step 3) Share the alert log directory with following command in the Oracle server rac1. # share -F nfs -o ro=rac1 /opt/oracle/base/diag Step 4) Create a local directory and mount the remote alert log directory into local directory. 4
# mkdir /mnt/rac1 # mount rac1:/opt/oracle/base/diag /mnt/rac1 3.5 Configure of TNSNAMES.ORA file in agent site. A new file tnsnames.ora is created in the agent server. It will be used by agent to resolve the connection information from the connection string. Following is sample steps: Step 1) use editor like Vi to create a new file tnsnames.ora file in the server tivsun21 at /opt/ibm/tivoli/config directory with following sample lines. The file permission should be set as rx-rx-rx. Note: Connection name of TIVRAC must start the far left of the line; other lines should not start from left of the line. # cat /opt/ibm/tivoli/config/tnsnames.ora #RAC database connection string TIVRAC = (DESCRIPTION= ) ) (LOAD_BALANCE=ON) (FAILOVER=on) (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521)) (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=myrac) #ASM database connection string TIVASM = (DESCRIPTION= ) ) (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521)) (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=+asm) #Dataguard database primary node connection string TIVDGPRI = (DESCRIPTION= (ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521)) 5
(CONNECT_DATA=(SERVICE_NAME=dgpri) ) ) #Dataguard database standby node connection string TIVDGSTD = (DESCRIPTION= (ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg2)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=dgstd) ) ) Step 2) Edit both file /opt/ibm/tivoli/config/rz.ini and /opt/ibm/tivoli/config/rz_<instance>.config to append one line below to ensure the variable TNS_ADMIN is set as the path directed to the new file tnsnames.ora. export TNS_ADMIN=/opt/IBM/Tivoli/config 3.6 Disable of sqlnet.log file in agent site A new file sqlnet.ora is created in the agent server tivsun21. It will be used by Oracle OCI library to get the Oracle Net Service settings; one of the settings will disable the sqlnet.log file in the Oracle client side, such as /opt/ibm/tivoli/bin/sqlnet.log. Fatal NI connect error 12514, connecting to: (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=orcl)(CID=(PROGRAM=krzstar t)(host=tivagent)(user=root)))(address=(protocol=tcp)(host =rac1.cn.ibm.com)(port=1521))) Step 1) use editor like Vi to create a new file sqlnet.ora file in the agent server tivsun21 at /opt/ibm/tivoli/config directory with following lines. The file permission should be set as rx-rx-rx. LOG_DIRECTORY_CLIENT=/dev/null LOG_FILE_CLIENT=/dev/null Step 2) (Optional) Edit both file /opt/ibm/tivoli/config/rz.ini and /opt/ibm/tivoli/config/rz_<instance>.config to append one line below to ensure the variable TNS_ADMIN is set as the path directed to the new file sqlnet.ora. 3.7 Agent configuration from command line Launch agent configuration from command line and input the specific value as following. 6
Launch configuration tool from command line. #/opt/ibm/tivoli/bin/itmcmd config -A rz When ask for the input filed of Oracle Home Directory, input a blank space to make sure there is no valid value for Oracle Home Directory Oracle home directory: (default is: ): [Input space and press Enter] When ask for the input field of Oracle Instant Client Installation Directory, input the path where the Oracle Instant Client is extracted. Oracle instant client installation directory: (default is: ): /opt/ibm/tivoli/instantclient_11_2 When ask for selection to edit the database connection settings, input 1 Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1 When ask for the input filed of Database Connection Name, input the database name racnew. Database connection name: (default is: ): rac When ask for the input field of Oracle Connection String, input the tnsnames.ora connection identifier which is defined in the file /opt/ibm/tivoli/config/tnsnames.ora, current value is TIVRAC. Or you can input the full connection string directly. Oracle connection string: (default is: ): TIVRAC Oracle connection string: (default is: ): (DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRE SS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDR ESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CON NECT_DATA=(SERVICE_NAME=myrac))) When ask for the input filed of Oracle alert log file paths, input the local alert log file full paths which is mounted from remote Oracle servers. The absolute file path of mapped alert log files of remote database instances in this database connection. Multiple files are separated by ";" on Windows systems, or ":" on UNIX systems. Each file is matched to a database instance, by the file name pattern alert_<instance>.log, or ignored if unmatched. Local database instance alert log files can be discovered automatically. If this item was not configured, the alter logs of remote database instances would be unable to be collected. Oracle alert log file paths: (default is: ): /mnt/sol_rac1/rdbms/rac/rac1/trace/alert_rac1.log: /mnt/sol_rac2/rdbms/rac/rac2/trace/alert_rac2.log 7
3.8 ASM database monitoring You can use existing agent instance to add one new connection to monitor one ASM database or create a new agent instance with a new connection. When ask for selection to edit the database connection settings, input 1 Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1 When ask for the input field of Database Connection Name, input the connection name like asm. Database connection name: (default is: ): asm Note: Only letters, Arabic number, underline and minus characters can be used in the connection name. Other connection name like +asm is invalid. When ask for the input field of Oracle Connection String, input the tnsnames.ora connection identifier which is defined in the file /opt/ibm/tivoli/config/tnsnames.ora, current value is TIVASM. Or you can input the full connection string directly. Oracle connection string: (default is: ): TIVASM Oracle connection string: (default is: ): (DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRE SS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDR ESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CON NECT_DATA=(SERVICE_NAME=+ASM))) When ask for the input field of Username, Password and Role, input the user information with SYSDBA role. The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user. Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1 Note: In Oracle 10g ASM database, a new Oracle user cannot be created in the sqlplus directly, you need to create the new Oracle user and grant as SYSDBA role in the RDBMS sqlplus and copy the RDB password file to the ASM password file, reference the technotes below: http://www-01.ibm.com/support/docview.wss?uid=swg21589173 3.9 Dataguard database monitoring You can use existing agent instance to add two new connections for Dataguard Primary database and Standby database separately, or you can create a new agent instance with two new connections. 8
When ask for selection to edit the database connection settings, input 1 for Primary Node connection. Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1 When ask for the input filed of Database Connection Name, input the connection name dgpri for the primary node connection. Database connection name: (default is: ): dgpri When ask for the input field of Oracle Connection String, input the tnsnames.ora connection identifier which is defined in the file /opt/ibm/tivoli/config/tnsnames.ora, current value is TIVDGPRI. Or you can input the full connection string directly. Oracle connection string: (default is: ): TIVDGPRI Oracle connection string: (default is: ): (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1 )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgpri))) When ask for the input field of Username, Password and Role, input the user with SYSDBA role. The user with SYSDBA role is not mandatory for Dataguard Primary Node, When ask for the selection to edit the database connection settings, input 1 for the Standby Node connection. Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1 When ask for the input filed of Database Connection Name, input the connection name dgpri for the primary node connection. Database connection name: (default is: ): dgstd When ask for the input field of Oracle Connection String, input the tnsnames.ora connection identifier which is defined in the file /opt/ibm/tivoli/config/tnsnames.ora, current value is TIVDGPRI. Oracle connection string: (default is: ): TIVDGSTD Oracle connection string: (default is: ): (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1 )(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgstd))) When ask for the input field of Username, Password and Role, input the user with SYSDBA role. The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user. 9
Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1 3.10 Use non-root user to startup agent to monitor Oracle database Use ITM utility $ITMHOME/bin/itmcmd to startup agent appending with agent instance name in the -o option by non-root user. Note: You need to grant the r+w permission of files installdir/config/.config/runinfo* and installdir/logs/*rz* to the non-root user before starting the agent. #su - oracle -c /opt/ibm/tivoli/bin/itmcmd agent -o inst1 start rz #su - tivmon -c /opt/ibm/tivoli/bin/itmcmd agent -o inst1 start rz 3.11 Check the status for connections. Go to the Oracle Database Extended root node and check the Database Status column in the view of Database Connection. If the Database Status is not Active or there is not data in the view of Database Connection, then check the Error Message column and Suggestion column in the Agent Event workspace. Figure 1: Blank Database Connection and Agent Event 10
Figure 2: Dataguard Database Connections Figure 3: ASM Database Connection 3.12 Summary Apply latest Interim Fix. Use latest Oracle Instant Client library instead of Oracle Home directory. Use remote monitoring Use file tnsnames.ora in the directory $ITMHOME/config, then specify the connection string using load balance and fail over, including all RAC instances listener address in one connection definition in the file tnsnames.ora. Mount the alert log directory from remote machine to local agent machine to monitor the alert log. 11
Use non-root user to startup agent. 4 Customization 4.1 Extend agent timeout You can set the environment variable below to extend the agent itmeout from default 60 seconds to more. Table 3: Environment variable for agent timeout Variable CDP_COLLECTION_TIMEOUT Description The maximum time of data collection. If the execution time exceeded, the agent stops the execution, and reports timeout. The default value is 60 (in seconds). The minimum value is 5 (in seconds). 4.2 Listener Monitoring 4.2.1 Stop the Listener/Net Service Monitoring Refer the readme file of 6.3.1-TIV-ITM_KRZ-IF0001. https://www-304.ibm.com/support/docview.wss?uid=swg24029579 Enhancement: INTERNAL Abstract: Providing the option to control whether monitoring the listener and Oracle netservice. Additional Information: The listener monitoring and net-service monitoring are started by default. We provide the option that if you do not want this monitoring; you can turn off the related monitoring functions. The environment variables KRZ_LISTENER_PING_INTERVAL and KRZ_TNS_PING_INTERVAL control the interval. If the value is less than 1, then it exits the thread and stops checking. KRZ_LISTENER_PING_INTERVAL=0 KRZ_TNS_PING_INTERVAL=0 4.2.2 Avoid the SQLNET.LOG file generated in the ITM directory When the listener monitoring is enabled and there is some ORA error, the Oracle OCI library file will generate the sqlnet.log file in both Oracle server directory and client directory. 12
There is such file in the ITM directory when agent is running and it will get grow more and more. User can avoid such sqlnet.log file by redirecting to another directory or set to /dev/null in the Oracle configuration file sqlnet.ora. #cat $ORACLE_HOME/network/admin/sqlnet.ora grep LOG LOG_DIRECTORY_CLIENT=/dev/null LOG_FILE_CLIENT=/dev/null 4.2.3 Selected Listener/Net Services to be monitored If the Oracle home directory is selected during the agent configuration, then agent will load Listener definition from file $ORACLE_HOME/network/admin/listener.ora and load Net Service definition from file $ORACLE_HOME/network/admin/tnsnames.ora automatically. If user do not want to monitor all Listener or Net Service from ORA files in the Oracle home directory and do not want to revise those files too, customer can specify the TNS_PATH during the agent configuration and create new listener.ora file and tnsnames.ora files in the new path and only the targeted Listener or Net Service is specified in the new files. This is the directory that contains Oracle database net configuration file. This directory is defined by the TNS_ADMIN environment variable for each Oracle database instance. The default directory is $ORACLE_HOME/network/admin on UNIX or Linux systems, and %ORACLE_HOME%\NETWORK\ADMIN on Windows systems. If there are multiple net configuration file directories, use ";" on Windows systems, or ":" on UNIX systems, to separate the directories. If this item was not configured, the default directory would be used. Net configuration files directories: (default is: ): /opt/ibm/tivoli/config/myrac 4.3 Alert Log Monitoring To customize the severity of Oracle alert log messages, reference Info Center: http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcama.doc_6.2.4/orac leextagent631_user24.htm?path=2_11_5_1_5#customize_alert_log You can set environment variable in the table below for the alert log monitoring. Table 4: Variable Environment variables for alert log monitoring Description KRZ_LOG_MAXREAD Only the latest part of alert logs is scanned by the Oracle Database Extended agent while the agent starts. The size of the scanned part is specified by the KRZ_LOG_MAXREAD attribute. The default value is 5M. The value must be 13
positive. The valid data suffixes are M/m presenting megabyte, and K/k presenting kilobyte. KRZ_LOG_INTERVAL KRZ_LOG_NOREPEAT This attribute defines the search interval (in seconds) for alert log files. The default value is 300 (seconds). Setting the value to 0 disables search. The valid range is from 60 to 3600. If the value is true, the Oracle Database Extended agent does not re-send alert log message entries that were found in the last running of the Oracle Database Extended agent. The default value is FALSE. The valid values are TRUE and FALSE. 4.4 Customized SQL To define a customized SQL statement to monitor in the Customized SQL workspace, reference Info Center: http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcama.doc_6.2.4/orac leextagent631_user25.htm?path=2_11_5_1_6#customize_sql http://www.ibm.com/support/docview.wss?uid=swg21588392 You can use the environment variable KRZ_CUSTOM_SQLXML to specify a customized krzsql.xml, reference the technotes 4.5 Product Predefined SQL Customization 4.6 Cluster Support Reference to the technotes of Agent clustering and positioning in the TEP Navigator http://www-01.ibm.com/support/docview.wss?uid=swg21326517 Table 5: Agent Affinities Affnities %IBM.OracleAgents %IBM.OracleAgentRDB %IBM.OracleAgentASM %IBM.OracleAgentDG Node in the TEP Oracle Database Extended Oracle RDBMS Oracle ASM Oracle Dataguard 14
5 Problem Determination Troubleshooting: Slow Database Monitoring http://www-01.ibm.com/support/docview.wss?uid=swg21508178 Question: How do you extended the timeout period of the oracle agent? Cause: The oracle database is slow on the system and the agent often times out during monitoring. Answer: Update the CDP_COLLECTION_TIMEOUT environment variable. The default is 60 seconds and the agent needs more time to verify database connection. Troubleshooting: TOP SQL historical collection http://www-01.ibm.com/support/docview.wss?uid=swg21511760 Question: How to configure the TOP SQL attribute group historical collection for the RZ agent Answer: To configure the historical collection for the TOP SQL attribute group, the user needs to click on the third tab ("Filters") and fill the requested filters: "begin hour","end hour", "order by" and "row order". This setting is required to get the historical collection enabled for this attribute group. Historical collection settings: 15
Figure 4: History Collection Configuration for TOPSQL Troubleshooting: DB connection fails during Oracle Extended Agent configuration Problem (Abstract): Attempts to configure result in error "ORA-01722: invalid number" despite IF0001 is already applied for the RZ agent v6.3.1. Symptom: Error message "ORA-01722: invalid number" Resolving the problem: If the IF0001 has been applied, you can safely ignore the error message for test connection, as follows. 1. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini NLS_NUMERIC_CHARACTERS=. Reconfigure agent and ignore the above error message. Try to start this agent, if agent is not started, try next step (2). 2. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini NLS_NUMERIC_CHARACTERS=., Reconfigure agent and ignore the above error message. 16
Try to start this agent, if agent is not started, please collect the debugging data for the IBM support investigation as explained at next step (3). 3. Set "KBB_RAS1" in <AGENT_INSTALL_HOME>/config/rz.ini KBB_RAS1=ERROR (UNIT:krz all) Start agent and send to IBM Support the output of "digup -a" command. Also, please use "sqlplus" to connect to database directly: issue following SQL and send the output to the IBM Support select * from nls_database_parameters; select * from nls_instance_parameters; select * from nls_session_parameters; select userenv('language') from dual; select * from v$nls_valid_values; Then quit "sqlplus" and setup the environment as follows: NLS_LANG=AMERICAN_AMERICA.AL32UTF8 && export NLS_LANG NLS_NUMERIC_CHARACTERS=. && export NLS_NUMERIC_CHARACTERS Use again "sqlplus" to connect to oracle db and issue: select userenv('language') from dual select * from nls_session_parameters Then quit "sqlplus" and setup system environment again as follows: NLS_NUMERIC_CHARACTERS='.,' && export NLS_NUMERIC_CHARACTERS Finally, issue the last SQL command: Select * from nls_session_parameters Please send the outputs from all the above SQLs statements to the IBM Support for more investigation. Troubleshooting: Monitoring Oracle alert logs with remote RZ agent Question: How to remotely monitor the Oracle alert log on AIX with an RZ agent installed on Windows? 17
Answer: For monitoring remote alert logs on AIX, UNIX or Linux systems, the file system containing the logs needs to be locally mounted on the RZ agent machine. To mount AIX file systems in Windows, you need to install Samba on AIX. Troubleshooting: Oracle Extended Agent (rz) fails loading Oracle OCI library Problem (Abstract): The Oracle Extended Agent is not running properly and produces the error: Loading Oracle OCI library failed! Symptom: The Oracle Extended Agent starts and connects to TEMS but the Agent does not display Oracle data. Cause: This error occurs because the Oracle Extended Agent(rz) can not use the Oracle Client Interface(OCI) to connect to the Oracle instance. Diagnosing the problem: The Oracle Extended Agent on unix/linux starts the krzagent and then launches the krzclient process. The krzclient process fails when it tries to connect to the Oracle database. instance_rz_server_krzclient_timestamp.log reflects the following errors: Fail to dynamically load library libclntsh.so.10.1, return code 8 Try to load another possible Oracle OCI library Fail to dynamically load library libclntsh.so.11.1, return code 2 Loading Oracle OCI library failed! Resolving the problem: Reconfigure the agent and correct Oracle home or Oracle Instant Client Library. Reconfigure the agent and restart. 18
APPENDIX 6 Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent Step 1) open Oracle Instant Client download page from Oracle Site below: http://www.oracle.com/technetwork/database/features/instant-client/index- 097480.html Step 2) click one suitable platform from list, especially for the 32-bit and 64-bit platforms. Figure 5: Selection of Oracle Instant Client Downloads Step 3) select Accept License Agreement. For example: vi
Instant Client Downloads for Solaris Operating System(SPARC 64-bit) Step 4) download Instant Client Package - Basic: All files requires to run OCI, OCCI, and JDBC-OCI applications. basic-11.2.0.2.0-solaris-sparc64.zip (64,102,971 bytes) (cksum - 4103480643) vii
REFERENCES [Reference List] (Use bibliography format. Style automatically creates a hanging indent when text runs over to the next line.)
错 误! 文 档 中 没 有 指 定 样 式 的 文 字 Best Practices for Oracle Monitoring with Oracle Extended Agent. Copyright IBM Corporation 2009 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC 27709 U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml. ==> The listed trademarks of the following companies require marking and attribution only if you mention any of them in your paper: 9
错 误! 文 档 中 没 有 指 定 样 式 的 文 字 Best Practices for Oracle Monitoring with Oracle Extended Agent. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office UNIX is a registered trademark of The Open Group in the United States and other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. 10