PURPOSE 14 th June, 2011 This paper demonstrates how to resolve the Oracle Enterprise Manager Database Control configuration errors in Oracle Database versions 10.2.0.4 or 10.2.0.5, arising due to the Root Certificate Expiry issue since 31st December, 2010. CASE STUDY We use Oracle Enterprise Manager Database Control for our 10.2.0.4 Production Box. All was fine until mid of February 11, when we noticed some stale information on Enterprise Manager Database Control. Despite carrying out the respective activities and the scheduled runs (like ADDM, Segment Advisor, etc.), for some reason the latest collection/information was not reflected on the Enterprise Manager Database Control Dashboard. All it showed was couple of days old data. By the end of March, we decided to drop the Enterprise Manager and create it afresh, so as to resolve the inconsistent data reflection issue. To our surprise, whenever we tried to setup Enterprise Manager (manually or using DBCA), it would fail to create the repository with exceptions/errors in the emca.log file. To simulate this issue and find the root cause of the problem, we tried the clean setup of Enterprise Manager Database Control on two different test databases, which subsequently resulted in similar repository creation errors, as follows: Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.emreposconfig invoke SEVERE: Error creating the repository Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.emreposconfig invoke INFO: Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\DBTEST\emca_repos_create_<date>.l og for more details. Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.emconfig perform SEVERE: Error creating the repository Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\DBTEST\emca_2011-04-06_12-36-45- PM.log for more details. Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.emconfig perform CONFIG: Stack Trace: oracle.sysman.emcp.exception.emconfigexception: Error creating the repository at oracle.sysman.emcp.emreposconfig.invoke(emreposconfig.java:204) at oracle.sysman.emcp.emreposconfig.invoke(emreposconfig.java:134) at oracle.sysman.emcp.emconfig.perform(emconfig.java:171) at oracle.sysman.emcp.emconfigassistant.invokeemca(emconfigassistant.java:486) at oracle.sysman.emcp.emconfigassistant.performconfiguration(emconfigassistant.java :1142) at oracle.sysman.emcp.emconfigassistant.statusmain(emconfigassistant.java:470) at oracle.sysman.emcp.emconfigassistant.main(emconfigassistant.java:419) 1
Upon investigation, we found out that the Root Certificate from Certification Authority, which is used to secure communications via the Secure Socket Layer (SSL) protocol, has expired on 31 st December, 2010 for Oracle Database versions 10.2.0.4 and 10.2.0.5. And, if anyone who installs or tries to secure Enterprise Manager Database Control on or after 31 st December, he is likely to face configuration errors, just as we did. In a nutshell, this is what My Oracle Support (MOS) had to say on the subject: ATTENTION! After 31-Dec-2010, creating/recreating/securing 10.2.0.4/10.2.0.5 EM DB Control will fail due to the expiration of the Certificate Authority. More informations in: NOTE 1217493.1 ATTENTION: Patch Required If You Plan To Configure Enterprise Manager Database Control With Oracle Database 10.2.0.4 Or 10.2.0.5 On Or After 31-Dec-2010 NOTE 1222603.1 Recovering From Database Control Configuration Errors Due to CA Expiry on Oracle Database 10.2.0.4 or 10.2.0.5 One needs to apply Patch 8350262 to the Oracle Home of 10.2.0.4 or 10.2.0.5 Databases, before configuring the Enterprise Manager Database Control. No database downtime is necessary to apply the patch. Also, the MOS Note says that the existing Database Control configurations are not impacted by the Root Certificate Expiry issue. Likewise, we encountered the Enterprise Manager configuration issues only after attempting the re-install of Enterprise Manager Database Control. Had we not noticed the stale information on our Enterprise Manager Dashboard, we would not have attempted the Enterprise Manager reinstall and would not have known about the Root Certificate Expiry issue either. Why the accurate information was not reflected on the Enterprise Manager Dashboard, could have been related to the Root Certificate Expiry issue or could have been not. We didn t investigate on this to conclude on it. But one thing we can say is that we haven t noticed the stale information issue after having patched the production box with Patch 8350262 till date. Here, I will demonstrate how we applied the Patch 8350262 to our Single-Instance, Test Oracle Database to resolve the Enterprise Manager configuration issues. The Operating System used was Microsoft Windows 2003 and the Oracle Database Product used was 10g Release 2 (10.2.0.4). 2
GETTING STARTED The two My Oracle Support Notes, namely 1222603.1 and 1217493.1, entail detailed explanation and resolution for successful Enterprise Manager Database Control configuration for both Single Instance as well as RAC Databases. In circumstances, where the Enterprise Manager Database Control re-install or re-creation has not been attempted yet, you can stop the Oracle Database Console and relevant services and directly apply the patch. Once the patch is successful, you can start the all the stopped services. In our case, as we had already tried to re-install the Enterprise Manager Database Control and had encountered the configuration errors, hence we had to clean the existing Enterprise Manager Database Control installation before we could attempt the patch. Before we apply the patch, we need to ensure that: 1. We have downloaded and extracted the Patch 8350262. 2. We have downloaded the supporting version of OPatch utility for 10.2.0.4 Oracle Home. 3. We have removed any trails of failed Enterprise Manager Installation, if applicable. NOTE: OPatch Version 10.2.0.5.1 is the version used to patch both 10.2.0.4 as well as 10.2.0.5 databases. The relevant OPatch utility for 10.2.0.4 Oracle Home and the Patch 8350262 can be downloaded using My Oracle Support website (http://support.oracle.com/). Once the necessary files are downloaded, the OPatch utility needs to be extracted to a reference directory. So, we created a new directory OPatch in our Oracle Home, extracted all the files in it. C:\oracle\product\10.2.0\db_1\OPatch>opatch version Invoking OPatch 10.2.0.5.1 OPatch Version: 10.2.0.5.1 OPatch succeeded. Then, we need to set the Oracle Home and set the path for OPatch such that the executables appears in the system PATH. C:\>set path=c:\oracle\product\10.2.0\db_1\opatch C:\>set oracle_home=c:\oracle\product\10.2.0\db_1 3
Next, we need to verify the OUI Inventory. If any errors are observed in this step, then we should contact Oracle Support for its resolution. Make sure this step is successful before attempting the patch. NOTE: Ensure that OUI Inventory verification is successful before attempting the patch. C:\>opatch lsinventory Invoking OPatch 10.2.0.5.1 Oracle Interim Patch Installer version 10.2.0.5.1 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home : C:\oracle\product\10.2.0\db_1 Central Inventory : C:\Program Files\Oracle\Inventory from : n/a OPatch version : 10.2.0.5.1 OUI version : 10.2.0.4.0 OUI location : C:\oracle\product\10.2.0\db_1\oui Log file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch2011-04-06_11-40-08AM.log Patch history file: C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch_history.txt Lsinventory Output file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\lsinv\lsinventory2011-04-06_11-40-08AM.txt -------------------------------------------------------------------------------- Installed Top-level Products (3): Oracle Database 10g 10.2.0.1.0 Oracle Database 10g Products 10.2.0.1.0 Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0 There are 3 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. -------------------------------------------------------------------------------- OPatch succeeded. Next, we need to create a directory to retain the Patch 8350262 files in. Accordingly, we created a new directory 8350262 under OPatch directory and then extracted the patch files from the archive file. C:\oracle\product\10.2.0\db_1\OPatch> C:\oracle\product\10.2.0\db_1\OPatch>cd 8350262 C:\oracle\product\10.2.0\db_1\OPatch\8350262> Before you begin to apply the patch, ensure that you have stopped the Database Console service. The database and listener services do not need to be stopped, but ensure that all java processes running from Oracle Home are stopped. 4
Using opatch apply command, you can now initiate the patch application process. After performing the prerequisite checks, the OPatch utility will prompt you to enter your My Oracle Support (MOS) username and password for receiving any future MOS Security updates. You may ignore this or set it up, as per your comfort. Choosing not to receive the MOS Security updates will not hinder the patch application process in any way. To ignore the MOS Setup, just leave the password blank or supply NONE when prompted for MOS configuration and Proxy information. Before applying the Patch 8350262, the OPatch utility will backup the necessary files for any possible rollbacks. NOTE: We have changed the MOS ID to a dummy ID (abc@xyz.com) for demonstration purpose. C:\oracle\product\10.2.0\db_1\OPatch\8350262>opatch apply Invoking OPatch 10.2.0.5.1 Oracle Interim Patch Installer version 10.2.0.5.1 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home : C:\oracle\product\10.2.0\db_1 Central Inventory : C:\Program Files\Oracle\Inventory from : n/a OPatch version : 10.2.0.5.1 OUI version : 10.2.0.4.0 OUI location : C:\oracle\product\10.2.0\db_1\oui Log file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch2011-04-06_11-49-09AM.log Patch history file: C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch_history.txt ApplySession applying interim patch '8350262' to OH 'C:\oracle\product\10.2.0\db_1' Running prerequisite checks... Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/user Name. Visit http://www.oracle.com/support/policies.html for details. Email address/user Name: abc@xyz.com Provide your My Oracle Support password to receive security updates via your My Oracle Support account. Password (optional): Unable to establish a network connection to Oracle. If your systems require a proxy server for outbound Internet connections, enter the proxy server details in this format: [<proxy-user>@]<proxy-host>[:<proxy-port>] If you want to remain uninformed of critical security issues in your configuration, enter NONE Proxy specification: NONE 5
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only. Backing up files and inventory (not for auto-rollback) for the Oracle Home Backing up files affected by the patch '8350262' for restore. This might take a while... Backing up files affected by the patch '8350262' for rollback. This might take a while... Patching component oracle.sysman.agent.core, 10.2.0.4.0a... "\sysman\jlib\emcore.jar\oracle\sysman\eml\sec\fsc\fswalletutil.class" "\sysman\jlib\emcore.jar\oracle\sysman\eml\sec\rep\repwalletutil.class" "\sysman\jlib\emcore.jar\oracle\sysman\eml\sec\util\rootcert.class" "\sysman\jlib\emcore.jar\oracle\sysman\eml\sec\util\secconstants.class" "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\fsc\fswalletutil.class" "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\rep\repwalletutil.class" "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\rootcert.class" "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\secconstants.class" ApplySession adding interim patch '8350262' to inventory Verifying the update... Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 8350262 are present in Oracle Home. OPatch succeeded. C:\oracle\product\10.2.0\db_1\OPatch\8350262> Once the patch is applied, the OPatch utility verifies the affected files and the OUI Inventory, before confirming that the patch process is a success. In the situation where the Enterprise Manager services were stopped simply to apply the patch, we can now start the Oracle Database Console and related services, and start using the Enterprise Manager Database Control Dashboard. For those of us who would want to install, secure or re-create the Enterprise Manager Database Control, then these can now be carried out smoothly. 6