DOCUMENTUM CONTENT SERVER CERTIFICATE BASED SSL CONFIGURATION AND TROUBLESHOOTING

Transcription

1 White Paper DOCUMENTUM CONTENT SERVER CERTIFICATE BASED SSL CONFIGURATION AND TROUBLESHOOTING Abstract This White Paper explains configuration for enabling Certificate based SSL for secure communication to Connection Broker and Content Server. It also describes steps to troubleshoot issues faced during configuration. March 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Part Number h

3 Table of Contents Executive summary... 4 Audience... 4 Configuration... 4 Docbroker... 5 Create Docbroker Keystore... 5 Create Docbroker Keystore's password file... 6 Docbroker configuration... 6 Start Docbroker... 7 Server... 7 Create Server Keystore... 7 Create Server Keystore's password file... 8 Create Server Trust-Store... 8 Server configuration... 9 Start Server... 9 DFC... 9 Create DFC Trust-Store... 9 DFC configuration Troubleshooting Docbroker startup fails Server startup fails Server not able to connect to Docbroker Clients not able to connect to Docbroker Clients not able to connect to Server Multiple Docbase Miscellaneous Conclusion References Appendix Content Server Error Messages Quick Reference

4 Executive summary Content Server and Connection Broker support connections in both native and secure modes. For secure connections, Anonymous SSL is used by default. Support for Non Anonymous or Certificate based SSL for communication is new feature introduced in Content Server to further enhance communication security. For using Certificate based SSL, Content Server, Docbroker and Clients needs to be configured after installation. This paper explains in detail configuration and troubleshooting steps. All the components (Server, Docbroker and Clients) should use same mode for secure communication. Mixed environments are not supported, i.e. either all the components have to use Anonymous SSL or Non Anonymous SSL for communication. For Certificate based SSL, only supported cipher is AES128-SHA for maintaining backward compatibility with RSA libraries. Audience The audience for this white paper comprises personnel responsible for the Installation, configuration and deployment of Content Server in production environments. This document is intended for internal EMC teams, partners and customers. Configuration This section describes the steps required to configure different components of the system (Docbroker, Content Server, Clients) to use certificate based SSL for communication. Following needs to be configured: 1. Docbroker as SSL server (Content Server & DFC as SSL clients) 2. Content Server as SSL server (DFC as SSL client) and SSL Client (Docbroker as SSL Server) 3. DFC as SSL client To enable clients to use Certificate based SSL communication for secure connections to Docbroker and Content Server, we need to install Docbroker and Content Server in secure or 'native & secure' mode. After installation is complete, all the services needs to be stopped (Docbroker, Content Server & Method Server) and then system needs to be configured for Certificate based SSL manually. Currently Content Server installer do not support automatic configuration of Certificate based SSL, so this configuration needs to be done manually after installation. 4

5 Docbroker Create Docbroker Keystore For configuring Docbroker as SSL server, a private key needs to be created for Docbroker and Certificate Signing Request (CSR) needs to be generated using private key. This CSR then needs to be presented to and signed by Certificate Authority (CA) for generating Docbroker's public certificate. Finally, we need to create a Keystore for Docbroker to store its private key and public certificate. All the above steps are described here in detail. In this paper OpenSSL tool is used for managing Public Key Infrastructure (PKI). Below command can be used to generate Docbroker's private key and CSR. It will generate private key (brokerkey.pem) and CSR (brokercsr.pem) in PEM (text) format. CSR (brokercsr.pem) can then be presented to some CA for generating Docbroker's public certificate. openssl req -newkey rsa:2048 -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -keyout brokerkey.pem -out brokercsr.pem Loading 'screen' into random state - done Generating a 2048 bit RSA private key writing new private key to 'brokerkey.pem' Enter PEM pass phrase: Verifying - Enter PEM pass phrase: For the purpose of this paper, we will show the configuration using self-signed certificates instead of CA signed certificates. Below command will generate Docbroker's private key (brokerkey.pem) and self-signed public certificate (brokercrt.pem), both in PEM format. Option -x509 is used to self-sign the public certificate. openssl req -newkey rsa:2048 -x509 -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -keyout brokerkey.pem -out brokercrt.pem 5

6 Docbroker will look for private key and public certificate in Keystore which should be in PKCS #12 format. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure as Docbroker will look in this directory for locating its keystore. To generate Docbroker's keystore (broker.p12) by storing key and self-signed public certificate, use below command: openssl pkcs12 -export -out broker.p12 -inkey brokerkey.pem -in brokercrt.pem -name broker Loading 'screen' into random state - done Enter pass phrase for brokerkey.pem: Enter Export Password: Verifying - Enter Export Password: When executed, it will ask for key s and Keystore's password. Option -name is specified to provide an alias for keys in keystore. Create Docbroker Keystore's password file Keystores are secured by making them password protected. Docbroker Keystore's password is encrypted and stored in a file which will be used by Docbroker for accessing Keystore. Docbroker will look in directory $DOCUMENTUM/dba/secure for locating password file. There is no restriction on password file's name or extension. Below command generates password file with keystore's password: dm_encrypt_password -encrypt <password> -file <file> e.g dm_encrypt_password -encrypt broker -file broker.pwd dm_encrypt_password utility is installed with Content Server and will work on machines on which Content Server is installed. Docbroker configuration Docbroker will use properties in Docbroker s configuration file (<Docbroker>.ini) for resolving Keystore and password file names. Property keystore_file is used to specify Keystore file name and keystore_pwd_file is used to specify file name in which Keystore's password is stored. 6

7 Put below mandatory properties in Docbroker s configuration file (<Docbroker>.ini) in $DOCUMENTUM/dba directory, for the files that we generated in previous steps. keystore_file=broker.p12 keystore_pwd_file=broker.pwd Start Docbroker Docbroker should start successfully on secure port (1490). Server Create Server Keystore Similar to Docbroker, private key and public certificates needs to be generated for Server and stored in Server's Keystore from where Server will access them. Below command can be used to generate Server's private key and CSR. openssl req -newkey rsa:2048 -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -keyout serverkey.pem -out servercsr.pem For generating private key (serverkey.pem) and self-signed public certificate (servercrt.pem), use above command with x509 option as below: openssl req -newkey rsa:2048 -x509 -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -keyout serverkey.pem -out servercrt.pem Loading 'screen' into random state - done Generating a 2048 bit RSA private key writing new private key to 'serverkey.pem' Enter PEM pass phrase: Verifying - Enter PEM pass phrase:

8 These keys generated should now be stored in Keystore. Server Keystore should be in PKCS #12 format. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure where server will look to locate its Keystore. Generate Server's Keystore (server.p12) containing private key and selfsigned public certificate using below command: openssl pkcs12 -export -out server.p12 -inkey serverkey.pem -in servercrt.pem -name server Loading 'screen' into random state - done Enter pass phrase for serverkey.pem: Enter Export Password: Verifying - Enter Export Password: Create Server Keystore's password file Server Keystore's password is encrypted and stored in a file in directory $DOCUMENTUM/dba/secure. There is no restriction on password file's name or extension. Below command generates password file with Keystore's password: dm_encrypt_password -encrypt server -file server.pwd Create Server Trust-Store When acting as SSL client to Docbroker, Server needs to verify public certificate sent by Docbroker. For verifying Docbroker's public certificate, Docbroker s public certificate or CA certificate which was used to sign Docbroker's public certificate needs to be stored in a Keystore. This Keystore will act as Server's trust-store and server will verify Certificates sent by Docbroker against certificates in this keystore. Server expects trust-store to be in PKCS #7 binary format. There is no restriction on trust-store's name or extension. It should be placed in location $DOCUMENTUM/dba/secure. Below command generates trust-store by importing Docbroker's certificate. openssl crl2pkcs7 -nocrl -certfile brokercrt.pem -outform der -out server-trust.p7b 8

9 Server configuration Server resolves file names of Server Keystore, trust-store and Keystore s password files by reading properties from Server configuration file (server.ini). Property keystore_file is used to specify the name of Server's keystore, keystore_pwd_file to specify password file name and truststore_file to specify trust-store name. All the below properties are mandatory. Put below properties in Server Configuration file (server.ini): keystore_file=server.p12 keystore_pwd_file=server.pwd truststore_file=server-trust.p7b Start Server Server should start successfully and errors related to connection to Docbroker should not be displayed. DFC Create DFC Trust-Store When DFC will be acting SSL client to Docbroker or Server, it will use a trust-store to store trusted certificates. Certificates sent by Docbroker or Server will be validated against the certificates in DFC trust-store. DFC trust store should be in JKS (Java Keystore) format. There is no restriction on store's name or extension as well as location. We will convert the Docbroker & Server public certificates that were generated in text format to binary format using OpenSSL. Then public certificates of Docbroker and Server in binary format will be imported to DFC trust-store. Below commands converts certificates from PEM to DER format: openssl x509 -outform der -in brokercrt.pem -out brokercrt.der openssl x509 -outform der -in servercrt.pem -out servercrt.der 9

10 Following commands create DFC trust-store (dfc.keystore) by importing Docbroker's & Server's public certificates: keytool -importcert -keystore dfc.keystore -file brokercrt.der -alias broker keytool -importcert -keystore dfc.keystore -file servercrt.der -alias server DFC configuration DFC will look in dfc.properties file to resolve trust-store location, name and password. Property dfc.security.ssl.truststore is used to specify trust-store's path and dfc.security.ssl.truststore_password to specify trust-store's password. Trust-store's password can be given in plain text or encrypted format. In dfc.properties, put entries for DFC trust-store and trust-store's password: dfc.security.ssl.truststore = c\:/secure/dfc.keystore dfc.security.ssl.truststore_password = password DFC trust-store password can be encrypted using DFC utility as below: java com.documentum.fc.tools.registrypasswordutils <password> E.g. java com.documentum.fc.tools.registrypasswordutils password MmKZP8KOlvrE6ndUolyl6Q== The encrypted password generated by this command can be copied and pasted to properties file. For this command to execute successfully, dfc.jar should be in java classpath. There is an additional property dfc.security.ssl.use_existing_truststore that is mutually exclusive to above two properties. When this property is specified, there is no need to put above two properties. In this case, Java Keystore will act as DFC truststore or trust-store can be specified using JVM parameter javax.net.ssl.truststore. For secure connections to Server and Docbroker, put the value of property dfc.session.secure_connect_default as secure. Same property controls behavior of connections to both Docbroker and Server. Put below property in dfc.properties file: 10

11 secure.dfc.session.secure_connect_default = secure Troubleshooting This section describes some of the common issues faced during configuration and steps to troubleshoot. Docbroker startup fails 1. Check if Docbroker Keystore and Keystore's password files are present in $DOCUMENTUM/dba/secure 2. Check if entry for above two files is there in Docbroker configuration file (<Docbroker>.ini) 3. Check if the format of Docbroker Keystore is PKCS #12. Below commands should list the keys in Keystore if it s in PKCS #12 format, else error will be displayed. Using OpenSSL: openssl pkcs12 -info -in <keystore> Using Keytool: keytool -list v -storetype pkcs12 -keystore <keystore> 4. Check if password in Keystore password file is correct. For testing, password can be put in plain-text (without encryption). Server startup fails 11

12 1. Check if Server Keystore, Server Keystore password and Server trust-store files are present in $DOCUMENTUM/dba/secure 2. Check if entry for above three files is there in Server configuration file (server.ini) 3. Check if the format of Server Keystore is PKCS #12. See commands in Docbroker startup fails section to verify. 4. Check if server trust-store is in PKCS #7 binary (der) format. For verifying, check if below command dumps it successfully: openssl pkcs7 -in <Keystore> -inform der E.g. openssl pkcs7 -in server-trust.p7b -inform der Server not able to connect to Docbroker 1. Check whether Docbroker Keystore has proper key and public certificate. Use below command to print the keys in keystore: Using OpenSSL: openssl pkcs12 -info -in <keystore> Using Keytool: keytool -list -storetype pkcs12 -keystore <keystore> 2. Check whether Docbroker is sending proper certificates. Below command will start a simple client which tries to connect to SSL Server and displays certificate chain sent by Server: openssl s_client -showcerts -debug -connect <SSL_Server_IP>:<SSL_Server_Port> 12

13 E.g. openssl s_client -showcerts -debug -connect : Check if Server's trust-store contains Docbroker's public certificate or CA certificate chain used to sign Docbroker's public certificate Use below command to display all certificates in trust store: openssl pkcs7 -in <trust-store> -inform der -print_certs -text Clients not able to connect to Docbroker 1. Check if proper entries are present in dfc.properties and trust-store file exists. 2. Check if DFC trust-store contains Docbroker's public certificate or CA certificate used to sign Docbroker s public certificate. To dump Trust-store's contents (no need to specify storetype as default type is JKS): keytool -list -keystore <keystore> -storepass <storepass> Clients not able to connect to Server 1. Check if proper entries are made in dfc.properties and trust-store file exists. 2. Check if DFC trust-store contains Server's public certificate or CA certificate used to sign Server s public certificate. To dump Trust-store's contents (no need to specify storetype as default type is JKS): keytool -list -keystore <keystore> -storepass <storepass> 3. Check if dfc.properties has right value for trust-store password. For verification, password can be given in plain-text. Multiple Docbase 13

14 If Sever and Docbroker are configured in secure mode with Certificate based SSL and second Docbase is created, installer will hang for some time in middle during step where docbase tries to connect to existing Docbroker. This issue occurs as Docbase will try to connect using Anonymous SSL by default and since mixed mode is not supported, it won t be able to connect. Follow below steps to resolve this issue: 1. While the installer is hanging, open server.ini of the new Server 2. Add the proper options for Certificate based SSL 3. Start/restart the new Server Miscellaneous 1. To verify that connection is secure and check encryption algorithm used, use jvm parameter -Djavax.net.debug=all, when starting the client. Conclusion Documentum support for Certificate based SSL communication to Connection Broker and Content Server improves security in Documentum products. This paper describes how to configure deployment to use this feature. For further information on installation, Content Server Enterprise Edition Installation Guide can be referenced. References 1. Content Server Enterprise Edition Installation Guide 2. OpenSSL Documentation: 3. Keytool Documentation (Java SE7): Appendix Content Server Error Messages Quick Reference 14

15 Table 1 Content Server Error Messages Error Message [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error (705) SSL error(10100) in call to SSL_CTX_new(). Winsock error: " [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_open(). Error (501) Network Library not initialized. Extended network error: 0." [ERROR] [AGENTEXEC 3088] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" javax.net.ssl.sslhandshakeexception: Received fatal alert: handshake_failure [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error Unknown error code (_nl_error_ = 0). Extended network error: 0." Possible Cause Server Keystore missing Server Keystore different format than PKCS #12 Server Keystore corrupt Server Keystore password file contains wrong password Server Keystore password file missing Server Keystore empty Server trust-store missing Server trust-store corrupt [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_open(). Error (501) Network Library not initialized. Extended network error: 0." [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_init(). Error Unknown error code 1795 (_nl_error_ = 0). Extended network error: 0." [DM_SERVER_E_START_NETWISE]error: "The server failed to start due to an error returned by the Netwise networking subsystem. Failed API: nl_open(). Error (501) Network Library not initialized. Extended network error: 0." [DM_DOCBROKER_E_NETWORK_ERROR]error: "An error occured performing a network operation: (Unknown error code 112 (_nl_error_ = 0)). Network specific error: (Extended network error: 0)." Server trust-store wrong format Server trust-store empty Docbroker's keystore empty [DM_DOCBROKER_E_CONNECT_FAILED_EX]error: "Unable to connect to DocBroker. Clients please check your dfc.properties file for a correct host. Server please check your server.ini or target 15

16 attributes in server config. Network address: (INET_ADDR: family: 2, port: <port>, host: <host_name> (<host_ip>, a))." [ERROR] [AGENTEXEC 2692] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker <docbroker_name>:<port> failed DFC trust-store missing [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" javax.net.ssl.sslexception: java.lang.runtimeexception: Unexpected error: java.security.invalidalgorithmparameterexception: the trustanchors parameter must be non-empty java.lang.runtimeexception: Unexpected error: java.security.invalidalgorithmparameterexception: the trustanchors parameter must be non-empty java.security.invalidalgorithmparameterexception: the trustanchors parameter must be non-empty [ERROR] [AGENTEXEC 2968] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker <docbroker_name>:<port> failed DFC trust-store missing Docbroker's Certificate [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" javax.net.ssl.sslhandshakeexception: sun.security.validator.validatorexception: PKIX path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target sun.security.validator.validatorexception: PKIX path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target 16

17 [ERROR] [AGENTEXEC 2436] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" DFC trust-store missing Server's Certificate javax.net.ssl.sslhandshakeexception: sun.security.validator.validatorexception: PKIX path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target sun.security.validator.validatorexception: PKIX path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target [ERROR] [AGENTEXEC 1672] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker <docbroker_name>:<port> failed DFC trust-store different format [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" java.net.socketexception: java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.io.ioexception: Invalid keystore format 17

18 [ERROR] [AGENTEXEC 2608] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker <docbroker_name>:<port> failed DFC trust-store corrupt [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" java.net.socketexception: java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.security.cert.certificateexception: Unable to initialize, java.io.ioexception: DerInputStream.getLength(): lengthtag=127, too big. java.io.ioexception: DerInputStream.getLength(): lengthtag=127, too big. [ERROR] [AGENTEXEC 5720] Detected during program initialization: Command Failed: connect,<server_name.docbase_name>,<user>,'',,,try_native_first, status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker <docbroker_name>:<port> failed [DM_SESSION_E_RPC_ERROR]error: "Server communication failure" java.net.socketexception: java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.security.nosuchalgorithmexception: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: com.sun.net.ssl.internal.ssl.defaultsslcontextimpl) java.io.ioexception: Keystore was tampered with, or password was incorrect java.security.unrecoverablekeyexception: Password verification failed DFC trust-store wrong password 18