DOCUMENTUM CONTENT SERVER CERTIFICATE BASED SSL CONFIGURATION WITH CLIENTS ABSTRACT This white paper is step-by-step guide for Content Server 7.2 and above versions installation with certificate based SSL for secure communication with clients. It explains creation of certificates for Connection Broker and Content Server. It also describes steps to configure with clients like standalone DFC and Documentum Administrator. July, 2015 EMC WHITE PAPER 1
To learn more about how EMC products, services, and solutions can help solve your business and IT challenges, contact your local representative or authorized reseller, visit www.emc.com, or explore and compare products in the EMC Store Copyright 2015 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware and <insert other VMware marks in alphabetical order; remove sentence if no VMware marks needed. Remove highlight and brackets> are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number H14393 2
TABLE OF CONTENTS ABSTRACT... 1 TABLE OF CONTENTS... 3 INTRODUCTION... 4 Intended audience... 4 Terminology... 4 CONFIGURATION... 4 DOCBROKER... 4 Create Docbroker Keystore... 4 Create Docbroker Kestore's password file... 5 SERVER... 5 Create Server Keystore... 5 Create Server Kestore's password file... 5 Create Server Trust Store... 6 DFC... 6 Create DFC Trust-Store... 6 CONTENT SERVER INSTALLATION... 6 Docbroker Configuration with Installer... 7 Manual Steps for Docbroker Configuration... 9 Server Configuration with Installer... 9 Manual Steps for Server Configuration... 12 UPGRADE... 12 CLIENTS CONFIGURATION... 14 DFC Configuration... 14 Steps to install DFC client with certificate based SSL by installer... 14 Manual Steps to configure DFC client with certificate based SSL... 15 Documentum Administrator Configuration... 16 Steps to configuration of DA... 16 TROUBLESHOOTING... 16 CONCLUSION... 18 REFERENCES... 18 3
INTRODUCTION Documentum release 7.2 support content server and connection broker connections in both native and secure modes. For secure connections, Anonymous SSL is used by default. Support for Non Anonymous or Certificate based SSL for communication is new feature introduced in Content Server to further enhance communication security. For using Certificate based SSL, Content Server, Docbroker and Clients needs to be configured either while installation or after installation. This paper explains in detail configuration and troubleshooting steps. All the components (Server, Docbroker and Clients) should use same mode for secure communication. Mixed environments are not supported, i.e. either all the components have to use Anonymous SSL or Non Anonymous SSL for communication. For Certificate based SSL deferent ciphers are supported. For all RSA libraries supported ciphers please refer 7.2 content server installation guide. Intended audience The audience for this white paper comprises personnel responsible for the Installation, configuration and deployment of Content Server in production environments. This document is intended for internal EMC teams, partners and customers. Terminology Connection Broker / Docbroker: A connection broker or Docbroker is name server for Content Server. Repository / Docbase: A repository or docbase is logical entity that provides abstraction to storage that stores native content files and RDBMS that stores content metadata. CONFIGURATION This section describes the steps required to configure different components of the system (Docbroker, Content Server and Clients) to use certificate based SSL for communication. Following needs to be configured: 1. Docbroker as SSL server (Content Server & DFC as SSL clients) 2. Content Server as SSL server (DFC as SSL client) and SSL Client (Docbroker as SSL Server) 3. DFC as SSL client To enable clients to use Certificate based SSL communication for secure connections to Docbroker and Content Server, we need to install Docbroker and Content Server in secure or 'native & secure' mode. Content Server 7.2 installer support automatic configuration of Certificate based SSL. So this configuration can do in two ways. One method is Configure the docbroker and docbase while installation by selecting the certificates. Second method is after installation is complete, all the services need to be stopped (Docbroker, Content Server & Method Server) and then system needs to be configured for Certificate based SSL manually. DOCBROKER Create Docbroker Keystore For configuring Docbroker as SSL server, a private key needs to be created for Docbroker and Certificate Signing Request (CSR) needs to be generated using private key. This CSR then needs to be presented to and signed by Certificate Authority (CA) for generating Docbroker's public certificate. Finally, we need to create a Keystore for Docbroker to store its private key and public certificate. In this paper we are creating self-signed certificates instead of CA signed certificates. 4 All the above steps are described here in detail. In this paper OpenSSL tool is used for managing Public Key Infrastructure (PKI). Below command will generate Docbroker s private key (brokerkey.pem) and self-signed public certificate (brokercrt.pem), both in PEM format.
openssl req -newkey rsa:2048 -x509 -nodes -sha256 -keyout brokerkey.pem -out brokercrt.pem -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.com/emailAddress=broker@dctm.com" Docbroker will look for private key and public certificate in Keystore which should be in PKCS #12 format. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure as Docbroker will look in this directory for locating its keystore. To generate Docbroker's keystore (broker.p12) by storing key and self-signed public certificate, use below command: openssl pkcs12 -export -out broker.p12 -inkey brokerkey.pem -in brokercrt.pem -name broker -descert When executed, it will ask for key s and Keystore's password. Option -name is specified to provide an alias for keys in keystore Create Docbroker Keystore s password file Keystores are secured by making them password protected. Docbroker Keystore's password is encrypted and stored in a file which will be used by Docbroker for accessing Keystore. Plain password also will work without any error. Docbroker will look in directory $DOCUMENTUM/dba/secure for locating password file. There is no restriction on password file's name or extension. Below command generates password file with keystore's password: dm_encrypt_password encrypt <password> -keyname <AEK Name> [-passphrase <AEK Passphrage>] -lockbox <lockbox> -lockboxpassphrase <Lockbox passphrase> -file <file> e.g : dm_encrypt_password -encrypt broker -file broker.pwd -keyname CSaek -passphrase Password@123 -lockbox lockbox.lb -lockboxpassphrase Password@123 dm_encrypt_password utility is installed with Content Server and will work on machines on which Content Server is installed. SERVER Create Server Keystore Similar to Docbroker, private key and public certificates needs to be generated for Server and stored in Server's Keystore from where Server will access them. Below command will generate Server s private key (serverkey.pem) and self-signed public certificate (servercrt.pem), both in PEM format. openssl req -newkey rsa:2048 -sha256 -x509 -nodes -keyout serverkey.pem -out servercrt.pem -days 365 -subj "/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.com/emailAddress=server@dctm.com" Server will look for private key and public certificate in Keystore which should be in PKCS #12 formats. There is no restriction on Keystore's filename or extension. It should be placed in $DOCUMENTUM/dba/secure where server will look in this directory for locating its keystore. To generate Server's keystore (server.p12) by storing key and self-signed public certificate, use below command: openssl pkcs12 -export -out server.p12 -inkey serverkey.pem -in servercrt.pem -name server descert Create Server Keystore s password file Server Keystore's password is encrypted and stored in a file in directory $DOCUMENTUM/dba/secure. Plain password also will work without any error. There is no restriction on password file's name or extension. Below command generates password file with Keystore's password: dm_encrypt_password encrypt <password> -keyname <AEK Name> [-passphrase <AEK Passphrage>] -lockbox <lockbox> -lockboxpassphrase <Lockbox passphrase> -file <file> 5
e.g : dm_encrypt_password -encrypt server -file server.pwd -keyname CSaek -passphrase Password@123 -lockbox lockbox.lb -lockboxpassphrase Password@123 Create Server Trust store When acting as SSL client to Docbroker, Server needs to verify public certificate sent by Docbroker. For verifying Docbroker's public certificate, Docbroker s public certificate or CA certificate which was used to sign Docbroker's public certificate needs to be stored in a Keystore. This Keystore will act as Server's trust-store and server will verify Certificates sent by Docbroker against certificates in this keystore. Server expects trust-store to be in PKCS #7 binary format. There is no restriction on trust-store's name or extension. It should be placed in location $DOCUMENTUM/dba/secure. Below command generates trust-store by importing Docbroker's certificate. openssl crl2pkcs7 -nocrl -certfile brokercrt.pem -outform der -out server-trust.p7b DFC Create DFC Trust-Store When DFC will be acting SSL client to Docbroker or Server, it will use a trust-store to store trusted certificates. Certificates sent by Docbroker or Server will be validated against the certificates in DFC trust-store. DFC trust store should be in JKS (Java Keystore) format. There is no restriction on store's name or extension as well as location. We will convert the Docbroker & Server public certificates that were generated in text format to binary format using OpenSSL. Then public certificates of Docbroker and Server in binary format will be imported to DFC trust-store. Below commands converts certificates from PEM to DER format: openssl x509 -outform der -in brokercrt.pem -out brokercrt.der openssl x509 -outform der -in servercrt.pem -out servercrt.der Following commands create DFC trust-store (dfc.keystore) by importing Docbroker's & Server's public certificates: keytool -importcert -keystore dfc.keystore -file brokercrt.der -alias broker keytool -importcert -keystore dfc.keystore -file servercrt.der -alias server CONTENT SERVER INSTALLATION First, install Content Server using serversetup.bin. In the last step installer provides option to launch Content Server configuration program and continue with Docbase configuration or configure this later using server configuration program. Configure now is express mode in which configuration program does not provide options to configure Docbroker and select connection modes. In this case docbroker is created in native mode. If we try to configure docbase in certificate based ssl then we will get docbase connection failure error. So need to install the docbase also in native mode. After installation we need to follow the manual steps to convert docbroker and docbase into certificate based ssl mode. 6
If we select Configure later and exiting from the installer then installer will support automatic configuration of Docbroker and docbase into certificate based SSL. Docbroker Configuration with Installer 1. First create the aek key manually with following command which is mentioned in CS admin guide. dm_crypto_create [-location <location>][-lockbox <lockbox>] [-lockboxpassphrase <lockboxpassphrase>] [-keyname <keyname>] [-location <location>] [-passphrase <passphrase>] [-noprompt] [-move] [-check] [-algorithm] [-help] 2. Copy Docbroker Keystore file and keystore password file and placed into $DOCUMENTUM/dba/secure folder. 3. Run Server Configuration program using $DM_HOME/install/Server_Configuration_Program.exe script to install Docbroker. 4. Select Secure or Native and Secure radio button in Select Connection Mode for Connection Broker page 7
5. Next page will come with option Use certificates check box 6. If select Use certificates check box it will display keystore filename, keystore password file name, cipher list etc entries in the same page. Need to change the proper keystore filename, keystore password file name and cipher list. Here installer supported default cipher is AES128-SHA. We need to enter the dfc keystore password in the password text box. Here Use Default Trust store check box is optional. 8
7. Proceed the further steps as normal installation. After installation cross check the entries in Docbroker configuration file (<Docbroker>.ini) under [DOCBROKER_CONFIGURATION] section at $DOCUMENTUM/dba directory keystore_file=broker.p12 keystore_pwd_file=broker.pwd cipherlist=aes128-sha crypto_keyname = CSaek crypto_lockbox=lockbox.lb And also new entries will be added in dfc.properties file at $DOCUMENTUM/config dfc.security.ssl.truststore=c\:\\documentum\\dba\\secure\\dfc.keystore dfc.security.ssl.truststore_password=aaaaeec7ixvob/bqfje57keqp3pqukuv0hqjrc1frrzvth29 Manual Steps for Docbroker Configuration Manual steps are useful when the docbroker is already installed without enable the certificate based ssl. 1. Install docbroker and stop the docbroker. 2. Add the bellow entries in Docbroker configuration file((<docbroker>.ini) at $DOCUMENTUM/dba directory keystore_file=broker.p12 keystore_pwd_file=broker.pwd cipherlist=aes128-sha crypto_keyname = CSaek crypto_lockbox=lockbox.lb 3. Add the bellow entries in in dfc.properties file under [DOCBROKER_CONFIGURATION] section at $DOCUMENTUM/config dfc.security.ssl.truststore=c\:\\documentum\\dba\\secure\\dfc.keystore dfc.security.ssl.truststore_password=<dfc trust store password> 4. Start the docbroker manually. Server configuration with installer 1. Copy Server Keystore file, server trust store file, keystore password file and DFC trust store file and placed in $DOCUMENTUM/dba/secure folder. 2. Run Server Configuration program using $DM_HOME/install/Server_Configuration_Program.exe script. 3. Enter the Connection broker details and select the check box Use Certificates and enter the valid DFC trust store password password in Docbroker information page. 9
10 4. Select Secure or Native and Secure radio button in Select Connection Mode page
5. Select Use certificates check box it will display keystore filename, keystore password file name, Trust store file name, cipher list etc entries in the same page. Need to change the proper keystore filename, keystore password file name, Trust store file name and cipher list. Here installer supported default cipher is AES128-SHA. We need to enter the valid dfc keystore password in the password text box. Here Use Default Trust store check box is optional. 6. Proceed the further steps as normal installation. After installation cross check the entries in Server configuration file (<server>.ini) under [SERVER_STARTUP] section at $DOCUMENTUM/dba/config/<docbasename> directory keystore_file=server.p12 keystore_pwd_file=server.pwd truststore_file=server-trust.p7b cipherlist=aes128-sha And also new entry will be added in dfc.properties file at $DOCUMENTUM/config dfc.session.secure_connect_default=try_secure_first 11
Manual Steps for Server configuration Manual steps are useful when the docbase is already installed without enable the certificate based ssl 1. Install Docbase and stop the docbase service. 2. Add the bellow entries in server configuration file((<server>.ini) under [SERVER_STARTUP] section at $DOCUMENTUM/dba/config/<docbasename> directory keystore_file=server.p12 keystore_pwd_file=server.pwd truststore_file=server-trust.p7b cipherlist=aes128-sha And also add the bellow entry in dfc.properties file at $DOCUMENTUM/config dfc.session.secure_connect_default=try_secure_first 3. Start the docbase manually UPGRADE: This section explaining the content server, docbroker and docbase upgrade steps with certificate based ssl. Irrespective of anonymous or non-anonymous ssl we need to follow the bellow steps to configure certificate based ssl in upgrade. 1. Stop docbase and docbroker. 2. Upgrade the content server. 3. Upgrade the docbroker. In upgrade docbroker, installer is not supporting the certificate based ssl settings. So we need to stop the docbroker and do the manual steps which are mentioned above. 4. Docbase installer will support certificate based SSL settings. Enter the Docbroker port and host which is already configured in certificate based ssl and select the Use certificate option in docbroker information page while upgrading the docbase. Enter proper dfc trust store location and dfc trust store password. 5. Select Secure or Native and Secure radio button in Select Connection Mode page 12
6. Select Use certificates check box it will display keystore filename, keystore password file name, Trust store file name, cipher list etc entries in the same page. Need to change the proper keystore filename, keystore password file name, Trust store file name and cipher list. Here installer supported default cipher is AES128-SHA. We need to enter the valid dfc keystore password in the password text box. Here Use Default Trust store check box is optional. 13
7. Proceed the further steps as normal upgrade. After installation cross check the entries in Server configuration file (<server>.ini) at $DOCUMENTUM/dba/config/<docbasename> directory keystore_file=server.p12 keystore_pwd_file=server.pwd truststore_file=server-trust.p7b cipherlist=aes128-sha And also new entry will be added in dfc.properties file at $DOCUMENTUM/config dfc.session.secure_connect_default=try_secure_first CLIENTS CONFIGURATION: DFC configuration DFC will look in dfc.properties file to resolve DFC trust-store location, name and password. Property dfc.security.ssl.truststore is used to specify trust-store's path and dfc.security.ssl.truststore_password to specify trust-store's password. Trust-store's password can be given in plain text or encrypted format. DFC installer will support automatic selection of password and trust store location. Steps to install DFC client with certificate based SSL by installer: 1. Select the Use certificates check box and enter the primary connection broker details. 14
2. Browse the trust store location and enter the password. Here Use Default Java Truststore check box is optional. 3. For secure connections to Server and Docbroker, put the value of property dfc.session.secure_connect_default as secure. Same property controls behavior of connections to both Docbroker and Server. Put below property in dfc.properties file: dfc.session.secure_connect_default = secure Manual steps to configure DFC client with certificate based SSL: 1. In dfc.properties, put entries for DFC trust-store and trust-store's password: dfc.security.ssl.truststore = c\:/secure/dfc.keystore dfc.security.ssl.truststore_password = password DFC trust-store password can be encrypted using DFC utility as below: java com.documentum.fc.tools.registrypasswordutils <password> E.g: java com.documentum.fc.tools.registrypasswordutils password The encrypted password generated by this command can be copied and pasted to dfc.properties file. For this command to execute successfully, dfc.jar should be in java classpath. 15
There is an additional property dfc.security.ssl.use_existing_truststore that is mutually exclusive to above two properties. When this property is specified, there is no need to put above two properties. In this case, Java Keystore will act as DFC trust-store or trust-store can be specified using JVM parameter javax.net.ssl.truststore. 2. For secure connections to Server and Docbroker, put the value of property dfc.session.secure_connect_default as secure. Same property controls behavior of connections to both Docbroker and Server. Put below property in dfc.properties file: dfc.session.secure_connect_default = secure Documentum Administrator configuration Documentum Administrator (DA) is a WDK web based client. When DA wants to connect certificate based ssl docbase DA contained DFC will be acting SSL client to Docbroker or Server, it will use a trust-store to store trusted certificates. Certificates sent by Docbroker or Server will be validated against the certificates in DFC trust-store. DA we will look in dfc.properties file to resolve DFC trust-store location, name and password. Property dfc.security.ssl.truststore is used to specify trust-store's path and dfc.security.ssl.truststore_password to specify trust-store's password. Trust-store's password can be given in plain text or encrypted format. We need to add the bellow entries in dfc.properties at <da-home>/web-inf/classes folder. dfc.session.secure_connect_default = secure dfc.security.ssl.truststore = c\:/secure/dfc.keystore dfc.security.ssl.truststore_password = password Steps to configuration of DA: 1. Copy dfc.keystore file into C:\Secure folder. There is no restriction on folder name or path. 2. Stop application server. 3. Extract the the da.war file under webapps folder in application server. 4. Add the global repository details along with bellow entires in dfc.properties at <da-home>/web-inf/classes folder. dfc.session.secure_connect_default = secure dfc.security.ssl.truststore = c\:/secure/dfc.keystore dfc.security.ssl.truststore_password = password 5. Start Application server. TROUBLESHOOTING: This section describes some of the common issues faced during configuration and steps to troubleshoot Docbroker startup fails 1. Check if Docbroker Keystore and Keystore's password files are present in $DOCUMENTUM/dba/secure 2. Check if entry for above two files is there in Docbroker configuration file (<Docbroker>.ini) 3. Check if the format of Docbroker Keystore is PKCS #12 Below commands should list the keys in Keystore if it s in PKCS #12 format, else error will be displayed 16
Using OpenSSL openssl pkcs12 -info -in <keystore> Using Keytool: keytool -list v -storetype pkcs12 -keystore <keystore>. 4. Check if password in Keystore password file is correct. For testing, password can be put in plain-text (without encryption). Docbase startup fails 1. Check if Server Keystore, Server Keystore password and Server trust-store files are present in $DOCUMENTUM/dba/secure 2. Check if entry for above three files is there in Server configuration file (server.ini) 3. Check if the format of Server Keystore is PKCS #12. See commands in Docbroker startup fails section to verify 4. Check if server trust-store is in PKCS #7 binary (der) format For verifying, check if below command dumps it successfully openssl pkcs7 -in <Keystore> -inform der E.g. openssl pkcs7 -in server-trust.p7b -inform der Server not able to connect to Docbroker 1. Check whether Docbroker Keystore has proper key and public certificate Use below command to print the keys in keystore: Using OpenSSL openssl pkcs12 -info -in <keystore> Using Keytool keytool -list -storetype pkcs12 -keystore <keystore> 2. Check whether Docbroker is sending proper certificates Below command will start a simple client which tries to connect to SSL Server and displays certificate chain sent by Server: openssl s_client -showcerts -debug connect <SSL_Server_IP>:<SSL_Server_Port> E.g. openssl s_client -showcerts -debug -connect 10.8.53.24:1490 3. Check if Server's trust-store contains Docbroker's public certificate or CA certificate chain used to sign Docbroker's public certificate Use below command to display all certificates in trust store: openssl pkcs7 -in <trust-store> -inform der -print_certs text Clients not able to connect to Docbroker 1. Check if proper entries are present in dfc.properties and trust-store file exists. 2. Check if DFC trust-store contains Docbroker's public certificate or CA certificate used to sign Docbroker s public certificate To dump Trust-store's contents (no need to specify storetype as default type is JKS): keytool -list -keystore <keystore> -storepass <storepass>. 3. Check the firewall settings are proper in client and docbroker machines. 17
Clients not able to connect to Server 1. Check if proper entries are made in dfc.properties and trust-store file exists 2. Check if DFC trust-store contains Server's public certificate or CA certificate used to sign Server s public certificate To dump Trust-store's contents (no need to specify storetype as default type is JKS): keytool -list -keystore <keystore> -storepass <storepass> 3. Check if dfc.properties has right value for trust-store password. For verification, password can be given in plain-text 4. Check the firewall settings are proper in client and server machines. CONCLUSION This white paper provides information required for understanding and quickly getting started with the process of installing Documentum Content Server 7.2 with certificate based SSL. For more details refer product documentation and references provided in next section. REFERENCES Documentum Content Server Installation Guide Version 7.2 Documentum Content Server Distributed Content Configuration Guide Version 7.2 Documentum Content Server Administration and Configuration Guide Version 7.2 18 www.emc.com