Data Center Real User Monitoring

Transcription

1 Data Center Real User Monitoring SSL Monitoring Administration Guide Release 12.1

2 Please direct questions about Data Center Real User Monitoring or comments on this document to: APM Customer Support FrontLine Support Login Page: Copyright 2013 Compuware Corporation. All rights reserved. Unpublished rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS-Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Disclosure is prohibited without the prior express written permission of Compuware Corporation. Use of this product is subject to the terms and conditions of the user's License Agreement with Compuware Corporation. Documentation may only be reproduced by Licensee for internal use. The content of this document may not be altered, modified or changed without the express written consent of Compuware Corporation. Compuware Corporation may change the content specified herein at any time, with or without notice. All current Compuware Corporation product documentation can be found at Compuware, FrontLine, Network Monitoring, Synthetic Monitoring, Server Monitoring, Transaction Trace Analysis, Compuware APM, VantageView, Compuware APM, Real-User Monitoring First Mile, and Gomez Performance Network are trademarks or registered trademarks of Compuware Corporation. Cisco is a trademark or registered trademark of Cisco Systems, Inc. Internet Explorer, Outlook, SQL Server, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation. Firefox is a trademark or registered trademark of Mozilla Foundation. Red Hat and Red Hat Enterprise Linux are trademarks or registered trademarks of Red Hat, Inc. J2EE, Java, and JRE are trademarks or registered trademarks of Oracle Corporation. VMware is a trademark or registered trademark of VMware, Inc. SAP and SAP R/3 are trademarks or registered trademarks of SAP AG. Adobe Reader is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. All other company and product names are trademarks or registered trademarks of their respective owners. Build: August 5, 2013, 10:49

3 Contents Contents Introduction Who Should Read This Guide Related Publications Organization of This Guide Customer Support Information Reporting a Problem Documentation Conventions Chapter 1 Process Overview of SSL Monitoring Chapter 2 Configuring SSL Monitoring on the AMD Configuring and Using RSA Private Keys Management of RSA Private Keys on AMD Using a List File to Specify RSA Private Keys SSL Hardware Accelerator Cards Selecting and Configuring SSL Engine Installing and Configuring NITROX XL FIPS Acceleration Board Supported NITROX XL FIPS Acceleration Board Security Levels Invoking Acceleration Board Management Utility Initializing the NITROX XL FIPS Acceleration Board Logging In and Out of the NITROX XL FIPS Acceleration Board RSA Key Management on NITROX XL FIPS RoHS Directive Compliance Installing and Configuring an ncipher SSL Card on a 32-bit AMD Installing and Configuring an ncipher SSL Card on a 64-bit AMD Installing and Configuring Sun Crypto Accelerator 6000 PCIe Card Initializing the Sun Crypto Accelerator 6000 PCIe Card Sun Crypto Accelerator 6000 PCIe Card - Key and Card Management Additional Configuration Settings and Administration for Sun Crypto Accelerator 6000 PCIe Card Reference Information for Sun Crypto Accelerator 6000 PCIe Card Sun Crypto Accelerator 6000 PCIe Card Known Issues Using KPA to Make Keys Available to the AMD Process

4 Contents Migrating from OpenSSL to Using SSL Hardware Accelerator Monitoring SSL-encoded Traffic without Decryption Chapter 3 Server-Based SSL Monitoring Configuration Defining Names of SSL Errors in Report Server Control Panel Managing SSL Alert Codes Customizing Definition of Availability Metrics Chapter 4 Tuning Configuration and Troubleshooting Problems Verification of Traffic Monitoring Quality SSL Diagnostics Troubleshooting SSL Monitoring Issues Guided Configuration Issues Appendix A SSL-Related rcon Commands SSLDECR CERTS SSLDECR HELP SSLDECR LOGLEVEL SSLDECR NAMES SHOW SSLDECR CERTS SHOW SSLDECR CIPHERS SHOW SSLDECR HELP SHOW SSLDECR KEYS SHOW SSLDECR LOGLEVEL SHOW SSLDECR NAMES SHOW SSLDECR SERVERS SHOW SSLDECR STATUS Appendix B Extracting Web Server Private SSL Keys Extracting Web Server Private RSA Keys for Apache/OpenSSL Server Extracting Web Server Private RSA Keys for Microsoft IIS 4.0 Server Extracting Web Server Private RSA Keys for Microsoft IIS 5.0 Server Extracting Web Server Private RSA Keys for Netscape (Old Format) Extracting Web Server Private RSA Keys for Netscape (New Format) Extracting Web Server Private RSA Keys for Zeus Extracting SSL Private Keys from an iplanet Web Server Appendix C SSL Support SSL Software Support SSL Hardware Support Index

5 INTRODUCTION Who Should Read This Guide This book is intended for users of Data Center Real User Monitoring who want to configure, diagnose, and troubleshoot monitoring of SSL traffic. Related Publications Documentation for your product is distributed on the product media. For DCRUM, it is located in the \Documentation directory. It can also be accessed from the Media Browser. You can also access online documentation for Compuware products via our FrontLine support site at FrontLine provides fast access to information about your Compuware products. You can download documentation and FAQs as well as browse, ask questions, and get answers on user forums (requires subscription). The first time you access FrontLine, you are required to register and obtain a password. Registration is free. PDF files can be viewed with Adobe Reader version 7 or later. If you do not have the Reader application installed, you can download the setup file from the Adobe Web site at Organization of This Guide This guide is organized as follows: Process Overview of SSL Monitoring [p. 11]contains overview of issues and considerations on monitoring of secure traffic based on SSL (Secure Socket Layer). Configuring SSL Monitoring on the AMD [p. 13] contains information on preparing private RSA keys, installing and configuring hardware accelerator cards, using OpenSSL, and migrating from OpenSSL to hardware SSL acceleration. It also includes information on monitoring SSL traffic without decryption. Server-Based SSL Monitoring Configuration [p. 49] explains how to change configuration of SSL monitoring related properties that affect DCRUM reports. 5

6 Introduction Tuning Configuration and Troubleshooting Problems [p. 53] addresses various configuration issues often encountered in SSL monitoring. SSL-Related rcon Commands [p. 65] is a collection of rcon commands related to SSL monitoring. Extracting Web Server Private SSL Keys [p. 77] explains in detail how to extract private SSL keys from different web servers. SSL Support [p. 85] gives reference information about hardware and software SSL support in DCRUM. Customer Support Information FrontLine Support Website You can access information for Compuware products via our FrontLine support site. You can review frequently asked questions, read or download documentation, access product fixes, or your questions or comments. The first time you access FrontLine, you are required to register and obtain a password. Registration is free. To access FrontLine, log in to Select your product from the Product Support dropdown list. Contacting Customer Support Phone USA and Canada: or All other countries: Contact your local Compuware office. Contact information is available at Web You can report issues via the Report and Track Calls tab on the FrontLine home page. NOTE Please report all high-priority issues by phone. APM Community You can find product documentation, forums with product experts, product fixes and more information at the Compuware APM Community. You must register and login to access the Community. apmsupport@compuware.com Mail Customer Support Compuware Corporation One Campus Martius Detroit, MI

7 Introduction Corporate Website To access the Compuware website, go to The Compuware site provides a variety of product and support information. Reporting a Problem When contacting APM Customer Support, please provide as much information as possible about your environment and the circumstances that led to the difficulty. You should be ready to provide: Client number. This number is assigned to you by Compuware and is recorded on your sales contract. The version number of the AMD, report servers, and RUM Console with RUM Console Server. Report Server Use the report server GUI by selecting Help Product Information About, or Tools Diagnostics System Status. AMD In RUM Console, navigate to Devices and Connections Manage Devices, select an AMD from the devices list and read the version. RUM Console and RUM Console Server Use the RUM Console GUI by selecting Help About menu item. TCAM Use the TCAM GUI by selecting Help About menu item. Environment information, such as the operating system and release (including service pack level) on which the product (AMD, report server) is installed, memory, hardware/network specifications, and the names and releases of other applications that were running. Problem description, including screen captures. Exact error messages, if any (screen captures recommended). Whether or not the problem is reproducible. If it is, include a sequence of steps for problem recreation. If it is not, include a description of the actions taken before the problem occurred. A description of the actions that may have been taken to recover from the problem, and their results. Debug information for specific components obtained from RUM Console. Information about the RUM Console itself. To export all the information, navigate to Help Export Console Diagnostics in the RUM Console menu. Information about the report servers. To export the information for a specific ADS or CAS, navigate to Devices and Connections Manage Devices, and choose Export diagnostic information from the context menu of the particular device. 7

8 Introduction Include data collector diag. Option to include diagnostic information on data collectors attached to the report server. Installation logs Option to include installation information logged and saved on the server. SQL trace logs Option to include trace logs of SQL queries. Save as Destination path and filename for the diagnostic package file. Information about the AMD. To export the information for a specific AMD, navigate to Devices and Connections Manage Devices, and choose Export diagnostic information from the context menu of the particular device. Include data files Option to include fragments of traffic data. Begin and End Time range of the monitoring data to be included with the diagnostics. Data file filter (RegEx) Regular expression filter for monitoring data files generated during the defined time range. Save as Destination path and filename for the diagnostic package file. Information from the TCAM System Event log of the machine where the TCAM is operating. TCAM logs which by default are stored in C:\ProgramData\Compuware\VTCAM for Windows Server 2008 and C:\Documents and Settings\All Users\Compuware\VTCAM for Windows Server NOTE Please compress all the files before sending them to Customer Support. Compuware values your comments and suggestions about the Compuware APM products and documentation. Your feedback is very important to us. If you have questions or suggestions for improvement, please let us know. Documentation Conventions The following font conventions are used throughout documentation: This font Bold Indicates Terms, commands, and references to names of screen controls and user interface elements. 8

9 Introduction This font Citation Documentation Conventions [p. 8] Fixed width Fixed width bold Fixed width italic Menu Item Screen Code block Indicates Emphasized text, inline citations, titles of external books or articles. Links to Internet resources and linked references to titles in Compuware documentation. Cited contents of text files, inline examples of code, command line inputs or system outputs. Also file and path names. User input in console commands. Place holders for values of strings, for example as in the command: cd directory_name Menu items. Text screen shots. Blocks of code or fragments of text files. 9

10 Introduction 10

11 CHAPTER 1 Process Overview of SSL Monitoring Monitoring of secure traffic requires more attention and preparation than monitoring of non-secure protocols. In addition, if the AMD is to decrypt SSL traffic, it needs third-party components, such as hardware or software SSL accelerators, to be preconfigured to seamlessly work with Data Center Real User Monitoring. Before You Begin Before you start configuration process: You should be familiar with DCRUM components and basic monitoring concepts. Refer to the Data Center Real User Monitoring Getting Started. You need to identify your monitoring goals. For more information, see Define and Prioritize Goals, Objectives, and Requirements in the Data Center Real User Monitoring Getting Started. You need to install the following DCRUM components: The latest version of AMD Refer to the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide. The latest version of RUM Console Refer to the Data Center Real User Monitoring RUM Console Installation Guide. The latest version of CAS Refer to the Data Center Real User Monitoring Central Analysis Server Installation Guide. Optionally: The latest version of ADS Refer to the Data Center Real User Monitoring Advanced Diagnostics Server Installation Guide. Make sure you have prepared your RSA keys and documentation on your SSL accelerator. 11

12 Chapter 1 Process Overview of SSL Monitoring The process of configuration of SSL monitoring with decryption consists of the following tasks. Note that you may want to monitor SSL traffic without decryption. For more information, see Monitoring SSL-encoded Traffic without Decryption [p. 48]. SSL-related configuration 1. Preparing RSA private keys for servers that are to be monitored The private keys in PEM format then have to be applied to the AMD in order to decrypt secure sessions.for more information, see Configuring and Using RSA Private Keys [p. 13] and Extracting Web Server Private SSL Keys [p. 77]. 2. Selecting the mode of RSA key management on the AMD For more information, see Management of RSA Private Keys on AMD [p. 14]. 3. Installing and configuring a hardware SSL accelerator, if a hardware accelerator is to be used In most deployments, hardware SSL accelerators are used because of performance reasons. However, there is an option to use a software alternative, OpenSSL. Depending on your SSL acceleration approach, refer to the topic appropriate for your hardware accelerator or use OpenSSL, the default cost-free SSL acceleration mode on the AMD. 4. Optional: Migrating from OpenSSL to an SSL hardware accelerator While OpenSSL is a cost-free solution to SSL decryption, it may not be sufficient in terms of performance. When your secure traffic stream overwhelms the AMD's software capabilities, you should consider deploying hardware SSL accelerators. For more information, see Migrating from OpenSSL to Using SSL Hardware Accelerator [p. 47]. Monitoring configuration 5. Setting up software service monitoring Monitoring of SSL traffic requires that you select an appropriate analyzer while defining a software service. For example, if you want to monitor an HTTPS (secure HTTP) software service, and you comply with the aforementioned steps of the configuration process, you ought to select the SSL Decrypted analyzer for such a service. Apart from selecting the analyzer for your software service, you may also want to configure more sophisticated features of HTTP analysis, such as user recognition, URL parameter parsing, and so on. Refer to the Data Center Real User Monitoring Web Application Monitoring User Guide. HTTPS, while the most dominant protocol, when considering SSL monitoring, is not the only protocol that can be encrypted with SSL. For more information, see Protocols Supported by CAS in the Data Center Real User Monitoring Administration Guide and Protocols Supported by ADS in the Data Center Real User Monitoring Administration Guide. What to Do Next In case of issues observed during monitoring of SSL traffic, you should consult the SSL-related FAQ to diagnose your problems before you contact Customer Support. For more information, see Troubleshooting SSL Monitoring Issues [p. 55] and SSL-Related rcon Commands [p. 65]. 12

13 CHAPTER 2 Configuring SSL Monitoring on the AMD Configuration of SSL monitoring with decryption requires you to extract and apply the RSA private keys and to install and configure the SSL hardware accelerator cards. Configuring and Using RSA Private Keys To process SSL decryption, an AMD needs to use RSA private keys for each monitored server. The keys need to be extracted from the monitored servers and can then be used either as PEM files or be stored on the accelerator card. Key extraction is described in Extracting Web Server Private SSL Keys [p. 77]. NOTE In the case of keys generated with OpenSSL, the keys are already in PEM format. If keys come from a Microsoft IIS or Netscape Web server, they are usually stored in hardware accelerators and must be exported to PEM format. A key can be encrypted with a password. For more information, see Using KPA to Make Keys Available to the AMD Process [p. 46]. SSL decryption can be performed either in the AMD software using OpenSSL or in a hardware SSL accelerator. If SSL decryption is performed in the AMD software, the AMD reads RSA private keys from PEM-encoded disk files during startup. If SSL decryption is performed in a hardware SSL accelerator, the keys may need to be stored in the accelerator card first: after extracting the keys from their servers as PEM-encoded disk files and writing them to the accelerator, the PEM files should be deleted for security reasons. The commands used for managing listing, organizing, and storing keys on an accelerator card are specific to the card and are described in topics dedicated to individual cards: Installing and Configuring NITROX XL FIPS Acceleration Board [p. 22], Installing and Configuring an ncipher SSL Card on a 32-bit AMD [p. 28] Installing and Configuring an ncipher SSL Card on a 64-bit AMD [p. 33] 13

14 Chapter 2 Configuring SSL Monitoring on the AMD Installing and Configuring Sun Crypto Accelerator 6000 PCIe Card [p. 38] Management of RSA Private Keys on AMD The AMD supports two mutually exclusive modes of using RSA private keys. A list of the private keys that are to be used for encryption can be contained in a text file on the AMD, with each entry containing a reference to a PEM-encoded file or a key stored on the accelerator card. The AMD can extract all keys from the accelerator card and use those for a pool of available keys. These two mutually exclusive modes of operation are governed by the following configuration properties in the rtm.config configuration file: server.key.dir The directory in which to store PEM-encoded key files (by default, this is /usr/adlex/config/keys). server.key.list The file in the above directory that describes what keys are to be used for the monitored servers. The default name of the file is keylist. Note that the file lists keys to be used, but does not provide a mapping of servers to keys. This is because the AMD is able to match keys to SSL sessions automatically. The advantage of this approach (of not mapping a specific IP address of the server to the private key) is that servers residing behind load balancers can also be monitored, even though the same IP address is then apparently using a number of different SSL private keys. ssl.import.all.keys.from.token Mode selector: Setting this configuration property to true overrides the settings specified in server.key.list and makes the AMD read the keys from the accelerator card. This is supported only for ssl.engine settings of nitroxfips, sca6000, or ncipher_pkcs11. For more information on setting ssl.engine, see Selecting and Configuring SSL Engine [p. 20]. Setting this property to false enables key resolution based on the information provided by the server.key.dir and server.key.list configuration properties. The file listing the keys, as specified in server.key.list, is a plain-text file with each line describing a single key and being composed of the following fields. Note that the square brackets ( [ ] ) imply that the given item is optional, and the brackets themselves should not be included in the actual entry. Note also that this file may also be used by other protocols, so entries of other types may also appear there. key_type, [app_name:]key_identifier[, comment] where: key_type specifies whether the private key is contained in a PEM-encoded file or in a hardware accelerator token: 14

15 Chapter 2 Configuring SSL Monitoring on the AMD file token key_type value file means that the private key is stored in a PEM-encoded file (possibly encrypted). key_type value token means that the private key is stored in a hardware accelerator. app_name is the application name within the ncipher context. The value of this parameter depends on, among other things, the method used for writing the key to the card. For example, if the following method is used:./generatekey --import simple pemreadfile=/usr/adlex/config/keys/s1.key protect=module ident=s1 the application name will be simple and the syntax of the entries in the list will be: token, simple:key_identifier[, comment] To determine the value you need to enter for each key on the card, use the rocs command provided with your ncipher card. For example: # cd /opt/nfast/bin #./rocs `rocs' key recovery tool Useful commands: `help', `help intro', `quit'. rocs> list keys No. Name App Protected by 1 k1 simple module rocs> exit For other accelerator cards, leave this field empty and do not include the colon in the syntax. key_identifier identifies the key: For keys stored in files, it is the name of the PEM-encoded file that contains an RSA private key. For keys stored on the accelerator card, it is the key identifier as given by the utilities that list keys. Note that some engines distinguish between key identifiers and key labels. Both of these identification methods can be used in the keylist file. However, you may need to specify the type of identification used, by setting the searchkeyby parameter of the ssl.engine.param property to id or label, as appropriate. See Selecting and Configuring SSL Engine [p. 20] for more information on configuring this option. For ncipher SSL cards, the identifier is an 8-digit hexadecimal value. For a NITROX XL FIPS Acceleration Board, the length of the identifier can vary. The comment part in square brackets [ ] is an optional comment describing the entry in the line. 15

16 Chapter 2 Configuring SSL Monitoring on the AMD Table 1. RSA key handling methods The following table lists the possible RSA key handling methods for the supported SSL engines. SSL engine entry of type file in keylist entry of type token in keylist can import all keys from token openssl YES nfast YES nshield YES YES ncipher_pkcs11 YES YES nitrox YES YES sca6000 YES YES Example 1. Sample entries with RSA private keys token,0a0412dc,key for stored in hardware file,server1.pem,key for on port 443 file,server2.pem,key for on port 444 file,server2.pem,key for on port 445 If the AMD is connected to a Central Analysis Server installation, then, for SSL decryption to be used for selected servers, you need to add service definitions for these servers using the report server graphical user interface, Monitoring Configuration. Here you should add an application (named, for example, SSL decoded ) and specify that the SSL (with decryption) analyzer is to be used for that application. Using a List File to Specify RSA Private Keys A list of the private keys that are to be used for encryption can be contained in a text file on the AMD, with each entry containing a reference to a PEM-encoded file or a key stored on the accelerator card. Before You Begin For the purpose of this procedure, it is assumed that you are using OpenSSL and have the required PEM-encoded keys ready. Key extraction is described in Extracting Web Server Private SSL Keys [p. 77]. To use a list file to specify RSA private keys: 1. Ensure that the AMD is configured to use keys listed in the list file. Edit the rtm.config configuration file and make sure that the ssl.import.all.keys.from.token configuration property is set to false: ssl.import.all.keys.from.token=false 2. Optional: Specify the directory in which to store the list file and the PEM-encoded key files. 16

17 Chapter 2 Configuring SSL Monitoring on the AMD This directory is by default /usr/adlex/config/keys. You do not need to modify this setting unless you want to store the files in a different location. To change the configuration, edit the rtm.config configuration file and modify the server.key.dir configuration property. The following example line shows the default setting: server.key.dir=/usr/adlex/config/keys 3. Optional: Specify the name of the list file. The default name of the file listing the keys is keylist. You do not need to modify this setting unless you want to use a different file name. To change the configuration, edit the rtm.config configuration file and modify the server.key.list configuration property. The following example line shows the default setting: server.key.list=keylist Note that the file lists keys to be used, but does not provide a mapping of servers to keys. This is because the AMD is able to match keys to SSL sessions automatically. The advantage of this approach of not mapping a specific IP address of the server to the private key is that servers residing behind load balancers can also be monitored, even though the same IP address is then apparently using a number of different SSL private keys. 4. Optional: Copy all key PEM-encoded key files to the correct directory. All the PEM-encoded key files if any are to be used should be copied to the directory specified in the server.key.dir configuration property. Example 2. Copying RSA key files Copying an individual file: # cp key1.pem /usr/adlex/config/keys/ or all the *.pem files in the current working directory: # cp *.pem /usr/adlex/config/keys/ 5. Optional: Write keys to the accelerator card. If an accelerator card is to be used, you may need to write the keys to the card before they can be used for encryption. Keys written to the card are referred to as tokens. Using tokens is more secure and therefore is recommended if the accelerator cards supports this option. For more information, see Management of RSA Private Keys on AMD [p. 14]. The commands used for managing listing, organizing, and storing keys on an accelerator card are specific to the card and are described in topics dedicated to individual cards: Installing and Configuring NITROX XL FIPS Acceleration Board [p. 22], Installing and Configuring an ncipher SSL Card on a 32-bit AMD [p. 28] Installing and Configuring an ncipher SSL Card on a 64-bit AMD [p. 33] Installing and Configuring Sun Crypto Accelerator 6000 PCIe Card [p. 38] 6. Optional: For ncipher cards on a 32-bit platform only, determine the values of the key application names. 17

18 Chapter 2 Configuring SSL Monitoring on the AMD These parameters are used only for ncipher keys on 32-bit platforms. To determine the value of the ncipher application name, use the rocs command provided with your ncipher card. For example: # cd /opt/nfast/bin #./rocs `rocs' key recovery tool Useful commands: `help', `help intro', `quit'. rocs> list keys No. Name App Protected by 1 k1 simple module rocs> exit In the above example, the name of the application is simple. 7. Optional: Specify the type of identification to be used as id or label. For engine values of ncipher_pkcs11 and sca6000, the searchkeyby parameter of the ssl.engine.param property can be set to id or label with the following default values for the respective engines: ncipher_pkcs11 Default key identification is by label. sca6000 Default key identification is by key identifier. Example 3. Specify the type of identification to be used ssl.engine.param=searchkeyby:id 8. Determine the values of the key identifiers for keys stored on the accelerator card. For keys stored in files, it is the name of the PEM-encoded file that contains an RSA private key. For keys stored on the accelerator card, it is the key identifier as given by the utilities that list keys. For the appropriate engines, distinguish between key identifiers and key labels as specified in Step 7 [p. 18]. For CryptoSwift and ncipher SSL cards, the identifier is an 8-digit hexadecimal value. For a NITROX XL FIPS Acceleration Board, the length of the identifier can vary. The commands used for managing listing, organizing and storing keys on an accelerator card, are specific to the card and are described in topics dedicated to individual cards: Installing and Configuring NITROX XL FIPS Acceleration Board [p. 22], Installing and Configuring an ncipher SSL Card on a 32-bit AMD [p. 28] Installing and Configuring an ncipher SSL Card on a 64-bit AMD [p. 33] Installing and Configuring Sun Crypto Accelerator 6000 PCIe Card [p. 38] 9. Create the list file. Use a text editor to create and edit the list file as a plain text file. The file should be located in the directory specified in the server.key.dir configuration property and named as specified in the server.key.list configuration property. 18

19 Chapter 2 Configuring SSL Monitoring on the AMD Each line should describe a single key and be composed of the following fields. Note that the square brackets ( [ ] ) imply that the given item is optional, and the brackets themselves should not be included in the actual entry. key_type, [app_name:]key_identifier[, comment] where: key_type specifies whether the private key is contained in a PEM-encoded file or in a hardware accelerator token: file token key_type value file means that the private key is stored in a PEM-encoded file (possibly encrypted). key_type value token means that the private key is stored in a hardware accelerator. app_name is the application name within the ncipher context. NOTE Specify this field only for ncipher cards, as explained in Step 6 [p. 17], and only in the case of files stored on the accelerator card. For other accelerator cards, or for files stored in PEM-encoded files, leave this field empty and do not include the colon in the syntax. key_identifier identifies the key: For keys stored in files, it is the name of the PEM-encoded file that contains an RSA private key. For keys stored on the accelerator card, it is the key identifier as given by the utilities that list keys. The comment part is optional. Example 4. Sample entries listing RSA private keys token,0a0412dc,key for stored in hardware file,server1.pem,key for on port 443 file,server2.pem,key for on port 444 file,server2.pem,key for on port Optional: Delete PEM files after keys have been loaded into the accelerator. After the keys have been loaded into the accelerator, it is advised, for security reasons, that the PEM files be deleted. You can securely delete the source files, by means of the shred command. This is a Linux command that allows secure deletion so that the information stored in the deleted file is not simply un-referenced by the file system but is actually overwritten. This makes it impossible for any disk recovery tool to re-created the deleted file. Use the -fuz options to the shred command to hide the shredding operation by overwriting the file with 0s and 19

20 Chapter 2 Configuring SSL Monitoring on the AMD to actually delete the file name form the directory listing while overriding any read protection. For example: [root@amd-35 keys]# shred -fuz my.pem CAUTION Secure deletion is not a necessary step. This is a security measure which you should follow if you do not want the un-encrypted file to remain on the system. Remember that this command will remove the file without any means of recovery of the removed information. 11. Optional: If using OpenSSL and the kpadmin utility, re-start the kpa daemon and re-run the kpadmin. After updating the keylist file you need to re-start the kpa daemon and re-run the kpadmin utility. For more information, see Using KPA to Make Keys Available to the AMD Process [p. 46]. 12. Apply the configuration changes. When the configuration is changed, you have to apply your changes to the AMD. To do so, you have to be logged in to the AMD as user root and execute the following commands: # ndstop # ndstart This will restart your AMD and apply all your configuration changes. What to Do Next If the AMD is connected to a Central Analysis Server installation, then, for SSL decryption to be used for selected servers, you need to add software service definitions for these servers using RUM Console. Here you should add a software service (named, for example, SSL decoded ) and specify that the SSL (with decryption) analyzer is to be used for that definition. SSL Hardware Accelerator Cards If the SSL card has been installed in the AMD during the manufacturing process, the software will also have been installed and it will detect the card, without the need for additional configuration. If, however, the AMD is upgraded and a new SSL accelerator card is added, you will need to install and configure the device driver. For the list of supported hardware accelerator cards see Tested Cards in the Data Center Real User Monitoring Hardware Recommendations. Selecting and Configuring SSL Engine To configure SSL monitoring, you must select the SSL engine to be used, which defines the type of accelerator card used or refers to software decryption. 20

21 Chapter 2 Configuring SSL Monitoring on the AMD Selecting engine type The type of the accelerator card is set in the configuration file rtm.config, in the configuration property named ssl.engine. The value to use depends on the accelerator card: openssl (for OpenSSL) nshield (for nshield 32-bit platform) nfast (for nfast 32-bit platform) ncipher_pkcs11 (for nshield 64-bit platform) nitroxfips (for NITROX) sca6000 (for Sun Crypto Accelerator 6000 supported but not recommended) Example usage: ssl.engine=nitroxfips Specifying the number of dedicated threads For the SSL cards that operate in synchronous mode, AMD spawns dedicated threads to wait for SSL operations on the accelerator. You can increase the number of threads to be executed for the given SSL engine, by setting the ssl.engine.param=threads:number configuration property in the rtm.config file. Specifying more than one thread may improve performance, depending on the performance capacity of the card. The SSL engines for which this setting is supported are: openssl ncipher_pkcs11 sca6000 Specifying key search criteria for the SSL engine The following engines distinguish between key identifiers and key labels. Both of these identification methods can be used to identify the keys in the keylist file. However, you may need to specify the type of identification to be used by editing the rtm.config file and setting the searchkeyby parameter of the ssl.engine.param property to id or label, as appropriate. ncipher_pkcs11 sca6000 Example usage: ssl.engine.param=searchkeyby:id Default key identification is by label. Default key identification is by key identifier. Applying configuration changes When the SSL engine type is chosen and other configuration changed according to your SSL accelerator, you have to apply your changes to the AMD. To do so, you have to be logged in to the AMD as user root and execute the following commands: # ndstop # ndstart 21

22 Chapter 2 Configuring SSL Monitoring on the AMD This will restart your AMD and apply all your configuration changes. You may also want verify that the changes are applied correctly, using the command show SHOW SSLDECR STATUS. For more information, see SHOW SSLDECR STATUS [p. 74] and SSL-Related rcon Commands [p. 65]. Installing and Configuring NITROX XL FIPS Acceleration Board If a new NITROX XL FIPS Acceleration Board has been added to your AMD (inserted into a free PCI slot), you will need to install the appropriate software. See Upgrading the AMD Software in the Data Center Real User Monitoring Agentless Monitoring Device Installation Guide for information on upgrading your AMD. In addition to ensuring that the driver software is present on the AMD, the accelerator card has to be initialized by creating superuser and user accounts, each with a password, as explained below. The configuration is performed using the nitrox-setup command line utility. NOTE NITROX XL FIPS Acceleration Board is referred to as Cavium NITROX XL CN1120-NFB Hardware Security Module or just HSM, in the configuration utility user interface, as described below. All of these names refer to the same entity. FIPS mode Level 3 is referred to as FIPS mode: on in the configuration utility user interface. FIPS mode Level 2 is referred to as FIPS mode: off in the configuration utility user interface. Supported NITROX XL FIPS Acceleration Board Security Levels The NITROX XL FIPS Acceleration Board, model CN NFB-1.1-G, can be configured to operate in the following security modes: FIPS Level 3 high security mode where it requires to be connected to a Pin Entry Device (PED). FIPS Level 2 mode, also referred to as the non-fips mode where connection to a PED device is not required and all operations on the card are performed solely through the hosting computer, that is through your AMD. You can use either of these modes for NITROX XL FIPS Acceleration Boards installed in an AMD. You should decide what mode to use, based on your specific security needs. For further information on security levels, please refer to Cavium Networks NITROX documentation. Invoking Acceleration Board Management Utility The nitrox-setup utility, located in /opt/nitrox_fips/bin, is used to perform configuration and management operations on the hardware security module as well as to facilitate actual card operation. 22

23 Chapter 2 Configuring SSL Monitoring on the AMD In addition to this software management utility, a Pin Entry Device (PED) might also be required to configure and operate the hardware security module, depending on the selected security level. To invoke the hardware security module management utility, log in to the AMD and execute the command: /opt/nitrox_fips/bin/nitrox-setup On startup, the utility displays a menu and information about the current hardware security module label and security level. Example 5. NITROX setup menu and configuration information Agentless Monitoring Configuration and management of Cavium NITROX XL FIPS Hardware Security Module (HSM) HSM label: testlabel1, HSM FIPS mode: off, USER logged in: no 1 - Display HSM status 2 - Initialize HSM 3 - Login as USER 4 - Logout USER 5 - Add RSA private key 6 - Remove RSA private key 7 - List RSA private keys X - Exit Select option and press [ENTER]: The exact function of the menu items is as follows: Display HSM status Displays current status information, including serial number, firmware version, memory size, capabilities and policies. Initialize HSM Initializes the card. This includes defining the security level, specifying SO and USER passwords or configuring and initializing PED keys. It also involves deleting all of the RSA keys currently stored on the card. Login as USER Logs into the card as USER. Logout USER Logs USER out of the card. Add RSA private key Imports an RSA private key to the hardware security module. Remove RSA private key Deletes an RSA private key from the hardware security module. List RSA private keys Lists RSA private keys stored on the hardware security module. Exit Exits the hardware security module management utility. 23

24 Chapter 2 Configuring SSL Monitoring on the AMD Initializing the NITROX XL FIPS Acceleration Board Before the card can be used, it has to be initialized. This includes defining the security level, specifying SO and USER passwords or configuring and initializing PED keys. It also involves deleting all of the keys currently stored on the card. The actual operation of writing initialization information to the acceleration board or deletion of RSA key information is performed in the last step of the initialization dialog. It is therefore possible to abort the initialization process at any point before the final confirmation. Initializing the hardware security module card will result in the deletion of all currently stored key information. To abort initialization before the final confirmation, type [Ctrl-C] to exit the hardware security module management utility. To initialize the NITROX XL FIPS accelerator: 1. Select the initialization option from the menu. To initialize the card, select the Initialize HSM option from the nitrox-setup menu. 2. Select the security level. You will be asked whether the hardware security module is to be initialized in the FIPS high security mode, that is mode Level 3, requiring the use of a PED device. The decision depends on your particular security requirements. Answer y for Yes or n for No, as appropriate. If you select the FIPS high security mode, you will be asked to initialize the PED keys. Please refer to Cavium Network PED documentation for details how to use PED and PED keys. If you select the non-fips mode, that is FIPS mode Level 2, you will be asked to type the new SO and USER passwords. 3. Provide a new acceleration board label. You will be asked for a new acceleration board label. This is an identification string written to the acceleration board. 4. Log in as the security officer (user SO). To be able to proceed with further initialization steps, nitrox-setup will attempt to log you into the card as the security officer (user SO). This means that, depending on the current security level (not the level you have just selected, but the currently active one) you will either need to supply the current SO password or the SO (blue) PED key with a PIN. The factory default setting is non-fips, that is FIPS mode Level 2. The default password can be found in the card manufacturer's documentation or in the /opt/nitrox_fips/doc/utils_readme.txt file, in the section entitled Initializing the board. If the FIPS high security (140-2 Level 3) mode is used, all PED operations, including SO identification, are deferred until you confirm initialization (see the last step of this procedure). CAUTION Three consecutive unsuccessful entries of the SO password will cause hardware security module reset. 5. Provide new SO and USER passwords. 24

25 Chapter 2 Configuring SSL Monitoring on the AMD As part of initialization, you will be asked to supply new security identification for user SO and user USER. If you are using a non-fips mode (FIPS mode Level 2), this you will simply need to enter new passwords for each of these users. In a FIPS high security mode Level 3, you will need to use a PED device and the appropriate keys. 6. Confirm initialization. Finally, you will be asked to confirm all of the above settings. Confirming initialization at this stage causes the hardware security module to be initialized as specified. If there were any PED operations pending, such as SO authorization or initialization of PED keys, they will be performed now. Please refer to the PED manufacturer's documentation for information on initializing and using PED keys. Note that the security officer (SO) will be logged out automatically as part of the initialization step. CAUTION The initialization process must not be aborted after the above (final) confirmation, else the hardware security module may be left in an undefined state, particularly if PED keys are being used. To remedy this situation, the manufacturer of the card has provided the Cfm1Util utility. Once the card falls in the indeterminate state, this tool can be used to reinitialize the card. The Cfm1Util utility is provided with the card software and usage syntax is described in the card's documentation. Example 6. Initializing Hardware Security Module in non-fips mode (FIPS mode Level 2) Agentless Monitoring Configuration and management of Cavium NITROX XL FIPS Hardware Security Module (HSM) HSM label: testlabel1, HSM FIPS mode: off, USER logged in: no 1 - Display HSM status 2 - Initialize HSM 3 - Login as USER 4 - Logout USER 5 - Add RSA private key 6 - Remove RSA private key 7 - List RSA private keys X - Exit Select option and press [ENTER]: 2 Initializing HSM... This step defines a new HSM label, security level and passwords and removes all RSA key information. Continue? (y or n): y Initialize HSM in FIPS mode (use of PIN Entry Device required)? (y or n): n Enter a new HSM label: testlabel1 ***************************************************************************** *** You need to enter the current HSM Security Officer (SO) password. *** *** WARNING: three consecutive unsuccessful entries will cause HSM reset! *** ***************************************************************************** Enter current HSM SO password: Enter a new HSM SO password (8 to 12 characters): Retype HSM SO password: Enter a new HSM USER password (8 to 12 characters, must be different from SO password): 25

26 Chapter 2 Configuring SSL Monitoring on the AMD Retype HSM USER password: *** WARNING: all key information will be deleted from HSM. *** Continue? (y or n): y Starting HSM initialization... Login successful. Initialization successful. Press [ENTER] to continue... Logging In and Out of the NITROX XL FIPS Acceleration Board The user USER must remain logged in order for AMD traffic monitoring software to be able to use the HSM card. Therefore, logging in will be usually the first operation performed after AMD is re-started. You should use the HSM management utility, nitrox-setup to log in and out of the HSM card as USER. HSM management operations, such as listing keys or adding or removing keys can only be performed if USER is logged in. Note that USER remains logged in after the nitrox-setup management utility exits, that is you can exit the menu without causing USER to be logged out. To log in or out of the card, select Login as USER or Logout USER from the nitrox-setup menu. CAUTION For security reasons, ten consecutive unsuccessful login attempts will disable the USER account. RSA Key Management on NITROX XL FIPS RSA key operations, including adding, deleting and listing stored keys, are performed using the nitrox-setup utility. The keys must be imported from unencrypted PEM files. Note that AMD with the hardware security module supports 1024-bit or 2048-bit RSA keys, even though 4096-bit keys can be stored on the hardware security module. For this reason, it is good practice, before loading they keys, to check the size of the keys, using the command: openssl rsa -in keyfile.pem -text Once keys are stored on the hardware security module, they are identified by hexadecimal numbers. Importing a key to acceleration board To import a new RSA key, select the Add RSA private key option from the nitrox-setup menu. Provide the appropriate PEM file name when prompted. If the specified file exists and contains a valid key, the key is imported with the default label PRV_KEY_IMPORT and a new key identifier is generated and displayed. 26