Server Automation Alert: Bootstrap SSL Certificate Expiration

Server Automation Alert: Bootstrap SSL Certificate Expiration (January 26, 2013) Action: Replace SA Bootstrap Secure Socket Layer (SSL) Certificates That Expire on February 3, 2013 Issues that Require Attention... 2 Bootstrap SSL Certificates Expiration... 2 Installation Media... 2 Recommended Actions... 2 Background... 2 Overview of the Patching Process... 3 Patch Actions... 3 Activities Impacted by the Certificate Expiration... 3 Components Impacted by the Certificate Expiration... 3 Installation Media... 4 Patch Installation Instructions... 4 Preinstallation Steps... 4 The bootcert_patch.tar.gz patch... 4 The bootcert_patch_osprov-<release_number>.tar.gz patch... 5 Installation Steps... 7 Patch Installation Error... 10 Standalone Agent Upgrade Tool (recertagentfile)... 11 Binaries Needed to Run this Utility... 11 Useful Commands... 11

Issues that Require Attention Bootstrap SSL Certificates Expiration The agent Bootstrap SSL certificates will expire on February 3, 2013. After this date, new managed Server Automation (SA) devices will be unable to register with the SA Core until the expired certificates are updated. To address the expired certificates issue, HP is issuing a patch containing replacement certificates. Once you install the patch, the new certificates will allow continued secure communications between the SA Core and the SA Agent. Note: SA 9.14 already has the updated agent Bootstrap SSL certificate. If you use 9.14, you do not have to apply the patch. Installation Media Having an expired certificate also impacts the addition of new infrastructure, such as SA Cores, satellites, and slices. After February 3, 2013, if an installation is attempted with the original SA media, the installation process will fail with a Certificate Expiration error during the SA Core Agent installation step. To address the infrastructure impact, HP will release installation media that contains the new certificate needed to install new SA components. Recommended Actions HP recommends that you apply the Bootstrap SSL certificate patch and the installation media to replace the expired certificates by February 3, 2013. You can delay the patch deployments, but you will be unable to add new managed devices (SA Agents, cores, slices, meshes, and satellites) until you apply the patch. Background Every SA release ships with a set of SSL certificates that provide secure communication between an SA Core and a new SA Agent during initial ( bootstrap ) registration, and during deployment of the new agent. These same certificates are used during installation of new SA Core components. The current certificates will expire on February 3, 2013. After this date, if the new certificates are not installed, any newly installed SA Agents will fail to register with the SA Core because they will be using the old certificates. Note: Current managed servers/core functionality will be unaffected you do not need to update their agent certificates. Note: For up-to-date information regarding this patch see the full documentation at: http://support.openview.hp.com/selfsolve/document/km00322074 Server Automation Alert Page 2

Overview of the Patching Process This section describes the patching process, and discusses the activities and components affected. Patch Actions HP will provide two patches that will perform the following actions: 1. Install new SA Agent certificate for all agents in the Software Repository on the Primary Core of your mesh. This will touch the agent executable in the Software Repository and modify the unit record in the database. 2. Install new Bootstrap SSL Certificate authority certificate for core components that have a Data Access Engine ( spin ) and Command Engine ( waybot ). Locations of these new certificates are: /opt/opsware/oi_util/opswarecerttool/common/bootstrap-ca.crt /var/opt/opsware/crypto/spin/bootstrap-ca.crt (not on satellites) /var/opt/opsware/crypto/waybot/bootstrap-ca.crt (not on infra cores, and satellites) 3. Install new agent certificate for Linux GFS Agents. Base directory location: /opt/opsware/boot/kickstart/opsware/ogfs-agent 4. Install new WinPE images. Base directory location: /opt/opsware/boot/tftpboot/winpe /sources/ 5. Clear, and then refresh, the agentcache. 6. Install new ISO images into the Software Repository. Activities Impacted by the Certificate Expiration The following activities are impacted by the certificate expiration: OS Provisioning Build Plans OS Sequences Agent Deployment Communication between new SA Agents and the SA Core. Installation of new Core components (slices, meshes, satellites) the installation will fail. Components Impacted by the Certificate Expiration The following components are impacted by the certificate expiration. Data Access Engine ( spin ) and Command Engine ( waybot ) The Data Access Engine and the Command Engine, which are involved in new agent registration, must have the new certificates to facilitate SA Agent installation. After the new certificate is installed, these components must be restarted. The patch will provide several scripts to restart these components. Current SA Agents All current agents stored in the Software Repository require new certificates. The new certificate allows an agent that is installed on a server to communicate with the SA Core. Server Automation Alert Page 3

ISO Images for OS Provisioning New ISO images (OPSWwinpex64/x86-ogfs-enabled, HPSA_ linux_boot_cd.iso) must be reloaded to the Software Repository after the new certificates are installed. Linux Global File System (GFS/ OGFS ) Build-Plan Agents (ogfs-agent) All Linux OGFS Agent certificates must be updated. Windows Build Images (OGFS-enabled WinPE) Existing OGFS-enabled WinPE images certificates will be outdated. HP will provide new OGFSenabled WinPE images that contain the new certificates. Agentcache The agentcache caches agents for use by other SA components. Although no new software is needed for this component, you must refresh it to ensure that the SA Agent containing the new certificate is in the cache. New Core component installations HP recommends you use the new GA installation media to add new core components, such as slices. Installation Media HP will provide new installation media to install new SA Core components. Note: HP recommends that you apply this patch by February 3, 2013. You can delay the patch deployment, but you will be unable to add the new managed devices (SA Agents, slices, meshes, and satellites) until you apply the patch. In addition, you will see a Certificate Expired error when you try to add new SA Agents. Patch Installation Instructions This section describes the patch preinstallation and installation steps. Preinstallation Steps You do not have to stop any SA processes before you install this patch. However, recently started OS Provisioning or new agent deployment processes will fail if they run concurrently with the patch installation and recertification processes. The bootcert_patch.tar.gz patch The bootcert_patch.tar.gz (approximately 24KB) applies to all patch-supported releases, and will update the Bootstrap SSL Certificate. This patch contains the following files: o coretographer o crypto/agent/agent.srv o new-core-bootstrap.sh o README.txt o recertagentfile o recertagentfile.pyc o recertword o recertword.pyc Server Automation Alert Page 4

This patch contains the following scripts: Name new-core-bootstrap.sh recertword coretographer recertagentfile Action(s)/Function This script: Installs the core bootstrapca.crt/certificate Authority on the core server. Verifies the authenticity of the client certificate (the agent Bootstrap SSL Certificate). Updates the Bootstrap SSL Certificates for the Linuxbased OGFS PXE Agents. This script: Replaces expired certificates of all the Agents in the system on your Primary SA Core. Takes no command-line arguments for normal use. This script: Replaces expired certificates of all the Agents in the system on your Primary SA Core. Takes no command-line arguments for normal use. This is a standalone utility that should only be used when you need to re-certify a standalone agent that was not present in the Software Repository. This utility can also be used to verify that the agent installer has the correct certificate. Check with HP Support for more information on how to use this utility. The bootcert_patch_osprov-<release_number>.tar.gz patch The bootcert_patch_osprov-<release_number>.tar.gz applies to all patch-supported releases, but each release stream has its own patch file. This file will update the SA media. This patch contains the following files: Name upgrade_iso_files.sh OPSWwinpe*.zip OPSWwinpex86-ogfs-enabled.iso OPSWwinpex64-ogfs-enabled.iso HPSA_linux_boot_cd.iso Action(s)/Function This script uploads the DHCP-less OS Provisioning ISOs to the Software Repository. These zip files are the OGFS-enabled WinPE image These OS Provisioning Media ISO files are stored in the Software Repository. This OS Provisioning Media ISO file is stored in the Software Repository. Note: This file is only available in the 9.1 patch bundles. Server Automation Alert Page 5

For SA 9.0x use: bootcert_patch_osprov_9.0x.tar.gz. For SA 9.1x use: bootcert_patch_osprov_<version_number>.tar.gz. Note: Use the 9.12 patch for SA 9.11. For 7.8x: There are no files for 7.8x. The files can range from approximately 700MB to 1.6GB. These file sizes are approximate and subject to change. bootcert_patch_osprov_9.0x.tar.gz bootcert_patch_osprov_9.10.tar.gz bootcert_patch_osprov_9.10.01.tar.gz bootcert_patch_osprov_9.10.02.tar.gz bootcert_patch_osprov_9.12.tar.gz bootcert_patch_osprov_9.13.tar.gz The following is the list of release-specific files contained in the OS Provisioning patches: bootcert_patch_osprov_9.0x: OPSWwinpe-40.0.0.0.64.zip OPSWwinpex64-ogfs-enabled-40.0.0.0.64.iso OPSWwinpex86-ogfs-enabled-40.0.0.0.64.iso upgrade_iso_files.sh bootcert_patch_osprov_9.10: HPSA_linux_boot_cd.iso OPSWwinpex64-ogfs-enabled.iso* upgrade_iso_files.sh OPSWwinpe-45.0.6172.0.zip OPSWwinpex86-ogfs-enabled.iso bootcert_patch_osprov_9.10.01: HPSA_linux_boot_cd.iso OPSWwinpex64-ogfs-enabled.iso upgrade_iso_files.sh OPSWwinpe-45.0.6172.0.zip OPSWwinpex86-ogfs-enabled.iso bootcert_patch_osprov_9.10.02: HPSA_linux_boot_cd.iso OPSWwinpex64-ogfs-enabled.iso upgrade_iso_files.sh OPSWwinpe-45.0.6172.0.zip OPSWwinpex86-ogfs-enabled.iso bootcert_patch_osprov_9.12: (Use these files if you have SA 9.11.) HPSA_linux_boot_cd.iso OPSWwinpex64-ogfs-enabled.iso upgrade_iso_files.sh OPSWwinpe-45.0.15366.0.zip OPSWwinpex86-ogfs-enabled.iso Server Automation Alert Page 6

bootcert_patch_osprov_9.13: HPSA_linux_boot_cd.iso OPSWwinpex64-ogfs-enabled.iso upgrade_iso_files.sh OPSWwinpe-45.0.22115.0.zip OPSWwinpex86-ogfs-enabled.iso Installation Steps This section contains the patch-installation steps. Each explains the actions you need to perform, where you need to perform them, what command you must use (if any), what the exceptions are, and the impact of performing the action. Step 1: Upgrade the Core server component Bootstrap SSL Certificate Authority Action: Copy the new-core-bootstrap.sh file and run the command. Where: On every SA component server (slices, infrastructure, satellites). Command: #./new-core-bootstrap.sh Exceptions: You do not need to run this script on managed servers or on database servers. Action Impact: Updates the bootstrap-ca.crt files located in the following directories: /opt/opsware/oi_util/opswarecerttool/common/bootstrap-ca.crt /var/opt/opsware/crypto/spin/bootstrap-ca.crt (not on satellites) /var/opt/opsware/crypto/waybot/bootstrap-ca.crt (not on infrastructure cores and satellites) Restarts the Command Engine ( waybot ) and Data Access Engine ( spin ). No satellite components will be restarted. For servers that are running the OS Provisioning component, this script will also update the Linux OGFS PXE certificates stored in: /opt/opsware/boot/kickstart/opsware/ogfsagent/<platform>/crypto/agent/agent.srv Step 2: Recertify the Agents in the Primary Software Repository (once per MESH) Pre-Action: This step must be run on the Primary SA Core infrastructure server, or on the first slice (slice 0), if the first slice (slice 0) has been installed on a separate server. Verify if the infrastructure server has been installed with the first slice (slice 0) by running the following command and verifying that mm_wordbot is present in the output: # /etc/init.d/opsware-sas list Server Automation Alert Page 7

Note: Running the coretographer script might also help you to locate the correct server. Your Primary SA Core is the facility that has the Spin Multimaster Central service level. Within your Primary SA Core, find a server that has the Opsware Word Service Level. There might be more than one server that fits this description. It does not matter which server you choose, as long as it is on your Primary SA Core. Action: Run the command once per mesh. Where: SA Core infrastructure server, or on the first slice (slice 0), if slice 0 has been installed on a separate server. Command: #./recertword Exceptions: None. Action Impact: Finds all Agents in the SA Software Repository and recertifies them. Logs messages to the screen. Logs detailed information to the recertword.pyc.log file, which is located in the same directory as the script. If run a second time on the same mesh, the script examines the certificate stored in each agent. If an agent contains the updated Bootstrap SSL Certificate, it skips that agent. If the agent does not contain the updated certificate, it will recertify it. Note: While processing the Windows version of the SA Agents you will see a warning message. Ignore it. It does not impact the Windows agent-recertification process. Sample warning message from a 9.13 core:... opsware-agent-45.0.23169.0-win32-5.0.exe warning [/var/opt/opsware/word/mmword_local/packages/opsware/nt/5.0/opswareagent-45.0.23169.0-win32-5.0.exe]: 257024 extra bytes at beginning or within zipfile (attempting to process anyway) Step 3 Fix the OS Provisioning media stored in the Software Repository Action: Run the script: #./upgrade_iso_files.sh install once per mesh. Where: On the same server as in Step 2. Command: Use the correct patch osprov bundle for your release: For 9.1x, use the specific release bundle: bootcert_patch_osprov_9.10.tar.gz bootcert_patch_osprov_9.10.01.tar.gz bootcert_patch_osprov_9.10.02.tar.gz Server Automation Alert Page 8

bootcert_patch_osprov_9.12.tar.gz Note: Use the SA 9.12 bundle for SA 9.11. bootcert_patch_osprov_9.13.tar.gz For 9.0x, use the following bundle: bootcert_patch_osprov_9.0x.tar.gz Note: There is no 7.8x bundle to install. Example for a 9.12 core: # tar -xvzf bootcert_patch_osprov_9.12.tar.gz # cd boot_cert_patch_osprov_9.12 #./upgrade_iso_files.sh install Exceptions: None. Action Impact: Uploads the following files into the Software Repository (using the 9.1x subdirectory for the example): bootcert_patch_osprov_9.12/hpsa_linux_boot_cd.iso bootcert_patch_osprov_9.12/opswwinpex86-ogfs-enabled.iso bootcert_patch_osprov_9.12/opswwinpex64-ogfs-enabled.iso Step 4: Install the OS Provisioning OGFS WIMs Action: Unzip the OPSWwinpe-45.0.22115.0.zip file from the root (/) file. This WinPE image is the OGFS agent-enabled PXE image. Note: No 7.8x OGFS WIMs are provided. Where: On each satellite and slice that has the OS Provisioning component installed. Command: Example for a 9.12 core: # cd / # unzip <path to patch>/bootcert_patch_osprov_9.12/opswwinpe- 45.0.22115.0.zip Exceptions: You do not need to run this script if you use SA version 7.8x. Action Impact: OGFS agent-enabled PXE images are installed. Step 5: Clear agent cache and restart the agentcache daemon. Action: Check for the agentcache daemon. Empty the cache and restart daemon. Where: On each Core, satellite, and slice that contains the agentcache daemon. Server Automation Alert Page 9

Command: To check for the agentcache daemon, run the following command and verify that agentcache is present in the output: # /etc/init.d/opsware-sas list grep agentcache agentcache To clear the cache, run the following command: # /etc/init.d/opsware-sas stop agentcache # rm -f /var/opt/opsware/agent_installers/* # /etc/init.d/opsware-sas start agentcache Exceptions: None. Action Impact: Cache is cleared. Note: Normally, the agent cache refreshes automatically when a higher agentversion number is detected in the Software Repository. However, because agents are being updated in place without version-number increases, a refresh must be done manually. Installation of the patch is now complete. Patch Installation Error If you have completed the patch installation, and you receive the following error when you are provisioning a new server or installing an agent on a new server, your patch installation was not successful. Contact HP Support for assistance. -----Unexpected Error----- Traceback (innermost last): File "./agent_reg_common.py", line 430, in blockingmainthread File "./agent_reg_common.py", line 413, in blocking_initial_register_or_update File "./agent_reg_common.py", line 198, in register_or_update_hardware_initial File "./spinwrapper.py", line 110, in func File "./spinwrapper.py", line 239, in getinfofromserver File "./xmlrpc/xmlrpclib.py", line 822, in call File "./xmlrpc/lcxmlrpclib.py", line 113, in request File "./xmlrpc/xmlrpclib.py", line 909, in request File "./SSLTransport.py", line 71, in request File "./asyncssl.py", line 108, in connect File "./asyncssl.py", line 157, in _connect_inner File "./asyncssl.py", line 314, in ssl_connect File "./asyncssl.py", line 339, in _timeout_io Error: error:14094418:ssl routines:ssl3_read_bytes:tlsv1 alert unknown ca Server Automation Alert Page 10

Standalone Agent Upgrade Tool (recertagentfile) Note: Do not run this upgrade tool (recertagentfile) unless you have been requested to do so by HP Support. Note: This tool should be run on an SA Slice as it requires other SA libraries/utilities to run. The recertagentfile tool can be used to upgrade the certificate information inside an agent that is not part of the Software Repository. It can also be used to verify the certificate in the agent installer. The tool is not part of the patch-installation process. This tool is useful for the following situations: Upgrading a dormant agent in a VM template. Upgrading/verifying a hotfix-delivered agent before installation. Troubleshooting suspected agent installers. Note: If you are using an agent in a customized VM image, you will need to either update the agent installer by downloading the recertified agent from the Software Repository, or use the recertagentfile utility to recertify that specific agent. Binaries Needed to Run this Utility The following files are needed to run this utility: bootcert_ patch/recertagentfile bootcert_ patch/recertagentfile.pyc bootcert_ patch/crypto/ bootcert_ patch/crypto/agent/ bootcert_ patch/crypto/agent/agent.srv Useful Commands To verify a file, use the following command: #./recertagentfile -v filename To recertify a standalone agent, use the following command: #./recertagentfile filename Server Automation Alert Page 11