CA ehealth. High Availability and Disaster Recovery Administration Guide. r6.1

Transcription

1 CA ehealth High Availability and Disaster Recovery Administration Guide r6.1

2 This documentation and any related computer software help programs (hereinafter referred to as the Documentation ) is for the end user s informational purposes only and is subject to change or withdrawal by CA at any time. This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and protected by the copyright laws of the United States and international treaties. Notwithstanding the foregoing, licensed users may print a reasonable number of copies of the Documentation for their own internal use, and may make one copy of the related software as reasonably required for back-up and disaster recovery purposes, provided that all CA copyright notices and legends are affixed to each reproduced copy. Only authorized employees, consultants, or agents of the user who are bound by the provisions of the license for the Product are permitted to have access to such copies. The right to print copies of the Documentation and to make a copy of the related software is limited to the period during which the applicable license for the Product remains in full force and effect. Should the license terminate for any reason, it shall be the user s responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed. EXCEPT AS OTHERWISE STATED IN THE APPLICABLE LICENSE AGREEMENT, TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION AS IS WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO THE END USER OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED OF SUCH LOSS OR DAMAGE. The use of any product referenced in the Documentation is governed by the end user s applicable license agreement. The manufacturer of this Documentation is CA. Provided with Restricted Rights. Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections , , and (c)(1) - (2) and DFARS Section (b)(3), as applicable, or their successors. All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies. Copyright 2008 CA. All rights reserved.

3 Contact CA Contact Technical Support For online technical assistance and a complete list of locations, primary service hours, and telephone numbers, contact Technical Support at Provide Feedback If you have comments or questions about CA product documentation, you can send a message to techpubs@ca.com. If you would like to provide feedback about CA product documentation, please complete our short customer survey, which is also available on the CA Support website.

4 CA Product References This document may reference the following CA products: CA ehealth AdvantEDGE View CA ehealth Application Response CA ehealth Business Service Console (ehealth BSC) CA ehealth Distributed ehealth CA ehealth Fault Manager CA ehealth Live Health Application CA ehealth Response CA ehealth Service Availability CA ehealth SystemEDGE CA ehealth TrapEXPLODER CA ehealth Voice Quality Monitor (VQM) CA ehealth AIM for Apache CA ehealth AIM for Microsoft Exchange CA ehealth AIM for Microsoft IIS CA ehealth AIM for Microsoft SQL Server CA ehealth AIM for Oracle CA Insight AIM for CA ehealth CA Insight Database Performance Monitor for Distributed Databases (CA Insight DPM for Distributed Databases) CA ehealth Integration for Alcatel (ehealth - Alcatel) CA ehealth Integration for Cisco IP Solution Center (ehealth - Cisco ISC) CA ehealth Integration for Cisco WAN Manager (ehealth - Cisco WAN Manager) CA ehealth Integration for HP OpenView (ehealth - OpenView) CA ehealth Integration for Lucent (ehealth - Lucent) CA ehealth Integration for Netcool (ehealth - Netcool) CA ehealth Integration for Nortel Preside (ehealth - Nortel Preside) CA ehealth Integration for Nortel Shasta SCS GGSN (ehealth - Nortel GGSN) CA ehealth Integration for Psytechnics (ehealth - Psytechnics)

5 CA ehealth Integration for Starent (ehealth - Starent) CA SPECTRUM CA Unicenter Network and Systems Management (Unicenter NSM) CA etrust Identity and Access Management (etrust IAM) CA Embedded Entitlements Manager (CA EEM) Note: CA Embedded Entitlements Manager (CA EEM) is the new name for etrust IAM. This product will be rebranded throughout the documentation in a future release.

6

7 Contents Chapter 1: Introduction to High Availability and Disaster Recovery 11 Overview High Availability Clusterware High Availability Clusters High Availability Clusters and Distributed ehealth Clusters Failures Fail over How ehealth Works with High Availability How Monitoring Works What Clusterware Monitors Chapter 2: Sun Cluster High Availability Configuration 21 How to Configure Sun Cluster with High Availability Step 1: Install and Configure Sun Cluster Software Step 2: Register Clusterware Resource Types Step 3: Create Resource Types and the Resource Group Step 4: Install ehealth Step 5: Configure ehealth to Run in the High Availability Cluster Step 6: (Optional) Install CA XOsoft Replication Software Step 7: Create, Install, and Register ehealth Resource Types Step 8: Create and Add Clusterware Resources to the Resource Group Step 9: Create and Add ehealth Resources to the Resource Group Step 10: Start the High Availability Service How to Configure Multiple Instances of ehealth High Availability Cluster Management Tasks Install a Patch, Service Pack, or Upgrade Upgrade the Sun Cluster High Availability Package Load the Database after High Availability Configuration Initiate a Planned Failover How to Enable or Disable SSL for Sun Cluster Chapter 3: Veritas High Availability Configuration 37 Critical Resource Functionality Critical Resources Non-critical Resources Contents 7

8 How to Configure Veritas with High Availability Step 1: Install and Configure VCS Software Step 2: Install ehealth Step 3: Configure ehealth to Run in the High Availability Cluster Step 4: Create a Service Group Step 5: (Optional) Install CA XOsoft Replication Software Step 6: Create and Install ehealth Resource Types Step 7: Start the High Availability Service Multiple Instances of ehealth in the HA Cluster How to Add Multiple Instances of ehealth High Availability Cluster Management Tasks Install a Patch or Service Pack Load the Database after High Availability Configuration Initiate a Planned Failover How to Enable or Disable SSL for Veritas Chapter 4: Disaster Recovery 53 How ehealth Works with Disaster Recovery System Guidelines System Terminology How CA XOsoft Replication Software Works Scenarios Replication Failover in a Disaster Recovery Environment Downtime and Data Loss Configure ehealth for Disaster Recovery Before You Configure ehealth for DR Windows Disaster Recovery Configuration Solaris Disaster Recovery Configuration Start the Failover Process How to Upgrade Your Disaster Recovery Environment Disaster Recovery Management Tasks Initiate Failover from the Standby (Active) System to the Primary System Install a Patch or Service Pack after Configuring Disaster Recovery Appendix A: Disaster Recovery in a High Availability Cluster 75 Configure ehealth for Disaster Recovery in a High Availability Cluster Failover from High Availability Cluster to Standby Disaster Recovery System Failover From Standby System to a System in a High Availability Cluster High Availability and Disaster Recovery Administration Guide

9 Appendix B: Sun Cluster Commands 87 Appendix C: Veritas Cluster Service Commands 89 Index 91 Contents 9

10

11 Chapter 1: Introduction to High Availability and Disaster Recovery This section contains the following topics: Overview (see page 11) High Availability (see page 12) How ehealth Works with High Availability (see page 17) How Monitoring Works (see page 17) Overview ehealth can be configured to run in the following environments: High availability (Solaris 2.9 or 2.10) Disaster recovery (Windows 2003 and Solaris 2.9 or 2.10) Running ehealth in these environments helps ensure that internal system failures (such as hardware or software malfunctions) and external failures (such as a natural disaster or operator error) will minimize ehealth system downtime and data loss. High Availability-High availability (HA) is a system implementation based on levels of redundancy that helps ensure a system or application can quickly come back online in the event of a failure. Highly available systems are often characterized by the ability of their components to fail over to backup systems in the event of a failure. For a strong HA solution, many large system environments are configured into clusters using clusterware or other application support. These HA clusters consist of two or more nodes that work together to build hardware and software redundancy: if one system fails, another can become active and resume its duties. This functionality should not be confused with load balance (or web farm) clusters, where multiple machines run concurrently with identical hardware and software configurations. When one system fails, the others continue to function as before, picking up the load dropped by the failed machine. Chapter 1: Introduction to High Availability and Disaster Recovery 11

12 High Availability Disaster Recovery-Disaster recovery (DR) is a network configuration that becomes vital when systems fail due to disasters like floods or fires. The DR service helps recover critical data after such events occur. DR includes the continual backup of data from a primary system to a standby system in a geographically separate location. After a catastrophic failure causes the primary system to go down, the standby system is manually started and resumes the first system s duties. When planning your DR environments, keep in mind that to configure ehealth for DR, you must first install CA XOsoft Replication software. You can also configure an ehealth Solaris HA cluster for DR using either Sun Cluster software or Veritas Cluster Server software. Note: In the DR environment, failover is a manual process. High Availability This section describes: HA clusterware and associated concepts, including clusters, failures, and failover. The process to implement HA in your infrastructure so ehealth can function as a highly available data service. Clusterware Clusterware is software that monitors hardware and connectivity issues and builds redundancy into a system cluster to eliminate single points of failure. Clusterware must be installed and configured on all machines in the environment before you install ehealth. High Availability Clusters Clusterware enables you to configure several similar systems that use shared storage into a highly available cluster, and deploy instances of applications into that cluster. When a failure occurs, the clusterware helps ensure the applications continue to run by restarting the instances on that system or failing over to a backup system. 12 High Availability and Disaster Recovery Administration Guide

13 High Availability Build the High Availability Cluster You begin building an HA cluster when you install clusterware on the first machine, or node. When you install clusterware on the second node, you tell it to join the group that the first node is a member of. Continue adding nodes to the group until your cluster is built. You need at least two nodes in a cluster to achieve redundancy. HA clusters are considered failover pools; machines that operate, fail over, and share resources together. Each cluster node is a stand-alone server that runs its own processes. This HA cluster solution uses a shared-storage model so that all applications connected to the cluster have read and write access to the stored data. High Availability Clusters and Distributed ehealth Clusters ehealth systems are configured to run as stand-alone machines (a single workstation) or part of a Distributed ehealth Cluster (multiple ehealth systems located centrally or across a geographic range). For distributed ehealth members to be highly available, they can be configured to be members of an HA cluster. While they may work in the same environment, each of these cluster types has its own job, separate from the other. For example, the nodes of an HA cluster function normally (as described in High Availability Clusters (see page 12)) whether or not they share members with a Distributed ehealth Cluster. Also, a Distributed ehealth Cluster is not aware when its members become part of an HA cluster and continue to poll and process data across multiple ehealth systems. Failover behavior does not change in a Distributed ehealth Cluster sharing membership with an HA cluster. Systems can be added or removed from either cluster type without interrupting functionality. Note: For more information about Distributed ehealth Clusters, see the Distributed ehealth Administration Guide. Failures No matter how well a system is designed or maintained, there will always be system failures. There are various types of failures that can have an impact on the availability of ehealth servers and applications. These include hardware problems such as a failed CPU, a corrupted disk, and bad memory; also software problems such as mis-configuration of the operating system and incorrectly specified networking parameters. Chapter 1: Introduction to High Availability and Disaster Recovery 13

14 High Availability ehealth failures can cause the following types of outages: ehealth is inaccessible, with ehealth administrators unable to access it through the OneClick for ehealth interface. ehealth is unable to poll or save data to the ehealth database. ehealth fails if all elements return a bad poll. ehealth reports run through the web are inaccessible. Live Health and reporting applications are inaccessible. Scheduled ehealth reports are not run. Reports being run at the time of failure are lost. When you configure your ehealth environment to be highly available, ehealth servers and applications continue to run and provide access to data with minimal interruption, even when failures occur. Fail over Failover is a service that initiates when a node or application failure is detected. To achieve failover, an automatic series of steps is performed by clusterware to migrate ehealth resources and applications from a failing node to a backup node in the HA cluster, providing HA through redundancy. During failover in an ehealth environment, the clusterware will either try to restart the ehealth application or start and run the application on the backup system, making the backup system active. After the failed application or system recovers, the backup system can be manually failed back to the primary system, and the primary system becomes active again. Note: For information about failover in a DR environment, see Failover in a Disaster Recovery Environment. High Availability Failover Configurations ehealth continually monitors the status of the poller. Depending on a default time-out threshold, ehealth determines when to restart or failover its monitored applications. In general, the clusterware you use determines which HA configurations you can have in your ehealth environment. Sun Cluster and Veritas software support the following cluster configurations: Two nodes (1 + 1); one primary and one backup. Multiple nodes (N + 1); several primary with a single shared backup. Multiple nodes (N * N) with multiple backups. 14 High Availability and Disaster Recovery Administration Guide

15 High Availability The following graphic details how ehealth is configured in an HA cluster with two nodes. As shown, the cluster is running properly, ready in the event of failover. Chapter 1: Introduction to High Availability and Disaster Recovery 15

16 High Availability Downtime and Data Loss There may be some downtime and data loss when your ehealth system fails over. Downtime-Downtime is a period of time when you cannot access the ehealth system. For example, you cannot run a report or the system does not register a poll. Note: If you are connected to an ehealth client application using a shared (virtual) hostname, such as OneClick for ehealth, after the failover you can use the session you were using before failover without having to log back in to the application. After Failover Planned Failover Data Loss- Data loss is a period of time when the ehealth system does not collect data, or the data that was collected is lost. After an ehealth system fails over there will be a short data gap as the backup servers start and the poller initializes. If ehealth fails over while running a report, the process will be interrupted and the data will be lost. Scheduled jobs in the run queue will also be lost during failover. After failover, the system on which the failure occurred is offline and its processes have been migrated to a backup system and started there, making the backup system active. While the backup system is active, you should perform any necessary maintenance in order to bring the failed system back online. You can then keep the system as a backup system or plan to fail back to the original system. You can manually bring ehealth online and offline using the command line interface. This is helpful when you perform planned maintenance on a system running ehealth. You can essentially force a failover at a convenient time allowing you to perform maintenance to the hardware or operating system without interrupting HA functionality. After the primary system is functional, bring the system back online and transfer operations back to the original system. When you bring ehealth up on the backup system, ehealth will experience missed polls causing some downtown, often minimal, depending on the number of elements you are polling and your system configuration. Note: When a system or application is considered online, it is turned on and active. When a system or application is considered offline, it is physically shut down. 16 High Availability and Disaster Recovery Administration Guide

17 How ehealth Works with High Availability How ehealth Works with High Availability ehealth can be configured with HA clusterware to run as a data service in a highly available cluster, where, on a separate disk, there is one instance of the ehealth server and Oracle. In an HA configuration, a number of different machines will be able to mount the disk and have access to ehealth. This disk can be RAID 1 configured: if one part of the disk fails, another takes over and continues the ehealth service without interruption. This adds another level of redundancy, buffering your system from potential downtime and data loss. The ehealth HA cluster functionality includes: The nhhasetup command, which allows an existing ehealth system to be managed by third-party clusterware. The ability for an HA cluster to join an ehealth system in a Distributed ehealth Cluster. The ability for the clusterware to speak with ehealth so it can monitor the ehealth applications and let ehealth monitor polling status. The use of shared hostnames in the ehealth environment. During failover, the clusterware moves the shared hostname (or names) from the primary system to the backup system; this lets you continue accessing ehealth applications with very little interruption. Note: When you connect to the ehealth server, Oracle, or any ehealth applications in an HA cluster, you must use the shared hostname (or names) confirmed when you initially configured ehealth for HA using the nhhasetup command. The use of one ehealth license across multiple systems in an HA cluster. When a failover occurs, you will not have to change or update any license files. How Monitoring Works After ehealth is configured for HA, all systems and applications in the HA cluster are monitored, helping ensure early detection of hardware and software failures. ehealth and the clusterware have specific applications and systems they monitor. Chapter 1: Introduction to High Availability and Disaster Recovery 17

18 How Monitoring Works To communicate with the clusterware, all ehealth applications and services must be organized using the following: Resource Type For monitoring purposes, all ehealth applications and processes must be configured as resource types to be managed by the clusterware. A resource type is a group of parameters that helps define the application in an HA cluster, defining where it is and what to monitor. For example, when a resource fails it can either be restarted on the same system or all the resources in a group can be failed over, depending on the group s configuration. Resource A resource is one instance of a resource type defined across an HA cluster. Resource/Service Group A resource group or service group is a collection of resources that should all fail over at once. When all ehealth applications reside in a group, an association of dependence is created. For example, ehealth startup may depend on the Oracle startup. Also, if a resource has to restart on a backup system, all resources in the group will restart on that backup system. What Clusterware Monitors High Availability Cluster Node Failure Clusterware monitors the HA cluster to determine if all cluster nodes are functioning properly. It also monitors most ehealth resources. Clusterware monitors hardware and network connectivity problems in the HA cluster. Each node contacts the other nodes with a heartbeat message to verify that it is connected and functioning properly. When the nodes get a response, the cluster continues functioning normally. If a node does not respond, each node continues to send a monitored heartbeat message to the other nodes, confirming that there is a unresponsive node. It then fails over the node and migrates resources to a backup node. 18 High Availability and Disaster Recovery Administration Guide

19 How Monitoring Works Clusterware Resources The HA clusterware monitors the following resources: Note: Resource failures result in failover, unless otherwise noted. Host (IP address)-the clusterware validates that the shared hostname that you specify is connected properly to the network. A failure occurs if the local machine thinks the IP address is no longer valid. Disk-The ehealth and Oracle processes reside on one disk in the HA cluster. The clusterware confirms that the disk is correctly mounted. A failure occurs if your disk connection fails. Oracle and Oracle Listener-The clusterware validates that Oracle and Oracle Listener are running and the processes are up. A failure occurs if the Oracle processes stop running. See the clusterware documentation regarding any additional monitoring capability. Apache Web Server-The clusterware validates that the Apache processes are up and running. A failure occurs if Apache processes stop running. See the clusterware documentation regarding any additional monitoring capability. WANSync (if you configure the primary HA system to be configured for DR)-The clusterware validates that the principal (ehealth data) scenario is running and the WANSync daemons exist in the process table. A failure occurs if the principal scenario is stopped or in an otherwise non-running state, or the daemons are not in place. A WANSync resource failure results in the following actions, depending on your clusterware: Veritas Cluster Service-The WANSync resource shows a fault but does not initiate failover. Sun Cluster-The WANSync resource initiates failover. Note: When you use Sun Cluster software, the Host and Disk resources must be set up before ehealth is installed. Oracle, Oracle Listener, and Apache resources must be set up after ehealth is installed. Licensing-The clusterware validates that the license processes are running. A failure occurs is the processes shut down. TrapEXPLODER-The clusterware validates that the trap processes are running. A failure occurs if these processes shut down. A TrapEXPLODER resource failure results in the following actions, depending on your clusterware: Veritas Cluster Service-The TrapEXPLODER resource shows a fault but does not initiate failover. Sun Cluster-The TrapEXPLODER resource initiates failover. Chapter 1: Introduction to High Availability and Disaster Recovery 19

20 How Monitoring Works Servlet (Tomcat)-The clusterware validates that the Tomcat processes are running. A failure occurs if these processes stop. A Tomcat resource failure results in the following actions, depending on your clusterware: Veritas Cluster Service-The Tomcat resource shows a fault but does not initiate failover. Sun Cluster-The Tomcat resource initiates failover. Report Center (if installed)-the clusterware confirms that Report Center has a valid connection. A failure occurs when the clusterware does not detect a connection. A Report Center resource failure results in the following actions, depending on your clusterware: Veritas Cluster Service-The Report Center resource shows a fault but does not initiate failover. Sun Cluster-The Report Center resource initiates failover. What ehealth Monitors The Poller Monitor ehealth monitors itself for poller status as well as the status of the ehealth startup server. The monitoring reads a configuration file to determine what thresholds should be used to control monitoring. An integral part of the ehealth program is its ability to poll devices in your infrastructure and write data to the ehealth database. In an HA cluster, ehealth runs its monitoring process once every minute and can detect and validate serious polling errors. Failure Thresholds Failure of the ehealth system to write data to the database by the statistics, traffic, or import poller within a certain time frame will trigger the system to restart or failover. For example, if the poller is configured to poll once every five minutes and after fifteen minutes has not written data to the database, the ehealth monitoring program will return a failed message. Thresholds, parameters by which failure is measured, are used as part of the monitoring process. When you configure ehealth for HA, ehealth loads threshold defaults for each application. The thresholds determine when an ehealth system or application has failed. Thresholds should only be adjusted by Technical Support. 20 High Availability and Disaster Recovery Administration Guide

21 Chapter 2: Sun Cluster High Availability Configuration This section contains the following topics: How to Configure Sun Cluster with High Availability (see page 21) How to Configure Multiple Instances of ehealth (see page 31) High Availability Cluster Management Tasks (see page 32) How to Configure Sun Cluster with High Availability The following list outlines the process to configure ehealth for HA with Sun Cluster: 1. Install and configure Sun Cluster software. 2. Register clusterware resource types. 3. Create resource types and the resource group. 4. Install ehealth. 5. Configure ehealth to run in the HA cluster. 6. (Optional) Install CA XOsoft Replication software. 7. Create, install, and register ehealth resource types. 8. Create and add clusterware resources to the resource group. 9. Create and add ehealth resources to the resource group. 10. Start the high availability service. Note: In the following procedures, rgroup-1 represents the name of the unique resource group. Chapter 2: Sun Cluster High Availability Configuration 21

22 How to Configure Sun Cluster with High Availability Step 1: Install and Configure Sun Cluster Software For ehealth to be configured for HA, Sun Cluster software version 3.1 or 3.2 must first be installed and configured, creating an HA cluster that ehealth can be managed by. When planning your HA cluster, follow these guidelines: Configure the primary system to use the shared (or virtual) hostname. Add the shared hostname and IP address to the /etc/hosts file on the primary system by doing the following: 1. Log on to the primary system as the root user and open a command line interface. 2. In the /etc/inet/hosts and /etc/inet/ipnodes files, add the shared IP address and hostname. These files map IP addresses to hostnames. Use the procedures in the Sun Cluster Software Installation Guide to do the following: Install the Sun Cluster software on all systems that will make up the HA Cluster. Configure the HA cluster and setup disk sets for locally mounted, highly available storage. Configure IP Multipath Networking in the HA cluster so that shared hostnames can be used. Install the Sun Cluster Apache agent and the Sun Cluster Oracle (not Oracle RAC) agent on each node in the HA cluster. Step 2: Register Clusterware Resource Types Before you install ehealth you must register the resource types that the clusterware will monitor. Note: Complete all the procedures in this chapter from any node in the HA cluster. 22 High Availability and Disaster Recovery Administration Guide

23 How to Configure Sun Cluster with High Availability To register clusterware resource types 1. From any system in the HA cluster, open a CLI. 2. Enter the following commands to install the Sun Cluster resource types: > su root # scrgadm -a -t SUNW.apache # scrgadm -a -t SUNW.oracle_server # scrgadm -a -t SUNW.oracle_listener # scrgadm -a -t SUNW.HAStoragePlus The following resource types are installed and registered: Apache, Oracle, Oracle Listener, and Disk (HAStoragePlus). Step 3: Create Resource Types and the Resource Group When you create the resource group it must point to the instance of ehealth you will be installing. You must supply the name of the directory where ehealth will be installed so that the resource group can indicate the location of support tools to the rest of the ehealth resources. During this step the following actions occur: The Host (IP) resource type is created. The Disk resource type is created. This will be a highly available local file system. Remember this location as you will need to install ehealth on the path you specify for this resource. The ehealth resource types are created. The resource group is brought online so that the disks are accessible. Choose a unique name for the resource group. If you are configuring multiple instances of ehealth to run in the HA cluster, each must reside in a separate resource group with a unique name. Also, each resource type you create must be given a unique name. Chapter 2: Sun Cluster High Availability Configuration 23

24 How to Configure Sun Cluster with High Availability To create the resource types and resource group Note: The file system should not already be mounted. The file system will be mounted during this step when you run the Sun command scswitch. 1. Open a CLI. 2. Enter the following command: > su root # scrgadm -a -g rgroup-1 -y PathPrefix=/mountpoint # scrgadm -a -j ehealth-host -g rgroup-1 -L -l sharedhost # scrgadm -a -j ehealth-storage -g rgroup-1 -t SUNW.HAStoragePlus \ -x FilesystemMountPoints=/mountpoint \ -x AffinityOn=True # scswitch -Z -g rgroup-1 rgroup-1 Represents the unique resource group name. mountpoint Represents the file system mountpoint sharedhost Represents the shared hostname that must always be used whenever you access the ehealth server and applications. Step 4: Install ehealth Install ehealth using the ehealth Installation Guide and the following guidelines: Install ehealth into the storage directory you created in Step 3: Create Resource Types and the Resource Group (see page 23). Confirm that you install ehealth, Oracle, and create the ehealth database on the path specified in the Disk resource in Step 3. For example, install ehealth in /mountpoint/software, install Oracle in /mountpoint/oracle, and create the database in /mountpoint/database. Before you install the ehealth license file, license.dat, you need to configure the file to include all host IDs, including all possible backup systems by doing the following: 1. Send all host IDs to the ehealth product licensing team. You will receive back one license file. 2. Copy and paste the license keys into the license.dat file located on the mountpoint/software/lmgr directory. It will reside on the disk to be accessed by all the ehealth systems in the HA cluster. 24 High Availability and Disaster Recovery Administration Guide

25 How to Configure Sun Cluster with High Availability Note: If you are configuring ehealth for both HA and DR, to save time you can install the WANSync software concurrent to ehealth. Do not create the replication scenarios until ehealth has been configured for HA. For complete steps, see Configuring ehealth for Disaster Recovery in a High Availability Cluster (Solaris Only). Step 5: Configure ehealth to Run in the High Availability Cluster After you install ehealth, you must run the nhhasetup command to configure ehealth to be started and managed by Sun Cluster. To configure ehealth to run in the HA cluster 1. (Optional) If your system is already running as part of a Distributed ehealth Cluster, you must complete this step before you run nhhasetup. From the primary system, run the following command: nhmodifyclustermember -cluster -name physicalhost -newname sharedhost physicalhost Represents the name of the primary HA system prior to being HA enabled sharedhost Represents the shared hostname. This prepares the Distributed ehealth Cluster to allow the cluster members to communicate with the primary system using the shared hostname. Note: If your system is already running as part of a Distributed ehealth Cluster and you have already run nhhasetup, you must enter the following commands to establish communication between all ehealth cluster members, from a system in the Distributed ehealth Cluster that nhhasetup was not run on: nhmodifyclustermember -all -name physicalhost -newname sharedhost You will receive an error from the Distributed ehealth Cluster member configured for HA because the other systems cannot communicate with it until the command finishes running. 2. Open a CLI and start the ehealth database by entering the following command: $NH_HOME/bin/nhStartDb Chapter 2: Sun Cluster High Availability Configuration 25

26 How to Configure Sun Cluster with High Availability 3. Enter the following command to configure TrapEXPLODER to run in the HA cluster by copying /opt/trapx to a shared file system: > su root # mv /opt/trapx /mountpoint mountpoint Represents a shared file system mountpoint. 4. Enable and verify the shared hostname on the system: a. As root, in the command line interface, enter the following: ifconfig XXXX:n plumb ifconfig XXXX:n ipaddress up b. Verify the shared hostname is properly set by entering the following: ifconfig -a The names of the system interfaces should appear in the output. 5. Enter the following command to configure ehealth to support shared hostnames: > $NH_HOME/bin/nhHaSetup -hostname sharedhost On WANSync (if installed): # rm /etc/rc*.d/[sk]*wansync 6. Enter the following command to remove the startup files to prevent ehealth from starting automatically at startup time: > su root # rm /etc/rc*.d/[sk]*httpd # rm /etc/rc*.d/[sk]*nethealth # rm /etc/rc*.d/[sk]*nhreportcenter # rm /etc/rc*.d/[sk]*trapexploder Step 6: (Optional) Install CA XOsoft Replication Software Complete this step only if you want your HA cluster to be configured for DR. For the complete procedure, see Configuring ehealth for Disaster Recovery in a High Availability Cluster (Solaris Only). Step 7: Create, Install, and Register ehealth Resource Types The resource types necessary to monitor ehealth must be installed on every node in the HA cluster. They also need to be registered. 26 High Availability and Disaster Recovery Administration Guide

27 How to Configure Sun Cluster with High Availability To create, install, and register ehealth resource types 1. As the ehealth user, stop the ehealth processes by entering the following commands: $NH_HOME/bin/nhServer stop $NH_HOME/bin/nhReportCenter stop $NH_HOME/bin/nhHttpd stop $NH_HOME/bin/nhLmgr stop $NH_HOME/bin/nhStopDb The ehealth processes are stopped. 2. On the machine on which the resource group is currently online, enter the following command to access the Solaris package: > su root # cd $NH_HOME/sys # pkgadd -d CAehealthHA.pkg 3. Change to the next machine on which you want to install the resources by entering the following commands: > su root # scswitch -z -g rgroup-1 -h system_name system_name Represents the name of the machine on which you want to install the resource types. 4. Log on to the next machine and repeat Step 2 and 3. Continue this until the resource types have been installed on every node in the cluster. 5. On any machine, enter the following commands to register the ehealth resource types: > su root # scrgadm -a -t CA.ehealthLicense # scrgadm -a -t CA.ehealth # scrgadm -a -t CA.ehealthRpt # scrgadm -a -t CA.ehealthServlet # scrgadm -a -t CA.ehealthTrap # scrgadm -a -t CA.ehealthWansync # scrgadm -a -t CA.ehealthWansync is only needed if you are setting up HA and DR. The ehealth resource types are now created, installed, and registered. Note: To view all registered resource types, enter the following command: scrgadm -p Chapter 2: Sun Cluster High Availability Configuration 27

28 How to Configure Sun Cluster with High Availability Step 8: Create and Add Clusterware Resources to the Resource Group The clusterware resources must be created. To do this, you create the Apache and Oracle resources and add them to the resource group. To configure clusterware resources 1. If you have not done so, source the nethealthrc ehealth file as root for your shell environment using one of the following commands: Bourne:../nethealthrc.sh C: source nethealthrc.csh Korn:./nethealthrc.ksh The ehealth environment is set. 2. Enter the following command to create an Apache resource to monitor the web server: > su root # scrgadm -a -j ehealth-web -g rgroup-1 \ -t SUNW.apache -y Network_resources_used=ehealth-host \ -y Scalable=False -y Port_list=$NH_HTTP_PORT/tcp \ -x Bin_dir=$NH_HOME/web/httpd/bin \ -y resource_dependencies=ehealth-storage Note: If SSL has been enabled, create the ehealth-web resource by using the NH_HTTP_PORT value found in the $NH_HOME/web/webCfg/servers.properties file. For example, if the Port value for SSL is now 443, the command to create the resource should read scrgadm a j ehealth-web..port_list=443/tcp. If SSL is not enabled, continue to use Port_list=$NH_HTTP_Port/tcp. 3. Enter the following command to create an Oracle resource to monitor the database: # scrgadm -a -j ehealth-db -g rgroup-1 -t SUNW.oracle_server \ -x Connect_string=$NH_USER/$NH_USER -x ORACLE_SID=$ORACLE_SID \ -x ORACLE_HOME=$ORACLE_HOME \ -x Alert_log_file=$ORACLE_HOME/admin/bdump/ $ORACLE_SID/alert_$ORACLE_SID.log \ -x Restart_type=RESOURCE_GROUP_RESTART \ -y resource_dependencies=ehealth-storage When you create the Oracle resource, you may receive errors indicating that the ORACLE_HOME directory is not currently accessible on all of the nodes in the HA cluster. This is normal when using locally mounted, highly available file systems and can be ignored. 28 High Availability and Disaster Recovery Administration Guide

29 How to Configure Sun Cluster with High Availability Note: When you enter this command, you set the password for the Oracle database. If you plan to change this password at a future date, use the following command: scrgadm -c -j ehealth-db -x Connect_string=new_user/new_password 4. Enter the following command to create a Listener resource to monitor the Oracle Listener: # scrgadm -a -j ehealth-db-listener -g rgroup-1 \ -t SUNW.oracle_listener \ -x LISTENER_NAME=listener \ -x ORACLE_HOME=$ORACLE_HOME \ -y resource_dependencies=ehealth-storage 5. Enter the following command to bring the new resources online: # scswitch -Z -g rgroup-1 The clusterware resources are created and added to the resource group. Step 9: Create and Add ehealth Resources to the Resource Group The ehealth resources must be created and added to the resource group so that when failover occurs, all ehealth processes, not just individual applications, start on the backup system. Adding resources to a group also establishes a dependency relationship between the associated ehealth processes that help determine how they are managed by the clusterware. The resources in this procedure are the default ehealth resources. For example, you can choose not to add the WANSync resource to the ehealth resource group at this time but at a later date. To add ehealth resources to the group 1. As root, specify the time zone environment variable (TZ) in your nethealthrc file. Example: Bourne: TZ= US/Eastern ; export TZ C: setenv TZ US/Eastern Korn: TZ= US/Eastern export TZ Note: Do not omit this step or errors will occur. Chapter 2: Sun Cluster High Availability Configuration 29

30 How to Configure Sun Cluster with High Availability 2. Enter the following commands to add the ehealth resources to the resource group: > su root # scrgadm -a -j ehealth-license -g rgroup-1 -t CA.ehealthLicense \ -y Network_resources_used=ehealth-host \ -x NH_HOME=$NH_HOME \ -x NH_USER=$NH_USER \ -y resource_dependencies=ehealth-storage # scrgadm -a -j ehealth -g rgroup-1 -t CA.ehealth \ -y Network_resources_used=ehealth-host \ -x NH_HOME=$NH_HOME \ -x NH_USER=$NH_USER \ -y resource_dependencies=ehealth-db,ehealth-db-listener, ehealth-license,ehealth-storage # scrgadm -a -j ehealth-rptctr -g rgroup-1 -t CA.ehealthRpt \ -y Network_resources_used=ehealth-host \ -x NH_HOME=$NH_HOME \ -x NH_USER=$NH_USER \ -y resource_dependencies=ehealth-db,ehealth-db-listener, ehealth-storage,ehealth-web # scrgadm -a -j ehealth-servlet -g rgroup-1 -t CA.ehealthServlet \ -y Network_resources_used=ehealth-host \ -x NH_HOME=$NH_HOME \ -x NH_USER=$NH_USER \ -y resource_dependencies=ehealth-db,ehealth-db-listener, ehealth-storage # scrgadm -a -j ehealth-trap -g rgroup-1 -t CA.ehealthTrap \ -y Network_resources_used=ehealth-host \ -x NH_HOME=$NH_HOME \ -x Trap_dir=/mountpoint/trapx \ -y resource_dependencies=ehealth-storage,ehealth-license Step 10: Start the High Availability Service To enable the HA cluster and bring it online, start the fully-configured HA service by enabling the resource group and all resources in the group. To start the HA service Enter the following command to bring the resource group online: > su root scswitch -Z -g rgroup-1 Note: Running the scswitch command with the -Z argument brings the resource group online and enables every resource within the group, even if those resources were previously disabled using the -n argument. 30 High Availability and Disaster Recovery Administration Guide

31 How to Configure Multiple Instances of ehealth Using the -z Argument Use the following guidelines when using the -z (lowercase) argument: -z lets you to choose on which machine the resource group is brought online. -z starts only those resources currently enabled within the group, ignoring the resources that have been disabled using the -n argument. How to Configure Multiple Instances of ehealth If you have more than two nodes in your HA cluster, you can configure more than one instance ehealth, creating a Distributed ehealth Cluster that is highly available. For example, if you have three nodes, you can run two instances of ehealth on two different primary nodes while one backup node sits in reserve, ready to take over active duties if either primary machine fails. Note: When multiple instances of ehealth are running in the same HA cluster, you must confirm that no two instances of ehealth are active on the same node at the same time. ehealth depends on certain global environment settings, therefore running multiple ehealth systems simultaneously on the same node will cause conflicts and crashes. Only one ehealth resource group should be run at a time on a single machine. You can control which resource groups (and therefore which instances of ehealth) are run on a particular system. You do this by configuring a strong negative affinity between each resource group using the RG_affinities property. For example, if you have three ehealth resource groups: rgroup-1, rgroup-2, and rgroup-3, the correct affinities could be configured using the following commands: # scrgadm -c -g rgroup-1 -y RG_affinities=--rgroup-2,--rgroup-3 # scrgadm -c -g rgroup-2 -y RG_affinities=--rgroup-1,--rgroup-3 # scrgadm -c -g rgroup-3 -y RG_affinities=--rgroup-1,--rgroup-2 Following the above example: rgroup-2 and rgroup-3 can never be online at the same time on the same machine as rgroup-1. rgroup-1 and rgroup-3 can never be online at the same time on the same machine as rgroup-2. rgroup-1 and rgroup-2 can never be online at the same time on the same machine as rgroup-3. Chapter 2: Sun Cluster High Availability Configuration 31

32 High Availability Cluster Management Tasks High Availability Cluster Management Tasks Use the following tasks to help manage ehealth in an HA Sun cluster. Install a Patch, Service Pack, or Upgrade After ehealth is fully integrated into the HA cluster you may have to run a patch or service pack installation program to update the ehealth and Oracle services. You must tell the clusterware that the system on which you want to install the patch will be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover. You use clusterware commands to achieve this functionality. Note: An alternate procedure to installing a patch or service pack is to run the nhhasetup -disable command to avoid resrouces being monitored during the upgrades and updates. To install a patch or service pack 1. Enter the following commands, including the name of the service as appropriate: > su root scswitch -n -j ehealthlicense scswitch -n -j ehealth scswitch -n -j ehealthrpt scswitch -n -j ehealthservlet scswitch -n -j ehealthtrap scswitch -n -j ehealthdb scswitch -n -j ehbdlistener All ehealth services are brought offline. 2. Start the ehealth server manually. Enter the following command: nhserver start 3. Start all other ehealth services. 4. Run the patch or service pack installer as you normally would on a standalone system. Note: After the patch or service patch completes installing, confirm that ehealth and all applications are working properly before restarting the HA service. 32 High Availability and Disaster Recovery Administration Guide

33 High Availability Cluster Management Tasks 5. Shut down any services started by the patch or service pack installation program. 6. Start the HA service again. See Step 10: Start the High Availability Service (see page 30). The patch or service pack is installed. Upgrade the Sun Cluster High Availability Package If you have already configured ehealth for HA with Sun Cluster software, this release of ehealth upgrades CAehealthHA, the Sun Cluster Solaris package that allows ehealth to be integrated and managed by the Sun clusterware. The new version of this package includes new command attributes, and clusterware and ehealth resources. Note: You must upgrade to ehealth r6.1 before you can upgrade CAehealthHA. To upgrade the CAehealthHA cluster package 1. Enter the following command to disable resource monitoring, including the name of the service as appropriate, for example: > su root # scswitch -M -n -j ehealth-license # scswitch -M -n -j ehealth # scswitch -M -n -j ehealth-web # scswitch -M -n -j ehealth-servlet # scswitch -M -n -j ehealth-trap 2. On the machine on which the resources are currently online, enter the following command to access and upgrade the Solaris package: > su root # cd $NH_HOME/sys # pkgadd -d CAehealthHA.pkg 3. Re-enable monitoring for each resource that you disabled by entering the following command, including the name of the service as appropriate: > su root # scswitch -M -e -j ehealth-license # scswitch -M -e -j ehealth # scswitch -M -e -j ehealth-web # scswitch -M -e -j ehealth-servlet # scswitch -M -e -j ehealth-trap Note: If you have multiple instances of ehealth installed in your HA environment, upgrade the Solaris package on each system. Chapter 2: Sun Cluster High Availability Configuration 33

34 High Availability Cluster Management Tasks Load the Database after High Availability Configuration After ehealth is fully integrated into the HA cluster you may need to load a saved ehealth database. Before you do this, the clusterware on the system on which you want to load the database must be shut down and the monitoring stopped. If you do not do this, the clusterware will continue to monitor the system, misread system and application failures, and induce failover. You use clusterware commands to achieve this functionality. To load the database 1. Shut down ehealth. As root, enter the following command, including the name of the service as appropriate. For example: scswitch -n -j ehealth-license scswitch -n -j ehealth scswitch -n -j ehealth-rpt scswitch -n -j ehealth-servlet scswitch -n -j ehealth-trap All ehealth services are brought offline. 2. Load the saved database. Note: For more information about the ehealth database and reasons for using the data restore feature, see the ehealth Administration Guide. When you start or stop servers in any ehealth database procedures on machines in an HA cluster environment, always shut down the clusterware and stop the monitoring first. 3. Start the ehealth server manually. Enter the following command: nhserver start 4. Start all other ehealth services. 5. Start the HA service again. See Step 10: Start the High Availability Service (see page 30). The database is loaded and HA service is started. Initiate a Planned Failover When you initiate a planned failover, you manually migrate ehealth resources from one system to another. 34 High Availability and Disaster Recovery Administration Guide