Automated Fail-Over Implementation of Control-M at First National Bank. Derick Pieterse - Blue Turtle Rhys Pride - First National Bank

Transcription

1 Automated Fail-Over Implementation of Control-M at First National Bank Derick Pieterse - Blue Turtle Rhys Pride - First National Bank

2 Infrastructure Components Machines in Different Physical Locations IBM Blades Operating System Red Hat Enterprise Linux Networking Disk Management Tools Storage - Hitachi Data Systems SAN

3 Machines in Different Locations

4 Networking Operating System Disk Management

5 Heartbeat Operating System Configure resources that we want to be highly available as well as any other operations that need to occur during failover process Heartbeat easily installed through rpms /etc/ha.d/ha.cf :... udpport 790 bcast bond0 auto_failback off node machinea node machineb /etc/ha.d/authkeys auth 3 3 md5 crypticpasswordhere /etc/ha.d/haresources : machinea truesplit em64prod cms64prod

6 Operating System Hitachi Truecopy Hitachi Open Remote Copy Manager (HORCM) Synchronous copy, each write on the primary volume is also performed on the secondary In normal operations the secondary volume is Read-Only, to the OS Installation and configuration is fairly simple, one config/instance per SAN controller ( 2 instances in our failover setup ) HORCM_MON #ip_address service poll(10ms) timeout(10ms) HORCM_CMD #dev_name dev_name dev_name /dev/mapper/command_disk_siteb HORCM_LDEV #dev_group dev_name Serial# CU:LDEV(LDEV#) MU# pri_truecopy pri_tc_disk :11 HORCM_INST #dev_group ip_address service pri_truecopy

7 Normal Operations

8 Failover Scenario

9 CONTROL-M/EM Failover Installation Create CONTROL-M/EM UNIX accounts Two user accounts were created, one on each node. Both users have identical names and identical user IDs (UID) Both users home directories should be the same The host name of the failover host are changed to have the same name and IP by scripts during failover Install CONTROL-M/EM server components Before starting the installation ensure to run the prerequisite check script(check_req) on each of the nodes Set the $BMC_HOST_INSTALL environment variable to the virtual network name designated for the CONTROL-M/EM Server. This will automatically configure CONTROL-M/EM and the TAO Naming Service to use this virtual network name. Install CONTROL-M/EM according to the instructions provided in the CONTROL-M Installation Guide. Install the latest fix pack to apply the most recent software updates. Managing CONTROL-M/EM server components A script is called during a failover scenario to first stop the CONTROL-M/EM server s components and then to start the CONTROL-M/EM server s components after the shared disk was moved: ## Stop Control-M/EM ## stop_config_agent -Uemuser -Ppassword stop_cms -Uemuser -Pempass stop_ns_daemon kill_gtw_proc -Uemuser -Ppassword stop_server -Usa -Ppassword ## Start Control-M/EM ## start_server sleep 5 start_ns_daemon start_cms -Uemuser -Ppassword start_config_agent -Uemuser -Ppassword

10 CONTROL-M/EM Failover Installation

11 CONTROL-M/Server Failover Installation Create CONTROL-M/Server UNIX accounts Create two user accounts, one on each node Both users must have identical names and identical user IDs (UID) Both users home directories should be the same The host name of the failover host are changed to have the same name and IP by scripts during failover Install CONTROL-M/Server Before starting the installation ensure to run the prerequisite check script (check_req) on each of the nodes. Install CONTROL-M/Server according to the instructions provided in the CONTROL-M Installation Guide. When prompted to provide the Local IP Host Interface Name, enter the virtual host name After the install also change the Local IP Host Interface Name to the virtual host name in ctm_menu. from ctm_menu select option 6 - Parameter Customization select option 1 - Communication and Operational Parameters These changes will set the OS_PRM_HOSTNAME parameter of the CONTROL-M/Server configuration file (${HOME}/ctm_server/data/config.dat), to the virtual host name as the parameter value. This change will not take effect until CONTROL-M/Server is shut down and restarted. Install the latest Fix Pack to apply the most recent software updates. Configure the fail-over Before failover start a script is used to stop CONTROL-M/Server, use the following commands shut_ca shut_ctm shutdb p password After the shared disk was moved the script uses the following commands to start CONTROL-M/Server startdb sleep 5 init_prflag start_ctm start_ca

12 CONTROL-M/Server Failover Installation

13 CONTROL-M/Agent Failover Installation Plan CONTROL-M/Agent configuration Each Agent should be configured to use the virtual host name for the communication with CONTROL-M/Server. As for the default, local Agent, when submitting jobs to it, the NODEID parameter value for the jobs should be the virtual host name. Create CONTROL-M/Agent UNIX accounts During the CONTROL-M/Server install, a default Agent is automatically install, but a separate Agent can be installed In case of a separate Agent create two user accounts, one on each node Both users must have identical names and identical user IDs (UID) Both users home directories should be the same The Agent s Logical name must be changed to the virtual host name: from ctmagcfg select option 7 - Advanced Parameters select option 4 - Logical Agent Name Install of separate CONTROL-M/Agent Install CONTROL-M/Agent on the relevant file system on the shared disk according to the instructions provided in the CONTROL-M Installation Guide. Install the latest Fix Pack to apply the most recent software updates. Run the CONTROL-M/Agent configuration utility (ctmagcfg) to configure the logical Agent name to be the virtual host name. In the configuration utility, select Logical Agent Name from the Advanced menu. In the CONTROL-M/Agent configuration menu, define the CONTROL-M/Server host name as authorized to submit jobs to this CONTROL-M/Agent. Only the virtual network name of the CONTROL-M/Server should be specified. Configure the fail-over Before failover the script use the following command (as root user) to stop CONTROL-M/Agent /controlm/cms64prod/ctm_agent/ctm/scripts/shut-ag -u cms64prod -p ALL After the shared disk was moved the script use the following command (as root user) to start CONTROL-M/Agent /controlm/cms64prod/ctm_agent/ctm/scripts/start-ag -u cms64prod -p ALL

14 CONTROL-M/Agent Failover Installation

15 Other Considerations Remote Hosts BMC Control-M agentless architecture is designed for high-availability configurations, in the FNB scenario it uses the CONTOL-M/Agent which have been setup to use the virtual host name and the shared disk The job owner s credentials (either user/passwords or SSH keys) are stored encrypted in the Control-M/Server database. The private keys are saved in the Control-M/Server database (encrypted) and the public keys are stored in files on the remote hosts SSH servers. Possible Job disappeared During our failover tests at FNB we did not have any instances of jobs going into a disappeared state, but below is the probable scenario explanation and optional workaround. In a failover scenario, while the job is still executing, the agent process is stopped and the agent file system is unmounted from the first host. In this case the job can keep running, but the procid file will not be updated when the job completes. Therefore, when the agent is started on the backup node, and the next AT track time arrives, it will find the original procid file but it will not find the actual process. This is why the job is marked as disappeared. You can define a JLOST ON statement for the jobs that run on the failover agent (Statement=*, Code=JLOST) and execute a DO RERUN command. Then the jobs will be automatically restarted (rerun) on the backup server when CONTROL- M/Server determines that they have disappeared. You must enter value greater than 0 in the MAX RERUN parameter in order for the job to be resubmitted.

16 Thank You Derick Pieterse - Blue Turtle Rhys Pride - First National Bank