Control-M for Hadoop. Technical Bulletin.

Transcription

1 Technical Bulletin Control-M for Hadoop Version September 30, 2014 Tracking number: PACBD BMC Software is announcing that Control-M for Hadoop now supports the following: Secured Hadoop cluster, implementing Kerberos authentication on a Linux 64bit computer (CAR ). Hive 2, which enables you to run Hive2 jobs (CAR ). This patch is accumulative and includes fixes from the following patches and : Support on Platform Linux CentOS 6.x 64-bit (x86_64) Verified on Framework version Apache Hadoop 2.x (Linux platforms only) and Hadoop distribution vendor Hortonworks Data Platform 2.x (HDP 2.x) The sqoop driver class should not be mandatory for using the JDBC connection string to connect to the database. For example, in the case of Teradata. Hadoop user interface plugins for Control-M Workload Change Manager and Control-M Self Service, which enables users to view and define Control-M for Hadoop jobs. NOTE: This Patch requires the following: Hadoop 2. Simple mode authentication is supported for both Hadoop 1 and Hadoop 2 APIs. In a secured Hadoop environment, the Agent User ID must be greater than If you have any questions, contact BMC Software Customer Support at (United States or Canada) or call your local support center.

2 Product distribution The following procedures describe how to obtain the installation files for this patch using EPD, FTP, or the BMC Sales Representative: Obtaining installation files through EPD on page 2 Obtaining installation files through FTP on page 2 Obtaining installation files through the BMC sales representative on page 2 Obtaining installation files through EPD This procedure describes how to obtain patch PACBD installation files through EPD. To obtain installation files via EPD: 1. Click and follow the instructions on the EPD site until you reach the Download Files page. 2. Download PACBD zip to a temporary directory. Obtaining installation files through FTP This procedure describes how to obtain the patch PACBD installation files through the FTP. To obtain installation files through FTP: Download the installation files to a temporary directory: ftp://ftp.bmc.com/pub/controlm/opensystem/bmc_control-m_for_hadoop/8.0.00/pacbd Obtaining installation files through the BMC sales representative This procedure describes how to obtain the patch PACBD installation files through the BMC sales representative. To obtain installation files through the BMC Sales representative: Contact your BMC Sales Representative. The list of representatives is available at Installation The following procedures describe how to install and uninstall patch PACBD : Installation on Control-M/EM on page 3. Installing patch PACBD on UNIX on page 4. Uninstalling PACBD from UNIX on page 5. Configuring Kerberos on Control-M for Hadoop on page 6. Configuring Hive2 connection details in the CCM on page 7. 2

3 Installation on Control-M/EM You must install this patch on the Control-M/EM client computer and Control-M/EM Server computer. These procedures must be performed on every computer that has Control-M/EM client or Control-M/EM Server components installed. The following procedures describe how to install the patch plug-in on Control-M/EM: Installing patch PACBD on Control-M/EM on UNIX on page 3. Installing patch PACBD on Control-M/EM on Windows on page 4. NOTE: If you have installed the GA version of Control-M/EM with fix pack 3 after you have installed this patch, you need to complete these procedures again. Installing patch PACBD on Control-M/EM on UNIX This procedure describes how to install patch PACBD plug-in on Control-M/EM on UNIX, which installs the CCM, Workload Change Manager, and self service plug-ins in Control-M/EM. To install on Control-M/EM: 1. Log in as the Control-M/EM user. 2. Navigate to a temporary directory and unzip the installation files. 3. Use one of the following modes: Display: If you install using GUI (or XServer), set your DISPLAY environment variable, according to your computer name. For csh or tcsh shell users: setenv DISPLAY <your computer name>:0.0 Console: It is not necessary to set DISPLAY. 4. From the temporary directory, activate the installation script with the following command: <temporary_directory>/setup.sh 5. Do one of the following: Interactive install: Continue with the on-screen instructions until the installation is complete. Automatic install: Do the following: a. Continue with the on-screen instructions until the Summary window. b. Click Generate and select the location to create the XML parameter file. c. Click Yes. d. Copy the automatic installation parameters file to a network location that is accessible to all computers where you want to perform an automatic installation. e. To run the installation script, type the following command: <source_path>/setup.sh -silent <filename.xml> The installation log can be found at the following location or in the %temp% directory: $HOME/BMCINSTALL/log/BMC_Control-M-CM_EM_Install_<date-time>.log 6. From the Control-M Configuration Manager, restart the Control-M GUI server. 3

4 The plug in is now installed on Control-M/EM. Installing patch PACBD on Control-M/EM on Windows This procedure describes how to install patch PACBD plug-in on Control-M/EM on Windows, which installs the CCM, Workload Change Manager, and self service plug-ins in Control-M/EM. To install on Control-M/EM: 1. Log in with a user ID with administrator privileges on the specific computer. 2. Navigate to a temporary directory and unzip the EM.zip installation file. From the temporary directory, activate the installation script with the following command: <temporary_directory>\setup.exe 3. Do one of the following: Interactive install: Continue with the on-screen instructions until the installation is complete. Automatic install: Do the following: a. Continue with the on-screen instructions until the Summary window. b. Click Generate and select the location to create the XML parameter file. c. Click Yes. d. Copy the automatic installation parameters file to a network location that is accessible to all computers where you want to perform an automatic installation. e. To run the installation script, type the following command: <source_path>\em\setup.exe -silent <filename.xml> The installation log can be found at the following location or in the %temp% directory: %temp%\bmc_control-m-cm_em_install_<date-time>.log 4. If the patch plug-in was installed on Control-M/EM Server, from the Control-M Configuration Manager, restart the Control-M GUI server. The plug-in is now installed. Installing patch PACBD on UNIX This procedure describes how to install patch PACBD on a UNIX computer. Before you begin Ensure that no Control-M jobs are running on the Control-M/Agent. NOTE: If Control-M/Agent was installed with user root, you must log in as user root every time you start up or shut down the Control-M/Agent throughout this procedure. To install patch PACBD on UNIX computer: 1. Log in as the Control-M/Agent user account. 2. Stop Control-M/Agent by typing the following command: shut-ag 4

5 3. Navigate to <agent root dir>/ctm/cm/hadoop/exe, and stop Control-M for Hadoop by typing the following command:./ctmhadoop_container stop 4. Create a temporary directory. For example: <agent root dir>/pacbd Extract the downloaded.tar file into the temporary directory by typing the following command: zcat PACBD _unix.tar.Z tar -xvf - 6. From the PACBD _unix directory, type the following command:./setup.sh Uninstalling PACBD from UNIX This procedure describes how to uninstall patch PACBD on a UNIX computer. Before you begin Ensure that no Control-M jobs are running on the Control-M/Agent. NOTE: If Control-M/Agent was installed with user root, you must log in as user root every time you start up or shut down the Control-M/Agent throughout this procedure To uninstall patch PACBD from UNIX: 1. Log in as the Control-M/Agent user account. 2. Stop Control-M/Agent by typing the following command: shut-ag 3. Navigate to <agent root dir>/ctm/cm/hadoop/exe, and stop Control-M for Hadoop by typing the following command:./ctmhadoop_container stop 4. Navigate to the following directory: <agent root dir>/ctm/patches/pacbd Type the following command that resides under the PACBD directory:./uninstall_pacbd sh The patch is uninstalled. 5

6 Configuring Kerberos on Control-M for Hadoop After you have installed Patch PACBD you need to configure Kerberos. To configure Kerberos on Control-M for Hadoop: 1. Log in as an admin user of Kerberos and create keytab for the Control-M/Agent user using the following script: <agent root dir>/ctm/cm/hadoop/exe/ctmhadoop_gen_keytab.sh <admin keytab> < admin principal> <realm> <keytab generated output dir> <CTM Agent principal> EXAMPLE: If the Realm is EXAMPLE.COM, the following script is used: <Agent home>/ctm/cm/exe/ctmhadoop_gen_keytab.sh /var/kerberos/krb5kdc/kadm5.keytab EXAMPLE.COM <agent root dir>/ctm/cm/hadoop/data <agent 2. Edit <agent root dir>/ctm/cm/hadoop/data/cm_container_config.xml file and add the following properties to the file: <KerberosKeytabPath>a path where the keytab file exists</kerberoskeytabpath> NOTE: The KerberosKeytabPath is the same as the keytab generated output dir parameter entered in step 1. <KerberosPrincipal>principal name of the CTM agent user</kerberosprincipal> <KerberosTicketLifeTimeMinutes>Kerberos ticket needed life time in minutes</kerberosticketlifetimeminutes> EXAMPLE: <KerberosKeytabPath> <agent root dir>/ctm/cm/hadoop/data </KerberosKeytabPath> <KerberosPrincipal><agent <KerberosTicketLifeTimeMinutes>300</KerberosTicketLifeTimeMinutes> (This means ticket renewal every 5 hours) 3. Complete the following steps if you want to impersonate another user (proxy user), other than the agent, in a Kerberos environment: a. In the target user s home directory, create a.k5login text file, which contains the source (the agent s) principal (for example, ctmagent@example.com). b. Log in as an admin user of Kerberos and create the keytab for the target user <target user root dir>/ctmhadoop.keytab. NOTE: You can use the Control-M provided script, which generates the file: <agent root dir>/ctm/cm/hadoop/exe/ctmhadoop_gen_keytab.sh <admin keytab> < admin principal> <realm> <target user home dir> <target user principal> 6

7 c. Once the keytab generates, change its ownership to the target user by typing the following chown command: chown <target user name>:<target user group> <target user root dir>/ctmhadoop.keytab d. The Control-M agent needs to be configured as a proxy user to perform actions as other users. Add to the core-site.xml file the following properties: <property> <name>hadoop.proxyuser.<control-m Agent user name>.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.<control-m Agent user name>.groups</name> <value>*</value> </property> EXAMPLE: In Cloudera, use cloudera manager to configure the hdfs service under the service-wide advanced section to add these properties. e. From the CCM in the Update Hadoop Connection Profile window, in the Run as user field enter the target user, which is used to connect. Configuring Hive2 connection details in the CCM The following procedure describes how to configure Hive2 connection details in the CCM. To configure Hive2 connection details in the CCM: 1. From Control-M Configuration Manager, select Control-M for Hadoop on the host that you want to manage and right-click Connection Profile Management. The Control-M for Hadoop - Connection Profile Management dialog box appears. 2. Select one of the following: To add a connection profile, click. To edit a connection profile, click. To copy a connection profile, click 7

8 For more information about the Connection profile management see, Connection Profile Management in the Control-M for Hadoop Administrator Guide. 3. Click on the Hive connection details tab to add the Hive connection paramaters (see screenshot below). Where to get the latest product information To view the latest BMC documents, see the Support Central website at Notices such as flashes, technical bulletins, and release notes are available on the website. You can subscribe to proactive alerts to receive messages when notices are issued or updated. For more information about proactive alerts, see the Support Central website. 8

9 Copyright 2014 BMC Software, Inc. BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners. IBM and z/os are trademarks of International Business Machines Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Office of Government Commerce and is used here by BMC Software, Inc., under license from and with the permission of OGC. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office, and is used here by BMC Software, Inc., under license from and with the permission of OGC. Linux is the registered trademark of Linus Torvalds. Oracle, Java, and Solaris are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. The information included in this documentation is the proprietary and confidential information of BMC Software, Inc., its affiliates, or licensors. Your use of this information is subject to the terms and conditions of the applicable End User License agreement for the product and to the proprietary and restricted rights notices included in the product documentation. BMC SOFTWARE INC 2101 CITYWEST BLVD, HOUSTON TX , USA Customer Support: (United States and Canada) or contact your local support center 9