SharePlex Connector for Hadoop 1.0. Installation and Setup Guide

Transcription

1 SharePlex Connector for Hadoop 1.0 Installation and Setup Guide

2 2013 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser s personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON- INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA legal@quest.com Refer to our Web site ( for regional and international office information. Trademarks Quest, Quest Software, the Quest Software logo and SharePlex are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Software s trademarks, please see Other trademarks and registered trademarks are property of their respective owners. SharePlex Connector for Hadoop Installation and Setup Guide Updated - July 2013 Software Version - 1.0

3 Table of Contents Welcome to SharePlex Connector for Hadoop 6 Getting Started 6 Maintaining Operations 6 Initial Setup 8 System Requirements 8 Java Development Kit 8 ActiveMQ 8 Download and Install 8 Start ActiveMQ 8 Start the SharePlex cop Process 9 Configure ActiveMQ to work with SharePlex 9 Set the total number of operations in the transaction 10 Additional - To replicate tables in Hive over HBase 10 SharePlex Connector for Hadoop 11 Unpack the Archive 11 Run install.sh 12 Run conn_setup.sh 14 Use Cases 16 Setup and Start Replication Start SharePlex for Oracle and sp_ctrl Define the Oracle tables to replicate Stop the post queue Run activate config SharePlex Connector for Hadoop - Run conn_snapshot.sh SharePlex for Oracle - Start the post queue Start SharePlex Connector for Hadoop 17

4 Table of Contents 4 Replication Paused or Data Inconsistent (Out of Sync) SharePlex for Oracle - Stop the post queue SharePlex Connector for Hadoop - Run conn_snapshot.sh SharePlex for Oracle - Start the post queue 18 Edit the list of tables being replicated SharePlex for Oracle - Copy the config file Edit the new config file Stop the post queue Activate the new config file SharePlex Connector for Hadoop - Run conn_snapshot.sh Start the post queue 20 Apache Derby Usage 20 How to install a new SharePlex Connector for Hadoop build without removing Apache Derby 20 How to start/stop the Derby Network Server 20 SharePlex Connector for Hadoop Command Reference 22 conn_setup.sh 22 conn_snapshot.sh 23 conn_ctrl.sh 26 conn_monitor.sh 26 conn_cleanup.sh 27 uninstall.sh 27 SharePlex Connector for Hadoop Log Files 29 How to configure the logs 29 Verifying the JMS Queue 31 Oracle Data Source 32 PK/UK supplemental logging 32

5 Table of Contents 5 Supported Data Types 32 Appendix A: Contact Quest 34 Contact Quest Support 34 Contact Quest Software 34 About Quest Software 34 Appendix B: Third Party Components 35 Index 39

6 1 Welcome to SharePlex Connector for Hadoop SharePlex Connector for Hadoop enables log-based replication of tables from Oracle to Hadoop (HDFS and HBase). Tables can further be replicated to Hive (Hive over HDFS and/or Hive over HBase). Getting Started 1. SharePlex Connector for Hadoop operates in conjunction with SharePlex for Oracle. Ensure SharePlex for Oracle is fully installed and operational in your Oracle environment. 2. SharePlex Connector for Hadoop replicates to Cloudera's distribution of Hadoop (CDH bit or CDH3u5 64-bit). Install Cloudera's distribution of Hadoop including libraries and configurations. Install and configure HBase (optional) to replicate tables to HBase and Hive over HBase. Install and configure Hive (optional) to replicate tables to Hive (Hive over HDFS or Hive over HBase). 3. Download and install Apache ActiveMQ. Tables are replicated through a JMS queue facilitated by Apache ActiveMQ. a. Install the Java Development Kit as required by Apache ActiveMQ. See "Java Development Kit" (page 8) for more information. b. Install ActiveMQ to operate with SharePlex. Start ActiveMQ, start the SharePlex cop process and set up the JMS queue. See "Download and Install" (page 8) for more information. 4. Install and configure SharePlex Connector for Hadoop. See "Unpack the Archive" (page 11) for more information. 5. Follow the use case to Setup and Start Replication (page 16) Maintaining Operations Now that SharePlex Connector for Hadoop is operational: SharePlex Connector for Hadoop Log Files Check the ALERT log messages. Actions may be required. See "SharePlex Connector for Hadoop Log Files" (page 29) for more information. Tip: Configure the logs to suit your environment.

7 Welcome to SharePlex Connector for Hadoop 7 conn_ ctrl.sh conn_ monitor.sh conn_ cleanup.sh uninstall.sh Control SharePlex Connector for Hadoop operations. See "conn_ctrl.sh" (page 26) for more information. Monitor SharePlex Connector for Hadoop operations. See "conn_monitor.sh" (page 26) for more information. Remove all data (HDFS, HBase, Hive) replicated from a specified table. See "conn_cleanup.sh" (page 27) for more information. Uninstall SharePlex Connector for Hadoop. See "uninstall.sh" (page 27) for more information. JMS Queue ActiveMQ Console SharePlex for Oracle Monitor activity in the JMS queue via the ActiveMQ Console / Web Interface. See "Download and Install" (page 8) for more information. From the sp_ctrl ()> prompt you can: Create additional JMS queues and show the configuration of the JMS queue. See "Download and Install" (page 8) for more information. Check the status of the queue and messages in the queue. See "Verifying the JMS Queue" (page 31) for more information.

8 2 Initial Setup System Requirements SharePlex Connector for Hadoop operates in conjunction with a number of third-party products. Ensure these products are installed and operational as documented in the SharePlex Connector for Hadoop Release Notes. Java Development Kit Download and Install the Java Development Kit or above if you have not already done so. The Java Development Kit is required on the computer where SharePlex Connector for Hadoop is to be installed. It is also required by AcitveMQ. Ensure the JAVA_HOME environment variable is set. Variable JAVA_HOME Example: export JAVA_HOME=/usr/java/jdk1.6.0_26 ActiveMQ Download and Install Download ActiveMQ from to the computer where SharePlex for Oracle is installed. In the lib/providers directory in the SharePlex for Oracle product directory, create the directory activemq and extract the ActiveMQ download to this location. $tar -xvfz apache-activemq-version-bin.tar.gz Copy the ActiveMQ JAR file and dependencies from providers/activemq/lib to providers/activemq/ Start ActiveMQ Change to the ActiveMQ bin directory lib/providers/activemq/bin/ and start ActiveMQ:./activemq start

9 Initial Setup 9 Open the web console to verify ActiveMQ is running ok: with the username and password admin/admin. Tip: To run ActiveMQ on a computer other than the one SharePlex for Oracle is installed: Download and install ActiveMQ as per the instructions above. Additionally, install ActiveMQ on a second computer. Run ActiveMQ from the second computer. The second computer requires network access to SharePlex for Oracle and SharePlex Connector for Hadoop. Start the SharePlex cop Process From the SharePlex for Oracle shareplex/prod/bin directory start the sp_cop process. /u01/app/shareplex/prod/bin >./sp_cop & Leave this session open. Configure ActiveMQ to work with SharePlex Set the environment parameters for the JNDI environment as follows. For more information on JNDI and ActiveMQ see Start a SharePlex sp_ctrl session /u01/app/shareplex/prod/bin >./sp_ctrl sp_ctrl ()> Set the Factory Class sp_ctrl ()> target x.jms queue PostQueueName set jms factory_class = org.apache.activemq.jndi.activemqinitialcontextfactory This creates the SharePlex post queue (name PostQueueName) and sets the JNDI property java.naming.factory.initial. Declare the ActiveMQ URL sp_ctrl ()> target x.jms queue PostQueueName set jms provider_url = url This sets the JNDI property java.naming.provider.url. Set it to the host name or IP address of the host running ActiveMQ. Example URLs are tcp://w2k3-64bit:61616 or tcp://localhost:61616 Declare the location of the ActiveMQ JAR files sp_ctrl ()> target x.jms queue PostQueueName set jms lib_location = activemq Set lib_location to the directory in the SharePlex /lib/providers directory where you copied the ActiveMQ JAR files. The above example assumes the JAR files were copied to the SharePlex /lib/providers/activemq directory. The JMS Queue name By default the JMS queue name is OpenTarget.

10 Initial Setup 10 You can create an additional JMS queue with the following command. The JMS queue name in the example is JMSQueueName. sp_ctrl> target x.jms queue PostQueueName set jms queuename = JMSQueueName NOTE: The following JMS queue names are reserved for internal use by SharePlex Connector for Hadoop. Do not use them. They are: Command_Queue, OpenTargetHBaseQueue and OpenTargetHDFSQueue. Further Commands Command sp_ctrl ()> target jms show sp_ctrl ()> help target jms Show the current configuration of the JMS. Ask for help on target jms. This shows all the options that can be configured for JMS and their usage. Set the total number of operations in the transaction Execute the following SharePlex for Oracle command. SharePlex Connector for Hadoop will use the information from this command to ensure the integrity of the data inside of checking for duplicate and missing messages. sp_ctrl ()> target jms set metadata size Tip: You can confirm the setting by running the target jms show command which will show size as a metadata parameter set on the queue. Refer to the SharePlex for Oracle documentation for more information. Additional - To replicate tables in Hive over HBase If you intend to replicate tables in Hive over HBase then complete the following additional setup steps. Ensure the Zookeeper Quorum and client port are configured in $HIVE_ HOME/conf/hive-site.xml. <property> <name>hbase.zookeeper.quorum</name> <value> ---- PLEASE SPECIFY ---- </value> <description>a comma separated list (with no spaces) of the IP addresses of all ZooKeeper servers in the cluster.</description> </property> <property>

11 Initial Setup 11 <name>hbase.zookeeper.property.clientport</name> <value> ---- PLEASE SPECIFY ---- </value> <description>the Zookeeper client port. Default clientport is 2181.</description> </property> Use --auxpath while entering into Hive hive --auxpath <HIVE_HOME>/lib/hive-hbase-handler-<version>.jar:<HBASE_ HOME>/hbase.jar:<HBASE_HOME>/lib/zookeeper.jar:<HBASE_HOME>/lib/guava- <version>.jar When using CDH4.2 with Kerberos authentication Export the HIVE_OPTS environment variable with Kerberos parameters as shown below to replicate Hive over HBase. export HIVE_OPTS="-hiveconf hbase.security.authentication=kerberos -hiveconf hbase.rpc.engine=org.apache.hadoop.hbase.ipc.securerpcengine -hiveconf -hiveconf -hiveconf hbase.zookeeper.quorum=zookeeperquorum" Tip: The Kerberos utility kinit allows you to identify yourself to the Kerberos server. kinit needs to be invoked (only once) if you are starting a new session. SharePlex Connector for Hadoop Unpack the Archive SharePlex Connector for Hadoop is distributed in the archive: shareplex-hadoop-connectorversion-date-clouderaversion.tar.gz where version identifies the SharePlex Connector for Hadoop release. Extract the archive with clouderaversion appropriate to your Hadoop installation on a machine where Hadoop libraries and configurations are present. Use the command: $tar -xzf shareplex-hadoop-connector-version-date-clouderaversion.tar.gz The archive contains the following files: File install.sh shareplex-hadoop- connector-version- A shell script that installs SharePlex Connector for Hadoop and other programs in the archive. SharePlex Connector for Hadoop archive.

12 Initial Setup 12 File date.tar db-derby bin.tar.gz sqoop-quest bin hadoopversion.tar.gz quest-oraoop date-version.tar.gz Apache Derby installable. SharePlex Connector for Hadoop uses the Apache Derby network server and creates a database for storing metadata and status information. Apache Sqoop installable. Sqoop is a tool designed to transfer bulk data between Apache Hadoop and structured data stores. Quest Data Connector for Oracle and Hadoop archive. Quest Data Connector for Oracle and Hadoop is an optional plugin to Sqoop. It facilitates the movement of data between Oracle and Hadoop. Run install.sh This shell script installs programs in the SharePlex Connector for Hadoop archive. Shell Script Usage [user@host bin]$./install.sh -d <INSTALL_DIR> [-h <HADOOP_HOME_DIR>] [-c <HADOOP_CONF_DIR>] [-b <HBASE_HOME_DIR>] [-v <HIVE_HOME_DIR>] [-p <DERBY_ PORT_NUMBER>] [--help] [--version] Options Parameter -d <INSTALL_ DIR> -h <HADOOP_ HOME_ DIR> -c <HADOOP_ CONF_ DIR> SharePlex Connector for Hadoop will be installed in the shareplex_ hadoop_connector directory inside this directory. The path to the Hadoop home directory. NOTE: This option overrides HADOOP_HOME in the environment. If this option is not set and the HADOOP_HOME environment variable is also not set, this parameter is set to /usr/lib/hadoop as default. The path to the Hadoop conf directory. NOTE: This option overrides HADOOP_CONF_DIR in the environment. If this option is not set and the HADOOP_CONF_DIR environment variable is also not set, this parameter is set to HADOOP_HOME/conf folder (for CDH3) HADOOP_HOME/etc/hadoop (for CDH4).

13 Initial Setup 13 Parameter -b <HBASE_ HOME_ DIR> -v <HIVE_ HOME_ DIR> -p <DERBY_ PORT_ NUMBER> --help --version The path to HBase home directory. NOTE: This option overrides HBASE_HOME in the environment. If this option is not set and the HBASE_HOME environment variable is also not set, this parameter is set relative to HADOOP_HOME. The path to Hive home directory. NOTE: This option overrides HIVE_HOME in the environment. The port number for the Apache Derby connection. NOTE: If not specified, this parameter is set to 1527 Show this help and exit. Show version information and exit. While install.sh is executing Please accept the Apache license to install Quest Data Connector for Oracle and Hadoop. When install.sh has finished executing install.sh starts the Apache Derby network server. About the new shareplex_hadoop_connector directory Files and Directories bin conf db-derbyversion-bin lib logs shareplex_ hadoop_ SharePlex Connector for Hadoop shell scripts as documented in this guide. In addition shareplex_hadoop_env.sh is used by SharePlex Connector for Hadoop. You can set the environment variables by executing source shareplex_hadoop_env.sh SharePlex Connector for Hadoop configuration files. Apache Derby application. SharePlex Connector for Hadoop required dependencies. SharePlex Connector for Hadoop log files. The Java archive file containing SharePlex Connector for Hadoop application code.

14 Initial Setup 14 Files and Directories connector.jar oraoopversion sqoop-questversion.bin Quest Data Connector for Oracle and Hadoop application Apache Sqoop application Run conn_setup.sh Run this script to setup SharePlex Connector for Hadoop and provide the necessary configuration details. This is usually a one time activity. Shell Script Usage [user@host bin]$./conn_setup.sh Tip: See conn_setup.sh (page 22) for a complete description of this command. Configuration Parameters The script will prompt you to respond to all configuration parameters, one-by-one. NOTE: Default values are provided within brackets. Press Enter to select the default value. Categories of detail HDFS HBase HBase parameters HDFS parameters You will be prompted to provide the following details... Do you want to enable Hadoop connector to copy data to HDFS? Answer yes to this question if you intend to replicate all (or most) tables by HDFS Near Real Time Replication. Do you want to enable Hadoop connector to copy data to HBase? Answer yes to this question if HBase is setup in your environment and you intend to replicate all (or most) tables by HBase Real Time Replication. You will be prompted for this detail if you have responded YES to replicate tables to HBase. The HBase column family name You will be prompted for this detail if you have responded YES to replicate tables to HDFS. The HDFS destination directory. You should consider this directory as used exclusively by SharePlex Connector for Hadoop. This directory may be cleaned up by conn_cleanup.sh and uninstall.sh.

15 Initial Setup 15 Categories of detail You will be prompted to provide the following details... How often do you want to copy data to HDFS? This is measured by time and number of changes. The first question relates to time (in minutes). If you say 10 minutes for example then the table will be replicated every 10 minutes. You should not set this to under 10 minutes. The second question relates to the number of changes. If you say 2 then replication is executed following 2 changes to the table. Replication is executed on the first condition met: on the given number of changes to the table or the set time period, whichever comes first. JMS parameters Oracle parameters The name of the JMS queue. For example: OpenTarget The name of the host running ActiveMQ. For example: localhost The port number used by the JNDI provider.url property. For example: The port number used to access the ActiveMQ admin web site. By default: 8161 Tip: For more information on each of these properties see Configure ActiveMQ to work with SharePlex (page 9) Host name (or TCP/IP address of the Oracle server) The port to connect to the Oracle server, By default: 1521 Oracle instance (SID) Oracle username NOTE: You will be prompted to enter the Oracle password when taking a snapshot. Refer to the use cases for more information. Configuration Complete SharePlex Connector for Hadoop shows this message. It is ready for use. connectorconfiguration.xml updated successfully. JMSConfiguration.xml updated successfully. OraOopConfiguration.xml updated successfully. Connector setup completed successfully.

16 3 Use Cases Setup and Start Replication NOTE: Ensure you complete all Initial Setup instructions first. 1. Start SharePlex for Oracle and sp_ctrl Ensure SharePlex for Oracle and sp_ctrl are running. The prompt should be sp_ctrl (host:port)>. Refer to the SharePlex for Oracle documentation for more information. /u01/app/shareplex/prod/bin >./sp_ctrl sp_ctrl ()> 2. Define the Oracle tables to replicate Use the SharePlex for Oracle create config command to create the file ConfigFile. The file is opened in vi. Declare all the Oracle tables you want captured into Hadoop, one table per line. sp_ctrl ()> create config ConfigFile ####################### datasource: O.OracleSID OracleSchema.OracleTable!jms[:TargetSchema.TargetTable] IPHostPostQueue [:PostQueueName] ##################### Example line: soo70.g_authors!jms :q2 IPHostPostQueue is the name or IP address of the host on which the SharePlex post queue is running. For more information on PostQueueName see Configure ActiveMQ to work with SharePlex (page 9). Tip: To verify there are no errors in the config file run command sp_ctrl ()> verify config ConfigFile 3. Stop the post queue SharePlex for Oracle uses the post queue to send messages to the JMS queue. sp_ctrl ()> stop post queue PostQueueName

17 Use Cases Run activate config Use the SharePlex for Oracle activate config command to activate the file ConfigFile. sp_ctrl ()> activate config ConfigFile Tip: If you see the error "minimal supplemental logging should be enabled" then See "Oracle Data Source" (page 32) for more information. 5. SharePlex Connector for Hadoop - Run conn_snapshot.sh Execute the SharePlex Connector for Hadoop conn_snapshot.sh script for each Oracle table in the ConfigFile. This makes a copy of each Oracle table to replicate. NOTE: Take a snapshot of all the tables defined in the config file before you start the post queue. Once you start the post queue, SharePlex Connector for Hadoop will ignore messages for those tables that have not had a snapshot taken. The conn_snapshot.sh script is fully customizable. It is fully documented in conn_snapshot.sh (page 23).» conn_snapshot.sh -t Schema.Table -s ';' NOTE: You will be prompted to enter the Oracle password. This is the password to the Oracle username supplied during configuration. See "Run conn_setup.sh" (page 14) for more information. 6. SharePlex for Oracle - Start the post queue Start the post queue so SharePlex for Oracle can send messages from the post queue to the JMS queue.» sp_ctrl ()> start post queue PostQueueName 7. Start SharePlex Connector for Hadoop Return to SharePlex Connector for Hadoop. For more on the conn_ctrl.sh command see conn_ ctrl.sh (page 26).» conn_ctrl.sh start Replication Paused or Data Inconsistent (Out of Sync) SharePlex Connector for Hadoop compares the values stored in HBase / HDFS with the lookup values received from SharePlex. If they don't match then data inconsistency is reported on the console and in the shareplex-connector-alert.log.

18 Use Cases 18 REPLICATION PAUSED Data Inconsistent (Out of Sync) - Take a fresh snapshot Scenarios that may lead to data inconsistency where SharePlex Connector for Hadoop pauses replication of a table include: For HDFS Near Real Time replication the entire merging job at Hadoop fails. Replication of the table is paused as there will be inconsistencies going ahead. Following changes to the schema (Alter). Scenarios that may lead to data inconsistency include: receiving a delete for a row which does not exist, receiving an update on a deleted row, receiving an insert on an already inserted row. SharePlex Connector for Hadoop suggests you take a new snapshot of the table Follow these instructions. 1. SharePlex for Oracle - Stop the post queue SharePlex for Oracle uses the post queue to send messages to the JMS queue. Stop the post queue before you take a snapshot. Tip: Enter command /u01/app/shareplex/prod/bin >./sp_ctrl to open the sp_ ctrl ()> prompt.» sp_ctrl ()> stop post queue PostQueueName For more information on PostQueueName see Configure ActiveMQ to work with SharePlex (page 9). 2. SharePlex Connector for Hadoop - Run conn_snapshot.sh Execute the SharePlex Connector for Hadoop conn_snapshot.sh script. In SharePlex Connector for Hadoop, run the conn_snapshot.sh script for the Oracle table associated with the REPLICATION PAUSED or DATA_INCONSISTENT message. This makes a fresh copy of that Oracle table. This script is fully documented in conn_snapshot.sh (page 23).» conn_snapshot.sh -t Schema.Table -s ';' NOTE: You will be prompted to enter the Oracle password. This is the password to the Oracle username supplied during configuration. See "Run conn_setup.sh" (page 14) for more information. 3. SharePlex for Oracle - Start the post queue Return to SharePlex for Oracle. Start the post queue so SharePlex for Oracle can send messages from the post queue to the JMS queue.» sp_ctrl ()> start post queue PostQueueName

19 Use Cases 19 Edit the list of tables being replicated SharePlex Connector for Hadoop replicates the tables listed in the file ConfigFile. Follow these steps to edit this list of tables. 1. SharePlex for Oracle - Copy the config file Use the SharePlex for Oracle copy config command to make a copy of ConfigFile. Tip: Enter command /u01/app/shareplex/prod/bin >./sp_ctrl to open the sp_ ctrl ()> prompt.» sp_ctrl ()> copy config ConfigFile to NewConfigFile 2. Edit the new config file Use the edit config command to edit the file NewConfigFile. The file is opened in vi. List the Oracle table(s) you want captured into Hadoop, one table per line.» sp_ctrl ()> edit config NewConfigFile ####################### datasource: O.OracleSID OracleSchema.OracleTable!jms[:TargetSchema.TargetTable] IPHostPostQueue [:PostQueueName] ##################### Example line: soo70.g_authors!jms :q2 IPHostPostQueue is the name or IP address of the host on which the SharePlex post queue is running. For more information on PostQueueName see Configure ActiveMQ to work with SharePlex (page 9). Tip: To verify there are no errors in the config file run command sp_ctrl ()> verify config NewConfigFile 3. Stop the post queue SharePlex for Oracle uses the post queue to send messages to the JMS queue. Stop the post queue.» sp_ctrl ()> stop post queue PostQueueName 4. Activate the new config file Activate the new config file.

20 Use Cases 20» sp_ctrl ()> activate config NewConfigFile 5. SharePlex Connector for Hadoop - Run conn_snapshot.sh Execute the SharePlex Connector for Hadoop conn_snapshot.sh script for each NEW Oracle table added to the list of tables in the config file. This makes a copy of the Oracle table for replication. The conn_snapshot.sh script is fully customizable. It is fully documented in conn_snapshot.sh (page 23).» conn_snapshot.sh -t Schema.Table -s ';' NOTE: You will be prompted to enter the Oracle password. This is the password to the Oracle username supplied during configuration. See "Run conn_setup.sh" (page 14) for more information. 6. Start the post queue Start the post queue so SharePlex for Oracle can send messages from the post queue to the JMS queue.» sp_ctrl ()> start post queue PostQueueName Apache Derby Usage How to install a new SharePlex Connector for Hadoop build without removing Apache Derby After SharePlex Connector for Hadoop is installed, to reinstall SharePlex Connector for Hadoop without losing the data in Apache Derby: 1. Backup the connector-derby-database folder in SharePlex Connector for Hadoop home. 2. Install SharePlex Connector for Hadoop. 3. Copy the connector-derby-database folder (from step 1) into the new SharePlex Connector for Hadoop home directory. How to start/stop the Derby Network Server SharePlex Connector for Hadoop installs Apache Derby and starts the Derby Network Server. First: Set the DERBY_HOME environment variable. Change the current working directory to the SharePlex Connector for Hadoop home, bin and execute $ source shareplex_ hadoop_env.sh To start Derby execute $ java -jar $DERBY_HOME/lib/derbynet.jar start p <PORT_NUM> &

21 Use Cases 21 To stop Derby execute $ java -jar $DERBY_HOME/lib/derbynet.jar shutdown p <PORT_NUM> Ensure you enter PORT_NUM as it was set when you ran install.sh. The default is To look up the PORT_NUM you set, in SharePlex Connector for Hadoop home, conf open the configuration file (by default connectorconfiguration.xml) and lookup the port number entry <entry key="derbyport">1527</entry>

22 4 SharePlex Connector for Hadoop Command Reference conn_setup.sh Use this script to setup SharePlex Connector for Hadoop and provide the necessary configuration details. Setup is usually a one time activity. This script is similar to ora_setup of SharePlex for Oracle. Shell Script Usage [user@host bin]$./conn_setup.sh [-c <INPUT_FILE>] [-r] [-n] [--help] [- -version] Options Parameter -c <INPUT_ FILE> Provide an input file that defines the configuration parameters with their values. Each configuration parameter should be specified on a new line of the input file. Format - Key=Value. Example enablerealtimecopy=true Tip: You may like to use this parameter to modify your original configuration. When conn_setup.sh is run without parameters the file it creates is conf/connectorconfiguration.xml There is an example input file located at conf/conn_setup_ template.properties -r Use to update the configuration Add/Update parameters for HBase Real Time Replication: the HBase column family name. Tables will now replicate to HBase Real Time Replication unless otherwise specified in conn_snapshot.sh -n Use to update the configuration Add/Update parameters for HDFS Near Real Time Replication: add the HDFS destination directory and how often do you want to copy data to HDFS? This is measured by time and number of changes. Tables will now replicate to HDFS Near Real Time Replication unless otherwise specified in conn_snapshot.sh

23 SharePlex Connector for Hadoop Command Reference 23 Parameter --help --version Show this help and exit. Show version information and exit. Configuration Parameters If conn_setup.sh is run without an input file then you will be prompted to supply a value to each of the configuration parameters. See "Run conn_setup.sh" (page 14) for more information. NOTE: Default values are provided within brackets. Press Enter to select the default value. conn_snapshot.sh Use this script to take a fresh copy of an Oracle table for replication. Shell Script Usage conn_snapshot.sh -t <TABLE_OWNER.TABLE_NAME> [-f <FILE_TYPE>] [-s <FIELD_ SEPARATOR>] [-e <CREATE_EXTERNAL_TABLE>] [-h <HIVE_HOME_DIR>] [-m <NUM_OF_ MAPPERS>] [-n <CHANGES_THRESHOLD>] [-i <TIME_THRESHOLD>] [-r] [-d] [-v] [-- help] [--version] Options Parameter -t <TABLE_ OWNER.TABLE_ NAME> -f <FILE_ TYPE> Name and owner of the table to import from Oracle. Required. Applicable to HDFS Near Real Time Replication. File type for import. [Text Sequence] (Default = Text. Use -f Sequence for sequence file type.) -f Sequence All data is replicated in the Sequence files on the HDFS. To read/write to the Sequence files you need access to the Writable classes used to generate the Sequence files. See the SharePlex Connector for Hadoop lib/sqoop-records directory and SharePlex Connector for Hadoop lib/sqooprecords.jar. NOTE: SharePlex Connector for Hadoop does not copy to Hive over HDFS for Sequence file type. -s <FIELD_ SEPARATOR> The separator between each field/column. The separator must be enclosed in single quotes.

24 SharePlex Connector for Hadoop Command Reference 24 Parameter -e <CREATE_ EXTERNAL_ TABLE> -h <HIVE_ HOME_DIR> -m <NUM_OF_ MAPPERS> -n <CHANGES_ THRESHOLD> -i <TIME_ THRESHOLD> Copy to Hive. [true false] (Default = false. Use -e true to enable.) If true then Copy to Hive over HDFS if a copy is taken for HDFS Near Real Time Replication. Copy to Hive over HBase if a copy is taken for HBase Real Time Replication. Path to the Hive home directory. If not specified the value of the HIVE_HOME environment variable is used. If this option is not set and the HIVE_HOME environment variable is also not set, this parameter will be set as relative to HADOOP_HOME. The number of mappers to be used. Use to override the default setting for how often SharePlex Connector for Hadoop replicates the table (measured by the number of changes to the table). NOTES: Applicable to HDFS Near Real Time Replication. The default setting was set in conn_setup.sh. Replication is executed on the first condition met: on the given number of changes to the table or the set time period -i, whichever comes first. SharePlex Connector for Hadoop remembers this setting and makes use of it during further execution of conn_snapshot. Use to override the default setting for how often SharePlex Connector for Hadoop replicates the table (measured by the number of minutes). NOTES: Applicable to HDFS Near Real Time Replication. The default setting was set in conn_setup.sh. Replication is executed on the first condition met: on the given number of changes to the table -n or the set time period, whichever comes first. SharePlex Connector for Hadoop remembers this setting and makes use of it during further execution of conn_snapshot. -r -d Use to override the settings in conn_setup.sh. If not specified, replicate this Oracle table using HBASE and/or HDFS as per the settings in conn_ setup.sh.

25 SharePlex Connector for Hadoop Command Reference 25 Parameter -r A copy of the table is taken for HBase Real Time Replication. Do not replicate this Oracle table using HDFS. This overrides the settings in conn_setup.sh. -d A copy of the table is taken for HDFS Near Real Time Replication. Do not replicate this Oracle table using HBASE. This overrides the settings in conn_setup.sh. -r -d A copy of the table is taken for HBase Real Time Replication and HDFS Near Real Time Replication. This overrides the settings in conn_setup.sh. NOTE: SharePlex Connector for Hadoop remembers these settings and makes use of them during further execution of conn_snapshot. -v Verbose - Show detailed information of each step. --help --version Show this help and exit. Show version information and exit. Example [user@host bin]$./conn_snapshot.sh -t Schema.Table -s ';' Use Cases Take a copy of the Oracle table Shema.Table for replication over HDFS and / or HBase as per the settings in conn_setup.sh As above and... Copy to Hive over HDFS if a copy is taken for HDFS Near Real Time Replication. Copy to Hive over HBase if a copy is taken for HBase Real Time Replication. Take a copy of the Oracle table Shema.Table for replication over HBase. Do not replicate over HDFS. Take a copy of the Oracle table Shema.Table for replication over HDFS. Do not replicate over HBase. Replication is set for every 20 minutes or 100 changes - whichever comes first. conn_snapshot.sh - t Schema.Table -s ';' conn_snapshot.sh - t Schema.Table -s ';' -e true conn_snapshot.sh - t Schema.Table -s ';' -r conn_snapshot.sh - t Schema.Table -s ';' d i 20 n 100

26 SharePlex Connector for Hadoop Command Reference 26 conn_ctrl.sh Use this script to start / restart / shutdown SharePlex Connector for Hadoop and reload SharePlex Connector for Hadoop configurations. Shell Script Usage conn_ctrl.sh [start restart shutdown reload-conf --help --version] Options Parameter start restart shutdown reloadconf --help --version Start SharePlex Connector for Hadoop. Stop and start SharePlex Connector for Hadoop. Stop SharePlex Connector for Hadoop. Reload some of the SharePlex Connector for Hadoop configurations without restarting SharePlex Connector for Hadoop. These configurations include HDFS Near Real Time Replication changes and time interval. Show this help and exit. Show version information and exit. Example [user@host bin]$./conn_ctrl.sh conn_monitor.sh Use this script to show a list of the tables being replicated. Monitor the status of a table under replication. Shell Script Usage conn_monitor.sh [--summary --status <TABLE_OWNER.TABLE_NAME> [--hdfs --hbase] --help --version] Options Parameter --summary --status <TABLE_ Show a summary of the tables under replication. Show the status of the given table.

27 SharePlex Connector for Hadoop Command Reference 27 Parameter OWNER.TABLE_NAME> [-- hdfs --hbase] --help --version Include the --hdfs option (optional) to limit the display to HDFS Near Real Time Replication. Include the --hbase option (optional) to limit the display to HBase Real Time Replication. Show this help and exit. Show version information and exit. conn_cleanup.sh Use this script to delete all data (HDFS, HBase and Hive) from a table under replication by SharePlex Connector for Hadoop. Shell Script Usage conn_cleanup.sh -t <TABLE_OWNER.TABLE_NAME> [-h <HIVE_HOME_DIR>] [--help] [--version] Options Parameter -t <TABLE_ OWNER.TABLE_ NAME> -h <HIVE_ HOME_DIR> --help --version Name and owner of the table data to cleanup. Path to the Hive home directory. If not specified the value of the HIVE_HOME environment variable is used. If this option is not set and HIVE_HOME environment variable is also not set, this parameter will be set as relative to HADOOP_HOME. Show this help and exit. Show version information and exit. Example [user@host bin]$./conn_cleanup.sh -t Schema.Table uninstall.sh Use this script to uninstall SharePlex Connector for Hadoop.

28 SharePlex Connector for Hadoop Command Reference 28 Tip: It is recommended to run the uninstall script from outside of the SharePlex Connector for Hadoop home directory. Shell Script Usage uninstall.sh [-q] [-h >HIVE_HOME_DIR>] [-t <LIST_OF_TABLES> -a] [-- help] [--version] Options Parameter -q Delete the point-to-point JMS Queue to which SharePlex Connector for Hadoop listens. -h <HIVE_ HOME_DIR> -t <LIST_ OF_ TABLES> Path to the Hive home directory. This parameter is used in conjunction with -t or -a when tables in Hive are involved in the cleanup. If not specified the value of the HIVE_HOME environment variable is used. If this option is not set and HIVE_HOME environment variable is also not set, this parameter will be set as relative to HADOOP_HOME. Delete all data (HDFS, HBase and Hive) from the list of tables under replication by SharePlex Connector for Hadoop. Use a comma to delimit the list of tables. NOTE: This option uses conn_cleanup.sh. -a Delete all data (HDFS, HBase and Hive) from all tables under replication by SharePlex Connector for Hadoop. NOTE: This option uses conn_cleanup.sh. --help --version Show this help and exit. Show version information and exit. Example [user@host bin]$./uninstall.sh

29 5 SharePlex Connector for Hadoop Log Files SharePlex Connector for Hadoop uses the Apache log4j library as a logging utility. Logs are created in the SharePlex Connector for Hadoop home, logs. Log File shareplexconnector.log shareplexconnectoralert.log shareplexconnectoricm.log Contains all log messages logged with levels DEBUG and above. Contains ALERT log messages which may require action, like: Data Inconsistency observed while replication Replication for a table failed Schema of table changed (alter) High priority messages and un-handled critical exceptions. Contains all installation, configuration and management related log messages from the SharePlex Connector for Hadoop shell scripts. For more information on these scripts see the section SharePlex Connector for Hadoop Command Reference. Note: This log file does not use the Apache log4j library as a logging utility. How to configure the logs The log4j configuration file is SharePlex Connector for Hadoop home, conf\opentarget-log4j.xml To change the size of the log file The following parameters set for 10 files each of 20MB maximum size. <param name="maxfilesize" value="20mb" /> <param name="maxbackupindex" value="10" /> To change the log level The default log level is DEBUG. Edit this parameter. <logger name="com.quest.shareplex"> <level value="debug" />

30 SharePlex Connector for Hadoop Log Files 30 Tip: To apply changes to log4j you need to restart SharePlex Connector for Hadoop. See "conn_ ctrl.sh" (page 26) for more information.

31 6 Verifying the JMS Queue Show the configuration of the JMS queue Command sp_ctrl ()> target jms show sp_ctrl ()> help target jms Show the current configuration of the JMS. Ask for help on target jms. This shows all the options that can be configured for JMS and their usage. Check if all the processes are running Command sp_ctrl()> show sp_ctrl()> status sp_ctrl()> qstatus Provide status information on the JMS queue. For example, when you run the show or status command if the Post Process is Pending then sp_cop may have experienced a problem starting the JMS bridge server. The bridge service log file can be found in the SharePlex for Oracle install directory var/log/openbridge.log When you run the qstatus command if there are messages in the backlog of the post queue then those messages have not been delivered to SharePlex Connector for Hadoop. Perhaps SharePlex Connector for Hadoop has stopped. Post process writing to the JMS queue: Command sp_ctrl()> stop post Use these commands to monitor the post process. sp_ctrl()> start post sp_ctrl()> show post

32 7 Oracle Data Source PK/UK supplemental logging Ensure PK/UK supplemental logging is enabled on the Oracle database. If you see the error "minimal supplemental logging should be enabled" then check the status of the supplemental_ log_data on the Oracle database and enable using the following SQL query: SQL> select supplemental_log_data_min from v$database; SUPPLEME NO SQL> alter database add supplemental log data; Database altered. Supported Data Types SharePlex Connector for Hadoop supports the following data types: Oracle Type CHAR DATE FLOAT NCHAR NUMBER NVARCHAR2 ROWID VARCHAR2 INTERVAL YEAR TO MONTH XML Type string datetime decimal string decimal string string string yearmonthduration

33 Oracle Data Source 33 Oracle Type INTERVAL DAY TO SECOND TIMESTAMP TIMESTAMP WITH TIME ZONE TIMESTAMP WITH LOCAL TIME ZONE XML Type daytimeduration datetime datetimestamp datetimestamp Tip: Disable the SharePlex for Oracle parameter SP_OPX_TRIM_TRAILING_ZEROS to avoid data inconsistency in SharePlex Connector for Hadoop. This is especially relevant to the INTERVAL DAY TO SECOND Oracle data type.

34 Appendix A: Contact Quest Contact Quest Support Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to our Support Portal at From our Support Portal, you can do the following: Retrieve thousands of solutions from our Knowledge Base Download the latest releases and service packs Create, update and review Support cases View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies and procedures. The guide is available at: Contact Quest Software Mail: Web site: info@quest.com Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA USA Refer to our Web site for regional and international office information. About Quest Software Now more than ever, organizations need to work smart and improve efficiency. Quest Software creates and supports smart systems management products helping our customers solve everyday IT challenges easier and faster. Learn more at

35 Appendix B: Third Party Components SharePlex Connector for Hadoop contains some third party components (listed below). Copies of their licenses may be found at Component activemqall jar 5.0 commonscodec 1.7 commonslang 2.6 Derby License or Acknowledgement Copyright The Apache Software Foundation Licensed under the Apache License, Version 2.0 ( Copyright The Apache Software Foundation Licensed under the Apache License, Version 2.0 ( Copyright The Apache Software Foundation Licensed under the Apache License, Version 2.0 ( Copyright The Apache Software Foundation Licensed under the Apache License, Version 2.0 ( This product includes software developed by The Apache Software Foundation ( ==================================================== Portions of Derby were originally developed by International Business Machines Corporation and are licensed to the Apache Software Foundation under the "Software Grant and Corporate Contribution License Agreement", informally known as the "Derby CLA". The following copyright notice(s) were affixed to portions of the code with which this file is now or was at one time distributed and are placed here unaltered. (C) Copyright 1997,2004 International Business Machines Corporation. All rights reserved. (C) Copyright IBM Corp ==================================================== The portion of the functiontests under 'nist' was originally developed by the National Institute of Standards and Technology (NIST), an agency of the United States Department of Commerce, and adapted by International Business Machines Corporation in accordance with the NIST Software Acknowledgment and Redistribution document at

36 Appendix B: Third Party Components 36 Component License or Acknowledgement ==================================================== The JDBC apis for small devices and JDBC3 (under java/stubs/jsr169 and java/stubs/jdbc3) were produced by trimming sources supplied by the Apache Harmony project. In addition, the Harmony SerialBlob and SerialClob implementations are used. The following notice covers the Harmony sources: Portions of Harmony were originally developed by Intel Corporation and are licensed to the Apache Software Foundation under the "Software Grant and Corporate Contribution License Agreement", informally known as the "Intel Harmony CLA". ==================================================== The Derby build relies on source files supplied by the Apache Felix project. The following notice covers the Felix files: Apache Felix Main Copyright 2008 The Apache Software Foundation I. Included Software This product includes software developed at The Apache Software Foundation ( Licensed under the Apache License 2.0. This product includes software developed at The OSGi Alliance ( Copyright (c) OSGi Alliance (2000, 2007). Licensed under the Apache License 2.0. This product includes software from Copyright (c) 2002,2003, Stefan Haustein, Oberhausen, Rhld., Germany. Licensed under BSD License. II. Used Software This product uses software developed at The OSGi Alliance ( Copyright (c) OSGi Alliance (2000, 2007). Licensed under the Apache License 2.0. III. License Summary - Apache License BSD License ==================================================== The Derby build relies on jar files supplied by the Apache Xalan project. The following notice covers the Xalan jar files: ==================================================== == NOTICE file corresponding to section 4(d) of the Apache License, == == Version 2.0, in this case for the Apache Xalan Java distribution. == ====================================================

37 Appendix B: Third Party Components 37 Component License or Acknowledgement Apache Xalan (Xalan XSLT processor) Copyright The Apache Software Foundation Apache Xalan (Xalan serializer) Copyright The Apache Software Foundation This product includes software developed at The Apache Software Foundation ( ==================================================== Portions of this software was originally based on the following: - software copyright (c) , Lotus Development Corporation., - software copyright (c) , Sun Microsystems., - software copyright (c) 2003, IBM Corporation., ==================================================== The binary distribution package (ie. jars, samples and documentation) of this product includes software developed by the following: - The Apache Software Foundation - Xerces Java - see LICENSE.txt - JAXP 1.3 APIs - see LICENSE.txt - Bytecode Engineering Library - see LICENSE.txt - Regular Expression - see LICENSE.txt - Scott Hudson, Frank Flannery, C. Scott Ananian - CUP Parser Generator runtime (javacup\runtime) - see LICENSE.txt ==================================================== The source distribution package (ie. all source and tools required to build Xalan Java) of this product includes software developed by the following: - The Apache Software Foundation - Xerces Java - see LICENSE.txt - JAXP 1.3 APIs - see LICENSE.txt - Bytecode Engineering Library - see LICENSE.txt - Regular Expression - see LICENSE.txt - Ant - see LICENSE.txt - Stylebook doc tool - see LICENSE.txt - Elliot Joel Berk and C. Scott Ananian - Lexical Analyzer Generator (JLex) - see LICENSE.txt ==================================================== Apache Xerces Java

38 Appendix B: Third Party Components 38 Component License or Acknowledgement Copyright The Apache Software Foundation This product includes software developed at The Apache Software Foundation ( Portions of Apache Xerces Java in xercesimpl.jar and xml-apis.jar were originally based on the following: - software copyright (c) 1999, IBM Corporation., - software copyright (c) 1999, Sun Microsystems., - voluntary contributions made by Paul Eng on behalf of the Apache Software Foundation that were originally developed at iclick, Inc., software copyright (c) ==================================================== Apache xml-commons xml-apis (redistribution of xml-apis.jar) Apache XML Commons Copyright ,2006 The Apache Software Foundation. This product includes software developed at The Apache Software Foundation ( Portions of this software were originally based on the following: - software copyright (c) 1999, IBM Corporation., - software copyright (c) 1999, Sun Microsystems., - software copyright (c) 2000 World Wide Web Consortium, Log4J Quartz Scheduler slf4j - Simple Logging Facade for Java Copyright , Apache Software Foundation. Licensed under the Apache License, Version 2.0 ( This product includes software developed by The Apache Software Foundation ( Copyright Terracotta, Inc. Licensed under the Apache License, Version 2.0 ( Copyright (c) QOS.ch All rights reserved. SLF4J source code and binaries are distributed under the MIT license.