Kognitio Technote Kognitio v8.x Hadoop Connector Setup

Transcription

1 Kognitio Technote Kognitio v8.x Hadoop Connector Setup For External Release Kognitio Document No Authors Reviewed By Authorised By Document Version Stuart Watt Date

2 Table Of Contents Document Control Introduction Installing Software For Kognitio Hadoop Access... 5 Appendices A. Testing MapReduce From The Linux Command Line A.1 Common MapReduce Failures For External Release Page 2

3 Document Control Distribution List Name Company Reason Kognitio For external release Revision History Version Revision Date Summary Of Changes First version Document Location Office Machine Filename Bracknell Babbage G:\Playground\Stuart.Watt\Hadoop\Kognitio Technote - v8.x Hadoop Connector Setup.docx For External Release Page 3

4 1. Introduction This document describes how to set up a Kognitio v8.x system to interoperate with a Hadoop cluster using Kognitio s external table technology. This document specifically covers the following Hadoop distributions currently: Hortonworks HDP v1.2.x; Cloudera CDH 4; Apache. Other Hadoop distributions may work with Kognitio, but as yet have not been verified. Note: The setup of the Hadoop cluster itself is outside the scope of this document. For External Release Page 4

5 2. Installing Software For Kognitio Hadoop Access This section outlines the steps necessary on all nodes to allow a Kognitio system, via its external table technology, to connect to a Hadoop cluster: 1. Install the latest Kognitio v8 Technology Preview Release (at least v which contains important MapReduce fixes), ensuring that it has a new generation licence that supports the v8.x features; 2. Install the IBM JVM on all the Kognitio nodes. Note that it needs to be the IBM JVM as the Oracle JVM won t work. Accept all the defaults as shown: % wget jre bin :53: Resolving kognitio-usa.s3.amazonaws.com Connecting to kognitio-usa.s3.amazonaws.com : connected. HTTP request sent, awaiting response OK Length: (65M) [application/octet-stream] Saving to: `ibm-java-i386-jre bin' 100%[==>] 68,596, M/s in 7.3s :53:08 (8.95 MB/s) - `ibm-java-i386-jre bin' saved [ / ] % chmod +x ibm-java-i386-jre bin %./ibm-java-i386-jre bin Preparing to install... Extracting the JRE from the installer archive... Unpacking the JRE... Extracting the installation resources from the installer archive... Configuring the installer for this system's environment... Launching installer... Graphical installers are not supported by the VM. The console mode will be used instead... Choose Locale Català 2- Deutsch ->3- English 4- Español 5- Français 6- Italiano 7- Português (Brasil) CHOOSE LOCALE BY NUMBER: 3 For External Release Page 5

6 IBM 32-bit Linux Runtime for Java v6 (created with InstallAnywhere) Preparing CONSOLE Mode Installation... License Agreement Installation and Use of IBM 32-bit Linux Runtime for Java v6 Requires Acceptance of the Following License Agreement: International License Agreement for Non-Warranted Programs Part 1 - General Terms BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE PROGRAM YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO THESE TERMS. IF YOU DO NOT AGREE TO THESE TERMS, - DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE PROGRAM; AND - PROMPTLY RETURN THE PROGRAM AND PROOF OF ENTITLEMENT TO THE PARTY FROM WHOM YOU ACQUIRED IT TO OBTAIN A REFUND OF THE AMOUNT YOU PAID. IF YOU DOWNLOADED THE PROGRAM, CONTACT THE PARTY FROM WHOM YOU ACQUIRED IT. "IBM" is International Business Machines Corporation or one of its subsidiaries. "License Information" ("LI") is a document that provides information specific to a Program. The Program's LI is available at The LI may also be found in a file in the Program's directory, by the use of a system command, or as a booklet which accompanies the Program. PRESS <ENTER> TO CONTINUE: etc. *** Need to press ENTER about 40 times 3. TRADEMARKS AND COPYRIGHT: YOUR RESPONSIBILITIES For External Release Page 6

7 a) You shall not modify, delete, suppress, or obscure any copyright, trademark or other legal notice (whether from IBM or any third party) which may be displayed by or included within the Program. b) Java and all Java-based Trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. c) You recognize IBM's and Sun Microsystems, Inc.'s ownership and title to their respective trademarks and of any goodwill attaching thereto, including goodwill resulting from use. You will not use or attempt to register any trademark which is confusingly similar to such IBM or Sun trademarks. 3. PROOF OF ENTITLEMENT This License Agreement constitutes your Proof of Entitlement. PRESS <ENTER> TO CONTINUE: D/N: L-RVEK-75GKYF P/N: L-RVEK-75GKYF DO YOU ACCEPT THE TERMS OF THIS LICENSE AGREEMENT? (Y/N): y Introduction InstallAnywhere will guide you through the installation of IBM 32-bit Linux Runtime for Java v6. It is strongly recommended that you quit all programs before continuing with this installation. Respond to each prompt to proceed to the next step in the installation. If you want to change something on a previous step, type 'back'. You may cancel this installation at any time by typing 'quit'. PRESS <ENTER> TO CONTINUE: Choose Install Folder Where would you like to install? For External Release Page 7

8 Default Install Folder: /opt/ibm/java-i ENTER AN ABSOLUTE PATH, OR PRESS <ENTER> TO ACCEPT THE DEFAULT : Pre-Installation Summary Please Review the Following Before Continuing: Product Name: IBM 32-bit Linux Runtime for Java v6 Install Folder: /opt/ibm/java-i Disk Space Information (for Installation Target): Required: 99,442,965 bytes Available: 7,987,286,016 bytes PRESS <ENTER> TO CONTINUE: Installing [== == == = =] [ ] Installation Complete Congratulations. IBM 32-bit Linux Runtime for Java v6 has been successfully installed to: /opt/ibm/java-i PRESS <ENTER> TO EXIT THE INSTALLER: % 3. Edit /etc/profile.local to add the following lines so that the JRE is in the PATH for all users: export JAVA_HOME=/opt/ibm/java-i386-60/jre export PATH=$PATH:$JAVA_HOME/bin 4. Log out and log back in and then check that Java is operational: % java -version For External Release Page 8

9 java version "1.6.0" Java(TM) SE Runtime Environment (build pxi3260sr _01(sr11)) IBM J9 VM (build 2.4, JRE IBM J9 2.4 Linux x86-32 jvmxi3260sr _ (JIT enabled, AOT enabled) J9VM _ JIT - r9_ _24176ifx1 GC _AA) JCL _01 5. Install the appropriate Hadoop client from the chosen Hadoop distribution on all the Kognitio nodes. This step is outside the scope of this document; 6. Edit /etc/profile.local so that the Hadoop client home is declared for all users: Hadoop Distribution Hortonworks HDP Cloudera Apache Hadoop Client Home /usr/lib/hadoop /usr/lib/hadoop/client /usr/lib/hadoop-x.xx-mapreduce export HADOOP_HOME=<hadoop_client_home> The exact directory required varies between Hadoop distributions. In general terms, the HADOOP_HOME setting needs to point at the Hadoop directory that contains the files of the form hadoop-xxxx-1.0.n.jar. 7. Log out and log back in and then check that the Hadoop client on each Kognitio node can connect to the Hadoop cluster: % hadoop fs -ls / Found 3 items drwx mapred hdfs drwxrwxrwx - hdfs hdfs drwxr-xr-x - hdfs hdfs :48 /mapred :12 /tmp :16 /user The exact directories shown will depend on the Hadoop distribution, e.g. the example above is from Hortonworks HDP. 8. Install the Kognitio-compiled version of libhdfs on all the Kognitio nodes: wget mkdir /usr/local/lib/hdfs cd /usr/local/lib/hdfs cp $HOME/libhdfs.so ln -s libhdfs.so libhdfs.so ln -s libhdfs.so libhdfs.so.0 echo -e '/opt/ibm/java-i386-60/jre/lib/i386\n/opt/ibm/java-i386-60/jre/lib/i386/j9vm\n/usr/local/lib/hdfs' >>/etc/ld.so.conf ldconfig 9. Make sure that the output from cat /etc/ld.so.conf looks similar to this: % cat /etc/ld.so.conf etc. /usr/lib64 /usr/lib include /etc/ld.so.conf.d/*.conf /opt/ibm/java-i386-60/jre/lib/i386 /opt/ibm/java-i386-60/jre/lib/i386/j9vm For External Release Page 9

10 /usr/local/lib/hdfs 10. The Cloudera Hadoop client keeps a key generic jar file in a different location to standard Hadoop, which means that the following commands must be run. This step is not required for other Hadoop distributions: # Allow MapReduce to work on Cloudera by allowing access to Hadoop client library cd /usr/lib/hadoop/client ln -s /usr/lib/hadoop-0.20-mapreduce/contrib contrib 11. Update the Kognitio database configuration file to ensure that the v8.x features are enabled: [boot options] external_tables=yes # Hadoop external tables external_scripts=yes # External scripting functionality fixed_pool_size=20 # Only needed if lots of external scripting 12. Create the Kognitio Hadoop module to reference the Hadoop client path set at Step 6 above. This must be done as soon as the Kognitio server is commissioned, i.e. before the Hadoop connectors are defined. The port numbers of the namenode and jobtracker nodes vary between Hadoop distributions: Hadoop Distribution Namenode Port Jobtracker Port Hortonworks HDP Cloudera Apache create module hadoop using '/opt/kognitio/wx2/current/software/linux/hadoop.wxpi hadoop_home=<hadoop_client_home_dir> java_home=/opt/ibm/java-i386-60/jre'; alter module hadoop set mode active; create connector hadoop_hdfs source hdfs target 'namenode <namenode_internal_ip_address>:<namenode_port>, user <hadoop_user>'; grant connect on hadoop_hdfs to public; -- If public access required create connector hadoop_mr source hadoopmap target 'namenode <namenode_internal_ip_address>:<namenode_port>, jobtracker <jobtracker_internal_ip_address>:<jobtrack_port>, subnets /8'; grant connect on hadoop_mr to public; -- If public access required The user clause should be specified if the Hadoop cluster is accessed with a user other than root. The subnets clause will definitely be required in Amazon EC2 environments and may be required in other environments as well. 13. Create an external table that connects to a test file in HDFS, e.g.: create external table test( <column_definitions>) from hadoop_hdfs -- Connector name defined above, could use hadoop_mr target 'file /user/zzzz/test.txt'; 1 Some HDP documentation claims it is port 8021 but this does not appear to work For External Release Page 10

11 A. Testing MapReduce From The Linux Command Line If a SQL statement fails when using the Kognitio hadoopmap (MapReduce) connector, it is sometimes helpful to run a test MapReduce job from the Linux command line to verify the basic operation of MapReduce independently of Kognitio: % hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming jar -fs hdfs://<namenode>:<namenode_port> -jt <jobtracker_node>:<jobtracker_port> -D mapred.reduce.tasks=0 -input hdfs://<namenode>:<namenode_port>/user/hadoop/test.txt -output hdfs://<namenode>:<namenode_port>/tmp/testoutput -mapper cat This MapReduce job will take the HDFS file /user/hadoop/test.txt and simply copy it ( cat ) to the HDFS directory /tmp/testoutput as a way of verifying that MapReduce is functioning correctly. The exact name and location of the hadoop-streaming jar file will depend on the Hadoop distribution being used. A.1 Common MapReduce Failures A common source of MapReduce job failures when running with the hadoopmap connector is HDFS file permissions issues. This is because MapReduce jobs always run as root, but the HDFS files being accessed may, by default, be inaccessible to the root user. The permissions on the HDFS files can be checked using the hadoop fs ls command: % hadoop fs -ls /user/hadoop Found 1 item -rwx hadoop hdfs :58 /user/hadoop/test.txt In this case, the HDFS file /user/hadoop/test.txt cannot be accessed by the root user and therefore the permissions need to be adjusted appropriately: % hadoop fs -chmod g+rx,o+rx /user/hadoop/test.txt % hadoop fs -ls /user/hadoop Found 1 item -rwxr-xr-x 3 hadoop hdfs :58 /user/hadoop/test.txt For External Release Page 11