unisys ClearPath Servers Hadoop Distributed File System(HDFS) Data Transfer Guide Firmware 2.0 and Higher December 2014 8230 6952-000
NO WARRANTIES OF ANY NATURE ARE EXTENDED BY THIS DOCUMENT. Any product or related information described herein is only furnished pursuant and subject to the terms and conditions of a duly executed agreement to purchase or lease equipment or to license software. The only warranties made by Unisys, if any, with respect to the products described in this document are set forth in such agreement. Unisys cannot accept any financial or other responsibility that may be the result of your use of the information in this document or software material, including direct, special, or consequential damages. You should be very careful to ensure that the use of this information and/or software material complies with the laws, rules, and regulations of the jurisdictions with respect to which it is used. The information contained herein is subject to change without notice. Revisions may be issued to advise of such changes and/or additions. Notice to U.S. Government End Users: This is commercial computer software or hardware documentation developed at private expense. Use, reproduction, or disclosure by the Government is subject to the terms of Unisys standard commercial license for the products, and where applicable, the restricted/limited rights provisions of the contract data rights clauses. Unisys, the Unisys logo, ClearPath, Forward! by Unisys, and the Forward! by Unisys logo are registered trademarks or trademarks of Unisys Corporation in the United States and other countries. All other brands and products referenced in this document are acknowledged to be the trademarks or registered trademarks of their respective holders.
Contents Section 1. Introduction 1.1. Documentation Updates........................... 1 1 1.2. Overview...................................... 1 2 1.3. Audience...................................... 1 2 1.4. Prerequisites................................... 1 3 1.5. Sequence of Procedures........................... 1 3 1.6. Considerations.................................. 1 4 1.7. Related Information.............................. 1 4 Section 2. Transferring RDMS and Enterprise Database Server Data Using Sqoop 2.1. About Sqoop................................... 2 1 2.2. Setting Up Hadoop to Use ClearPath JDBC............. 2 2 2.3. Transferring the Current Contents of a Table............. 2 3 Section 3. Transferring Enterprise Database Server Data without Using Sqoop 3.1. Extracting Data from an Enterprise Database Server Database.................................... 3 1 3.2. Using Query Design Center to Extract Data from an Enterprise Server Data Database................... 3 2 3.3. Transferring the Data into an HDFS................... 3 2 3.3.1. Transferring Data into HDFS Using PUT Command.... 3 3 3.3.2. Transferring Data into HDFS Using Java Program..... 3 3 Section 4. Transferring DMS Data Using QLP 4.1. Overview...................................... 4 1 4.2. Extracting DMS Data into a Report Using the QLP Processor.................................... 4 2 4.3. Sample DMS Database............................ 4 2 4.4. Extracting DMS Data into a Report................... 4 3 4.5. Transferring the Report into HDFS.................... 4 5 Section 5. Next Steps 8230 6952-000 iii
Contents iv 8230 6952-000
Figures 1 1. Transferring ClearPath Data to a Hadoop Cluster Running on the Forward! Fabric................................................... 1 2 2 1. Transferring RDMS and Enterprise Database Server Data Using Sqoop...... 2 2 3 1. Transferring Enterprise Database Server Data without Using Sqoop........ 3 1 4 1. Transferring DMS Data Using QLP................................ 4 2 8230 6952-000 v
Figures vi 8230 6952-000
Section 1 Introduction This section provides the information an administrator needs before transferring data from a ClearPath database to a Hadoop Distributed File System (HDFS). 1.1. Documentation Updates This document contains all the information that was available at the time of publication. Changes identified after release of this document are included in problem list entry (PLE) 19054420. To obtain a copy of the PLE, contact your service representative or access the current PLE from the product support Web site: http://www.support.unisys.com/all/ple/19054420 Note: If you are not logged into the product support site, you will be asked to do so. 8230 6952-000 1 1
Introduction 1.2. Overview This document describes how to transfer ClearPath data into a Hadoop Distributed File System (HDFS) running on the Forward! fabric (see Figure 1 1). This enables an enterprise to use data from ClearPath database systems, along with data from other sources, in their analytic or business intelligence system. Figure 1 1. Transferring ClearPath Data to a Hadoop Cluster Running on the Forward! Fabric 1.3. Audience This document is for the ClearPath database administrator. It helps the administrator transfer Relational Database Management System (RDMS), Database Management System (DMS), or Enterprise Database Server data stored on ClearPath systems into an HDFS cluster on the Forward! fabric. This document assumes the administrator is familiar with the following topics: ClearPath databases and either the OS 2200 or MCP operating environment Forward! fabric Windows and Linux operating systems 1 2 8230 6952-000
Introduction Hadoop and Sqoop Depending on the ClearPath database and the method chosen to transfer the data, the administrator may also need to be familiar with the following tools: ClearPath software such as Query Design Center or the Query Language Processor (QLP 2200). Open-source tools such as WinSCP. 1.4. Prerequisites Before you begin, make sure you address the following prerequisites: A Hadoop cluster is running on the Forward! fabric. See the Forward! by Unisys Installation, Administration and Operations Guide for more information on configuring a Hadoop cluster on the Forward! fabric. You know which tables in the RDMS, DMS, or Enterprise Database Server database system you want to transfer to the HDFS cluster. 1.5. Sequence of Procedures The method for transferring ClearPath data to a HDFS depends on the database where the data currently resides. Relational Database Management System (RDMS) If the data resides in an RDMS database, use the following procedures: 1. Set up Hadoop to use ClearPath Java Database Connectivity (JDBC). 2. Transfer the current contents of the table from the database to the HDFS cluster using Sqoop. 3. Periodically run Sqoop to transfer any data that has not already been transferred. See Section 2, Transferring RDMS and Enterprise Database Server Data Using Sqoop, for the details. Enterprise Database Server For Enterprise Database Server, use the following procedures if you are using Sqoop: 1. Set up Hadoop to use ClearPath Java Database Connectivity (JDBC). 2. Transfer the current contents of the table from the database to the HDFS cluster using Sqoop. 3. Periodically run Sqoop to transfer any data that has not already been transferred. See Section 2, Transferring RDMS and Enterprise Database Server Data Using Sqoop, for the details. 8230 6952-000 1 3
Introduction Use the following procedures if you are not using Sqoop: 1. Extract the data from the Enterprise Database Server database using Query Design Center. 2. Transfer the data into the HDFS cluster using either a PUT command or a Java program. See Section 3, Transferring Enterprise Database Server Data without Using Sqoop, for the details. Database Management System (DMS) If the data resides in a DMS database, use the following procedures: 1. Use QLP processor to extract the data into a report. 2. Transfer the report into the HDFS cluster. See Section 4, Transferring DMS Data Using QLP, for the details. 1.6. Considerations A Hadoop Distributed File System (HDFS) has its own methods for controlling access. When ClearPath database tables and its data are moved into a HDFS, none of the access control mechanisms are transferred with the data. For example, if the ClearPath data contains the following control mechanisms, they are not transferred with the data: Permission settings on tables or columns. The administrator must set any desired access controls in the HDFS. Mechanisms for filtering on specific parts of a database. Mechanisms for control at a more specific level are not available. 1.7. Related Information See the following Web sites or documents for more information about Hadoop or ClearPath database environments. Apache Hadoop Project Hadoop: http://hadoop.apache.org Ambari: http://ambari.apache.org Sqoop: http://sqoop.apache.org ClearPath MCP Documents JDBC for ClearPath MCP User s Guide (3847 5588) Enterprise Database Server for ClearPath MCP Application Program Interfaces Programming Guide (8600 2409) 1 4 8230 6952-000
Introduction SQL Query Processor for ClearPath MCP Installation and Operations Guide (3850 8206) System Software Utilities Operations Reference Manual (8600 0460) ClearPath OS 2200 Documents Relational JDBC Driver for ClearPath OS 2200 User Guide (7862 7395) Relational Database Server for ClearPath OS 2200 SQL Programming Reference Manual (7830 8160) Query Language Processor (QLP 2200) End Use Guide, Volume 1: Basic Commands (7830 7493) ODBC Access for ClearPath OS 2200 Installation and Operations Guide (7862 5753) Forward! by Unisys Documents Forward! by Unisys Overview and Planning Guide (8222 4528) Forward! by Unisys Installation, Administration and Operations Guide (8222 4544) 8230 6952-000 1 5
Introduction 1 6 8230 6952-000
Section 2 Transferring RDMS and Enterprise Database Server Data Using Sqoop This section describes how to transfer RDMS or Enterprise Database Server data to a HDFS using Sqoop. 2.1. About Sqoop Sqoop is an open-source, command-line tool from the Apache Software Foundation. It transfers bulk data between Hadoop and structured databases. Using Sqoop, you can import individual tables into files into an HDFS. For information about Sqoop, see http://sqoop.apache.org/docs/1.4.0-incubating/sqoopuserguide.html. Figure 2 1 illustrates how Sqoop is used to transfer data from RDMS and Enterprise Database Server databases to a HDFS. 8230 6952-000 2 1
Transferring RDMS and Enterprise Database Server Data Using Sqoop Figure 2 1. Transferring RDMS and Enterprise Database Server Data Using Sqoop 2.2. Setting Up Hadoop to Use ClearPath JDBC Once the Hadoop cluster is established on the Forward! fabric, make sure that the Sqoop environment has the necessary JDBC driver library files to access your ClearPath server. The server provides the necessary files. If you encounter an error message about Mismatched JDBC levels when you run Sqoop against your server, update the JDBC driver jar files on the Hadoop cluster using the following procedure: 1. Obtain the jar files appropriate to the level of the JDBC server to which you are going to connect. For Enterprise Database Server on MCP, the files are jdbcformcp.jar, provider.jar, and xlateebcdic.jar. For RDMS on OS 2200, the file is rdmsdriver.jar. 2. Transfer these files to the /usr/jdbc/ folder on the ResourceManager/NameNode of your Hadoop cluster. You must log in to that partition as root and enter:./usr/jdbc/distjdbcdrivers.sh This script distributes the correct drivers to all the nodes of the cluster. 3. Remove the old files from the /usr/sqoop/lib folder. That is, if you are updating drivers for RDMS on an OS 2200 system, enter: 2 2 8230 6952-000
Transferring RDMS and Enterprise Database Server Data Using Sqoop rm -f /usr/sqoop/lib/rdmsdriver.jar If you are updating drivers for Enterprise Database Server on MCP, enter: rm -f /usr/sqoop/lib/jdbcformcp.jar rm -f /usr/sqoop/lib/provider.jar rm -f /usr/sqoop/lib/xlateebcdic.jar 2.3. Transferring the Current Contents of a Table The /usr/lib/sqoop folder on the client node of the Hadoop cluster contains a number of scripts to help you transfer data from your ClearPath server to HDFS. 1. Log in to a client node on the Hadoop cluster as hdadmin. 2. Enter cd /usr/lib/sqoop This changes your current directory to /usr/lib/sqoop. 3. Execute the sqooprdms.sh script with the following parameters: Schema name (database name) Table name Host name (IP address or host name with suffix) Username to access database Column name on which to split the table (name of the key column of the table by which the data gets split across the cluster) For example,./sqooprdms.sh dbase dtable rdhost Usera T_ID This example loads the rows of dtable from the database dbase on the OS 2200 rdhost. It uses the userid Usera and splits the job on T_ID. 8230 6952-000 2 3
Transferring RDMS and Enterprise Database Server Data Using Sqoop 2 4 8230 6952-000
Section 3 Transferring Enterprise Database Server Data without Using Sqoop This section describes how to transfer Enterprise Database Server data to an HDFS without using Sqoop. 3.1. Extracting Data from an Enterprise Database Server Database To extract data from Enterprise Database Server without using Sqoop, extract the data to a text file or comma-separated file. Figure 3 1 depicts the possible ways of extracting the data from the Enterprise Database Server without using Sqoop. Figure 3 1. Transferring Enterprise Database Server Data without Using Sqoop 8230 6952-000 3 1
Transferring Enterprise Database Server Data without Using Sqoop 3.2 Using Query Design Center to Extract Data from an Enterprise Server Data Database describes using Query Design Center (QDC) to extract the data from the database and copy it to the NameNode. A PUT command or a user-written Java program is used to transfer the text file or commaseparated file into an HDFS. See 3.3 Transferring the Data into an HDFS. 3.2. Using Query Design Center to Extract Data from an Enterprise Server Data Database Query Design Center is an interactive query tool that developers and system administrators use to issue queries against SQL Query Processor for ClearPath MCP. To use Query Design Center to extract the data, perform the following steps: 1. In Query Design Center, right-click the database and then select Analyze Query. 2. Enter and execute SQL statements to fetch the data from the database to Query Design Center. The result is displayed. 3. On the File menu, click Save Results Set As and then specify the name of the file on the Windows PC to which the data is extracted. 4. Using the WinSCP tool, copy the data files from the Windows PC to the NameNode in the Linux system. For more information about Query Design Center, refer to the SQL Query Processor for ClearPath MCP Installation and Operations Guide. For more information about WinSCP, see http://winscp.net. 3.3. Transferring the Data into an HDFS You can transfer a text file or comma-separated file into an HDFS using one of the following methods: PUT command The PUT command is a Hadoop shell command used to transfer data into an HDFS. See 3.3.1 Transferring Data into HDFS Using PUT Command for the details. Java program You can write a Java program to transfer data into an HDFS. See 3.3.2 Transferring Data into HDFS Using Java Program for the details. 3 2 8230 6952-000
Transferring Enterprise Database Server Data without Using Sqoop 3.3.1. Transferring Data into HDFS Using PUT Command Use one of the following procedures to transfer a file into an HDFS using the PUT command: To transfer a file into an HDFS, execute the following command from the Hadoop directory: hadoop dfs -put /home/testfile.txt /SampleFile.txt where Testfile.txt is the name of the input text file and SampleFile.txt is the name of the Hadoop file into which you want to transfer the input file. To transfer a file into a folder created in an HDFS, execute the following commands: hadoop dfs -mkdir hfiles hadoop dfs -put /home/testfile.txt /hfiles/samplefile.txt where hfiles is the name of the HDFS folder. To transfer multiple files into a folder created in an HDFS, execute the following commands: hadoop dfs -mkdir InputFolder hadoop dfs -put /home/testinput/*.txt InputFolder where InputFolder is the name of the HDFS folder and TestInput is the folder containing multiple files that need to be transferred into the HDFS. For more information, see http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html 3.3.2. Transferring Data into HDFS Using Java Program Use the following procedure to transfer data into HDFS using a user-written Java program. Prerequisites The Hadoop Conf folder has been added into the classpath of the Java program. The following additional jar files have been copied to your Eclipse workspace. - hadoop-core-1.0.1.jar - commons-logging-1.1.1.jar - commons-logging-api-1.0.4.jar - commons-configuration-1.6.jar - commons-lang-2.4.jar Procedure To transfer data into HDFS using a user-written Java program, perform the following steps: 1. Create a java file (for example, HadoopAppendFile.jar) using the following sample: package com.unisys.hadoop; import java.io.bufferedreader; 8230 6952-000 3 3
Transferring Enterprise Database Server Data without Using Sqoop import java.io.bufferedwriter; import java.io.file; import java.io.filereader; import java.io.outputstreamwriter; import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; public class AppendFileHadoop public static void main (String [] args) throws Exception{try{ } String uri =args[0]; Path pt=new Path(uri); FileSystem fs = FileSystem.get(new Configuration()); BufferedWriter br; if(fs.isfile(pt)){ br=new BufferedWriter(new OutputStreamWriter(fs.append(pt))); br.newline(); }else{ br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true))); } String source=args[1]; File src = new File(source); BufferedReader reader; reader = new BufferedReader( new FileReader(src)); String input=""; while ((input=reader.readline())!=null) { br.write(input); br.newline(); } reader.close(); br.close(); }catch(exception e){ e.printstacktrace(); System.out.println("File not found"); } } 2. Place the Java files in the project classpath. 3. Using the Eclipse tool, import the Java file and then export it as a JAR file. 4. Execute the following command: hadoop jar /home/ HadoopAppendFile.jar com.unisys.hadoop.appendfilehadoop hdfs://<host name>:<port number>/user/hduser/hfiles/customers.txt /home/cust_update.txt where HadoopAppendFile.jar is the Java JAR file. com.unisys.hadoop.appendfilehadoop is the package and Java class name. 3 4 8230 6952-000
Transferring Enterprise Database Server Data without Using Sqoop hdfs://<host name>:<port number>/user/hduser/hfiles/ is the path to the HDFS file - Replace <host name> with the appropriate host name of the NameNode. - Replace <port number> with the appropriate port number of the NameNode. - user/hduser/hfiles is the directory folder path. CUSTOMERS.TXT is the Hadoop file name to be updated. CUST_Update.TXT is the input file name. 8230 6952-000 3 5
Transferring Enterprise Database Server Data without Using Sqoop 3 6 8230 6952-000
Section 4 Transferring DMS Data Using QLP This section describes how to extract DMS data using the Query Language Processor (QLP 2200) and then transferring the data into an HDFS. 4.1. Overview The DMS database engine on the OS 2200 does not have a supported JDBC server interface. To transfer data from a DMS database, you need to directly extract the data using one of the following methods: Code written by the user (COBOL or C, for example). ODBC Access to retrieve data using the ODBC interface to the data. For more information on ODBC Access, refer to the documents on ODBC Access for ClearPath OS 2200 in the ClearPath documentation suite. Using the QLP processor to extract data into a report. The following topics describe this method. For general information about the QLP processor, refer to the Query Language Processor (QLP 2200) End Use Guide, Volume 1: Basic Commands. Figure 4 1 illustrates the use of the QLP processor to transfer DMS data to an HDFS. 8230 6952-000 4 1
Transferring DMS Data Using QLP Figure 4 1. Transferring DMS Data Using QLP 4.2. Extracting DMS Data into a Report Using the QLP Processor To use the QLP processor, you need to declare the subschema and process it by invoking the Subschema Data Definition Language (SDDL) processor. This subschema is referenced to invoke the QLP processor and then generate a report containing the data selected as a series of rows with comma-delimited values. The example in 4.3 Sample DMS Database through 4.5 Transferring the Report into HDFS demonstrates the procedure for using the QLP processor to extract DMS data to a report and then transfer the report to an HDFS. 4.3. Sample DMS Database This example uses the following schema for a DMS database: *IDENTIFICATION DIVISION SCHEMA NAME IS COMPOUND IN FILE QLP-COMPOUND DATA DIVISION AREA SECTION AREA NAME IS INORGANIC AREA CODE IS 1 ALLOCATE 20 PAGES PAGES ARE 140 WORDS 4 2 8230 6952-000
Transferring DMS Data Using QLP RECORD SECTION RECORD NAME IS ELEM-NAME RECORD CODE IS 1 LOCATION MODE IS DIRECT DBASE-K, DBASE-N WITHIN INORGANIC RECORD MODE IS ASCII 02 NAME-E PIC X(12) RECORD NAME IS ELEMENT RECORD CODE IS 2 LOCATION MODE IS VIA UNKNOWN SET WITHIN INORGANIC RECORD MODE IS ASCII 02 FORMULA PIC X(12) 02 SPEC-GVTY PIC 9(3)V99 USAGE IS COMP 02 MOL-WGT PIC 9(5)V9(6) USAGE IS COMP 02 MELT-POINT PIC S9(4)V9(4) 02 BOIL-POINT PIC S9(4)V9(4) 02 SOLUBILITY-C PIC X(2) 02 SOLUBILITY-H PIC X(2) 02 FLT-NBRS 03 CELL-VOLUME PIC 9(15) USAGE IS COMP 03 XRAY-VOLUME PIC 9(10) USAGE IS COMP 02 BUSINESS-DATA USAGE IS DISP 03 COSTS 04 WHOLESALE-P PIC 9(10)V99 04 RETAIL-P PIC 9(12)V99 03 SALE-CODES 04 REGION-NBR PIC 99PPPP 04 DIST-NBR PIC 99PPP USAGE IS COMP 04 SUB-DIST-NBR PIC 9999P USAGE IS DISPLAY 04 SALESMAN 05 REG-MGR PIC 9V99999 USAGE IS DISPLAY 05 DIST-MGR PIC VPPP9 USAGE IS DISPLAY-1 05 SUB-MGR PIC VPPPP9 USAGE IS COMP-4 05 SELLER PIC VPPPPP999999 USAGE IS DISPLAY-1 04 ACCNT-STATUS PIC SVPP9999 SET SECTION SET NAME IS UNKNOWN SET CODE IS 1 MODE IS CHAIN LINKED PRIOR ORDER IS NEXT OWNER IS ELEM-NAME MEMBER IS ELEMENT AUTOMATIC SET SELECTION IS THRU CURRENT OF SET 4.4. Extracting DMS Data into a Report The following procedure lists the steps to extract the data into a report: 8230 6952-000 4 3
Transferring DMS Data Using QLP 1. Create a subschema that can be used with QLP as in the following example: IDENTIFICATION DIVISION SUBSCHEMA NAME IS HADDMP IN FILE QLP-COMPOUND OF SCHEMA COMPOUND HOST LANGUAGE IS QLP DATA DIVISION DATA NAME SECTION DATA NAMES ARE ALL AREA SECTION AREAS ARE ALL RECORD SECTION RECORDS ARE ALL SET SECTION SETS ARE ALL QLP SECTIONPATH NAME IS ANALYSIS ROOT IS ELEM-NAME DIRECT CURRENCY ASSUMED DATABASE-KEY = 000100100001 THRU UNKNOWN TO ELEMENT PATH NAME IS DIRECT1 ROOT IS ELEM-NAME THRU UNKNOWN TO ELEMENT 2. Process the subschema using the following add stream: @. Down and up the data base file @sys$lib$*iru.iru,e down file uds$$src*qlp-compound;act appl udssrc; up file uds$$src*qlp-compound;act appl udssrc; end @. Clear the schema @uds$$src*abs$.suds,de udssrc sa compound sccompound exit @. Process the subschema and install it @uds$$src*dmrmt$.sddl,sd hdp-qlp*compound.haddmp,uds$$src*qlpcompound.haddmp @sys$lib$*urep.dd,e,,udssrc process subschema haddmp for schema compound install. exit. @. 3. Use the following run stream to pull rows containing formula, boiling point, melting point, retail price, and account status: @use qlp-compound.,uds$$src*qlp-compound. @use qlp$.,sys$lib$*qlp. @delete,c haddmp*hdout. @cat,p haddmp*hdout.,f///9999 @use hdout.,haddmp*hdout. @qlp$.qlp,d invoke haddmp of qlp-compound report haddump for transfer FILE DETAIL LINE column 1 formula col plus 1,, boil-point,,,melt-point,,,retail-p,,,accntstatus 4 4 8230 6952-000
Transferring DMS Data Using QLP END REPORT GENERATE haddump TO FILE hdout exit The output is placed in the data file, haddmp*hdout, as follows: PX012-1, 868.0056, 1000.9997, 275049.63,-.005687 PX012-2,- 436.5556,- 32.3800, 3230422.73,-.002863 PX012-3, 1.7765, 59.8823, 1756433.56,-.005688 PX012-4,- 53.9776, 5.8993, 323042274.00,.002863 4.5. Transferring the Report into HDFS Transfer the DMS data in the generated report into HDFS using the following procedure. 1. On the Hadoop system, mount the OS 2200 file system as /mnt/dmssys using the following bash command: Mount -t cifs //dmsys/os2200 -o username=myuid,password=mypwrd /mnt/ dmssys where //dmssys/os2200 is the name of the IP address of the OS 2200 system where the database resides. 2. Place the generated report into the HDFS location, /DMSSYS/compound1, using the following command: hadoop dfs -copyfromlocal /mnt/dmssys/haddmp/hdout /DMSSYS/compound1 8230 6952-000 4 5
Transferring DMS Data Using QLP 4 6 8230 6952-000
Section 5 Next Steps This document contains the procedures you can use to transfer RDMS, DMS, or Enterprise Database Server data into an HDFS cluster. Once you have transferred the ClearPath data into an HDFS cluster, use any of the open source or commercial tools for business intelligence or analytics to process and analyze the data. This document does not describe how to use these tools to analyze large data sets. See the documentation from your tool vendor for additional information. 8230 6952-000 5 1
Next Steps 5 2 8230 6952-000
.
2014 Unisys Corporation. All rights reserved. *82306952-000* 8230 6952-000