TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data User s Guide

Transcription

1 TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data User s Guide Software Release 1.0 November 2013 Two-Second Advantage

2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE. USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE LICENSE FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME. This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc. TIBCO, Two-Second Advantage, TIBCO ActiveMatrix BusinessWorks, TIBCO Designer, TIBCO Administrator, TIBCO Rendezvous, TIBCO Runtime Agent, and TIBCO Hawk are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. Enterprise Java Beans (EJB), Java Platform Enterprise Edition (Java EE), Java 2 Platform Enterprise Edition (J2EE), and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle Corporation, Inc. in the U.S. and other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only. THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM. THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME. THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES. Copyright 2013 TIBCO Software Inc. ALL RIGHTS RESERVED. TIBCO Software Inc. Confidential Information

3 iii Contents Figures vii Tables ix Preface xi Related Documentation xii TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Documentation xii Other TIBCO Product Documentation xii Typographical Conventions xiii Connecting with TIBCO Resources xv How to Join TIBCOmmunity xv How to Access TIBCO Documentation xv How to Contact TIBCO Support xv Chapter 1 Introduction Overview of the Plug-in Chapter 2 HDFS Palette and Hadoop Palette Palette Overview HDFSConnection Configuration HDFS Operation Configuration Input Output Error Output ListFileStatus Configuration Input Output Error Output Read Configuration Input Output

4 iv Contents Error Output Write Configuration Input Output Error Output HCatalogConnection Configuration Database Configuration Table Configuration Advanced Hive Configuration Input Output Error Output Mapreduce Configuration Input Output Error Output Pig Configuration UDF Input Output Error Output WaitForJobCompletion Configuration Input Output Error Output Chapter 3 Getting Started Overview Creating a Project Configuring an HCatalogConnection Shared Resource Creating a Process Adding Activities to a Process

5 Contents v Testing a Process Deploying a Project Chapter 4 Using the Sample Project Overview of the Sample Project DataCollectionAndQuery Project Process Description Setting Up the Sample Project Running the Sample Project Expected Result Appendix A Managing Big Data Logs Managing Logs Appendix B Trace Messages HDFS Error Messages Big Data Error Messages Appendix C Configuration for Amazon EMR Configuring for Amazon EMR Cluster Index

6 vi Contents

7 Figures vii Figures Figure 1 HDFS Palette Figure 2 HCatalog Palette Figure 3 TIBCO Designer Start Screen Figure 4 TIBCO Designer Save Project Screen Figure 5 The DemoWorkflow Process Figure 6 The ErrorHandler Process Figure 7 The DemoWorkflow Process Output Figure 8 The ErrorHandler Process Result

8 viii Figures

9 Tables ix Tables Table 1 General Typographical Conventions xiii Table 2 HDFSConnection Configuration Table 3 HDFS Operation Configuration Table 4 HDFS Operation Input Table 5 HDFS Operation Output Table 6 HDFS Operation Error Output Table 7 ListFileStatus Configuration Table 8 ListFileStatus Input Table 9 ListFileStatus Output Table 10 ListFileStatus Error Output Table 11 Read Configuration Table 12 Read Input Table 13 ListFileStatus Output Table 14 Read Error Output Table 15 Write Configuration Table 16 Write Input Table 17 Write Output Table 18 Write Error Output Table 19 HCatalogConnection Configuration Table 20 Database Configuration Table 21 Table Configuration Table 22 Table Advanced Table 23 Hive Configuration Table 24 Hive Input Table 25 Hive Output Table 26 Hive Error Output Table 27 Mapreduce Configuration Table 28 Mapreduce Input

10 x Tables Table 29 Mapreduce Output Table 30 Mapreduce Error Output Table 31 Pig Configuration Table 32 Pig UDF Table 33 Pig Input Table 34 Pig Output Table 35 Pig Error Output Table 36 WaitForJobCompletion Configuration Table 37 WaitForJobCompletion Input Table 38 WaitForJobCompletion Output Table 39 WaitForJobCompletion Error Output Table 40 The DemoWorkflow Process Activities Table 41 The ErrorHandler Process Activities Table 42 Log Properties

11 xi Preface TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data, integrated with TIBCO ActiveMatrix BusinessWorks, addresses the needs of the TIBCO ActiveMatrix BusinessWorks customers who want a user-friendly way to deal with big data. It is assumed that you are familiar with the basic concepts of the TIBCO environment and big data. Topics Related Documentation, page xii Typographical Conventions, page xiii Connecting with TIBCO Resources, page xv

12 xii Related Documentation Related Documentation This section lists documentation resources you may find useful. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Documentation The following document form the TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data documentation set: TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Installation Read this manual for instructions on site preparation and installation. Read this manual for instructions on using this product. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Release Notes Read the release notes for a list of new features. This document also contains a list of known issues. Other TIBCO Product Documentation You may find it useful to read the documentation for the following TIBCO products: TIBCO Designer TIBCO Administrator TIBCO ActiveMatrix BusinessWorks TIBCO Rendezvous TIBCO Hawk TIBCO Runtime Agent

13 Preface xiii Typographical Conventions The following typographical conventions are used in this manual. Table 1 General Typographical Conventions Convention ENV_NAME TIBCO_HOME Use TIBCO products are installed into an installation environment. A product installed into an installation environment does not access components in other installation environments. Incompatible products and multiple instances of the same product must be installed into different installation environments. An installation environment consists of the following properties: Name Identifies the installation environment. This name is referenced in documentation as ENV_NAME. On Microsoft Windows, the name is appended to the name of Windows services created by the installer and is a component of the path to the product shortcut in the Windows Start > All Programs menu. Path The folder into which the product is installed. This folder is referenced in documentation as TIBCO_HOME. TIB_BWBD_HOME code font TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data installs into a directory within a TIBCO_HOME. This directory is referenced in documentation as TIB_BWBD_HOME. The default value of TIB_BWBD_HOME depends on the operating system. For example on Windows systems, the default value is C:\tibco\bw\plugins\bigdata. Code font identifies commands, code examples, filenames, pathnames, and output displayed in a command window. For example: Use MyCommand to start the foo process. bold code font Bold code font is used in the following ways: In procedures, to indicate what a user types. For example: Type admin. In large code samples, to indicate the parts of the sample that are of particular interest. In command syntax, to indicate the default parameter for a command. For example, if no parameter is specified, MyCommand is enabled: MyCommand [enable disable]

14 xiv Typographical Conventions Table 1 General Typographical Conventions (Cont d) Convention italic font Key combinations Use Italic font is used in the following ways: To indicate a document title. For example: See TIBCO ActiveMatrix BusinessWorks Concepts. To introduce new terms. For example: A portal page may contain several portlets. Portlets are mini-applications that run in a portal. To indicate a variable in a command or code syntax that you must replace. For example: MyCommand PathName Key name separated by a plus sign indicate keys pressed simultaneously. For example: Ctrl+C. Key names separated by a comma and space indicate keys pressed one after the other. For example: Esc, Ctrl+Q. The note icon indicates information that is of special interest or importance, for example, an additional action required only in certain circumstances. The tip icon indicates an idea that could be useful, for example, a way to apply the information provided in the current section to achieve a specific result. The warning icon indicates the potential for a damaging situation, for example, data loss or corruption if certain steps are taken or not taken.

15 Preface xv Connecting with TIBCO Resources How to Join TIBCOmmunity TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCOmmunity offers forums, blogs, and access to a variety of resources. To register, go to How to Access TIBCO Documentation You can access TIBCO documentation here: How to Contact TIBCO Support For comments or problems with this manual or the software it addresses, contact TIBCO Support as follows: For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site: If you already have a valid maintenance or support contract, visit this site: Entry to this site requires a user name and password. If you do not have a user name, you can request one.

16 xvi Connecting with TIBCO Resources

17 1 Chapter 1 Introduction This chapter gives an overview of big data and TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Topics Overview of the Plug-in, page 2

18 2 Chapter 1 Introduction Overview of the Plug-in TIBCO BusinessWorks is an easy-to-use integration software that provides world-class integration technology in a rapidly deployable solution that manages the entire life cycle of integration projects. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data plugged into TIBCO BusinessWorks, allows you to use the established non-code approach to integrate with Hadoop family projects, such as Hadoop Distributed File System (HDFS). Before running operations by using this plug-in, ensure that the user has appropriate permissions on HDFS server. The plug-in contains the following two palettes: The HDFS palette allows you to do some operations on the files in HDFS. The HDFSConnection shared resource contains all necessary parameters to connect to HDFS. The HDFS Operation activity is used to do basic operations on files in HDFS. The ListFileStatus activity is used to list the status of the specified file or directory. The Read activity is used to read data from a file in HDFS. The Write activity is used to write data to a file in HDFS. The Hadoop palette allows you to utilize the benefits of Hive, Mapreduce, and Pig based on Hadoop. The HCatalogConnection shared resource contains all the necessary parameters to connect to HCatalog. The Database shared resource is an administrative container for a set of tables. The Table shared resource provides shared virtual storage for data. The Hive activity is used to facilitate querying and managing large datasets residing in distributed storage. The Mapreduce activity is used to create and queue a standard Mapreduce job or a streaming Mapreduce job. The Pig activity is used to create and queue a Pig job. The WaitForJobCompletion activity is used to wait for the specified job to complete until it reaches the specified value of timeout.

19 3 Chapter 2 HDFS Palette and Hadoop Palette This chapter specifies the HDFS palette and Hadoop palette. After you go through this chapter, you will be familiar with the functions provided by TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Topics Palette Overview, page 4 HDFSConnection, page 6 HDFS Operation, page 7 ListFileStatus, page 10 Read, page 12 Write, page 15 HCatalogConnection, page 18 Database, page 20 Table, page 21 Hive, page 23 Mapreduce, page 26 Pig, page 30 WaitForJobCompletion, page 34

20 4 Chapter 2 HDFS Palette and Hadoop Palette Palette Overview This plug-in contains two palettes: HDFS palette and Hadoop palette. The HDFS palette includes one shared resource and four activities, as shown in Figure 1. Figure 1 HDFS Palette Shared resource: HDFSConnection Activities: HDFS Operation ListFileStatus Read Write The Hadoop palette includes three shared resources and four activities, as shown in Figure 2. Figure 2 HCatalog Palette

21 Palette Overview 5 Shared resources: HCatalogConnection Database Table Activities: Hive Mapreduce Pig WaitForJobCompletion

22 6 Chapter 2 HDFS Palette and Hadoop Palette HDFSConnection Shared Resource The HDFSConnection shared resource contains all necessary parameters to connect to HDFS. It can be used by the HDFS Operation, ListFileStatus, Read, Write activities and the HCatalogConnection shared resource. Configuration The Configuration tab contains the following fields. Table 2 HDFSConnection Configuration Field Global Var? Description Name No The name of the shared resource. The default value is HDFSConnection. Description No A short description of the shared resource. Click to add the description on the Description page. HDFS Url Yes The web HDFS URL to connect to HDFS. The format is User Name Yes A unique user name to identify each user connecting to HDFS. Test Connection Button Clicking Test Connection allows you to test whether the specified configuration fields result in a valid connection to HDFS for design time configuration and runtime execution of the activities.

23 HDFS Operation 7 HDFS Operation Activity The HDFS Operation activity is used to do basic operations on files in HDFS, including copying local files to HDFS, copying files from HDFS to local system, and renaming or deleting files in HDFS. Folders cannot be copied between HDFS and local system. File names containing space characters in HDFS cannot be operated by all activities. Configuration The Configuration tab contains the following fields. Table 3 HDFS Operation Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is HDFS Operation. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes An HDFSConnection shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. HDFSOperation Yes Specifies the HDFS operation. PUT_LOCAL_TO_HDFS: Copy local files to HDFS. GET_HDFS_TO_LOCAL: Copy files from HDFS to the local system. RENAME_HDFS: Rename files in HDFS. DELETE_FROM_HDFS: Delete files from HDFS.

24 8 Chapter 2 HDFS Palette and Hadoop Palette Input The input of this activity varies depending on the HDFS operation you chose in the Configuration tab. Table 4 specifies the possible input of the activity. Table 4 HDFS Operation Input Input Item Data Type Description HDFS sourcefilepath string Specifies the path of the source file. destinationfilepath string Specifies the path of the destination file. overwrite boolean Specifies whether to overwrite the existing file if a file that has the same name already exists in the specified destination path, 1 (true) or 0 (false). blocksize integer Specifies the block size of the file. The value in this field must be greater than 0. replication integer Specifies the number of replications of the file. The value in this field must be greater than 0. permission integer Specifies the permission of the file. The value in this field must be in the range 0 to 777. offset integer Specifies the starting byte position. The value in this field must be 0 or greater. length integer Specifies the number of bytes to be processed. buffersize integer Specifies the size of the buffer used in transferring data. The value in this field must be 0 or greater. recursive boolean Specifies whether to operate on the content in the subdirectories, 1 (true) or 0 (false).

25 HDFS Operation 9 Output The output for the activity is as follows Table 5 HDFS Operation Output Output Item Data Type Description HDFS status integer Returns standard HTTP status code to indicate whether the execution has succeeded or not. msg string Returns the execution message. Error Output The Error Output tab lists the exceptions that can be thrown by this activity Table 6 HDFS Operation Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

26 10 Chapter 2 HDFS Palette and Hadoop Palette ListFileStatus Activity The ListFileStatus activity is used to list the status of the specified file or directory. Regarding the specified directory, the activity returns the status of all files and directories in the specified directory. Configuration The Configuration tab contains the following fields. Table 7 ListFileStatus Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is ListFileStatus. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Input Table 8 specifies the input of the activity. Table 8 ListFileStatus Input Input Item path Description Specifies the path of a file or a directory. Note: If you specify an empty folder in this field, the Output tab will return null.

27 ListFileStatus 11 Output The output for the activity is as follows. Table 9 ListFileStatus Output Output Item Data Type Description fileinfo accesstime integer Returns the access time of the file in milliseconds. blocksize integer Returns the block size of the file. length integer Returns the number of bytes in a file. modificationtime integer Returns the modification time of the file in milliseconds. replication integer Returns the number of replications of the file. owner string Returns the user name of the file owner. type string Returns the type of the path object, FILE or DIRECTORY. group string Returns the group associated with the file. permission string Returns the permission of the file. pathsuffix string Returns the path suffix of the file. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 10 ListFileStatus Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

28 12 Chapter 2 HDFS Palette and Hadoop Palette Read Activity The Read activity is used to read data from a file in HDFS and place its content into the Output tab of the activity. Due to the limitations of TIBCO ActiveMatrix BusinessWorks, this activity cannot read more than 2 GB data at one time. You can use the group to iteratively read the data in a file of more than 2 GB. See TIBCO ActiveMatrix BusinessWorks Process Design Guide for details about the group. Configuration The Configuration tab contains the following fields. Table 11 Read Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Read. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Read As Yes Specifies in what format the file to be read as, text or binary.

29 Read 13 Input Table 12 specifies the input of the activity. Table 12 Read Input Input Item filename offset length buffersize Description Specifies the path of the file to be read. Specifies the starting byte position to be read. The value in this field must be greater than 0. Specifies the number of bytes to be read. Specifies the size of the buffer used in transferring data. The value in this field must be greater than 0. Output The output for the activity is as follows. Table 13 ListFileStatus Output Output Item Data Type Description filecontent binarycontent string Returns the file content in binary format. Note: This item is displayed when binary is selected in the Read As field in the Configuration tab. textcontent string Returns the file content in text format. Note: This item is displayed when text is selected in the Read As field in the Configuration tab. end boolean Returns whether the file has been read to the end. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 14 Read Error Output HDFSException msg Description The error message description returned by the plug-in.

30 14 Chapter 2 HDFS Palette and Hadoop Palette Table 14 Read Error Output (Cont d) HDFSException msgcode exception message javaclassname Description The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

31 Write 15 Write Activity The Write activity is used to write data to a specified file in HDFS. Configuration The Configuration tab contains the following fields. Table 15 Write Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Write. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Append Yes Specifies whether to append the data to an existing file. Note: Data cannot be appended to the same file in HDFS by multiple processes simultaneously. Write From Yes Specifies the format of the data to be written. text binary file StreamObject

32 16 Chapter 2 HDFS Palette and Hadoop Palette Table 15 Write Configuration (Cont d) Field Global Var? Description The following field is displayed when the append check box is cleared. Overwrite Yes Specifies whether to overwrite the existing data in the specified file. Input The input of this activity varies depending on the data format you chose in the Configuration tab. Table 16 specifies the possible input of the activity. Table 16 Write Input Input Item filename filecontent binarydata sourcefilepath inputstreamobject overwrite blocksize replication Description Specifies the path of the file to be written in. Specifies the content to be written in the file. Note: This field is displayed when text is selected in the Write From field in the Configuration tab. Specifies the binary data to be written in the file. Note: This field is displayed when binary is selected in the Write From field in the Configuration tab. Specifies the path of the file to be written. Note: This field is displayed when file is selected in the Write From field in the Configuration tab. Specifies the streaming object to be written. Note: This field is displayed when StreamObject is selected in the Write From field in the Configuration tab. Specifies whether to overwrite the existing data in the source file. Specifies the block size of the file. The value in this field must be greater than 0. Specifies the number of replications of the file. The value in this field must be greater than 0.

33 Write 17 Table 16 Write Input (Cont d) Input Item permission buffersize Description Specifies the permission of the file. The value in this field must be in the range 0 to 777. Specifies the size of the buffer used in transferring data. The value in this field must be greater than 0. Output The output for the activity is as follows. Table 17 Write Output Output Item Data Type Description status integer Returns standard HTTP status code to indicate whether the execution has succeeded or not. msg string Returns the execution message. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 18 Write Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

34 18 Chapter 2 HDFS Palette and Hadoop Palette HCatalogConnection Shared Resource The HCatalogConnection shared resource contains all the necessary parameters to connect to HCatalog. It can be used by the Hive, Mapreduce, Pig, and WaitForJobCompletion activities. Configuration The Configuration tab contains the following fields. Table 19 HCatalogConnection Configuration Field Global Var? Description Name No The name of the shared resource. The default value is HCatalogConnection. Description No A short description of the shared resource. Click to add the description on the Description page. HCatalog Url Yes The web HCatalog URL to connect to HCatalog. The format is User Name Yes A unique user name to identify each user connecting to the HCatalog. HDFS Connection Yes An HDFSConnection shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Test Connection Button Clicking Test Connection allows you to test whether the specified configuration fields result in a valid connection to HCatalog for design time configuration and runtime execution of the activities.

35 HCatalogConnection 19 Introspect Database Button When you click Introspect Database, the plug-in introspects all the databases that currently exist in the specified HCatalog. When you select one of them from the Introspect Databases list, the plug-in generates a Database shared resource with definitions. Introspected databases are read-only.

36 20 Chapter 2 HDFS Palette and Hadoop Palette Database Shared Resource The Database shared resource is an administrative container for a set of tables. Creating new databases is not supported by this plug-in temporarily. You can introspect the existing databases in the specified HCatalog. See Introspect Database Button on page 19 for more information. Configuration The Configuration tab contains the following fields. Table 20 Database Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Database. Description No A short description of the activity. Click to add the description on the Description page. Database Location Database Comment Database Properties No No No Specifies the directory where the database is located. Specifies comments for the database. Specifies properties for the database. A property is associated with a name and a value. Introspect Table Button When you click Introspect Table, the plug-in introspects all tables that currently exist in the specified database. When you select one of them from the Introspect Tables list, the plug-in generates a Table shared resource with definitions. Introspected tables are read-only.

37 Table 21 Table Shared Resource The Table shared resource provides shared virtual storage for data. It is a shared entity that can be accessed by multiple applications concurrently, each one of these has the same coherent view of the data contained in the table. Creating new tables is not supported by this plug-in temporarily. You can introspect the existing tables in the specified database. See Introspect Table Button on page 20 for more information. Configuration The Configuration tab contains the following fields. Table 21 Table Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Table. Description No A short description of the activity. Click to add the description on the Description page. Comment No Specifies comments for the table. Location No Specifies the directory where the table is located. Table Columns No Specifies columns of the table. A column is associated with a name, a type, an advancetype, and a Comment. Note: The advancetype column indicates the type of complex data type.

38 22 Chapter 2 HDFS Palette and Hadoop Palette Advanced The Advanced tab contains the following fields. Table 22 Table Advanced Field Global Var? Description Partitioned No Specifies whether the table is partitioned. Output Format No Specifies the output format of the table. Owner No Specifies the owner of the table. Input Format No Specifies the input format of the table. Permission No Specifies the permission of the table. Group No Specifies the group associated with the table. External No Specifies the external table to be created. Partition Columns Table Properties No No Specifies the partition columns of the table. A partition column is associated with a name, a type, and a comment. Specifies the properties of the table. A table property is associated with a name and a value. Generate Schema Clicking Generate Schema allows you to generate the schema of a table. After clicking Generate Schema, input the name of the schema file, which is stored in the BigdataTableSchema folder in your project.

39 Hive 23 Hive Activity The Hive activity is used to facilitate querying and managing large datasets residing in distributed storage. This activity do not support to upload Hive data from local clusters and file systems. Configuration The Configuration tab contains the following fields. Table 23 Hive Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Hive. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. IsFileBase Yes Select this check box if Hive scripts are from a file. Hive File Yes Specifies the path of the file containing Hive scripts. Note: This field is displayed when the IsFileBase check box is selected.

40 24 Chapter 2 HDFS Palette and Hadoop Palette Table 23 Hive Configuration (Cont d) Field Global Var? Description Hive Editor Yes Specifies Hive scripts. The keywords of the scripts are highlighted automatically. Click to open the Hive Editor page that can be magnified. Note: This field is displayed when the IsFileBase check box is cleared. Define Yes Specifies Hive configuration variables. A variable is associated with a name and a value. Click to add variables. Status Directory Yes Specifies the directory where the status of the Hive job is located. Wait for Hive Result Yes Select this check box if you want the process to wait for the Hive operation to complete. Note: When this check box is selected, the Hive activity does not support to query more than 2 GB result data at one time due to the limitations of TIBCO ActiveMatrix BusinessWorks. Input The values specified in this tab takes precedence over the ones in the corresponding fields in the Configuration tab. Table 24 specifies the possible input of the activity. Table 24 Hive Input Input Item HiveFile HiveScript Description Specifies path of the HDFS file that contains commands. Note: This item is displayed when the IsFileBase check box is selected. Specifies Hive scripts directly. Note: This item is displayed when the IsFileBase check box is cleared. Define(s) Name Specifies the name of the Hive configuration variable. Value Specifies the value of the Hive configuration variable. StatusDirectory Specifies the directory where the status of the Hive job is located.

41 Hive 25 Output The output for the activity is as follows. Table 25 Hive Output Output Item Data Type Description jobid string Returns the job ID of the Hive operation. Note: This item is displayed when the Wait for Hive Result check box is cleared. You can use the WaitForJobCompletion activity to wait for the job to complete. The exitvalue in the Output tab of the WaitForJobCompletion activity shows the exit value of Hive SQL execution. content string Returns the result of the job. Note: This item is displayed when the Wait for Hive Result check box is selected. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 26 Hive Error Output HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in.

42 26 Chapter 2 HDFS Palette and Hadoop Palette Mapreduce Activity The Mapreduce activity is used to create and queue a standard Mapreduce job or a streaming Mapreduce job. Configuration The Configuration tab contains the following fields. Table 27 Mapreduce Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Mapreduce. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. Streaming Yes Select this check box to create and run Mapreduce streaming jobs. The following four fields are displayed when the Streaming check box is selected. Input Yes Specifies the path of the input data in Hadoop. Output Yes Specifies the path of the output data. Mapper Yes Specifies the path of the mapper program in Hadoop. Reducer Yes Specifies the path of the reducer program in Hadoop. The following four fields are displayed when the Streaming check box is cleared.

43 Mapreduce 27 Table 27 Mapreduce Configuration (Cont d) Field Global Var? Description JarName Yes Specifies the name of the.jar file for Mapreduce to use. ClassName Yes Specifies the name of the class for Mapreduce to use. Libjars Yes Specifies the comma separated.jar file to be included in the classpath. Files Yes Specifies the comma separated.jar files to be copied to the Mapreduce cluster. StatusDirectory Yes Specifies the directory where the status of Mapreduce jobs are stored. Arguments Yes Specifies the program arguments. If the Streaming check box is cleared, specify Java main class arguments. If the Streaming check box is selected, specify a list of arguments that contain space-separated strings to pass to the Hadoop streaming utility. For example, - files /user/hdfs/file - D mapred.reduce.task=0 - inputformat org.apache.hadoop.mapred.lib.nlineinputformat - cmdenv info=wc-reducer The following field is displayed when the Streaming check box is cleared. Define Yes Specifies the Hadoop configuration variables. A variable is associated with a name and a value.

44 28 Chapter 2 HDFS Palette and Hadoop Palette Input The values specified in this tab take precedence over the ones in the corresponding fields in the Configuration tab. Table 28 specifies the possible input of the activity. Table 28 Mapreduce Input Input Item Description The following four fields are displayed when the Streaming check box is selected. Input Output Mapper Reducer Specifies the path of the input data in Hadoop. Specifies the path where the output data is stored. Specifies the path of the mapper program in Hadoop. Specifies the path of the reducer program in Hadoop. The following four fields are displayed when the Streaming check box is cleared. JarName ClassName Libjars Files StatusDirectory Specifies the name of the.jar file for Mapreduce to use. Specifies the name of the class for Mapreduce to use. Specifies the comma separated.jar file to be included in the classpath. Specifies the comma separated.jar files to be copied to the Mapreduce cluster. Specifies the directory where the status of Mapreduce jobs are stored. Arguments Argument Specifies the program argument. The following fields are displayed when the Streaming check box is cleared. Define Name Specifies the name of Hadoop configuration variables. Value Specifies the value of Hadoop configuration variables.

45 Mapreduce 29 Output The output for the activity is as follows. Table 29 Mapreduce Output Output Item Data Type Description jobid string Returns the job ID of the Mapreduce operation. Note: You can use the WaitForJobCompletion activity to wait for the job to complete. The exitvalue in the Output tab of the WaitForJobCompletion activity shows the exit value of Mapreduce execution. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 30 Mapreduce Error Output HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in.

46 30 Chapter 2 HDFS Palette and Hadoop Palette Pig Activity The Pig activity is used to create and queue a Pig job. Configuration The Configuration tab contains the following fields. Table 31 Pig Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Pig. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. IsFileBase Yes Select this check box if Pig scripts are from a file. Pig File Yes Specifies the path of the file contains Pig scripts. Note: This field is displayed when the IsFileBase check box is selected. Pig Editor Yes Specifies Pig scripts. The keywords of the scripts are highlighted automatically. Click to open the Pig Editor page that can be magnified. Note: This field is displayed when the IsFileBase check box is cleared.