TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data User s Guide

Transcription

1 TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data User s Guide Software Release 1.0 November 2013 Two-Second Advantage

2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE. USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE LICENSE FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME. This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc. TIBCO, Two-Second Advantage, TIBCO ActiveMatrix BusinessWorks, TIBCO Designer, TIBCO Administrator, TIBCO Rendezvous, TIBCO Runtime Agent, and TIBCO Hawk are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. Enterprise Java Beans (EJB), Java Platform Enterprise Edition (Java EE), Java 2 Platform Enterprise Edition (J2EE), and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle Corporation, Inc. in the U.S. and other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only. THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM. THIS DOCUMENT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME. THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES. Copyright 2013 TIBCO Software Inc. ALL RIGHTS RESERVED. TIBCO Software Inc. Confidential Information

3 iii Contents Figures vii Tables ix Preface xi Related Documentation xii TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Documentation xii Other TIBCO Product Documentation xii Typographical Conventions xiii Connecting with TIBCO Resources xv How to Join TIBCOmmunity xv How to Access TIBCO Documentation xv How to Contact TIBCO Support xv Chapter 1 Introduction Overview of the Plug-in Chapter 2 HDFS Palette and Hadoop Palette Palette Overview HDFSConnection Configuration HDFS Operation Configuration Input Output Error Output ListFileStatus Configuration Input Output Error Output Read Configuration Input Output

4 iv Contents Error Output Write Configuration Input Output Error Output HCatalogConnection Configuration Database Configuration Table Configuration Advanced Hive Configuration Input Output Error Output Mapreduce Configuration Input Output Error Output Pig Configuration UDF Input Output Error Output WaitForJobCompletion Configuration Input Output Error Output Chapter 3 Getting Started Overview Creating a Project Configuring an HCatalogConnection Shared Resource Creating a Process Adding Activities to a Process

5 Contents v Testing a Process Deploying a Project Chapter 4 Using the Sample Project Overview of the Sample Project DataCollectionAndQuery Project Process Description Setting Up the Sample Project Running the Sample Project Expected Result Appendix A Managing Big Data Logs Managing Logs Appendix B Trace Messages HDFS Error Messages Big Data Error Messages Appendix C Configuration for Amazon EMR Configuring for Amazon EMR Cluster Index

6 vi Contents

7 Figures vii Figures Figure 1 HDFS Palette Figure 2 HCatalog Palette Figure 3 TIBCO Designer Start Screen Figure 4 TIBCO Designer Save Project Screen Figure 5 The DemoWorkflow Process Figure 6 The ErrorHandler Process Figure 7 The DemoWorkflow Process Output Figure 8 The ErrorHandler Process Result

8 viii Figures

9 Tables ix Tables Table 1 General Typographical Conventions xiii Table 2 HDFSConnection Configuration Table 3 HDFS Operation Configuration Table 4 HDFS Operation Input Table 5 HDFS Operation Output Table 6 HDFS Operation Error Output Table 7 ListFileStatus Configuration Table 8 ListFileStatus Input Table 9 ListFileStatus Output Table 10 ListFileStatus Error Output Table 11 Read Configuration Table 12 Read Input Table 13 ListFileStatus Output Table 14 Read Error Output Table 15 Write Configuration Table 16 Write Input Table 17 Write Output Table 18 Write Error Output Table 19 HCatalogConnection Configuration Table 20 Database Configuration Table 21 Table Configuration Table 22 Table Advanced Table 23 Hive Configuration Table 24 Hive Input Table 25 Hive Output Table 26 Hive Error Output Table 27 Mapreduce Configuration Table 28 Mapreduce Input

10 x Tables Table 29 Mapreduce Output Table 30 Mapreduce Error Output Table 31 Pig Configuration Table 32 Pig UDF Table 33 Pig Input Table 34 Pig Output Table 35 Pig Error Output Table 36 WaitForJobCompletion Configuration Table 37 WaitForJobCompletion Input Table 38 WaitForJobCompletion Output Table 39 WaitForJobCompletion Error Output Table 40 The DemoWorkflow Process Activities Table 41 The ErrorHandler Process Activities Table 42 Log Properties

11 xi Preface TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data, integrated with TIBCO ActiveMatrix BusinessWorks, addresses the needs of the TIBCO ActiveMatrix BusinessWorks customers who want a user-friendly way to deal with big data. It is assumed that you are familiar with the basic concepts of the TIBCO environment and big data. Topics Related Documentation, page xii Typographical Conventions, page xiii Connecting with TIBCO Resources, page xv

12 xii Related Documentation Related Documentation This section lists documentation resources you may find useful. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Documentation The following document form the TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data documentation set: TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Installation Read this manual for instructions on site preparation and installation. Read this manual for instructions on using this product. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data Release Notes Read the release notes for a list of new features. This document also contains a list of known issues. Other TIBCO Product Documentation You may find it useful to read the documentation for the following TIBCO products: TIBCO Designer TIBCO Administrator TIBCO ActiveMatrix BusinessWorks TIBCO Rendezvous TIBCO Hawk TIBCO Runtime Agent

13 Preface xiii Typographical Conventions The following typographical conventions are used in this manual. Table 1 General Typographical Conventions Convention ENV_NAME TIBCO_HOME Use TIBCO products are installed into an installation environment. A product installed into an installation environment does not access components in other installation environments. Incompatible products and multiple instances of the same product must be installed into different installation environments. An installation environment consists of the following properties: Name Identifies the installation environment. This name is referenced in documentation as ENV_NAME. On Microsoft Windows, the name is appended to the name of Windows services created by the installer and is a component of the path to the product shortcut in the Windows Start > All Programs menu. Path The folder into which the product is installed. This folder is referenced in documentation as TIBCO_HOME. TIB_BWBD_HOME code font TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data installs into a directory within a TIBCO_HOME. This directory is referenced in documentation as TIB_BWBD_HOME. The default value of TIB_BWBD_HOME depends on the operating system. For example on Windows systems, the default value is C:\tibco\bw\plugins\bigdata. Code font identifies commands, code examples, filenames, pathnames, and output displayed in a command window. For example: Use MyCommand to start the foo process. bold code font Bold code font is used in the following ways: In procedures, to indicate what a user types. For example: Type admin. In large code samples, to indicate the parts of the sample that are of particular interest. In command syntax, to indicate the default parameter for a command. For example, if no parameter is specified, MyCommand is enabled: MyCommand [enable disable]

14 xiv Typographical Conventions Table 1 General Typographical Conventions (Cont d) Convention italic font Key combinations Use Italic font is used in the following ways: To indicate a document title. For example: See TIBCO ActiveMatrix BusinessWorks Concepts. To introduce new terms. For example: A portal page may contain several portlets. Portlets are mini-applications that run in a portal. To indicate a variable in a command or code syntax that you must replace. For example: MyCommand PathName Key name separated by a plus sign indicate keys pressed simultaneously. For example: Ctrl+C. Key names separated by a comma and space indicate keys pressed one after the other. For example: Esc, Ctrl+Q. The note icon indicates information that is of special interest or importance, for example, an additional action required only in certain circumstances. The tip icon indicates an idea that could be useful, for example, a way to apply the information provided in the current section to achieve a specific result. The warning icon indicates the potential for a damaging situation, for example, data loss or corruption if certain steps are taken or not taken.

15 Preface xv Connecting with TIBCO Resources How to Join TIBCOmmunity TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCOmmunity offers forums, blogs, and access to a variety of resources. To register, go to How to Access TIBCO Documentation You can access TIBCO documentation here: How to Contact TIBCO Support For comments or problems with this manual or the software it addresses, contact TIBCO Support as follows: For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site: If you already have a valid maintenance or support contract, visit this site: Entry to this site requires a user name and password. If you do not have a user name, you can request one.

16 xvi Connecting with TIBCO Resources

17 1 Chapter 1 Introduction This chapter gives an overview of big data and TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Topics Overview of the Plug-in, page 2

18 2 Chapter 1 Introduction Overview of the Plug-in TIBCO BusinessWorks is an easy-to-use integration software that provides world-class integration technology in a rapidly deployable solution that manages the entire life cycle of integration projects. TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data plugged into TIBCO BusinessWorks, allows you to use the established non-code approach to integrate with Hadoop family projects, such as Hadoop Distributed File System (HDFS). Before running operations by using this plug-in, ensure that the user has appropriate permissions on HDFS server. The plug-in contains the following two palettes: The HDFS palette allows you to do some operations on the files in HDFS. The HDFSConnection shared resource contains all necessary parameters to connect to HDFS. The HDFS Operation activity is used to do basic operations on files in HDFS. The ListFileStatus activity is used to list the status of the specified file or directory. The Read activity is used to read data from a file in HDFS. The Write activity is used to write data to a file in HDFS. The Hadoop palette allows you to utilize the benefits of Hive, Mapreduce, and Pig based on Hadoop. The HCatalogConnection shared resource contains all the necessary parameters to connect to HCatalog. The Database shared resource is an administrative container for a set of tables. The Table shared resource provides shared virtual storage for data. The Hive activity is used to facilitate querying and managing large datasets residing in distributed storage. The Mapreduce activity is used to create and queue a standard Mapreduce job or a streaming Mapreduce job. The Pig activity is used to create and queue a Pig job. The WaitForJobCompletion activity is used to wait for the specified job to complete until it reaches the specified value of timeout.

19 3 Chapter 2 HDFS Palette and Hadoop Palette This chapter specifies the HDFS palette and Hadoop palette. After you go through this chapter, you will be familiar with the functions provided by TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Topics Palette Overview, page 4 HDFSConnection, page 6 HDFS Operation, page 7 ListFileStatus, page 10 Read, page 12 Write, page 15 HCatalogConnection, page 18 Database, page 20 Table, page 21 Hive, page 23 Mapreduce, page 26 Pig, page 30 WaitForJobCompletion, page 34

20 4 Chapter 2 HDFS Palette and Hadoop Palette Palette Overview This plug-in contains two palettes: HDFS palette and Hadoop palette. The HDFS palette includes one shared resource and four activities, as shown in Figure 1. Figure 1 HDFS Palette Shared resource: HDFSConnection Activities: HDFS Operation ListFileStatus Read Write The Hadoop palette includes three shared resources and four activities, as shown in Figure 2. Figure 2 HCatalog Palette

21 Palette Overview 5 Shared resources: HCatalogConnection Database Table Activities: Hive Mapreduce Pig WaitForJobCompletion

22 6 Chapter 2 HDFS Palette and Hadoop Palette HDFSConnection Shared Resource The HDFSConnection shared resource contains all necessary parameters to connect to HDFS. It can be used by the HDFS Operation, ListFileStatus, Read, Write activities and the HCatalogConnection shared resource. Configuration The Configuration tab contains the following fields. Table 2 HDFSConnection Configuration Field Global Var? Description Name No The name of the shared resource. The default value is HDFSConnection. Description No A short description of the shared resource. Click to add the description on the Description page. HDFS Url Yes The web HDFS URL to connect to HDFS. The format is User Name Yes A unique user name to identify each user connecting to HDFS. Test Connection Button Clicking Test Connection allows you to test whether the specified configuration fields result in a valid connection to HDFS for design time configuration and runtime execution of the activities.

23 HDFS Operation 7 HDFS Operation Activity The HDFS Operation activity is used to do basic operations on files in HDFS, including copying local files to HDFS, copying files from HDFS to local system, and renaming or deleting files in HDFS. Folders cannot be copied between HDFS and local system. File names containing space characters in HDFS cannot be operated by all activities. Configuration The Configuration tab contains the following fields. Table 3 HDFS Operation Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is HDFS Operation. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes An HDFSConnection shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. HDFSOperation Yes Specifies the HDFS operation. PUT_LOCAL_TO_HDFS: Copy local files to HDFS. GET_HDFS_TO_LOCAL: Copy files from HDFS to the local system. RENAME_HDFS: Rename files in HDFS. DELETE_FROM_HDFS: Delete files from HDFS.

24 8 Chapter 2 HDFS Palette and Hadoop Palette Input The input of this activity varies depending on the HDFS operation you chose in the Configuration tab. Table 4 specifies the possible input of the activity. Table 4 HDFS Operation Input Input Item Data Type Description HDFS sourcefilepath string Specifies the path of the source file. destinationfilepath string Specifies the path of the destination file. overwrite boolean Specifies whether to overwrite the existing file if a file that has the same name already exists in the specified destination path, 1 (true) or 0 (false). blocksize integer Specifies the block size of the file. The value in this field must be greater than 0. replication integer Specifies the number of replications of the file. The value in this field must be greater than 0. permission integer Specifies the permission of the file. The value in this field must be in the range 0 to 777. offset integer Specifies the starting byte position. The value in this field must be 0 or greater. length integer Specifies the number of bytes to be processed. buffersize integer Specifies the size of the buffer used in transferring data. The value in this field must be 0 or greater. recursive boolean Specifies whether to operate on the content in the subdirectories, 1 (true) or 0 (false).

25 HDFS Operation 9 Output The output for the activity is as follows Table 5 HDFS Operation Output Output Item Data Type Description HDFS status integer Returns standard HTTP status code to indicate whether the execution has succeeded or not. msg string Returns the execution message. Error Output The Error Output tab lists the exceptions that can be thrown by this activity Table 6 HDFS Operation Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

26 10 Chapter 2 HDFS Palette and Hadoop Palette ListFileStatus Activity The ListFileStatus activity is used to list the status of the specified file or directory. Regarding the specified directory, the activity returns the status of all files and directories in the specified directory. Configuration The Configuration tab contains the following fields. Table 7 ListFileStatus Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is ListFileStatus. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Input Table 8 specifies the input of the activity. Table 8 ListFileStatus Input Input Item path Description Specifies the path of a file or a directory. Note: If you specify an empty folder in this field, the Output tab will return null.

27 ListFileStatus 11 Output The output for the activity is as follows. Table 9 ListFileStatus Output Output Item Data Type Description fileinfo accesstime integer Returns the access time of the file in milliseconds. blocksize integer Returns the block size of the file. length integer Returns the number of bytes in a file. modificationtime integer Returns the modification time of the file in milliseconds. replication integer Returns the number of replications of the file. owner string Returns the user name of the file owner. type string Returns the type of the path object, FILE or DIRECTORY. group string Returns the group associated with the file. permission string Returns the permission of the file. pathsuffix string Returns the path suffix of the file. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 10 ListFileStatus Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

28 12 Chapter 2 HDFS Palette and Hadoop Palette Read Activity The Read activity is used to read data from a file in HDFS and place its content into the Output tab of the activity. Due to the limitations of TIBCO ActiveMatrix BusinessWorks, this activity cannot read more than 2 GB data at one time. You can use the group to iteratively read the data in a file of more than 2 GB. See TIBCO ActiveMatrix BusinessWorks Process Design Guide for details about the group. Configuration The Configuration tab contains the following fields. Table 11 Read Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Read. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Read As Yes Specifies in what format the file to be read as, text or binary.

29 Read 13 Input Table 12 specifies the input of the activity. Table 12 Read Input Input Item filename offset length buffersize Description Specifies the path of the file to be read. Specifies the starting byte position to be read. The value in this field must be greater than 0. Specifies the number of bytes to be read. Specifies the size of the buffer used in transferring data. The value in this field must be greater than 0. Output The output for the activity is as follows. Table 13 ListFileStatus Output Output Item Data Type Description filecontent binarycontent string Returns the file content in binary format. Note: This item is displayed when binary is selected in the Read As field in the Configuration tab. textcontent string Returns the file content in text format. Note: This item is displayed when text is selected in the Read As field in the Configuration tab. end boolean Returns whether the file has been read to the end. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 14 Read Error Output HDFSException msg Description The error message description returned by the plug-in.

30 14 Chapter 2 HDFS Palette and Hadoop Palette Table 14 Read Error Output (Cont d) HDFSException msgcode exception message javaclassname Description The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

31 Write 15 Write Activity The Write activity is used to write data to a specified file in HDFS. Configuration The Configuration tab contains the following fields. Table 15 Write Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Write. Description No A short description of the activity. Click to add the description on the Description page. HDFS SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Select one to apply to your current activity. Append Yes Specifies whether to append the data to an existing file. Note: Data cannot be appended to the same file in HDFS by multiple processes simultaneously. Write From Yes Specifies the format of the data to be written. text binary file StreamObject

32 16 Chapter 2 HDFS Palette and Hadoop Palette Table 15 Write Configuration (Cont d) Field Global Var? Description The following field is displayed when the append check box is cleared. Overwrite Yes Specifies whether to overwrite the existing data in the specified file. Input The input of this activity varies depending on the data format you chose in the Configuration tab. Table 16 specifies the possible input of the activity. Table 16 Write Input Input Item filename filecontent binarydata sourcefilepath inputstreamobject overwrite blocksize replication Description Specifies the path of the file to be written in. Specifies the content to be written in the file. Note: This field is displayed when text is selected in the Write From field in the Configuration tab. Specifies the binary data to be written in the file. Note: This field is displayed when binary is selected in the Write From field in the Configuration tab. Specifies the path of the file to be written. Note: This field is displayed when file is selected in the Write From field in the Configuration tab. Specifies the streaming object to be written. Note: This field is displayed when StreamObject is selected in the Write From field in the Configuration tab. Specifies whether to overwrite the existing data in the source file. Specifies the block size of the file. The value in this field must be greater than 0. Specifies the number of replications of the file. The value in this field must be greater than 0.

33 Write 17 Table 16 Write Input (Cont d) Input Item permission buffersize Description Specifies the permission of the file. The value in this field must be in the range 0 to 777. Specifies the size of the buffer used in transferring data. The value in this field must be greater than 0. Output The output for the activity is as follows. Table 17 Write Output Output Item Data Type Description status integer Returns standard HTTP status code to indicate whether the execution has succeeded or not. msg string Returns the execution message. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 18 Write Error Output HDFSException msg msgcode exception message javaclassname Description The error message description returned by the plug-in. The error code returned by the plug-in. Occurs when the plug-in has internal errors. The error message returned by the server. The name of the Java Class where an error occurred.

34 18 Chapter 2 HDFS Palette and Hadoop Palette HCatalogConnection Shared Resource The HCatalogConnection shared resource contains all the necessary parameters to connect to HCatalog. It can be used by the Hive, Mapreduce, Pig, and WaitForJobCompletion activities. Configuration The Configuration tab contains the following fields. Table 19 HCatalogConnection Configuration Field Global Var? Description Name No The name of the shared resource. The default value is HCatalogConnection. Description No A short description of the shared resource. Click to add the description on the Description page. HCatalog Url Yes The web HCatalog URL to connect to HCatalog. The format is User Name Yes A unique user name to identify each user connecting to the HCatalog. HDFS Connection Yes An HDFSConnection shared resource. Click to open the Select A Resource page. All usable HDFSConnection shared resources are listed on the page. Test Connection Button Clicking Test Connection allows you to test whether the specified configuration fields result in a valid connection to HCatalog for design time configuration and runtime execution of the activities.

35 HCatalogConnection 19 Introspect Database Button When you click Introspect Database, the plug-in introspects all the databases that currently exist in the specified HCatalog. When you select one of them from the Introspect Databases list, the plug-in generates a Database shared resource with definitions. Introspected databases are read-only.

36 20 Chapter 2 HDFS Palette and Hadoop Palette Database Shared Resource The Database shared resource is an administrative container for a set of tables. Creating new databases is not supported by this plug-in temporarily. You can introspect the existing databases in the specified HCatalog. See Introspect Database Button on page 19 for more information. Configuration The Configuration tab contains the following fields. Table 20 Database Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Database. Description No A short description of the activity. Click to add the description on the Description page. Database Location Database Comment Database Properties No No No Specifies the directory where the database is located. Specifies comments for the database. Specifies properties for the database. A property is associated with a name and a value. Introspect Table Button When you click Introspect Table, the plug-in introspects all tables that currently exist in the specified database. When you select one of them from the Introspect Tables list, the plug-in generates a Table shared resource with definitions. Introspected tables are read-only.

37 Table 21 Table Shared Resource The Table shared resource provides shared virtual storage for data. It is a shared entity that can be accessed by multiple applications concurrently, each one of these has the same coherent view of the data contained in the table. Creating new tables is not supported by this plug-in temporarily. You can introspect the existing tables in the specified database. See Introspect Table Button on page 20 for more information. Configuration The Configuration tab contains the following fields. Table 21 Table Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Table. Description No A short description of the activity. Click to add the description on the Description page. Comment No Specifies comments for the table. Location No Specifies the directory where the table is located. Table Columns No Specifies columns of the table. A column is associated with a name, a type, an advancetype, and a Comment. Note: The advancetype column indicates the type of complex data type.

38 22 Chapter 2 HDFS Palette and Hadoop Palette Advanced The Advanced tab contains the following fields. Table 22 Table Advanced Field Global Var? Description Partitioned No Specifies whether the table is partitioned. Output Format No Specifies the output format of the table. Owner No Specifies the owner of the table. Input Format No Specifies the input format of the table. Permission No Specifies the permission of the table. Group No Specifies the group associated with the table. External No Specifies the external table to be created. Partition Columns Table Properties No No Specifies the partition columns of the table. A partition column is associated with a name, a type, and a comment. Specifies the properties of the table. A table property is associated with a name and a value. Generate Schema Clicking Generate Schema allows you to generate the schema of a table. After clicking Generate Schema, input the name of the schema file, which is stored in the BigdataTableSchema folder in your project.

39 Hive 23 Hive Activity The Hive activity is used to facilitate querying and managing large datasets residing in distributed storage. This activity do not support to upload Hive data from local clusters and file systems. Configuration The Configuration tab contains the following fields. Table 23 Hive Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Hive. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. IsFileBase Yes Select this check box if Hive scripts are from a file. Hive File Yes Specifies the path of the file containing Hive scripts. Note: This field is displayed when the IsFileBase check box is selected.

40 24 Chapter 2 HDFS Palette and Hadoop Palette Table 23 Hive Configuration (Cont d) Field Global Var? Description Hive Editor Yes Specifies Hive scripts. The keywords of the scripts are highlighted automatically. Click to open the Hive Editor page that can be magnified. Note: This field is displayed when the IsFileBase check box is cleared. Define Yes Specifies Hive configuration variables. A variable is associated with a name and a value. Click to add variables. Status Directory Yes Specifies the directory where the status of the Hive job is located. Wait for Hive Result Yes Select this check box if you want the process to wait for the Hive operation to complete. Note: When this check box is selected, the Hive activity does not support to query more than 2 GB result data at one time due to the limitations of TIBCO ActiveMatrix BusinessWorks. Input The values specified in this tab takes precedence over the ones in the corresponding fields in the Configuration tab. Table 24 specifies the possible input of the activity. Table 24 Hive Input Input Item HiveFile HiveScript Description Specifies path of the HDFS file that contains commands. Note: This item is displayed when the IsFileBase check box is selected. Specifies Hive scripts directly. Note: This item is displayed when the IsFileBase check box is cleared. Define(s) Name Specifies the name of the Hive configuration variable. Value Specifies the value of the Hive configuration variable. StatusDirectory Specifies the directory where the status of the Hive job is located.

41 Hive 25 Output The output for the activity is as follows. Table 25 Hive Output Output Item Data Type Description jobid string Returns the job ID of the Hive operation. Note: This item is displayed when the Wait for Hive Result check box is cleared. You can use the WaitForJobCompletion activity to wait for the job to complete. The exitvalue in the Output tab of the WaitForJobCompletion activity shows the exit value of Hive SQL execution. content string Returns the result of the job. Note: This item is displayed when the Wait for Hive Result check box is selected. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 26 Hive Error Output HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in.

42 26 Chapter 2 HDFS Palette and Hadoop Palette Mapreduce Activity The Mapreduce activity is used to create and queue a standard Mapreduce job or a streaming Mapreduce job. Configuration The Configuration tab contains the following fields. Table 27 Mapreduce Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Mapreduce. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. Streaming Yes Select this check box to create and run Mapreduce streaming jobs. The following four fields are displayed when the Streaming check box is selected. Input Yes Specifies the path of the input data in Hadoop. Output Yes Specifies the path of the output data. Mapper Yes Specifies the path of the mapper program in Hadoop. Reducer Yes Specifies the path of the reducer program in Hadoop. The following four fields are displayed when the Streaming check box is cleared.

43 Mapreduce 27 Table 27 Mapreduce Configuration (Cont d) Field Global Var? Description JarName Yes Specifies the name of the.jar file for Mapreduce to use. ClassName Yes Specifies the name of the class for Mapreduce to use. Libjars Yes Specifies the comma separated.jar file to be included in the classpath. Files Yes Specifies the comma separated.jar files to be copied to the Mapreduce cluster. StatusDirectory Yes Specifies the directory where the status of Mapreduce jobs are stored. Arguments Yes Specifies the program arguments. If the Streaming check box is cleared, specify Java main class arguments. If the Streaming check box is selected, specify a list of arguments that contain space-separated strings to pass to the Hadoop streaming utility. For example, - files /user/hdfs/file - D mapred.reduce.task=0 - inputformat org.apache.hadoop.mapred.lib.nlineinputformat - cmdenv info=wc-reducer The following field is displayed when the Streaming check box is cleared. Define Yes Specifies the Hadoop configuration variables. A variable is associated with a name and a value.

44 28 Chapter 2 HDFS Palette and Hadoop Palette Input The values specified in this tab take precedence over the ones in the corresponding fields in the Configuration tab. Table 28 specifies the possible input of the activity. Table 28 Mapreduce Input Input Item Description The following four fields are displayed when the Streaming check box is selected. Input Output Mapper Reducer Specifies the path of the input data in Hadoop. Specifies the path where the output data is stored. Specifies the path of the mapper program in Hadoop. Specifies the path of the reducer program in Hadoop. The following four fields are displayed when the Streaming check box is cleared. JarName ClassName Libjars Files StatusDirectory Specifies the name of the.jar file for Mapreduce to use. Specifies the name of the class for Mapreduce to use. Specifies the comma separated.jar file to be included in the classpath. Specifies the comma separated.jar files to be copied to the Mapreduce cluster. Specifies the directory where the status of Mapreduce jobs are stored. Arguments Argument Specifies the program argument. The following fields are displayed when the Streaming check box is cleared. Define Name Specifies the name of Hadoop configuration variables. Value Specifies the value of Hadoop configuration variables.

45 Mapreduce 29 Output The output for the activity is as follows. Table 29 Mapreduce Output Output Item Data Type Description jobid string Returns the job ID of the Mapreduce operation. Note: You can use the WaitForJobCompletion activity to wait for the job to complete. The exitvalue in the Output tab of the WaitForJobCompletion activity shows the exit value of Mapreduce execution. Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 30 Mapreduce Error Output HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in.

46 30 Chapter 2 HDFS Palette and Hadoop Palette Pig Activity The Pig activity is used to create and queue a Pig job. Configuration The Configuration tab contains the following fields. Table 31 Pig Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is Pig. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. IsFileBase Yes Select this check box if Pig scripts are from a file. Pig File Yes Specifies the path of the file contains Pig scripts. Note: This field is displayed when the IsFileBase check box is selected. Pig Editor Yes Specifies Pig scripts. The keywords of the scripts are highlighted automatically. Click to open the Pig Editor page that can be magnified. Note: This field is displayed when the IsFileBase check box is cleared.

47 Pig 31 Table 31 Pig Configuration (Cont d) Field Global Var? Description Argument Yes Specifies Pig arguments containing space-separated string. Click to add arguments. Status Directory Yes Specifies the directory where the status of the Pig job is located. Files Yes Specifies the comma separated files to be copied to the Mapreduce cluster. UDF The UDF tab contains the following fields. Table 32 Pig UDF Field Global Var? Description UDF Directory Yes Specifies the directory where user define functions (UDF) is located. The format is hdfs://namenode_hostname:port/filepath. Available UDF Files Upload UDF File Yes Yes Specifies the UDF file to be applied. Click to list available UDF files under the specified UDF directory. Select a UDF file and click, the UDF file is displayed in the Hive Editor field in the Configuration tab. Specifies the UDF file to be uploaded. Click to open the Select File page. Select the UDF file to be uploaded, and then click Upload to upload the file to the specified directory.

48 32 Chapter 2 HDFS Palette and Hadoop Palette Input The values specified in this tab takes precedence over the ones in the corresponding fields in the Configuration tab. Table 33 specifies the possible input of the activity. Table 33 Pig Input Input Item PigScript PigFile Description Specifies Pig scripts. Note: This item is displayed when the IsFileBase check box is cleared. Specifies the comma separated files to be copied to the Mapreduce cluster. Note: This item is displayed when the IsFileBase check box is selected. Arguments Argument Specifies Pig arguments. StatusDirectory Files Specifies the directory where the status of the Pig job is located. Specifies the comma separated files to be copied to the Mapreduce cluster. Output The output for the activity is as follows. Table 34 Pig Output Output Item Data Type Description jobid string Returns the job ID of the Pig operation. Note: You can use the WaitForJobCompletion activity to wait for the job to complete. The exitvalue in the Output tab of the WaitForJobCompletion activity shows the exit value of Pig scripts execution. Error Output The Error Output tab lists the exceptions that can be thrown by this activity.

49 Pig 33 Table 35 Pig Error Output HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in.

50 34 Chapter 2 HDFS Palette and Hadoop Palette WaitForJobCompletion Activity The WaitForJobCompletion activity is used to wait for the specified jobs to complete until it reaches the specified value of timeout. Configuration The Configuration tab contains the following fields. Table 36 WaitForJobCompletion Configuration Field Global Var? Description Name No The name of the activity in the process definition. The default value is WaitForJobCompletion. Description No A short description of the activity. Click to add the description on the Description page. SharedResource Yes Specifies a shared resource. Click to open the Select A Resource page. All usable HCatalogConnection shared resources are listed on the page. Select one to apply to your current activity. Timeout Yes Specifies how long to wait for a job to complete (in milliseconds). The default value is milliseconds. Interval Yes Specifies the time interval in milliseconds, to check the job execution. The default value is 5000 milliseconds.

51 WaitForJobCompletion 35 Input Table 37 specifies the input of the activity. Table 37 WaitForJobCompletion Input Input Item jobid timeout interval Description Specifies the ID of the job to be waited for. Specifies how long to wait for a job to complete (in milliseconds). Note: It overrides the value in the Timeout field in the Configuration tab at runtime. Specifies the time interval in milliseconds to check the job execution. Note: It overrides the value in the Interval field in the Configuration tab at runtime. Output The output for the activity is as follows. Table 38 WaitForJobCompletion Output Output Item Job Data Type Description status starttime string Returns the start time of the job. username string Returns the user name of the job owner. jobid string Returns the ID of the job. jobacls string Returns ACLs for the job. schedulingin fo string Returns the scheduling information associated with the job. failureinfo string Returns the reason for the failure of the job. jobid string Returns the ID of the job. jobpriority string Returns the priority of the job. runstate string Returns the current running state of the job.

52 36 Chapter 2 HDFS Palette and Hadoop Palette Table 38 WaitForJobCompletion Output Output Item Data Type Description State string Returns the state of the job. jobcomplete string Returns the completion status of the job. profile url string Returns the URL of the job. jobid string Returns the ID of the job. user string Returns the user name of the job owner. queuename string Returns the queue name of Mapreduce. jobfile string Returns the name of the job file. jobname string Returns the name of the job. jobid string Returns the ID of the job. id string Returns the ID of the job. parentid string Returns the parent ID of the job. percentco mplete string Returns the completion percentage of the job. exitvalue string Returns the exit value of the job. If a value of 0 is returned, it indicates that the job has been successful. Otherwise, the job completed with errors. Note: You can get error messages of the job from the StatusDirectory\stderr directory. StatusDirectory is the value specified in the Status Directory field in the Configuration tab of the activity that executed the job. The ErrorHandler process in the sample project shipped with the installer shows how to get error messages of a job. See Process Description on page 51 for more information. user string Returns the user name of the job owner. callback string Returns the callback URL. completed string Returns the completion status of the job. For example, done.

53 WaitForJobCompletion 37 Error Output The Error Output tab lists the exceptions that can be thrown by this activity. Table 39 WaitForJobCompletion Error Output ActivityErrors HadoopException msg msgcode Description The error message description returned by the plug-in. The error code returned by the plug-in. ActivityTimedOutException msg msgcode The error message description returned by TIBCO ActiveMatrix BusinessWorks. The error code returned by the TIBCO ActiveMatrix BusinessWorks.

54 38 Chapter 2 HDFS Palette and Hadoop Palette

55 39 Chapter 3 Getting Started This chapter specifies the basic steps that are required to configure and run TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data in TIBCO Designer. Topics Overview, page 40 Creating a Project, page 41 Configuring an HCatalogConnection Shared Resource, page 43 Creating a Process, page 44 Adding Activities to a Process, page 45 Testing a Process, page 46 Deploying a Project, page 47

56 40 Chapter 3 Getting Started Overview TIBCO ActiveMatrix BusinessWorks is a scalable, extensible, and easy to use integration platform that allows you to develop and test integration projects. TIBCO ActiveMatrix BusinessWorks includes a graphical user interface, TIBCO Designer for defining business processes, and an engine that executes the processes. For detailed information about how to configure processes, see TIBCO Designer documentation, which can be accessed from the Help > Designer Help menu option in TIBCO Designer. A typical configuration session contains the following steps: 1. Creating a Project, page Configuring an HCatalogConnection Shared Resource, page Creating a Process, page Adding Activities to a Process, page Testing a Process, page Deploying a Project, page 47

57 Creating a Project 41 Creating a Project The TIBCO Designer GUI is used to build a project. When starting TIBCO Designer, you must create or select a project. A project contains the configuration files that define options used during runtime. To create a project, complete the following steps: 1. Start TIBCO Designer. Execute one of the following platform-specific commands to open the TIBCO Designer window: On Microsoft Windows Click Start > All Programs > TIBCO > TIBCO Designer version_number > TIBCO Designer. Or From a command window, run TIBCO_HOME\designer\version_number\bin\designer.exe. On UNIX From a command window, run TIBCO_HOME/designer/version_number/bin/designer. 2. Click New Empty Project on the TIBCO Designer page, as shown in Figure 3. The Save Project page is displayed. Figure 3 TIBCO Designer Start Screen

58 42 Chapter 3 Getting Started 3. Click in the Multi-File Project tab. Navigate to the location where you intend to save the project and specify a name for the project. Click OK to continue, as shown in Figure 4. Figure 4 TIBCO Designer Save Project Screen The TIBCO Designer GUI appears with the newly created project.

59 Configuring an HCatalogConnection Shared Resource 43 Configuring an HCatalogConnection Shared Resource After creating the project, you need to add an HCatalogConnection shared resource and configure it to set up the connection between the HCatalog and the plug-in. After dragging the HCatalogConnection shared resource from the Hadoop Palette panel to the design panel, you need to configure each field in the Configuration tab. See HCatalogConnection on page 18 for details about the shared resource.

60 44 Chapter 3 Getting Started Creating a Process After getting the metadata in the TIBCO Designer window, you need to create a process to deal with certain workflows. To create a process, complete the following steps: 1. Select the root directory in the project panel. 2. Expand the Process palette in the palette panel and drag the Process Definition icon into the design panel. 3. Configure the process. For more information, see TIBCO Designer User s Guide. 4. Click Apply to save the configuration.

61 Adding Activities to a Process 45 Adding Activities to a Process After creating the process, you can add the plug-in activities to it. To add an activity to the process, complete the following steps: 1. Click the process that was previously created in Creating a Process on page 44. The Start and End activities appear in the design panel. 2. Expand the Hadoop palette in the palette panel and drag one or more activities to the design panel. You can also add activities from other palettes to the process, for example, File activities. If the Hadoop palette is not shown in the palette panel, select Palettes > Activities > Hadoop from the menu bar to make it visible. 3. Create transitions between activities in the design panel. 4. Configure each activity in the process.

62 46 Chapter 3 Getting Started Testing a Process After creating and configuring the process, you can test it in TIBCO Designer. This allows you to check if the process works properly before you deploy it. See TIBCO BusinessWorks Process Design Guide for detailed information on using test mode.

63 Deploying a Project 47 Deploying a Project To deploy a project that you have configured, you need to generate an Enterprise Archive file (EAR file) that contains configurations for the process definition you want to deploy. You can upload the archive to TIBCO Administrator to deploy the associated application. See TIBCO ActiveMatrix BusinessWorks Administration for more information. Before deploying a project of this plug-in, make sure that the repository location specified in TIBCO Designer is consistent with the directory where the repository file is located in the machine that you plan to deploy on. The following is the basic procedure: 1. Save the project in TIBCO Designer window. 2. Create an enterprise archive and then create a process archive in it. 3. Add processes defined in the project into the process archive. 4. Build the archive by clicking under the Enterprise Archive panel. When you click the Build Archive, TIBCO Designer generates an enterprise archive (.ear) file that you can then deploy from TIBCO Administrator. 5. Start TIBCO Administrator and create an application related to the project. 6. Deploy the application and start the process.

64 48 Chapter 3 Getting Started

65 49 Chapter 4 Using the Sample Project This chapter describes the sample project packaged with TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Working through the sample project helps you to understand how TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data works. Topics Overview of the Sample Project, page 50 DataCollectionAndQuery Project, page 51

66 50 Chapter 4 Using the Sample Project Overview of the Sample Project TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data packages one sample project and one sample data file within the installer. These sample files are located in the TIB_BWBD_HOME\examples\DataCollectionAndQuery directory. DataCollectionAndQueryProj This folder contains the DataCollectionAndQuery project. The sample project shows how to use TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data to manage and query data, for example, how to operate files in HDFS, transfer files between HDFS and local directory, create Hive Table, and query data. SampleData This folder contains the customers.csv sample file. It can be used as a sample object of operation.

67 DataCollectionAndQuery Project 51 DataCollectionAndQuery Project The DataCollectionAndQuery sample project shows how to use TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data to manage and query data, for example, how to operate files in HDFS, transfer files between HDFS and local directory, create Hive Table, and query data. Process Description This project contains a main process and a subprocess. The DemoWorkflow process is the main process, as shown in Figure 5. Figure 5 The DemoWorkflow Process The activities in this process are described in Table 40. Table 40 The DemoWorkflow Process Activities Activity Timer RemoveExistingFiles CopyDataToHDFS CreateHiveTableRefertToData Description Starts the process when the specified time interval expires. Deletes the customers.csv file from the user\hdfs\bwdemo directory in HDFS. Copies the customers.csv file from the TIB_BWBD_HOME\ examples\datacollectionandquery\sampledata directory to the user\hdfs\bwdemo directory in HDFS. Creates an external Hive Table named customers in HDFS.

68 52 Chapter 4 Using the Sample Project Table 40 The DemoWorkflow Process Activities (Cont d) Activity CheckError HiveQuery Generate Error Catch End Description Performs the subprocess. If no error occurs during the ErrorHandler process works, the process goes to the HiveQuery activity. If any error occurs, the process goes to the Generate Error activity. Queries 100 records from the customers.csv Hive table created by the CreateHiveTableRefertToData activity. Generates error messages from the Output tab of the ErrorHandler subprocess. Catches error messages and displays in the Output tab when any error occurs in the DemoWorkflow process. Ends the process. The DemoWorkflow process also includes the ErrorHandler subprocess, as shown in Figure 6. Figure 6 The ErrorHandler Process

69 DataCollectionAndQuery Project 53 The activities in this process are described in Table 41. Table 41 The ErrorHandler Process Activities Activity Start WaitForJobCompletion Activity ReadResult ReadError Generate Error Catch Render XML End Description Starts the process. Waits for the CreateHiveTableRefertToData activity to complete the job. If the job in the CreateHiveTableRefertToData activity is completed, the process goes to the ReadResult activity. If any error occurs, the process goes to the ReadError activity. Reads the output of the job executed by the CreateHiveTableRefertToData activity, which is in the user\hdfs\hive\createstatus\stdout directory. Reads error messages of the job executed by the CreateHiveTableRefertToData activity, which is in the user\hdfs\hive\createstatus\stderr directory. Generates error messages from the Output tab of the ReadError activity. Catches the error messages and displays in the Output tab when any error occurs in the ErrorHandler subprocess. Renders the error messages in XML string. Ends the process. Setting Up the Sample Project To set up the project, follow these tasks: Task A Open the Project 1. Start TIBCO Designer. 2. Click Open Existing Project from the Project menu. 3. Click the Multi-File Project tab on the Open Project page. 4. Specify the sample project directory in the Project Directory field as follows:

70 54 Chapter 4 Using the Sample Project TIB_BWBD_HOME\examples\DataCollectionAndQuery\DataCollectionAn dqueryproj. 5. Click OK. The DataCollectionAndQuery project is displayed in the Project panel. Task B Configure the Project and Test the Connection 1. Configure the required global variables: HCATALOG_URL=localhost_IP HDFS_URL=localhost_IP Project_Path=TIB_BWBD_HOME\examples\DataCollectionAndQuery 2. Configure the optional global variables: CUSTOMER_HDFSFILE refers to the sample file that you delete from HDFS and copy from local directory to HDFS. HiveQueryStatus refers to the status directory of the HiveQuery activity. HiveStatus refers to the status directory of the CreateHiveTableRefertToData activity. HiveTableCdrCustomers refers to the name of Hive table create by the CreateHiveTableRefertToData activity. UserPath refers to the User HDFS path. The default value is user\hdfs. The hdfs variable should be consistent with the user name specified in the HDFSConnection shared resource. 3. Test the connection in shared resources to confirm the URLs are valid: Click Test Connection in the Configuration tab of the HCatalogConnection and HDFSConnection SharedResources. Running the Sample Project After setting up the project, complete the following steps to run the sample project: 1. Click the Tester tab on the left of the Project panel to open the Test panel. 2. Click. The Selected Processes To Load page is displayed. 3. Select the DemoWorkflow process and click Load Selected. The process starts. 4. Click to stop the testing and return to Design mode. See TIBCO BusinessWorks Process Design Guide for detailed information on using test mode and setting breakpoints.

71 DataCollectionAndQuery Project 55 Expected Result If the process performs successfully, 100 records in the customers.csv file that is created by the CreateHiveTableRefertToData activity are returned in the Output tab of the HiveQuery activity, as shown in Figure 7. Figure 7 The DemoWorkflow Process Output Figure 8 shows the expected result of the ErrorHandler process. Figure 8 The ErrorHandler Process Result

72 56 Chapter 4 Using the Sample Project

73 57 Appendix A Managing Big Data Logs This appendix describes how to manage Big Data logs. Topics Managing Logs, page 58

74 58 Appendix A Managing Big Data Logs Managing Logs Table 42 Log Properties Logs are used to trace and troubleshoot the plug-in exceptions at design time and runtime. You can specify the log level, the log filename, and the output destination for trace messages. To check debug information, see log properties listed in Table 42. See the TIBCO ActiveMatrix BusinessWorks documentation for detailed information about logs. Property com.tibco.plugin.bigdata.debug Description The debug role properties of the plug-in. The default value is False. Only when bwengine property Trace.Debug.*=true and com.tibco.bigdata.debug=true, it will display debug info.

75 59 Appendix B Trace Messages This appendix lists the error messages used by TIBCO ActiveMatrix BusinessWorks Plug-in for Big Data. Topics HDFS Error Messages, page 60 Big Data Error Messages, page 63

76 60 Appendix B Trace Messages HDFS Error Messages BW-HDFS : Please specify HDFS connection shared resource. Role: errorrole Category: BW-Plugin Description: The HDFSConnection shared resource is not specified in the activity. Resolution: Specify the HDFSConnection shared resource. BW-HDFS : Remote Error while run %1. Role: errorrole Category: BW-Plugin Description: Errors occurred in the HDFS server. Resolution: No action. BW-HDFS : Error Occurred: %1. Role: errorrole Category: BW-Plugin Description: Describes the general error information. Resolution: No action. BW-HDFS : Redirect URL cannot be empty. Role: errorrole Category: BW-Plugin Description: Redirect URL cannot be empty. Resolution: No action.

77 HDFS Error Messages 61 BW-HDFS : Invalid HDFS URL, Please specify a correct one. Role: errorrole Category: BW-Plugin Description: Incorrect HDFS URL parameters cause an invalid connection to the HDFS. Resolution: Specify correct HDFS URL. BW-HDFS : %1 is not a valid buffer size, valid buffer size should > 0. Role: errorrole Category: BW-Plugin Description: The specified value is not a valid buffer size. The buffer size should be greater than 0. Resolution: Specify a value greater than 0. BW-HDFS : %1 is not a valid replication, valid replication should > 0. Role: errorrole Category: BW-Plugin Description: The specified value is not a valid replication. The replication value should be greater than 0. Resolution: Specify a value greater than 0. BW-HDFS : %1 is not a valid offset, valid offset should >= 0. Role: errorrole Category: BW-Plugin Description: The specified value is not a valid offset. The offset value should be 0 or greater. Resolution: Specify a value greater than or equal to 0.

78 62 Appendix B Trace Messages BW-HDFS : %1 is not a valid length, valid length should >= 0. Role: errorrole Category: BW-Plugin Description: The specified value is not a valid length. The length should be 0 or greater. Resolution: Specify a value greater than or equal to 0. BW-HDFS : %1 is not a valid block size, valid block size should > 0. Role: errorrole Category: BW-Plugin Description: The specified value is not a valid block size. The block size should be greater than 0. Resolution: Specify a value greater than 0. BW-HDFS : %1 is not a valid permission, valid values should Role: errorrole Category: BW-Plugin Description: The specified value is not a valid permission. The permission value should be in the range Resolution: Specify a value in the range

79 Big Data Error Messages 63 Big Data Error Messages BW-BigData : Please specify HCatalog connection. Role: errorrole Category: BW-Plugin Description: The HCatalogConnection shared resource is not specified in the activity. Resolution: Specify the HCatalogConnection shared resource. BW-BigData : Remote Error while run %1. Role: errorrole Category: BW-Plugin Description: Errors occurred in the HCatalog server. Resolution: No action. BW-BigData : Error Occurred: %1. Role: errorrole Category: BW-Plugin Description: Describes the general error messages. Resolution: No action. BW-BigData : HCatalog URL cannot be empty. Role: errorrole Category: BW-Plugin Description: The HCatalog URL field cannot be empty. Resolution: Specify the HCatalog web URL.

80 64 Appendix B Trace Messages BW-BigData : Redirect URL is null, please check. Role: errorrole Category: BW-Plugin Description: The redirect URL is null. Resolution: Specify the redirect URL. BW-BigData : Hive error occurred: %1. Role: errorrole Category: BW-Plugin Description: Errors occurred in Hive. Resolution: No action. BW-BigData : Introspect database error occurred: %1. Role: errorrole Category: BW-Plugin Description: Errors occurred when introspecting database. Resolution: No action. BW-BigData : Introspect table error occurred: %1. Role: errorrole Category: BW-Plugin Description: Errors occurred when introspecting table. Resolution: No action. BW-BigData : Please specify UDF file path. Role: errorrole Category: BW-Plugin Description: Specify the UDF file path in the Pig activity.

81 Big Data Error Messages 65 Resolution:.Specify the UDF file path in the UDF Directory field in the UDF tab. BW-BigData : UDF Path cannot empty, please specify UDF file path. Role: errorrole Category: BW-Plugin Description: The UDF Path cannot be empty. Resolution: Specify the UDF file path. BW-BigData : UDF Path invalid, valid UDF path should like: hdfs://namenodehostname:port/tmp/.. Role: errorrole Category: BW-Plugin Description: The specified UDF path is invalid. Resolution:.Specify a valid UDF path. BW-BigData : Invalid HDFS Connection %1, please specify one. Role: errorrole Category: BW-Plugin Description: The specified HDFS connection is invalid. Resolution:.Specify a valid HDFS connection. BW-BigData : HDFS URL is empty, please specify one. Role: errorrole Category: BW-Plugin Description: HDFS URL is empty. Resolution:.Specify an HDFS web URL.

82 66 Appendix B Trace Messages BW-BigData : Status directory is empty, please specify one. Role: errorrole Category: BW-Plugin Description: The Status Directory field is empty. Resolution:.Specify a status directory. BW-BigData : No database found. Role: errorrole Category: BW-Plugin Description: No tables are introspected. Resolution: No action. BW-BigData : No table found. Role: errorrole Category: BW-Plugin Description: No tables are introspected. Resolution: No action. BW-BigData : %1. Role: errorrole Category: BW-Plugin Description: Describes the general Error information. Resolution: No action. BW-BigData : Hive script cannot be null. Role: errorrole Category: BW-Plugin

83 Big Data Error Messages 67 Description: The Hive Editor field cannot be empty if the IsFileBase check box is cleared. Resolution: Specify Hive script. BW-BigData : Hive file script cannot be null, please select one script file. Role: errorrole Category: BW-Plugin Description: The Hive File field cannot be empty if the IsFileBase check box is selected. Resolution: Specify the file contains Hive scripts. BW-BigData : Pig script cannot be null. Role: errorrole Category: BW-Plugin Description: The Pig Editor field cannot be empty if the IsFileBase check box is cleared. Resolution: Specify Pig script. BW-BigData : Pig file script cannot be null, please select one script file. Role: errorrole Category: BW-Plugin Description: The Pig File field cannot be empty if the IsFileBase check box is selected. Resolution: Specify the file contains Pig scripts. BW-BigData : Mapreduce jar name is required, please provide one. Role: errorrole Category: BW-Plugin Description: The Jar Name field is requied if the Streaming check box is cleared.

84 68 Appendix B Trace Messages Resolution: Specify the Jar name in the Jar Name field. BW-BigData : Mapreduce main class is required, please provide one. Role: errorrole Category: BW-Plugin Description: The Main Class field is requied if the Streaming check box is cleared. Resolution: Specify the main class in the Main Class field. BW-BigData : Mapreduce streaming input is required, please provide one. Role: errorrole Category: BW-Plugin Description: The Input field is requied if the Streaming check box is selected. Resolution: Specify the input in the Input field. BW-BigData : Mapreduce streaming mapper is required, please provide one. Role: errorrole Category: BW-Plugin Description: The Mapper field is requied if the Streaming check box is selected. Resolution: Specify the mapper in the Mapper field. BW-BigData : Mapreduce streaming reducer is required, please provide one. Role: errorrole Category: BW-Plugin Description: The Reducer field is requied if the Streaming check box is selected. Resolution: Specify the reducer in the Reducer field.

85 Big Data Error Messages 69 BW-BigData : Generate schema error occurred: %1. Role: errorrole Category: BW-Plugin Description: Errors occurred when generating schema of table. Resolution: No action. BW-BigData : Invalid name node host or port, please verify. Role: errorrole Category: BW-Plugin Description: The node host or port is invalid. Resolution: Specify a valid node host or port. BW-BigData : %1 Role: inforole Category: BW-Plugin Description: Describes the general Info information. Resolution: No action. BW-BigData : %1 is done (%2ms) Role: inforole Category: BW-Plugin Description: Indicates the activity is done successfully in [%2] milliseconds. Resolution: No action.

86 70 Appendix B Trace Messages

87 71 Appendix C Configuration for Amazon EMR This appendix describes how to configurate for the Amazon EMR cluster. Topics Configuring for Amazon EMR Cluster, page 72

88 72 Appendix C Configuration for Amazon EMR Configuring for Amazon EMR Cluster If you run this plug-in on the Amazon Elastic MapReduce (Amazon EMR) cluster, you need to complete the following configuration: 1. Add the following property to the designer.tra file and bwenging.tra file: java.property.com.tibco.plugin.bigdata.resolve.hosts.file.enabl e=true 2. Add the following information of DNS and IP to the hosts file. public_ip public_dns internal_dns internal_ip For example, ec compute-1.amazonaws.com ip ec2.internal If you do not have permission to modify the hosts file, follow these steps: a. Create a file containing the above information of DNS and IP. b. Add the following property in the designer.tra file and bwenging.tra file. java.property.com.tibco.plugin.bigdata.resolve.hosts.file.lo cation=filepath/filename

89 73 Index A activities HDFS Operation 7 Hive 23 ListFileStatus 10 Mapreduce 26 Pig 30 Read 12 WaitForJobCompletion 34 Write 15 append 15 argument 32 arguments 27, 28, 31 HDFS url 6 HDFSOperation 7 I interval 34, 35 L Logs 58 B blocksize 8, 11 buffersize 8, 13, 17 M mapper 26, 28 msg 9, 11, 13, 17, 25, 29, 33, 37, 37 msgcode 9, 11, 14, 17, 25, 29, 33, 37, 37 C customer support xv O offset 8, 13 E error messages 59 H HCatalog url 18 P palette Hadoop palette 4, 4 HDFS palette 4, 4 palette overview 4 pathsuffix 11 permission 8, 11

90 74 Index R recursive 8 reducer 26, 28 replication 8, 11 S shared resource 6, 18 Database 20 HCatalogConnection 18 HDFSConnection 6 Table 21 streaming 26 support, contacting xv T technical support xv TIBCO ActiveMatrix BusinessWorks 40 TIBCO Designer 40 TIBCO_HOME xiii timeout 34, 35 U UDF 31