Informatica Cloud (Version Winter 2015) Hadoop Connector Guide

Transcription

1 Informatica Cloud (Version Winter 2015) Hadoop Connector Guide

2 Informatica Cloud Hadoop Connector Guide Version Winter 2015 March 2015 Copyright (c) Informatica LLC. All rights reserved. This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S. and/or international Patents and other Patents Pending. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us in writing. Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange Informatica On Demand, Informatica Identity Resolution, Informatica Application Information Lifecycle Management, Informatica Complex Event Processing, Ultra Messaging and Informatica Master Data Management are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rights reserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rights reserved.copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright Isomorphic Software. All rights reserved. Copyright Meta Integration Technology, Inc. All rights reserved. Copyright Intalio. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. All rights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved. Copyright Rogue Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright Glyph & Cog, LLC. All rights reserved. Copyright Thinkmap, Inc. All rights reserved. Copyright Clearpace Software Limited. All rights reserved. Copyright Information Builders, Inc. All rights reserved. Copyright OSS Nokalva, Inc. All rights reserved. Copyright Edifecs, Inc. All rights reserved. Copyright Cleo Communications, Inc. All rights reserved. Copyright International Organization for Standardization All rights reserved. Copyright ejtechnologies GmbH. All rights reserved. Copyright Jaspersoft Corporation. All rights reserved. Copyright International Business Machines Corporation. All rights reserved. Copyright yworks GmbH. All rights reserved. Copyright Lucent Technologies. All rights reserved. Copyright (c) University of Toronto. All rights reserved. Copyright Daniel Veillard. All rights reserved. Copyright Unicode, Inc. Copyright IBM Corp. All rights reserved. Copyright MicroQuill Software Publishing, Inc. All rights reserved. Copyright PassMark Software Pty Ltd. All rights reserved. Copyright LogiXML, Inc. All rights reserved. Copyright Lorenzi Davide, All rights reserved. Copyright Red Hat, Inc. All rights reserved. Copyright The Board of Trustees of the Leland Stanford Junior University. All rights reserved. Copyright EMC Corporation. All rights reserved. Copyright Flexera Software. All rights reserved. Copyright Jinfonet Software. All rights reserved. Copyright Apple Inc. All rights reserved. Copyright Telerik Inc. All rights reserved. Copyright BEA Systems. All rights reserved. Copyright PDFlib GmbH. All rights reserved. Copyright Orientation in Objects GmbH. All rights reserved. Copyright Tanuki Software, Ltd. All rights reserved. Copyright Ricebridge. All rights reserved. Copyright Sencha, Inc. All rights reserved. Copyright Scalable Systems, Inc. All rights reserved. Copyright jqwidgets. All rights reserved. This product includes software developed by the Apache Software Foundation ( and/or other software which is licensed under various versions of the Apache License (the "License"). You may obtain a copy of these Licenses at Unless required by applicable law or agreed to in writing, software distributed under these Licenses is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the Licenses for the specific language governing permissions and limitations under the Licenses. This product includes software which was developed by Mozilla ( software copyright The JBoss Group, LLC, all rights reserved; software copyright by Bruno Lowagie and Paulo Soares and other software which is licensed under various versions of the GNU Lesser General Public License Agreement, which may be found at The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvine, and Vanderbilt University, Copyright ( ) , all rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistribution of this software is subject to terms available at and This product includes Curl software which is Copyright , Daniel Stenberg, <[email protected]>. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. The product includes software copyright ( ) MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at license.html. The product includes software copyright , The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at kawa/software-license.html. This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & Wireless Deutschland. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by Boost ( or under the Boost software license. Permissions and limitations regarding this software are subject to terms available at / This product includes software copyright University of Cambridge. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at and at This product includes software licensed under the terms at license.html, httpunit.sourceforge.net/doc/ license.html,

3 license.html, license-agreement; /copyright-software ; forge.ow2.org/projects/javaservice/, license.html; protobuf.googlecode.com/svn/trunk/src/google/protobuf/descriptor.proto; current/doc/mitk5license.html; blob/master/license; page=documents&file=license; blueprints/blob/master/license.txt; and This product includes software licensed under the Academic Free License ( the Common Development and Distribution License ( the Common Public License ( the Sun Binary Code License Agreement Supplemental License Terms, the BSD License ( the new BSD License ( licenses/bsd-3-clause), the MIT License ( the Artistic License ( and the Initial Developer s Public License Version 1.0 ( This product includes software copyright Joe WaInes, XStream Committers. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by the Indiana University Extreme! Lab. For further information please visit This product includes software Copyright (c) 2013 Frank Balluffi and Markus Moeller. All rights reserved. Permissions and limitations regarding this software are subject to terms of the MIT license. This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,501; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,162,643; 7,243,110; 7,254,590; 7,281,001; 7,421,458; 7,496,588; 7,523,121; 7,584,422; 7,676,516; 7,720,842; 7,721,270; 7,774,791; 8,065,266; 8,150,803; 8,166,048; 8,166,071; 8,200,622; 8,224,873; 8,271,477; 8,327,419; 8,386,435; 8,392,460; 8,453,159; 8,458,230; 8,707,336; 8,886,617 and RE44,478, International Patents and other Patents Pending. DISCLAIMER: Informatica Corporation provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of noninfringement, merchantability, or use for a particular purpose. Informatica Corporation does not warrant that this software or documentation is error free. The information provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is subject to change at any time without notice. NOTICES This Informatica product (the "Software") includes certain drivers (the "DataDirect Drivers") from DataDirect Technologies, an operating company of Progress Software Corporation ("DataDirect") which are subject to the following terms and conditions: 1. THE DATADIRECT DRIVERS ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. 2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACH OF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS. Part Number: IC-HCG

4 Table of Contents Preface Informatica Resources Informatica Documentation Informatica Web Site Informatica Cloud Web Site Informatica Cloud Communities Informatica Cloud Marketplace Informatica Cloud Connector Documentation Informatica Knowledge Base Informatica Cloud Trust Site Informatica Global Customer Support Chapter 1: Overview Chapter 2: Hadoop Description Chapter 3: Hadoop Plugin Chapter 4: Supported Objects and Task Operations Chapter 5: Enabling Hadoop Connector Instructions while installing the Secure Agent Chapter 6: Creating a Hadoop Connection as a Source JDBC URL JDBC Driver class Setting Hadoop Classpath for various Hadoop Distributions Setting Hadoop Classpath for Amazon EMR_ HortonWorks_ Pivotal and MapR Chapter 7: Creating Hadoop Data Synchronization Task Chapter 8: Enabling a Hadoop Connection as a Target Chapter 9: Creating Hadoop Data Synchronization Task Chapter 10: Data Filters Chapter 11: Troubleshooting Increasing Secure Agent Memory Additional Troubleshooting Tips Table of Contents

5 Chapter 12: Known Issues Index Table of Contents 5

6 Preface Hadoop user guide provides a brief introduction on cloud connectors and its features. The guide provides detailed information on setting up the connector and running data synchronization tasks (DSS). A brief overview of supported features and task operations that can be performed using Hadoop connector is mentioned. Informatica Resources Informatica Documentation The Informatica Documentation team makes every effort to create accurate, usable documentation. If you have questions, comments, or ideas about this documentation, contact the Informatica Documentation team through at [email protected]. We will use your feedback to improve our documentation. Let us know if we can contact you regarding your comments. The Documentation team updates documentation as needed. To get the latest documentation for your product, navigate to Product Documentation from Informatica Web Site You can access the Informatica corporate web site at The site contains information about Informatica, its background, upcoming events, and sales offices. You will also find product and partner information. The services area of the site includes important information about technical support, training and education, and implementation services. Informatica Cloud Web Site You can access the Informatica Cloud web site at This site contains information about Informatica Cloud editions and applications. It also provides information about partners, customers, and upcoming events. Informatica Cloud Communities Use the Informatica Cloud Community to discuss and resolve technical issues in Informatica Cloud. You can also find technical tips, documentation updates, and answers to frequently asked questions. Access the Informatica Cloud Community at: 6

7 Developers can learn more and share tips at the Cloud Developer community: Informatica Cloud Marketplace Visit the Informatica Marketplace to try and buy Informatica Cloud Connectors, Informatica Cloud integration templates, and Data Quality mapplets. Cloud Connectors Mall: Cloud Integration Templates Mall: Data Quality Solution Blocks: Informatica Cloud Connector Documentation You can access documentation for Informatica Cloud Connectors at the Informatica Cloud Community: Informatica Knowledge Base As an Informatica customer, you can access the Informatica Knowledge Base at Use the Knowledge Base to search for documented solutions to known technical issues about Informatica products. You can also find answers to frequently asked questions, technical white papers, and technical tips. If you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base team through at Informatica Cloud Trust Site You can access the Informatica Cloud trust site at This site provides real time information about Informatica Cloud system availability, current and historical data about system performance, and details about Informatica Cloud security policies. Informatica Global Customer Support You can contact a Customer Support Center by telephone or online. For online support, click Submit Support Request in the Informatica Cloud application. You can also use Online Support to log a case. Online Support requires a login. You can request a login at The telephone numbers for Informatica Global Customer Support are available from the Informatica web site at Preface 7

8 C H A P T E R 1 Overview Informatica cloud connector SDKs are off-cycle, off release add-ins that provide data integration to SaaS and on-premise applications, which are not supported natively by Informatica cloud. The cloud connectors are specifically designed to address most common use cases such as moving data into cloud and retrieving data from cloud for each individual application. Once the Hadoop cloud connector is enabled for your ORG Id, you need to create a connection in Informatica cloud to access the connector. 8

9 C H A P T E R 2 Hadoop Description The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The project includes these modules: Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS ): A distributed file system that provides high-throughput access to application data. Hadoop YARN: A framework for job scheduling and cluster resource management. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. Other Hadoop-related projects at Apache include: Ambari : A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner. Avro : A data serialization system. Cassandra : A scalable multi-master database with no single points of failure. Chukwa : A data collection system for managing large distributed systems. HBase : A scalable, distributed database that supports structured data storage for large tables. Hive : A data warehouse infrastructure that provides data summarization and ad hoc querying. Mahout : A Scalable machine learning and data mining library. Pig : A high-level data-flow language and execution framework for parallel computation. ZooKeeper : A high-performance coordination service for distributed applications. 9

10 C H A P T E R 3 Hadoop Plugin The Informatica Hadoop connector allows you to perform the Query and Insert operations on Hadoop. The plug-in supports CloudEra, HortonWorks, Amazon EMR, MapR and Pivotal Hadoop and has been certified to work on CDH 4.2 and HDP 1.1 Cloudera 5.0, MapR 3.1, Pivotal HD 2.0, Amazon EMR and Horton Works 2.1. The Informatica Cloud Secure Agent must be installed on one of the nodes of the Hadoop Cluster when the plug-in is used as a target to insert data into Hadoop. The plug-in connects to Hive and Cloudera Impala to perform relevant data operations. The plug-in can easily be integrated with the Informatica Cloud. The plugin supports all operators supported in HiveQL. The plug-in supports the AND conjunction between filters. It supports both AND and OR conjunctions in advanced filters. The plug-in supports filtering on all filterable columns in Hive/Impala tables. 10

11 C H A P T E R 4 Supported Objects and Task Operations The table below provides the list of objects and task operations supported by ReST connector. Objects DSS Source DSS Target Query Insert Update Upsert Delete Data Preview Look Up All tables in Hive All tables in Impala NA NA NA NA NA NA NA NA NA NA Supported NA Not Applicable 11

12 C H A P T E R 5 Enabling Hadoop Connector To enable Hadoop connector, get in touch with Informatica support or Informatica representative. It usually takes 15 minutes for the connector to download to secure agent, after it is enabled. Instructions while installing the Secure Agent Follow the given instructions while installing the secure agents: You must install the secure agent on Hadoop cluster. If you install it outside the Hadoop cluster you can only read from Hadoop, but you cannot write into the Hadoop. You must also install the secure agent on the node where hive server 2 is running. 12

13 C H A P T E R 6 Creating a Hadoop Connection as a Source To use Hadoop connector in data synchronization task, you must create a connection in Informatica Cloud. See Also: Creating a connection for Linux environment. The following steps help you to create Hadoop connection in Informatica Cloud. 1. In Informatica Cloud home page, click Configure. 2. The drop-down menu appears, select Connections. 3. The Connections page appears. 4. Click New to create a connection. 5. The New Connection page appears. 13

14 6. Specify the values to the connection parameters. Connection Property Connection Name Description Type Secure Agent Username Password JDBC Connection URL Driver Commit Interval Hadoop Installation Path HDFS Installation Path HBase Installation Path Implala Installation Path Miscellaneous Library Path Enable Logging Description Enter a unique name for the connection. Provide a relevant description for the connection. Select Hadoop from the list. Select the appropriate secure agent from the list. Mention the username of Schema of Hadoop component. Mention the password of the schema of Hadoop component. Mention the JDBC URL to connect to the Hadoop Component. Refer JDBC URL on page 15. Mention the JDBC driver class to connect to the Hadoop Component. Refer Setting Hadoop Classpath for various Hadoop Distributions on page 15. Mention the commit interval. It is the Batch size (in rows) of data loaded into hive. Mention Hadoop Installation path. The Installation path of the Hadoop component* used to connect to Hadoop. Only one of these installation Mention the HDFS Installation Path. Mention HBase Installation Path. Mention Implala Installation Path. Mention the Miscellaneous Library Path. This is an additional library that could be used to communicate with Hadoop. Check the Enable Logging box. This Enables verbose log messages. Note: Installation paths are the paths where Hadoop jar is listed. The connector loads and set one of these or more. Connector loads the libraries from these paths before sending any instructions to Hadoop. If you do not want to mention the installation path, you can set the Hadoop classpath.sh file for amazon, HortonWorks, MapR and Cloudera. Refer Setting Hadoop Classpath for various Hadoop Distributions on page Click Test to evaluate the connection. 8. Click Ok to save the connection. 14 Chapter 6: Creating a Hadoop Connection as a Source

15 JDBC URL The connector connects to different components of Hadoop using JDBC. The URL format and parameters vary among components. Hive uses the JDBC URL format mentioned below:. jdbc:<hive/hive2>://<server>:<port>/<schema> The significance of URL parameters is discussed below: hive/hive2 protocol information depending on the version of the Thrift Server used, hive forhiveserver and hive2 for HiveServer2. Server, port server and port information where the Thrift Server is running. Schema hive schema to which the connector needs to access. For example, jdbc:hive2://invrlx63iso7:10000/default connects the default schema of Hive, using a Hive Thrift server HiveServer2 that stars on the server invrlx63iso7 on port The Hive thrift serve runs for the connector to communicate with Hive. The command to start the Thrift server is hive service hiveserver2. Cloudera Impala uses the JDBC URL format given below: jdbc:hive2://<server>:<port>/;auth=<auth mechanism> In this case, the parameter auth must be set to the security mechanism used by the Impala Server, Kerberos. For example, jdbc:hive2://invrlx63iso7:21050/;auth=nosasl connects to the default schema of Impala. JDBC Driver class The JDBC Driver class tends to vary among Hadoop components. For example, org.apache.hive.jdbc.hivedriver for Hive and Impala: Setting Hadoop Classpath for various Hadoop Distributions In the connection parameters if you do not mention the installation paths, you can still go ahead and perform the connection operations. In order to set the class path, you must simply set the classpath for the respective distributions. This section helps you to set the classpath for the distributions of Hadoop and procedure to set Classpath for Mapr alone. JDBC URL 15

16 Setting Hadoop Classpath for Amazon EMR_ HortonWorks_ Pivotal and MapR Follow the procedure for generating sethadoopconnectorclasspath.sh for Amazon, Horton works, Pivotal and MapR. 1. Start the Agent as shown in the below command prompt. 2. Create the Hadoop Connection using the connector. 3. Test the connection. This will generate the sethadoopconnectorclasspath.sh file in Infa_Agent_DIR/ main/tomcat path. 4. From Infa_agent_DIR, execute the../main/tomcat/sethadoopconnectorclasspath.sh using the command. 5. Restart the Agent. And execute the DSS tasks Chapter 6: Creating a Hadoop Connection as a Source

17 Note: If you want to generate the classpath.sh file again, then delete the existing one and regenerate. Directing the Hadoop classpath to the correct classpath In certain cases the Hadoop classpath may point to the incorrect classpath. Follow the procedure given below to direct it to the correct classpath. 6. Enter the command hadoop classpath from the terminal. This will display the stream of jars. and paste the above stream in a notepad 2. Copy 7. Delete the following entries from the notepad file: :/opt/mapr/hadoop/hadoop /bin/../hadoop*core*.jar :/opt/mapr/hadoop/hadoop /bin/../lib/commons-logging-api jar (retain the latest version and delete the previous) 8. Copy the remaining content and export it to a variable called HADOOP_CLASSPATH 9. In saas-infaagentapp.sh file make the following entry Setting Hadoop Classpath for Amazon EMR_ HortonWorks_ Pivotal and MapR 17

18 10. Now follow Steps for generating sethadoopconnectorclasspath.sh mentioned above. Refer Setting Hadoop Classpath for various Hadoop Distributions on page Chapter 6: Creating a Hadoop Connection as a Source

19 C H A P T E R 7 Creating Hadoop Data Synchronization Task Note: You need to create a connection before getting started with data synchronization task. The following steps help you to setup a data synchronization task in Informatica Cloud. Let us consider the task operation Insert (Fetch/Read) to perform the Data synchronization task. 1. In Informatica Cloud home page, click Applications. 2. The drop-down menu appears, select Data Synchronization. 3. The Data Synchronization page appears. 4. Click New to create a data synchronization task. 5. The Definition tab appears. 6. Specify the Task Name, provide a Description and select the Task Operation Insert. 7. Click Next. 19

20 8. The Source tab appears. 9. Select the source Connection, Source Type and Source Object to be used for the task. 10. Click Next. 11. The Target tab appears. Select the target Connection and Target Object required for the task. 12. Click Next. 13. In Data Filters tab by default, Process all rows is chosen. See Also Chapter 10, Data Filters on page 27. It is mandatory to assign _FLT_URL_Input_Parameters_Config_File_Path data filter in DSS task. 14. Click Next. 20 Chapter 7: Creating Hadoop Data Synchronization Task

21 15. In Field Mapping tab, map source fields to target fields accordingly. 16. Click Next. 17. The Schedule tab appears. 18. In Schedule tab, you can schedule the task as per the requirement and save. 19. If you do not want schedule the task, click Save and Run the task. After you Save and Run the task, you will be redirected to monitor log page. In monitor log page, you can monitor the status of data synchronization tasks. 21

22 C H A P T E R 8 Enabling a Hadoop Connection as a Target To use Hadoop connector in data synchronization task, you must create a connection in Informatica Cloud. See Also: Creating a connection for Linux environment. The following steps help you to create Hadoop connection in Informatica Cloud. 1. In Informatica Cloud home page, click Configure. 2. The drop-down menu appears, select Connections. 3. The Connections page appears. 4. Click New to create a connection. 22

23 5. The New Connection page appears. 6. Specify the values to the connection parameters. Refer Creating a Hadoop Connection as a Sourceon page Click Test to evaluate the connection. 8. Click Ok to save the connection. 23

24 C H A P T E R 9 Creating Hadoop Data Synchronization Task Note: You need to create a connection before getting started with data synchronization task. The following steps help you to setup a data synchronization task in Informatica Cloud. Let us consider the task operation Insert (Fetch/Read) to perform the Data synchronization task. 1. In Informatica Cloud home page, click Applications. 2. The drop-down menu appears, select Data Synchronization. 3. The Data Synchronization page appears. 4. Click New to create a data synchronization task. 5. The Definition tab appears. 6. Specify the Task Name, provide a Description and select the Task Operation Insert. 7. Click Next. 24

25 8. The Source tab appears. 9. Select the source Connection, Source Type and Source Object to be used for the task. 10. Click Next. 11. The Target tab appears. Select the target Connection and Target Object required for the task. 12. Click Next. 13. In Data Filters tab by default, Process all rows is chosen. See Also Chapter 10, Data Filters on page 27. It is mandatory to assign _FLT_URL_Input_Parameters_Config_File_Path data filter in DSS task. 14. Click Next. 25

26 15. In Field Mapping tab, map source fields to target fields accordingly. 16. Click Next. 17. The Schedule tab appears. 18. In Schedule tab, you can schedule the task as per the requirement and save. 19. If you do not want schedule the task, click Save and Run the task. After you Save and Run the task, you will be redirected to monitor log page. In monitor log page, you can monitor the status of data synchronization tasks. 26 Chapter 9: Creating Hadoop Data Synchronization Task

27 C H A P T E R 1 0 Data Filters Data filters help you to fetch specific data based on the APIs configured in Config.csv file. The data synchronization task will process the data based on the filter field assigned. Note: Advanced data filters are not supported by Hadoop Connector The following steps help you to use data filters. 1. In Data synchronization task, select Data Filters tab. 2. The Data Filters tab appears. 3. Click New as shown in the figure below. 4. The Data Filter dialog box appears. 27

28 5. Specify the following details. Field Type Object Filter By Operator Filter Value Description Select Object for which you want to assign filter fields Select the Filter Field Select Equals operator. Only Equals operator is supported with this release. Enter the Filter value 6. Click Ok. 28 Chapter 10: Data Filters

29 C H A P T E R 1 1 Troubleshooting This chapter includes the following topics: Increasing Secure Agent Memory, 29 Additional Troubleshooting Tips, 31 Increasing Secure Agent Memory To overcome memory issues faced by secure agent follow the steps given below. 1. In Informatica Cloud home page, click Configuration. 2. Select Secure Agents. 3. The secure agent page appears. 4. From the list of available secure agents, select the secure agent for which you want to increase memory. \ 5. Click pencil icon corresponding to the secure agent. The pencil icon is to edit the secure agent. 6. The Edit Agent page appears. 7. In System Configuration section, select the Type as DTM. 29

30 8. Edit JVMOption1 as -Xmx512m as shown in the figure below. 9. Again in System Configuration section, select the Type as TomCatJRE. 10. Edit INFA_memory to -Xms256m -Xmx512m as shown in the figure below. 11. Restart the secure agent The secure agent memory has been increased successfully. 30 Chapter 11: Troubleshooting

31 Additional Troubleshooting Tips When the connection is used as a target, the last batch of the insert load is not reflected in the record count. Refer the session logs for the record count of the last batch inserted. For example, if the commit interval is set to 1 million and the actual rows inserted are 1.1 million, the record count in the UI shows 1 million and the session logs reveal the row count of the reminder 100k records. Set the commit interval to the highest value possible before java.lang.outofmemoryerror is encountered. When the connection is used as a target to load data into Hadoop, ensure that all the fields are mapped. After a data load in Hive, Impala needs to be refreshed manually for the latest changes to the table to be reflected in Impala. In the current version, the connector does not automatically refresh Impala upon a Hive dataset insert. Additional Troubleshooting Tips 31

32 C H A P T E R 1 2 Known Issues The connector is currently certified to work with Cloudera CDH 4.2. and HortonWorks HDP 1.1. The connector may encounter java.lang.outofmemory exception while fetching large data sets for tables with a large number of columns (for example, 5 million for a 15 column table). In such scenarios, restrict the resultset by adding appropriate filters or by decreasing the number of field mappings. The Enable Logging connection parameter is place-holder for a future release, and its state has no impact on connector functionality. The connector has been certified and tested on Hadoop s pseudo-distributed mode. Performance is a factor of Hadoop s cluster setup. Ignore log4j initialization warnings in the session logs. 32

33 I n d e x C Cloud Developer community URL 6 I Informatica Cloud Community URL 6 Informatica Cloud web site URL 6 Informatica Global Customer Support contact information 7 T trust site description 7 33