Talend Platform for Data Management. Installation and Upgrade Guide 5.2.1

Transcription

1 Talend Platform for Data Management Installation and Upgrade Guide 5.2.1

2 Talend Platform for Data Management Adapted for v Supersedes any previous Installation and Upgrade Guide. Copyright Notices All brands, product names, company names, trademarks and service marks are the properties of their respective owners.

3 Table of Contents Preface... v 1. General information... v 1.1. Purpose... v 1.2. Audience... v 1.3. Typographical conventions... v 2. Feedback and Support... v Chapter 1. Prior to installing the Talend products License key Installation requirements Memory usage Disk usage Environment variable configuration Naming conventions for software packages Talend Installer specific prerequisites User interface and installation modes of Talend Installer Studio specific prerequisites Installing database client software (for bulk mode) Compatible Platforms Talend Studio Talend server modules: Talend Administration Center, JobServer, CommandLine,Talend Runtime, Talend Data Quality Portal Compatible Databases Talend Administration Center Talend Activity Monitoring Console Talend Data Quality Portal Talend Data Stewardship Console Compatible Runtime Containers Talend Administration Center Talend Data Quality Portal Compatible execution servers Job servers (JobServer and Talend Runtime) ESB OSGI servers (Talend Runtime) Compatible SVN Compatible Web browsers Port information Talend Studio and CommandLine Talend Administration Center Talend Runtime JobServer Talend Artifact Repository Talend Data Quality Portal Chapter 2. Introducing Talend Platform for Data Management Introduction to Talend Platform for Data Management Apache Tomcat Server Database SVN Talend Artifact Repository Talend Runtime JobServer Talend Studio Talend Activity Monitoring Console log database Operating principles of Talend Platform for Data Management Building and administrating Deploying and executing Monitoring Chapter 3. Installing Talend Platform for Data Management for the first time Setting up your project server on SVN Windows Debian, apache Debian, ssh Redhat (Fedora), Apache Mac OS X Deploying Talend Administration Center on an application server Deploying the Web application on Tomcat Deploying Talend Administration Center on JBoss Increasing the memory of Tomcat Synchronizing Web application and server time zones Defining the log storage mode Setting up High Availability Configuring Talend Administration Center (Web application) Configuring the Web application access Managing the connection pool via Tomcat Oracle-specific configuration Advanced configuration Installing CommandLine Accessing user-defined components from the CommandLine Installing and configuring the artifact repository Talend Artifact Repository Installing the execution servers (JobServers) Installing the Jobservers Configuring the JVM (optional) Configuring the SSL Keystore (optional) Installing Talend Runtime Installing the Talend Runtime containers Enabling the SSL encryption Installing Talend Data Quality Portal Customizing and restoring the Talend Data Quality Portal configuration database Copying the libraries, the resources and the.war file Editing the servertemplate file Starting Tomcat Configuring the portal Installing the Talend Activity Monitoring Console Web application Deploying the Talend Activity Monitoring Console on Tomcat Configuring the Talend Activity Monitoring Console in Talend Administration Center Installing a Business Rules Management System (Drools) Talend Platform for Data Management Installation and Upgrade Guide

4 Talend Platform for Data Management Installing the standalone Talend Data Stewardship Console (optional) Replacing the default database Setting up the automatic log purge (optional) Chapter 4. Configuring Talend Studio and Talend Activity Monitoring Console Configuring the Talend Studio Installing external modules Memory configuration and launching issues Configuring the.ini file on Linux/Windows For 64-bit platforms on Linux/ Windows For non Linux/Windows platforms For 64-bit platform on non Linux/Windows OS Configuring Talend Activity Monitoring Console Chapter 5. Upgrading your Talend products Backing up the environment Saving the configuration parameters Saving the local projects Stopping the services Backing up the administration database (MySQL, H2, MS SQL Server, or Oracle) Backing up the SVN repository Backing up the archive folder holding the generated tasks Re-installing CommandLine Upgrading the administration database For MySQL, SQL Server, or Oracle For H Re-installing JobServer Upgrading Talend Administration Center Web application Clearing your previous Talend Administration Center files Setting up your Talend Administration Center configuration back Retrieving the previously generated tasks Getting and managing the License key Upgrading the Talend projects in the Studio Importing your local projects Retrieving your remote projects Upgrading the Talend Data Quality Portal Chapter 6. Installing services Configuring JSL and.ini file for the CommandLine and JobServer services Configuring the CommandLine parameters Configuring the JobServer parameters On Windows Tomcat CommandLine JobServer SVN Talend Runtime Talend Artifact Repository On Linux Tomcat CommandLine JobServer SVN Talend Runtime Talend Artifact Repository Appendix A. Installing services on non- Debian distributions A.1. Example of CommandLine daemon file A.1.1. On Redhat A.1.2. On Solaris A.2. Example of Job Server init file A.2.1. On Redhat A.2.2. On Solaris A.2.3. On OpenSuse A.3. Example of Tomcat init file Appendix B. H2 Database Administration & Maintenance B.1. About H2 embedded database B.2. Administrating the H2 database through the Web console B.2.1. Connecting to the H2 Web Console B.2.2. Backing up the H2 database B.3. Setting up the H2 database for access from other machines Appendix C. Supported Third-Party System/Database Versions C.1. Supported systems and databases iv Talend Platform for Data Management Installation and Upgrade Guide

5 Preface 1. General information 1.1. Purpose This Installation Guide explains how to install and configure Talend Platform for Data Management modules and related applications. For detailed explanation on how to use and fine-tune Talend Platform for Data Management applications, please refer to the appropriate Administrator or User Guides of Talend Platform for Data Management solutions. Information presented in this document applies to release of Talend Platform for Data Management Audience This guide is devoted for administrators of Talend Platform for Data Management solutions. The layout of GUI screens provided in this document may vary slightly from your actual GUI Typographical conventions This guide uses the following typographical conventions: text in bold: window and dialog box buttons and fields, keyboard keys, menus, and menu and options, text in [bold]: window, wizard, and dialog box titles, text in courier: system parameters typed in by the user, text in italics: file, schema, column, row, and variable names, The icon indicates an item that provides additional information about an important point. It is also used to add comments related to a table or a figure, The icon indicates a message that gives information about the execution requirements or recommendation type. It is also used to refer to situations or information the end-user needs to be aware of or pay special attention to. Any command is highlighted with a grey background or code typeface. 2. Feedback and Support Your feedback is valuable. Do not hesitate to give your input, make suggestions or requests regarding this documentation or product and find support from the Talend team, on Talend s Forum website at: Talend Platform for Data Management Installation and Upgrade Guide

6 Feedback and Support vi Talend Platform for Data Management Installation and Upgrade Guide

7 Chapter 1. Prior to installing the Talend products This chapter provides useful information on software and hardware prerequisites you should be aware of, prior to starting the installation of the Talend modules. There are two ways to install the Talend solutions. You can either install them manually, one by one, or use the Talend Installer wizard to get out-of-the-box solutions. The sections below provide information on both installation modes. In the following documentation: certified: designates an environment certified by Talend through QA tests prior to the release of the software; supported: designates an environment that can be provided by Talend for problem reproduction and testing within 24 hours; supported with limitations: designates an environment that is supported by Talend under certain conditions explained in notes. Talend Platform for Data Management Installation and Upgrade Guide

8 License key 1.1. License key You should have received an from Talend, including the following information: your personal license key in a file with no extension: The license key is mandatory to be able to access each module of Talend Platform for Data Management. Keep this file at hand in a safe place. For more information on how to use your license key, see section Configuring the Web application access and section Configuring the Talend Studio. The availability of different perspectives in the Studio you want to install and use depends on your license or on the type of the remote project to which you connect. For further information on licenses and perspectives in the Studio, see the appropriate Administrator or User Guide. the software parts in archive files: Unzip each archive file at the root or in a short path of the machine where they are to be deployed (Talend Studio on a developer s machine and Talend Administration Center on the execution server for example). If you are on Windows, we recommend that you use 7-zip to unzip files, as it handles long file names well. the documentation (including the one you are reading now): Each module of Talend has a dedicated Administrator or User Guide available in pdf Installation requirements To make the most out of the Talend products you subscribed to, please consider the following hardware and software requirements Memory usage Memory usage heavily depends on the size and nature of your Talend projects. However, in summary, if your Jobs include many transformation components, you should consider upgrading the total amount of memory allocated to your servers, based on the following recommendations. Product Client/Server Recommended alloc. memory Talend Administration Center + Talend Activity Monitoring Console Web application Server 4GB minimum, 8GB recommended CommandLine Server 2GB minimum, 5 GB recommended JobServer Server 1GB minimum, more recommended 1 Studio Client 3GB minimum, 4 GB recommended Talend Runtime Server 2GB minimum, 4 GB recommended 1 1. Memory requirements depend on the executed processes Disk usage The same requirements also apply for disk usage. It also depends on your projects but can be summarized as: 2 Talend Platform for Data Management Installation and Upgrade Guide

9 Environment variable configuration Product Client/Server Required disk space for installation Talend Administration Center Server with Talend Artifact Repository (for Software update) + Talend Activity Monitoring Console Web application 800MB + more than 50MB for Talend Activity Monitoring Console Web application Required disk space for use 800MB minimum + project size = 20GB+ recommended CommandLine Server 3GB 2GB minimum + project size = 20 GB+ recommended JobServer Server 20MB 2GB minimum + Jobs deployed = 20+ GB recommended Studio Client 3GB 3+ GB Talend Runtime Server 400MB 400+ MB Environment variable configuration Prior to installing your Talend solutions, you have to set the JAVA_HOME Environment variable: 1. Define your JAVA_HOME environment variable so that it points to the JDK directory. For example, if the JDK path is C:\Java\JDKx.x.x\bin, you must set the JAVA_HOME environment variable to point to: C:\Java\JDKx.x.x. It is highly recommended that the full path to the server installation directory is as short as possible and does not contain any space character. If you already have a suitable JDK installed in a path with a space, you simply need to put quotes around the path when setting the values for the environment variable. If you use Talend Installer, you also have to set the Path system variable: 2. Add the previously defined JAVA_HOME variable to the Path environment variable. For example, <PathVariable>;%JAVA_HOME%\bin Naming conventions for software packages This section applies only to manual installations. In the you received from Talend are listed a number of links to software modules for you to download. The file naming convention is as follows: Zip/jar file naming convention Talend-Installer-Starter-rYYYYY-VA.B.C.zip Talend-Studio-rYYYYY-VA.B.C.zip Talend-AdministrationCenter-rYYYYY-VA.B.C.zip Talend-JobServer-rYYYYY-VA.B.C.zip Talend-Runtime-VA.B.C.zip Talend-AMC_Web-rYYYYY-VA.B.C.zip Description Talend Installer: wizard-based application which guides you step by step through the installation and configuration of the Talend modules. CommandLine interface to the IDE + Studio IDE (GUI) Talend Administration Center: Web-based application used to administrate Talend Platform for Data Management projects and users + Talend Artifact Repository. JobServer: Standalone execution server Talend Runtime: OSGi Container including JobServer. Talend Runtime is a standalone equivalent to the Talend ESB OSGi Container (container folder) of Talend ESB. Talend Activity Monitoring Console Web application used to monitor Talend Jobs and projects. Talend Platform for Data Management Installation and Upgrade Guide 3

10 Talend Installer specific prerequisites Zip/jar file naming convention Talend-BRMS-rYYYY-VA.B.C.zip Talend-DQPortal-rYYYY-VA.B.C.zip Talend-DSC-rYYYY-VA.B.C.jar Description Drools: Business Rules Management System based on JBoss Rules Governor. Talend Data Quality Portal: portal which provides customizable web-based data quality monitoring and reporting. Talend Data Stewardship Console: stewardship tool that helps to manage manually data resolution and data integration tasks that are the output of data matching and data integrity processes. Where: YYYYY: Revision number, A.B.C.: Major. Minor. Patch: revision level if relevant. The software modules must be all in the same versions/revisions! This means that both YYYY and A.B.C must match on both client side and server side Talend Installer specific prerequisites Prior to launching the Talend Installer, please check that the following software and modules are properly set up on the station where you are to install the Talend modules: Operating System Linux 32/64 bits, Windows 32/64 bits or Mac OS X; JDK or higher must be installed; Talend Installer automatically checks for JDK version 1.6 or higher version. If no instance of JDK is found, the installer will shut down..net version 2 or higher is required by Talend Installer to install services on Windows; (optional) a mail server (to send notifications by ). For Linux users: Note that Talend Installer does not support the sdshell utility. IMPORTANT: Talend Installer allows you to get out-of-the-box Talend solutions that do not require any manual installation. However, these solutions are not provided in a production-ready environment as they may require additional configurations or optimizations according to your specific needs. For example, you may want to change the H2 database that is embedded by default in Talend Administration Center with your own database (MySQL or Oracle for example). Talend Installer is used only for first installations of Talend solutions. Therefore, if you want to know more about the migration and upgrade processes, please refer to the chapter Upgrading your Talend products User interface and installation modes of Talend Installer This section provides information about the different installation modes that Talend Installer can run in. Note that the log files generated during the installation can be found in C:\Users\<username>\AppData\Local\Temp on Windows, and in /tmp/ on Linux. 4 Talend Platform for Data Management Installation and Upgrade Guide

11 User interface and installation modes of Talend Installer Three main installation modes are available: The Full Graphic mode: allows full interactivity through a graphical user interface. The Text mode: provides full interactivity with users in the command line. It is equivalent to any GUI mode but the pages are displayed in text mode in a console. The image below shows an example of text mode where the user enters the --mode text option from the command line on a OS X system: This installation mode is only available on Unix platforms. It is automatically used if no graphical mode is available but it also can be forced using the --mode text command. The Unattended mode: is especially useful for automating the installation processes. This silent mode will perform an unattended installation that will not prompt the user for any information. To perform this Unattended installation: 1. Write a simple.txt script in which you will define the options values: The list of values available can be displayed by entering the help command in the command line. In this example, the script details the silent installation of the Client type. The installation directory that will be created is called Install and the license file used is located in the C:/licenses directory. 2. Launch the silent installation using the --optionfile <filename> command, where <filename> is the name of the script which contains the list of pairs <key>=<value>. An unattended installation is performed. If you want to install Talend products as services via the Installer and you are on Windows Vista or Windows 7, you are required to run the application as Administrator OR to disable User Account Control. For more information on these installation modes, please refer to the online Bitrock documentation: installbuilder.bitrock.com/docs/installbuilder-userguide.html. Talend Platform for Data Management Installation and Upgrade Guide 5

12 Studio specific prerequisites 1.5. Studio specific prerequisites To use the Studio properly, you first need to install external programs specific to bulk components (if you want to use Oracle, Sybase, Informix or Ingres bulk functionality). On Windows XP and Windows Server 2003, the GDI is already installed. However, on Windows 2000, this installation is required. The GDI can be downloaded from Microsoft s Website. For further information, visit Eclipse s FAQ Installing database client software (for bulk mode) Some bulk components, like Oracle, Sybase, Informix or Ingres, require database client software to run properly: OracleBulkExec uses the sqlldr external utility. This utility is available in Oracle clients that must be installed on the computer. Informix uses the dbload external utility. Ingres uses the sql external utility. Sybase uses the bcp.exe external utility. This utility is asked for in the Sybase bulk components Basic Settings view. For more information, see tsybasebulkexec, tsybaseoutputbulk and tsybaseoutputbulkexec components on the appropriate Talend Components Reference Guide Compatible Platforms Despite our intensive tests, you might encounter some issues when installing our products on some Operating Systems. Please refer to the grids below for a summary of supported OS and Java Runtime environments Talend Studio On Linux, the xulrunner package is required to run the Studio. Please install mozilla-xulrunner192 Mozilla Runtime Environment and add the line -Dorg.eclipse.swt.browser.XULRunnerPath=</usr/lib/xulrunner > at the end of the.ini file that corresponds to your Studio (where </usr/lib/xulrunner > is the xulrunner installation path). OS Name Version Processor Java JDK/JRE 1 Support type Unix Linux Ubuntu bits Oracle Java 7 certified Unix Linux Ubuntu /64 bits Oracle Java 6 supported Unix Linux Ubuntu 11.10/ /64 bits Oracle Java 6/7 supported Unix Unix Redhat Linux Enterprise Server Edition/CentOS Redhat Linux Enterprise Server Edition/CentOS 5.3 to /64 bits Oracle Java 6 supported 6.X (>=6.1) 64 bits Oracle Java 6/7 supported Unix SUSE SLES 10/11 32/64 bits Oracle Java 6/7 supported 6 Talend Platform for Data Management Installation and Upgrade Guide

13 Talend server modules: Talend Administration Center, JobServer, CommandLine,Talend Runtime, Talend Data Quality Portal OS Name Version Processor Java JDK/JRE 1 Support type Windows Microsoft Windows 8 64 bits Oracle Java 7 certified Windows Microsoft Windows 7 64 bits Oracle Java 6 certified Windows Microsoft Windows XP SP3 32/64 bits Oracle Java 6 supported Windows Microsoft Windows Vista SP1 32/64 bits Oracle Java 6/7 supported Windows Microsoft Windows 7 32 bits Oracle Java 6/7 supported MAC OS MAC OS Lion/ bits Oracle Java 6 certified 2 MAC OS MAC OS Lion/ bits Oracle Java 7 supported MAC OS MAC OS Mountain Lion/ It is recommended to use a recent update of JDK 1.6 (Update 11 or higher). 64 bits Oracle Java 6/7 supported 2. Need to set security settings to accept non MAC-registered applications Talend server modules: Talend Administration Center, JobServer, CommandLine,Talend Runtime, Talend Data Quality Portal OS Name Version Processor Java JDK/JRE 1 Support type Unix Linux Ubuntu bits Oracle Java 7 certified Unix Red Hat Linux Enterprise Server Edition bits Oracle Java 7 certified Unix SUSE SLES bits IBM Java 6 certified 2 Unix Linux Ubuntu /64 bits Oracle Java 6 supported Unix Linux Ubuntu 11.10/ /64 bits Oracle Java 6/7 supported Unix Unix Red Hat Linux Enterprise Server Edition/CentOS Red Hat Linux Enterprise Server Edition/CentOS 5.3 to /64 bits Oracle Java 6 supported 6.X (>=6.2) 64 bits Oracle Java 6/7 supported Unix SUSE SLES 10/11 32/64 bits Oracle Java 6/7 supported Unix AIX bits IBM Java 6 supported 3 Unix Solaris (SunOS) 10/11 64 bits Oracle Java 6/7 supported Windows Microsoft Windows Server 2008 R2 64 bits Oracle Java 6 certified Windows Microsoft Windows Server bits Oracle Java 7 certified Windows Microsoft Windows Server /64 bits Oracle Java 6 supported 1. It is recommended to use a recent update of JDK 1.6 (Update 11 or higher). 2. Certification/support of IBM JDK is restricted to CommandLine, Jobserver. Not supported: Talend Administration Center, DQ Portal. No Installer support. 3. Only for the Runtime Container and the JobServer Compatible Databases Please refer to the grids below for a summary of supported databases. Talend Platform for Data Management Installation and Upgrade Guide 7

14 Talend Administration Center Talend Administration Center Databases Version Support type MySQL 5.5 certified 1 Oracle 11 certified 2 H2 1.3 certified 3 MySQL 5.0/5.1 supported MS SQL Server 2005 supported MS SQL Server 2008 supported Oracle 10 supported H2 is provided by Talend with Talend Administration Center Talend Activity Monitoring Console Databases Version Support type MySQL 5.5 certified Oracle 11 certified IBM DB certified MySQL 5.0/5.1 supported H2 1.3 supported 1 MS SQL Server 2005 supported MS SQL Server 2008 supported Oracle 10 supported Derby DB >10.8 supported with limitations 2 1. Requires an H2 Database Server separated from Talend Administration Center. 2. Only for development and test purposes Talend Data Quality Portal Databases Version Support type MySQL 5.0+ certified Oracle 10g supported HSQL supported If you are using MySQL, the my.ini file of your MySQL server installation directory must be edited in order to add the following line: max_allowed_packet = 64M The 64M value is only given as example. It represents the maximum size of a query packet the server can handle as well as the maximum query size the server can process. You can replace it with any value superior to 16M. 8 Talend Platform for Data Management Installation and Upgrade Guide

15 Talend Data Stewardship Console Talend Data Stewardship Console The table below indicates the Talend Data Stewardship Console embedded databases and their supported versions. Databases Version MySQL 4.1/5.0 1 HypersonicSQL Support of versions 5.1 to 5.5 requires the installation of a Connector/J driver v Versions 1.61 to 1.8 are also supported. It is possible to make Talend Data Stewardship Console work with other databases (HSQL, Oracle, SQLServer, etc.) by editing the properties of the tdsc-database.properties file which can be found in the <DSC>\WEB-INF\classes folder where <DSC> is the Talend Data Stewardship Console installation directory. For more information, see the section Replacing the default database Compatible Runtime Containers Please refer to the grids below for a summary of supported Web application server Talend Administration Center Runtime Containers Version Support type Apache Tomcat 6 certified Apache Tomcat 7 supported Weblogic 11 supported JBoss Application Server supported Talend Data Quality Portal Runtime Containers Version Support type Apache Tomcat 6 certified Apache Tomcat 7 supported 1.9. Compatible execution servers Make sure the execution server version is compatible with the Talend Administration Center, CommandLine and Studio versions. Note that the information contained in this section is valid at the date of publication, but may be subject to change at a later date. Talend Platform for Data Management Installation and Upgrade Guide 9

16 Job servers (JobServer and Talend Runtime) Job servers (JobServer and Talend Runtime) Talend Administration Center, Studio, CommandLine Talend Administration Center, Studio, CommandLine Talend Administration Center, Studio, CommandLine Talend Administration Center, Studio, CommandLine JobServer JobServer & Talend JobServer & Talend JobServer & Talend Runtime (holding Runtime (holding Runtime (holding Job-Server) Job-Server) Job-Server) Version 4.2.x 5.0.x 5.1.x 5.2.x 4.2.x 5.0.x 5.1.x 5.2.x ESB OSGI servers (Talend Runtime) Compatible SVN Please refer to the tables below for a summary of supported SVN clients/servers that you can use to store your projects. Subversion client Version Support type Apache Subversion (former Tigris) 1.7 or earlier supported Subversion servers Version Support type Visual SVN Server 2.5 or earlier certified 1 Bitnami Subversion Stack based on SVN v.1.7 or earlier supported Svnserve based on SVN v.1.7 or earlier supported Compatible Web browsers Please refer to the table below for a summary of supported Web browsers. Web browser Mozilla Firefox >= 3.6 Microsoft Internet Explorer 8/9 Talend Administration Center certified certified 10 Talend Platform for Data Management Installation and Upgrade Guide

17 Port information Web browser Talend Administration Center Apple Safary 5 supported Google Chrome >= 18 supported with limitations 1 1. Graphical restrictions. 2. Graphical restrictions, a patch can be provided on demand Port information This section describes the most important TCP/IP ports the Talend products use. Please make sure that your firewall configuration is compatible with these ports or change the default ports where needed. Table Information : Port: a TCP/IP port or a range of ports Active: Active for a standard installation of the product (Standard Installation is defined here as Server or Client installation using Talend Installer with the default values provided in the Installer User Interface) Direction: In (Inbound); Out (Outbound) - related to the communication direction (e.g. a HTTP Port for a Service we listen on request) will be an Inbound port. A Browser who send a request e.g. to port 7080 will have this port as Outbound port in this list. Usage: which part of the Product component uses this port (e.g is used by the JMX Monitoring component of Talend Runtime). Config: the file or location where the value can be changed. Remark: anything which is important to mention additionally Talend Studio and CommandLine Port Direction Usage Config Remark 8002 IN CommandLine port Windows: commandline.bat Active: Y Linux: commandline-linux.sh MacOS: commandline-mac.sh Talend Administration Center Port Direction Usage Config Remark 8080 Active: Y 8009 Active: Y (none) Active: Y* IN Talend Administration Center Server - Apache Tomcat HTTP Port IN Talend Administration Center Server - Apache Tomcat AJP Connector Port OUT Talend Administration Center Server - Database /conf/server.xml /conf/server.xml Configuration Page in TAC Web-UI * By default an embedded H2 Database is used (not network accessible). If another database should be used Talend Platform for Data Management Installation and Upgrade Guide 11

18 Talend Runtime Port Direction Usage Config Remark the port is related to the type and configuration of this database Talend Runtime Port Direction Usage Config (./etc/ ) Remark 8000 Active: Y 8001 Active: Y 8888 Active: Y IN Talend JobServer - Command Port IN Talend JobServer - File Transfer Port IN Talend JobServer - Monitoring Port org.talend.remote.jobserver.server. cfg org.talend.remote.jobserver.server. cfg org.talend.remote.jobserver.server. cfg JobServer Port Direction Usage Config Remark 8000 Active: Y 8001 Active: Y 8888 Active: Y IN Talend JobServer - Command Port IN Talend JobServer - File Transfer Port IN Talend JobServer - Monitoring Port org.talend.remote.jobserver.server. cfg org.talend.remote.jobserver.server. cfg org.talend.remote.jobserver.server. cfg Talend Artifact Repository Port Direction Usage Config Remark 8082 Active: Y IN Talend Artifact Repository - Jetty HTTP Port /conf/jetty.xml 8443 Active: Y IN Talend Artifact Repository - Jetty HTTPS Port /conf/jetty.xml (not required) (*) Active: N* OUT Talend Artifact Repository - Database (/conf/jetty.xml)* *By default an embedded Apache Derby Database is used (not network accessible). If another database should be used, the port is related to the type and configuration of this database Talend Data Quality Portal Port Direction Usage Config Remark 8580 IN Standard port /tdqp/apache-tomcat/conf/ server.xml 12 Talend Platform for Data Management Installation and Upgrade Guide

19 Talend Data Quality Portal Port Direction Usage Config Remark Active: Y Talend Platform for Data Management Installation and Upgrade Guide 13

20 Talend Platform for Data Management Installation and Upgrade Guide

21 Chapter 2. Introducing Talend Platform for Data Management This chapter introduces Talend Platform for Data Management. It provides a detailed description of the architecture and lists all modules that are required for the installation and configuration of Talend Platform for Data Management. Talend Platform for Data Management Installation and Upgrade Guide

22 Introduction to Talend Platform for Data Management 2.1. Introduction to Talend Platform for Data Management The present section lists all the elements required for using Talend Platform for Data Management. To ease their management, we recommend you to centralize all the server modules on one single system. All Talend applications to be installed must be the same version. An application server (Apache Tomcat server + CommandLine) that hosts Talend Administration Center. A database server storing the administration metadata of Talend Administration Center (by default, an embedded H2 database is used). A database server storing data quality reports and analyses managed by Talend Data Quality Portal. A database server storing golden or master data records managed by Talend Data Stewardship Console (if your license includes the Data Stewardship option). A SVN server for Project metadata. A Web browser to access Web application: Talend Administration Center where projects, users and processes can be managed and administrated. For more information, see the Talend Administration Center User Guide. Data Quality Portal that extends the dynamic reporting and monitoring capabilities of Talend Studio. For more information, see the Talend Data Quality Portal User and Administrator Guides. An Artifact Repository (based on Apache Archiva) for software updates. Execution servers (JobServers) or Talend Runtime execution containers (based on Apache Karaf) to deploy and execute processes. A Studio API to carry out technical processes. For more information, see the Talend Studio User Guide. A Talend Activity Monitoring Console, providing detailed monitoring capabilities that can be used to consolidate the collected log information. For more information, see the Talend Activity Monitoring Console User Guide. Each of these elements is detailed in the following sub-sections Apache Tomcat Server The Apache Tomcat server is an application server that hosts Talend Administration Center. This Web application gives access to all management and administration functionalities for an integration project, allowing users to (depending on their role): Create and manage projects. Create and manage user accounts and roles/rights. Access the Job Conductor to schedule, deploy and execute Jobs. Access the Dashboard to monitor the execution of Jobs. Note that the best recommended versions for Tomcat are 6.0 and 7.0, however the 5.5 version is also supported. Talend Administration Center can also be hosted by JBoss application server. For more information on how to deploy Talend Administration Center on Apache Tomcat or JBoss, see the section Deploying Talend Administration Center on an application server. 16 Talend Platform for Data Management Installation and Upgrade Guide

23 Database Database The administration database server is used to store administration information and manage the persistence in Talend Administration Center. By default an embedded H2 database is used, but you can also use MySQL, MS SQL Server, or Oracle to store all cross-project data (users, projects, authorization, license, tasks, triggers, dashboard). The administration database will be named <talend_administrator> in the rest of this document. The <talend_administrator> administration database will contain all the data related to project information and administration including: administration data, project declaration, user declaration and authorization, task list, etc. The tables in this database are automatically created when connecting for the first time to Talend Administration Center. The created tables include (among others): a Users table, a Projects table, a Rights table. These tables are created, populated and managed automatically by Talend Platform for Data Management, users do not need to take any action. Additionally, to perform Data Quality operations, a HSQL, MySQL or Oracle database is used to store report and analysis results and a stewardship console database is used to store golden or master data records. For more information about those databases, see the relevant sections in the present guide SVN The SVN server is used to manage the persistence of all the data relating to the objects (Jobs, Business Models, Metadata, Routines, Documentation, etc.) stored in svn mode in the Repository of Talend Studio. We recommend you to store several projects per repository, simply in order not to have too many repositories to deal with. However you can choose to store only one project per SVN repository, if you prefer so. You can also have several SVN repositories each containing several projects. For more information on how to create projects and store them in an SVN repository, see the Talend Administration Center User Guide Talend Artifact Repository Talend Artifact Repository, based on Apache Archiva, is a preconfigured application centralizing the management and usage of the Software Update. Software Update is used to manage application updates (versions and patches) distributed by Talend. By default two repositories are embedded within Software Update: one to publish the updates and one to retrieve them. These repositories allow the user to visualize the updates available. For detailed information, see the section Software update repositories Software update repositories The following image shows the architecture of Software Update linked to Talend Administration Center and to the Studio. Talend Platform for Data Management Installation and Upgrade Guide 17

24 Talend Runtime To download and install some software updates, you need to use Software Update (integrated within Talend Artifact Repository) and its two embedded repositories: an admin repository named talend-update-admin used to publish the updates; a user repository named talend-update-user used to retrieve the updates. To do so, you must install Talend Artifact Repository on your machine and log in once to its Web interface following the procedure described in the section Installing and configuring the artifact repository. The Talend Artifact Repository archive file is embedded in the Talend Administration Center archive file that is provided by Talend. In Talend Administration Center, the new versions and patches are detected and the administrator can accept them. Talend Studio is connected to Talend Administration Center to retrieve the repository connection information and the version updates are detected and installed automatically. For more information on how to install and configure Talend Artifact Repository and Software Update, see the section Installing and configuring the artifact repository. For more information on how to check updates via these repositories, see the Talend Administration Center and Talend Studio User Guides Talend Runtime Talend Runtime (based on Apache Karaf) is an execution container in which you can deploy and execute all your Jobs stored on your SVN repository. 18 Talend Platform for Data Management Installation and Upgrade Guide

25 JobServer For more information on the installation of Talend Runtime, see the section Installing Talend Runtime JobServer The JobServer is an application that allows a system installed on the same network as Talend Administration Center to declare itself as an execution server. These systems must obviously have a working JVM version 1.6. For more information on the installation of JobServers, see the section Installing the execution servers (JobServers) Talend Studio Talend Studio is a rich client that allows the user (such as a project manager, a developer or a DBA) to work on any Talend project for which he has authorization. Talend Studio offers a comprehensive set of tools and functions for all its key capabilities including: Integration Activity monitoring Console Data profiling Data quality These tools are ALL accessible in different perspectives from one Studio. The availability of perspectives in your Studio depends either on the license you have when you are working in a local project, or on the type of the remote project itself when you are working in remote projects. For further information on user authorization on remote project, see the Talend Administration Center User Guide. For further information about the different perspectives available in the studio, see the user guide corresponding to each perspective Talend Activity Monitoring Console log database If you want to use the Talend Activity Monitoring Console, an <AMC> log database must be created, which can be installed on any server. This <AMC> database will initially be empty. Its name may be modified, but you must take into account this modification in the rest of this document. The <AMC> database will contain three tables that collect data allowing users to monitor Jobs. The three tables will collect data from the following components: tflowmetercatcher, tlogcatcher, tstatcatcher. Instructions on how to create these tables and their structure is provided in the Talend Activity Monitoring Console User Guide. Talend Platform for Data Management Installation and Upgrade Guide 19

26 Operating principles of Talend Platform for Data Management A corresponding SQL user must be created and thus mapped to have access to this database. This user should be granted the create and update rights Operating principles of Talend Platform for Data Management The operating principles of Talend Platform for Data Management could be summarized as briefly as the following topics: building technical or business-related processes and data profiling analyses (graphically represented in light blue and yellow), administrating users, projects, access rights and processes and their dependencies (graphically represented in violet), deploying and executing technical processes (graphically represented in red and dark blue), monitoring the execution of technical processes (graphically represented in orange). Depending on your license, some of the functional blocks may not be available to you. So, each of the above topics can be isolated in different functional blocks and the different types of blocks and their interoperability can be described as below: Building and administrating The light blue block includes one or more Talend Studio APIs and Web browsers that could be on the same or on different machines. 20 Talend Platform for Data Management Installation and Upgrade Guide

27 Deploying and executing From the Studio API, end-users can carry out technical processes and data profiling analyses and reports regardless of data volume and process complexity. The Talend Studio allows the user to work on any project for which he has authorization. For more information, see the Talend Studio User Guide. From a Web browser, end-users connect to the remotely based Administration Center through a secured HTTP protocol. The end-user category in this description may include developers, project managers, administrators and any other person involved in building data flows, Each of these end-users will use either the Studio or the Administration Center or both of them depending on the company policy. Additionally, end-users also use a Web browser to connect to Talend Data Quality Portal that plugs to the data quality data mart (yellow block) and extends the dynamic reporting and monitoring capabilities of the Talend Studio, and to Talend Data Stewardship Console to resolve the data issues included in the listed tasks to reach the golden data record (if they subscribed to this option). The violet block includes a web-based Administration Center (application server) connected to two shared repositories: one based on an SVN server and one based on a database server. The Administration Center enables the management and administration of all projects. Administration metadata (user accounts, access rights and project authorization for example) is stored in the database server and project metadata (Jobs, Business Models, Routines for example) is stored in the SVN server (to easily share them between the different end-users). The Administration Center also enables to configure the tasks that handle job executions and triggers. It also looks after the job generation and deployment to the execution servers. For more information, see the Talend Administration Center User Guide Deploying and executing The dark blue block represents the Talend Artifact Repository that stores all the: Software Updates available for download. The red block represents the execution servers that run technical processes according to the execution scheduling set up in the Talend Administration Center Web application. Those execution servers can be of: One or more Talend Runtimes (execution container) deployed inside your information system. The Talend Runtime deploys and executes the technical processes according to the set up defined in the Talend Administration Center Web application. Those processes are Jobs built from the Studio and centralized on the SVN server.. One or more JobServers deployed inside your information system that run technical processes (Jobs) according to scheduled time, date or event set in the Talend Administration Center Web application. The end-user can transfer technical processes to a remote execution server directly from the Studio (distant run). You must install the JobServer files ( Agent ), delivered by Talend, on each of the execution servers to become operational. For more information, see the section Installing the execution servers (JobServers) Monitoring The orange block represents the monitoring dashboard: the Activity Monitoring Console. Talend Platform for Data Management Installation and Upgrade Guide 21

28 Monitoring The Activity Monitoring Console allows end-users to monitor the execution of technical processes. It provides detailed monitoring capabilities that can be used to consolidate log information collected, understand the interaction between underlying data flows, prevent faults that could be unexpectedly generated and support system management decisions. For more information on the Activity Monitoring Console, see the Talend Activity Monitoring Console User Guide. 22 Talend Platform for Data Management Installation and Upgrade Guide

29 Chapter 3. Installing Talend Platform for Data Management for the first time We strongly encourage you to read the chapter Prior to installing the Talend products and the chapter Introducing Talend Platform for Data Management before starting this chapter. This chapter details the procedures required to install each part of Talend Platform for Data Management. The following parts usually need to be installed in the following order: 1. SVN Server, see the section Setting up your project server on SVN. 2. Application server (Tomcat or JBoss), see the section Deploying the Web application on Tomcat. 3. High Availability, see the section Setting up High Availability. The implementation of this architecture depends on your license. 4. Administration center Web application, see the section Configuring Talend Administration Center (Web application). 5. CommandLine, see the section Installing CommandLine. 6. Talend or Nexus Artifact Repository, see the section Installing and configuring the artifact repository. 7. Execution servers (Jobservers or "agents"), see the section Installing the execution servers (JobServers). 8. Talend Runtime, see the section Installing Talend Runtime. 9. Talend Data Quality Portal, see the section Installing Talend Data Quality Portal. 10.Activity Monitoring Console Web application, see the section Installing the Talend Activity Monitoring Console Web application. 11.JBoss Business Rules Management System, see the section Installing a Business Rules Management System (Drools). Talend Platform for Data Management Installation and Upgrade Guide

30 12.Data Stewardship Console as a standalone application (optional), see the section Installing the standalone Talend Data Stewardship Console (optional). 13.Automatic log purge (optional), see the section Setting up the automatic log purge (optional). In the following documentation, <TomcatPath> designates the Tomcat installation path and <ApplicationPath> designates the directory in <TomcatPath>/Webapps where Talend Administration Center war file has been deployed. 24 Talend Platform for Data Management Installation and Upgrade Guide

31 Setting up your project server on SVN 3.1. Setting up your project server on SVN This section describes how to install the SVN server. This server will store all your project data (Jobs, Database connections, Routines, Joblets, etc.). The installation instructions for Subversion will differ depending on: your development environment, the access method, the authentication mode. The access methods and authentication modes to the Subversion server include: Apache, a Web server and can be used to access a Subversion repository. Only the authenticated access is documented here. Note that VisualSVN has both Apache and Subversion embedded SSH, another method for accessing a Subversion repository. Authentication is mandatory and brings a higher level of security thanks to encrypted communication. Choose the procedure that is most relevant to your environment among the following instruction sections. These sections are independent from each other Windows 1. Download the SVN server installer, for example: 2. Launch the wizard. 3. Clear the Use secure connection ( check box so that you use unsecured HTTP. 4. Complete the installation process without changing the other parameters. 5. Launch VisualSVN. In the main window of VisualSVN Server Manager: Check in the server URL that the port is 80, if not then perform the following operation: 1. Right-click on VisualSVN Server on the left tree view. 2. Select Properties. 3. Click on the Network tab. 4. Change the port selection to Click Apply then OK. Then create a new repository: 1. Right-click on Repositories in the main window of VisualSVN. 2. Click Create new repository... Talend Platform for Data Management Installation and Upgrade Guide 25