WebSphere MQ Managed File Transfer. Parineeta Mattur

Transcription

1 WebSphere MQ Managed File Transfer Parineeta Mattur

2 Agenda Basic FTP What is Managed File Transfer? WebSphere MQ File Transfer Edition The Three Key Components of FTE Integration with MQ Networks Data Transfer FTE Tools Overview of FTE facilities Advantages of MFT A Recap Demo 02/21/14 WMQ MFT 2

3 How do most organizations move files today? Currently, many business critical applications connect by exchanging files Most organizations have several products, and different techniques for doing file transfer Typically there is a mix of FTP, homegrown, and other file transfer products 02/21/14 WMQ MFT 3

4 Why FTP Lowest common denominator Quick fix repent at leisure Simple concepts low technical skills to get started FTP products are free, simple, intuitive and ubiquitous 02/21/14 WMQ MFT 4

5 Shortcomings of Basic FTP Limited Reliability Not every implementation supports resuming failed transfers Limited Visibility Often no monitoring facilities No facility to centralize control over what gets transferred Limited Traceability Logs often held on the machine performing the transfer Difficult to track a file s entire journey from start to finish 02/21/14 WMQ MFT 5

6 What is Managed File Transfer? There is no standard that dictates what Managed File Transfer is and is not Software that overcomes the problems inherent with traditional methods of transferring files WebSphere MQ File Transfer Edition is a Managed File Transfer Solution 02/21/14 WMQ MFT 6

7 Introducing WebSphere MQ Managed File Transfer Adds file transfer capabilities to WebSphere MQ to enable movement of files regardless of size in a managed way (reliable, auditable, secure) Flexible backbone for transfers move files from anywhere to anywhere in your network Multi-purpose use for both files and messages Auditable with logging subsystem that tracks transfer at source and at destination for audit purposes 02/21/14 WMQ MFT 7

8 Flexible backbone for transfers Contd... Centralized control and configuration Integration with MQ-enabled apps and ESBs No need to program no need to use APIs Automatic file conversion and compression Security of file payload using SSL 02/21/14 WMQ MFT 8

9 The Three Key Components of FTE Agents Long running MQ application Run on system where files are to be transferred from / to An access point for sending or receiving files. Each Agent is associated to a queue manager(local/remote) For Agents to communicate with each other there needs to be a communication path between associated QM. SYSTEM A SYSTEM B AGENT AGENT WebSphere MQ Network 02/21/14 WMQ MFT 9

10 Transfer Control AGENT Command Command AGENT Negotiate Transfer Data transfer Negotiation Reply Completion Simplified version - actual exchange has more messages. 02/21/14 WMQ MFT 10

11 Agent Transactions Each stage of a transfer is protected by a transaction. It remembers the status of the transfer No transfers are lost. No repeated transferring of files. For large files, transferring will continue from where it last stopped (check point). Each transaction includes updating internal status of a transfer which is also stored in a queue. AGENT Agent QM Persist 02/21/14 WMQ MFT 11

12 Agent Network Agent connections to MQ Client mode (TCP/IP) Local or Remote agent Server mode (Binding) Local only agents AGENT AGENT AGENT Agent QM AGENT AGENT AGENT Use QM interconnection facilities to communicate between Qms Channels Clusters Agent QM Agent QM AGENT AGENT 02/21/14 WMQ MFT 12

13 The Three Key Components of FTE: 2) Commands Instruct agents. For example, to start or cancel a transfer Use MQ messaging to relay the instructions A queue manager that the commands connect to is playing the command queue manager role GUI or command line 02/21/14 WMQ MFT 13

14 The Three Key Components of FTE: 3) Logging / Monitoring Acts as a collection point for monitoring and logging data How are my transfers progressing? What files were transferred? The queue manager that the logging / monitoring tools connect to is performing the coordination queue manager role The coordination queue manager must be MQ V7 Uses publish-subscribe capabilities 02/21/14 WMQ MFT 14

15 Coordination Queue Manager Two additional QMs for each FTE network Command Coordination AGENT AGENT Command QM All command requests (transfers etc) for the FTE network are sent to this QM. This QM will have been configured to forward the request onto the relevant agent s QM. Agent QM Agent QM Coordination QM Information generated by an agent (status,logs etc) is forwarded to this QM and is stored on a topic. Other applications can subscribe and collect the information from all the agents within an FTE network. Command QM Coordination QM Command and Coordination can be the same QM. MQ v7 02/21/14 WMQ MFT 15

16 Integration with MQ Networks Using a Single Queue Manager 02/21/14 WMQ MFT 16

17 MQ Network Integration: Many Queue Managers 02/21/14 WMQ MFT 17

18 Data Transfer Subdivides data into frames and panels Incoming reply queue Rebuilds the panels and then frames. Acknowledge completed frames Outgoing data queue Sender Side Source Data is loaded into frames Frames are subdivided into panels Destination A panel is transferred as a single MQ message Receiver Side Read message into a panel Acknowledge frames Generate data from frames 02/21/14 WMQ MFT 18

19 Tools The following commands generate XML messages and submit them to the Command QM and then await a reply on a temporary queue. ftecreatetransferrequest create a transfer request based on the given arguments ftecanceltransfer cancel a previously requested transfer. ftepingagent simple ping request to a named agent and await reply. 02/21/14 WMQ MFT 19

20 Tools The following commands subscribe to the coordination topic ftelistagents give a simple list of agents that have reported status to the topic. fteshowagentdetails an expand status of agents. The following commands are for configuration ftecreateagent generates the necessary file structure for an agent including MQSC script files to configure the agent s QM. ftesetupcommands/ftesetupcoordination generates the necessary file structure for a Command/coordination definition and the relevant MQSC scripts for the coordination topics. 02/21/14 WMQ MFT 20

21 Overview of FTE facilities Transfer in binary or text. Text transfers support Code page translation (perform on destination agent) End of line translation Multiple files in single transfers Source directory with or without recursion. Send one or more files to a named directory Send one or more files to a data set. 02/21/14 WMQ MFT 21

22 Overview of FTE facilities Source Disposition - Delete source file on successful transfer Destination file already present - Error mark that file as failed (continue with others) Overwrite the existing file. User exits Allow service or clients to extend the capabilities of FTE. - Virus scanning on received files. - Informing a system a file has been received ( ). 02/21/14 WMQ MFT 22

23 Overview of FTE facilities Resource Monitor Agent has a resource monitor which can monitor directories for file activity. Create a monitor definition to monitor a directory. Trigger can filter for files in the directory or optionally in sub-directories. Each monitor polls the resource at a given period. Monitor fires when a file matching the filter has Been created Last modified date has changed since last polled. 02/21/14 WMQ MFT 23

24 Overview of FTE facilities Resource Monitor Once fired, the monitor modifies a supplied transfer request by substituting values into the message based on the file that triggered the monitor. Practical application - File Sorting at office A monitor polls directories within the monitor directory (root) for files. When a file is detected, the name of the directory it was found in is the name of the destination agent. A transfer request is generated and the destination agent is changed for the directory name. Source disposition of delete. 02/21/14 WMQ MFT 24

25 Overview of FTE facilities Database logger The database logger captures transfer log information and stores on a database Ideally lives on the coordination QM. Independent of any agent in the FTE network. Supports DB2 or Oracle. 02/21/14 WMQ MFT 25

26 Overview of FTE facilities Diagnostics Passive Transfer logs / Agent status stored on the topic. Output.log file storing event information one per agent C:\Program Files (x86)\ibm\websphere MQ\mqft\logs\QM\agents\IVTAGENT\logs. Generates FFDC. Active Trace log to file based on the MQ trace. Filter by class and/or level. ftepingagent is it up test. 02/21/14 WMQ MFT 26

27 Overview of FTE facilities GUI Plugin for the MQ explorer Create transfer, monitors, etc View transfer log Show progress of a transfer 02/21/14 WMQ MFT 27

28 Advantages of MFT A Recap A Consolidated Transport for Files and MessagesTraditional approaches to file transfer result in parallel infrastructures One for files typically built on FTP One for application messaging based on WebSphere MQ, or similar High degree of duplication in creating and maintaining the two infrastructures Consolidating messaging and file transports yields: Operational savings and simplification Reduced administration effort Reduced skills requirements and maintenance 02/21/14 28 WMQ MFT

29 Advantages of MFT A Recap Contd... A managed file transfer. Transfers a file from one location to another Large files Lots of small files. Managed it will complete the transfer (eventually). record the transfer event. Translations text mode Code page End of line Platforms File types (plain, datasets). 02/21/14 WMQ MFT 29

30 Advantages of MFT A Recap Contd... Secure SSL Move a block of data from System A to System B Fastest possible way Fast transport Overlapping actions Acknowledgement on completion. 02/21/14 WMQ MFT 30

31 Questions??

32 Thank you