Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition IBM Corporation

Transcription

1 Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition

2 Agenda 2 Introduction MQ FTE Overview DataPower File handling Overview File transfer scenarios with DataPower / MQ FTE DataPower MQ FTE Configuration

3 File Transfer within the Enterprise Moving Files and Documents around internally within Head-office Moving Files and Documents between Head-office and remote locations Branches Regional Locations Head Office Regional Offices Subsidiaries CRM Packaged Apps Stores Accounting Inventory Sales Warehouses Orders

4 How do most organizations move files today? Most organizations rely on a mix of homegrown code, several legacy products and different technologies and even people! FTP Typically File Transfer Protocol (FTP) is combined with writing and maintaining homegrown code to address its limitations Why is FTP use so widespread? FTP is widely available Lowest common denominator Promises a quick fix repent at leisure Simple concepts low technical skills needed to get started FTP products seem free, simple, intuitive and ubiquitous Legacy File Transfer products A combination of products often used to provide silo solutions Often based on proprietary versions of FTP protocol Can t transport other forms of data besides files Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure especially with SOA People From IT Staff to Business staff and even Security Personnel Using a combination of , fax, phone, mail, memory keys 4

5 Shortcomings of Basic FTP Limited Reliability Unreliable delivery Lacking checkpoint restart Files can be lost Transfers can terminate without notification or any record corrupt or partial files can be accidentally used File data can be unusable after transfer lack of Character Set conversion Limited Flexibility Changes to file transfers often require updates to many ftp scripts that are typically scattered across machines and require platform-specific skills to alter All resources usually have to be available concurrently Often only one ftp transfer can run at a time Typically transfers cannot be prioritized 5 Limited Security Often usernames and passwords are sent with file as plain text! Privacy, authentication and encryption often not be available Non-repudiation often lacking Limited visibility and traceability Transfers cannot be monitored and managed centrally or remotely Logging capabilities may be limited and may only record transfers between directly connected systems Cannot track the entire journey of files not just from one machine to the next but from the start of its journey to its final destination

6 What is Managed File Transfer? Reliable, controlled, auditable movement of files No agreed specification for managed file transfer products to certify against General consensus that managed file transfer involves the following: Auditable Reliable Secure Automated Who transferred a file? Where? When? Was it this file? Centralized Any file size Integrated Cost Effective Can be monitored and managed from one central location 6 Automatic resumption of interrupted transfers. No partial file data left lying around Limits access to authorized users. Protects file data in transit Designed for lights out operation Imposes no practical limits on file sizes. Efficient regardless of file size Integrates well with applications that typically perform file processing Reuses existing skills and infrastructure

7 Agenda 7 Introduction MQ FTE Overview DataPower File handling Overview File transfer scenarios with DataPower / MQ FTE DataPower MQ FTE Configuration

8 What is WebSphere MQ File Transfer Edition? Adds managed file transfer capabilities to WebSphere MQ WebSphere MQ File Transfer Edition A Auditable Reliable Secure Automated Centralized Any file size Integrated Cost Effective 8 B C X Y Z Full logging and auditing of file transfers + archive audit data to a database Checkpoint restart. Exploits solid reliability of WebSphere MQ Protects file data in transit using SSL. Provides end-to-end encryption using AMS Providing scheduling and file watching capabilities for event-driven transfers Provides centralized monitoring and deployment of file transfer activities Efficiently handles anything from bytes to terabytes Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct Reuses investment in WebSphere MQ. Wide range of support ( inc. z/os and IBM i)

9 A consolidated transport for both files and messages Traditional approaches to file transfer result in parallel infrastructures One for files typically built on FTP One for application messaging based on WebSphere MQ, or similar High degree of duplication in creating and maintaining the two infrastructures File Transfers Application Messaging File Transfer Edition reuses the MQ network for managed file transfer and yields: Operational savings and simplification Reduced administration effort Reduced skills requirements and maintenance 9 Consolidated Transport for Messages & Files

10 Components of a typical WMQ FTE network Applications exchanging file data s WebSphere MQ Coordination Queue Manager A historical record of file transfers Coordination queue manager Gathers together file transfer events Commands 10 Send instructions to agents Log database Commands The endpoints for managed file transfer operations Log Database

11 Components of a typical WMQ FTE network s Applications exchanging file data Act as the end points for file transfers Long running MQ applications that transfer files by splitting them into MQ messages Multi-threaded file transfers WebSphere MQ Coordination Queue Manager This can be used for audit purposes Associated with one particular queue manager (either v6 or v7) 11 Can both send and receive multiple files at the same time Generate a log of file transfer activities which is sent to the coordination queue manager Efficient transfer protocol avoids excessive use of MQ log space or messages building up on queues state on queues Commands Log Database

12 NOTES Notes on: s FTE agent processes define the end-points for file transfer. That is to say that if you want to move files off a machine, or onto a machine that machine would typically need to be running an agent process processes are long running MQ applications that oversee the process of moving file data in a managed way. Each agent monitors a command queue waiting for messages which instruct it to carry out work, for example file transfers The FTE agent process needs connectivity to an MQ queue manager to do useful work. It can connect either directly to a queue manager running on the same system, or as an MQ client using an embedded version of the MQ client library (which is kept completely separate to any other MQ client libraries that may or may not already have been installed onto the system)* Each agent requires its own set of MQ queues which means that an agent is tied to the queue manager where these queues are defined However one queue manager can support multiple agents * Note: availability of direct (bindings) connectivity or MQ client based connectivity is dependent on the version of MQ FTE in use WebSphere MQ File Transfer Edition on z/os does not support the MQ client style of connectivity File Transfer Edition on distributed platforms has a server and client offering. The agent component of the client offering is restricted to only supporting MQ client style connectivity. The agent component of the server offering may be used either connectivity options 12

13 Example usage of agent monitoring + program execution Existing Application Application writes file to file system FTE transports file to destination WMQ FTE 2. monitors file system, spots arrival of file and based on rules, transfers the file WMQ FTE 4. FTE can also start another application to process the file p* a t * Existing Application At destination MQ FTE writes file to file system

14 NOTES Notes on: 14 Monitoring & Program Execution Resource monitors work in two stages: 1. Poll a resource (in this case the file system but as we ll see later the resource can also be an MQ queue) and identify that a condition has been met (perhaps the appearance of a file matching a particular pattern) 2. Perform an action (which can include starting a managed file transfer, or running a script), optionally propagating information about the resource (for example the name of the file triggered on) into the action As shown on the previous slide, resource monitors are typically used to provide integration with an existing system without needing to make changes to the system Another function of FTE, used for integration with existing systems is the ability to execute programs or scripts both on the source or destination systems for a file transfer. This can be used to: Start a program, on the source system, which generates the file data to be transferred prior to performing the managed file transfer Start, or notify, a program on the destination system when the file data has been transferred allowing it to process the data without having to poll

15 Components of a typical WMQ FTE network Commands Applications exchanging file data 15 Send instructions to agents and display information about agent configuration Via MQ messages Many implementations of commands: MQ Explorer plug-in Command line programs Open scripting language JCL Documented interface to program to WebSphere MQ Coordination Queue Manager Commands Log Database

16 NOTES Notes on: 16 Commands Commands is the name we have given to anything which instructs the FTE agent. As described on the previous slide, there are a wide range of command implementations including graphical and non-graphical command-line based commands Commands instruct the FTE agent by sending it messages. The messages themselves use a documented format which can easily be incorporated into your own applications The commands that are supplied with FTE can connect either as an MQ client (again based on embedded client libraries) or directly to a queue manager located on the same system

17 Components of a typical WMQ FTE network Log Database Applications exchanging file data Keeps a historical account of transfers that have taken place Implemented by the database logger component which connects to the coordination queue manager WebSphere MQ Coordination Queue Manager Stand alone application Or JEE application Queryable via Web Gateway 17 Who, where, when etc. Also a documented interface Commands Log Database

18 NOTES Notes on: 18 Log Database File Transfer Edition can record a historical account of file transfers to a database using the database logger component. This component is available to run as a stand alone process or as a JEE application The information used to populate the log database is generated, as MQ messages, by the FTE agents participating in file transfers. This is routed to a collection point in the MQ network, referred to as the coordination queue manager (see next slides). The database logger component subscribes to the messages produced by agents and reliably enters them into a database.

19 Components of a typical WMQ FTE network Coordination Queue Manager Applications exchanging file data Gathers together information about events in the file transfer network Not a single point of failure WebSphere MQ Coordination Queue Manager MQ v7 publish / subscribe 19 Can be made highly available Messages stored + forwarded Allows multiple log databases, command installs Documented interface Commands Log Database

20 NOTES Notes on: Coordination queue manager The coordination queue manager is used as the gathering point for all the information about file transfers taking place between a collection of FTE agents The queue manager uses publish/subscribe (so it must be MQ version 7) to distribute this information to interested parties which typically include: The WebSphere MQ Explorer plug-in, which provides a graphical overview of FTE activity The database logger component, which archives the information to a database Some of the command line utilities which are part of the FTE product The format used to publish information is documented and can be used to develop 3rd party applications which process this data Although there is only a single coordination queue manager for a given collection of agents it does not represent a point of failure: MQ stores and forwards messages to the coordination queue manager when it is available so if the coordination queue manager is temporarily unavailable no log data is lost The coordination queue manager can be made highly available using standard HA techniques such as MQ multi-instance queue manager or via a HA product such as PowerHA 20

21 WebSphere MQ File Transfer Edition Tools Command Line tools For operations (for example transferring a file) Administration (for example starting an agent) GUI eclipse based plug-in to MQ Explorer Used for operations only (for example transferring files, browsing transfer log history)

22 How to start a File Transfer Command Line Interface is consistent across all supported platforms Transfer commands can be invoked from the supported Operating Systems shell environment Commands can be invoked from anywhere across the file transfer network i.e. Command could be invoked from a Windows machine for transfers taking place between z/os and Unix machines Developers can use any native command line language on the OS that can invoke these commands (shell, bat, cmd, etc.) Application Programs can place a request using a messaging interface in XML Examples: ftecreatetransfer ftestart ftestop fteshowdetails fteshows Page 22 Starts a new file transfer from the command line Starts a File Transfer agent from the command line Stops a File Transfer agent in a controlled way Displays the details of a particular File Transfer agent Displays the status of all known Transfer agents

23 Eclipse GUI integrated into WMQ Explorer

24 Scheduling & Triggering File Transfers Choose when to start the scheduled transfer Choose advanced options Choose when to repeat the scheduled transfer and how often Choose the trigger for the transfer Deploy file transfer to network Page 24

25 Auditing & Monitoring File Transfers Page 25

26 Agenda 26 Introduction MQ FTE Overview DataPower File handling Overview File transfer scenarios with DataPower / MQ FTE DataPower MQ FTE Configuration

27 The Modern Enterprise is a Network of Complex Interactions Wholesalers Affiliates Retailer Internal Employees B2C B2B

28 Rich Portfolio of Workload Optimized Systems and Appliances Information Netezza Retail Analytic Appliance Netezza Skimmer Netezza TwinFin Smart Analytics System Performance WebSphere DataPower Caching Appliance WebSphere DataPower Low Latency Appliance WebSphere DataPower XML Accelerator Power7 with WebSphere and DB2 Smart Analytics Optimizer Management Application Manager for Smart Business IBM CloudBurst Rational Power Appliance Express Rational Power Appliance Workgroup Rational Power Appliance Enterprise Service Manager for Smart Business Smart Business Appliance Virtual Desktop for Smart Business WebSphere CloudBurst Appliance Storage Information Archive SAN Volume Controller Scale Out Network Attached Storage System Storage Productivity Center Real Time Compression Appliance 28 Connectivity WebSphere Cast Iron Cloud Integration WebSphere DataPower B2B Appliance WebSphere DataPower Integration Security InfoSphere Guardium Real-time Database Monitoring and Security Proventia Management SiteProtector System Proventia Network Enterprise Scanner Proventia Network Multi-Function Security Security Network Active Bypass Security Network Intrusion Prevention System WebSphere DataPower XML Security Gateway

29 An Appliance Analogy SIMPLE PURPOSE BUILT WORKHORSE

30 The Connectivity Appliance Recipe 1) Take some characteristics from integration software stacks Programmability by developers Message level routing and security (including by partner for B2B) Integration with other SOA infrastructure & governance software such as Registry, Repository, etc Conditional Processing (if, else, split, join) Transformation of any-2-any formats them into a hardware device (aka hardware form factor) = 3)thatCombine has characteristics of both application stacks and networking gear. 2) Take some characteristics of networking & firewall devices High throughput Many concurrent connections IP level security and routing (can t see inside message but route by source and target) Simplified Management and reduced maintenance costs through firmware updates Reliability

31 IBM WebSphere DataPower appliances Simple architecture: microcode firmware + purpose-built hardware Delivered from the factory with everything you need to connect to the network and start working No need to provision anything but the Ethernet network and CAT cables to get started All computationally-significant components sealed within a temper-proof casing Chips Memory Boards and cards Flash-based file system (signed and encrypted) Parsing and xform accelerators (patented) Cryptographic accelerators (patented)

32 Why use WebSphere DataPower appliances? Business Requirements Configuration driven, not programmable Relief from maintaining diverse development and system administrator skills Simplified IT deployments Appliances are Fit for Purpose, Easily Consumable A drop-in, plug-n-work device Hardened, with numerous functions integrated into a single device Easy configuration-driven set up and management Purpose-built hardware / software that delivers on specific business needs 32

33 WebSphere DataPower XI52 and XB62 Appliances (9005) 2U high density rackmount design New customer-replaceable Ethernet modules with far more ports than the current generation (4) Two 10 Gbps ports Eight 1 Gbps ports Far more memory than ever before Memory aggregation at the domain level (announced for firmware 4.01) Enhanced hardware diagnostics Customizable intrusion detection Higher performance chips, memory, flash, and hard drives RAID RAIDmirroring mirroring and andstriping striping across acrossfour four drives drives Gigabit 1-Gigabit Ethernet EthernetNICs NICs Gigabit 10-Gigabit Ethernet EthernetNICs NICs

34 DataPower Devices DataPower appliances have been the best selling WebSphere product since the DataPower company acquisition in 2005 The Russian Doll Guiding philosophy is to take rote, repeatable integration tasks and lock them down in the appliance form factor, including Parsing Validation Transformation Security Integration B2B (XML, Flat) (XML, Flat) (XML, Flat) (Transport, XML, other) (Multi-transport, SOA, ESB) (Partner Mgt, Persistent Tx Store)

35 Common usage patterns Internet DMZ Trusted Domain Applications 1 Web Gateway Security policy enforcement SLA policy enforcement Web Application Firewall Devices Web Apps 2 Integration Gateway Web Gateway functions + Any-to-any transformation Protocol Bridging Partner Apps 3 B2B Gateway 2.1 ESB Integration Gateway functions + Routing Database connectivity more... System z 2.2 Service Enablement ESB functions + Rapid web service façade creation Integration Gateway functions + Partner Management Enhanced qualities of service 35

36 Web Gateway Connection from client Proxying and Enforcement Terminate incoming connection Terminate transport-level security Partner App Enforce Service Level Agreement policies Inspect message content, filter, pattern-match Enforce security policies on message content Call out to Access Control List(s) Detach binaries and call out to virus checker ACL Virus Scanner Transform content (XSLT, XML-to-XML) Establish a new connection to pass results New connection to target Internal App 36

37 Integration Gateway All the same capabilities as the XS40 to proxy and protect traffic, PLUS Support for additional transport protocols: WTX Design Studio FTP(s), SFTP(SSH), WebSphere MQ, WebSphere JMS, TIBCO EMS, MQ FTE Database connectivity (ODBC) File system access (NFS) Web 2.0 standards: ReST, JSON Onboard engine for parsing and transforming virtually any message format WebSphere Transformation Extender (WTX) 37

38 B2B Gateway ebms / All of the capabilities of the XS40 to proxy and enforce policies All of the additional capabilities of the XI50 for protocol bridging and universal transformation, PLUS Partner Management functions: Define partners with the web management console Associate partners with network endpoints Attach metadata about the partners to their definitions Enhanced Qualities of Service Onboard persistent transaction store Search messages by partner, time, etc Replay messages if necessary ebxml/ebms, AS1, AS2, and AS3 protocol bindings for greater reliability across traditionally unreliable protocols

39 Business to Business (B2B) Appliance XB60 / XB62 Purpose-built B2B hardware for simplified deployment, exceptional performance and hardened security Extend integration beyond the enterprise with B2B Hardened Security for DMZ deployments Easily manage and connect to trading partners using industry standards Simplified deployment and ongoing management Trading Partner Management for B2B Governance; B2B protocol policy enforcement, access control, message filtering, and data security Application Integration with B2B Gateway Service capabilities supporting B2B patterns for AS1, AS2, AS3, ebms v2.0, FTP(S), SFTP, HTTP(S), SFTP/POP3, MQ, MQ FTE, JMS and more Full featured User Interface for B2B configuration and transaction viewing; correlate documents and acknowledgments displaying all associated events Simplified deployment, configuration and management providing a quicker time to value by establishing rapid connectivity to trading partners

40 XB60 complements IBM s family of WebSphere software solutions Internet DMZ Trusted Domain WebSphere MQ FTE WebSphere DataPower XB60 Applications Partner WebSphere Transformation Extender / Trading Manager WPS, WESB, WMB, DP WebSphere Partner Gateway / Sterling Integrator Standalone Consolidated Patterns Deploy standalone for purpose built B2B gateway functionality in the DMZ utilizing exceptional security and B2B transaction volumes for quickly connecting to trading partners Enterprise B2B Integration Pattern Deploy with MQFTE for B2B enabled Managed File Transfer Deploy with WTX-TM for end-to-end EDI Processing Deploy as B2B entry point for BPM and ESB solutions Supplement WPG or Sterling Integrator by offloading security functions and advanced web services

41 Agenda Introduction MQ FTE Overview DataPower File handling Overview File transfer scenarios with DataPower / MQ FTE DataPower MQ FTE Configuration 41

42 XB60 MFT Integration Pattern Integrate to MQ FTE for endto-end file transfer Enterprise Trading Partner Browser (Admin) XB60 2 Multi-Protocol Gateway Service Server Server 3 WS MQ WS MQ 1 Internet 4a B2B Gateway Service Profile Mgmt 01 4b WAN Data Store XB60 Data Store Transaction Viewer 02 6 MQ Explorer FTE Viewer 5 Application Browser (Partner view) Browser (LOB User)

43 Message Segmentation for Large Files If the payload of the received message is too big, the message payload will be segmented into smaller pieces before forwarding to the queue Can also be configured for streaming 43

44 Outbound Logical flow of data MQ File Transfer Edition portion of transfer File Transfer Sender Source MQFTE Network DataPower portion of transfer Partner Destination Appliance Message MQ queue Message 1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower 2. Destination agent places file data as one or more messages in a group. 3. MQFTE front-side handler consumes message(s) from the queue, reassemble the message if there is more than one segments, and sends the data to the external partner 44

45 Agenda Introduction MQ FTE Overview DataPower File handling Overview File transfer scenarios with DataPower / MQ FTE DataPower MQ FTE Configuration 45

46 Outbound from MQ FTE to DataPower... FTE : APC QMGR : QMPC MQ queue : FTE2DP Appliance Network

47 Using a File to Queue transfer to the FTE2DP queue... FTE : APC QMGR : QMPC MQ queue : FTE2DP Appliance Network

48 MQFTE & DataPower XB60 Integration Headers MQFTE Source can optionally configure the following metadata when integrating with XB60. When set, B2B Gateway will use these information to match the right partner profile settings. DPMQFTESenderID : the business ID of internal partner DPMQFTEReceiverID : the business ID of external partner 48 DPMQFTEContentType : the content type of message payload

49 Set up of the queue manager connections within DataPower... FTE : APC QMGR : QMPC MQ queue : FTE2DP Appliance Network

50 MQFTE Front Side Protocol Handler Configuration The Queue manager that host the shared queue The queue where we retrieve the message segments 50

51 Transactionality Support Logical flow of data DataPower portion of transfer MQ File Transfer Edition portion of transfer File Transfer Sender Source MQFTE Network 4 (error) Partner Destination 1 Get Queue 2 3 Appliance Backout Queue 1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower 2. Destination agent places file data as one or more messages in a group. 3. MQFTE front-side handler consumes message(s) from the queue 4. Error happens in transaction processing and MQFTE front-side handler rolls the transfer back and. Go back to step 1 to retry the transfer. 5. If the number of retries reaches the backout threshold, then MQFTE front-side handler will send the transfer to the backout queue.. 51

52 Inbound to DataPower and MQ FTE Network Appliance MQ queue Network...

53 The DataPower Multi-Protocol Gateway with FTE backend URL Network Appliance MQ queue Network dpmqfte://ftewa/? RequestQueue=DP2FTE&Dest =APC& DestQM=QMPC&DestFile=testcompl ete.xml...

54 MQFTE URLOpener Parameters Mandatory Dest : the destination agent in MQFTE network that will ultimately receive the file DestQM : the destination queue manager the source agent should send the messages to DestFile : the filename to use when the destination agent need to store the received messages into file. Unique filename is supported. Optional SenderID : the business ID of sender (e.g. external partner ) ReceiverID : the business ID of receiver (e.g. internal partner) ContentType : the content type of the message payload Example : dpmqfte://qm_demo/? RequestQueue=Q_in&Dest=Kai&DestQM=dummyQM&DestFile=dummyPath 54

55 Nothing MQ FTE specific in the Policy

56 The FTE Monitor configured to accept routing from DataPower Network agent="${dpmqftedestination}" Appliance QMgr="${DPMQFTEDestinationQM}" MQ queue <file>c:\temp\$ {DPMQFTEDestinationFile}</file> Network FTE : AWA QMGR : FTEWA...

57 View MQ FTE Transactions in the B2B Viewer(XB60 only) DataPower portion of transfer Outbound GW MQ File Transfer Edition portion of transfer Inbound GW Appliance Back-end System MQ queue 4 1. Outbound gateway sends a B2B message to inbound gateway 2. The DataPower appliance writes data to an MQ queue as one or more messages in a group. The message ID assigned to the first message in the group becomes the Integration ID. Other information can be passed in the RFH 2 header of the first message 57 Network MQFTE Logger 3. FTE consumes the group of messages and uses the information present in the first message in the group to route the data to a back end system, where it is written as a file 4. B2B Viewer of the outbound gateway retrieve the transaction metadata from the MQFTE Logger Database

58 New Integration ID Column in B2B Viewer ( XB60 only) A new column to store the integration ID which could be used to correlate a B2B Transaction with a backend transaction For MQFTE integration, the integration ID maps to the transfer ID of MQFTE network For inbound, the integration-id is the Message ID of first message we sent to the shared queue For outbound, the integration-id is the Message ID of the first message we received from the shared queue 58

59 MQ FTE Data View in XB60 - The metadata display popup windows A pop-up to display the MQFTE metadata when the integration-id is clicked ( for MQFTErelated transaction only) Depending on weather the transaction is inbound or outbound, the popup window will show different metadata 59

60 DataPower and MQ FTE solution benefits Integration between the B2B appliance and backend application is over WebSphere MQ instead of a shared file system. Files transferred between the B2B appliance and MQ FTE can be correlated using the integration ID from MQ; this ID can also be seen in the B2B Viewer. The combined B2B messaging flow through the B2B appliance and file transfer flow through MQ FTE can be viewed through the B2B Viewer on the appliance. This provides the user with an end-to-end view of the file transfer. File transfers can be set up to occur at specified times or dates, or repeated at specified intervals. File transfers can also be triggered by a range of system events, such as new files or updated files.

61