WhiteWave's Integrated Managed File Transfer (MFT) Spotlight on webmethods ActiveTransfer Scott Jaynes ecommerce Architect, WhiteWave Foods Ramesh Venkat Director, Product Management, Software AG 10/17/12
October 24, 2012 2 Agenda WhiteWave Foods Introduction File transfer environment & Requirements Lessons Learned File Transfer Patterns & Challenges Introducing webmethods ActiveTransfer Key Take-aways
October 24, 2012 3 WhiteWave Foods CPG company Based in Broomfield, Colorado SAP is company s ERP system (7 years) previously disparate inherited systems webmethods selected as middleware (7 years) B2B Platform Many Standard EDI doc types as well as XML over AS2 Web Services FTP Scott Jaynes ecommerce Architect (7 years)
WhiteWave Foods North American Brands October 24, 2012 4
WhiteWave European Brand October 24, 2012 5
October 24, 2012 6 What is the need for file transfers? Financial documents (secured / encrypted) POS detail (sales reporting / trending) Flat File data feeds from 3 rd party partners
October 24, 2012 7 Typical qualities of file transfers Traverse different networks - LAN/WAN (not same subnet) Usually no processing involved (don t have to open file) Often orders of magnitude larger data than typical ecommerce transactions (EDI, XML, etc.) resulting in long transfer times File polling usually involved at some point during process Very large files (+1GB) not uncommon
October 24, 2012 8 What could go wrong? FTP process picks up file before it is finished writing Network connection severed midstream Login times out Too much traffic at the same time - network bandwidth can be exceeded
October 24, 2012 9 Project Perform bi-directional flat file transfers of sensitive data between WhiteWave and 3 rd party vendor over the internet Only secure protocol 3 rd party vendor supports is FTPS implicit (not common) -current webmethods IS built-in services do not support FTPS implicit Some files over 1GB Event driven architecture desired Tight SLA on some transfers (for FTP)
October 24, 2012 10 Possible architectures 1. Find java libraries to support FTPS implicit, build IS services to perform needed functionality All functionality, including low level services supported in-house Longer time to deliver solution Any issues require core personnel to drop everything and support 2. Use 3 rd party solution to handle FTPS and transport to target systems Usually requires file polling for back-end systems to get file Event driven architecture requires APIs into all systems leading to increased complexity Sub-optimal error handling / notification 3. Hybrid - use 3 rd party solution to handle FTPS and hand off to webmethods to trigger back end system (SAP) 3 rd party solution seeds good APIs / messaging to properly handle all scenarios (esp. errors) Already good APIs / messaging between SAP and webmethods Future ease in replacing 3 rd party solution with integrated wm solution
October 24, 2012 11 Hybrid solution architecture Outbound data flow example 1. SAP generates file and writes it to NFS mount on webmethods server (same network) 2. SAP calls RFC in webmethods to trigger event 3. webmethods calls API in 3 rd party software to trigger event 4. 3 rd party software connects to target external vendor, initiates FTPS session and transfers file Inbound data flow example 1. 3 rd party partner generated file and writes it to their FTP server 2. 3 rd party FTP client software polls (via FTP) for file and transfers it to local directory. At completion, writes done.txt file 3. webmethods polls local directory for done.txt file 4. When found, webmethods service moves and renames file to shared SAP directory (NFS mount) and triggers SAP processing via RFC call
October 24, 2012 12 Conclusions Lessons Learned Traversing 4 systems makes it hard to implement comprehensive consistent error handling 3rd party FTP solutions are not always enterprise ready Added 3 more servers to administer / monitor (Dev, QA, Prod) without adding more personnel Error messaging not consistent messaging out of the 3 rd party solution not as informative as what we can build in webmethods Troubleshooting is complicated, usually requiring logging in and looking at all 4 systems. Usually no one person has access to all involved systems
October 24, 2012 13 Next Steps Interested in upcoming webmethods ActiveTransfer MFT solution Leverage existing webmethods environment batch servers dedicated IS cluster for large, slow, low SLA interfaces Unified error messaging Reports Trending are we heading for a future bottleneck? When? By interface / endpoint is one particular interface slow compared to the rest? (if so, issue is probably not on our side) Points out infrastructure issues (bandwidth, time of day chokepoints, etc.) Simplified troubleshooting
File Transfer Patterns & Challenges
File Transfer Patterns October 24, 2012 15 B2B A2A Cloud Ad hoc Supply Chain Payroll Processing Insurance reimbursements Expense reimbursements Multi-Site Integration (Store/Bank and Headquarters) synchronization File transfer between Open systems and Mainframe Back-up data to the Cloud Data synchronization between Cloud and On-premise apps Human to System File Transfer through a Web interface Human to Human Send files through email
File Transfer Challenges October 24, 2012 16 B2B A2A Cloud Ad hoc Standards support Automated & Process driven Flexible security Control & Visibility Batch Process Automated & Event driven Multiple Operating environments Control & Visibility Batch Driven Large file transfers Control & Visibility Audit and Governance Ease of Use Onboarding Control & Visibility
File Transfer Spaghetti Look Familiar? October 24, 2012 17 Public Internet Trusted Zone FTP Public Cloud Private Cloud FTP Email Suppliers SFTP HTTP Legacy Systems Customers Shippers etc. FTP HTTP Unix, Linux & Windows Servers
October 24, 2012 18 Introducing ActiveTransfer Secure and Control your file transfers Public Internet DMZ Trusted Zone SFTP Private Cloud Public Cloud ActiveTransfer Web Client HTTP/S ActiveTransfer Proxy ActiveTransfer Server Legacy Systems FTP/S Suppliers WebDAV/S Customers SCP Unix, Linux & Windows Servers
ActiveTransfer Server
ActiveTransfer Server: Centralized Administration October 24, 2012 20 Configure and manage all server instances centrally Broad protocol support IP Banning Filename/Folder restrictions Manage users and their access privileges Prioritize file transfers Restrict file transfer actions Inherit user settings from templates
ActiveTransfer Server: Centralized Administration cont d October 24, 2012 21 Post-processing & Alerting File transfer criteria based triggers Perform basic file operations Invoke webmethods services (IS, TN, Broker) Execute native scripts (JCL, ETL jobs) Scheduling & Event-driven transfers Trigger file transfers based on a set schedule Monitor folders for file events and trigger transfers
ActiveTransfer Server: File Acceleration October 24, 2012 22 Accelerate File transfers Accelerate file transfers in high latency connections Accelerate files through HTTP(S) or FTP much more firewall friendly than UDP 15 to 20x increase in speed 22 21 20 19 Active Transfer Tunnel 19 20 21 Active Transfer Server 22 21 20 19 22
ActiveTransfer Server: Centralized Monitoring October 24, 2012 23 Track all file transfers Filter transactions based on transfer criteria View meta data on files exchanged (Filename, Status, Size, Date & Time, etc.) Manually retransfer failed transfers View audit trail of all activities related to file transfer Track file transfer activities including post-processing actions.
ActiveTransfer Server: MFT Analytics October 24, 2012 24 Executive Dashboards Actionable intelligence on file transfers Analyze key metrics based on file transfer volume, average transfer rates, etc. Present Charts/graphs with drill-downs Slice/dice data displayed through filters Export data as CSV or images
ActiveTransfer Proxy Server & Web Client
Trusted Zone October 24, 2012 26 ActiveTransfer Proxy Server Runs on Reverse Gateway Integration Server Public Internet DMZ Supports all open protocols (FTP/S, HTTP/S, SFTP, etc.) SFTP Secure Authentication No ports opened in the inner firewall File streaming support No mandatory storage of files in DMZ Public Cloud ActiveTransfer Web Client Suppliers HTTP/S FTP/S ActiveTransfer Proxy High Availability support WebDAV/S SCP Customers
October 24, 2012 27 ActiveTransfer Web Client Web interface for all file transfer operations (Upload, Download, Rename, etc.) Quick Filtering for downloads Ad hoc file transfer for sharing files Automatic zipping of files for uploads/downloads Checkpoint restart and Acceleration supported Customize web interface
Key Take-aways
Remember WhiteWave s Issues? October 24, 2012 29 How can we address them? FTP process picks up file before it is finished writing ActiveTransfer can pick up files only after completion Network connection severed midstream ActiveTransfer can resume transfers from the point of interruption Login times out ActiveTransfer can restart transfers Too much traffic at the same time - network bandwidth can be exceeded ActiveTransfer can prioritize file transfers and ensure the transfer of most important files meeting SLAs
What to Remember October 24, 2012 30 3 Key things to remember from this presentation Secure and Control all of your File Transfers Enhance your ESB with File Oriented Integration Consolidate on one platform for your B2B & MFT needs
Thank You!