Encrypted File Transfer - Customer Testing

Encrypted File Transfer - Customer Testing V1.0 David Wickens McKesson

CLASSIFICATION McKesson Technical Guidance Documentation: NOT PROTECTIVELY MARKED VERSION 1.0 SCOPE This guidance document is aimed at skilled IT staff involved in testing connections to the McKesson HUB which are in the process of migrating from FTP to FTPS. Encrypted File Transfer - Customer Testing Contents 1. Encrypted File Transfer - Customer Testing 1. Summary 2. Test Objectives 3. Test Location 4. Performance Testing Summary 5. FTPS 1. 2. 3. 4. Test Tools 1. curl 2. FileZilla Testing a FTPS Server using curl 1. Basic Diagnostic Options 2. Testing Guidelines Testing a FTPS Server using the FileZilla Client 1. Connection Configuration: FTP over TLS 2. Testing Guidelines Common FTP over TLS Problems 1. Client Fails to Connect to Port 21 2. Client Cannot Connect in Encrypted Mode 3. Client Fails to Switch to Encrypted Mode 4. Client Fails to Open Data Channel 5. Client Has Excessive Filesystem Access 6. sftp 1. Test Tool 2. Testing an OpenSSH sftp server from Linux 1. Creating and Deploying a Public Key 2. Connection Test 3. Common sftp Problems 1. Client Fails to Connect to Port 22 2. Client's Public Key is Ignored 3. Interactive Access Allowed 4. Client has Excessive Filesystem Access 7. References This document is intended to provide assistance for ESR customers in testing the encrypted transfer functionality of their FTP or sftp servers. This testing must be done prior to requesting McKesson to change their transfer details from FTP to either FTP over TLS (FTPS) or sftp (OpenSSH v2). Test Objectives You need to know: 1/12

Whether the interface delivers files to your server; Whether the interface collects files from your server; The technical set-up details for either FTP over TLS (FTPS) or sftp. The technical set-up details are present in the FTPS Project Customer General Advice Guidance document. You need to show that: Your server can accept encrypted transfers from ESR interfaces that deliver files; Your server will allow encrypted collection from ESR interfaces that collect files. Test Location It is often possible to perform the testing either: on the same machine as the FTPS or sftp server; on a different machine on the same IP subnet; on a different machine on another IP subnet. If testing on the same machine as the FTPS or sftp server, any network routing or security measures are often eliminated. This can make initial testing easier so you can establish that the FTPS or sftp server has basic functionality. On the other hand, it can give an overly optimistic view of how this server will work on a real network. Testing from a different machine on the same IP subnet will give a better view of real use but, on many networks, the wide area network considerations are still minimised or eliminated. Testing from a different machine on a different IP subnet gives some assurance that your FTPS or sftp server will respond as expected from at least one other subnet. The testing may encounter your typical local network conditions and overcoming these problems may reduce later problems. One difficulty with this sort of testing is that it may be difficult or costly if firewall access has to be arranged. If you can test from the N3 network edge to your server, you have the optimal test position. Performance Testing Encryption has an overhead and this is primarily in the processor time taken to do the encryption and decryption. On relatively new equipment, small isolated transfers would not be expected to have a significant impact; however large or numerous transfers might impact on the system performance. To make a reasonable assessment, you need to know the typical load in terms of frequency and size of files transferred. Measure the time for a single typical transfer with and without encryption and multiply this up to obtain an estimate of the total impact of the change. It is most important to do this assessment when large or numerous transfers are part of the expected load. FTPS Test Tools For testing FTP over TLS (FTPS), we recommend that you use the command line curl tool. This can provide useful diagnostics and is available on Windows or Linux. 2/12

If you need a graphical transfer client for FTPS transfers, we suggest the FileZilla client. Please note that there are some special conditions when using this software with older vsftpd releases. curl curl is licensed under a MIT/X derivate license and is Open Source/Free Software. The curl recommended version at 23/03/2009 was 7.19.4 and this has been used in a range of tests below without any observed problems. You need at least version 7.16.0 for reliable FTP over TLS and preferably should be above 7.17.0 to avoid an important but rare bug. At 27/03/2009 the McKesson wide area testing is performed with version 7.18.2 with curl compiled from source code. Note that some pre-compiled RPMs for curl 7.18.2 on Linux exhibit some problems with FTPS (seen in Fedora 8 and 9) but these problems have not been seen when curl is compiled from source. Recent curl versions have support for sftp - however this is largely redundant on Linux and awkward to install on Windows as you need a ssh implementation as well as curl. Installing curl on Windows The curl version used was 7.19.4 and the OpenSSL version was 0.9.8j. You need: The Windows curl with SSL support The OpenSSL kit for Windows Microsoft Visual C++ 2008 Redistributables For the curl kit go to the curl download page at http://curl.haxx.se/download.html For the OpenSSL kit go to Shining Light Productions at http://www.shininglightpro.com/products/win32openssl.html (there is a link on the curl download page). You need the 'Light' edition and you need to choose depending on whether you have 32 or 64 bit Windows (most desktops/laptops are still 32 bit). The Shining Light page has a link to the Microsoft Visual C++ 2008 Redistributables. They recommend a particular version and again you have to choose depending on whether you have 32 or 64 bit. In the example installation on 26/03/2009 for curl 7.19.4 on a 32 bit machine these are the kits acquired: vcredist_x86.exe (Microsoft Visual C++ 2008 Redistributables) Win32OpenSSL_Light-0_9_8j.exe (the OpenSSL 'Light' edition) curl-7.19.4-win32-ssl.zip (curl with SSL support) The Microsoft Visual C++ 2008 Redistributables and the OpenSSL 'Light' edition are fairly normal Windows installations. The curl zip is unpacked and the curl.exe can be moved to an appropriate location or one in your path so that it can be invoked from the command line. Installing curl on Linux The FTP client tools curl and lftp that shipped with Red Hat Enterprise Linux 4 and 5 are too old for reliable testing. The related Linux community distribution Fedora has viable curl versions from Fedora 8 3/12

onwards. You can compile the curl command from source code but you need the GNU gcc compiler and related tools and libraries. Download the source code curl-17.19.4.tar.gz from http://curl.haxx.se/download.html. The sequence of commands on a suitable linux system is: tar xzf curl-7.19.4.tar.gz cd curl-7.19.4./configure make make test make install curl --version The make install has to be performed as 'root' and copies your built curl program to /usr/local/bin. Normally this will be earlier in the directory search path so the newer curl will be invoked rather than the older provided version. In the 'configure' output you expect to see SSL support as enabled (OpenSSL). The 'configure' output from CentOS 5/Red Hat Enterprise Linux 5 looks like: curl version: 7.19.4 Host setup: x86_64-unknown-linux-gnu Install prefix: /usr/local Compiler: gcc SSL support: enabled (OpenSSL) SSH support: no (--with-libssh2) zlib support: enabled krb4 support: no (--with-krb4*) GSSAPI support: no (--with-gssapi) SPNEGO support: no (--with-spnego) c-ares support: no (--enable-ares) ipv6 support: enabled IDN support: enabled Build libcurl: Shared=yes, Static=yes Built-in manual: enabled Verbose errors: enabled (--disable-verbose) SSPI support: no (--enable-sspi) ca cert bundle: /etc/pki/tls/certs/ca-bundle.crt ca cert path: no LDAP support: enabled (OpenLDAP) LDAPS support: no (--enable-ldaps) The 'configure' output from CentOS 4/RHEL 4 is slightly different in that the location of the certificate bundle is different. 4/12

curl version: 7.19.4 Host setup: x86_64-unknown-linux-gnu Install prefix: /usr/local Compiler: gcc SSL support: enabled (OpenSSL) SSH support: no (--with-libssh2) zlib support: enabled krb4 support: no (--with-krb4*) GSSAPI support: no (--with-gssapi) SPNEGO support: no (--with-spnego) c-ares support: no (--enable-ares) ipv6 support: enabled IDN support: enabled Build libcurl: Shared=yes, Static=yes Built-in manual: enabled Verbose errors: enabled (--disable-verbose) SSPI support: no (--enable-sspi) ca cert bundle: /usr/share/ssl/certs/ca-bundle.crt ca cert path: no LDAP support: enabled (OpenLDAP) LDAPS support: no (--enable-ldaps) Some tests in 'make test' are bound to fail as this is a general test of curl functionality - however you are interested that most succeed. Here is the final output from a CentOS 5 test. TESTDONE: 475 tests out of 475 reported OK: 100% TESTDONE: 524 tests were considered during 1268 seconds. TESTINFO: 49 tests were skipped due to these restraints: TESTINFO: "rlimit problem: fds needed 1050 > system limit 1024" 1 times (518) TESTINFO: "curl lacks scp support" 10 times (601, 603, 605, 607, 617, 619, 621, 623, 629, 631) TESTINFO: "Resolving IPv6 'ip6-localhost' didn't work" 2 times (241, 1083) TESTINFO: "openssl engine not supported" 1 times (307) TESTINFO: "curl lacks netrc_debug support" 6 times (130, 131, 132, 133, 134, 257 ) TESTINFO: "curl lacks sftp support" 29 times (600, 602, 604, 606, 608, 609, 610, 611, 612, 613, 614, 615, 616, 618, 620, 622, 624, 625, 626, 627, 628, 630, 632, 633, 634, 635, 636, 637, 2004) FileZilla This is a GPL licensed product with FTP over TLS/SSL and sftp support. 5/12

Testing Older vsftpd Installations FileZilla made the TLS protocol handling stricter in 2008 and this revealed a bug in the older vsftpd installations (such as found on Red Hat Enterprise Linux 4 and 5, CentOS 4 and 5 and Fedora 7). The nature of the bug is that newer FileZilla versions fail to get the directory listing after the FTP over TLS connection. This does not appear to stop uploads working but, as you cannot see the files, it does prevent downloads. The bug is fixed in vsftpd source for 2.0.7 and it appears the fix was ported back into the Fedora from version 8 onwards. No fix has been found for CentOS 4 and 5 and it is not known whether Red Hat have issued a fix for RHEL 4 and 5. Although the bug is in vsftpd, using an older FileZilla without the check on TLS protocol handling does allow you to proceed with testing using a graphical transfer client. The last version found without this check is V3.0.11.1. Installing FileZilla on Windows Download the latest stable version from http://filezilla-project.org/. The easiest installation is by using the *_win32-setup.exe kit. If testing older vsftpd installations, download the older FileZilla_3.0.11.1_win32-setup.exe. You can find old versions at http://sourceforge.net/projects/filezilla/ then take the Download, Browse all packages, FileZilla option. This gives you a list of all the downloads for previous versions. FileZilla has an uninstall option. Installing FileZilla on Linux Recent community Linux distributions like Fedora have pre-built versions of FileZilla; however the last version on, for example, Fedora 8 was FileZilla 3.1.0.1. If testing older vsftpd installations, this version is not useful as it has the protocol conformity check discussed earlier. Testing a FTPS Server using curl Basic Diagnostic Options Check the version. curl --version Get a directory listing, and require encryption plus verbose information, to host called myhost.mysite.com with username testdept. curl -kv --ftp-ssl-reqd -utestdept ftp://myhost.mysite.com Without verbose information. 6/12

curl -k --ftp-ssl-reqd -utestdept ftp://myhost.mysite.com With a trace file. curl -k --ftp-ssl-reqd -utestdept --trace traceout.txt ftp://myhost.mysite.com With a trace to your screen. curl -k --ftp-ssl-reqd -utestdept --trace - ftp://myhost.mysite.com Testing Guidelines Once you have basic connectivity. curl -k --ftp-ssl-reqd -utestdept ftp://myhost.mysite.com Use curl to download a file to your screen. curl -k --ftp-ssl-reqd -utestdept ftp://myhost.mysite.com/myfile.txt Use curl to download a file to a file. curl -k --ftp-ssl-reqd -utestdept -o mylocalfile.txt ftp://myhost.mysite.com/myfile.txt Use curl to upload a file to a file. curl -k --ftp-ssl-reqd -utestdept -T mylocalfile.txt ftp://myhost.mysite.com Finally this curl command is testing that a file called test.txt can be uploaded and renamed on the remote host, the file is uploaded to a temporary name then renamed after successful transfer. 7/12

curl -k --ftp-ssl-reqd -utestdept -T test.txt ftp://myhost.mysite.com/test.txt_txfr -Q '-RNFR test.txt_txfr' -Q '-RNTO test.txt' Testing a FTPS Server using the FileZilla Client FileZilla is a complex transfer client which can handle a number of different protocols. Its strongest feature is its ability to handle a transfer queue. If you are using this as your test tool, use it on a known working server first so you understand the different aspects of the software. Essentially there are 4 different areas on the screen - most of these can be removed via the View menu. Message Log Local Directory and Files Remote Directory and Files Transfer Queue For connection diagnostic work, the 'Message Log' screen is usually the most important. You can increase the amount of information shown via Settings, Debug option on the Edit menu. To create a new connection use Site Manager on the File menu. Connection Configuration: FTP over TLS This is a typical General panel for a FTP over TLS connection. and here is the Settings panel. 8/12

Testing Guidelines You should test: Directory changes (if your interface delivery or collection has a directory structure for different files) File uploads File downloads It is not easy to simulate the real delivery or collection behaviour with a graphical client, however, it is easy to do volume testing with FileZilla to assess the reliability of your server. Common FTP over TLS Problems Client Fails to Connect to Port 21 Your connection fails or hangs on the initial connect to the FTP control connection port 21. Either: Your FTP server is not running; A firewall is blocking connection; There is no routing to your FTP server. Client Cannot Connect in Encrypted Mode You can make a normal FTP connection in unencrypted mode but the encrypted mode fails. Either: Your FTP server has not been configured for encrypted mode; Your FTP server does not have the ability for encrypted mode; 9/12

Your FTP client set-up is wrong. Client Fails to Switch to Encrypted Mode The FTP server responds with a message similar to: 530 Non-anonymous sessions must use encryption. This means that the FTP server is set-up with mandatory encryption. Either: Your FTP client has not been set-up for encrypted connections; Your FTP client does not have the functionality for encrypted connections; You did not intend to configure the FTP Server for mandatory encryption. Client Fails to Open Data Channel The session hangs and/or timeouts after the initial connection. On observing the logging or tracing, you notice that the connection hangs at the point the FTP session attempts to open the data channel. This is an access mismatch between your FTP server and the your firewall configuration. The passive port range defined in your FTP server must be allowed by your firewall. Client Has Excessive Filesystem Access Once connected you can change directory to places you do not expect or want this user to reach. A well secured FTP server will stop remote users leaving their home or specified directory - this is called chrooting or jailing. It is part of the FTP server configuration. sftp Test Tool For testing sftp, we recommend the use of the sftp program on Linux systems. The sftp program is installed on virtually all modern Linux machines and many Unix machines by default. It is part of the OpenSSH v2 suite. Testing an OpenSSH sftp server from Linux To test a sftp server, we suggest a system running Linux. The OpenSSH suite should be installed by default including the client ssh/scp/sftp commands. The IP port 22 has to be open through firewalls to connect with the sftp server. Creating and Deploying a Public Key On the client system create a no passphrase key pair as below: $ ssh-keygen -t rsa Generating public/private rsa key pair. 10/12

Enter file in which to save the key (/home/testdept/.ssh/id_rsa): Created directory '/home/testdept/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/testdept/.ssh/id_rsa. Your public key has been saved in /home/testdept/.ssh/id_rsa.pub. The key fingerprint is: 2e:d4:91:be:a7:07:e0:4d:50:2d:e8:9e:5e:db:8d:0b testdept@centos5.mckesson.co.uk Transfer the public key file.ssh/id_rsa.pub to the destination machine by any means and place it in the.ssh/authorized_keys file. Make sure the protection on the authorized_keys and the.ssh is restricted to the user. Connection Test On the client use the sftp program to connect to the remote system - a successful connection would simply be seen as this: sftp testdept@centos4.mckesson.co.uk sftp> Check that you can: Transfer files to and from the remote system with put and get Remotely rename a file with rename Your access via sftp is restricted to certain areas of the file system That you cannot ssh to the remote system Common sftp Problems Client Fails to Connect to Port 22 Network connection is fairly straight-forward with sftp as long as the IP port 22 is open. If it is blocked by a firewall, initial connection will be impossible. Client's Public Key is Ignored If the public key is ignored by the OpenSSH server, the client will usually prompt for a password. This is called the password fallback method. This is no use for automated transfers so you have to find out why the key is being ignored. Finding the reason why a key is ignored can be difficult but try the following: Check the access to the authorized_keys file and its directory - it should not be open Run the openssh server with logging to determine the problem 11/12

Interactive Access Allowed The sftp connects to the remote system but so does a ssh command allowing an interactive shell session. The OpenSSH suite was conceived as a system management replacement for older facilities; so by default it allows ssh for interactive connection, scp as a replacement to rcp and sftp as transfer agent with similar facilities to an ftp client. Consult your Linux/Unix support to restrict the connection to sftp. Client has Excessive Filesystem Access Good FTP servers like vsftpd can be restricted to give remote FTP clients a very limited view of the remote system -this is called chrooting or jailing. Until 2008, sftp servers on Linux/Unix could not easily be restricted to be like FTP servers and, for older systems, you must either: Make sure filesystem security is good enough to prevent unwanted security intrusion; Obtain the very latest OpenSSH software. On the newer systems, sftp server can be run in a chrooting/jailing mode. References Title: Encrypted File Transfer - Customer Testing Version: 1.0 Author: David Wickens Company: McKesson (UK) Reviewer: Document Type: Knowledge Document Ref: Manager: ESR Technical Services Manager Classification: UNCLASSIFIED Review Date: Edited By: David Pugh & David Wickens Edit Date: 11/06/2009 CategoryKnowledge 12/12