Comparison of FTP and Signiant

Transcription

1 Comparison of FTP and Signiant An In depth Comparison of FTP with Methodologies from Signiant Ian Hamilton, CTO, Signiant

2 Abstract FTP (File Transfer Protocol) is used to perform file transfers over Internet Protocol (IP) networks such as the public Internet and corporate intranets. FTP is widely deployed and is widely accepted as a file transfer solution. However, are there downsides to using FTP for file transfers? This paper compares FTP with file transfer methodologies of Signiant s software applications and identifies the shortcomings of FTP that Signiant was designed to overcome. FTP FTP, as a term, is often used in multiple ways. As such, it is important to distinguish between FTP as a network protocol and FTP client and server implementations. File Transfer Protocol The FTP protocol is a set of formal rules that specify how to transmit files and file system related information between two computer systems. FTP is a client server protocol. The FTP client initiates a connection to an FTP server and can issue requests to STORE or RETRIEVE individual files. The base protocol also supports file system listing and manipulation commands and simple user authentication. The protocol provides mechanisms to specify data types and file structures with the assumption that two incompatible systems can provide conversion between data representations and storage formats. In practice, translation features are only for text conversion (e.g. EBCDIC 1 to ASCII 2 conversions). FTP Clients and Servers The FTP applications that most computer users are familiar with are interactive computer programs that implement the client portion of the FTP protocol. Users interact with the FTP client using a command line or graphical user interface. Many FTP clients support operations that are not part of the FTP protocol but rather are implemented by executing multiple FTP primitives in sequence for user convenience. FTP servers are usually deployed on back office server infrastructure; however, FTP servers can also run on desktop and notebook computers. An FTP server typically has no user interface and runs as a background service or daemon 3. Some FTP server implementations support graphical management and monitoring interfaces. An FTP client connects to the FTP server and usually provides, via the FTP protocol, a username and password for authentication. Most FTP servers support an anonymous mode of operation that allows clients to access a portion of the file system after a user supplies an address instead of a password. Anonymous FTP is not a function of the protocol or the client, but rather a local implementation feature of the server. 2

3 FTP SECURITY EXTENSIONS Security extensions for the FTP protocol were proposed in the late 1990 s to promote secure file exchange. These security extensions support strong authentication, data integrity, and confidentiality. However, the extensions only specify how the security messages are conveyed in the FTP control protocol and don t specify the actual security mechanisms used to implement authentication, message privacy, and integrity. As a result, FTP security extensions are not widely implemented and, when security extensions are implemented, systems are often incompatible. Many organizations that use FTP for secure data exchange use a third party tool to encrypt files before and decrypt the files after transfer. This process provides some confidentiality and integrity; however, securely transferring files then involves multiple manual steps or the use of homegrown scripts to automate the process. Further, if it is necessary to implement tasks such as key distribution after transfers have taken place, the security utilities then require a management infrastructure of their own, adding more complexity to the overall solution. Another factor to consider is that encrypting files does not address FTP authentication issues. A common approach to addressing the authentication void is to use a semi secure host in a demilitarized zone (DMZ 4 ) to relay messages. This further complicates the end to end data transfer process and does not solve the authentication problem but rather reduces the impact of security breaches. DMZ FTP relays are a poor solution at best. FTP AND FIREWALLS The FTP control protocol establishes a separate TCP 5 (Transport Control Protocol) connection for each file transferred. A new TCP port number, in the dynamic (or short lived) port range, is chosen for each new connection and firewalls must be application aware to manage FTP traffic. That is, the firewall must interpret the FTP protocol to determine which TCP ports are being used and dynamically alter the firewall rules. Low end firewalls and filtering routers cannot put adequate controls on FTP traffic. FTP PERFORMANCE The FTP protocol transfers each file over a new TCP connection. Not only does this introduce connection setup overhead for each file, but in high latency environments, additional round trip delays are introduced. When large numbers of files are transferred, the per file setup overhead can significantly reduce overall throughput. TCP PERFORMANCE Because FTP relies on the TCP protocol for basic data transfer, it is impacted by the throughput limitations of TCP in high latency environments. The TCP protocol provides reliability on top of IP by sending acknowledgements for data in the reverse direction to the data flow. When data is not acknowledged within a timeout window, it is retransmitted. TCP is a Sliding Window Protocol which means that only so much unacknowledged data can be sent before an acknowledgement is received 3

4 from the receiving end. When an acknowledgement is received, the window is advanced and more data can be sent. TCP attempts to scale the window size to match the time it takes for packets to make a round trip on the network, however, there is a limit to the TCP window size and the higher the latency and bandwidth of the network becomes, the worse TCP performs. As the bandwidth and latency of the network grow, the TCP window is exhausted, resulting in a stop / start behavior. Typically, if the bandwidth latency product (bandwidth in bits/second multiplied by latency in seconds) of the network is greater than 256 Kbits, throughput is impacted. TCP is a stream oriented protocol that only acknowledges how many bytes into the stream the target has received. The TCP retransmission scheme always retransmits the stream in full from the last acknowledged byte when data is lost. As such, loss of a single packet can result in resending the entire send window, including data after the lost packet that may have been received by the target. The TCP flow control mechanism determines the rate that data is placed on the network. The TCP flow control mechanism attempts to measure the capacity of the network by looking for dropped packets on the network. The assumption being that, when a packet is lost, the capacity of the network has been exceeded. The TCP flow control mechanism uses an additive increase and multiplicative decrease scheme, where the rate is increased linearly until packet loss occurs at which point the send rate is cut by a factor of the overall send rate. This result is greater fairness between multiple TCP streams competing for resources on the network, but also impacts overall throughput rather dramatically. Further, when there is inherent loss on the network (e.g. the public Internet), throughput can be extremely poor. SIGNIANT SOFTWARE and DESIGN APPROACH While FTP was designed for simple interactive file exchange, Signiant software was designed to address all of the requirements of inter and intra enterprise, automated and interactive managed file movement. File transfer is one of the basic capabilities of Signiant software; however, the Signiant solution encompasses much additional functionality. To build comparable solutions with FTP would require extensive in house development including third party software integration to address the following Signiant capabilities: Central management including: o Fault Management o Configuration Management o Accounting, and o Performance Management Security including: o Authentication o Authorization o Data Integrity o Data Confidentiality o Non Repudiation (Certified Delivery) 4

5 Application integration and workflow automation Enhanced file transfer features including: o File metadata management o Incremental Transfer o Mirroring o Versioning o Multipoint transfer o Advance file selection criteria o Atomic transfer and rollback Notification Firewall issues, and Reliability and redundancy Even when in house development is undertaken, the resulting solutions are typically: Inflexible, Difficult to scale, and Expensive to maintain. Further, these solutions still have all of the inherent performance limitations of the FTP protocol and a TCP based transport. MANAGED PEER TO PEER FLEXIBILITY AND SCALABILITY The Signiant solution was fundamentally designed to be easy to manage in large networks and to run unattended without human intervention. To accomplish these objectives, a managed peer to peer model was implemented instead of a client server model. Rather than installing client and server binaries on each host computer system, Signiant installs a single agent that incorporates both server and client functionality. A central manager administers the distribution of business rules, including file transfer control and application integration information, to agents that, in turn, establish direct connection to other agents to exchange data. Users of the system can create new rules (in the form of job templates) that specify how data is moved and various application interaction points. Users can also use predefined job templates provided within the system for simple tasks such as hot folder transfer and person to person transfers. Jobs are created by binding a job template to input parameters (e.g. the source and target agent). Jobs can be run on a scheduled basis or in response to an external event. All job templates and jobs can be viewed, managed, and tracked from the multi user web based management console. 5

6 Signiant also supports interactive, accelerated, secure file transfers to web based users, with no requirement for pre installed software. A web plug in that implements the client portion of the Signiant protocol can be dynamically downloaded and used as part of any web experience. The advantage of this model is that it is much easier to scale and maintain than a traditional clientserver model. To automate a data transfer process using FTP, rules must be embedded in individual automation scripts that reside within each FTP client or server. INTEGRATED SECURITY Security is built into Signiant technology. Each agent installation automatically generates, at install time, the public key security credentials necessary to mutually authenticate with other agents and guarantee the privacy and integrity of data. Signiant uses standards based public and private key cryptographic techniques. Signiant operates a Certificate Authority (CA) and an on line Registration Authority (RA) that interacts with the agent installer to process the agent s Certificate Signing Request (CSR) and creates the agent certificate. Signiant manages Certificate Revocation Lists (CRLs) and automatic certificate renewal. Signiant can also optionally work with third party certificate authorities. In contrast, third party security products, with their own associated support infrastructure, are required to provide message security for FTP based transfers. Further, FTP authentication mechanisms are weak and managing user IDs and passwords used by FTP scripts is labor intensive and insecure. FIREWALL FRIENDLY Signiant uses a single port for all control and data transfer traffic. This approach simplifies firewall identification and classification of Signiant traffic. The relay feature of Signiant s technology can be enabled to minimize firewall rule administration. By configuring an agent as a relay inside and outside your firewall, you can establish secure connections to any Signiant agent inside or outside your corporate network with a single address port pair firewall rule. The relay capability also allows you to build transfer rules that take advantage of redundant or parallel Internet connections. Tunneling can also be used to establish transport layer connections in the opposite direction to data transfer. Signiant can even use the FTP protocol to transfer files to locations that have an FTP server and no Signiant Agent. PEFORMANCE The Signiant protocol has been optimized at both the application and transport layer to minimize the impact of network latency. Using techniques like selective acknowledgement, dynamic windows scaling, differential loss monitoring, intelligent rate management and other proprietary patent pending techniques, Signiant maximizes the use of the available bandwidth. Signiant eliminates round trip delays in the application level protocol by pipelining application layer acknowledgements. 6

7 Signiant is also optimized to work with Network Attached Storage (NAS) and minimizes Common Internet File System (CIFS) or Network File System (NFS) operations when writing files to NAS. Signiant can perform hundreds of time faster than FTP transfers over high latency networks when transferring both large numbers of small files and small numbers of large files. BANDWIDTH MANAGEMENT Signiant provides advanced bandwidth management capabilities that assist in ensuring that network resources are allocated to the highest priority business activities first before lower priority business activities are addressed. REPORTING AND LOGGING Signiant logs all data transfer activity and provides a simple browser based user interface for building reports for performance management, compliance and billing. Some FTP implementations provide limited server side logging of file transfer activity but central collection and reporting capabilities must be custom coded. CONCLUSION While FTP is a commonly used tool for simple interactive file transfer within an enterprise, it lacks essential management, security, acceleration, and process automation capabilities necessary to support secure automated system to system data transfer. Signiant was designed from its inception to handle secure, automated, and reliable system to system data transfer. The peer to peer communications model, with centralized management of data transfer business rules, is ideal for building cost effective, highly scalable, and easy to deploy solutions that address data movement requirements of all sizes and complexities. About the Author Ian Hamilton, Chief Technology Officer, Signiant Ian Hamilton is Signiant's CTO and Vice President of Development. Ian has been an innovator and entrepreneur in Internet and system software for over 20 years. About Signiant Founded in 2000, Signiant ( is the digital media supply chain company that telco, media and entertainment executives rely on to manage the movement of media to the right place at the right time. With its open software platform that unites best in class management, acceleration, automation and security technologies, Signiant ensures that some of the world s most recognized brands are able to deliver their digital assets and drive new revenue models. Signiant is headquartered in Burlington, MA with development facilities in Ottawa, Ontario, Canada and offices in New York, Los Angeles and Manchester, UK. Read the Signiant blog at 7

8 Footnotes: 1 EBCDIC Extended Binary Coded Decimal Interchange Code (EBCDIC) is an 8 bit character encoding (code page) used on IBM mainframe operating systems such as z/os, OS/390, VM and VSE, as well as IBM minicomputer operating systems such as OS/400 and i5/os. It is also employed on various non IBM platforms such as Fujitsu Siemens' BS2000/OSD, HP MPE/iX, and Unisys MCP. 2 ASCII is the acronym for the American Standard Code for Information Interchange. ASCII is a code for representing English characters as numbers, with each letter assigned a number from 0 to 127. For example, the ASCII code for uppercase M is 77. Most computers use ASCII codes to represent text, which makes it possible to transfer data from one computer to another. 3 Daemon In Unix and other computer multitasking operating systems, a daemon is a computer program that runs in the background, rather than under the direct control of a user; they are usually initiated as processes. 4 A DMZ is a network between the corporate intranet and the Internet that operates with a level of security between that of the corporate network and the Internet. 5 TCP is the connection oriented protocol built on top of Internet Protocol (IP). TCP adds reliable communication and flow control and provides full duplex, process to process connections. Each TCP stream is identified by a source and destination IP address/port pair. The ports identify the sending and receiving processes on the hosts identified by the sending and receiving addresses. Firewalls filter IP traffic using, among other things, port information. 8