FTP : File Transfer Protocol and Z/OS

FTP : File Transfer Protocol and Z/OS Laura Jeanne Knapp Technical Evangelist 1-919-224-2205 Laura@lauraknapp.com www.lauraknapp.com z/os technical content provided by Alfred Christensen MoreIP_ 010

TCP/IP Layered Architecture Browser HTTP Server Application TCP/UDP/RSVP WWW, mail, file transfer, remote access Application interfaces End-to-end delivery Application TCP/UDP/RSVP IP Internet Protocol Best effort delivery IP Internet Protocol Network Interface and hardware Physical connection Network Interface and hardware MoreIP_ 020

Internet Capabilities (Basics) Terminal Emulation (TELNET) World Wide Web (WWW) File Transfer Protocol (FTP) News Discussions (NNTP) Internet Network Management Protocol (SNMP) Internet Mail Simple Mail Transfer Protocol (SMTP) MoreIP_ 030

FTP - File Transfer Program FTP Control Internet FTP Data Users System Server Copy Single files, multiple files, append files File management List files, identify directory, change directory, create, rename, and remove Control Identify ASCII/EBCDIC/binary text, bytes or record sequences MoreIP_ 040

TCP/IP Socket Overview FTP Control Internet FTP Data Client CONNECT Server BIND LISTEN SELECT ACCEPT Allows the client to open a connection to a server's Port Associates a socket with a Port number Tells TCP/IP that this process is listening for connections Waits for activity on a Socket Accepts a connection from a client Client and Server SOCKET Allocates a Socket to read/write from SEND Sends data to the process on the other host RECV Receives data from the other host CLOSE Terminates a connection, deallocating the Socket MoreIP_ 050

TCP/IP Socket Flow Simple Example of Application socket calls Client/Server Client Application TCP/IP TCP/IP Server Application Socket function Connection state Network Flows Connection state Socket function Socket Socket Connect (IP Address x, Port 3000) SynSent Established SYN SYN/ACK ACK Listen Bind (Port 3000) inaddrany Listen incoming connection Accept Write Rc=100 Read Rc=10 for read 100 bytes 100 bytes ACK +100 10 bytes 10 bytes ACK + 10 100 bytes of data 10 bytes Read Rc=100 from read call Write Rc=10 Close FIN_wait1 FIN_Wait2 Time_Wait FIN ACK FIN ACK Close_wait Last_Ack Read RC=0 for Read (remote side closed) Close MoreIP_ 060

Z/OS FTP FTP is one of the most widely used TCP/IP applications on z/os Both an FTP client and server are included as part of the base z/os CS functions The FTP functions on z/os support both traditional MVS data sets and files in the hierarchical file system Conversion occurs between ASCII and EBCDIC when copying text files or data sets in or out of z/os FTP was developed for a stream oriented file system and configuration options abound in z/os to handle the aspects of how to map MVS data set structures FTP File Transfer MoreIP_ 070

FTP - Port Numbers and Data Connections FTP Client Client IP address 10.1.1.1 FTP Server Server IP address 10.2.2.2 ACTIVE mode FTP aka. standard mode FTP 50001 Control connection setup to server port 21 PORT/EPRT command - 10.1.1.1, 50002 200 Port request OK Data connection setup to client port 50002 21 50002 20 Active socket is Bound to 10.2.2.2, port 20 PASSIVE mode FTP aka. Firewall Friendly FTP 50001 50002 Control connection setup to server port 21 PASV/EPSV command 227 Entering Passive Mode (10.2.2.2, 1025) Data connection setup to server port 1025 21 1025 Passive (listening) socket is bound to 10.2.2.2, port 1025 MoreIP_ 090

FTP - Transfer Attributes Every FTP transfer operation is characterized by three attributes: Data Type - ASCII / EBCDIC / IMAGE - how should the transmitted octets be interpreted by the receiver data<lf> data<lf> Structure - File / Record - structure of data file on-the-wire Transfer Mode - Stream / Block / Compressed - which service should the receiver perform on the received data before storing it FTP Client 1. Strip off local line terminators 2. Add CRLF as defined by the FTP protocol Type=ASCII, Structure=File, Mode=Stream data<crlf> data<crlf> FTP Server 1. Strip off lthe line terminators that were added by the FTP protocol 2. Add local line terminators Why you can't just do a binary transfer of a text file between a Windows workstation and an ASCII UNIX host! data<lf> data<lf> or MVS data set: data data MoreIP_ 100

FTP - Server Functional Overview Workstation: 1. FTP client command line interface 2. FTP client GUI interface 3. Web browser UNIX-based clients FTP. DATA Security Database SAF z/os FTP Server SBCS DBCS MBCS Unicode SMF Records SyslogD DB2 MVS Data Sets HFS Files SQL Queries z/os TSO or UNIX Shell-based clients Customizable ASCII/EBCDIC translation tables and standard code set names supported by iconv on OS/390. JESx JESx spool Submit Batch Jobs Retrieve JES Sysout XMIT data to NJE destination Batch job Copying data sets and files NJE Node MoreIP_ 110

FTP Client Functional Overview FTP. DATA User-level or system-level FTP client configuration options MVS Data sets HFS Files Batch job Interactive TSO user Interactive Shell user Imbedded in REXX program Workstation FTP Servers UNIX-based FTP Servers z/os FTP Server MoreIP_ 120

FTP Graphical Connection FTP Voyager from RhinoSOft MoreIP_ 130

FTP Command Line Connection Has minimal information MoreIP_ 140

FTP Enhanced Command Line Connection C:\WINDOWS>ftp mvs098.tcp.raleigh.ibm.com Connected to mvs098.tcp.raleigh.ibm.com. 220-FTPABC1 FTP CS V1R2 at MVS098, 15:31:39 on 2002-02-02. 220-* 220-* Welcome to the FTP server on MVS098 220-* This system is used by Alfred for testing purposes. 220-* Any issues should be reported to alfredch@us.ibm.com 220-* Your host name is sig-9-15-22-215.mts.ibm.com 220-* 220 Connection will not timeout. User (mvs098.tcp.raleigh.ibm.com:(none)): user1 331 Send password please. Password: 230-* 230-* USER1 - welcome to the FTP server on MVS098 230-* Login time and date is Sat Feb 2 15:31:47 2002 230-* The current working directory is /u/user1 230-* 230 USER1 is logged on. Working directory is "/u/user1". ftp> Welcome banner is sent as 220 reply messages right after the connection Login message is sent as 230 reply message after successful login Directory info is sent as 250 reply message after first CD directory ftp> cd 'user1.alfred.cntl' 250-* 250-* This is Alfred's JCL library 250-* USER1.ALFRED.CNTL 250-* 250 "USER1.ALFRED.CNTL" partitioned data set is working directory ftp> cd /u/ftp/pub 250-********************************************************************** 250-* 250-* This is the public directory on MVS098.tcp.raleigh.ibm.com 250-* All files in this directory and in its subdirecyories can 250-* be downloaded to your workstation. 250-* 250-********************************************************************** 250 HFS directory /u/ftp/pub is the current working directory ftp> MoreIP_ 150

FTP Server Exits z/os FTP Server FTCHKIP FTCHKPWD Accept/reject connection from certain clients Accept/reject logins from certain user IDs FTCHKCMD Accept/reject individual FTP commands FTCHKJES Accept/reject/modify JCL FTPSMFEX Accept/reject writing of SMF record FTPOSTPR File transfer postprocessing exit FTP server exit If these exits routines are present they will be loaded and called at the defined exit points The FTCHKIP exit is called by the FTP daemon, while the others are called by the FTP server (after the new address space has been created) The command check routine is the most widely used. It has information about the current command from the client, what the current working directory is, what filetype we are using, etc. It may reject the command or it may modify the command options, such as the file or data set name on a STOR or RETR command. There are a number of samples in the the TCPIP install library (tcpip.sezainst) The exits are normally coded in assembler, but we have seen examples where they were coded in C. In the next release (fall 2002), there will be changes to the exit interfaces to support IPv6 addresses and to implement a communication area that can be used by the exits to pass information between them (does not include the FTCHKIP and FTCHKPWD exits) MoreIP_ 160

Anonymous FTP Server Support on z/os Anonymous user's view / FTP.DATA RACF / /pub /bin /incoming Jane Doe When anonymous logs in, server does a chroot() to the anonymous user's home directory. /u /etc /u/guest /u/guest/pub /u/guest/incoming /u/guest/bin /sbin CWD / -> positioned at /u/guest The anonymous user cannot see any parts of the HFS above or outside the subtree that is defined as the anonymous user's home directory in RACF. There are FTP server options to control the anonymous user's access to MVS data sets or HFS files, use of file type SEQ, JES, and SQL, default permission bits for files and directories created by the anonymous user, etc. Certain SITE commands are disabled - such as a CHMOD and UMASK command. MoreIP_ 170

Anonymous User s Home Directory ls bin sh ftpdns /u/ftp usr pub incoming extract sbin Place your public files here - make sure permissions do not allow anonymous user to update/delete Write access allowed to this directory - enabling anonymous users to store files on the server Anonymous user can get from, but not list this directory. This directory structure with correct permission bits can be built using a sample FTPANDIR shell script. drwx--x--x 7 USER 0 8192 Sep 21 17:23 u/ftp drwx--x--x 2 USER 0 8192 Sep 21 17:23 bin -rwx--x--x 1 USER 0 126976 Sep 21 17:23 ls -rwx--x--t 1 USER 0 0 Sep 21 17:23 sh drwx--x--x 2 USER 0 8192 Sep 21 17:23 extract drwx-wx-wx 2 USER 0 8192 Sep 21 17:23 incoming drwxr-xr-x 2 USER 0 8192 Sep 21 17:23 pub drwx--x--x 3 USER 0 8192 Sep 21 17:23 usr drwx--x--x 2 USER 0 8192 Sep 21 17:23 sbin -rwx--x--t 1 USER 0 0 Sep 21 17:23 ftpdns MoreIP_ 180

Browser Access to z/os FTP Server Web browsers use anonymous by default. You can encode the FTP URL as follows to change that: ftp://userid:password@hostname:portnumber/ MoreIP_ 190

Accessing MVS Data Sets from a Browser /MVSDS/ to instruct the FTP server that this URL refers to an MVS data set and not an HFS file ;type=a to force an ASCII transfer to have data converted from EBCDIC to ASCII by the FTP server MoreIP_ 200

Enhanced ASCII Support by FTP local_ascii_file type=a site sbdataconn=(ibm-1047, iso8859-1) put local_ascii_file zos_ebcdic_file zos_ebcdic_file The SBDATACONN SITE command is used to instruct the FTP server that data is sent by the remote client in ASCII encoding ISO8859-1 and that the server is requested to convert the received data to EBCDIC codepage -1047 before storing the file on z/os. USER1:/u/user1: >ls -T zos* t -1047 T=on zos_ebcdic_file For text-type transfer to z/os, the FTP server always knows what the z/os codepage is. With the support added by Enhanced ASCII, FTP can now preserve that information by storing it as a file tag for HFS files. The file tag can be used by anyone accessing the file - incl. FTP if the file is to be transferred back to an ASCII platform later. The file tag can also be used by the Enhanced ASCII auto conversion function to automatically convert HFS file encoding to whatever ASCII codepage a program is executing in (currently only ISO8859-1). Warning : Never set global AUTOCVT ON in the BPXPRMxx PARMLIB member - results may be unpredictable!!! MoreIP_ 230

Socksified FTP Client in z/os SOCKS is a very common relay server on firewalls. SOCKS allow TCP connections to be relayed on a firewall in such a way that intranet IP addresses are not revealed to the public network - it is an application-layer gateway type of firewall technology. Clients that connect through a SOCKS server must have specific support for doing so (to be socksified). Servers that are being used through a SOCKS server do not need any modifications. The SOCKS configuration file instructs the FTP client which destinations (IP address ranges) can be reached directly and which require use of a SOCKS server. Intranet destinations directly; everything else via a SOCKS server at a given intranet IP address. SOCKS configuration file z/os FTP client Private network (intranet) Firewall SOCKS server Public network (Internet) FTP server No longer a need to stage data on a workstation if you need to transfer data sets or files directly to/from z/os to/from an Internet destination. One example is when transferring data to/from the support site. MoreIP_ 240

FTP Secure Transfers in z/os SSL/TLS support has been added to both the z/os FTP client and the z/os FTP server. The server can be started in one of two modes: SSL/TLS required - only secure connections will be accepted SSL/TLS allowed - secure connections will be established if the client requests it Server options specify whether a client certificate is required or optional If a connection is secure, the control connection will always be encrypted. Server options are used to control whether the data connections may or is required to be secure also. Most widely used workstation SSL/TLS enabled FTP software is WS-FTP Pro. z/os FTP client Business-to-Business File Transfer Firewall Firewall Firewall Company B intranet Company A intranet Internet Secure FTP connection Company B Security Zone Client Certificate Firewalls may be application gateway firewalls (ie. SOCKS), or plain filtering firewalls. Since the FTP control connection is encrypted, boxes in the middle cannot investigate/change control connection data. SSL/TLS capable FTP Server Staging File Server Server Certificate MoreIP_ 250

FTP Tuning Information FTP Control Internet FTP Data Users System Server MVS CPU decreases as packet size (MTU) increases MVS Throughput increases and MVS CPU decreases as workstation TCP/IP window size increases (recommend window size = 64 KB at workstation) MVS Throughput increases and MVS CPU decreases as MVS TCP/IP window size increases (Defualt window size is 64KB) MoreIP_ 280

FTP Tuning Information FTP Control Internet FTP Data Users System Server For CLAW devices (CISCO CIP) set read/write buffers on CLAW device statement (TCP/IP profile) to 50 (default =15) MVS throughput increases as MVS dataset blocksize increases (recommended DS blocksize - 1/2 DASD track) Kepp CHKPTINT parm set to 0 (tcpip.ftp.data) MoreIP_ 290

FTP Capacity Planning MVS CPU Requirements: Max KB CPU secs CPU secs ------------- * --------------- = --------------- Elap secs KB Elap secs Example: (49.8 MB/S, WS--> MVS, Bin Put, CS/390 R12, OSAE-GBE) 50995.2 KB.0000127 N1.648 CPU secs ------------- * ----------------- = --------------- Elap secs KB Elap secs N1: MVS TCP/IP + VTAM + FTP Addr Spaces (2064-108 4 CP LPAR) If the CPU secs/elap sec ratio is greater than 1, one would need more than one processor (CS/390 R4 - R12). MoreIP_ 320

MVS CPU Utilization: FTP Capacity Planning CPU secs/elap Sec ------------------------------ * 100 % = CPU Util % #ofprocessors # of processors: Should be equal to the number of number of processors (CS/390 R4 - R12). Example: (49.8 MB/S, WS--> MVS, Bin Put, CS/390 R12, OSAE-GBE) 0.648 CPU secs/elap sec ------------------------------------- * 100 % = 16.2 % 4processor Thus, MVS TCP/IP's + VTAM's + FTP addr spaces CPU requirement for FTP Binary PUT would require 16.2 % of a four processor 2064-108 LPAR system. LSPR can be used to adjust for other processors types. MoreIP_ 330

Application Advances Real-time audio/video Internet Talk Radio USENET Animation Voice over IP Commerce IRC (Internet Relay Chat) Listserve JAVA/ActiveX MUD (Multiple User Dimension/Dialogue) LDAP/DEN/Active Directory VRML (Virtual Reality Markup Language) 340 MoreIP_

TCP/IP Protocol Suite Most Real time NFS comm apps Telnet FTP SMTP HTTP POP DNS apps RTP/RTCP DNS RPC TCP UDP SNMP RSVP IP ICMP ARP RARP Token-Ring, Ethernet, FDDI, Frame Relay, Dial, Leased Line, ATM, ISDN, SMDS, Sonet, X.25, Fibre Channel, PPP, SLIP IP - Internet Protocol ICMP - Internet Control Message Protocol ARP - Address Resolution Protocol RARP - Reverse Address Resolution Protocol TCP - Transmission Control Program UDP - User Datagram Protocol POP - Post Office Protocol DNS - Domain Name System Telnet - Teletype Network FTP - File Transfer Protocol SMTP - Simple Mail Transfer Protocol HTTP - Hypertext Transport Protocol NFS - Network File System RPC - Remote Procedure Call SNMP - Simple Network Management Protocol MoreIP_ 350

Resources URL http://www.ibm.com/software/network http://www.ibm.com/software/network/commserver http://www.ibm.com/software/network/commserver/support http://www.ibm.com/software/network/commserver/library http://s390.ibm.com/networking http://www.ibm.com/software/network/hostondemand http://www.ibm.com/software/network/pcomm http://www.ibm.com/software/network/technology http://www.ibm.com/redbooks http://www.ibm.com/support/techdocs/ http://www.rfc-editor.org/rfcsearch.html Content Network and Communications software Communications Server Communications Server Support CS White Papers, Product Doc, etc. S/390 Networking Host On-Demand Personal Communications Networking Technologies ITSO Redbooks Advanced Technical Support (Flashes, Presentations, White Papers, etc.) Request for Comments (RFCs) MoreIP_ 360

FTP Commands MoreIP_ 370

FTP Replies 200 Command okay. 500 Syntax error, command unrecognized. This may include errors such as command line too long. 501 Syntax error in parameters or arguments. 202 Command not implemented, superfluous at this site. 502 Command not implemented. 503 Bad sequence of commands. 504 Command not implemented for that parameter. 110 Restart marker reply. 211 System status, or system help reply. 212 Directory status. 213 File status. 214 Help message. 215 NAME system type. 150 File status okay; about to open data connection. 250 Requested file action okay, completed. 257 "PATHNAME" created. 350 Requested file action pending further information. 450 Requested file action not taken. File unavailable (e.g., file busy). 550 Requested action not taken. File unavailable (e.g., file not found, no access). 451 Requested action aborted. Local error in processing. 551 Requested action aborted. Page type unknown. 452 Requested action not taken. Insufficient storage space in system. 552 Requested file action aborted. Exceeded storage allocation (for current directory or data set). 553 Requested action not taken. File name not allowed. 120 Service ready in nnn minutes. 220 Service ready for new user. 221 Service closing control connection. Logged out if appropriate. 421 Service not available, closing control connection. 125 Data connection already open; transfer starting. 225 Data connection open; no transfer in progress. 425 Can't open data connection. 226 Closing data connection. Requested file action successful (for example, file transfer or file abort). 426 Connection closed; transfer aborted. 227 Entering Passive Mode (h1,h2,h3,h4,p1,p2). 230 User logged in, proceed. 530 Not logged in. 331 User name okay, need password. 332 Need account for login. 532 Need account for storing files. MoreIP_ 380