From the Ridiculous to the Sublime: Getting Files from There to Here

Transcription

1 From the Ridiculous to the Sublime: Getting Files from There to Here Robert H. Upson - Virginia Community College System Gail M. Barnes - Southside Virginia Communhy College Tamara R. Fischell - Timely Technical Solutions Introduction: Determining the procedures that successfully transport SAS datasets across different hardware platforms assumes critical importance as sites move further from single-vendor, single protocol systems. And, although SAS Institute's Multi-Vendor-Architecture, as impressive as it is, attempts to address many of these problems, it does not eliminate the difficulties you will encounter in transporting remote-stored data across different environments. There are a number of good reasons to become familiar with SAS transport files, and to learn how to get them from one place to another. Data and applications can be created and can reside at the most appropriate location, as long as you know how to transport them when they are needed. The large resource investment needed to rekey, reproof, and recode data that are already available is eliminated if you can move the data at will. And finally, in this multi-protocol world, SAS transport files are just ONE example of the utility of being able to move proprietary format files as needed. Other analogous applications include text (Le., PostScript, Word-Perfect) and executable (Le., public domain programs) that you may transport without (much) trepidation. This paper briefly describes SASjCONNECT, offers several alternative methods if SAS jconnect is not available, and suggests how you might select the method most likely to guarantee data integrity. But First, Some Definitions: In order to better understand the process, and select the best method of file transfer, it is necessary to understand the relationship between the hardware and software used in the transfer process. There are SOURCE and REMOTE machines, and there are TARGET and LOCAL machines, which mayor may not be the same physical machines. ** The SOURCE machine creates the file to be transported. In order to create a SAS transport file, the SOURCE machine must be a SAS machine. You will need to know the SAS release used to create the file, the type of operating system, and how the file was created, either by PROC COPY or PROC CPORT. ** The transport file will be imported by a TARGET machine. In order to use a transported file the TARGET machine must be a SAS machine. You will need to know the SAS release and the operating system installed on the target machine. ** SAS FILES refers to datasets and catalogs which are in the SAS proprietary format. That format is specific to a release and platform. 419

2 SAS TRANSPORT FILES refer to a format which will allow SAS data and catalogs to be moved electronically from machine to machine, and reconstructed by SAS software, on the same or a different platform. * In addttion, there are two kinds of activities involved. The first, EXPORTING and IMPORTING are the procedures which code and decode the SAS file for transmission, and are performed by the SAS software. The second actmty uses FILE TRANSFER software to move the transport file from machine to machine. Because both the SOURCE and TARGET machines are running under SAS, this portion of the file transfer is relatively straight forward. However, once the transport file leaves the control of SAS software; * The term REMOTE machine is used to refer to the current location of the file. * The term LOCAL machine is used to refer to the file's next destination. You will need to know the operating system and file transfer software that will be used on the REMOTE and on the LOCAL machines, to control the file transfer. The Easy Way! SAS/CONNECT is a cooperative processing product that enables a local SAS session to establish conversations with one or more remote SAS sessions. The software gives you access to the files, hardware resources, and SAS software on various remote hosts to use with a SAS session on the local host.. It eliminates the need to specify most operating system dependent codes or procedures, and has a built in compression and decompression function which is advantageous for large files with missing data. CONNECT came bundled with Base SAS in Release However, there were a number of restrictions on hardware, software, and protocols. These restrictions have been largely eliminated in newer releases, and compatibility issues virtually disappear in Release 6.07, and higher. Where the appropriate release is available on both the SOURCE and TARGET machines, CONNECT can transport both data and catalog type datasets, as well as external files. Older releases can transport data type datasets and external files. Once CONNECT is running on both machines, PROC's UPLOAD and DOWNLOAD use the following syntax to move SAS datasets or external files. PROC UPLOAD/DOWNLOAD DATA = Lib.Mem INCAT = Lib.Cat INFILE = Fileref ;RUN; OUT = Lib.Mem OUTCAT=Lib.Cat OUTFILE = Fileref Both procedures are executed on the REMOTE SAS machine: (SAS data) (SAS catalog) (external file) Use RSUBMIT for both! 420

3 So, You DonI Have CONNECT? - File Transfer of Transport Format Files: The first decision you must make in considering the transport of SAS datasets is, "Will I be able to use the transported data?" This question is not as silly as it sounds, because catalog datasets created under different SAS releases may not be compatible. Catalog Datasets, Compatibility and Different Releases: Compatibility is a major concern in transporting catalog type datasets, since catalog datasets are not downwards compatible. If a catalog dataset was created under SAS Version 6.0B, for example, then you should not attempt to import that catalog to SAS Under this scenario, you should consider transporting the SAS source code for execution on the your machine. Three Rules for Catalog Datasets: 1. Do not attempt to transport catalogs to a lower release. 2. When you ignore Rule #1, don't think you will find all the bugs. 3. When you ignore Rule #2, your users will find all the bugs. One at a time. After determining the compatibility of the data set to the SAS release on your machine, two specific activities are necessary to transport files. The first involves the SAS IMPORT jexport functions which construct and deconstruct the file. The second involves moving the file across platforms between the SOURCE and TARGET machine. You te Going To Try It? - Export and Import: In order to transport files created on the SOURCE machine, from SAS library members, it is necessary to first convert the dataset, (eithei catalog or data, or both) into one file in SAS Transport format. When the file arrives at the TARGET machine, the process is reversed. Because SAS libraries can contain data type datasets, catalog type datasets, or a combination of both, the procedure which you employ to export and import transport libraries will depend on the library contents. The available methods are: PROC CPORT and PROC CIMPORT - Exporting and importing catalog type datasets. Note: Data type datasets CAN be exported and imported using these two proced ures. PROC COPY - Exporting and importing data type datasets only. The appropriate transport engine (the means of converting SAS format to transport format and back) must be used. 421

4 Creating an Export File: PROCCPORT CPORT CAT = Lib.Cat File=(File Name) Options CPORT DATA = Lib.Mem File=(File Name) CPORT LIBRARY = Lib File=(File Name) {Select or Exclude = } {MEMTYPE = } (other options); RUN; Creating SAS Datasets From an Export File: CIMPORT CAT = Lib.Cat CIMPORT DATA = Lib.Mem CIMPORT LIBRARY = Lib {Select or Exclude = }; RUN; InFile= (File Name) InFile=(File Name) InFile= (File Name) PROC CIMPORT Options Exporting and Importing Type DATA Only: L1BNAME ddnamel (export engine) 'Transport Library Name'; L1BNAME ddname2 'SAS Library Name'; To create a transport file from SAS files: PROC COpy INDD=ddname2 OUTDD=ddnamel (MT = ); {Select or Exclude = }; RUN; TO create SAS files from a transport file: PROC COPY INDD=ddnamel OUTDD=ddname2 (MT = ); {Select or Exclude = };RUN; Moving the File Between Platforms: Once you have established that the SOURCE machine has created a transport file that can be used on your TARGET machine, the next step will be electronic file transfer. This electronic file transfer will involve some number of intermediate machines. A number of issues must be addressed before successful transport can be completed. For example, libraries must be transported in binary mode, and intermediate operating systems may need to be given specific instructions. The intermediate platforms and processes frequently use defaults that do not preserve the SAS transport file format. There are several ways to preserve the correct file attributes. 422

5 Transferring files requires the programmer to know about transfer protocols and some of the nuances of spechic operating systems. At each step along the transport process, the file should look to each machine through which it passes, as though it could be run on that machine. One can specify attributes within the file transfer software, but the results are sometimes unpredictable. You will find it more predictable to create a transport file "space" w~h the correct attributes and replace it with the transport file. For example, SAS, running under IBM mainframe operating systems (MVS, VM) require transport files to have the following attributes: 1. A format of fixed block; 2. A logical record length of 80 bytes; 3. Block size of 8,000. As an example, under MVS, the following TSO statement can be used: ALlOC FI(trans) DA(file.name) VOL(identfier) NEW DSORG(PS) RECFM(F,B) LRECL(80) BLKSIZE(8000) SPACE(primary,secondary) unit Under VM the following CMS statements can be used: FILEDEF trans DISK file name A (RECFM FB LRECL 80 BLKSIZE 8000 DSORG PS FILEDEF temp DISK Profile Exec A MOVEFILE temp trans Methods of Electronic File Transfer: File Transfer Protocol (FTP), is one of the fastest, most efficient ways to move transport files across different hardware platforms. On-line help is available, and the syntax is virtually the same, regardless of platform. Here is a sample of the FTP commands needed to write a remote file over the temporary MVS file created in the previous section, from the TSO 'ready': FTP remote.machine.name LOGIN FTP (or your FTP login) PASSWORD your address (or your FTP password) BINARY GET remote filename file. name (replace QUIT 423

6 Non-FTP Protocols: II you are not using FTP protocols there are a number of alternative ways you can transfer files, depending on your environment. Using operating system utilities: To send a transport file from VM to MVS: SENDFILE file name A (10) user at node To receive the file on the MVS system, issue the TSO RECEIVE command. RECEIVE BLKSIZE(8000) To send a transport file from MVS to VM: XMIT node.id DA(file.name) SEQ When the file appears in the VM reader queue, RECEIVE it on to your disk. The user may RECEIVE the file as is, or RECEIVE the file into a pre-built file with the proper file attributes, as suggested above. Modem Protocols (eg, xmodem, ymodem, and zmodem): The modem protocol you use must be available on both the LOCAL and the REMOTE machines. For example, to download a transport file from UNIX, to your PC, using the XMODEM protocol: a. From the UNIX command line start the transfer with the XMODEM SEND command, usually SX filename, or XMODEM -S filename. b. Hotkey to the PC, and start your modem's RECEIVE software, specifying XMODEM and binary, if necessary. Miscellaneous Protocols: A variety of software supports binary file transfer. Examples include KERMIT, and PCj3270 emulation. The syntax is specific to your software, but be sure to select binary transfer and to preserve file attributes, as discussed above. 424

7 Desperation Protocols: You may find yourself in a situation where you have no direct method of moving files. For example, you may be on the NJE ("BITNET") protocol with no direct INTERNET connection. Don't despair! A number of sites support a facility called BITFTP, which allows you to send MAIL to the BITFTP site containing a subset of FTP commands to be executed. The file will be returned to your BITNET machine in due time. For more information about this facility, send a note to [email protected] containing the text HELP. You will receive information on using BITFTP. Alternatively, you can encode a binary file so that it may be sent as MAIL and decoded correctly on the TARGET machine. XXENCODE and UUENCODE are perhaps the most widely used of these encoding schemes, and of course require the corresponding DECODE program on the TARGET machine. Ask your site support personnel about the availability of this software at your installation. Selecting a Good/Better/Best Way: If you are fortunate enough to have more than one method of file transfer available to you, among the elements you might consider are the following: a. Machine Resources Batch jobs that can be run during 'off-hours' are less resource intensive than 'on-line' jobs that require your intervention. b. Human Resources - You will spend less time with the methods with which you are most familiar. Some methods e.g., SAS/CONNECT allow you to remain within one environment from start to finish. Ask your support personnel which methods they recommend, and which they are willing to help you with. c. Line and Resource Charges - The modem protocols, while "friendly" and convenient will involve long distance phone charges. The INTERNET connection will not. You may incur charges for intermediate files stored on disk, as well. d. Compressing and Archiving - A variety of software to reduce the overhead of file transfer may be available. Compression utilities such as ZIP, and archiving facilities such as TAR can minimize the size and number of files being transferred. Some modems support built in, automatic compression. Golcha's: The following list of "things that go wrong" is intended to alert you to several predictable errors in file tranfer: VMS users have several things to be aware of: 1. Creating transport files with the correct Carriage Control Option. Prior to release 6.08, transport files were created by SAS with a CC=CR; Unfortunately, software like FTP and Kermit added the "missing" carriage return during transfer, even while in BINARY mode. 425

8 In release 6.08 and up, ALWAYS use CC=NONE on statements creating the transport file. On prior releases, use the FDL job refered to in USAGE NOTES to correct the CC option so the file can be ported successfully. Usage Note: V6-COPY Kermit users on VMS should be aware that the commands are different, depending on whether VMS is the LOCAL, or the REMOTE machine. PC to VMS: VMS to PC: SET FILE BLOCK 80 SET FILE TYPE Fixed SET FILE BLOCK 80 SET FILE TYPE Block FTP users, especially of I BM machines, should be aware that 'bugs' have been reported regarding using FTP commands to perform file transfer in the FB 80/6000 format. Writing files on top of pre-existing files, to the best of the available information, eliminates this problem. For example, the following has been reported NOT to work correctly: BINARY FIXED 60, as has using the "magic" file name "FTP.DATA", which is intended for FB 60/6000 files by default. It can not be emphasized enough that importing catalogs created by later releases is a procedure guaranteed to create ambiguity. These catalogs may seem to import "correctly" and can give all signs of running "correctly"; however, this is definitely an "AT YOUR OWN RISK" exercise. All users should be aware that some operating systems (MVS, VM, VMS) use record length and blocksize attributes. Many others don't. It is your job to be sure that a "stream" file, is correctly formatted prior to importing to SAS on the target machine. As suggested above, pre-building a file to receive the "stream" will eliminate this difficulty. Mail users should be aware of the possibility of untranslatable characters. XX is preferable to UU, but neither is 100% perfect. There are others. Because many problems arise as a result of the multi-platform, multioperating system, and multi-personnel environments you will be traversing, assistance with other problems may best be obtained from a specialized users group on the net, among them, SAS-L. In Summary: Moving libraries, files, and catalogs from a distant location for use on your local machine is far preferable to data reentry and re-coding. The easiest way to accomplish the transport is through SAS/CONNECT. If you do not have CONNECT, or are running a release of SAS that is not compatible with the version under which the file was created, you have a number of additional options, including FTP, and BITFTP. 426

9 Each of these options contain pitfalls which can corrupt the file and create problems in using the transported data. As a general rule. creating an empty file with the attributes specified. and transporting the SAS file into the file you have created will go a long way toward maintaining the file integrity. It is important to understand the relationship between your machine. the machine where the desired file is stored. and the intervening platforms that handle the actual transfer. Finally. you must know a good deal about the file you wish to move. including the SAS release used in its creation. the operating system in use where it is stored. and the types of SAS files you wish to move. Obtaining this information prior to transfer is critical to successful transport. A special thanks is due to: Gary J. B. Pfeifer. University of Colorado. for his assistance wtth OS/2 and providing a Transport Application Library Kernon J. B. Gibes. of Monsanto Corporation. for assistance with VMS. SAS References You Will Need: Technical Report P-195: Transporting SAS Files Between Host Systems. SAS/CONNECT if you are using the software. The SAS Companion for (your operating system) SAS Procedures Guide (PROC's CPORT. CIMPORT. and COPY) 427