SFTP SHELL SCRIPT USER GUIDE

Similar documents
Unix Scripts and Job Scheduling

List of FTP commands for the Microsoft command-line FTP client

Shell Scripts (1) For example: #!/bin/sh If they do not, the user's current shell will be used. Any Unix command can go in a shell script

Using sftp in Informatica PowerCenter

IBM WebSphere Application Server Version 7.0

Unix Shell Scripts. Contents. 1 Introduction. Norman Matloff. July 30, Introduction 1. 2 Invoking Shell Scripts 2

Unix Sampler. PEOPLE whoami id who

DaimlerChrysler EBMX HTTP/s Quick Start Guide

Introduction to Shell Programming

An Introduction to the Linux Command Shell For Beginners

Installing IBM Websphere Application Server 7 and 8 on OS4 Enterprise Linux

Fundamentals of UNIX Lab Networking Commands (Estimated time: 45 min.)

File Transfer Protocol (FTP) Chuan-Ming Liu Computer Science and Information Engineering National Taipei University of Technology Fall 2007, TAIWAN

Command Line Crash Course For Unix

Linux command line. An introduction to the Linux command line for genomics. Susan Fairley

Thirty Useful Unix Commands

Setting Up the Site Licenses

Linux Shell Script To Monitor Ftp Server Connection

Network Security In Linux: Scanning and Hacking

Web Hosting: Pipeline Program Technical Self Study Guide

SAS 9.4 In-Database Products

TS-800. Configuring SSH Client Software in UNIX and Windows Environments for Use with the SFTP Access Method in SAS 9.2, SAS 9.3, and SAS 9.

embeo Getting Started and Samples

SSL Tunnels. Introduction

The Basics of FTP. Basic Order of Operations: Commands: FTP (File Transfer Protocol) allows a user to transfer files to/from a remote network site.

Managed File Transfer with Universal File Mover

Lecture 4: Writing shell scripts

Introduction to Shell Scripting

Enhanced Connector Applications SupportPac VP01 for IBM WebSphere Business Events 3.0.0

Syntax: cd <Path> Or cd $<Custom/Standard Top Name>_TOP (In CAPS)

Beginners Shell Scripting for Batch Jobs

HP-UX Essentials and Shell Programming Course Summary

SendMIME Pro Installation & Users Guide

Answers to Even-numbered Exercises

Cisco Networking Academy Program Curriculum Scope & Sequence. Fundamentals of UNIX version 2.0 (July, 2002)

Secure Shell Demon setup under Windows XP / Windows Server 2003

Basic C Shell. helpdesk@stat.rice.edu. 11th August 2003

IBM Pure Application Create Custom Virtual Image Guide - Part 1 Virtual Image by extending

UNIX: Introduction to TELNET and FTP on UNIX

Table of Contents Introduction Supporting Arguments of Sysaxftp File Transfer Commands File System Commands PGP Commands Other Using Commands

CS Unix Tools & Scripting Lecture 9 Shell Scripting

A Crash Course on UNIX

ICS 351: Today's plan

High-Performance Reservoir Risk Assessment (Jacta Cluster)

Configuring System Message Logging

NETWORK ADMINISTRATION

WinSCP PuTTY as an alternative to F-Secure July 11, 2006

Tera Term Telnet. Introduction

MATLAB on EC2 Instructions Guide

MAX_RMAN_08137_IGNORE=5 DISK_RETENTION_POLICY='RECOVERY WINDOW OF 7 DAYS'

Automated Offsite Backup with rdiff-backup

Using the AVR microcontroller based web server

File Transfer Protocol

2 Advanced Session... Properties 3 Session profile... wizard. 5 Application... preferences. 3 ASCII / Binary... Transfer

Secure Shell. The Protocol

FTP protocol (File Transfer Protocol)

Hands-On UNIX Exercise:

SSH Connections MACs the MAC XTerm application can be used to create an ssh connection, no utility is needed.

Bash shell programming Part II Control statements

IBM Campaign Version-independent Integration with IBM Engage Version 1 Release 3 April 8, Integration Guide IBM

AN INTRODUCTION TO UNIX

Installation Guide for WebSphere Application Server (WAS) and its Fix Packs on AIX V5.3L

Monitoring Clearswift Gateways with SCOM

HIGH AVAILABILITY SETUP USING VERITAS CLUSTER SERVER AND NETAPP SYNCHRONOUS SNAPMIRROR. Jorge Costa, NetApp June 2008

LSN 10 Linux Overview

Centers for Medicare and Medicaid Services. Connect: Enterprise Secure Client (SFTP) Gentran. Internet Option Manual

How to set FTP Server (IIS)

Shellshock Security Patch for X86

1 Reflection ZFE 5. 2 Security Considerations Troubleshooting the Installation 19. Contents 1

Tivoli Access Manager Agent for Windows Installation Guide

BASIC TCP/IP NETWORKING

BASH Scripting. A bash script may consist of nothing but a series of command lines, e.g. The following helloworld.sh script simply does an echo.

Automating admin tasks using shell scripts and cron Vijay Kumar Adhikari.

Router CLI Overview. CradlePoint, Inc.

Make a folder named Lab3. We will be using Unix redirection commands to create several output files in that folder.

The Einstein Depot server

IBM WebSphere MQ File Transfer Edition, Version 7.0

UNIX, Shell Scripting and Perl Introduction

Universal Event Monitor for SOA Reference Guide

CA Workload Automation Agent for UNIX, Linux, or Windows

Using LDAP Authentication in a PowerCenter Domain

Sametime Gateway Version 9. Deploying DMZ Secure Proxy Server

Mass Deploying Bomgar Software to Macs

Configuring IBM WebSphere Application Server 6.1 to Support SAS 9.2 Web Applications

Moxa Device Manager 2.3 User s Manual

Guide to the Configuration and Use of SFTP Clients for Uploading Digital Treatment Planning Data to IROC RI

Appendix. Web Command Error Codes. Web Command Error Codes

INASP: Effective Network Management Workshops

CS 2112 Lab: Version Control

Cloud Server powered by Mac OS X. Getting Started Guide. Cloud Server. powered by Mac OS X. AKJZNAzsqknsxxkjnsjx Getting Started Guide Page 1

Scheduling in SAS 9.3

PMOD Installation on Linux Systems

WebSphere MQ FTP client Version 1.3

Configuring Logging. Information About Logging CHAPTER

GDC Data Transfer Tool User s Guide. NCI Genomic Data Commons (GDC)

How to upload large files to a JTAC Case

THE HONG KONG POLYTECHNIC UNIVERSITY Department of Electronic and Information Engineering

Transcription:

SFTP SHELL SCRIPT USER GUIDE FCA US INFORMATION & COMMUNICATION TECHNOLOGY MANAGEMENT

Overview The EBMX SFTP shell scripts provide a parameter driven workflow to place les on the EBMX servers and queue the les for processing. SFTP is the protocol used to transmit and queue the data to EBMX. SFTP requires that each environment (yours and EBMX) exchange Public Keys before any transmissions can occur. In order to expedite the setup, please submit a Trouble Ticket requesting SFTP setup to the TTTS group EBMX. [Please include server name/ip, application name, AMS#, and technical contact(s)]. The EBMX group will work with you during the setup and testing if you are unfamiliar with SFTP. The shell scripts (Korn & Bourne shell) were developed using Ubuntu Linux and tested on AIX and Solaris. The EBMX Group provides these scripts as-is, if you nd there is an issue or have a question about functionality, please submit a Trouble Ticket and route to EBMX. The EBMX SFTP shell script is one of two options for applications to access the functions of EBMX. EBMX also provides an interface based on IBM WebSphere MQ Series. These options should be discussed with an EBMX analyst to determine the best choice when selecting an interface. Features Transfer le using SFTP Accept parameters as arguments Check for expected response Full activity logging Return code setting Korn shell version (ksh) Bourne shell version (sh) Operating Environment The EBMX SFTP shell script is expected to run on UNIX variants such as Solaris, AIX, and Linux. The operating system must be able to support either the Korn shell or the Bourne shell and have a standard SFTP client installed. 04/29/2015 1 FCA US

Operation The EBMX SFTP shell script has two versions: Description Korn Shell Version Bourne Shell Version Shell Script Name sftp_put_rename.ksh sftp_put_rename.sh The EBMX SFTP shell script requires eight arguments: Name Description Value User ID The EBMX member ID of the sender of the data. Supplied by the EBMX administrative team. SFTP Host Name The IP address or DNS name of the EBMX system. EBMX Production = ebmxprod.appl.chrysler.com EBMX Development = ebmxdev.appl.chrysler.com SFTP Port SFTP port number. 50322 Local File Name Name of the le to send including path, if necessary. Supplied by the user of the script. Sender ID The EBMX member ID of the sender of the data. Supplied by the EBMX administrative team. Receiver ID The EBMX member ID of the receiver of the data. Supplied by the EBMX administrative team. Document Type The document type of the data Supplied by the EBMX Log le name being sent. The log le name for activity logging including path, if necessary. administrative team. Supplied by the user of the script. All arguments are mandatory. The shell script exits with at return code of zero if the process was successful. The shell script exits with at return code of 1 if the process fails. It is the responsibility of the user of the shell script to check the return code for success or failure. If the process fails, interrogate the log le to determine the reason for the failure. The EBMX administrative team can assist in the interpretation of the log le. If the log le does not exist, the shell script will create a new one. If the log le does exist, the shell script will append the results to the end of the existing log. If the put and rename is successful, the log le will contain the character string Success!. If an error occurs at any time during the workflow, the character string Failure! is placed in the log le. Therefore, the EBMX team recommends that log les are kept for a minimum of seven days. 04/29/2015 2 FCA US

Examples Bourne Shell./sftp_put_rename.sh testdoc clmsd058b.cldc.chrysler.com 50322 test_le_1 TESTDOC LOOPTEST LOOPBACK mylog.log Korn shell./sftp_put_rename.ksh testdoc clmsd058b.cldc.chrysler.com 50322 test_le_1 TESTDOC LOOPTEST LOOPBACK mylog.log Note: The above examples should be entered as one continuous command and not broken into individual lines. 04/29/2015 3 FCA US

Example Log File Each run of the EBMX SFTP shell script will produce the following log: Success Example: Running on: 2010-11-03:14:52:21 number of arguments received: 8 ARG[1] remoteuser: testdoc ARG[2] remotehost: clmsd058b.cldc.chrysler.com ARG[3] remoteport: 50322 ARG[4] remotele: test_le_1 ARG[5] ecxsender: TESTDOC ARG[6] ecxreceiver: LOOPTEST ARG[7] ecxdoctype: LOOPBACK ARG[8] remotelog: mylog.log localle for put is: test_le_1.20101103145221.30236 Connecting to clmsd058b.cldc.chrysler.com... You are accessing FCA (Fiat Chrysler Automobiles) systems. Connected to EBMX SFTP server. Connection accepted. Access to FCA's computer systems is controlled. FCA authorizes use for its business purposes only. Unauthorized access is prohibited due to risk of irreparable harm to FCA. FCA management may monitor use to ensure compliance with its policies. FCA may terminate access privileges, take disciplinary action and/or institute civil or criminal proceedings to enforce this policy. If any part of this policy is unacceptable to you, please disconnect now!!! Thank you. YOUR ACCESS IS BEING LOGGED & MONITORED!!!! sftp> dir ERRORS RESULTS TESTDOC TMP sftp> cd /TMP sftp> put test_le_1 test_le_1.20101103145221.30236 Uploading test_le_1 to /TMP/test_le_1.20101103145221.30236 sftp> rename test_le_1.20101103145221.30236 /TESTDOC/LOOPTEST/LOOPBACK/test_le_1.20101103145221.30236 sftp> quit End of SFTP PUT and RENAME About to check exit status... Success! 04/29/2015 4 FCA US

Failure Example: (Invalid le) Running on: 2010-11-03:14:52:41 number of arguments received: 8 ARG[1] remoteuser: testdoc ARG[2] remotehost: clmsd058b.cldc.chrysler.com ARG[3] remoteport: 50322 ARG[4] remotele: xtest_le_1 ARG[5] ecxsender: TESTDOC ARG[6] ecxreceiver: LOOPTEST ARG[7] ecxdoctype: LOOPBACK ARG[8] remotelog: mylog.log localle for put is: xtest_le_1.20101103145241.24721 Connecting to clmsd058b.cldc.chrysler.com... You are accessing FCA (Fiat Chrysler Automobiles) systems. Connected to EBMX SFTP server. Connection accepted. Access to FCA's computer systems is controlled. FCA authorizes use for its business purposes only. Unauthorized access is prohibited due to risk of irreparable harm to FCA. FCA management may monitor use to ensure compliance with its policies. FCA may terminate access privileges, take disciplinary action and/or institute civil or criminal proceedings to enforce this policy. If any part of this policy is unacceptable to you, please disconnect now!!! Thank you. YOUR ACCESS IS BEING LOGGED & MONITORED!!!! sftp> dir ERRORS RESULTS TESTDOC TMP sftp> cd /TMP sftp> put xtest_le_1 xtest_le_1.20101103145241.24721 stat xtest_le_1: No such le or directory sftp> rename xtest_le_1.20101103145241.24721 /TESTDOC/LOOPTEST/LOOPBACK/xtest_le_1.20101103145241.24721 Couldn't rename le "/TMP/xtest_le_1.20101103145241.24721" to "/TESTDOC/LOOPTEST/LOOPBACK/xtest_le_1.20101103145241.24721": No such le or directory sftp> quit End of SFTP PUT and RENAME About to check exit status... Failure! 04/29/2015 5 FCA US

Failure Example: (Invalid Public Key) Running on: 2010-11-03:14:53:16 number of arguments received: 8 ARG[1] remoteuser: testdoc ARG[2] remotehost: clmsd058b.cldc.chrysler.com ARG[3] remoteport: 50322 ARG[4] remotele: xtest_le_1 ARG[5] ecxsender: TESTDOC ARG[6] ecxreceiver: LOOPTEST ARG[7] ecxdoctype: LOOPBACK ARG[8] remotelog: mylog.log localle for put is: xtest_le_1.20101103145316.20968 Connecting to clmsd058b.cldc.chrysler.com... You are accessing FCA (Fiat Chrysler Automobiles) systems. Connected to EBMX SFTP server. Connection accepted. Access to FCA's computer systems is controlled. FCA authorizes use for its business purposes only. Unauthorized access is prohibited due to risk of irreparable harm to FCA. FCA management may monitor use to ensure compliance with its policies. FCA may terminate access privileges, take disciplinary action and/or institute civil or criminal proceedings to enforce this policy. If any part of this policy is unacceptable to you, please disconnect now!!! Thank you. YOUR ACCESS IS BEING LOGGED & MONITORED!!!! Permission denied (publickey). Couldn't read packet: Connection reset by peer End of SFTP PUT and RENAME About to check exit status... Failure! 04/29/2015 6 FCA US

Appendix A. Shell Scripts (Korn) Copy and save to a le with the extenstion.ksh #!/bin/ksh # # This script will accept parameters/arguments and connect to the EBMX app # server via the OpenSourceSSH SFTP server and PUT a le. The reply # is examined for any errors - this determines success/failure of this # script. This script is intended for use with ksh. # # store the date for display/le naming purposes datetime=`date '+%Y%m%d%H%M%S'` displaydatetime=`date '+%Y-%m-%d:%H:%M:%S'` sftp_response_log=" " # store the number of arguments received and the arguments themselves numargsreceived=$# argumentsreceived=$* remoteuser=$1 remotehost=$2 remoteport=$3 remotele=$4 ecxsender=$5 ecxreceiver=$6 ecxdoctype=$7 remotelog=$8 # construct the name of the local le to be PUT localle="$remotele.$datetime.$random" # this routine echos the date, arguments and localle echo_args () { echo "Running on: $displaydatetime"; echo "number of arguments received: $numargsreceived"; echo; # check to see if the number of arguments is correct if [ $numargsreceived -eq 8 ]; then echo "ARG[1] remoteuser: $remoteuser"; echo "ARG[2] remotehost: $remotehost"; echo "ARG[3] remoteport: $remoteport"; echo "ARG[4] remotele: $remotele"; echo "ARG[5] ecxsender: $ecxsender"; echo "ARG[6] ecxreceiver: $ecxreceiver"; echo "ARG[7] ecxdoctype: $ecxdoctype"; echo "ARG[8] remotelog: $remotelog"; echo; elif [ $numargsreceived -gt 8 ]; then echo "Too many arguments received!!!"; echo "Received $numargsreceived arguments"; echo "Arguments received: $argumentsreceived"; echo "Arguments expected: <remoteuser> <remotehost> <remoteport> <remotele> <ecxsender> <ecxreceiver> <ecxdoctype> [<remotelog>]"; exit 1; elif [ $numargsreceived -lt 8 ]; then echo "Too few arguments received!!!"; echo "Received $numargsreceived arguments"; echo "Arguments received: $argumentsreceived"; echo "Arguments expected: <remoteuser> <remotehost> <remoteport> <remotele> <ecxsender> <ecxreceiver> <ecxdoctype> [<remotelog>]"; exit 1; echo "localle for put is: $localle"; echo; } 04/29/2015 7 FCA US

Appendix A. Shell Scripts (Korn) (continued) # this routine is responsible for the PUT of the le and RENAME # we also do a DIR and PWD for completeness sftp_put_rename () { # open sftp as a co-process (ksh only) so that we may write to/read from the # process. The reads are stored in a shell variable sftp -oport=$remoteport $remoteuser@$remotehost 2>&1 & print -p dir print -p cd /TMP print -p put $remotele $localle print p chmod 666 $localle print -p rename $localle /$ecxsender/$ecxreceiver/$ecxdoctype/$localle print -p quit print -p : # read all the responses at the end of the process while read -p sftp_response do # add a new line so it looks right - otherwise, it wraps on one line without # newlines xx="$sftp_response"; echo "$xx" >> $remotelog; sftp_response_log="$sftp_response_log$xx"; done } : # this runs the echo_args routine - the eval is necessary for lename # expansion prior to redirection eval echo_args 1>>$remotelog 2>&1 # run the sftp_put_rename routine - since we use a coprocess to store output # in a shell variable, we don't need to worry about redirection sftp_put_rename # grep the SFTP session log looking for Couldn denied - if we nd one, it failed echo "End of SFTP PUT and RENAME" >>$remotelog 2>&1 echo "About to check exit status..." >>$remotelog 2>&1 if [ `echo $sftp_response_log egrep -c '(Couldn denied)'` -eq 0 ] then echo "Success!" >>$remotelog 2>&1; exit 0; else echo "Failure!" >>$remotelog 2>&1; exit 1; 04/29/2015 8 FCA US

Appendix B. Shell Scripts (Bourne Shell) Copy and save to a le with the extenstion.sh #!/bin/sh # # This script will accept parameters/arguments and connect to the EBMX app # server via the OpenSourceSSH SFTP server and PUT a le. The reply # is examined for any errors - this determines success/failure of this # script. This script is intended for use with ksh. # # store the date for display/le naming purposes datetime=`date '+%Y%m%d%H%M%S'` displaydatetime=`date '+%Y-%m-%d:%H:%M:%S'` # store the number of arguments received and the arguments themselves numargsreceived=$# argumentsreceived=$* remoteuser=$1 remotehost=$2 remoteport=$3 remotele=$4 ecxsender=$5 ecxreceiver=$6 ecxdoctype=$7 remotelog=$8 #echo $datetime; # dene temporary log... tempremotelog="./sftp_log.$datetime"; # construct the name of the local le to be PUT localle="$remotele.$datetime" if [ $# -gt 0 ] then numargsreceived=$#; else numargsreceived=0; argsreceived=$*; # this routine echos the date, arguments and localle echo_args () { echo "Running on: $displaydatetime"; echo "number of arguments received: $numargsreceived"; # check to see if the number of arguments is correct if [ $numargsreceived -eq 8 ]; then echo "Start of FTP PUT and SUBMIT"; exiterror=0; elif [ $numargsreceived -eq 7 ]; # generate a remote log le when only 7 arguments are passed then remotelog=$tempremotelog; exiterror=0; elif [ $numargsreceived -gt 8 ]; then echo "Too many arguments received!!!"; echo "Received $numargsreceived arguments"; echo "Arguments received: $argsreceived"; echo "Arguments expected: <remoteuser> <remotehost> <remoteport> <remotele> <ecxsender> <ecxreceiver> <ecxdoctype> [<remotelog>]"; exiterror=1; elif [ $numargsreceived -lt 7 ]; then echo "Too few arguments received!!!"; echo "Received $numargsreceived arguments"; echo "Arguments received: $argsreceived"; echo "Arguments expected: <remoteuser> <remotehost> <remoteport> <remotele> <ecxsender> <ecxreceiver> <ecxdoctype> [<remotelog>]"; exiterror=1; 04/29/2015 9 FCA US

Appendix B. Shell Scripts (Bourne Shell) (continued) echo; echo "ARG[1] remoteuser: $remoteuser"; echo "ARG[2] remotehost: $remotehost"; echo "ARG[3] remoteport: $remoteport"; echo "ARG[4] remotele: $remotele"; echo "ARG[5] ecxsender: $ecxsender"; echo "ARG[6] ecxreceiver: $ecxreceiver"; echo "ARG[7] ecxdoctype: $ecxdoctype"; echo "ARG[8] remotelog: $remotelog"; echo "Local File: $localle"; echo "Temp Log: $tempremotelog"; echo; } # this routine is responsible for the PUT of the le and ensuing SUBMIT # we also do a DIR and PWD for completeness sftp_put_rename () { sftp -oport=$remoteport $remoteuser@$remotehost <<!EOF cd /TMP put $remotele $localle chmod 666 $localle rename $localle /$ecxsender/$ecxreceiver/$ecxdoctype/$localle quit!eof echo End of SFTP PUT and RENAME; : } # this runs the echo_args routine - the eval is necessary for lename # expansion prior to redirection eval echo_args 1>>$tempremotelog 2>&1 # this runs the sftp_put_rename routine - the eval is necessary for lename # expansion prior to redirection if [ $exiterror -ne 1 ]; then eval sftp_put_rename 1>>$tempremotelog 2>&1; else echo "Incorrect number of Arguments, please see $tempremotelog"; exit; echo About to check exit status... >>$tempremotelog # grep the remote log looking for a tracking ID - if we nd one, it # worked if [ $exiterror -ne 1 ]; then it_worked=`egrep -c '(Couldn denied)' $tempremotelog`; else it_worked=0; # if we nd one tracking ID, it worked. If we nd more than one, the log # contains multiple sessions, and we can't tell for sure if the last one # worked if [ $it_worked -eq 0 ]; then echo "Success!" >>$tempremotelog; exitcode=0; else echo "Failure!" >>$tempremotelog; exitcode=1; if [ "$tempremotelog"!= "$remotelog" ]; then cat $tempremotelog >>$remotelog; rm $tempremotelog; exit $exitcode; 04/29/2015 10 FCA US