Introduction to the UNIX command-line shell Riccardo Murri Grid Computing Competence Center, University of Zurich

Transcription

1 GC3: Grid Computing Competence Center Introduction to the UNIX command-line shell Riccardo Murri Grid Computing Competence Center, University of Zurich Sept. 27, 2012

2 Outline The goal of this session is to recap some UNIX system concepts and introduce the command line shell. Slides available for download from: A cheatsheet with a quick summary of syntax and commands is available at teaching/lsci2012/lab01/sh-cheatsheet.html

3 Layers of UNIX shell commands A user interacts with the shell: the purpose of a shell is to execute users commands. Command is just another word for a program. kernel

4 Programs and processes A process is an instance of a program that is being executed. Several processes may be associated with the same program: e.g., the same collection of instructions can be performed on multiple data sets. A command is the textual specification of a program and data to turn into a process.

5 Interaction with the shell The shell: 1. reads commands from the user one line at a time, 2. parses the command-line, and 3. executes it.

6 Interaction with the shell: reading The shell reads commands one line at a time. An interactive shell will print a prompt to signal it s ready to accept input.

7 Interaction with the shell: parsing Any line input to the shell is split into words, by scanning it character by caracter: Each occurrence of a quote character (, or ") accumulates all the text up to the next matching quote character into the current word. There are differences among the different quote characters, and some special treatment of other characters (e.g., \,, #,... ) but we shall see them aftwerwards. Each occurrence of a (sequence of) blank characters ends the current word. A new word is started at the next (unquoted) non-blank character. A command-line is terminated at the first (unquoted) occurrence of newline, ;,, &. Exact rules available at: pubs.opengroup.org/ onlinepubs/ / utilities/ xcu chap02.html#tag 02 03

8 Interaction with the shell: execution Each UNIX process is initialized with a list (1D array) of strings, called the argv vector. Each word on the command-line is assigned to a string in the argv vector: the initial word (the command name) has index 0, and the index increases left-to-right.

9 Hello, world from the shell At the shell prompt, type: echo Hello, world. What s the difference if we type this instead? echo "Hello, world." And with this? echo Hello, world;

10 The most useful command of all The command man displays the manual page of any other command on the system. Example: # shows the manual for the shell itself man sh

11 A few other commands ls cat zcat more less gzip gunzip wget gedit, emacs, vi, nano List the contents of a directory Show the contents of a file Show the contents of a compressed file Show file contents, one screenful at a time A more with more advanced functionality Compress a file Uncompress a file Download a URL to a file Text editors

12 Shell scripts A shell read its input line by line, but not necessarily from a terminal! You can tell the shell to read and execute commands from a file, by passing it as the first argument to the shell. A file containing shell commands is called a shell script. Exercise 1: Write a shell script that displays the single line Hello, world.

13 Shell variables Shell variables are automatically created when they are first assigned a value: 1 ANSWER=42 name=john The value of a variable is taken by prefixing its name with the character $; i.e., the string $name is substituted with the value of the variable name, as if it had been typed directly on the command-line. 1 Note: no spaces allowed around the equal sign =.

14 Variable expansion, I Substituting the value of a variable in a command line is called variable expansion. Examples: # What do you think this will display? name=john; echo Hello, $name; # and this one? name="john Doe"; echo Hello, $name; You can use ${name} instead of $name. (When could this be useful?)

15 Variable expansion, II Variable expansion occurs inside double quotes (") as well, but not inside single quotes ( ). Try out: name=john; # variable is expanded echo "Hello, $name"; # this one is not echo Hello, $name ;

16 Special Variables: Positional arguments The special variables $0, $1, $2, etc. expand to the corresponding string in the argv vector. Quiz: How do you get the 11-th command-line argument? Exercise 2: Write a hello.sh script, that takes one argument on the command-line (a person s name), and outputs a single line, consisting of the words Hello, followed by the name. Example: $ sh hello.sh Joe Hello, Joe.

17 UNIX process I/O UNIX processes perform Input/Output operations on streams. Three streams are available to any process: standard input (stdin): data which is read by the program. standard output (stdout): the stream where a process writes its main output data. standard error (stderr): an additional stream for writing error messages, diagnostics or progress messages. More streams can be opened by a program while it s running. Quiz: Can you think of an example to motivate the stdout/stderr separation?)

18 I/O redirection A process stdin, stdout and stderr are normally connected to whatever streams its parent shell was connected to. There is special shell syntax for specifying where the streams of a command should be redirected to. command > file command >> file command >& file command < file c1 c2 stdout of command overwrites file stdout of command is appended to file stdout and stderr of command are written to file stdin of command is read from file stdout of c1 is stdin of c2

19 Exercises on I/O redirection Exercise 3: Write a script that, given a file name and a person name on the command-line, will write the usual Hello greeting to the specified file. Exercise 4: Write a script that, given a matrix file name (e.g., M0,5-D4): 1. Downloads the compressed matrix file from (compressed files have a.gz extensions) 2. Uncompresses the matrix file (e.g., M0,5-D4.sms.gz) into a plain-text file (e.g., M0,5-D4.sms)

20 The UNIX filesystem, I UNIX directories are organized in a single tree-like structure. Files are the leaves of this tree. There is one and only one root directory, denoted by \. The directory tree is not confined into a single disk; every accessible disk/partition/network storage corresponds to a subtree. The branch where this subtree is rooted is called a mount point.

21 The UNIX filesystem, II Files are the leaves of the directory tree. Each file can be identified by specifying the path along the tree to reach that leaf from the current node. Nodes encountered along the path (so-called path compoments) are separated by /. /home/murri/lsci/lab1.tex absolute path (= starting from the root) lsci/lab1.tex relative path (= starting from the current node).. relative path to upper-level directory

22 Filesystem commands pwd cd mkdir rmdir ls cp mv rm path to current directory change current directory make a new directory remove/delete an existing directory list contents of a directory copy files move/rename files remove/delete files

23 Options, I Options select an alternate behavior of a command. By convention, options start with the - character. Examples: ls -s *.tex ls -sh ls -s -h ls --size ls --help one option, one argument two options, combined two options, separately written long form of the -s option quick help on command usage

24 Options, II Some options require an argument as well. tar --create --file=tex.tar *.tex tar --create --file tex.tar *.tex tar --create -f tex.tar *.tex tar -c -ftex.tar *.tex

25 The exit code Upon termination, each UNIX process specifies an exit code, which is an 8-bit unsigned integer. By convention, exit code 0 means OK/success/true. Any exit code different from 0 is normally interpreted as error/failure/false. Note: This is the opposite convention than used, e.g., in C and Java. The exit code of the last executed command is available as the special shell variable $?

26 Conditionals, I The shell supports conditional execution of commands: if command; then # do this when exitcode 0... else # otherwise do this... fi The if statement only checks the exit code of command.

27 Conditionals, 2 Any command can be used for the test clause in a conditional. However, the test command is the one more frequently used: test -r path True if path is readable. test -w path True if path is writeable. test -d path True if path is a directory. test -z "string" True if string is null. test -n "string" True if string is not null. test "s1" = "s2" True if the strings are equal. test "s1"!= "s2" True if the strings are different.... and many more: man test is your friend! The open bracket [ is an alias for test, but it requires a closing ] at the end; e.g., [ -d /tmp ] # do something...

28 Conditionals, III Exercise 5: Modify the hello.sh script: if the name given on the command-line is Mary, then it should print Happy birthday, Mary. ; if any other name is given, print Hello,...

29 Conditionals, IV Other syntaxes for conditional execution of commands: cmd1 && cmd2 cmd1 cmd2 Execute cmd2 if and only if cmd1 exited with code 0. Execute cmd2 if and only if cmd1 exited with code non-zero. The && and operators are left-associative.

30 while -loops while command; do # do this while exit code is 0 done The specified command is executed at each iteration: if its exit code is 0, then the body of the loop is executed; otherwise the shell continues with the next instructions after the done statement.

31 for -loops, I for variable in value1 value2...; do # do this for every value of $variable done The body of the loop is executed once for each value to be assigned to variable. Note 1: The list of values is terminated by ;. Note 2: Each value is a word in shell syntax. In particular, the list of values is space-separated.

32 for -loops, II The special shell variable expands to the list of arguments passed on the command-line. Exercise 6: Write a multihello.sh script that takes a variable number of names on the command-line and, for each of these names prints Hello, name on a separate line.

33 Background execution The & operator may be specified at the end of a command-line to indicate that the command should be executed in the background, i.e., concurrently with the shell and with no interaction with the terminal. Example: # decompress a file in the background gunzip M0,5-D4.sms.gz & A backgrounded process cannot read from stdin nor write to stdout. (It will be stopped by the kernel if it tries to.)

34 Controlling UNIX processes, I Each process is assigned a unique numeric process identifier (PID). The shell variable $! contains the PID of the most recently-started background command. The ps and kill commands can be used to control a process given its PID: ps [PID] kill [-l SIGNAL] PID Print information on a process. Send a signal to a process. (By default, SIGTERM)

35 ps usage Example of ps usage: $ ps 1051 PID TTY STAT TIME COMMAND 1051 pts/3 T 0:01 emacs -nw src.c++/examples/ra The naming of parts: PID The process ID. TTY The controlling terminal. STAT Kernel process status. TIME CPU time consumed so far. COMMAND Command-line that started this process. The ps output can be customized to print almost every field of the kernel process table. (Use man ps.)

36 Process statuses D R S T Uninterruptible sleep (usually I/O). Running or runnable (on run queue). Interruptible sleep (waiting for an event to complete). Stopped, either by a job control signal or because it is being traced. Z Zombie process, terminated but not reaped by its parent. More info:

37 Controlling UNIX processes, II Using the kill command a process can be sent a signal, e.g., to terminate it, or temporarily suspend execution. SIGTERM Ask process to terminate. (The process may choose to ignore this signal.) SIGKILL Terminate process forcibly, here and now. SIGHUP Sent to a process when the controlling terminal has been disconnected (e.g., ssh session closed). Example: # forcibly terminate a process kill -l KILL PID;

38 References [1] A stream is an ordered sequence of data bytes which can be read until the end of file ; from: streams [2] U. Waldmann, A Guide to Unix Shell Quoting, uwe/lehre/unixffb/ quoting-guide.html (A readable introduction and reference on shell quoting.) [3] The POSIX Shell Command Language Specification, onlinepubs/ /utilities/xcu chap02.html

39 Further reading, I Steve s Bourne / Bash shell scripting tutorial by S. Parker, Bash Guide for Beginners by M. Garrels, Bash-Beginners-Guide.html Linux Shell Scripting Tutorial - A Beginner s handbook by V. G. Gite, Advanced Bash-Scripting Guide by M. Cooper,

40 Further reading, II sh cheat-sheet by S. parker, Unix/Linux Command Cheat Sheet, by J. Peddicord, unixlinux-command-cheat-sheet/