FEEG Applied Programming 7 - Shell scripts and Makefiles

Transcription

1 FEEG Applied Programming 7 - Shell scripts and Makefiles Sam Sinayoko / 44

2 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 2 / 44

3 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 3 / 44

4 Learning outcomes Give three reasons for writing shell scripts Learn the basics of shell scripts: Make files defining variables passing arguments Give three reasons for writing make files Learn the basics: Using make to build a program from source Using make to (re)generate a series of files 4 / 44

5 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 5 / 44

6 Introduction Motivation for shell scripts Automate simple tasks: creating/copying/moving files or directories simple data processing running a series of programs successively Scalable: run the same commands on a 1000 files Makes it easy to pass arguments to commands Examples: script for downloading lecture notes from feeg6002 website into a specific directory script for downloading a C program from feeg6002 course into a specific directory and compiling it with gcc 6 / 44

7 Introduction Motivation for make files Easily build / re-build a set of files Particularly useful to build programs in C or other compiled language Reproducible papers: makes it easy to update the paper and its figures after changing something (e.g. one data file, or one program) Captures dependencies between files so only the necessary files are re-built when some of the source files are changed (unlike a shell script which runs all commands) Independent commands can be run in parallel Other similar programs exist (scons, ant, cmake), but: make is already installed on any Linux/OS X machine philosophy is similar Example: feeg6002 website and lecture notes, most open source programs (e.g. emacs, vim). 7 / 44

8 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 8 / 44

9 Elementary shell scripts Simply put the list of successive commands you want to run into a file and run with the "bash" shell command. cmd1 cmd2 cmd3... It is common to append the suffix ".sh" to make it clear the file is a shell script. 9 / 44

10 Elementary shell scripts Example: downloading lecture notes into some folder On Linux/OS X, create file =feeg6002pdf.sh= 1 : mkdir -p /tmp/feeg6002/applied_programming/ wget feeg6002_applied_programming01.pdf mv feeg6002_applied_programming01.pdf \ /tmp/feeg6002/applied_programming/ Run the script (see Appendix A for an alternative): bash feeg6002pdf.sh ls /tmp/feeg6002/applied_programming feeg6002_applied_programming01.pdf 1 The "\" is for splitting long lines to fit the command to the screen. 10 / 44

11 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 11 / 44

12 Shell script variables Basic syntax No need to declare variable type as in C. By default everything is a string Variables are defined and used like this NAME=World echo Hello $NAME Hello World Notes: no space around equal sign. Use dollar sign $ to expand variable name. 12 / 44

13 Shell script variables Variable names in strings Variable in double quoted "strings" are expanded Variable in single quoted strings are not expanded. NAME=world echo "Hello double quoted $NAME" echo Hello single quoted $NAME (Oops) Hello double quoted world Hello single quoted $NAME (Oops) 13 / 44

14 Shell scripts variables Protecting variable names with ${} Use curly brackets to separate variable name from the rest FILE="image" echo $FILE01.png "(broken)" echo ${FILE}01.png "(works)".png (broken) image01.png (works) Note: there is no variable FILE01 so it defaults to the empty string, the shell doesn t throw an error. 14 / 44

15 Shell script Improved script feeg sh FOLDER=/tmp/feeg6002/applied_programming/ N=01 # lecture number filename=feeg6002_applied_programming$n.pdf mkdir -p $FOLDER wget mv $filename $FOLDER 15 / 44

16 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 16 / 44

17 Command line arguments Command line arguments It is very easy to define command line arguments The following variables are available automatically: $0 command name $1 argument 1 $2 argument $9 argument 9 $@ all arguments $# number of arguments 17 / 44

18 Command line arguments Command line arguments: example (1/2) Define file "arg-test.sh": echo "command: $0" echo "arg 1: $1" echo "arg 2: $2" echo "arg 3: $3" echo "all args: echo "number of args: $#" 18 / 44

19 Command line arguments Command line arguments: example (2/2) Test file "arg-test.sh": bash arg-test.sh I love Southampton command: arg-test.sh arg 1: I arg 2: love arg 3: Southampton all args: I love Southampton number of args: 3 19 / 44

20 Shell script Improved script feeg sh FOLDER=/tmp/feeg6002/applied_programming/ N=$(printf "%02d" $1) # lecture number filename=feeg6002_applied_programming$n.pdf mkdir -p $FOLDER wget mv $filename $FOLDER Usage: bash feeg6002pdf-3.sh 1 bash feeg6002pdf-3.sh 2 ls /tmp/feeg6002/applied_programming feeg6002_applied_programming01.pdf feeg6002_applied_programming02.pdf 20 / 44

21 Self study Exercise Write a shell script getcode.sh that: 1. takes any C file cmd.c (e.g. hello.c) from Professor Fangohr s FEEG6002 lecture notes as an argument 2. downloads the file from 3. moves it to an existing directory of your choice 4. (optional) compiles the source file with gcc and creates executable cmd Example usage bash getcode.sh hello.c cd code # or wherever getcode.sh sends the files./hello 21 / 44

22 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 22 / 44

23 Make: introduction The creation of a series of files via a list of commands can be described through: a flow chart: the intuitive way. a dependency graph: the more powerful way used by the make program. Flow charts An intuitive way to describe a build is through a flow chart. The program flows from the source to the target. Flow chart 1: "Hello world" C program. gcc Source: hello.c Target: hello 23 / 44

24 Make: introduction Flow chart 2: plotting data (from lab 3, tabulatesin.c gcc tabulatesin figure.png plotfile.py data.txt redirect STDOUT 24 / 44

25 Make: introduction Dependency graph In a Makefile, one defines a dependency graph instead. Dependency graph example for hello depends on Target: hello Source: hello.c Command: gcc 25 / 44

26 Make: introduction Dependency graph Dependency graph example for lab 3 tabulatesin.c depends on tabulatesin figure.png depends on data.txt depends on depends on plofile.py 26 / 44

27 Make: defining the Makefile and running make The make looks for a file called Makefile in the working directory. Makefile syntax Makefiles take the form target: source [tab] command The file "target" depends on file "source" The program "command" is used to generate file "target" The tab is necessary! Running make In a terminal, running make target builds target by executing command. 27 / 44

28 Make: hello example Dependency graph Target: hello depends on Source: hello.c Command: gcc Makefile Save the following Makefile to the directory containing hello.c. hello: hello.c gcc -Wall -ansi -pedantic -o hello hello.c 28 / 44

29 Make: hello example Building hello make hello gcc -Wall -ansi -pedantic -o hello hello.c Note: make prints the commands it runs to the screen. If we run make again, we get make: hello is up to date. 29 / 44

30 Make: Makefiles with multiple sources and targets Makefiles can declare multiple targets and sources: target1: source1 source2 command1a command1b target2: target1 command2 Important: the targets can be defined in any order! Make builds the dependency graph from the Makefile and only builds the necessary files. Example Building "target2" with "make target2" will automatically build "target1" if it is missing it is out of date ("source1" or "source2" have changed). 30 / 44

31 Make: lab3 example Dependency graph tabulatesin.c figure.png depends on depends on tabulatesin data.txt depends on depends on plofile.py 31 / 44

32 Make: lab3 example Makefile figure.png: plotfile.py data.txt python plotfile.py data.txt data.txt: tabulatesin./tabulatesin > data.txt tabulatesin: tabulatesin.c gcc -Wall -ansi -pedantic -o tabulatesin tabulatesin.c clean: rm data.txt tabulatesin plotfile.py plotfile.py: wget tools/plotfile2.py mv plotfile2.py plotfile.py 32 / 44

33 Make: lab3 example Build figure.png Copy plotfile.py and tabulatesin.c to the working directory. Modify plotfile.py so that it saves the figuret make clean make figure.png rm data.txt tabulatesin plotfile.py wget gcc -Wall -ansi -pedantic -o tabulatesin tabulatesin.c./tabulatesin > data.txt python plotfile.py data.txt Data in data.txt has 10 rows and 2 columns 33 / 44

34 Make: lab3 example Figure 34 / 44

35 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 35 / 44

36 Conclusions Use shell scripts to automate repetitive tasks Use make and Makefiles for to manage complexity only runs the steps that are necessary useful to build complex programs useful for reproducible science 36 / 44

37 References Bash programming HOW-TO: Blog post on make for scientists: Reproducible papers with make: GNU Make manual: O Reilly (open) book on Make: 37 / 44

38 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 38 / 44

39 Appendix A: running scripts with a shebang To having to call the appropriate program (bash, python, zsh etc) everytime, one can include a shebang as the first line of the script. For bash, the shebang is of the form: #!/usr/bin/env bash One can also provide the path to the executable #!/home/user/ss1234/apps/anaconda/bin/python For security reasons, you must also run the following command to make the script executable: chmod u+x feeg6002pdf.sh The script can then by run with./feeg6002pdf.sh The./ in not needed if the script is in one of the folders listed in the PATH environmental variable (display it with echo $PATH). 39 / 44

40 Appendix B: storing command output into a variable Storing command output into a variable with $() You can store the output of a command in a variable. For example the following command prints the value of π. python -c "import math; print math.pi" You can assign the output to a variable with $(): PI=$(python -c "import math; print math.pi") echo $PI / 44

41 Appendix C: Makefile for simple C program This Makefile can be used to compile a C program called myprogram.c into an executable myprogram, with the appropriate flags and compiler. Change the value of TARGET to customize the file to your needs. TARGET = myprogram # name of program CC = gcc # compiler CFLAGS = -g -Wall # flags all: $(TARGET) $(TARGET): $(TARGET).c $(CC) $(CFLAGS) -o $(TARGET) $(TARGET).c clean: $(RM) $(TARGET) This complex Makefile compiles all.c files in a directory. 41 / 44

42 Outline Learning outcomes Introduction Elementary Shell scripts Variables Command line arguments Make files Conclusions Appendix Questionmark Perception Exam 42 / 44

43 Questionmark Perception Exam Watch isolution s introductory video General information about Questionmark Perception 43 / 44

44 How to revise See Test section on FEEG6002 website. Topics that will not be examined Cython, Weave, Ctypes, IPython, SymPy, Numerical Methods. C part Review your own C programs from each lab. Make sure you remember the syntax. You should be able to read/write/debug short programs. Applied Programming Review each lecture / tutorial Focus on the learning outcomes: these are the things I expect you to be able to do Are you able to predict the output of each code block? Would you be able to write such code blocks yourself? 44 / 44