Useless Use of * Slide 1. Useless Use of * Jan Schaumann jschauma@netmeister.org PGP: 136D 027F DC B42 47D6 7C5B 64AF AF22 6A4C

Transcription

1 Useless Use of * Slide 1 Useless Use of * jschauma@netmeister.org PGP: 136D 027F DC B42 47D6 7C5B 64AF AF22 6A4C

2 whoami Useless Use of * Slide 2 $ ME=$(id -un) $ grep ${ME} /etc/passwd cut -d: -f5 $

3 whoami Useless Use of * Slide 3 $ ME=$(id -un) $ grep ${ME} /etc/passwd cut -d: -f5 $ groups ${ME} netbsd sa yahoo $

9 Useless Use of... what? Useless Use of * Slide 9 Back in the day...

11 Useless Use of... what? Useless Use of * Slide 11 Back in the day... The Operator

12 Useless Use of... what? Useless Use of * Slide 12 Back in the day... BOFH

17 Useless Use of... what? Useless Use of * Slide 17 comp.unix.shell

18 Useless Use of * Slide 18 Useless Use of... what? I have a bunch of text files that contain strings containing /u. I no longer want them to contain /u ex: /u/appl/ should be /appl/... Should I use awk? sed? Help!

19 Useless Use of * Slide 19 Useless Use of... what? > I have a bunch of text files that contain strings > containing /u. I no longer want them to contain > /u ex: /u/appl/ should be /appl/... > > Should I use awk? sed? Help! cat file sed -e "s!/u/!/!"

20 Useless Use of * Slide 20 Useless Use of Cat

21 Useful Use of Cat (?) Useless Use of * Slide 21 The obvious: cat file grep pattern cat file awk { print $2; }

22 Useless Use of Cat Useless Use of * Slide 22 The obvious: cat file grep pattern cat file awk { print $2; }

23 Useless Use of Cat Useless Use of * Slide 23 The obvious: cat file grep pattern grep pattern file cat file awk { print $2; } awk { print $2; } < file

24 Useful Use of Cat (?) Useless Use of * Slide 24 cat file1 file2 file3 wc -l cat file1 file2 file3 wc -w

25 Useless Use of Cat Useless Use of * Slide 25 cat file1 file2 file3 wc -l cat file1 file2 file3 wc -w

26 Useless Use of Cat Useless Use of * Slide 26 cat file1 file2 file3 wc -l awk END { print NR } file1 file2 file3 cat file1 file2 file3 wc -w awk {w = w + NF} END { print w } file1 file2 file3

27 Useful Use of Cat (?) Useless Use of * Slide 27 cat file1 file2 file3 grep pattern if [ $(cat files grep -c pattern) -gt 0 ]; then

28 Useless Use of Cat Useless Use of * Slide 28 cat file1 file2 file3 grep pattern if [ $(cat files grep -c pattern) -gt 0 ]; then

29 Useless Use of Cat Useless Use of * Slide 29 cat file1 file2 file3 grep pattern grep -h pattern file1 file2 file3 if [ $(cat files grep -c pattern) -gt 0 ]; then

30 Useless Use of Cat Useless Use of * Slide 30 cat file1 file2 file3 grep pattern grep -h pattern file1 file2 file3 awk /pattern/ { print } file1 file2 file3 if [ $(cat files grep -c pattern) -gt 0 ]; then

31 Useless Use of Cat Useless Use of * Slide 31 cat file1 file2 file3 grep pattern grep -h pattern file1 file2 file3 awk /pattern/ { print } file1 file2 file3 if [ $(cat files grep -c pattern) -gt 0 ]; then if [ -n $(grep -l pattern files) ]; then

32 Useless Use of Cat Useless Use of * Slide 32 cat file1 file2 file3 grep pattern grep -h pattern file1 file2 file3 awk /pattern/ { print } file1 file2 file3 if [ $(cat files grep -c pattern) -gt 0 ]; then if [ -n "$(grep -l pattern files)" ]; then if grep pattern files >/dev/null 2>&1; then

33 Useful Use of Cat Useless Use of * Slide 33 concatenate and print files

34 Useful Use of Cat Useless Use of * Slide 34 concatenate and print files (D oh!) cat * > file

35 Useful Use of Cat Useless Use of * Slide 35 concatenate and print files (D oh!) cat * > file feed file as input to another command in a particular order

36 Useful Use of Cat Useless Use of * Slide 36 concatenate and print files (D oh!) cat * > file feed file as input to another command in a particular order { echo $VAR1; cat file; cmd1; } command

37 Useful Use of Cat Useless Use of * Slide 37 concatenate and print files (D oh!) cat * > file feed file as input to another command in a particular order { echo $VAR1; cat file; cmd1; } command use as a NOOP

38 Useful Use of Cat Useless Use of * Slide 38 concatenate and print files (D oh!) cat * > file feed file as input to another command in a particular order { echo $VAR1; cat file; cmd1; } command use as a NOOP if condition; then cmd1 cmd2 cmd3 else cmd1 cmd3 fi

39 Useful Use of Cat Useless Use of * Slide 39 concatenate and print files (D oh!) cat * > file feed file as input to another command in a particular order { echo $VAR1; cat file; cmd1; } command use as a NOOP if condition; then filter=cmd2 else filter=cat fi cmd1 ${filter} cmd3

40 Useful Use of Cat Useless Use of * Slide 40 CAT5 CAT6

41 Why bother? Useless Use of * Slide 41

42 Why bother? Useless Use of * Slide 42 $2.95

51 But... Useless Use of * Slide 51

52 But... Useless Use of * Slide I run my scripts only once!

53 But... Useless Use of * Slide I have super fast hardware!

54 But... Useless Use of * Slide I have super fast hardware! $ uptime 10:36AM up 254 days, 4 users, load averages: 80.12, 75.51, $

55 But... Useless Use of * Slide nobody else but me uses my code!

56 Oh, really? Useless Use of * Slide 56 Unfortunately, you really have no control over your code:

57 Oh, really? Useless Use of * Slide 57 Unfortunately, you really have no control over your code: code grows

58 Oh, really? Useless Use of * Slide 58 Unfortunately, you really have no control over your code: code grows code is reused

59 Oh, really? Useless Use of * Slide 59 Unfortunately, you really have no control over your code: code grows code is reused code moves with you

60 Oh, really? Useless Use of * Slide 60 Unfortunately, you really have no control over your code: code grows code is reused code moves with you code is left behind

61 Oh, really? Useless Use of * Slide 61 Unfortunately, you really have no control over your code: code grows code is reused code moves with you code is left behind It s alive!

62 Oh, really? Useless Use of * Slide 62

63 Oh, really? Useless Use of * Slide 63

64 Useless Use of * Useless Use of * Slide 64 Useless Use of Cat

65 Useless Use of * Useless Use of * Slide 65 Useless Use of *

66 Useless Use of * Slide 66 Useless Use of Grep

67 Useless Use of Grep Useless Use of * Slide 67 Of course, all use of grep(1) is useless!

68 Useless Use of Grep Useless Use of * Slide 68 Of course, all use of grep(1) is useless! echo g/re/p ed -s file

69 Useful Use of Grep (?) Useless Use of * Slide 69 host hostname grep has address awk { print $NF } echo ${string} grep pattern sed -e s/foo/bar/

70 Useless Use of Grep Useless Use of * Slide 70 host hostname grep has address awk { print $NF } echo ${string} grep pattern sed -e s/foo/bar/

71 Useless Use of Grep Useless Use of * Slide 71 host hostname grep has address awk { print $NF } host hostname awk /has address/ { print $NF } echo ${string} grep pattern sed -e s/foo/bar/

72 Useless Use of Grep Useless Use of * Slide 72 host hostname grep has address awk { print $NF } host hostname awk /has address/ { print $NF } echo ${string} grep pattern sed -e s/foo/bar/ echo ${string} sed -ne /pattern/ s/foo/bar/p

73 Useful Use of Grep (?) Useless Use of * Slide 73 grep pattern1 file... grep -v pattern2 grep pattern1 file grep -v ^# grep -v pattern2

74 Useless Use of Grep Useless Use of * Slide 74 grep pattern1 file... grep -v pattern2 grep pattern1 file grep -v ^# grep -v pattern2

75 Useless Use of Grep Useless Use of * Slide 75 grep pattern1 file... grep -v pattern2 awk /pattern1/ &&!/pattern2/ { print } file... grep pattern1 file grep -v ^# grep -v pattern2

76 Useless Use of Grep Useless Use of * Slide 76 grep pattern1 file... grep -v pattern2 awk /pattern1/ &&!/pattern2/ { print } file... grep pattern1 file grep -v ^# grep -v pattern2 awk /pattern1/ &&!/(ˆ#) (pattern2)/ { print } file

77 Useful Use of Sed (?) Useless Use of * Slide 77 for p in $(echo ${PATH} sed -e s/:/ / ); do done ls ${p}

78 Useless Use of Sed Useless Use of * Slide 78 for p in $(echo ${PATH} sed -e s/:/ / ); do done ls ${p}

79 Useless Use of Sed Useless Use of * Slide 79 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do done ls ${p}

80 Useless Use of Sed Useless Use of * Slide 80 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Sed

81 Useless Use of Sed Useless Use of * Slide 81 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Mknod?

82 Useless Use of Sed Useless Use of * Slide 82 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Groff?

83 Useless Use of Sed Useless Use of * Slide 83 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Shlock?

84 Useless Use of Sed Useless Use of * Slide 84 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Route?

85 Useless Use of Sed Useless Use of * Slide 85 for p in $(echo ${PATH} sed -e s/:/ / ); do IFS=":"; for p in ${PATH}; do ls ${p} done awk BEGIN { srand() } { if (NR == int(rand()*100)) { print }} Dump?

86 Useless Use of Sed Useless Use of * Slide 86

87 Useful Use of Sed (?) Useless Use of * Slide 87 VAR1="foo-bar-baz" VAR2=$(echo ${VAR1} sed -e s/-baz// )

88 Useless Use of Sed Useless Use of * Slide 88 VAR1="foo-bar-baz" VAR2=$(echo ${VAR1} sed -e s/-baz// )

89 Useless Use of Sed Useless Use of * Slide 89 VAR1="foo-bar-baz" VAR2=$(echo ${VAR1} sed -e s/-baz// ) VAR2=${VAR1%-baz}

90 Useful Use of Sed (?) Useless Use of * Slide 90 Looping over values in a variable: VAR1="foo-bar-baz" for i in $(echo ${VAR1} sed -e s/-/ /g ); do the_needful $i done

91 Useless Use of Sed Useless Use of * Slide 91 Looping over values in a variable: VAR1="foo-bar-baz" for i in $(echo ${VAR1} sed -e s/-/ /g ); do IFS=for i in ${VAR1}; do the_needful $i done

92 Useful Use of Sed (?) Useless Use of * Slide 92 Assigning variables: VAR1="foo-bar-baz" VAR_A=$(echo ${VAR} sed -e s/-*// ) VAR_B=$(echo ${VAR} sed -e s/[^-]*-$[^-]*$-.*/\1/ ) VAR_C=$(echo ${VAR} sed -e s/.*-// )

93 Useless Use of Sed Useless Use of * Slide 93 Assigning variables: VAR1="foo-bar-baz" VAR A=$(echo ${VAR} sed -e s/-*// ) VAR B=$(echo ${VAR} sed -e s/[^-]*-[ˆ ]-.*/\1/ ) VAR C=$(echo ${VAR} sed -e s/.*-// ) IFS=set -- ${VAR1} VAR_A="${1}" VAR_B="${2}" VAR_C="${3}"

94 Useless Use of... Useless Use of * Slide 94 Dude, IFS + shell is Teh Roxor!!!1 Look, Ma: Reading a CSV with Shell Only! IFS="," while read -r field1 waste field3 field4 waste; do echo "${field1}: ${field4} --> ${field3}" done <file

95 Useless Use of Shell Only Useless Use of * Slide 95 $ cat >script.sh IFS="," while read -r waste field2 field3 waste; do echo "${field1}: ${field4} --> ${field3}" done <file ^D $ wc -l file file $ time sh script.sh >/dev/null 12.33s real 3.30s user 8.54s system $

96 Useless Use of Shell Only Useless Use of * Slide 96 Awk to the rescue! $ cat >script2.sh awk -F"," { print $1 ": " $4 " --> " $3 } <file ^D $ time sh script2.sh >/dev/null 1.08s real 1.08s user 0.00s system $

97 Useful Use of Ls (?) Useless Use of * Slide 97 for file in $(ls *pattern*); do done the_needful

98 Useless Use of Ls Useless Use of * Slide 98 for file in $(ls *pattern*); do for file in *pattern*; do done the_needful

99 Useful Use of Wc (?) Useless Use of * Slide 99 VAR=$(cat file wc -l sed -e s/ *//g)

100 Useful Use of Wc (?) Useless Use of * Slide 100 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g)

101 Useless Use of...? Useless Use of * Slide 101 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g)

102 Useless Use of...? Useless Use of * Slide 102 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g)

103 Useless Use of...? Useless Use of * Slide 103 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g) VAR=$(awk END { print NR } file)

104 Useless Use of Sed Useless Use of * Slide 104 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g) VAR=$(wc -l <file)

105 Useless Use of Sed Useless Use of * Slide 105 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g) VAR=$(wc -l <file) echo "${VAR}"

106 Useless Use of Sed Useless Use of * Slide 106 VAR=$(cat file wc -l sed -e s/ *//g ) VAR=$(wc -l <file sed -e s/ *//g) VAR=$(wc -l <file) echo "${VAR}" echo ${VAR}

107 Useful Use of Head (?) Useless Use of * Slide 107 command1 head -1 sed -e s/pattern/string/ command1 head -10 sed -e s/pattern/string/

108 Useless Use of Head Useless Use of * Slide 108 command1 head -1 sed -e s/pattern/string/ command1 sed -e s/pattern/string/;q command1 head -10 sed -e s/pattern/string/

109 Useless Use of Head Useless Use of * Slide 109 command1 head -1 sed -e s/pattern/string/ command1 sed -e s/pattern/string/;q command1 head -10 sed -e s/pattern/string/ command1 awk { if (NR <= 10) { print gensub("pattern","string",0); } }

110 Useful Use of Tail (?) Useless Use of * Slide 110 command1 tail -1 sed -e s/pattern/string/

111 Useless Use of Tail Useless Use of * Slide 111 command1 tail -1 sed -e s/pattern/string/ command1 awk END { print gensub("pattern","string",0); exit; }

112 Useful Use of Tail (?) Useless Use of * Slide 112 command1 tail -10 sed -e s/pattern/string/

113 Useful Use of Tail Useless Use of * Slide 113 command1 tail -10 sed -e s/pattern/string/

114 Useful Use of Expr (?) Useless Use of * Slide 114 echo $(expr $i + $i)

115 Useless Use of Expr Useless Use of * Slide 115 echo $(expr $i + $i) echo $(( $i + $i ))

116 Useless Use of Expr Useless Use of * Slide 116 echo $(expr $i + $i) echo $(( $i + $i )) echo $(( $i << 1 )) This even lets you do binary manipulation (binary and, or, xor, not, leftshift, rightshift).

117 Useless Use of Expr Useless Use of * Slide 117 echo $(expr $i + $i) echo $(( $i + $i )) echo $(( $i << 1 )) This even lets you do binary manipulation (binary and, or, xor, not, leftshift, rightshift). $ x=5 $ y=12 $ x=$(( ${x} ^ ${y} )) $ y=$(( ${x} ^ ${y} )) $ x=$(( ${x} ^ ${y} )) $ echo "x=${x}; y=${y}" x=12; y=5 $

118 Most Egregiously Useless Use of Perl Useless Use of * Slide 118 perl -e "print \"y\\nn\\n\";" cmd

119 Most Egregiously Useless Use of Perl Useless Use of * Slide 119 perl -e "print \"y\\nn\\n\";" cmd ( echo y; echo n; ) cmd

120 Most Egregiously Useless Use of Perl Useless Use of * Slide 120 perl -e "print \"y\\nn\\n\";" cmd ( echo y; echo n; ) cmd { echo y; echo n; } cmd printf "y\nn\n" cmd

121 Shell Coding: Performance Useless Use of * Slide 121 avoid file I/O avoid Useless Use of *: use builtins when you can avoid builtins when it makes sense use the right tool for the job avoid spawning subshells (+1 fork) or pipes (n+1 forks)

122 Shell Coding: Performance Useless Use of * Slide 122 avoid file I/O avoid Useless Use of *: use builtins when you can avoid builtins when it makes sense use the right tool for the job avoid spawning subshells (+1 fork) or pipes (n+1 forks) cut -f2 vs. awk {print $2} $ stat -f "%z %N" /usr/bin/awk /usr/bin/cut /usr/bin/awk /usr/bin/cut $

123 Testing Shell Code Useless Use of * Slide 123 make your code modular clearly define each function: pre-conditions valid input post-conditions prepare valid input and desired output of each function prepare invalid input and desired output of each function

124 Shell Coding Style Useless Use of * Slide 124 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;)

125 Shell Coding Style Useless Use of * Slide 125 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) $ cat >script.sh set -u echo ${FOO} echo "done" ^D $ sh script.sh script.sh: FOO: parameter not set $ echo $? 2 $

126 Shell Coding Style Useless Use of * Slide 126 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) $ cat >script.sh set -e ls /nowhere echo "done" ^D $ sh script.sh ls: /nowhere: No such file or directory $

127 Shell Coding Style Useless Use of * Slide 127 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) $ cat >script.sh set -e if! ls /nowhere; then echo "ls failed" fi echo "done" ^D $ sh script.sh ls: /nowhere: No such file or directory ls failed done $

128 Shell Coding Style Useless Use of * Slide 128 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables $ cat >script.sh readonly VAR="foo" VAR="bar" echo ${VAR} ^D $ sh script.sh script.sh: VAR: is read only $

129 Shell Coding Style Useless Use of * Slide 129 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables

130 Shell Coding Style: local variables Useless Use of * Slide 130 $ cat s.sh func () { input="${1}" # do something with ${input} } $ cat script.sh../s.sh input="var1 var2 var3" for var in ${input}; do func "${var}" done echo ${input} $ sh script.sh var3 $

131 Shell Coding Style: local variables Useless Use of * Slide 131 $ cat s.sh func () { local input="${1}" # do something with ${input} } $ cat script.sh../s.sh input="var1 var2 var3" for var in ${input}; do func "${var}" done echo ${input} $ sh script.sh var1 var2 var3 $

132 Shell Coding Style Useless Use of * Slide 132 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code

133 Shell Coding Style Useless Use of * Slide 133 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code Bad: # this adds 1 to num num=$(( ${num} + 1 ))

134 Shell Coding Style Useless Use of * Slide 134 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code Better: input="pair1l:pair1r pair2l:pair2r" # extract first pair in input string, throw away rest p1l=${input%%:*} rest=${input#*:} p1r=${rest% *}

135 Shell Coding Style Useless Use of * Slide 135 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code write your code in a modular way

136 Shell Coding Style Useless Use of * Slide 136 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code write your code in a modular way write test cases for your shell code

137 Shell Coding Style Useless Use of * Slide 137 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code write your code in a modular way write test cases for your shell code be consistent in your style

138 Shell Coding Style Useless Use of * Slide 138 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code write your code in a modular way write test cases for your shell code be consistent in your style be consistent in your user interface

139 Shell Coding Style Useless Use of * Slide 139 Writing Shell Code is just like writing any other code: set -eu (perl-mongers: think -w and use strict;) use readonly variables use local variables comment your code write your code in a modular way write test cases for your shell code be consistent in your style be consistent in your user interface be willing to sacrifice performance for readability

140 Shell Coding: Performance vs. Readability Useless Use of * Slide 140 awk -F: -v ME="${ME}" { if ($0 ~ ME) { print $5 }} file vs. grep ${ME} file cut -d: -f5 awk {w = w + NF} END { print w } file1 file2 file3 vs. cat file1 file2 file2 wc -w