Statement Sequences

The simplest form of a computer program is a sequence of statements:

cd $HOME
mkdir png
cd png
cp ../images/*.png .
rm _*.png

This script changes into your personal home-directory, creates a directory named "png", goes into it, copies all png files from $HOME/images to there, then removes all files that start with an underscore.

A shell script is a collection of commands you enter on a terminal screen. There were times when everything was done like that, and you could automate quickly just by collecting your daily commands into a script, and then let the script do the work :-)

Order Significant?

Sometimes the order of statements matters, sometimes the order could be changed. In the above example, the mkdir png command is a precondition for the following cd png command. But in the following script no command depends on some predecessor:

echo "Logged in at `date +'%Y-%m-%e %H:%M'`" >>logfile
rm -rf WASTEBASKET/*
mount /windisk
ls -la

This appends date and time to the file "logfile", cleans the wastebasket, mounts a drive and then lists the current directory. We could do these statements in any order.

What I want to make clear is that programming-languages have no means to distinguish between these two types of sequences (statement order significant or not). Which is an underestimated cause of bugs. When seeing a sequence of statements, you never know whether the programmer ordered them intentionally or not. Best is to write conditions instead of sequences when order is significant.

Pipes

A pipe is a special way to chain a sequence of commands. The output (stdout) of the leftmost command is fed into the input (stdin) of the next command, which again might produce output for the next command, and so on. That way a lot of applications can be used to process a stream of data.

ls -la | grep myfile | sed 's/png/jpg/g'

This code lists the current directory, then filters out all lines that contain "myfile", and then replaces in all those lines every occurrence of "png" by "jpg" (sed = stream editor).

Conditions

The UNIX Shell finds out whether a command was successful or not be checking its exit-code. The $? variable always holds the exit code of the latest launched command.

$ test -n hello
$ echo $?
0

This uses the test command to check whether "hello" is a non-zero (-n) string. Then it outputs the exit code of the latest command.

An exit-code of 0 (zero) indicates success, anything else is to be interpreted as error number.
(Which is exactly the opposite to most other programming languages, where if (0) would be false!)

if Conditions

So when we write sequences of commands that depend on each other, we can make this explicit by testing whether the previous command succeeded or not.

if mkdir png
then
  cp images/*.png png
else
  echo "Could not create directory png!"
fi

This looks much more robust, right? The "if" is a reserved shell keyword. The command after it gets executed, its exit code is queried, and the sequence after "then" is executed when the exit code indicated success. When not, it executes the sequence in the "else" section. Any such condition must be closed by "fi". The "else" is optional. There is also an "elif" for catenation of conditions:

if test "$1" = "me"
then
  echo "First parameter was me"
elif test "$1" = "you"
then
  echo "First parameter was you"
else
  echo "First parameter was >$1<"
fi

This condition catenation uses the test command to check the first script (or function) parameter $1for values "me" and "you", and outputs it by calling the echo command.

The test command is the most used with if-conditions, because it can do a lot of different things. And it can be written in a way that gives the feeling of a real programming language.
Following two conditions do exactly the same, but the second uses the built-in "shortcut" for the test command, which materializes as [ condition ]:

if test 2 -gt 1; then echo "test can compare numbers!"; fi

if [ 2 -gt 1 ]; then echo "test can be written as brackets!"; fi

Mind how the semicolon ; can replace newlines. This example shows that the test command also can be used to compare numbers.

You can combine several conditions within the test command using the options -a (AND) or -o (OR). Following script outputs when the variable count is greater than 0 (-gt) and less equal 10 (-le).

count=4
if [ -n "$count" -a "$count" -gt 0 -a "$count" -le 10 ]
then
  echo "Not empty and between 1 and 10: $count"
fi

You can also use the operators && (AND) or || (OR) outside the test command to build composite conditions:

count=-1
if test -n "$count" && ( test "$count" -lt 0 || test "$count" -gt 10 )
then
  echo "Not empty and less than 0 or greater than 10: $count"
fi

This script uses ( parentheses ) to group conditions together, because && binds stronger than ||.
As you might suspect, using the brackets and parentheses together can result in quite cryptic expressions, so try to stay "human readable":-)

one-armed Conditions

Following kind of condition I call "one-armed". It does not have "else" or "elif".

pwd || echo "Print Working Directory failed!"

The || demands that the part to the right is executed only when the left condition failed. Reads as "print current directory or echo error". So when the pwd command failed (which is not very likely:-), this outputs an error message.

grep "you" logfile && echo "Found you in logfile!"

As expected, the && demands that the part to the right is executed only when the left condition succeeded. Reads as "when finding 'you' in logfile, echo success". The grep command is a text search application that understands regular expressions.

The one-armed condition is not restricted to just one follower command. You can enclose any number of follower commands into braces:

grep "$pattern" logfile >/dev/null && {
  echo "Found $pattern in logfile!"
  exit 3
}

This checks whether a pattern (given as shell variable) is to be found in logfile. The output redirection ">/dev/null" causes the output of the grep command to be dismissed. When the pattern was found, the block within the braces gets executed, and the script exits with code 3.

case Conditions

What is switch-case in most programming languages, is a pattern-matching case keyword in shell.

case "$parameter" in
  no\ version)
    echo "Not a version: $parameter"
    ;;
  *[Dd]ocumentation* | [Ss]pecification* )
    echo "Is documentation or specification: $parameter"
    ;;
  *)
    echo "Unknown: $parameter"
    ;;
esac

The case gives you the opportunity to test a variable or comand output against different patterns that can contain a reduced set of regular expressions. The above case tests for the value "no version", for some text containing "documentation" or "specification" (whereby the first letter could be capitalized), and outputs "Unknown" in any other case.

Loops

To find things we have to look at a lot of instances and compare them to some criteria. Or we want to print any element of a collection. We might want to wait until a condition gets true. There are a lot of use cases for a loop. But mind that normally loops are the part of an application that consumes the most performance.

for Loops

The shell has no C-like for (int i = 0; i < limit; i++). Instead you have a for-in loop.

todos="
  a
  b
  c"
for todo in $todos
do
  echo $todo
done

The shell would split the $todos array by spaces or newlines, and output every word of it in the loop.
To have a loop count you would need to write:

count=0
for todo in x y z
do
  echo "$count: $todo"
  count=`expr \$count + 1`
done

The basic shell can not do arithmetics. But the expr command can. The above script defines a count variable and uses expr in a so-called "command substitution" to increment it in every loop pass (see chapter about procedure calls).

while Loops

A conditional loop is a merge of loop and condition. The while loop repeats as long as its condition is true.

while read line
do
  echo $line
done <.profile

This loop is fed by lines from the file .profile that normally resides in $HOME directory. It outputs any line from that file. Mind that the input redirection has to be written after the loop end, not after the read command in first line.

while read input && [ "$input" != "OVER" ]
do
  echo $input
done

This script will output any line you write on the keyboard, it reads from stdin. You can terminate the loop by typing OVER, which makes the second condition false. Or you can type Ctrl-d, this is the UNIX end-of-input key code, and it will make read exit with a code different from 0, which will make the first condition false.

For your pleasure there is also an until-loop:

count=0
until [ "$count" -gt 10 ]
do
  count=`expr \$count + 1`
  echo $count
done

Procedures

Now that we can write sequences, conditions and loops, we can write a computer program. But we want to reuse other programs in it, and maybe we want to nest our program into an even bigger program.

Script Calls

When our ready-made shell script is in some directory that is in our execution PATH (echo $PATH), any other script should be able to call our script. A convenient way to do this is to have a $HOME/bin directory where all the scripts are, and to have that in $PATH (can be done in .profile or .bashrc). Do not forget to set the script executable: chmod 754 scriptfile.sh.

With Parameters

What we need to know is how scripts can receive the information they need to do their work (parameters).

if [ -z "$1" -o -z "$2" -o -z "$3" ]
then
  echo "Not enough parameters"
  exit 1
fi
echo "Starting to work ..."

This script checks its command line arguments whether they are non-empty and whether there are three of them. If less than three parameters were passed to it, or when one of them is empty, the script terminates with exit code 1. Store that script in a file checkarg.sh, call chmod 754 checkarg.sh, and then test it:

./checkarg.sh
Not enough parameters
$ ./checkarg.sh 1
Not enough parameters
$ ./checkarg.sh 1 2
Not enough parameters
$ ./checkarg.sh 1 2 3
Starting to work ...

With Environment

The second way to receive information is via the environment. The calling script can do the following:

JAVA_HOME=$HOME/jdk18
export JAVA_HOME

The difference between shell variables and environment variables is that environment variables are already exported, while shell variables need to be explicitly exported to be present in a called script.

The called script that needs a pre-set JAVA_HOME environment variable should do the following:

[ -z "$JAVA_HOME" ] && {
  echo "Need JAVA_HOME to work!"
  exit 1
}

On a UNIX shell you can list the environment using printenv or env.

Functions

A way to structure script code is to use functions. The word function is a reserved shell keyword. Functions can have parameters, but they do not declare it. There are three different ways to define functions:

function foo {
  echo "foo: $1"
}

bar() {
  echo "bar: $1"
}

function foobar () {
  echo "foobar: $1 $2"
}

foo Hello
bar World
foobar Hello World

Functions receive their parameters like a script, but they do not see thier script's top-level paremeters.
The above script outputs:

foo: Hello
bar: World
foobar: Hello World

You can write "library scripts" that contain just functions and no main section. When you want to use these functions in some script, you can read the functions using the source command, better known as . command.

Store the foo(), bar() and foobar() functions above in script called foobar.sh (remove their calls on bottom). Then create a script called libtest.sh where you write the following:

source foobar.sh    # kind of import

foo Hello
bar World
foobar Hello World

The foobar.sh library must be in PATH, or in same directory as the calling script.

Command Substitutions

This is a smart procedure call mechanism to quickly receive and use the output of some command.

oldDir=`pwd`
doSomeUncalculableWork
cd $oldDir   # change to directory having been before

The command enclosed in `backquotes` will be executed by a subshell, and its output pasted to where the backquotes have been. In modern shells, the backquotes could also be replaced by $(pwd).

In the script above, the output of pwd will be the current working directory. The script remembers this directory in variable oldDir, then executes some function that can change directory to wherever it pleases, the caller then will safely return to where it has been before. (Mind that this will fail when doSomeUncalculableWork also uses a variable oldDir, because all shell variables are global!)

case "`uname`" in
  CYGWIN*)
    echo "This is s special operating system"
    ;;
  *)
    echo "We are on some UNIX system"
    ;;
esac

The above code calls the uname command that outputs the operating system. The output then is tested for being on UNIX or a CYGWIN system.

Warning

You might already have noticed that interpreted script languages are not suitable for bigger projects. They are made for small tasks, and to do it quick and dirty.

The UNIX shell is no exception, there is no data-typing, there are no encapsulation mechanisms, every variable you declare is global. A big problem is to keep control over the working directory when using the cd command (change directory), especially when using functions that change the directory.

Shell scripts are small and efficient, but developing them takes some time and nerves. Their escaping mechanism and quote-precedences must be studied thoroughly. Every shell command has its own set of regular expressions. What you can apply in a case you can not use with grep, what egrep understands is not known to sed. Batch editors like ed and ex are very useful, but developing their commands always is an extra task.

Nevertheless scripts can be very useful replacements for big administration applications. Remember that make had the possibility to define shell commands. So use it, and enjoy your scripts!