Unix Administration with Self-Parsing Shell Scripts

John Kozubik - john@kozubik.com - http://www.kozubik.com


Unix system administration commonly consists of repeating common and similar
operations on different sets of targets.  The notion of using, and stringing
together, building block primitives such as cat, sort, strings, wc, sed and awk
is a core tenet of Unix philosophy.  Sometimes jobs are too complicated to
solve with a single command line of pipelined or redirected primitives, or
perhaps the job will be repeated often enough that it makes sense to create
a more permanent script to produce the desired outcome.  The inevitable
question that arises is how to allow the script to handle a variety of
related inputs, allowing it to be useful in the future when different data
needs to be processed using the same procedures.

The obvious solution to this problem is to write a script that operates on
data files.  That way, the script can exist on its own, and be used to process
any number of new data sets as they are introduced in the future.  A good
example of this is the rc system used for system initialization in FreeBSD.
In this arrangement, a number of rc scripts (such as rc.network, rc.sendmail
and rc.diskless[1,2]) are run, who in turn each parse through a single
user-edited configuration file, /etc/rc.conf.  In this configuration, changes
to /etc/rc.conf alter the way that rc scripts run and what they perform.

This solution is a useful and elegant one, and lends itself to modular
organization and an increase in overall ease of use - especially when
implemented as well as it is in the FreeBSD rc system.  However, there
are more discreet tasks (tasks that do not fit into some larger framework,
such as system initialization) that can be made even simpler by removing
even the data files that the script operates on.  By incorporating the data
<italic>into the script itself</italic>, one can create powerful system
administration tools, in the form of simple shell scripts, that consist
merely of a single file.

The basic organization of such a script is a set of one or more data items
which are commented out, as they are not actual commands, but commented
in such a way that they can be distinguished from normal comments:


01: #!/bin/sh
02: #
03: # Here is the data set, and perhaps we will add some other comments here
04: #
05: ##DATA var1 var2 var3
06: ##DATA var1 var2 var3
07: ##DATA var1 var2 var3


As you can see, normal comments are commented out with one # character,
but data items are commented with ##.  Not only does this allow us the
ability to parse through the script and easily identify which lines are
data lines (as opposed to normal comments), but it also allows us to
quickly disable a data line that we temporarily do not wish to use.
Simply remove one of the # characters from the data line that is not
to be used - it will not be parsed because it has no leading ##, but it
still starts with #, and thus does not affect the script as it is still
commented out.

The body of the script consists of a variable defining the path of the
script itself (obviously the script needs to know the path to itself if
it is to parse itself), and then a while loop that reads in every line
of the script, but filters out (using grep) only those lines that are
data lines, which begin with ##DATA :


08: myself=/usr/local/bin/script.sh
09:
10: while read line
11: do
12:
13:    if echo $line | grep "^##DATA"
14:    then
15:
16:      var1=`echo $line | awk '{print $2}'`
17:      var2=`echo $line | awk '{print $3}'`
18:      var3=`echo $line | awk '{print $4}'`
19:
20:      diff $var1 $var2 >> $var3
21:
22:    fi
23:
24: done < $myself


What is happening here is that the script, in line 08, defines the path
to itself, and uses a "while read line" construct, the end of which
in line 24 takes as input the name of the script itself.  While the
script reads every line of itself, it selects, through the use of grep,
only the lines that begin with ##DATA.  As noted above, disabling a
particular line of data can be done by simply removing one of the #
characters, as it will still be commented out, but will not be selected
as an input line by grep.

What the script above actual does is trivial - as you can see, in lines
16,17 and 18, we simply echo each line and pipe the output through awk to
grab the second, third or fourth word in that line (since the first word
is ##ENTRY) and assign it to a variable.  Then we perform a simple
operation on the variables from that line.

Because we have three total lines of data in this script (lines 05, 06
and 07) the diff action in line 20 will be run a total of three times,
each time with a different group of data.  There is no limit to the
number of data lines we could put in the script.  It should be noted,
however, that every line in the entire script will be parsed and tested
by the grep conditional in line 13, regardless of whether it is a data
line or not.

What are some practical uses for self-parsing shell scripts such as
this ?  In June 2000, I wrote a simple, but powerful backup utility
that, three years later, I now use to back up every system in my
organization.  The script, which can be found here:

http://www.kozubik.com/software/kozubik_backup-1.0.tar.gz

Contains data lines like this:


##ENTRY /usr/local/etc www7.kozubik.com-usr-local-etc /mnt/backup1
##ENTRY /var/mail www7.kozubik.com-var-mail /mnt/backup4


The first piece of data is a local path in the filesystem.  The second
piece of data is a label for that backup, and the final piece of
data is a destination for the backup to be placed onto.

The body of this backup script is somewhat complicated, but can be
simplified as follows:

00: myself=/usr/local/etc/backup.sh
01: while read line
02: do
03:
04:   if echo $line | grep "^##ENTRY"
05:   then
06:
07:   directory=`echo $line | awk '{print $2}'`
08:   name=`echo $line | awk '{print $3}'`
09:   backupdir=`echo $line | awk '{print $4}'`
10:   date=`date '+%y-%m-%d'`
11:
12:   if test -d $backupdir
13:   then
14:     if test -d $dir
15:     then
16:       tar cvzf $backupdir/$date-$name.tar $dir
17:     else
18:       echo "Directory to back up does not exist"
19:     fi
20:   else
21:     echo "Target directory does not exist"
22:   fi
23: done < $myself


The end result is, given the two data lines above, the creation of
two gzipped tar files - one named 03-07-20-www7.kozubik.com-usr-local-etc
that is placed in /mnt/backup1, and one named 03-07-20-www7.kozubik.com-var-mail
that is placed in /mnt/backup4.  Note that different source directories
for the backup can be placed in different target directories, as that
is simply a variable defined in each data line.

Another, more complicated (but very useful) script can be found here:

http://www.kozubik.com/software/kozubik_pulse-1.0.tar.gz

This script contains data lines like this:


##QUERY www.example.com 80 "GET /index.html HTTP/1.0" "HTTP/1.1 200 OK" "responds to HTTP requests" "DOES NOT respond to HTTP requests" 2
##QUERY www.example.com 25 "QUIT" "221" "responds to SMTP requests" "DOES NOT respond to SMTP requests" 2


The first piece of data is an address.  The second piece of data is a port
number, the third piece of information is a string to send to that socket,
and the fourth piece of information is a string we expect to be returned
to us.  The fifth and sixth data items are strings to return on success or
failure, respecively.  Finally, the seventh data item is the number of tries
the script should make to query this network service before returning a
negative report.

The body of the script is more complicated, as it calls netcat (nc) to
query the services in the data lines of the script, but again, the
basic functionality can be simplified as follows:


00: myself=/usr/local/etc/pulse.sh
01: while read line
02: do
03:
04:   if echo $line | grep "^##QUERY"
05:
06:   then
07:
08:   address=`echo $line | awk '{print $3}'`
09:   port=`echo $line | awk '{print $4}'`
10:   send=`echo $line | awk -F\" '{print $2}'`
11:   receive=`echo $line | awk -F\" '{print $4}'`
12:   upmsg=`echo $line | awk -F\" '{print $6}'`
13:   downmsg=`echo $line | awk -F\" '{print $8}'`
14:   retries=`echo $line | awk -F\" '{print $9}'`
15:   count=0
16:   successes=0
17:
18:   while ( test $count -lt $retries )
19:   do
20:
21:    output=`printf "$send\r\n\r\n" | nc -v -w 3 $address $port`
22:    count=`$expr $count + 1`
23:    if echo $output | grep "$receive"
24:    then
25:    successes=`expr $successes + 1`
26:    fi
27:
28:    done
29:
30:    if test $successes -eq 0
31:    then
32:    echo "$address $downmsg on port $port" >> /tmp/kozubik.pulse.fail.$$
33:    else
34:    echo "$address $upmsg on port $port" >> /tmp/kozubik.pulse.succeed.$$
35:    fi
36:
37:  fi
38:  done < $myself


In this script, the end result is that each line in the data portion of the
script causes netcat to run with the corresponding destination and port
number, and is given a number of retries within which to receive a positive
response from the server daemon that is presumed to be running at that
remote location.  Although we do not show it here, the file that this script
outputs (either failure or success) can then be emailed to one or more email
addresses defined earlier in the script.

Please note one significant difference between the backup script shown above,
and this pulse.sh script.  Because the backup script has data lines whose
elements always consist of single words (such as /usr/local/etc) we can
simply parse the lines using awk to grab words in the data line by their number.
In the pulse.sh script, however, some of the data elements have multiple
words, and look like "responds to HTTP requests", which means when we parse
the data lines and pull out individual fields with awk, we need to specify a
field seperator and offset, like you see in lines 10-14:

	upmsg=`echo $line | awk -F\" '{print $6}'`




The original basic template, shown as the first code example above, combined
with the specifc examples of a backup script and a remote monitoring script,
show how using a self-parsing shell script can dramatically simplify common
Unix system administration tasks.  Compare the simplicity of the pulse.sh
script above, which monitors remote server daemons - which I personally use
to monitor over 800 servers across the world - to the complexity of other
remote monitoring applications which encompass complicated software
installations and utilize hundreds of program and data files.  The abilities
to quickly alter one or more data lines, disable lines (but leave them in
place) by simply removing one of the two leading # characters, and transport
the abilities of these programs in a single, self contained file represent
a significant gain in simplicity and ease of use.

I encourage the reader to download and begin using the three fully
functional examples at http://www.kozubik.com, and to begin grouping
common tasks into new self-parsing shell scripts to extend this simplicity
and ease to other tasks I have not yet written examples for.