UNIX for Dummies Questions & Answers

I’m trying to create a shell script to get the server stats of 100 servers and load the details into a table.

Initially I’m creating a parameter file which has the list of all the servers, then I ‘m connecting these servers through ssh and run df –k. ssh keys are already setup.

Issues I’m facing is that I’m not able to associate server name to the result, I want server name added as a column to the df –k output.
Also the output format cannot be loaded into a table as there is no delimiter or tab or space properly formatted to load. I have tried sed & various other options but no luck.

Code:

#!/bin/ksh
PARMFILE=/opt/sdw/scripts/db_scripts/server_stats.txt
 
value=$(<server_list1.txt)
echo "$value"
sourceservers=`grep =/opt/sdw/scripts/db_scripts/server_stats.txt |cut -d= -f2`
 
#Input array passed as parameter file to the script
set -A array_value $value
vLen=${#array_value[@]}
echo $vLen
 
for(( j=0; j<$vLen; j++))
do
#echo "${array_value[$j]}"
#ssh -q "${array_value[$j]}"; df -k
ssh -q "${array_value[$j]}" 'df -h' >> df.out
ssh -q "${array_value[$j]}" df -h | column -t  >> df1.out
ssh -q "${array_value[$j]}" df -k | tr -s " " | sed 's/ /, /g' | sed '1 s/, / /g' | column -t >> df3.out
 [[ ! $? = 0 ]] && echo Failure, errno $?, cannot connect to host "${array_value[$j]}" >> sshfailed.list
done

Output

Code:

Filesystem                                             Size  Used  Avail  Use%
Mounted                  on
/dev/mapper/vg00-lvol3                                 1.5G  434M  923M   32%

Desired output

Code:

Filesystem,                                Size,       Used ,  Avail,    Use%,   Mounted  on, Servername
/dev/mapper/vg00-lvol3,      1.5G,     434M,  923M,   32%,      /                        br724

First off: i am more or less on my vacation and writing this on a Windows-machine, lacking a *NIX-system to test my words against. Please bear with me that i will give you untested instead of working code therefore and forgive me some errors or typos my failing memory and lack of testing equipment might produce.

Another thing: you haven't mentioned the system you are working on. Guessing from the output of "df" you are on some Linux-system, but then this guess might as well be wrong. I will try to accomodate a Linux-system with the following until you say otherwise.

Let us start with the way you read the server names and process them. It is not wrong to read the names into an array like you do but it is not a very "shell-like" way either. It is inherently dangerous too, because, depending on the version of the Korn shell you are using, your arrays might be (too) limited in possible array elements. ksh88 (the default shell on AIX systems, for instance) allows only for 1024 array elements, so if your server list grows your array might be unable to hold it. Set up a small loop fed by the server list as input file. I have added a little error handling, because it is good style to let a script state the reason it can't go on instead of just just breaking.

Further, i have added a little refinement to your input file with the list of server names. It pays to have the possibility to add comments to such a file as a way to document the reason why some entries are there or quick-remove one by "outcommenting" it:

[code]

Code:

#!/bin/ksh
PARMFILE=/opt/sdw/scripts/db_scripts/server_stats.txt
sourceservers=`grep =/opt/sdw/scripts/db_scripts/server_stats.txt |cut -d= -f2`          # i do not understand what this does

typeset fInput=""
typeset chServer=""
 
if [ ! -r "$1" ] ; then            # exit if input file with server list is not readable
     print -u2 "ERROR: cannot read input file $1, exiting."
     exit 2
fi
fInput="$1"

sed 's/#.*$//;s/^ *//;s/ *$//;/^$/d' "$fInput" |\
while read chServer ; do
     print - "processing $chServer"
done

exit 0

Call this script like: ./script /path/to/server_list_file, where the input file could look like this. Comment rules are like in scripts:

Code:

# server list file version 1
# last change: <today> by: <me>
# -------------------------------------------------------------

server1.network.com      # the initial one
server2.network.com      # the second one

# these added by request of X. Y.
server3.network.com
server4.network.com
# server5.network.com # do not need that one

If this produces the desired result (a list of the server names) we are going to replace the "print"-statement with a more elaborate processing. You said you will have exchanged the keys prior to running this script but it is better not to depend on this. Add a little error-handling for the case that the communication between the two systems is failing somehow - broken network, system down, not exchanged keys, whatever. When you query 100 systems almost every time the one or other will not respond for one or the other reason. The way you wrote it:

Code:

ssh -q "${array_value[$j]}" df -k | tr -s " " | sed 's/ /, /g' | sed '1 s/, / /g' | column -t >> df3.out
 [[ ! $? = 0 ]] && echo Failure, errno $?, cannot connect to host "${array_value[$j]}" >> sshfailed.list

would not work. First, because you are testing against the exit code of a pipeline (which is the exit code of the last command), not that of "ssh". (I doubt you have set the "-o pipefail" setting in "ksh"). Further, because the special variable "$?" is set anew upon completion of every command. After (successfully) executing "[[ ! $? = 0 ]]" it will be set to "0" and therefore "..errno $?, cannot..." will hold no usable data, even if it would have held it before (which is unlikely).

This is version two of the script. In case you might wonder about my variable naming: i am using sort-of hungarian style notation prefixes to indicate the type - "ch" for "char" (string), "i" for "integer", "f" for "file" (a path name), etc., to keep track of my variable types. I suggest to read up upon "print -u<n>" in the man page for ksh to understand the following, as it is (alas!) not common knowledge:

[code]

Code:

#!/bin/ksh
PARMFILE=/opt/sdw/scripts/db_scripts/server_stats.txt
sourceservers=`grep =/opt/sdw/scripts/db_scripts/server_stats.txt |cut -d= -f2`          # i do not understand what this does

typeset fInput=""
typeset chServer=""
typeset fLog="/path/to/some/logfile"

if [ ! -r "$1" ] ; then            # exit if input file with server list is not readable
     print -u2 "ERROR: cannot read input file $1, exiting."
     exit 2
fi
fInput="$1"
exec 3>"$fLog"                   # set up logging to logfile

sed 's/#.*$//;s/^ *//;s/ *$//;/^$/d' "$fInput" |\
while read chServer ; do
     if ping -qc 1 "$chServer" ; then
          if ssh -qTo BatchMode yes "$chServer" date ; then
               ssh -qTo BatchMode yes "$chServer" 'df -kP'
          else
               print -u3 "Warning: $chServer refuses ssh-communication."
          fi
     else
          print -u3 "Warning: $chServer cannot be reached via network."
     fi 
done

exec 3>&-                        # close the logfile
exit 0

Notice that i got rid of the "-h" flag of "df", as it produces the line breaks which make parsing the output harder. Instead i added the "-P" flag for POSIX-compatibility, so that output over different systems are better comparable and parsing the output is made easier.

Right now we dump the output of our command to the screen unrefined. As soon as you are satisfied with the results we give it some finishing touches by pocessing it a bit. We remove the headers (we need them only once if at all) and put the host name in front. This way it is easily possible to sort for hosts or grep for lines from one host and the resulting output can easily be parsed. Script version 3:

[code]

Code:

#!/bin/ksh
PARMFILE=/opt/sdw/scripts/db_scripts/server_stats.txt
sourceservers=`grep =/opt/sdw/scripts/db_scripts/server_stats.txt |cut -d= -f2`          # i do not understand what this does

typeset fInput=""
typeset chServer=""
typeset fLog="/path/to/some/logfile"

if [ ! -r "$1" ] ; then            # exit if input file with server list is not readable
     print -u2 "ERROR: cannot read input file $1, exiting."
     exit 2
fi
fInput="$1"
exec 3>"$fLog"                   # set up logging to logfile

sed 's/#.*$//;s/^ *//;s/ *$//;/^$/d' "$fInput" |\
while read chServer ; do
     if ping -qc 1 "$chServer" ; then
          if ssh -qTo BatchMode yes "$chServer" date ; then
               ssh -qTo BatchMode yes "$chServer" 'df -kP' |\
               sed '1d; s/^/'"$chServer"': /'
          else
               print -u3 "Warning: $chServer refuses ssh-communication."
          fi
     else
          print -u3 "Warning: $chServer cannot be reached via network."
     fi 
done

exec 3>&-                        # close the logfile
exit 0

Once satisfied call the script with

Code:

./script /path/to/server.list > /path/to/outfile

to save the output to a file.

It is easily possible to add more functionality to the script: suggestions would include a configurable option file and better command line handling. See the "getopts" man page for details. The server list file format could be refined to include certain user accounts to use, etc.. Also the error reporting is sketchy at best. I prefer a layered approach with two logfiles (severe errors and warnings) in my scripts and log start- and end-times of runs, process-IDs, etc., so that it is easy to find out which script instance has started to go wrong and why.

Anyway, i hope to have shed some light on how to improve your script and that you find my comments useful.

I hope this helps.

bakunin

Thank you, you helped me to understand and improve the way script should be written