Shell Programming and Scripting

Hello,

I am working on a Perl script to run from an AIX Nim server to extract hardware errors from our Linux server using various forms of grep statements -

I have all my hosts in a text file and a command is run from the nim server to each host to pull out data from /var/log/messages and compile it into an email.

I seem to be strruggling with the following line:

Code:

my $errRpt = `ssh $hostname "grep -s -e '$shortDate' -e '$shortDate2' /var/log/messages|egrep -v ssh|egrep -v cron|egrep -v 'su:'|egrep -v warn|egrep -v segfault|egrep -v success|egrep -v syslog|egrep -v passwd|egrep -v usersense|egrep -v -i patrol|egrep -v -m 10 multipath`;

in that I am getting thousands of multipath io messages and only want to print it in my report if it occurs once.

e.g.

Code:

Mar  6 00:00:09 sap7000 multipathd: sdd: tur checker reports path is down
Mar  6 00:00:09 sap7000 multipathd: 360060e80100aec80053026d00000000e: switch to path group #1
Mar  6 00:00:14 sap7000 multipathd: sde: tur checker reports path is down
Mar  6 00:00:14 sap7000 multipathd: sdd: tur checker reports path is down
Mar  6 00:00:14 sap7000 multipathd: 360060e80100aec80053026d00000000e: switch to path group #1
Mar  6 00:00:19 sap7000 multipathd: sde: tur checker reports path is down
Mar  6 00:00:19 sap7000 multipathd: sdd: tur checker reports path is down
Mar  6 00:00:19 sap7000 multipathd: 360060e80100aec80053026d00000000e: switch to path group #1
Mar  6 00:00:24 sap7000 multipathd: sde: tur checker reports path is down
Mar  6 00:00:24 sap7000 multipathd: sdd: tur checker reports path is down

Is there a way I can do this through using egrep/grep? I have been using '-m' option but it doesn't seem to be helping.

i just want my script to produce a report from each server's messages file for those hardware faults but not filling up my mailbox.

Hope that makes sense.

Thanks
Neil.

Don't take this too harshly, but:

Code:

# Parts that are perl
# Parts that are shell

my $errRpt = `ssh $hostname "grep -s -e '$shortDate' -e '$shortDate2' /var/log/messages|egrep -v ssh|egrep -v cron|egrep -v 'su:'|egrep -v warn|egrep -v segfault|egrep -v success|egrep -v syslog|egrep -v passwd|egrep -v usersense|egrep -v -i patrol|egrep -v -m 10 multipath`;

...fully 93% of this is shell code -- literally running in a shell, since perl creates whole, actual shells every time you use a set of backticks. In a large enough program you could be wasting dozens of shell instances.

I, too, used to write perl code like this. Eventually I threw away the perl, since it did nothing but hold together 19 different shells.

Code:

# Line counts of programs of identical function and logic
$ wc -l perl-backup.pl backup.sh
  49 perl-backup.pl
  25 backup.sh

$

Anyway. About your logic, you can also bundle those handfuls of egrep -v's into a single

Code:

egrep -v 'thing1|thing2|thing3|thing4|thing5|...|thing9

I don't think the -m option makes much sense when using -v.

...but grep doesn't have if-then logic. It's not a language. It can't make decisions later about things that happen now.

awk is a language, which can do absolutely all of that:

Code:

nawk -v SD1="$SHORTDATE1" -v SD2="$SHORTDATE2" '
# Ignore things that don't match SD1 and SD2
!((SD1 ~ \$0) || (SD2 ~ \$0)) { next }
# Ignore things that match these strings
/ssh|cronsu:|warn|segfault|success|syslog|passwd|usersense|[Pp][Aa][Tt][Rr][Oo][Ll]/ { next }

/multipathd/ {
        STR=\$0
        # Blank the date fields for easier comparison
        for(N=1; N<=5; N++) \$N=""
        # Skip messages that've happened before
        if(U[\$0]++) next;
        # Put it back
        \$0=STR
}

1' /var/log/messages

All the red backslashes are junk you need to do this in perl, because it will turn awk's $0/$1/etc into its own variables without it. They're not necessary in shell.

Hello,

Thank you for coming back to me and I inherited this perl script to gather our errors onto our Nim server. I will look at combining the egreps together as you suggest and -m option doesnot make any sense so have removed that from my code and will look at your awk code and see if I can adapt it to get what I need.

Thank you and it is much appreciated.

The awk code should be a nearly-exact replacement for what you have, except that it won't repeat messages.

Code:

Mar  6 00:00:09 sap7000 multipathd: sdd: tur checker reports path is down

It ought to look at the highlighted part of the line and not repeat multipath messages with the same content.

You should try awk on linux, most other systems use nawk

Hello,

Thanks for your help so far. I've tried putting the awk script into my Perl routine but I'm not sure if I'm missing something so please forgive my ignorance. This is my Perl code:

#my $errRpt = `ssh $hostname "grep -s -e '$shortDate' -e '$shortDate2' /var/log/messages|egrep -v 'ssh|cr
on|warn|segfault|success|syslog|passwd|usersense|command|patrol' \n"`;
my $errRpt = `ssh $hostname "awk -v SD1='$shortDate' -v SD2='$shortDate2' '
# Ignore things that don't match SD1 and SD2
!((SD1 ~ $0) || (SD2 ~ $0)) { next }
# Ignore things that match these strings
/ssh|cron|warn|segfault|success|syslog|passwd|usersense|[Pp][Aa][Tt][Rr][Oo][Ll]/ { next }
/multipathd/ {
STR=$0
# Blank the date fields for easier comparison
for(N=1; N<=5; N++) $N=""
# Skip messages that've happened before
if(U[$0]++) next;
# Put it back
$0=STR
}
1' /var/log/messages "`;
$errRpt =~ s/$hostname //g;
if ( $errRpt ne "" ) {
$message .= "###############################################\n";
$message .= "## Host $hostname\n";
$message .= "###############################################\n";
$message .= "TIMESTAMP DESCRIPTION\n";
$message .= $errRpt;
$message .= "\n";
}


and running it produces:

# ./test.pl
Global symbol "$N" requires explicit package name at ./test.pl line 74.
Execution of ./test.pl aborted due to compilation errors.
#

If I define $N within the script I still get errors:

# ./test.pl
bash: -c: line 2: syntax error near unexpected token `{'
bash: -c: line 2: `!((SD1 ~ ./test.pl) || (SD2 ~ ./test.pl)) { next }'
#


I am running the script on my AIX nim server which is to ssh onto the various Linux servers running a single command in it's shell - in this case the awk statement.

Can you advise please?

Thanks,
Neil.

---------- Post updated at 06:04 AM ---------- Previous update was at 04:47 AM ----------

Sorry, this is my test within the script:

#my $errRpt = `ssh $hostname "grep -s -e '$shortDate' -e '$shortDate2' /var/log/messages|egrep -v 'ssh|cr
on|warn|segfault|success|syslog|passwd|usersense|command|patrol' \n"`;
my $errRpt = `ssh $hostname "awk -v SD1='$shortDate' -v SD2='$shortDate2' '
# Ignore things that don't match SD1 and SD2
!((SD1 ~ \$0) || (SD2 ~ \$0)) { next }
# Ignore things that match these strings
/ssh|cron|warn|segfault|success|syslog|passwd|usersense|[Pp][Aa][Tt][Rr][Oo][Ll]/ { next }
/multipathd/ {
STR=\$0
# Blank the date fields for easier comparison
for(N=1; N<=5; N++) \$N=""
# Skip messages that've happened before
if(U[\$0]++) next;
# Put it back
\$0=STR
}
1' /var/log/messages "`;

And this is what I get when I run it so not sure if I've got the awk syntax correct within the Perl script.

# ./test.pl
bash: -c: line 2: syntax error near unexpected token `{'
bash: -c: line 2: `!((SD1 ~ sh) || (SD2 ~ sh)) { next }'
#


Thanks
Neil.