Shell Programming and Scripting

Hi All,

I have made the following code which is taking 10-15 mins to execute completely. Here the file am using is very big( around 1050993843). Can any one make some changes in my code which reduces the time it takes to execute as am very much new to Unix(learner).

One more thing, i have to run this script in 4 of my App Servers. Is there any way that this script should automatically login into the all the servers and get me the final output..

I have tried with public key generation and it is not working properly... it is connecting to the second server and not coming back to the script.....
Please do help me, its very urgent..

#!/bin/ksh

fname=sample.log
server=$(hostname)
cp /dev/null /axphome/mvelmu/scriptcount.txt
check()
{
count=`cat $1 | grep -c "$2"`
if [ $count -eq 0 ]
then
echo "There is no $3 in the log file" >> /axphome/mvelmu/scriptcount.txt
else
echo "$3 = $count" >> /axphome/mvelmu/scriptcount.txt
fi
}
check2()
{
count=`cat $1 | grep -i "$2" | grep -i -c "$3"`
if [ $count -eq 0 ]
then
echo "There is no $4 in the log file" >> /axphome/mvelmu/scriptcount.txt
else
echo "$4 = $count" >> /axphome/mvelmu/scriptcount.txt
fi
}
echo "$server" >> /axphome/mvelmu/scriptcount.txt
echo "*******" >> /axphome/mvelmu/scriptcount.txt
check "$fname" "SuspendCardWebA2AServiceCall.suspendCard Entry" "Suspend card"
check "$fname" "MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.DataException" "Suspend card Data Exception"
check "$fname" "MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.BusinessSystemException" "Suspend card BusinessSystemExceptions"
check "$fname" "ApplyForACardHelper.insertStatus Entry" "Apply card"
check "$fname" "ApplyForACardFacade.getBcaProfile com.americanexpress.gcbs.exception.BusinessSystemException" "Apply card BusinessSystemException"
check "$fname" "ApplyForACardFacade.getBcaProfile com.americanexpress.gcbs.exception.UnexpectedException" "Apply card UnExpected Exceptions"
check "$fname" "ApplyForACardBusinessLogic.getBcaProfileProvider exception occured while executing the command" "Apply card BCA profile error"
check "$fname" "MaintainAccountStatusFacade.cancel card Entry" "Cancel card"
check "$fname" "MaintainAccountStatusFacade.cancel card com.americanexpress.gcbs.exception.BusinessSystemException" "Cancel card BusinessSystemException"
check "$fname" "MaintainAccountStatusFacade.cancel card com.americanexpress.gcbs.exception.UnexpectedException" "Cancel card UnExpectedException"
check2 "$fname" "ChangeLimitsServiceBean.process" "entry" "Change limits"
check2 "$fname" "ChangeLimitsBusinessLogic" "DBUnexpectedException" "Change limits DBUnexpected Exception"
check2 "$fname" "ChangeLimitsBusinessLogic" "UnexpectedException" "Change limits UnExpected Exception"
check "$fname" "ChangeCMDetailsServiceBean.validateRequest Entry" "Change CMDetails"
check2 "$fname" "ChangeCMDetailsBusinessLogic" "DBCommunicationException" "Change CMDetails DB exception"
check2 "$fname" "ChangeCMDetailsBusinessLogic" "MYCACommunicationException" "Change CMDetails MYCA exception"
check2 "$fname" "ChangeCMDetailsBusinessLogic" "MultipleExceptions" "Change CMDetails Multiple exception"
#cat /axphome/mvelmu/scriptcount.txt
cat /axphome/mvelmu/scriptcount.txt | mail -s "Script count for $server" Madhav.K.Sunduru@aexp.com,Arun.V.Shankar@aexp.com,Ramasubramanian.Z.Murugesan@aexp.com,Bini.R.Mathew @aexp.com,Rafikul.H.Sekh@aexp.com,Uma.M.Subburayan@aexp.com,Kavitha.V.Madanagopal@aexp.com


Thanks in Advance...

I guess you should not use the below line
count=`cat $1 | grep -c "$2"`
Instead use the above command with slight modification
count=`grep -c "$2" $1`

I want to mention you are using cat command unneccessary(useless use of cat).The line
cat $1 | grep -c "$2"
will read file twice first by "cat" and then by command "grep" and if your file is large it will cost a time to be read twice.

Yes agree with dhruva. Dont use pipes unless u really require it.

- nilesh

The Useless Use of Cat is not particularly slowing down the script by much; the main problem is that you are grepping the file over and over for each search string. Perhaps you could change it to grep for all the strings at the same time, then change the output into the format you want. A simple sed script should work for this.

Code:

sed -n -f - sample.log <<HERE | sort | uniq -c
s/.*SuspendCardWebA2AServiceCall.suspendCard Entry.*/Suspend card/p
s/.*MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.DataException.*/Suspend card Data Exception/p
s/.*MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.BusinessSystemException.*/Suspend card BusinessSystemExceptions/p
# etc
# and then the check2 cases
/ChangeLimitsServiceBean.process/s/.*entry.*/Change limits/p
/ChangeLimitsBusinessLogic/s.*DBUnexpectedException.*/Change limits DBUnexpected Exception/p
# etc
HERE

This is just a proof of concept, and doesn't handle the case-insensitive searching you have in the original script. Other than that, it will find matching lines, and change them into the corresponding label; then count the number of occurrences of these labels in the resulting output.

Anyway, the point is, in order to speed it up, see if you can run only one pass over the log file, and end up with a much smaller amount of data which you could process further into the report you want.

This doesn't print anything at all for fields which are not found; if you want explicit zero counts, adding that as a post-processing stage should be a fairly trivial exercise.

Perhaps you would do well to separate the string and label data from the script itself. That's also left as an exercise.

ThanQ very much for all your Quick response...

Hi Dhruva,

the command what you have suggest using only one grep is taking me more time than what i was using previously... Can you suggest me some other command which reduces exec time...

Here's a slightly more elaborate awk script which works on the same principle as the sed I posted previously, but includes the zero counts and avoids the need to sort the output. It also somewhat separates the data from the logic. Still missing is a facility to make some of the matching case-insensitive.

Code:

#!/bin/sh

awk 'NR==FNR {
    f = split ($0, a, ":");
    if (f>2) {
        c[a[3]] = a[2];
        r[a[3]] = a[1];
        i[NR] = a[3];
    }
    else {
        c[a[2]] = ".";
        r[a[2]] = a[1];
        i[NR] = a[2];
    }
    next
}
{
    for (l in r) {
        if ($0 ~ r[l] && $0 ~ c[l])
            ++n[l]
    }
}
END {
    for (j = 1; j in i; ++j) {
        if (n[i[j]])
            printf "%s = %i\n", i[j], n[i[j]];
        else
            printf "There is no %s in the log file\n", i[j];
    }
}' - $1 <<"HERE"
SuspendCardWebA2AServiceCall.suspendCard Entry:Suspend card
MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.DataException:Suspend card Data Exception
MaintainAccountStatusFacade.suspend card com.americanexpress.gcbs.exception.BusinessSystemException:Suspend card BusinessSystemExceptions
... etc ...:... etc ...
ChangeLimitsServiceBean.process:entry:Change limits
ChangeLimitsBusinessLogic:DBUnexpectedException:Change limits DBUnexpected Exception
... etc ...:... etc ...:... etc ...
HERE