Slow performance filtering file Post: 302495437

Sponsored Content

Top Forums Shell Programming and Scripting Slow performance filtering file Post 302495437 by Miila on Thursday 10th of February 2011 08:50:33 AM

02-10-2011

Registered User

Slow performance filtering file

Please, I need help tuning my script. It works but it's too slow.
The code reads an acivity log file with 50.000 - 100.000 lines and filters error messages from it. The data in the actlog file look similar to this:

Code:

02/08/2011 00:25:01,ANR2034E QUERY MOUNT: No match found using this criteria. (SESSION: 211433)
or
02/08/2011 01:05:50,"ANE4991I (Session: 211857, Node: xxx) blablabla (SESSION: 211857)"

I was trying to do the same with awk but it seems it's over my head. Any ideas will be appreciated. Thanks!

here is the code:

Code:

while read LINE
do
    A=$(echo ${LINE} | cut -c 21)
    if [[ ${A} == "\"" ]];then  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 22-24)
        MSGNUM=$(echo ${LINE} | cut -c 25-28)
        MSGTYPE=$(echo ${LINE} | cut -c 29)
        MESSAGE=$(echo ${LINE} | cut -c 30-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    else  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 21-23)
        MSGNUM=$(echo ${LINE} | cut -c 24-27)
        MSGTYPE=$(echo ${LINE} | cut -c 28)
        MESSAGE=$(echo ${LINE} | cut -c 29-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    fi
done <${ACTLOGFILE}

Miila

View Public Profile for Miila

Find all posts by Miila

8 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo

2. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other...

3. Filesystems, Disks and Memory

Slow Copy(CP) performance

Hi all We have got issues with copying a 2.6 GB file from one folder to another folder. Well, this is not the first issue we are having on the box currently, i will try to explain everything we have done from the past 2 days. We got a message 2 days back saying that our Production is 98%...

4. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we...

5. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%....

6. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system...

7. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are...

8. Filesystems, Disks and Memory

Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system -...

LEARN ABOUT MOJAVE

ualarm

UALARM(3)						   BSD Library Functions Manual 						 UALARM(3)

NAME

     ualarm -- schedule signal after specified time

LIBRARY

     Standard C Library (libc, -lc)

SYNOPSIS

     #include <unistd.h>

     useconds_t
     ualarm(useconds_t useconds, useconds_t interval);

DESCRIPTION

     This is a simplified interface to setitimer(2).

     The ualarm() function waits a count of useconds before asserting the terminating signal SIGALRM.  System activity or time used in processing
     the call may cause a slight delay.

     If the interval argument is non-zero, the SIGALRM signal will be sent to the process every interval microseconds after the timer expires
     (e.g., after useconds number of microseconds have passed).

     Due to a setitimer(2) restriction, the maximum number of useconds and interval is limited to 100,000,000,000,000 (in case this value fits in
     the unsigned integer).

RETURN VALUES

     When the signal has successfully been caught, ualarm() returns the amount of time left on the clock.

NOTES

     A microsecond is 0.000001 seconds.

SEE ALSO

     getitimer(2), setitimer(2), sigpause(2), sigvec(2), alarm(3), signal(3), sleep(3), usleep(3)

HISTORY

     The ualarm() function appeared in 4.3BSD.

BSD
								  April 19, 1994							       BSD