Sponsored Content
Top Forums Shell Programming and Scripting Slow performance filtering file Post 302495437 by Miila on Thursday 10th of February 2011 08:50:33 AM
Old 02-10-2011
Slow performance filtering file

Please, I need help tuning my script. It works but it's too slow.
The code reads an acivity log file with 50.000 - 100.000 lines and filters error messages from it. The data in the actlog file look similar to this:
Code:
02/08/2011 00:25:01,ANR2034E QUERY MOUNT: No match found using this criteria. (SESSION: 211433)
or
02/08/2011 01:05:50,"ANE4991I (Session: 211857, Node: xxx) blablabla (SESSION: 211857)"

I was trying to do the same with awk but it seems it's over my head. Any ideas will be appreciated. Thanks!

here is the code:
Code:
while read LINE
do
    A=$(echo ${LINE} | cut -c 21)
    if [[ ${A} == "\"" ]];then  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 22-24)
        MSGNUM=$(echo ${LINE} | cut -c 25-28)
        MSGTYPE=$(echo ${LINE} | cut -c 29)
        MESSAGE=$(echo ${LINE} | cut -c 30-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    else  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 21-23)
        MSGNUM=$(echo ${LINE} | cut -c 24-27)
        MSGTYPE=$(echo ${LINE} | cut -c 28)
        MESSAGE=$(echo ${LINE} | cut -c 29-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    fi
done <${ACTLOGFILE}

 

8 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo (6 Replies)
Discussion started by: Neo
6 Replies

2. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

3. Filesystems, Disks and Memory

Slow Copy(CP) performance

Hi all We have got issues with copying a 2.6 GB file from one folder to another folder. Well, this is not the first issue we are having on the box currently, i will try to explain everything we have done from the past 2 days. We got a message 2 days back saying that our Production is 98%... (3 Replies)
Discussion started by: b_sri
3 Replies

4. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we... (0 Replies)
Discussion started by: kopper
0 Replies

5. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%.... (2 Replies)
Discussion started by: draiphod
2 Replies

6. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system... (3 Replies)
Discussion started by: susindram
3 Replies

7. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are... (7 Replies)
Discussion started by: christr
7 Replies

8. Filesystems, Disks and Memory

Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system -... (3 Replies)
Discussion started by: ces55
3 Replies
CUT-DIFF(1)							  Cutter's manual						       CUT-DIFF(1)

NAME
cut-diff - show difference between 2 files with color SYNOPSIS
cut-diff [option ...] file1 file2 DESCRIPTION
cut-diff is a diff command that uses diff feature in Cutter. It shows difference with color. It's recommended that you use a normal diff(1) when you want to use with patch(1) or you don't need color. OPTIONS
--version cut-diff shows its own version and exits. -c [yes|true|no|false|auto], --color=[yes|true|no|false|auto] If 'yes' or 'true' is specified, cut-diff uses colorized output by escape sequence. If 'no' or 'false' is specified, cut-diff never use colorized output. If 'auto' or the option is omitted, cut-diff uses colorized output if available. The default is auto. -u, --unified cut-diff uses unified diff format. --context-lines=LINES Shows diff context around LINES. All lines are shown by default. When unified diff format is used, 3 lines are shown by default. --label=LABEL, -L=LABEL Uses LABEL as a header label. The first--label option value is used as file1's label and the second --label option value is used asfile2's label. Labels are the same as file names by default. EXIT STATUS
The exit status is 0 for success, non-0 otherwise. TODO: 0 for non-difference, 1 for difference and non-0 for errors. EXAMPLE
In the following example, cut-diff shows difference between file1 and file2: % cut-diff file1 file2 In the following example, cut-diff shows difference between file1 and file2 with unified diff format: % cut-diff -u file1 file2 SEE ALSO
diff(1) Cutter February 2011 CUT-DIFF(1)
All times are GMT -4. The time now is 09:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy