Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Making script run faster

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Making script run faster
# 1  
Old 05-11-2012
Making script run faster
Can someone help me edit the below script to make it run faster?

Shell: bash
OS: Linux Red Hat

The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK".

For each chunk, the beginning starts with "service {", and ends with "}".

I should point out that the status.log file is about 25MB.

Code:
grep -Pnw "MEMORY_CHECK" /apps/status.log | while read line

do

        NUMP=$(echo $line | awk -F":" '{print $1}')

        BEGINNINGA=$NUMP

        BEGINNING=$(NUM=1

        while [ $NUM -lt 70 ]

        do

                VAL=$NUMP

                MINUSED=$(echo $VAL $NUM | awk '{print $1 - $2}')

                DEFOU=$(sed -n ${MINUSED}p /apps/status.log | grep -Pv "#" | grep -P "service {")

                if [ ! -z "$DEFOU" ] ; then

                        echo "${MINUSED}:$DEFOU" | awk -F":" '{print $1}'

                        break

                fi

                NUM=$(( $NUM + 1 ))

        done)

##############################

        END=$(NUM=1

        while [ $NUM -lt 70 ]

        do

                VAL=$NUMP

                MINUSED=$(echo $VAL $NUM | awk '{print $1 + $2}')

                DEFOU=$(sed -n ${MINUSED}p /apps/status.log | grep -Pv "#" | grep -P "}")

                if [ ! -z "$DEFOU" ] ; then

                        echo "${MINUSED}:$DEFOU" | awk -F":" '{print $1}'

                        break

                fi

                NUM=$(( $NUM + 1 ))

        done)


        echo "==================================================="
        echo ""

        BAD=$(sed -n ${BEGINNING},${END}p/apps/status.log  | egrep "CRITICAL:|UNKNOWN:")

        if [ ! -z "$BAD" ] ; then

                sed -n ${BEGINNING},${END}p /apps/status.log  | egrep "host_name=|check_command=|^plugin_output="

        fi

        echo ""
        echo "==================================================="

done

Last edited by SkySmart; 05-11-2012 at 02:29 PM..
# 2  
Old 05-11-2012
You're running grep, awk, and sed hundreds of thousands of times to process thousands of lines. I think your script may need a rewrite. You could probably do it all in one awk instance.

Can you show the input data you have, and the output you want?
# 3  
Old 05-11-2012
Quote:
Originally Posted by Corona688
You're running grep, awk, and sed hundreds of thousands of times to process thousands of lines. I think your script may need a rewrite. You could probably do it all in one awk instance.

Can you show the input data you have, and the output you want?


here's the input data. Notice the below is what i refer to as a chunk. in this particular case, the chunk below belongs to the service called "MEMORY_CHECK".

so there's a file that contains several of these chunks for several services.

lets say the file is called status.log. for processing, I want to pull out all chunks called "service_description=MEMORY_CHECK" and then pull out other attributes of each of the chunk as needed.

attributes can be, i.e what is the name of the hostname that this MEMORY_CHECK is on? in this case, the host name would be "sky.log.net".

what is the plugin output, in this case, it would be "CRITICAL: Process was not found".

hope this helps.

Code:
service {
host_name=sky.log.net
service_description=MEMORY_CHECK
modified_attributes=1
check_command=blah!blah!blah
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.453
check_latency=1.701
check_type=0
current_state=2
last_state=2
last_hard_state=2
last_event_id=2522339
current_event_id=2523376
current_problem_id=1127556
last_problem_id=0
current_attempt=1
max_attempts=1
normal_check_interval=30.000000
retry_check_interval=2.000000
state_type=1
last_state_change=1335466482
last_hard_state_change=1335466482
last_time_ok=1334123115
last_time_warning=0
last_time_unknown=1335465427
last_time_critical=1336750629
plugin_output=CRITICAL:  Process was not found
long_plugin_output=
performance_data=
last_check=1336750629
next_check=1336752429
check_options=0
notified_on_unknown=1
notified_on_warning=0
notified_on_critical=1
current_notification_number=364
current_notification_id=2530346
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=0
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

# 4  
Old 05-11-2012
Yes, and what output do you want now? "Pull out" isn't too specific, what does the data you want actually look like?
# 5  
Old 05-11-2012
Here's a start at least, which only prints service { sections containing service_description=MEMORY_CHECK:

Code:
$ cat memorycheck.awk
BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
} D["service_description"]=="MEMORY_CHECK"

$ awk -f memorycheck.awk data

service {
host_name=sky.log.net
service_description=MEMORY_CHECK
modified_attributes=1
check_command=blah!blah!blah
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.453
check_latency=1.701
check_type=0
current_state=2
last_state=2
last_hard_state=2
last_event_id=2522339
current_event_id=2523376
current_problem_id=1127556
last_problem_id=0
current_attempt=1
max_attempts=1
normal_check_interval=30.000000
retry_check_interval=2.000000
state_type=1
last_state_change=1335466482
last_hard_state_change=1335466482
last_time_ok=1334123115
last_time_warning=0
last_time_unknown=1335465427
last_time_critical=1336750629
plugin_output=CRITICAL:  Process was not found
long_plugin_output=
performance_data=
last_check=1336750629
next_check=1336752429
check_options=0
notified_on_unknown=1
notified_on_warning=0
notified_on_critical=1
current_notification_number=364
current_notification_id=2530346
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=0
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

$

If awk doesn't work for you, try nawk.

All the various fields for each section should be available inside the D array inside awk, for you to do with as you please.

If you could actually tell me what you wanted to do with them, all the better.
Last edited by Corona688; 05-11-2012 at 06:29 PM..
This User Gave Thanks to Corona688 For This Post:
SkySmart
# 6  
Old 05-11-2012
Quote:
Originally Posted by Corona688
Here's a start at least, which only prints service { sections containing service_description=MEMORY_CHECK:

Code:
$ cat memorycheck.awk
BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
} D["service_description"]=="MEMORY_CHECK"

$ awk -f memorycheck.awk data

service {
host_name=sky.log.net
service_description=MEMORY_CHECK
modified_attributes=1
check_command=blah!blah!blah
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.453
check_latency=1.701
check_type=0
current_state=2
last_state=2
last_hard_state=2
last_event_id=2522339
current_event_id=2523376
current_problem_id=1127556
last_problem_id=0
current_attempt=1
max_attempts=1
normal_check_interval=30.000000
retry_check_interval=2.000000
state_type=1
last_state_change=1335466482
last_hard_state_change=1335466482
last_time_ok=1334123115
last_time_warning=0
last_time_unknown=1335465427
last_time_critical=1336750629
plugin_output=CRITICAL:  Process was not found
long_plugin_output=
performance_data=
last_check=1336750629
next_check=1336752429
check_options=0
notified_on_unknown=1
notified_on_warning=0
notified_on_critical=1
current_notification_number=364
current_notification_id=2530346
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=0
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

$

If awk doesn't work for you, try nawk.

All the various fields for each section should be available inside the D array inside awk, for you to do with as you please.

If you could actually tell me what you wanted to do with them, all the better.
you are a genius.

just one last question, i don't want to have to run the script like this:

Code:
awk -f memorycheck.awk data

i modified the code and put it all in one script, but it's not working:

Code:
awk BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
} D["service_description"]=="MEMORY_CHECK" status.log

i get the following errors:

./script: line 1: syntax error near unexpected token `}'
./script: line 1: `awk BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }'

and also, i want to get (from each chunk), information about the "host_name=" and "plugin_output="

how do i specify that in the script? i want to be able to specify in the script to only output the MEMORY_CHECK information about so and so host.
Last edited by SkySmart; 05-11-2012 at 09:26 PM..
# 7  
Old 05-12-2012
I'm not being cranky about this - but you do realize that some of us have possibly been doing this stuff since before you were born. So it is at least remotely possible that we know what are doing, in a collective sense. I started UNIX in 1975.

Ok?

So - could you please find the time to tell us what you are trying to do - not what you think you should do to get to your goal? We need a description of your goal, devoid of your idea of how to do it. Thanks. It will help all of us in the long run.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script works, but I think it could be better and faster

Hi All, I'm new to the forum and to bash scripting. I did some stuff with VB.net, Batch, and VBScripting in the past, but because I shifted over to Linux, I am learning to script in Bash at this moment. So bear with me if I seem to script like a newbie, that's just because I am ;-) OK, I... (9 Replies)
Discussion started by: cornelvis
9 Replies

2. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1... (8 Replies)
Discussion started by: SkySmart
8 Replies

3. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

4. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each... (10 Replies)
Discussion started by: AlbertGM
10 Replies

5. Shell Programming and Scripting

Script to parse a file faster

My example file is as given below: conn=1 uid=oracle conn=2 uid=db2 conn=3 uid=oracle conn=4 uid=hash conn=5 uid=skher conn=6 uid=oracle conn=7 uid=mpalkar conn=8 uid=anarke conn=1 op=-1 msgId=-1 - fd=104 slot=104 LDAPS connection from 10.10.5.6 to 10.18.6.5 conn=2 op=-1 msgId=-1 -... (7 Replies)
Discussion started by: sags007_99
7 Replies

6. UNIX for Advanced & Expert Users

Making things run faster

I am processing some terabytes of information on a computer having 8 processors (each with 4 cores) with a 16GB RAM and 5TB hard drive implemented as a RAID. The processing doesn't seem to be blazingly fast perhaps because of the IO limitation. I am basically running a perl script to read some... (13 Replies)
Discussion started by: Legend986
13 Replies

7. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make... (10 Replies)
Discussion started by: shew01
10 Replies

8. UNIX for Advanced & Expert Users

Country Codes script faster response ;please help

Dear all I have group of input lines which look like this These input lines is placed in a file named phonelines.txt and there is a script which match $4 and $5 with country codes placed in another file named country-codes.txt and its contents is : Italy 39 Libyana 21892 Thuraya... (12 Replies)
Discussion started by: zanetti321
12 Replies

9. UNIX for Dummies Questions & Answers

making ssh run without password

Hello Everybody, Could anyone please tell me how to get ssh to work without asking for passwords? (i want to do a ssh <hostname> without getting a request for a password but getting connected straight away) I have attempted the following but to no avail :( ... I tried to generate a SSH... (5 Replies)
Discussion started by: rkap
5 Replies
Login or Register to Ask a Question