Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Optimize shell script to run faster

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimize shell script to run faster
# 1  
Old 01-06-2015
Optimize shell script to run faster
data.file:

Code:
contact {
contact_name=royce-rolls
modified_attributes=0
modified_host_attributes=0
modified_service_attributes=0
host_notification_period=24x7
service_notification_period=24x7
last_host_notification=0
last_service_notification=0
host_notifications_enabled=1
service_notifications_enabled=1
}
servicecomment {
host_name=pgphhram01
service_description=Free Space All Disks
entry_type=4
comment_id=32
source=0
persistent=0
entry_time=1416610977
expires=0
expire_time=0
author=hpsm
comment_data=DM02782504
}
servicecomment {
host_name=pgphhram02
service_description=Free Space All Disks
entry_type=4
comment_id=32
source=0
persistent=0
entry_time=1420561982
expires=0
expire_time=0
author=hpsm
comment_data=DM02902504
}

My data.file is about 60MB big. So i need to trim it. To trim it, i need to identify which chunks are comment chunks. and when these chunks are identified, i need to check their entry time. If their entry time is older than 60 minutes from right now, i ignore that particular chunk and move on to the next chunk. in the above data.file, there are 3 chunks in it. chunks begin with a

Code:
"   {"

and ends with a

Code:
}

here is the code i'm using:

Code:
FILE=data.file

FFNUM=$(wc -l < ${FILE})

awk '{print NR","$0}' ${FILE} | egrep " {" | awk -F"," '{print $1}' | while read CLNUM
do
        NTIME=${CLNUM}

        LINENUMS=$(while [ $NTIME -le $FFNUM ]
        do
                ENDY=$(sed -n ${NTIME}p ${FILE} | egrep "^}")

                if [ ! -z "${ENDY}" ] ; then
                        echo "${CLNUM},${NTIME}"
                        break
                fi
                NTIME=$((${NTIME} + 1))
        done)

        FOUND=$(sed -n ${LINENUMS}p ${FILE})
        ISITCOMMENT=$(echo "${FOUND}"  | egrep "comment {")
        DNOW=$(date +%s)

        if [ ! -z "${ISITCOMMENT}" ] ; then

                ENTRYTIME=$(echo "${FOUND}" | egrep "entry_time" | awk -F"=" '{print $2}')

                ELAPSEDTIME=$(awk "BEGIN{print $DNOW - $ENTRYTIME}")

                if [ ${ELAPSEDTIME} -lt ${AMINUTES} ] ; then
                        echo "${FOUND}"
                fi
        else
                echo "${FOUND}"
        fi
done

This code works and does exactly what i need. however, it runs very slow. can anyone think of anyway i can augment this script so it runs faster?
# 2  
Old 01-06-2015
what's the expected output given a sample input?
# 3  
Old 01-06-2015
Quote:
Originally Posted by vgersh99
what's the expected output given a sample input?
if i updated the entry time of the third chunk with the timestamp of right now (date +%s), then, when i run this script, it should print the following:

Code:
contact {
contact_name=royce-rolls
modified_attributes=0
modified_host_attributes=0
modified_service_attributes=0
host_notification_period=24x7
service_notification_period=24x7
last_host_notification=0
last_service_notification=0
host_notifications_enabled=1
service_notifications_enabled=1
}
servicecomment {
host_name=pgphhram02
service_description=Free Space All Disks
entry_type=4
comment_id=32
source=0
persistent=0
entry_time=1420567480
expires=0
expire_time=0
author=hpsm
comment_data=DM02902504
}

the chunk i updated with the timestamp of right now is:

Code:
servicecomment {
host_name=pgphhram02
service_description=Free Space All Disks
entry_type=4
comment_id=32
source=0
persistent=0
entry_time=1420567480
expires=0
expire_time=0
author=hpsm
comment_data=DM02902504
}

# 4  
Old 01-06-2015
try this: awk -v now="$(date +%s)" -f sky.awk myFile.txt
where sky.awk is:
Code:
BEGIN {
  FS="="
  if (!now) now=systime()

  tDiff=60*60
  p=1
}
/{/ {rec=$0;p=1;next}
/}/ && rec && p {print rec ORS $0;next}
$1=="entry_time" {
  if (now-$2>tDiff)p=0
}
{rec=rec ORS $0}

This User Gave Thanks to vgersh99 For This Post:
SkySmart
# 5  
Old 01-06-2015
The final while read loop should be replaced by more advanced sed. You can read a whole xxx{ ... } into the buffer using 'N' and test it.
Last edited by DGPickett; 01-06-2015 at 04:03 PM..
# 6  
Old 01-06-2015
Quote:
Originally Posted by vgersh99
try this: awk -v now="$(date +%s)" -f sky.awk myFile.txt
where sky.awk is:
Code:
BEGIN {
  FS="="
  if (!now) now=systime()

  tDiff=60*60
  p=1
}
/{/ {rec=$0;p=1;next}
/}/ && rec && p {print rec ORS $0;next}
$1=="entry_time" {
  if (now-$2>tDiff)p=0
}
{rec=rec ORS $0}

thanks a million!
any way to put this into one script so i can run script like this:

Code:
./script  data.file  60m

# 7  
Old 01-06-2015
Quote:
Originally Posted by SkySmart
thanks a million!
any way to put this into one script so i can run script like this:

Code:
./script  data.file  60m

why not:
myScript.sh data.file 3600
myScript.sh
Code:
#!/bin/sh
#
awk -v now="$(date +%s)" -v tDiff="${2}" '
   BEGIN {   
      FS="=" 
      if (!now) now=systime()    
      if (!tDiff) tDiff=60*60
      p=1 
  } 
   /{/ {rec=$0;p=1;next} 
   /}/ && rec && p {print rec ORS $0;next} 
   $1=="entry_time" { if (now-$2>tDiff)p=0 } 
   {rec=rec ORS $0}' "${1}"

you can elaborate on the parameter passing on your own....
This User Gave Thanks to vgersh99 For This Post:
SkySmart
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script run in a case statement call to run a php file, also Perl

Linux System having all Perl, Python, PHP (and Ruby) installed From a Shell script, can call a Perl, Python, PHP (or Ruby ?) file eg eg a Shell script run in a case statement call to run a php file, also Perl or/and Python file??? Like #!/usr/bin/bash .... .... case $INPUT_STRING... (1 Reply)
Discussion started by: hoyanet
1 Replies

2. UNIX for Dummies Questions & Answers

optimize if block : shell script

Hi, I need a shell script to determine if a no. is either even, greater than 4, less than 8 SHELL : ksh OS : RHEL 6 this is the if block of the script mod=`expr $num % 2` if || || then echo "No. is either even or greater than 4 or less than 8" fi this code works... (2 Replies)
Discussion started by: sam05121988
2 Replies

3. Shell Programming and Scripting

Making script run faster

Can someone help me edit the below script to make it run faster? Shell: bash OS: Linux Red Hat The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK". For each chunk, the beginning starts with "service {", and ends with "}". I should... (15 Replies)
Discussion started by: SkySmart
15 Replies

4. Shell Programming and Scripting

Optimize shell code

#!/usr/bin/perl use strict; use warnings; use Date::Manip; my $date_converted = UnixDate(ParseDate("3 days ago"),"%e/%h/%Y"); open FILE,">$ARGV"; while(<DATA>){ my @tab_delimited_array = split(/\t/,$_); $tab_delimited_array =~ s/^\ =~ s/^\-//; my $converted_date =... (2 Replies)
Discussion started by: sandy1028
2 Replies

5. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

I've a script to do some snapshots but the time it does so is very different... once i got a snapshot under 1 sec, on the other hand it took 3 sec, but nothing else changed, i didnt even move the cursor or something. I put the script on a ramdisk and its faster, but still swing from under 1... (1 Reply)
Discussion started by: mcW
1 Replies

6. UNIX for Advanced & Expert Users

Making things run faster

I am processing some terabytes of information on a computer having 8 processors (each with 4 cores) with a 16GB RAM and 5TB hard drive implemented as a RAID. The processing doesn't seem to be blazingly fast perhaps because of the IO limitation. I am basically running a perl script to read some... (13 Replies)
Discussion started by: Legend986
13 Replies

7. Shell Programming and Scripting

Help need to make a shell script run for ffmpeg vhook watermaking in shell

i have a small problem getting a batxh shell script to run in shell this is the code the problem seems to be centered around the ffmpeg command, something maybe to do with the ' ' wrapping around the vhook part command this is a strange problem , if i take the ffmpeg command and... (1 Reply)
Discussion started by: wingchun22
1 Replies

8. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make... (10 Replies)
Discussion started by: shew01
10 Replies
Login or Register to Ask a Question