Visit Our UNIX and Linux User Community


Script optimalisation


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Script optimalisation
# 1  
Old 07-30-2009
Script optimalisation

Hi,

I have a script that basically interegates a large file (833594152), extracts data and creates an number of files with data that fits within certain criteria. An example of the scripts is below:


Code:
for number in `cat linenum`

do

    if [ $counter = 1 ]

         then NO=$number
              counter=`expr $counter + 1`

    elif [ $counter != 1 ]

         then marker=$number
              end=`expr $number - 1`

              filename=exfile$counter

              awk -v NO=$NO -v end=$end 'NR == NO, NR == end' $file1 > $filename

              filename2=compfile$counter

              grep "WORD1 WORD2" $filename > $filename2

                   if [ ! -s $filename2 ]

                       then rm $filename
                            rm $filename2

                       else rm $filename2

                   fi

              NO=$marker
              counter=`expr $counter + 1`
    fi
done



The problem is that it takes a long time to run (i'm estimating atleast 6 years, probably more!!), I know that it is quite "clunky" scripting, but does anyone know how I can possibly streamline this. It seems that the AWK statement is the biggest user of CPU both in time and percentage.

I'm presuming it's the file size that's the problem, but any help would be appreciated.

Thanks

Chris[COLOR="#738fbf"]

---------- Post updated at 03:52 PM ---------- Previous update was at 03:31 PM ----------

Please ignore the spelling

Last edited by chris01010; 08-03-2009 at 06:01 AM.. Reason: Bad spelling
# 2  
Old 07-30-2009
You know, if you told us what the script is supposed to do it would help.

One place to save time is to lose the expr's and use shell arithmetic operators -
Code:
number=$(( $number + 1 ))
# or
number=$(( $somevar - 1 ))

are far more efficient. each execution of awk & grep eat cpu and time as well.
# 3  
Old 07-30-2009
To increase chance you might want to edit your post and insert CODE tags, this little #-button up there in the edit bar when you write/edit a post. It will enhance readability of your code, data or logs and preserve formatting like indention etc.. Please have a look into this, ty.
# 4  
Old 07-31-2009
Ok well, the first 2 lines give a description (albeit vague) of what the script is meant to do. But basically it locates the start and end of non-fixed length data messages within a large file, puts the data in a new file on it's own, checks if a certain detail is present and then either keeps the data message in the file or removes it.

I believe it is the awk statement that is taking the most time as it will have to read in the whole file everytime.

I'll look into the tags and update the post.
 

Previous Thread | Next Thread
Test Your Knowledge in Computers #343
Difficulty: Medium
Scott McNealy, Andy Bechtolsheim, and Vinod Khosla, all Caltech graduate students, founded Sun Microsystems.
True or False?

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to block first bash script until second bash script script launches web server/site?

I'm new to utilities like socat and netcat and I'm not clear if they will do what I need. I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes. I'm using Cygwin bash on Windows 10. My... (3 Replies)
Discussion started by: siegfried
3 Replies

2. Shell Programming and Scripting

Shell script works fine as a standalone script but not as part of a bigger script

Hello all, I am facing a weird issue while executing a code below - #!/bin/bash cd /wload/baot/home/baotasa0/sandboxes_finance/ext_ukba_bde/pset sh UKBA_publish.sh UKBA 28082015 3 if then echo "Param file conversion for all the areas are completed, please check in your home directory"... (2 Replies)
Discussion started by: ektubbe
2 Replies

3. UNIX for Dummies Questions & Answers

Calling a script from master script to get value from called script

I am trying to call a script(callingscript.sh) from a master script(masterscript.sh) to get string type value from calling script to master script. I have used scripts mentioned below. #masterscript.sh ./callingscript.sh echo $fileExist #callingscript.sh echo "The script is called"... (2 Replies)
Discussion started by: Raj Roy
2 Replies

4. Shell Programming and Scripting

Script will keep checking running status of another script and also restart called script at night

I am using blow script :-- #!/bin/bash FIND=$(ps -elf | grep "snmp_trap.sh" | grep -v grep) #check snmp_trap.sh is running or not if then # echo "process found" exit 0; else echo "process not found" exec /home/Ketan_r /snmp_trap.sh 2>&1 & disown -h ... (1 Reply)
Discussion started by: ketanraut
1 Replies

5. Shell Programming and Scripting

create a shell script that calls another script and and an awk script

Hi guys I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file. #!/bin/sh echo " $2 $3 $4 $5 $6 $7 isql -w400 -U$2 -S$5 -P$3 << xxx use $4 go print"**Changes to the table... (0 Replies)
Discussion started by: magikminox
0 Replies

Featured Tech Videos