I have a script that basically interegates a large file (833594152), extracts data and creates an number of files with data that fits within certain criteria. An example of the scripts is below:
Code:
for number in `cat linenum`
do
if [ $counter = 1 ]
then NO=$number
counter=`expr $counter + 1`
elif [ $counter != 1 ]
then marker=$number
end=`expr $number - 1`
filename=exfile$counter
awk -v NO=$NO -v end=$end 'NR == NO, NR == end' $file1 > $filename
filename2=compfile$counter
grep "WORD1 WORD2" $filename > $filename2
if [ ! -s $filename2 ]
then rm $filename
rm $filename2
else rm $filename2
fi
NO=$marker
counter=`expr $counter + 1`
fi
done
The problem is that it takes a long time to run (i'm estimating atleast 6 years, probably more!!), I know that it is quite "clunky" scripting, but does anyone know how I can possibly streamline this. It seems that the AWK statement is the biggest user of CPU both in time and percentage.
I'm presuming it's the file size that's the problem, but any help would be appreciated.
Thanks
Chris[COLOR="#738fbf"]
---------- Post updated at 03:52 PM ---------- Previous update was at 03:31 PM ----------
Please ignore the spelling
Last edited by chris01010; 08-03-2009 at 06:01 AM..
Reason: Bad spelling
To increase chance you might want to edit your post and insert CODE tags, this little #-button up there in the edit bar when you write/edit a post. It will enhance readability of your code, data or logs and preserve formatting like indention etc.. Please have a look into this, ty.
Ok well, the first 2 lines give a description (albeit vague) of what the script is meant to do. But basically it locates the start and end of non-fixed length data messages within a large file, puts the data in a new file on it's own, checks if a certain detail is present and then either keeps the data message in the file or removes it.
I believe it is the awk statement that is taking the most time as it will have to read in the whole file everytime.
I'm new to utilities like socat and netcat and I'm not clear if they will do what I need.
I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes.
I'm using Cygwin bash on Windows 10.
My... (3 Replies)
Hello all,
I am facing a weird issue while executing a code below -
#!/bin/bash
cd /wload/baot/home/baotasa0/sandboxes_finance/ext_ukba_bde/pset
sh UKBA_publish.sh UKBA 28082015 3
if
then
echo "Param file conversion for all the areas are completed, please check in your home directory"... (2 Replies)
I am trying to call a script(callingscript.sh) from a master script(masterscript.sh) to get string type value from calling script to master script. I have used scripts mentioned below.
#masterscript.sh
./callingscript.sh
echo $fileExist
#callingscript.sh
echo "The script is called"... (2 Replies)
I am using blow script :--
#!/bin/bash
FIND=$(ps -elf | grep "snmp_trap.sh" | grep -v grep) #check snmp_trap.sh is running or not
if
then
# echo "process found"
exit 0;
else
echo "process not found"
exec /home/Ketan_r /snmp_trap.sh 2>&1 & disown -h ... (1 Reply)
Hi guys
I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file.
#!/bin/sh
echo " $2 $3 $4 $5 $6 $7
isql -w400 -U$2 -S$5 -P$3 << xxx
use $4
go
print"**Changes to the table... (0 Replies)
