I have a script that basically interegates a large file (833594152), extracts data and creates an number of files with data that fits within certain criteria. An example of the scripts is below:
The problem is that it takes a long time to run (i'm estimating atleast 6 years, probably more!!), I know that it is quite "clunky" scripting, but does anyone know how I can possibly streamline this. It seems that the AWK statement is the biggest user of CPU both in time and percentage.
I'm presuming it's the file size that's the problem, but any help would be appreciated.
---------- Post updated at 03:52 PM ---------- Previous update was at 03:31 PM ----------
Please ignore the spelling
Last edited by chris01010; 08-03-2009 at 06:01 AM..
Reason: Bad spelling
To increase chance you might want to edit your post and insert CODE tags, this little #-button up there in the edit bar when you write/edit a post. It will enhance readability of your code, data or logs and preserve formatting like indention etc.. Please have a look into this, ty.
Ok well, the first 2 lines give a description (albeit vague) of what the script is meant to do. But basically it locates the start and end of non-fixed length data messages within a large file, puts the data in a new file on it's own, checks if a certain detail is present and then either keeps the data message in the file or removes it.
I believe it is the awk statement that is taking the most time as it will have to read in the whole file everytime.
I'm new to utilities like socat and netcat and I'm not clear if they will do what I need.
I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes.
I'm using Cygwin bash on Windows 10.
My... (3 Replies)
I am facing a weird issue while executing a code below -
sh UKBA_publish.sh UKBA 28082015 3
echo "Param file conversion for all the areas are completed, please check in your home directory"... (2 Replies)
I am trying to call a script(callingscript.sh) from a master script(masterscript.sh) to get string type value from calling script to master script. I have used scripts mentioned below.
echo "The script is called"... (2 Replies)
I am using blow script :--
FIND=$(ps -elf | grep "snmp_trap.sh" | grep -v grep) #check snmp_trap.sh is running or not
# echo "process found"
echo "process not found"
exec /home/Ketan_r /snmp_trap.sh 2>&1 & disown -h ... (1 Reply)
I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file.
echo " $2 $3 $4 $5 $6 $7
isql -w400 -U$2 -S$5 -P$3 << xxx
print"**Changes to the table... (0 Replies)