I've not read the whole thing, but assuming that you are working through the file, line by line and this design is actually reading the whole file each and every time round your loop, calling 10 processes to split up your line, how about this construct instead:-
Okay, so this is sh / ksh / bash, but it is far neater and has far lower overheads. I've not got a true csh available and the manual page I have uses all sorts of bash phrasing so it is only imitating some csh scripting so I can't really test anything I write in csh.
You might be better with this csh mangle instead of calling awk all over the place:-
This reads the file once, and for each line it splits up the record to the variables you want without calling external commands (except the cat that I can't find a way to remove)
Overall though, it is worth the effort to convert to sh based scripts. I hope your code is not riddled with goto statements like I have had to decipher before. That poor programming can leave serious headaches in re-designing.
I hope that this helps,
Robin
These 2 Users Gave Thanks to rbatte1 For This Post:
looking for solution to replicate 1.5GB files to a remote location...
Currently, this process looks like the following:
move 1.5GB files into a staging area.
compress files.
rsync files to remote server.
remove compressed files.
I have performed some timings, and compress seems more... (5 Replies)
Hi,
I have one file stat.
Stat file contents are as follows: for example.
H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000
H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000
Now i... (1 Reply)
Hi,
I am trying to send oracle archives over WAN and it is taking hell a lot of time. To reduce the time, I tried to gzip the files and send over to the other side. That seems to reduce the time. Does anybody have experienced this kind of problem and any possible ways to reduce the time.
... (1 Reply)
We have an egrep search in a while loop.
egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new
${PICKUP_DIR}/new_update is 210 MB file
In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
HI Experts ,
I'm using the following code to remove spaces appearing at the end of the file.
sed "s/*$//g" <filename> > <new_filename>
mv <new_filename> <filename>
this is working fine for volumes upto 20-25 GB.
for the bigger files it is taking more time that it is required... (5 Replies)
Hello everyone,
suppose there is a very big text file (>800 mb) that each line contains an article from wikipedia. Each article begins with a tag (<..>) containing its url. Currently there are 10^6 articles in the file.
I want to take random N articles, eliminate all non-alpharithmetic... (14 Replies)
Hello, I've been searching for something that slows down a process for some time now. Slow down as in make time pass by slower. I have rarely turned to asking a forum in the past but at this point I've given up.
For example: if I made a program that would print "Hello" in 5 seconds, I would use... (18 Replies)
Hi,
I have a HP-UX server were I need to list all the files in the entire file system, their directory path, last modified date, owner and group. I do not need to search the file contents. I created the script given below and I am excluding directories and files of type tmp, temp and log. The... (4 Replies)
I have "inherited" a OmniOS (illumos based) server.
I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware.
Using same hardware, same command with same source and target disks, OmniOS r151026 gives:
test@omniosce:~# time... (11 Replies)
Hello,
For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)