Shell Programming and Scripting

hello all,

kindly i need your help, i made a script to print a specific lines from a huge file about 3 million line. the output of the script will be about 700,000 line...the problem is the script is too slow...it kept working for 5 days and the output was only 200,000 lines !!!

the script is so simple:

for i in `cat file` ------> file is the file that contains the line no. to be printed from a file.


sed '$i q;d' file1 > file2 ----> where file1 is the huge file 3 millions lines and file 2 is the output file which will be 700.000 lines

so plz could anyone tell me how can i decrease the processing time of that script and why is it taking all that time !!!?

thanks in advance

Do you know the to and from line no of the file which you want to print??

because cat file will be really slow if file has 3 million records please avoid that.

From my experience, running a pile of small(er) files take much less time than working with a single "fatty" thus, you may want to split up both source files in junks, then process them (possibly in parallel) and finally concatenate the results.

i tried not to use the cat in for and i did with head -line no | tail -1 and it also worked but still too slow....i can't print specific range because the lines not know i run script to get the line no. which differs from file to file.
i really don't know what is the problem with it !?

---------- Post updated at 09:11 PM ---------- Previous update was at 09:09 PM ----------

dr house, how many lines per file do u think it will be good to split the fatty file?

As this is very machine-dependent, I'd start with a ten-percent split, process one (!) file, time this, then either process the remaining nine splits - or split, process and time the "guinea pig" again (thus going down to e.g. five-percent splits). You get the idea ...

Can you provide a couple sample lines from file ?
The sed command simply prints the appropriate line, right?

Code:

awk 'FNR==NR{n[$0];next}FNR in n'  file file1 > file2