Shell Programming and Scripting

Thanks!!
well, the actual text file has 59850 records where I have to pick every 3325 records and output to different files!! So, i have to place some number around 18 which still gives me the same error code " too many output files" !!
Can you please help me more? Also, what if i have more than one field in the input file? will a for loop inside AWk structure do?

Actually, I've just run this with 600,000 records using the original awk

Code:

awk '{print > "file" (NR-1)%3+1}' infile

and it did not complain, which in hindsight makes sense since it's only opening three files.

And with 10000 files:

Code:

 awk '{print > "file" (NR-1)%10000+1}' infile

No problem there either.

You can have as many fields as you like, it writes the whole record to the files.

Which OS / awk version are you using?

Here a version that works without running into the open file limit. It's an awk script and should be in a file:

Code:

BEGIN {
        print RF_CNT;
        print LN;
}

(NR <= RF_CNT) {                # output first records
                a[NR] = "file" NR;
                print >a[NR];
                close(a[NR]);   # expensive but stays under fopen limit
}

((i = NR%LN +1) in a) {         # see if this line is a candidate
                print >>a[i];   # add it
                close(a[i]);    # close it to stay under fopen limit
}

Use this command line for it:

Code:

awk -f num.awk -v RF_CNT=20 -v LN=2000 num.txt

The value of RF_CNT is the number of files to make. The first line of each file will contain line N of the input file (ie. file 4 contains line 4 of the input file).

The value of LN is the line offset/modulo to determine if line need to go to a file and which file.

num.txt is input file. I used a file with the numbers 1-8000 on each line.

It opens and closes the output files to avoid the open file limits...

The BEGIN clause is just to confirm inputs and can be removed....

Dear Scottn,

I switched to SUN systems and the code 'kinda' worked. I mean, the numer of output files were correct but I had only one line in each output file. For the test case, I had one input file with 40950 records and I wanted multipliations of 2275 to be printed in each file which should result in 40950/2275=18 lines in each file.
In the last file, file2275, i had the last line of input file, for the file2274 i had 40949th line of the input file printed and so on to file1!!!!
I do not know what's wrong and am really confused.
The code works for small number of modulos like up to 10 but doesn't like big numbers!!

JP2542a

I tried your code but it gives me an error on the line begining with (( i = NR....). syntax error and bailing out near this line!!

Thank you so much

We'll get there!

Use /usr/xpg4/bin/awk on Solaris. I say this every time the word Solaris crops up. Gives me time to think of something more useful to say!!

I will try it on Solaris tomorrow.

Cheers

I ran the script on both CentOS and cygwin and it worked. What exactly did you type and what was the exact output?

Dear Scottn,

Surprisingly, when i tried your first code on SUN, i.e.

Code:

awk '{print > "file" (NR-1)%3+1}' infile

it worked pretty well!! whereas when i tried the code with close (out) the outputs were garbage!
So, if i have 10 fields and i want to have each field processed seperatly, what should i do? By this I mean if 2000 files is generated from one column I would like to end up with 2000*10 files. Does a for loop
(for i=1;i<=FN;i++) works? if so, where in the code?

Dear JP2542a,
The following is the code i've used. The bug shows up when i try it on Solaris machines while when i run it on SUNs it doesn't complain but it doesn't generate any files. Don't bother yourself if it's sth wierd coz i've been succefully run Sottn's code.

I should very much appreciate your help
I'll probably get back here with some other questions.

Cheers

Code:

  BEGIN {print RF_CNT;print LN;}(NR <= RF_CNT) { a[NR] = "file" NR;print >a[NR];close(a[NR]);}(( i = NR %LN + 1 ) in a ) {print >> a[i];close(a[i]);}