Shell Programming and Scripting

Hi

I have a large file 2.6 million records and I am trying to split the file based on last column.

I am doing

awk -F"|" '{ print > $NF }' filename1

After around 1000 splits it gives me a error
awk: can't open file 3332332423
input record number 1068, file filename1
source line number 1

For different input files it gives error in different line numbers. I expect to create around 50,000 files. Is there any limitation?

Adarsh

What do you mean split the file based on the last column? You want to split the file into many files?

Yes

I have records like

a|b|b|c|c|dsda|ada|123
a|b1|b|c1|c|dsda|ada|223
a|b1|b|c1|c|dsda|ada|223
a|b|b|c|c|dsda|ada|423
a|b|b|c|c|dsda|ada|1234
a|b|b|c|c|dsda|ada|123

I require

123
a|b|b|c|c|dsda|ada|123
a|b|b|c|c|dsda|ada|123

223
a|b1|b|c1|c|dsda|ada|223
a|b1|b|c1|c|dsda|ada|223

1234
a|b|b|c|c|dsda|ada|1234

423
a|b|b|c|c|dsda|ada|423


Actually awk -F"|" '{ print > $NF }' filename1 is doing the job. My input file has 2.6 million records. It created some 1000 files and then giving a strange error

Please advice me

Adarsh

Code:

awk -F"|" '{ print > $NF ; close($NF)}' filename1

Thanks vgersh99
Its working

Was it something related to file not closing?

thanks again

if you don't close the output stream you run the risk of running over the MAXIMUM number of the opened streams which varies depending on your version of 'awk'.

thx again