Hi guys,
I'm been struggling for some time to split a big file into smaller ones based on some conditions without any luck so far. I've been able to complete this with a small shell script with loops but it takes around 1h for a file with 600k records . I'm sure there is a simpler solution with awk.

The file I'm trying to split (comma separated file):
field1, ..., ..., ...
field1, ..., ..., ...
.....
field1, ..., ..., ...
field2, ..., ..., ...
field2, ..., ..., ...
.....
field2, ..., ..., ...
.....
fieldn, ..., ..., ...

What I need is to split this file into multiple files (with name F1, F2, etc) normally with no more than 2k records in each file. However, the first field values should not span into separate files. For example if in the record 2001 the first field is the same as in the records 2000 it should go also in the first file, a new file should be created only when the value of the field changes.

Appreciate all your help.

Like this?

awk -F, '{print > "File_" $1}' file

Thanks for the reply.
This script will split the file into multiple files each one containing the distinct values of first field. I would like if the output file does not contain 2000 lines to keep adding also in the same file other values of the first field also.

Try something like:

awk 'BEGIN {
        FS = OFS = ","
        fn = "F" ++fc
}
c >= 2000 && $1 != last {
        close fn
        fn = "F" ++fc
        c = 0
}
{       print > fn
        last = $1
        c++
}' input

I'm getting an error I think at close fn:
awk: cmd. line:5: close fn
awk: cmd. line:5: ^ syntax error

Sorry about that. Try close(fn) instead of close fn . The awk I'm using on OS X accepts either form, but the standards only require it to work when the parentheses are provided.

awk -F, 'c[i]>=2 && p1!=$1{i++}
{print > "File_" $1;c[i]++}
{p1=$1}' i=1 file