This may sound like a trivial problem, but I still need some help:

I have a file with ids and I want to split it 'n' ways (could be any number) into files:

1
1
1
2
2
3
3
4
5
5

Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may look like (1,1,1,), (2,2,3,3),(4,5,5).

Thanks guys,

- CB

You're aware there's a limited number of combinations for a given fixed number of ids, right?
So, if you have 1 1 1 2 2 3 3 4 5 5 in the example you gave, and this happens more than once, the first partition could be (1,1,1),(2,2,3,3),(4,5,5) and a second occurrence of those ids could generate a partition like (1,1), (1, 2,2,3,3),(4,5,5), right?

The second combination that you listed is incorrect because partition 1 and 2 both have id '1' in them.

Thanks,

- CB

I don't understand what you want then. Give me an example, if I have the same ID twice, how should the program handle it?

I was hoping for a better solution, but here is a crude way that i thought of:

1. split the file 'n' ways (n=3 for this example):

part 1 part 2 part 3
1 2 3
1 2 4
1 3 5

2. if n%(size of orig file) = 3%10 > 0 then append remaining id to the last partition

part 3
3
4
5
5

3. Compare part 1 with part 2 and see if ids are matched. If found, then move row from part 2 to part 1. Move to the next part and do the same.

part 1
1
1
1

part 2
2
2
3
3

part 3
3
4
5
5
Hopefully, someone will present a sleeker solution with some syntax.

Thanks,

- CB

by the way, the original file is sorted by id

The OP wants all the 1's in a single file, 2's in a single file possibly with all 3's in the same file as well.

The problem is you have to know the split count as well as the complete key list and count of unique keys and how to group them before you attempt a split. I would create a list of unique key fields, divide the count by 3 and let any extras fall into the last split.

The problem with this is that you can get splits of enormously different sizes depending on how skewed the distribution of keys is in the data file. It defeats splitting altogether - IMO. And what happens when you ask for more splits than there are keys?

The only thing that that makes sense to me is a one-to-one split - one distinct key per file or leave everything in one big file.