UNIX for Dummies Questions & Answers

Hi,

I have some files with 2 million odd records which i need to split into chunks of 0.5 millions. I have the file sorted with a key column in order. The same key value can appear as 4 or 5 records in the file.
Hence after splitting we are checking whether all the key values are present in the same file.
After splitting we will be running all the files in parallel, so we need all the key values in one file.

For ex key column is column A

column A column B
--------- --------
AAAAAAA 1234244
BBBBBBBB 8734793
BBBBBBBB 3925873
BBBBBBBB 9085000
CCCCCCC 3094823
DDDDDDD 9084509

Here when i split this file into two. i need the first 4 columns in the same file since the the Column A is the key column and i want the key column value BBBBBBB in the same file.

When i split the file using split command, it looks like it doesnt split in the order the file is.

Is there a way to split the file in the same order as it it.

i.e
AAAAAAA 1234244
BBBBBBBB 8734793
BBBBBBBB 3925873
in one file and

BBBBBBBB 9085000
CCCCCCC 3094823
DDDDDDD 9084509
in another file. after this i can manually check for the key columns and move up or down as needed.

how can i split it in the same order as it is?

if you wanna split it as chunk of .5 million you can go for aplit by line count

Code:

split -l 50000 filename

hi,

i did that, but the sort order is not maintained in the split files

ok, i think it works like this.

if the file has 27 records

the last 9 records are stored in the first split file
then 10-18 records are stored in the second split file
then 1-9 records are stored in the third split file..

am i right?

its working fine for me..

Code:

> cat q
AAAAAAA 1234244
BBBBBBBB 8734793
BBBBBBBB 3925873
BBBBBBBB 9085000
CCCCCCC 3094823
DDDDDDD 9084509
> split -l 3 q
>cat xaa
AAAAAAA 1234244
BBBBBBBB 8734793
BBBBBBBB 3925873
> cat xab
BBBBBBBB 9085000
CCCCCCC 3094823
DDDDDDD 9084509

this is what you needed right??

thanks you so much..it works now.