Shell Programming and Scripting

Thanks in advance : )

I try for a long time searching for a way to split a large gzip csv file into many gzip files (except for the last sub-file which is to joint the next big file's children.) All the subfiles are to be named by the field.

But I only managed to split them into the uncompressed subfiles.

Code:

zcat bigfile | awk ‘{ print $0 >> $1 }’ FS=”,”

If I can pass $1 outside awk and if gz could append lines then it will save me a lot of i/o time and capacity. One of my gzip file is 124 gb.

I.e. something like

Code:

zcat file | awk ‘{ print $0 }’ FS=”,” |gzip >> "$PassOut1".gz

If you kwon what I mean.

may be you can try something like this.

First write the uncompressed data in subfile1 and when you are about to strat the subfile2, run gzip for subfile1.

I am new to this forum. Thanks Kumaran. I am not sure how to do that, i.e. testing the change of $1. Also the source data is not sorted by $1 so it's hard to tell tell if I get more of the same over the entire bigfile.

we need sample data and criteria to split the file.(where and when to split the file)

Code:

mkdir subfolder
zcat file | awk -F, '{print $0 > "subfolder/" $1}' 
gzip subfolder/*

Thanks rdcwayx!! Thats neat, but the file is not gz until it's all split.

The data goes like this, with a lot of fields

Code:

VTLJ.J	2-Jan-96	08:07:07.310	2	Quote
VTLJ.J	2-Jan-96	09:30:00.320	2	Quote
BKLJ.J	2-Jan-96	10:56:38.660	2	Quote
LJ.C	3-Jan-96	09:23:34.070	2	Quote

If I can pass $1 outside awk and if gz could be appended line then it will save me a lot of i/o time and capacity. One of my gzip file is 124 gb.

I.e. something like

Code:

Zcat file | awk ‘{ print $0 }' FS=”,” |gzip >> $PassOut1

If you kwon what I mean.

That, is I like to do 1 pass of the data and split the bigfile into

File VTLI.J.gz

Code:

VTLJ.J	2-Jan-96	08:07:07.310	2	Quote
VTLJ.J	2-Jan-96	09:30:00.320	2	Quote

File BKLI.J.gz
Etc

First, I don't see "," in your input file.

Second, Do you mean the big file has been sessioned or sorted by column 1

For example, if column 1 is VTLJ.J, and after I see BKLJ.J, there should be no line with VTLJ.J.

You need confirm it first.

If it is not, you have to wait until awk go through the whole file, then gzip the output files.