Split a large file with patterns and size

sudhamacs · #1 (**permalink**) 06-23-2008

Hi,
I have a large file with a repeating pattern in it. Now i want the file split into the block of patterns with a specified no. of lines in each file.
i.e. The file is like

1...
2...
2...
3...
1...
2...
3...
1...
2...
2...
2...
2...
2...
3...

where 1 is the start of the block and 3 is the end of the block.
Now i want the file to be split up in blocks. But each file can have more than one block, but specified no. of lines or less.

What i have done is i split up the file using

Code:

awk '/^1/{close("file"f);f++}{print $0 > "file"f}'  testfile

But this gives one file for each block. I needed to concat the files to get a file with some specified no. of lines, say 10000.

Please give me an efficient way to do this. I feel we can do it in the awk command only.
Thanks,
CSS

Franklin52 · #2 (**permalink**) 06-24-2008

Code:

awk 'NR%10000==1{close("file"f);f++}{print $0 > "file"f}'  testfile

Regards

sudhamacs · #3 (**permalink**) 06-24-2008

But that is writing 10000 line per file. But not patternwise.

rubin · #4 (**permalink**) 06-25-2008

Maybe this will help:

Code:

awk  '/^1/{f=1} f{ print $0 > "file_"n ; c++} c==x+1{ n++; c=1; close("file_"n)} /^3/{f=0}'  x=10000 c=1 n=1  filename

Change x to whatever number of lines per file you need to have. Whereas c and n are just counters, that don't need to be changed.

sudhamacs · #5 (**permalink**) 06-25-2008

Quote:

Originally Posted by rubin

Maybe this will help:

Code:

awk  '/^1/{f=1} f{ print $0 > "file_"n ; c++} c==x+1{ n++; c=1; close("file_"n)} /^3/{f=0}'  x=10000 c=1 n=1  filename

Change x to whatever number of lines per file you need to have. Whereas c and n are just counters, that don't need to be changed.

Even this is giving files with 10000 lines but i want to split in patterns.

sudhamacs · #6 (**permalink**) 06-25-2008

Quote:

Originally Posted by sudhamacs

Even this is giving files with 10000 lines but i want to split in patterns.

Thanks Rubin,
But i had to make some minor changes and this worked.
awk '/^1/{f=1} f{ print $0 > "file_"n ; c++} c>10000 && /^3/ { n++; c=1; close("file_"n) }' c=1 n=1 testfile

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Split large file and add header and footer to each small files	ashish4422	Shell Programming and Scripting	7	07-07-2008 03:13 PM
Split large file and add header and footer to each file	ashish4422	Shell Programming and Scripting	1	04-15-2008 07:12 AM
Need to split a large data file using a Unix script	SAIK	HP-UX	1	03-29-2006 05:05 PM
Split A Large File	nbvcxzdz	Shell Programming and Scripting	4	11-14-2005 08:48 AM
prevent file size is too large	ust	UNIX for Dummies Questions & Answers	2	03-10-2005 02:04 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by