Split a file with no pattern -- Split, Csplit, Awk

madhunk · #1 (**permalink**) 12-13-2007

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this.

For example:

Code:

split -l 3000000 filename.txt

This is very slow and it splits the file with 3 million records in each file. But I would like to give the number of files as a parameter and output the user defined file names and not xaa, xab and so on.

I am also trying awk and I know it will be very fast and simple. I read the forum and they are all splitting the files on a specific pattern and I don't require any pattern.

Please give me your input on this..

Smiling Dragon · #2 (**permalink**) 12-13-2007

I would have thought dd would be a more appropriate choice for this?

madhunk · #3 (**permalink**) 12-14-2007

If you can recommend a fast way like awk, that would be very much appreciated. The split taking up a lot of time.

jim mcnamara · #4 (**permalink**) 12-14-2007

If disk i/o is not making split "too slow" then try awk. But you should consider that a big I/O request queue length on that filesystem is a likely candidate for slow splitting, rather than split being a bad performer.
awk version of split:

Code:

awk ' {
          if(NR<300000) { print $0 > "smallfile1"}
          if (NR>300000 && NR < 600000) { print $0 > "smallfile2" }
          if (NR>60000) {print $0 > "smallfile3" }
       }'  bigfile

radoulov · #5 (**permalink**) 12-14-2007

Another approach - you can pass multiple arguments and control the filenames:

Code:

awk 'FNR == 1 { c = 1 }
{ close(FILENAME c-1)
	print > (FILENAME (!(FNR%30000000) ? ++c : c))
}'  file_1 file_2 ... file_n

or:

Code:

awk 'FNR == 1 { c = 1 }
	      { print > (FILENAME c) }
!FNR%30000000 { close(FILENAME c); ++c }
' file_1 file_2 ... file_n

Use nawk or /usr/xpg4/bin/awk on Solaris.

madhunk · #6 (**permalink**) 12-17-2007

Thank you Radoulov...When I ran your code, it is saying file1, file2 or file3 is not found. It seems like the code is assuming that those are the input files. However, Jim's code is working fine.

The whole environment is on Windows. But I am using MKS Tool kit and invoking bash shell to execute awk. Never worked on Windows before and it is not quite nice..

radoulov · #7 (**permalink**) 12-17-2007

Quote:

Originally Posted by madhunk

Thank you Radoulov...When I ran your code, it is saying file1, file2 or file3 is not found. It seems like the code is assuming that those are the input files. However, Jim's code is working fine.

The whole environment is on Windows. But I am using MKS Tool kit and invoking bash shell to execute awk. Never worked on Windows before and it is not quite nice..

Sorry,
just realized I misread your question
(you don't want to pass multiple input files).

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
how to split a file	aarif	UNIX for Dummies Questions & Answers	2	03-01-2008 03:36 PM
Split files using Csplit	savitha	UNIX for Dummies Questions & Answers	7	12-01-2007 11:55 AM
Split file	mpang_	Shell Programming and Scripting	3	09-12-2006 08:37 PM
Split a file	Reza Nazarian	UNIX for Dummies Questions & Answers	1	08-09-2006 06:01 AM
multiple pattern split in perl	umen	Shell Programming and Scripting	3	08-01-2006 02:43 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: