UNIX for Dummies Questions & Answers

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this.

For example:

Code:

split -l 3000000 filename.txt

This is very slow and it splits the file with 3 million records in each file. But I would like to give the number of files as a parameter and output the user defined file names and not xaa, xab and so on.

I am also trying awk and I know it will be very fast and simple. I read the forum and they are all splitting the files on a specific pattern and I don't require any pattern.

Please give me your input on this..

I would have thought dd would be a more appropriate choice for this?

If you can recommend a fast way like awk, that would be very much appreciated. The split taking up a lot of time.

If disk i/o is not making split "too slow" then try awk. But you should consider that a big I/O request queue length on that filesystem is a likely candidate for slow splitting, rather than split being a bad performer.
awk version of split:

Code:

awk ' {
          if(NR<300000) { print $0 > "smallfile1"}
          if (NR>300000 && NR < 600000) { print $0 > "smallfile2" }
          if (NR>60000) {print $0 > "smallfile3" }
       }'  bigfile

Another approach - you can pass multiple arguments and control the filenames:

Code:

awk 'FNR == 1 { c = 1 }
{ close(FILENAME c-1)
	print > (FILENAME (!(FNR%30000000) ? ++c : c))
}'  file_1 file_2 ... file_n

or:

Code:

awk 'FNR == 1 { c = 1 }
	      { print > (FILENAME c) }
!FNR%30000000 { close(FILENAME c); ++c }
' file_1 file_2 ... file_n

Use nawk or /usr/xpg4/bin/awk on Solaris.

Thank you Radoulov...When I ran your code, it is saying file1, file2 or file3 is not found. It seems like the code is assuming that those are the input files. However, Jim's code is working fine.

The whole environment is on Windows. But I am using MKS Tool kit and invoking bash shell to execute awk. Never worked on Windows before and it is not quite nice..

Sorry,
just realized I misread your question
(you don't want to pass multiple input files).