UNIX for Dummies Questions & Answers

Just for completeness: the first parameter (n) is the number of lines per file,
the second - the custom_name with a numeric suffix:

Code:

awk 'FNR == 1 { c = 1 }
	      { print > (f c) }
!FNR%n { close(f c); ++c }
' n=<number_of_lines> f=<custom_name> big_file

If you want the number of files/parts,
it's important if the total number of lines in the bigfile is known a priori
(otherwise it will be slower, because we have to read it twice: first to get
the number of lines, and then again to split it).

That is true Radoulov....It is very very slow. The only way I can get the number of lines is by doing a wc -l on the file. The records keep changing and we have ten different files that are incoming from a different source. We will be using this script company wide...

Hi, jim mcnamara.
The number in red appears to be missing a zero, suggesting that the last part of the file beyond 60K (not 600K) ends up on smallfile3 ... cheers, drl

Hi.
This does not appear to need absolute precision. You can read the first 100 (or few hundreds) of lines, then seek to near the end, read another 100 or so until the end, then calculate the mean length of the lines you saw. Obtaining the length of the file is cheap -- Linux has a stat command, but one could cut the length out from "ls -l". Divide the length in bytes by the estimated mean length and you get an estimate of the number of lines.

If necessary, one could also read a section from the middle (or elsewhere) to increase the accuracy.

The key is using a seek for positioning, and stat for the length -- both are very fast ... cheers, drl