Shell Programming and Scripting

How to split the file and have suffix with formatted numbers

Tried the following code

Code:

awk '{filename="split."int((NR-1)/2)".txt"; print >> filename}' split.txt

Current Result Expected Result

You should use a printf statement, like this with the d printing directive...

Code:

{printf("split.%03d.txt\n","1")}

how do I embed the printf in awk. I also need the content of original file in all these split file based on no. of lines I like to split

sprintf prints to a string.

Code:

filename=sprintf("split.%03d.txt",(NR-1)/2)

Since you're specifying 3 digit file sequence numbers, I assume you expect that you'll be producing more than a hundred files with this script. There is a good chance that awk will run out of file descriptors if you keep all of them open. You might want to consider something like:

Code:

awk '
NR%2 {	# Odd lines:
	fn = sprintf("split.%03d.txt", (NR - 1) / 2)
}
{	# all lines:
	print >> fn
}
(NR%2) == 0 {
	# Even lines:
	close(fn)
}' split.txt

If there are existing split.xxx.txt files when you start this script do you really want to append data to them, or do you want to remove any data that was there before and just keep what you find in the current input file?

If you want to append to existing files, the script above should work.

If you want to replace data instead of appending data, change:

Code:

	print >> fn

to:

Code:

	print > fn

As always, if you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.

I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.

Besides, how do I get header (n records) and trailer with file number or some content in it.

Simple, you slightly modify the code I gave you to put 100000 lines per output file instead of 2 lines per output file. The code I gave you already closes files when it is done with them so it only keeps one output file open at a time.

You're going to have to give us a lot more than "get header (n records) and trailer with file number or some content in it" to guess at what you want to put as headers and trailers in your files. Show us sample input and show us sample output! How is your script supposed to identify which lines are headers, which lines are trailers, and what data you want added to or removed from those headers as you copy parts of the input file to your hundreds of output files?