Split files with formatted numbers Post: 302907416

Sponsored Content

Top Forums Shell Programming and Scripting Split files with formatted numbers Post 302907416 by Don Cragun on Friday 27th of June 2014 04:52:40 PM

06-27-2014

Registered User

If you don't have GNU awk (or if you want code that should work on any system), you could try something like:

Code:

#!/bin/ksh
lc=$(wc -l < split.txt)
awk -v lc="$lc" '
BEGIN {	lpf = 1000000	# Lines per output file.
}
FNR == 1 {
	# This is not in the BEGIN section to allow the default value of lpf
	# to be overridden by an assignment before the filename operand.
	nf = int((lc + lpf - 1) / lpf)	# Total number of files to be created.
}
NR % lpf == 1 {
	# 1st line of output file:
	fn=sprintf("split.%03d.txt", ++ofc)
	printf("%03d of %03d files\n", ofc, nf) > fn
}
{	# all lines:
	print > fn
}
NR % lpf == 0 {
	# Last line of output file:
	trailer()
}
END {	if(NR % lpf)
		trailer()
}
function trailer() {
	close(fn)
	cmd = sprintf("date >> \"%s\"\n", fn)
	system(cmd)
}' "$@" split.txt

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.

Although I normally use the Korn shell (and this was tested using ksh), it will work with any shell that supports basic POSIX shell standard syntax (such as bash and ksh).

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to remove improperly formatted fortran output line from files, tried sed

I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in...

2. Shell Programming and Scripting

Generating formatted reports from log files

3. UNIX for Dummies Questions & Answers

Split Function Prefix Numbers

Hello, Hello, I use the following command to split a file: split -Number_of_Lines Input_File MyPrefix_ output is MyPrefix_a MyPrefix_b MyPrefix_c ...... Instead, how can I get numerical values like: MyPrefix_1 MyPrefix_2 MyPrefix_3 ......

4. Shell Programming and Scripting

Extracting formatted text and numbers

Hello, I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh. For example, given the following data I want to extract: and...

5. UNIX for Dummies Questions & Answers

Breaking a fasta formatted file into multiple files containing each gene separately

Hey, I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time: for i in *.rtf.out do awk '/^>/{f=++d".fasta"} {print > $i.out}' $i done

6. Shell Programming and Scripting

Split a file into multiple files based on line numbers and first column value

Hi All I have one query,say i have a requirement like the below code should be move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines. This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first...

7. Shell Programming and Scripting

awk split numbers

I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into -dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25...

8. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce...

9. Shell Programming and Scripting

Sum up formatted numbers with comma separation

I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values. 1,775,947,633 4,738 7,300 16,610 15,279 0 0 I tried the following code in a shell script. awk '{sum+=$5} END{print...

LEARN ABOUT OPENSOLARIS

endusershell

getusershell(3C)					   Standard C Library Functions 					  getusershell(3C)

NAME

       getusershell, setusershell, endusershell - get legal user shells

SYNOPSIS

       #include <unistd.h>

       char *getusershell(void);

       void setusershell(void);

       void endusershell(void);

DESCRIPTION

       The  getusershell()  function  returns  a  pointer  to  a  legal  user  shell  as defined by the system manager in the file /etc/shells. If
       /etc/shells does not exist, the following locations of the standard system shells are used in its place:

	 /bin/bash		    /bin/csh
	 /bin/jsh		    /bin/ksh
	 /bin/ksh93		    /bin/pfcsh
	 /bin/pfksh		    /bin/pfsh
	 /bin/sh		    /bin/tcsh
	 /bin/zsh		    /sbin/jsh
	 /sbin/pfsh		    /sbin/sh
	 /usr/bin/bash		    /usr/bin/csh
	 /usr/bin/jsh		    /usr/bin/ksh
	 /usr/bin/ksh93 	    /usr/bin/pfcsh
	 /usr/bin/pfksh 	    /usr/bin/pfsh
	 /usr/bin/sh		    /usr/bin/tcsh
	 /usr/bin/zsh		    /usr/sfw/bin/zsh
	 /usr/xpg4/bin/sh

       The getusershell() function opens the file /etc/shells, if it exists, and returns the next entry in the list of shells.

       The setusershell() function rewinds the file or the list.

       The endusershell() function closes the file, frees any memory used by getusershell() and setusershell(), and rewinds the file /etc/shells.

RETURN VALUES

       The getusershell() function returns a null pointer on EOF.

BUGS

       All information is contained in memory that may be freed with a call to endusershell(), so it must be copied if it is to be saved.

NOTES

       Restricted shells should not be listed in /etc/shells.

SunOS 5.11							    1 Nov 2007							  getusershell(3C)