Dynamically split file Post: 302961718

Sponsored Content

Top Forums Shell Programming and Scripting Dynamically split file Post 302961718 by Don Cragun on Wednesday 2nd of December 2015 07:17:53 PM

12-02-2015

Registered User

If you create a file named file_list.txt that contains the names of the output files you want to produce (one filename per line), the following script seems to do what you want:

Code:

awk '
FNR == NR {
	of[++ofc] = $0
	next
}
{	m = q[ms = 1]
	for(i = 2; i <= ofc; i++)
		if(q[i] < m)
			m = q[ms = i]
	print > of[ms]
	q[ms] += $2
}' file_list.txt Input_file.txt

Note that this code makes two simplifying assumptions:

the quantity field in each of the first [B]n[/B ](where [B]n[/B is the number of lines in file_list.txt]) lines of Input_file.txt is greater than zero, and
your version of awk can keep open as many output files as you are trying to create. (For any version of awk that should be at least 9 files; for many versions of awk, it will be several dozen files.)

If file_list.txt contains:

Code:

FileA
FileB
FileC
FileD
FileE

it produces the output you said you wanted in each of the output files you listed post #3 when Input_file.txt contains the sample input data also listed in post #3.

If someone else wants to try this script on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

dynamically linked file

Hi friends , how do i view a dynamically linked file in unix ? its there on other system and do i have to ftp it in ASCII format or binary ? and after the ftp how do i view it ? thanks in advance veeras

2. UNIX for Advanced & Expert Users

dynamically linked file

Hi friends, i have a dynamically linked file on my solaris system.this is script that runs regularly. How can i read the contents of that ? when i tried to say "vi filename " then it says executable and nothing is seen. Please help. thanks in advance Veera

3. Shell Programming and Scripting

dynamically split the records

Hi, I was wandering would it be possible to split the record dynamically based on the certain values, for an instance i have a file with record with predefined split value i.e 10 col1 col2 col3 col4 ------------------------ aaaa bbbb 2 44aaaabbbb55cccddd1110 mmn xnmn 3...

4. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each...

5. Shell Programming and Scripting

Spliting the file dynamically

i am creating the file , when this file reaches the size 2 GB, i need one message or fire

6. Shell Programming and Scripting

Dynamically locating a file

Hi, I have a requriement where in I need to install a s/w by executing the installable file through a script. The script currently contains the path of the installable file. I need to now update the script accordingly such tht it identifies the location of the installable file automatically and...

7. UNIX for Dummies Questions & Answers

Read a file dynamically

Hi my requriment is read the file name dynamically my code is #!/bin/sh file="/c/work/loan/" Header_Trailer_move() { sed '1d;$d' $file| cat >sam.txt } Header_Trailer_move in above given path my list of files or there i have to read file dyanamically when i entered particular file name...

8. Shell Programming and Scripting

Shell Script to Dynamically Extract file content based on Parameters from a pdf file

Hi Guru's, I am new to shell scripting. I have a unique requirement: The system generates a single pdf(/tmp/ABC.pdf) file with Invoices for Multiple Customers, the format is something like this: Page1 >> Customer 1 >>Invoice1 + invoice 2 >> Page1 end Page2 >> Customer 2 >>Invoice 3 + Invoice 4...

9. Shell Programming and Scripting

Dynamically creating file names from portions of another file name

Not sure how to do the following, but any help would be appreciated. Has to be done using C shell (sorry about that). I have about 300 files that I need this done for, but I am only going to give one example. I will just need to know how to execute your solution through some type of loop to get...

10. Shell Programming and Scripting

Split file Dynamically

Hello , I have a flat file ( comma separated ) and want to split dynamically . If I provide input 3 then rows 1,4,7 will o/p to a file and rows 2,5,8 will redirect to 2nd file and 3,6,9 rows will go to 3rd file So 3 files will be generated . Could it be possible in Unix?

LEARN ABOUT LINUX

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

dynamically linked file

Discussion started by: sveera

2. UNIX for Advanced & Expert Users

dynamically linked file

Discussion started by: sveera

3. Shell Programming and Scripting

dynamically split the records

Discussion started by: braindrain

4. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

Discussion started by: madhunk

5. Shell Programming and Scripting

Spliting the file dynamically

Discussion started by: kingganesh04

6. Shell Programming and Scripting

Dynamically locating a file

Discussion started by: yoursdavinder

7. UNIX for Dummies Questions & Answers

Read a file dynamically

Discussion started by: sgoud

8. Shell Programming and Scripting

Shell Script to Dynamically Extract file content based on Parameters from a pdf file

Discussion started by: DIps

9. Shell Programming and Scripting

Dynamically creating file names from portions of another file name

Discussion started by: jclanc8

10. Shell Programming and Scripting

Split file Dynamically

Discussion started by: Pratik4891

LEARN ABOUT LINUX

igawk