Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Dynamically split file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Dynamically split file
# 1  
Old 12-02-2015
Dynamically split file
Hi guys,

I have a file with 10000 entries (there are 2 columns. the first column contains the the product name and the second column contains the quantity of each product).

I would like to split this file into 5 different files. I want the 1st entry to go to the fileA the 2nd entry to fileB the 3rd entry to fileC the 4th entry to fileD and the 5th entry to fileE.

Then the 6th entry (and the rest of the entries) I would like to be added to the file with the lowest product quantity.

So somehow I want to check the files (based on the current quantity) before a new entry will be added.

Is there any way to do that in unix shell?

Thank you in advance.
# 2  
Old 12-02-2015
Is this a homework assignment? (Homework and coursework questions can only be posted in this forum under special homework rules.)

What separates fields in your file?

Please show us a small sample input file (~15 lines) and show us the output files that should be produced (with the contents of each file shown in CODE tags).

What operating system are you using?

What shell are you using?

What have you tried to solve this problem on your own?
This User Gave Thanks to Don Cragun For This Post:
rbatte1
# 3  
Old 12-02-2015
Hi Don

No that's not a homework/coursework (I wish I were a studentSmilie)
The fields are separated with tabs (I have another file where the fields are separated with commas).

Please check below a sample of the file that has been generated and it will be used as input:
Code:
25467Quill	948
38120Sarra	289
75283Moonstone	734
18359Mineral	400
63456Breeze	1022
78345Oxford	158
78456Duke	278
12726Iris	540
43361Velvet	195
65611Boutique	655
37556Exotica	452
11590Olana	794
70146Felicity	245
44498Quartz	852
61455Bella	693

The output files should be as below:
FileA
Code:
25467Quill	948
11590Olana	794

FileB
Code:
38120Sarra	289
78345Oxford	158
12726Iris	540
70146Felicity	245
61455Bella	693

FileC
Code:
75283Moonstone	734
65611Boutique	655

FileD
Code:
18359Mineral	400
78456Duke	278
43361Velvet	195
37556Exotica	452

FileE
Code:
63456Breeze	1022
44498Quartz	852

The OS is RedHat5 and I am using bash shell.

So far I 've tried the following:
Code:
split -l 2000 Input_file.txt

The above does not meet my needs as I am created 5 file with 2000 lines in each

Code:
awk '{print > "NewFile" ((NR-1)%x)}' x=5 Input_file.txt

That one is more close to what I need, however, the new entries are not checking the file with the lowest quantity before they will be added.
Last edited by coweb; 12-02-2015 at 07:28 PM..
# 4  
Old 12-02-2015
If you create a file named file_list.txt that contains the names of the output files you want to produce (one filename per line), the following script seems to do what you want:
Code:
awk '
FNR == NR {
	of[++ofc] = $0
	next
}
{	m = q[ms = 1]
	for(i = 2; i <= ofc; i++)
		if(q[i] < m)
			m = q[ms = i]
	print > of[ms]
	q[ms] += $2
}' file_list.txt Input_file.txt

Note that this code makes two simplifying assumptions:
  1. the quantity field in each of the first [B]n[/B ](where [B]n[/B is the number of lines in file_list.txt]) lines of Input_file.txt is greater than zero, and
  2. your version of awk can keep open as many output files as you are trying to create. (For any version of awk that should be at least 9 files; for many versions of awk, it will be several dozen files.)

If file_list.txt contains:
Code:
FileA
FileB
FileC
FileD
FileE

it produces the output you said you wanted in each of the output files you listed post #3 when Input_file.txt contains the sample input data also listed in post #3.

If someone else wants to try this script on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
coweb
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split file Dynamically

Hello , I have a flat file ( comma separated ) and want to split dynamically . If I provide input 3 then rows 1,4,7 will o/p to a file and rows 2,5,8 will redirect to 2nd file and 3,6,9 rows will go to 3rd file So 3 files will be generated . Could it be possible in Unix? (2 Replies)
Discussion started by: Pratik4891
2 Replies

2. Shell Programming and Scripting

Dynamically creating file names from portions of another file name

Not sure how to do the following, but any help would be appreciated. Has to be done using C shell (sorry about that). I have about 300 files that I need this done for, but I am only going to give one example. I will just need to know how to execute your solution through some type of loop to get... (2 Replies)
Discussion started by: jclanc8
2 Replies

3. Shell Programming and Scripting

Shell Script to Dynamically Extract file content based on Parameters from a pdf file

Hi Guru's, I am new to shell scripting. I have a unique requirement: The system generates a single pdf(/tmp/ABC.pdf) file with Invoices for Multiple Customers, the format is something like this: Page1 >> Customer 1 >>Invoice1 + invoice 2 >> Page1 end Page2 >> Customer 2 >>Invoice 3 + Invoice 4... (3 Replies)
Discussion started by: DIps
3 Replies

4. UNIX for Dummies Questions & Answers

Read a file dynamically

Hi my requriment is read the file name dynamically my code is #!/bin/sh file="/c/work/loan/" Header_Trailer_move() { sed '1d;$d' $file| cat >sam.txt } Header_Trailer_move in above given path my list of files or there i have to read file dyanamically when i entered particular file name... (2 Replies)
Discussion started by: sgoud
2 Replies

5. Shell Programming and Scripting

Dynamically locating a file

Hi, I have a requriement where in I need to install a s/w by executing the installable file through a script. The script currently contains the path of the installable file. I need to now update the script accordingly such tht it identifies the location of the installable file automatically and... (1 Reply)
Discussion started by: yoursdavinder
1 Replies

6. Shell Programming and Scripting

Spliting the file dynamically

i am creating the file , when this file reaches the size 2 GB, i need one message or fire (4 Replies)
Discussion started by: kingganesh04
4 Replies

7. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

8. Shell Programming and Scripting

dynamically split the records

Hi, I was wandering would it be possible to split the record dynamically based on the certain values, for an instance i have a file with record with predefined split value i.e 10 col1 col2 col3 col4 ------------------------ aaaa bbbb 2 44aaaabbbb55cccddd1110 mmn xnmn 3... (6 Replies)
Discussion started by: braindrain
6 Replies

9. UNIX for Advanced & Expert Users

dynamically linked file

Hi friends, i have a dynamically linked file on my solaris system.this is script that runs regularly. How can i read the contents of that ? when i tried to say "vi filename " then it says executable and nothing is seen. Please help. thanks in advance Veera (5 Replies)
Discussion started by: sveera
5 Replies

10. Shell Programming and Scripting

dynamically linked file

Hi friends , how do i view a dynamically linked file in unix ? its there on other system and do i have to ftp it in ASCII format or binary ? and after the ftp how do i view it ? thanks in advance veeras (1 Reply)
Discussion started by: sveera
1 Replies
Login or Register to Ask a Question