How to reduce multiple files into a specific number of files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to reduce multiple files into a specific number of files
# 1  
Old 04-05-2009
How to reduce multiple files into a specific number of files

Can anyone please let me know how do I reduce files into a specific number of files by cat'ing files?

For example:

15 files must be reduced to 1 or 5 or 9 (possible values 1 to 14)
# 2  
Old 04-05-2009
Please expand your question and give examples of your input and expected output.
# 3  
Old 04-06-2009
Sorry if I wasn't clear. I have a set of input files ranging anywhere from 4 to 15
Input File: File 1,File2 ....File 15

I need to merge/cat these input files into fewer number of files.

For example:
1) If the argument is 1 (meaning if I need just 1 output file) then all the 15 files must be concatenated into 1.
2) If the argument is 5, File1,2,3 cat'ed to 1st output file and so on
3) If the argument is 3 , Files 1,2,3,4,5 cat'ed to 1st output file and so on
# 4  
Old 04-06-2009
what if parameter is 4 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 how are the files going to be distibuted then?

it still pretty unclear to me...but hopefully you understand this.
Code:
for i in $list
do
cat $i >>mergedfile
done

u get the idea right?
variable list contains the filenames set as per parameter passed
# 5  
Old 04-06-2009
What you have done is cat'ing all files into 1, but the number of input files is a variable and can be anything from 4 to 15 and number of files to be merged/cat'ed is also variable.

If no_of_input files is 15, and no_of_output files is any of the following (samples)

1) 1 then all the 15 files must be concatenated into 1.
2) 5, File1,2,3 cat'ed to 1st output file and so on
3) 3 Files 1,2,3,4,5 cat'ed to 1st output file and so on
4) 0-->Invalied entry
5) 16 -->Invalied entry
6 15 -->Do nothing ( because there are already 15 files)

If no_of_input files is 10, and no_of_output files is
1) 1 then all the 10 files must be concatenated into 1.
2) 5, File1,2 cat'ed to 1st output file and so on
3) 10- Do nothing (because there is already 10 files)
# 6  
Old 04-06-2009
Use c_d's script - then add the split command
Code:
outfiles=4
for i in $list
do
cat $i 
done  > mergedfile

wc -l mergedfile | read linecnt  throwaway
limit=$(( $linecnt / $outfiles ))
limit=$(( $limit + $linecnt%limit ))
split -l $limit mergedfile newfile

The output of this is : newfileaa newfileab newfileac newfilead
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing carriage returns from multiple lines in multiple files of different number of columns

Hello Gurus, I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file. 1|ABC DEF|100|10 2|PQ RS T|200|20 3| UVWXYZ|300|30 4| GHIJKL|400|40... (7 Replies)
Discussion started by: dJHa
7 Replies

2. Shell Programming and Scripting

Extracting specific files from multiple .tgz files

Hey, I have number of .tgz files and want to extract the file with the ending *results.txt from each one. I have tried for file in *.tgz; do tar --wildcards -zxf $file *results.txt; doneas well as list=$(ls *.tgz) for i in $list; do tar --wildcards -zxvf $i *.results.txt; done... (1 Reply)
Discussion started by: jfern
1 Replies

3. Shell Programming and Scripting

Process a specific number of files ina list

Hello, I have a list of files that was created with, FILES='./'$FOLD'/'$FOLD'_continue/'$OPTIMIZE_ON'/'*'out.txt' I am doing a loop on this list for INPUT in $FILES do ... done but I may not want to process everything. Is there a simple way to just process the first 5,10,n, etc in... (2 Replies)
Discussion started by: LMHmedchem
2 Replies

4. Shell Programming and Scripting

Compare multiple files with multiple number of columns

Hi, input file1 abcd 123 198 xyz1:0909090-0909091 ghij 234 999 xyz2:987654:987655 kilo 7890 7990 xyz3:12345-12357 prem 9 112 xyz5:97-1134 input file2 abcd 123 198 xyz1:0909090-0909091 -9.122 0 abed 88 98 xyz1:98989-090808 -1.234 1.345 ghij 234 999 xyz2:987654:987655 -10.87090909 5... (5 Replies)
Discussion started by: jacobs.smith
5 Replies

5. Shell Programming and Scripting

averaging specific column of multiple files

Hi all, I'm needing help again on scripting. I have weekly files with 3 columns, and I need to do monthly averaging on the values on column 3, the file naming convention is as follows: 20000105.u- 2000:year 01:month 05:day 20000112.u 20000119.u 20000126.u 20000202.u 20020209.u I need to... (15 Replies)
Discussion started by: ida1215
15 Replies

6. Shell Programming and Scripting

Need script to count specific word and iterate over number of files.

Hi Experts, I want to know the count of specific word in a file. I have almost 600+ files. So I want to loop thru each file and get the count of the specific word. Please help me on achieving this... Many thanks (2 Replies)
Discussion started by: elamurugu
2 Replies

7. Shell Programming and Scripting

highly specific search and replace for a large number of files

hey guys, I have a directory with about 600 files. I need to find a specific word inside a command and replace only that instance of the word in many files. For example, lets say I have a command called 'foo' in many files. One of the input arguments of the 'foo' call is 'bar'. The word 'bar'... (5 Replies)
Discussion started by: ksubrama
5 Replies

8. Shell Programming and Scripting

Replace text in SPECIFIC multiple files

I have a list of files with different file names and ext that i need replace(saved as file_list.txt.) i just want to replace from a specific file list. i obtain an error saying lint not found. Plz help. Thx perl -e "s/$NAME/$T_NAME/gi;" -pi $(find . -type f | xargs -0 lint -e < file_list.txt) (0 Replies)
Discussion started by: yvmy
0 Replies

9. UNIX for Dummies Questions & Answers

Unix command to count the number of files with specific characters in name

Hey all, I'm looking for a command that will search a directory (and all subdirectories) and give me a file count for the number of files that contain specific characters within its filename. e.g. I want to find the number of files that contain "-a.jpg" in their name. All the searching I've... (6 Replies)
Discussion started by: murphysm
6 Replies

10. Shell Programming and Scripting

Creating large number of files of specific size

Hi I am new to shell scripting.I want to create a batch file which creates a desired number of files with a specific size say 1MB each to consume space.How can i go about it using for loop /any other loop condition using shell script? Thanks (3 Replies)
Discussion started by: swatideswal
3 Replies
Login or Register to Ask a Question