Loop for file merging in a folder


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Loop for file merging in a folder
# 1  
Old 01-26-2014
Question Loop for file merging in a folder

Dear all,
I have a few files in a folder (lets say 5) and each have a few lines in them. I want to create merges of each of them with the other ones e.g.
Quote:
files = march, april,may, june, july,
I need the following merges
Quote:
combinations of 2 == march+arpil, march+may, march+june, march+july ... and the same for the others
combinations of 3 == march+april+may ...
combinations of 4 == march+april+may+june
I tried to write a loop and started with combinations of 2, I get the write file names but for each combination it merges all the files
Code:
mkdir output
FZ="result/*/*"
	for Z in $FZ
	do
		docz=$(basename $Z)
		FV="result/*/*"
		for V in $FV
		do
			docv=$(basename $V)
			cat $FZ $FV | sort -u > output/${docz}-${docv}
	done  
done

can you point where the problems is and how I can fix it ?
# 2  
Old 01-26-2014
Which OS do you use?
As for me...
Code:
[sea@localhost ~]$ FZ="result/*/*"
[sea@localhost ~]$ echo $FZ
result/*/*

And I see no counters for your diffrent merge types (2,3,4).
Using sort after cat'ing 2 files, might mix up their lines.
Also you'll merge March with March, April with April, and so on...

But before going closer/deeper, sorry for asking, is this schoolwork?
hth

Last edited by sea; 01-26-2014 at 10:42 AM.. Reason: Asking because of: result/*/* not working on fedora linux
# 3  
Old 01-26-2014
Hello Sea, I switch between unix and cygwin but now on cygwin
no i have not yet considered them, i was trying to go step by step (which might be the wrong approach)
yes, because some of the details are the same and I dont want to have duplications (e.g. costumer details)

no this is not a schoolwork Smilie, this is me trying to analyse a big data which i thought it might be easier with a code then manually of excel Smilie

btw, I figured out the way to fix the problem with the code above as it was a silly mistake
Code:
	cat $FZ $FV | sort -u > output/${docz}-${docv}

should be
Code:
		cat $Z $V | sort -u > output/${docz}-${docv}

it should be possible to add some more loops to cover all but is there a better way to code this? (thanks)

Last edited by A-V; 01-26-2014 at 11:06 AM..
# 4  
Old 01-26-2014
Ok then, and yes that should be the case Smilie

Untested but hope it gets you started:
Code:
#!/bin/sh
#
#	Vars
#
	FZ=*
	FV=$FZ
	I=1
	THESE=""
#
#	Action
#
	mkdir output

	for C in 2 3 4
	do
		for Z in $FZ
		do
			THESE=$Z

			for V in $FV
			do
				while [[ $I -lt $C ]]
				do
					[[ ! $Z = $V ]] && THESE+=" $V" && let I++
				done
			done
			str=$(echo $THESE|sed s,' ','-',g)
			cat $THESE|sort > output/$str
			I=1
			THESE=""
		done
	done

Note its Shell, i dont know cygwin.

EDIT:
However, this should merge just any 2/3/4 combinations, but not counting +1 from current starting month, so it might be quite january 'oriented'...
eg: march-april but also april-march, so you'll have some 'duplicates'

Edit2:
Just comming back from holidays (fri), so i'm a bit rusty

EDIT3:
So to improve it, you could change the list (FZ=*) to an array (FZ=(*) ), which might help for only merges with files 'higher' to the count of the current.
To easy do so, i would use a Z-counter int and a V counter int to 'index'/work with the array.

Last edited by sea; 01-26-2014 at 11:27 AM..
# 5  
Old 01-26-2014
the code is not working but will try to figure out y
yes I have realized in my previous code that I had duplications which I have to try to get rid off as well Smilie
haha... still way better than my skills
# 6  
Old 01-26-2014
The try with array
Code:
#!/bin/sh
#
#	Vars
#
	FZ=(*)
	ZC=0 # Z Counter
	I=1
	THESE=""
	echo ${FZ[@]}
#
#	Action
#
	mkdir output

	for C in 2 3 4
	do
		for Z in ${FZ[@]}
		do
			THESE=$Z

			while [[ $I -gt $C ]]
			do
				NUM=$[ $ZC + $I ]
				THESE+=" ${FZ[$NUM]}"
				let I++
			done
			str=$(echo $THESE|sed s,' ','-',g)
			cat $THESE|sort -u > output/$str

			let ZC++
			I=1
			THESE=""
		done
	done

Edit2:
looked right, but was wrong

Last edited by sea; 01-26-2014 at 11:44 AM..
This User Gave Thanks to sea For This Post:
# 7  
Old 01-26-2014
I guess the structure of my data is not compatible with the naming
Code:
FZ=(result/*)

so i get errors
Quote:
test.sh: line 27: output/result/april-result/may: No such file or directory
test.sh: line 27: output/result/july: No such file or directory
test.sh: line 27: output/result/june: No such file or directory
test.sh: line 27: output/result/may: No such file or directory
test.sh: line 27: output/result/april: No such file or directory
test.sh: line 27: output/result/july: No such file or directory
test.sh: line 27: output/result/june: No such file or directory
test.sh: line 27: output/result/may: No such file or directory
test.sh: line 27: output/result/april: No such file or directory
test.sh: line 27: output/result/july: No such file or directory
test.sh: line 27: output/result/june: No such file or directory
test.sh: line 27: output/result/may: No such file or directory
Edit 1:
I have tried to add basename to get it right but I only get 5 files with the name of the months Smilie I think the naming system is not the way it should)
Code:
str=$(basename $(echo $THESE|sed s,' ','-',g))

Edit 2:
so I only get e.g. july at the end instead of a name like aril-may-june.txt

Last edited by A-V; 01-26-2014 at 12:00 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging two columns from two files with similar names into a loop

I have two files like this: fileA.net A B C fileA.dat 1 2 3 and I want the output output_expected A 1 B 2 C 3 I know that the easier way is to do a paste fileA.net fileA.dat, but the problem is that I have 10,000 couple of files (fileB.net with fileB.dat; fileC.net with... (3 Replies)
Discussion started by: valente
3 Replies

2. Shell Programming and Scripting

For loop for number of files in a folder

Hi All, Need a for loop which should run for number of files in a folder and should pass the file name as parameter to another shell script for each loop. Please help me. Thanks. (2 Replies)
Discussion started by: chillblue
2 Replies

3. Shell Programming and Scripting

For loop to read folder which are not under processing

Hi, I want to read folders which do not have a file "processing" in a for loop ordered by timestamp. Currently im doing like this. Like cd /home/working for i in `ls -c1` do some command... done I want to exclude folders which have that "processing" file. The directory... (2 Replies)
Discussion started by: chetan.c
2 Replies

4. Shell Programming and Scripting

File merging based on different counter loop

hello, File 1 main Group sub group MIT VAR_1D_DATA_TYPE 23-03-2012 MIT VAR_1D_DATA_TYPE 22-03-2012 MIT VAR_10D_DATA_TYPE 23-03-2012 MIT VAR_10D_DATA_TYPE 22-03-2012 MIT ... (0 Replies)
Discussion started by: manas_ranjan
0 Replies

5. Shell Programming and Scripting

Loop through text file > Copy Folder > Edit XML files in bulk?

I have a text file which contains lines in this format - it contains 105 lines in total, but I'm just putting 4 here to keep it short: 58571,east_ppl_ppla_por 58788,east_pcy_hd_por 58704,east_pcy_ga_por 58697,east_pcy_pcybs_por It's called id_key.txt I have a sample folder called... (9 Replies)
Discussion started by: biscuitcreek
9 Replies

6. Shell Programming and Scripting

Merging folder of files according to name and contents.

I have absolutely no idea how to do this and everything I have done doesn't even come close. Here's the scenario: There are a number of files in a folder named in this manner: agd.txt stv.txt frk.txt dqp.txt There is also a series of other files in the same folder with like file... (5 Replies)
Discussion started by: Trapper
5 Replies

7. Shell Programming and Scripting

File in a folder and loop until time

Hi All, I am stuck with this requirement. My requirement is that a job will run daily at anytime before and would look for a file named filename.txt in a folder and mv to a different folder and if not found until 9 A.M it will exit with a success code. Now I did the follwing ..but there... (8 Replies)
Discussion started by: RubinPat
8 Replies

8. Shell Programming and Scripting

Shell script for merging lines in a loop

Dear All, I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl). I/P File------------------------- APaul,,,,SDH,,23,,,PPH,,2 ,,,,KKH,,19,,,MMH,,12, ,,,,CCH,,22,,,MNH,,19, ,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies

9. Shell Programming and Scripting

File Management: How do I move all JPGS in a folder structure to a single folder?

This is the file structure: DESKTOP/Root of Photo Folders/Folder1qweqwasdfsd/*jpg DESKTOP/Root of Photo Folders/Folder2asdasdasd/*jpg DESKTOP/Root of Photo Folders/Folder3asdadfhgasdf/*jpg DESKTOP/Root of Photo Folders/Folder4qwetwdfsdfg/*jpg DESKTOP/Root of Photo... (4 Replies)
Discussion started by: guptaxpn
4 Replies

10. Shell Programming and Scripting

Parse the .txt file for folder name and FTP to the corrsponding folder.

Oracle procedure create files on UNIX folder on a regular basis. I need to FTP files onto windows server and place the files, based on their name, in the corresponding folders. File name is as follows: ccyymmddfoldernamefile.txt; Folder Name length could be of any size; however, the prefix and... (3 Replies)
Discussion started by: MeganP
3 Replies
Login or Register to Ask a Question