Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Loop for file merging in a folder

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Loop for file merging in a folder
# 8  
Old 01-26-2014
Yes the script was bogus, a cig helped me think clearer (at least i tell me so)...

Now this one is tested and the output looks much better...
Code:
#!/bin/sh
#
#	Vars
#
	FZ=(*) # all files/dirs of current location
	ZC=0 # Z Counter
	THESE=""
#
#	Action
#
	[[ -d output ]] || mkdir output

	for Z in ${FZ[@]}
	do
		THESE=$Z

		for I in 1 2 3
		do
			NUM=$[ $ZC + $I ]
			THESE+=" ${FZ[$NUM]}"
			str=$(echo $THESE|sed s,' ','-',g)
			cat $THESE|sort -u > output/${str}.txt
		done

		let ZC++
		THESE=""
	done

Note that i ran the script inside the very same folder the other files existed.

So it'll expect:
SOMEDIR
- test-script
- january
- february
- ...
+ output-dir

Hope this helps better

EDIT:
Example how i tested it: (where tmp is the script)
Code:
[sea@localhost ~]$ ls
mm  net  notepad  output  priv  prjs  rpmbuild  tmp
[sea@localhost ~]$ ls output/
[sea@localhost ~]$ sh tmp
[sea@localhost ~]$ ls output/
mm-net-notepad-output.txt      output-priv-prjs.txt
mm-net-notepad.txt             output-priv.txt
mm-net.txt                     priv-prjs-rpmbuild-tmp.txt
net-notepad-output-priv.txt    priv-prjs-rpmbuild.txt
net-notepad-output.txt         priv-prjs.txt
net-notepad.txt                prjs-rpmbuild-tmp.txt
notepad-output-priv-prjs.txt   prjs-rpmbuild.txt
notepad-output-priv.txt        rpmbuild-tmp.txt
notepad-output.txt             tmp.txt
output-priv-prjs-rpmbuild.txt
[sea@localhost ~]$

Last edited by sea; 01-26-2014 at 12:17 PM..
# 9  
Old 01-26-2014
I know exactly what u mean :P
I have try to put in a different folder as I create my files as I go but it does not like that
Quote:
FZ=(result/*)
but there is still things missing e.g. from the files your created there is no
Quote:
mm-notepad.txt
and there are combinations with the code file
Quote:
priv-prjs-rpmbuild-tmp.txt
and thanks a lot for trying to help

edit
I have changed the place of the loop and for now I get months and combinations of u right but still the same problem with script and the rest of combinations
Code:
	for I in 1 2 3 4 
	do
		for Z in ${FZ[@]}
		do
			THESE=$Z
			NUM=$[ $ZC + $I ]
			THESE+=" ${FZ[$NUM]}"
			str=$(echo $THESE|sed s,' ','-',g)
			cat $THESE|sort -u > output/${str}.txt
		done
		let ZC++
		THESE=""
	done

Last edited by A-V; 01-26-2014 at 12:35 PM..
# 10  
Old 01-26-2014
Just a couple of observations that might help you progress in this thread, that seems to be wandering about a bit...

A-V:
could you please post the exact directory and file structure that you are using (and if there are other files/directories present) and also specify from which directory you plan to call the script. Also please state which shell you plan on using on what OS and in what order you would like the files to be merged or concatenated.


@sea:
  • #!/bin/sh does not know arrays, use #!/bin/bash instead
  • use double quotes around the enumeration of array elements "${array[@]}"
  • use double quotes when using echo echo "$text" (unless you want field splitting to occur)
  • echo $FZ will only produce result/*/* if there are no files that match this pattern.
  • [[ ... ]] is not proper syntax for #!/bin/sh, use [ ... ] instead or use #!/bin/bash instead, but then use == instead of =
  • $[ ... ] is deprecated in bash, use $(( ... )) instead
  • Don't use the external program let. Instead of let var++, use var=$((var+1)) or (in the case of bash) use (( var++ ))
Last edited by Scrutinizer; 01-26-2014 at 01:41 PM..
This User Gave Thanks to Scrutinizer For This Post:
A-V
# 11  
Old 01-26-2014
@ Scrutinizer
At times i bark back Smilie (meaning i want to be right)
Unix - Using Shell Arrays this tutorial says sh does support arrays.
Code:
sh-4.2$ uname -sr
Linux 3.11.9-200.fc19.x86_64
sh-4.2$ [[ -d bin ]] && echo "doubles work"
doubles work
sh-4.2$ [ -d bin ] && echo "single work"
single work
sh-4.2$

* I didnt use result/*/* at all
* Quotes, yes you're absolutly right, i was too lazy Smilie
* ((var++)) is my preference, at times i use let - and now figured its a bash-builtin.. thank you Smilie
# 12  
Old 01-26-2014
Dear Scrutinizer, Thank you so much for information...
my folder at this stage looks like
Quote:
Combine (main folder) == test.sh (the code) + result (the folder with N [4 for now] in it)
each file[e.g. april, may, june, july] has data separeated by "|" in 5 rows (costumer details)
and there might be duplications in both.
the code I have now with changes looks like
Code:
	FZ=(*)
	ZC=0 # Z Counter
	THESE=""
#
#	Action
#
	[[ -d output ]] || mkdir output

	for Z in "${FZ[@]}"
	do
		THESE=$Z
		for I in 1 2 3
		do		
			NUM=$[ $ZC + $I ]
			THESE+=" ${FZ[$NUM]}"
			str=$(echo "$THESE"|sed s,' ','-',g)
			cat $THESE|sort -u > output/${str}.txt
		done
		let ZC++ 
		THESE=""
	done

but it keeps giving me errors one way or another === as well as create wrong files as the code should be place in the same directory for this two work

I am now using a Linux Terminal
# 13  
Old 01-26-2014
Quote:
Originally Posted by sea
@ Scrutinizer
At times i bark back Smilie (meaning i want to be right)
Unix - Using Shell Arrays this tutorial says sh does support arrays.
Then that tutorial is wrong! /bin/sh on every POSIX-compliant modern system means a POSIX shell (or near POSIX), which is defined here: , except on Solaris <= 10 where it means the classic Bourne shell which also does not know arrays. On older legacy systems it also means Bourne Shell..
Quote:
Code:
sh-4.2$ uname -sr
Linux 3.11.9-200.fc19.x86_64
sh-4.2$ [[ -d bin ]] && echo "doubles work"
doubles work
sh-4.2$ [ -d bin ] && echo "single work"
single work
sh-4.2$

That is no proof. That just means that the shell you are using happens to understand more than the POSIX syntax. Try executing that with /bin/sh on Ubuntu or Debian and the former will fail...
Quote:
* I didnt use result/*/* at all
Yes you did in post #2
Quote:
* Quotes, yes you're absolutly right, i was too lazy Smilie
* ((var++)) is my preference, at times i use let - and now figured its a bash-builtin.. thank you Smilie
You are right, let is a bash-builtin (and a ksh93 builtin)...
Last edited by Scrutinizer; 01-26-2014 at 03:14 PM..
This User Gave Thanks to Scrutinizer For This Post:
sea
# 14  
Old 01-26-2014
Quote:
Originally Posted by A-V
Dear Scrutinizer, Thank you so much for information...
my folder at this stage looks like
Quote:
Combine (main folder) == test.sh (the code) + result (the folder with N [4 for now] in it)
each file[e.g. april, may, june, july] has data separeated by "|" in 5 rows (costumer details)
and there might be duplications in both.
the code I have now with changes looks like
[..]

I am now using a Linux Terminal
You're welcome, please be more specific and give examples and/or directory (tree) listings instead of an anecdotal description, also post the content of the files if that is relevant for you problem..

A Linux Terminal, so are you working on a Linux host? What is the shell you are using in your script? Can you show the shebang, or how do you execute the script?
Last edited by Scrutinizer; 01-26-2014 at 03:00 PM..
This User Gave Thanks to Scrutinizer For This Post:
A-V
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging two columns from two files with similar names into a loop

I have two files like this: fileA.net A B C fileA.dat 1 2 3 and I want the output output_expected A 1 B 2 C 3 I know that the easier way is to do a paste fileA.net fileA.dat, but the problem is that I have 10,000 couple of files (fileB.net with fileB.dat; fileC.net with... (3 Replies)
Discussion started by: valente
3 Replies

2. Shell Programming and Scripting

For loop for number of files in a folder

Hi All, Need a for loop which should run for number of files in a folder and should pass the file name as parameter to another shell script for each loop. Please help me. Thanks. (2 Replies)
Discussion started by: chillblue
2 Replies

3. Shell Programming and Scripting

For loop to read folder which are not under processing

Hi, I want to read folders which do not have a file "processing" in a for loop ordered by timestamp. Currently im doing like this. Like cd /home/working for i in `ls -c1` do some command... done I want to exclude folders which have that "processing" file. The directory... (2 Replies)
Discussion started by: chetan.c
2 Replies

4. Shell Programming and Scripting

File merging based on different counter loop

hello, File 1 main Group sub group MIT VAR_1D_DATA_TYPE 23-03-2012 MIT VAR_1D_DATA_TYPE 22-03-2012 MIT VAR_10D_DATA_TYPE 23-03-2012 MIT VAR_10D_DATA_TYPE 22-03-2012 MIT ... (0 Replies)
Discussion started by: manas_ranjan
0 Replies

5. Shell Programming and Scripting

Loop through text file > Copy Folder > Edit XML files in bulk?

I have a text file which contains lines in this format - it contains 105 lines in total, but I'm just putting 4 here to keep it short: 58571,east_ppl_ppla_por 58788,east_pcy_hd_por 58704,east_pcy_ga_por 58697,east_pcy_pcybs_por It's called id_key.txt I have a sample folder called... (9 Replies)
Discussion started by: biscuitcreek
9 Replies

6. Shell Programming and Scripting

Merging folder of files according to name and contents.

I have absolutely no idea how to do this and everything I have done doesn't even come close. Here's the scenario: There are a number of files in a folder named in this manner: agd.txt stv.txt frk.txt dqp.txt There is also a series of other files in the same folder with like file... (5 Replies)
Discussion started by: Trapper
5 Replies

7. Shell Programming and Scripting

File in a folder and loop until time

Hi All, I am stuck with this requirement. My requirement is that a job will run daily at anytime before and would look for a file named filename.txt in a folder and mv to a different folder and if not found until 9 A.M it will exit with a success code. Now I did the follwing ..but there... (8 Replies)
Discussion started by: RubinPat
8 Replies

8. Shell Programming and Scripting

Shell script for merging lines in a loop

Dear All, I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl). I/P File------------------------- APaul,,,,SDH,,23,,,PPH,,2 ,,,,KKH,,19,,,MMH,,12, ,,,,CCH,,22,,,MNH,,19, ,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies

9. Shell Programming and Scripting

File Management: How do I move all JPGS in a folder structure to a single folder?

This is the file structure: DESKTOP/Root of Photo Folders/Folder1qweqwasdfsd/*jpg DESKTOP/Root of Photo Folders/Folder2asdasdasd/*jpg DESKTOP/Root of Photo Folders/Folder3asdadfhgasdf/*jpg DESKTOP/Root of Photo Folders/Folder4qwetwdfsdfg/*jpg DESKTOP/Root of Photo... (4 Replies)
Discussion started by: guptaxpn
4 Replies

10. Shell Programming and Scripting

Parse the .txt file for folder name and FTP to the corrsponding folder.

Oracle procedure create files on UNIX folder on a regular basis. I need to FTP files onto windows server and place the files, based on their name, in the corresponding folders. File name is as follows: ccyymmddfoldernamefile.txt; Folder Name length could be of any size; however, the prefix and... (3 Replies)
Discussion started by: MeganP
3 Replies
Login or Register to Ask a Question