Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge files from different folders into one

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge files from different folders into one
# 1  
Old 08-20-2014
Merge files from different folders into one
Gents

I want to merge various files located in different folders into one folder.

Example I have 4 folders increment number example ( 1080 to 1084 ) where there is various files with the same name in each folder. Please I need to merge all files with the same name to other folder called 223.

Example

INPUT
folder 1
Code:
1080
223.txt
223.csv
223.prn

folder2
Code:
1081
223.txt
223.csv
223.prn

folder3
Code:
1082
223.txt
223.csv
223.prn

folder4
Code:
1083
223.txt
223.csv
223.prn

output
Code:
223
223.txt
223.csv
223.prn

Thanks for your support and help Smilie
# 2  
Old 08-20-2014
Quote:
Originally Posted by jiam912
I have 4 folders increment number example ( 1080 to 1084 )
Those are 5 folders, though Smilie

Try this for txt files, if it works, adjust it for csv and prn:
Code:
$ txt=223.txt; find 108[01234] -type f -name "$txt" -exec cat {} >>save/223/$txt +

# 3  
Old 08-21-2014
Dear jiam912,
I have a few to questions pose in response first:-
  • What have you tried so far?
  • What output/errors do you get?
  • Does the order of reading files matter? (ls or find & sort)
  • Does the order of the content matter? (Concatenate or sort)
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.


Regards,
Robin
# 4  
Old 08-21-2014
Dear rbatte1

I am trying to learn awk and I am using shell.. using linux redhat 5

This what I did till now to merge files .txt only

Use my script like this

Code:
script 1080 1084 223

Here is what I did
Code:
#!/bin/csh

set jd = $3
set fsw = $1
set lsw = $2
if ( $#argv == 1 ) then
    set fsw = $1
    set lsw = $1
endif
if ( $fsw > $lsw ) then
   set tmp = $fsw
   set fsw = $lsw
   set lsw = $tmp
endif

if ( $fsw > "0" ) then

foreach sw ( `awk 'BEGIN {for (i='$fsw';i<='$lsw';i++) printf ("%s\n", i) }'` )
        printf "\n/------------------------> sw : $sw\n"

cd /ect/usr/$jd
cd /ect/usr/$jd/$sw/
cp $jd.rps /ect/usr/$jd/$sw/$jd"_1".txt
cd /ect/usr/$jd/$sw/
end 
cd /ect/usr/$jd/$sw
cat *.txt > $jd.txt
cd /ect/usr/$jd

Thanks
Last edited by rbatte1; 08-21-2014 at 05:02 PM..
# 5  
Old 08-21-2014
Provided "merge" means "concatenate", and source file is $jd.txt not $jd.rps
Code:
...
cd /etc/usr/$jd || exit # must exist - or exit
# now we can refer files relative to the current work directory
foreach ext ( txt csv prn )
  printf "" > $jd.$ext # clear an old merge file
  foreach sw ( ... )
    printf "\n $sw/$jd.$ext\n"
    cat $sw/$jd.$ext >> $jd.$ext # append to merge file
  end
end

--
Consider rewriting your C-shell script to a shell script!
A shell script has increased robustness and allows a more efficient concatenation.
This User Gave Thanks to MadeInGermany For This Post:
jiam912
# 6  
Old 08-24-2014
Dear MadeInGermany

I have modified my script and it works fine.. But As I no to good in scripts I, kindly can you see the script and try to check if there is something extra.

When I merge the files each file has header starting with "H" letter I would like to delete all lines ^H in all files but from line 50 to the end.. can you please give and advise how I can do this.

here the script.
Code:
#!/bin/bash

clear
date

printf "==================================================================================================\n"

      printf "Please enter the swath numbers.\n"
      printf "\n"
             read -p "First swath: " fsw
      printf "\n"
             read -p "Last swath : " lsw
      printf "\n"
             read -p "Julian Day : " jd

printf "==================================================================================================\n"


cd $IMP1_DIR/$jd 
rm -fR mkdir FF_files
mkdir FF_files

tt="FF_files"

for ext in  rps sps xps aps cog raw
do
  printf "" > $IMP1_DIR/$jd/$tt/$jd.$ext 

  for sw in  `awk 'BEGIN {for (i='$fsw';i<='$lsw';i++) printf ("%s\n", i) }'`

   do

file=$jd.$ext 

cd $IMP1_DIR/$jd/"sw"$sw
if [[ ! -f $file  ]] ; then
echo  "***" File "$file" not found in swath $sw , aborting. 
exit
else
	if [ ! -s $file ] ; then
echo  "***" File $jd.$ext in swath $sw is empty. aborting.
exit
	fi
fi

if [[ $file == $jd.rps  ]] ; then

cat $jd.rps | awk 'FNR==50 {print $0}' | awk '{\
                  ptype = substr($0,1,1);\
		  if (ptype != "R") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi

if [[ $file == $jd.sps  ]] ; then

cat $jd.sps | awk 'FNR==50 {print $0}' | awk '{\
                  ptype = substr($0,1,1);\
		  if (ptype != "S") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi
if [[ $file == $jd.xps  ]] ; then

cat $jd.xps | awk 'FNR==50 {print $0}' | awk '{\
                  ptype = substr($0,1,1);\
		  if (ptype != "X") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi
if [[ $file == $jd.aps  ]] ; then

cat $jd.aps | awk 'FNR==50 {print $0}' | awk '{\
                  ptype = substr($0,1,1);\
		  if (ptype != "A") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi
if [[ $file == $jd.cog  ]] ; then

cat $jd.cog | awk 'FNR==50 {print $0}' | awk '{\
                  ptype = substr($0,1,1);\
		  if (ptype != "C") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi
if [[ $file == $jd.raw  ]] ; then

cat $jd.raw | awk 'FNR==1 {print $0}' | awk '{\
                  ptype = substr($0,1,3);\
		  if (ptype != "Obs") {\
                                      printf ("    Type point WRONG FORMAT in file '$file' swath %4d\n",'$sw' )}}'
fi

cd $IMP1_DIR/$jd/"sw"$sw

cat $IMP1_DIR/$jd/"sw"$sw/$jd.$ext >> $IMP1_DIR/$jd/$tt/$jd.$ext

sed '50,10000{/^H/d;}' $jd.rps
sed '50,10000{/^H/d;}' $jd.sps
sed '50,10000{/^H/d;}' $jd.xps
sed '50,10000{/^H/d;}' $jd.sps
sed '50,10000{/^H/d;}' $jd.cog

  done
done
printf "==================================================================================================\n"
echo          New files were created under $IMP1_DIR/$jd/$tt
printf "==================================================================================================\n"


date

# 7  
Old 08-25-2014
Some comments on your script (without entirely understanding what you're about):

- rm -fR mkdir FF_files doesn't work on mkdir; -f suppresses error output. (And, to save some chars typed, why not use tt=FF_files; rm -R $tt; mkdir $tt?)


- why not use seq $fsw $lsw instead of`awk 'BEGIN {for (i='$fsw';i<='$lsw';i++) printf ("%s\n", i) }'` ?

- the entire sequence of if blocks like
Code:
if [[ $file == $jd.rps  ]] ; then  cat $jd.rps | awk 'FNR==50 {print $0}' | awk '{ptype = substr($0,1,1); if (ptype != "R") { printf ("Type point WRONG FORMAT ..." )}}' fi

could be replaced - if I understand your logics correctly - by one single
Code:
awk -v PAR=${TYP[$ext]} 'FNR==4 && ! ($0 ~ "^"PAR){print "type WRONG etc..."} $file'

, provided you define an associative array upfront:declare -A TYP=( [rps]=R [sps]=S [xps]=X [aps]=A [cog]=C [raw]=Obs ) (and you could use
Code:
echo ${!TYP[@]}
rps raw sps aps cog xps

to list all extensions in the for loop...)


- the sed '50,10000{/^H/d;}' $jd.xxx are not working on the right directory, and they could make use of $ext as well, and, why stop at line 10000 and not work through the EOF by using 50,$, and they print to the screen instead of a file, and you could issue them after all loops have finished.
This User Gave Thanks to RudiC For This Post:
jiam912
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to copy files/folders and show the files/folders?

Hi, So i know we use cp -r as a basic to copy folders/files. I would like this BUT i would like to show the output of the files being copied. With the amazing knowledge i have i have gone as far as this: 1) find source/* -exec cp -r {} target/ \; 2) for ObjectToBeCopied in `find... (6 Replies)
Discussion started by: Imre
6 Replies

2. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

3. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

4. Shell Programming and Scripting

List all the files in the present path and Folders and subfolders files also

Hi, I need a script/command to list out all the files in current path and also the files in folder and subfolders. Ex: My files are like below $ ls -lrt total 8 -rw-r--r-- 1 abc users 419 May 25 10:27 abcd.xml drwxr-xr-x 3 abc users 4096 May 25 10:28 TEST $ Under TEST, there are... (2 Replies)
Discussion started by: divya bandipotu
2 Replies

5. UNIX for Dummies Questions & Answers

Searching for folders/parent folders not files.

Hello again, A little while back I got help with creating a command to search all directories and sub directories for files from daystart of day x. I'm wondering if there is a command that I've overlooked that may be able to search for / write folder names to an output file which ideally... (2 Replies)
Discussion started by: Aussiemick
2 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Move files to Folders

Hi Friends, Below is my requirement and i am not clear how to approach this issue in unix programming. I have a folder with 2500 files. The files are in below format. 1234_name1.txt 1234_name123.txt 4567_name1.txt 4567_name123.txt and i need a program which will read each file from this... (5 Replies)
Discussion started by: diva_thilak
5 Replies

8. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

9. HP-UX

to get the timestamp of files from the files and folders in Unix

Hi, I had a directory and many subdirectories and files with in it. Now i want to get the timestamp of files from the files and folders recursively. :( Please help me to generate a script fort he above mentioned requirement! Appreciate for ur qick response Thanks in advance! ... (2 Replies)
Discussion started by: kishan
2 Replies

10. Shell Programming and Scripting

removing old files except configuration files and folders

Dear all, I want to remove files older than 2 months in the /home/member directory. But except the configuration files (like .bash_profile .config/ .openoffice/ .local/ .kde/ etc..) I have tried with the command find . -mtime +60 -wholename './.*' -prune -o -print -exec mv {} \; but it... (1 Reply)
Discussion started by: jamcalicut
1 Replies
Login or Register to Ask a Question