Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge files from different folders into one

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge files from different folders into one
# 8  
Old 08-25-2014
RudiC
Thanks a lot for your help
I have change some of your advises in my script but I don't understand how to do the associative array upfront... Smilie
Please can you modify my script..

About to delete the lines content "H" after row 50,, There is a way to do it touching the file and done the change in the same file in order to no create a new cleaned.

Again Thanks for your support

Code:
#!/bin/bash


      printf "Please enter the swath numbers.\n"
      printf "\n"
             read -p "First swath: " fsw
      printf "\n"
             read -p "Last swath : " lsw
      printf "\n"
             read -p "Julian Day : " jd

cd $IMP1_DIR/$jd 

tt="QC_Files"; rm -fR $tt; mkdir $tt

for ext in  rps sps xps aps cog raw
do
  printf "" > $IMP1_DIR/$jd/$tt/$jd.$ext 

  for sw in `seq $fsw $lsw`

   do

file=$jd.$ext 

cd $IMP1_DIR/$jd/"sw"$sw
if [[ ! -f $file  ]] ; then
echo  "***" File "$file" not found in swath $sw , aborting. 
exit
else
	if [ ! -s $file ] ; then
echo  "***" File $jd.$ext in swath $sw is empty. aborting.
exit
	fi
fi

declare -A TYP=([rps]=R [sps]=S [xps]=X [aps]=A [cog]=C [raw]=Obs )
echo ${!TYP[@]}
rps sps xps aps cog raw
awk -v PAR=${TYP[$ext]} 'FNR==50 && ! ($0 ~ "^"PAR) {print "Type WRONG FORMAT in file '$file' '$sw'" }' $file 

cd $IMP1_DIR/$jd/"sw"$sw

cat $IMP1_DIR/$jd/"sw"$sw/$jd.$ext >> $IMP1_DIR/$jd/$tt/$jd.$ext



  done
done

sed '50,${/^H/d;}' $jd.$ext

# 9  
Old 08-25-2014
You were pretty close:

Code:
#!/bin/bash


printf "Please enter the swath numbers.\n\n"
read -p "First swath: " fsw
printf "\n"
read -p "Last swath : " lsw
printf "\n"
read -p "Julian Day : " jd

cd $IMP1_DIR/$jd

tt="QC_Files"; rm -fR $tt; mkdir $tt

declare -A TYP=([rps]=R [sps]=S [xps]=X [aps]=A [cog]=C [raw]=Obs )
for ext in  ${!TYP[@]}
do
  printf "" > $IMP1_DIR/$jd/$tt/$jd.$ext

  for sw in `seq $fsw $lsw`
  do

     file=$jd.$ext
     cd $IMP1_DIR/$jd/"sw"$sw
     if [[ ! -f $file  ]] ; then
         echo  "***" File "$file" not found in swath $sw , aborting.
         exit
     fi
     if [ ! -s $file ] ; then
         echo  "***" File $jd.$ext in swath $sw is empty. aborting.
         exit
     fi

     awk -v PAR=${TYP[$ext]} 'FNR==50 && ! ($0 ~ "^"PAR) {print "Type WRONG FORMAT in file '$file' '$sw'" }' $file

     cd $IMP1_DIR/$jd/"sw"$sw

     cat $IMP1_DIR/$jd/"sw"$sw/$jd.$ext >> $IMP1_DIR/$jd/$tt/$jd.$ext

  done
done

sed '50,${/^H/d;}' $jd.$ext

This User Gave Thanks to Chubler_XL For This Post:
jiam912
# 10  
Old 08-26-2014
Not sure I understood your logics completely. Check if this can help you.
Code:
#!/bin/bash

printf "Please enter the swath numbers.\n\n"
read -p "First swath: " fsw
printf "\n"
read -p "\nLast swath : " lsw
printf "\n"
read -p "Julian Day : " jd


cd $IMP1_DIR/$jd

tt="QC_Files"; rm -fR $tt; mkdir $tt

declare -A TYP=([rps]=R [sps]=S [xps]=X [aps]=A [cog]=C [raw]=Obs )

for ext in  ${!TYP[@]}
     do file="$jd.$ext" 
        touch "$tt/$file" 

        for sw in $(seq $fsw $lsw)
             do cd "sw"$sw
                [ -f "$file"  ] || { echo  "***" File $file not found in swath $sw, aborting.; exit; }
                [ -s "$file"  ] || { echo  "***" File $file in swath $sw is empty. aborting.; exit; } 
                awk -v PAR=${TYP[$ext]} 'FNR==50 && ! ($0 ~ "^"PAR) {print "Type WRONG FORMAT in file " FILENAME " " SW }' SW="$sw"  $file 
                cd ..

                cat "sw"$sw/$jd.$ext >> $tt/$jd.$ext
             done
             
        sed '50,${/^H/d;}' $jd.$ext
     done

sed on some systems has the -i option to replace files "in place". Going a bit deeper into the problem, you could do all of it (appending to the central file, deleting H- records) in the one awk command.
This User Gave Thanks to RudiC For This Post:
jiam912
# 11  
Old 08-26-2014
Dears XL and RudiC

Thanks a lot for your support.

The system give a error :
Code:
declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

# 12  
Old 08-26-2014
Note: declare -A is bash 4 . It does not work in bash 3 or lower.
# 13  
Old 08-26-2014
Not all is lost. Try to replace that associative array stuff with sth like
Code:
EXTARR=(rps sps xps aps cog raw)
TYPARR=(R S X A C Obs)
for ((IDX=1; IDX<${#EXTARR[@]}; IDX++)) 
  do ext=${EXTARR[$IDX]}

and then, in the awk line,
Code:
-v PAR=${TYPARR[$IDX]}

This User Gave Thanks to RudiC For This Post:
jiam912
# 14  
Old 08-27-2014
Dear Rudi C.
Thanks a lot for your help and support.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to copy files/folders and show the files/folders?

Hi, So i know we use cp -r as a basic to copy folders/files. I would like this BUT i would like to show the output of the files being copied. With the amazing knowledge i have i have gone as far as this: 1) find source/* -exec cp -r {} target/ \; 2) for ObjectToBeCopied in `find... (6 Replies)
Discussion started by: Imre
6 Replies

2. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

3. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

4. Shell Programming and Scripting

List all the files in the present path and Folders and subfolders files also

Hi, I need a script/command to list out all the files in current path and also the files in folder and subfolders. Ex: My files are like below $ ls -lrt total 8 -rw-r--r-- 1 abc users 419 May 25 10:27 abcd.xml drwxr-xr-x 3 abc users 4096 May 25 10:28 TEST $ Under TEST, there are... (2 Replies)
Discussion started by: divya bandipotu
2 Replies

5. UNIX for Dummies Questions & Answers

Searching for folders/parent folders not files.

Hello again, A little while back I got help with creating a command to search all directories and sub directories for files from daystart of day x. I'm wondering if there is a command that I've overlooked that may be able to search for / write folder names to an output file which ideally... (2 Replies)
Discussion started by: Aussiemick
2 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Move files to Folders

Hi Friends, Below is my requirement and i am not clear how to approach this issue in unix programming. I have a folder with 2500 files. The files are in below format. 1234_name1.txt 1234_name123.txt 4567_name1.txt 4567_name123.txt and i need a program which will read each file from this... (5 Replies)
Discussion started by: diva_thilak
5 Replies

8. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

9. HP-UX

to get the timestamp of files from the files and folders in Unix

Hi, I had a directory and many subdirectories and files with in it. Now i want to get the timestamp of files from the files and folders recursively. :( Please help me to generate a script fort he above mentioned requirement! Appreciate for ur qick response Thanks in advance! ... (2 Replies)
Discussion started by: kishan
2 Replies

10. Shell Programming and Scripting

removing old files except configuration files and folders

Dear all, I want to remove files older than 2 months in the /home/member directory. But except the configuration files (like .bash_profile .config/ .openoffice/ .local/ .kde/ etc..) I have tried with the command find . -mtime +60 -wholename './.*' -prune -o -print -exec mv {} \; but it... (1 Reply)
Discussion started by: jamcalicut
1 Replies
Login or Register to Ask a Question