How to combine data files using for loop


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to combine data files using for loop
# 1  
Old 07-28-2008
How to combine data files using for loop

Hi,

I have 5 files basically;namely file1.txt situated each at folder A to E respectively.

I would like to extract out third column from each of these file1.txt from folder A to folder E. Also, I wanted to extract the first and second column which are common. In other words, e.g

A/file1.txt
1 1 10

B/file1.txt
1 1 20

C/file1.txt
1 1 30

D/file1.txt
1 1 40

E/file1.txt
1 1 50

I tried

dirs=(A,B,C,D,E);
files=(file1.txt);

for file in "${files[@]}";

do for dir in "${dirs[@]}";
do awk {'print $3 '} "$dir/$file" >test.txt

done

done

This only able to print out the 3rd column separately. Ultimately, I wanted to have the below output in an external file (test.txt):-
1 1 10 20 30 40 50

The file1.txt might have more than one line. The above file1.txt data is just an example.

Thanks.
# 2  
Old 07-28-2008
Try this:

Code:
dirs=(A,B,C,D,E);
files=(file1.txt);

for file in "${files[@]}";
do
    v=$(awk '{print $1,$2}' ${dirs[1]}/$file)
    for dir in "${dirs[@]}";
    do
        v="$v $(awk '{ print $3 }' $dir/$file)"
    done
    echo "$v" >> test.txt
done

I'm presuming the "1 1" on the beginning is the same in all files, and you just obtain it from the file in the first directory.
# 3  
Old 07-28-2008
Not nice but works:
Code:
awk '{print $1,$2; while (getline) {print $3} }' `find A B C D E -type f -name "file1.txt" -print`| sed -e :a -e '{ N; s/\n/ /g;ta }'

1 1 20 30 40 50

# 4  
Old 07-28-2008
Hi Annihilannic.

I still could not get the right output.

It doesnt combine the third element until the last line read from the first directory's file.

I modified to be something like below:-
Code:
dirs=(A B C D E); 
files=(file1.txt); 

for file in "${files[@]}"; 
do
     v=$(awk '{print $1,$2}' ${dirs[1]}/$file)
	  
     for dir in "${dirs[@]}"; 
	do 
	    w=$(awk '{print $5 }' $dir/$file)
         
        done
        echo "$v $w" >> ../test.txt
done

But its not working too, Basically i wanted to have 1 1 20(from dir A) 30(from dir B) 40 (from dir C) and 50(from dir D) corresponded with respect to each line.

E.g :-

A/file1.txt
1 1 10
2 2 100
B/file1.txt
1 1 20
2 2 1000

C/file1.txt
1 1 30
2 2 10000

D/file1.txt
1 1 40
2 2 100000

E/file1.txt
1 1 50
2 2 1000000

Output:- 1 1 10 20 30 40 50
2 2 100 1000 10000 100000 1000000

Please advise. Thanks.
# 5  
Old 07-28-2008
Try (and adapt) the following script :
Code:
awk -v File=file1.txt -v Dirs_list="A,B,C,D,E" '
BEGIN {
   dirs_count = split(Dirs_list, dirs, ",");
   for (i=1; i<=dirs_count; i++)
      files[i] = dirs[i] "/" File;
   while (getline < files[1]) {
      out = $1 OFS $2 OFS $3;
      for (i=2; i<=dirs_count; i++) {
         getline < files[i];
         out = out OFS $3;
      }
      print out
   }
}
'

Input files:
Code:
> for dir in A B C D E
> do
> echo "--- $dir/file1.txt ---"
> cat $dir/file1.txt
> done
--- A/file1.txt ---
1 1 A11
1 2 A12
--- B/file1.txt ---
1 1 B11
1 2 B12
--- C/file1.txt ---
1 1 C11
1 2 C12
--- D/file1.txt ---
1 1 D11
1 2 D12
--- E/file1.txt ---
1 1 E11
1 2 E12
>

Output:
Code:
1 1 A11 B11 C11 D11 E11
1 2 A12 B12 C12 D12 E12

Jean-Pierre.
# 6  
Old 07-28-2008
Hi Jean,

Thanks for the reply. I havent try the suggested solution but I am thinking how is that possible if i have about 30 files to perform the above operation.

Is that possible the first solution suggested by Annihilannic to be tweaked ?

Please advise. Thanks.
# 7  
Old 07-28-2008
Another way :

Code:
for dir in A B C D E
do
   echo $dir/file1.txt
done | 
xargs paste |
awk '{out = $1 OFS $2; for (i=3; i<=NF; i+=3) out = out OFS $i; print out}'

Execution (step by step):
Code:
> for dir in A B C D E
do
    echo $dir/file1.txt
done
A/file1.txt
B/file1.txt
C/file1.txt
D/file1.txt
E/file1.txt
>

Code:
> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs echo paste  # echo for debug purpose
paste A/file1.txt B/file1.txt C/file1.txt D/file1.txt E/file1.txt
>

Code:
> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs paste
1 1 A11 1 1 B11 1 1 C11 1 1 D11 1 1 E11
1 2 A12 1 2 B12 1 2 C12 1 2 D12 1 2 E12
>

Code:
> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs paste |
awk '{out = $1 OFS $2; for (i=3; i<=NF; i+=3) out = out OFS $i; print out}'
1 1 A11 B11 C11 D11 E11
1 2 A12 B12 C12 D12 E12
>

Jean-Pierre.

Last edited by aigles; 07-28-2008 at 09:06 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. Shell Programming and Scripting

Combine data out of 3 files into one new file

Hi, How can I combine the data of of three files into one new file? I try to give as much informations as possible. The three existing files are called file1 file2 and file3 the new file should named output_combined. The size of the files will be around 900 words/lines each .. but always... (5 Replies)
Discussion started by: MyMemberName
5 Replies

3. Shell Programming and Scripting

Combine data from two files base on uniq data

File 1 ID Name Po1 Po2 DD134 DD134_4A_1 NN-1 L_0_1 DD134 DD134_4B_1 NN-2 L_1_1 DD134 DD134_4C_1 NN-3 L_2_1 DD142 DD142_4A_1 NN-1 L_0_1 DD142 DD142_4B_1 NN-2 L_1_1 DD142 DD142_4C_1 NN-3 L_2_1 DD142 DD142_3A_1 NN-41 L_3_1 DD142 DD142_3A_1 NN-42 L_3_2 File 2 ( Combination of... (1 Reply)
Discussion started by: pareshkp
1 Replies

4. Shell Programming and Scripting

Combine/omit data from 2 files

i made a script on my own. this is for the inventory to all of my AWS servers, and i run it to all of my servers to get the hostname, please look at file2. Then i need some data in file3 as well,. i need to combine them #cat file1 192.10.1.41 server.age.com ###### 192.10.0.40 ssh cant... (10 Replies)
Discussion started by: kenshinhimura
10 Replies

5. Shell Programming and Scripting

How to combine the data of files?

I have a main file as follows aaa 3/2 = 1.5 aba 55+6 = 61 aca 67+8 = 75 hjk 3+3 = 67 ghd 66+30 = 96 ghj 99-3 = 96 ffg 67+3 = 70 I have 4 sub files named sub1, sub2, sub3, sub4 content of sub1 aaa 23+5 = 28 hjk 45+6 = 51 ghd 40-20 = 20 ... (2 Replies)
Discussion started by: jackevan
2 Replies

6. Shell Programming and Scripting

How to combine data from 2 file ? How to print ?

Dear friends, I am just trying to print data from 2 file,namely file_1.txt and file_1.dat (specific column of ROW 1) file_1.txt 12 13 14 15 99 AMC 69 36 89 12 13 14 15 99 AMC 69 84 -12 12 13 14 ... (6 Replies)
Discussion started by: nex_asp
6 Replies

7. Shell Programming and Scripting

get data from files combine them to a file

hi people; this is my file1.txt:192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 ... this is my file2.txt:portnames usernames maxusercap ... i want to write to file3.txt:l ./getports 192.168.1.1 'get all;l+;get . portnames;l-' l ./getports 192.168.1.1 'get all;l+;get . usernames;l-'... (4 Replies)
Discussion started by: gc_sw
4 Replies

8. Shell Programming and Scripting

combine data of 2 files by variable

my first post ... please be gentle. I have been working on a script to get info out of mysql. Its a support ticket system database OTRS. I can write the subject of open tickets to a text file with a unique user id. I also have a text file with the unique user id, username and email adres. I... (11 Replies)
Discussion started by: dicenl
11 Replies

9. UNIX for Dummies Questions & Answers

Recover data from 2 files then combine

Using dd or similar tools to recover data from 2 damaged cdroms, I need a way to then combine the 2 files, 1 from each cd, and make a good file: this all result from finding that certain cd's tops scratch easily even when using the "proper" cd markers, hence making the file useless, however the... (1 Reply)
Discussion started by: saint65
1 Replies

10. Shell Programming and Scripting

i want to combine two awk scripts which is having same loop and filelist

hi, to all give me some idea below scripts file=`cat filelist` for filename in $file do awk '{ if ($0 ~ /system/ && flag == 0) { flag=1; print $0; } else if ($0 ~ /system/ && flag == 1) { printf("%s SYSLAY %s %s %s\n",$1,$3, $4, $5); } else print $0; }' $filename >... (6 Replies)
Discussion started by: LAKSHMI NARAYAN
6 Replies
Login or Register to Ask a Question