Combine columnes from different files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combine columnes from different files
# 1  
Old 03-16-2011
Combine columnes from different files

Hi all,
I found some similar probs and solutions but not working for my case. I have multiple files like this:


file1:
Code:
g1.1 2
g2.2 4
g2.1 5
g4.1 3

file2:
Code:
g1.1 2
g1.2 3
g4.1 4

file3:
Code:
g1.2 3
g5.1 3

I need out put like
Code:
gene file1 file2 file3
g1.1 2 2 -
g1.2 - 3 3
g2.2 4 - -
g2.1 5 - -
g4.1 3 4 -
g5.1 - - 3

So compare the first columns and collect the value for that from files. if the entry is missing then it should give -. Most of the entries will be shared between files but some of them are missing. I have around 8 files like this with two columns .

Thanks in advance

Moderator's Comments:
Mod Comment Please use [code] and [/code] tags when posting code, data or logs etc. to preserve formatting and enhance readability, thanks.

Last edited by zaxxon; 03-16-2011 at 09:54 AM.. Reason: code tags
# 2  
Old 03-16-2011
take a look below statements
Code:
kent$ awk 'ARGIND==1{a[$1]=$0 " - -";next;} 
ARGIND==2{ if($1 in a)sub(/ - /, " "$2" ",a[$1]);else a[$1]=$1" - "$2" -";next; }
ARGIND==3{if($1 in a)sub(/ -$/," "$2" ",a[$1]); else a[$1]=$1" - - "$2;}
END{for(i in a)print a[i]}' file1 file2 file3 |sort
g1.1 2 2 -
g1.2 - 3 3 
g2.1 5 - -
g2.2 4 - -
g4.1 3 4 -
g5.1 - - 3

# 3  
Old 03-16-2011
@sk1418: thanks for the help, but I may have more file then I think i have to extent the script with more ARGIND check. Is it possible to make it some loop or something? I think I have to check with ARGIND==0 ?

Even then I am only getting the last file counts and for the rest just a - .
# 4  
Old 03-16-2011
hi, this script is just for 3 or less then 3 files as input. If you have more files, the logic needs to be changed as well, not only the ARGIND part. say you have 10 files, then one line could be
Code:
g5.1 - - - - - 7 - - - -

the script above cannot give you that output. So "extend" doesn't work. you have to rewrite.

awk can do this, but not so easy, at least I haven't thought a shortcut for that, glad to know if someone has. You may try write a python script. I think that wouldn't be complicated.

---------- Post updated at 14:45 ---------- Previous update was at 14:07 ----------

here is the py script to do your job.

PHP Code:
#!/usr/bin/python
files=['file1','file2','file3']
dict={}
len files.__len__()
for 
s in files:
    
idx files.index(s)
    
open(s)
    
line = [x.replace("\n","") for x in f.readlines()]
    for 
l in line:
        
k,l.split(" ")[0], l.split(" ")[1]
        if(
not dict.has_key(k)):
            
dict[k] = list("-"*len)
        
dict[k][idx] = v
    f
.close()

keys dict.keys()
keys.sort()
for 
k in keys:
    print 
k+" "reduce(lambda x,y" " y,dict[k]) 
save above codes to a .py file. e.g. t.py
then:
Code:
kent$ python t.py
g1.1 2 2 -
g1.2 - 3 3
g2.1 5 - -
g2.2 4 - -
g4.1 3 4 -
g5.1 - - 3

you could change the "files" list, add as many files as you want. of course, you can get the files list by a given path. e.g. os.listdir(path)...
# 5  
Old 03-16-2011
How about perl?
Invocation
Code:
perl merger.pl file1 .. fileN

merger.pl
Code:
#!/usr/bin/perl

while (<@ARGV>){
open (FH,$_)|| die "FAIL- $!\n";
while (<FH>) {
        @fld=split;
        if ($i > 0) {
                if (exists  $hash{$fld[0]} ) {
                        $sc= scalar @{$hash{$fld[0]}};
                        if ( $sc == $i ) {
                        push(@{$fld[0]},$fld[1]);
                        } else {
                        $sc=scalar @{$hash{$fld[0]}};
                        $sc=$i - $sc;
                        for($j=1;$j<=$sc;$j++) { push(@{$fld[0]},"-");}
                        push(@{$fld[0]},$fld[1]);
                        }
                } else {
                $sc=$i;
                for($j=1;$j<=$sc;$j++) { push(@{$fld[0]},"-");}
                push(@{$fld[0]},$fld[1]);
                $hash{$fld[0]}=\@{$fld[0]};
                }
        } else {
        push(@{$fld[0]},$fld[1]);
        $hash{$fld[0]}=\@{$fld[0]};
        }
}
$i=$i+1;
close(FH);
}

foreach (keys(%hash)) {
        if (scalar @{$hash{$_}} != $i ) {
        $sc=$i-scalar @{$hash{$_}};
        for($j=1;$j<=$sc;$j++) { push(@{$hash{$_}},"-");}
        }
}

print $_," ","@{$hash{$_}}","\n" foreach (keys(%hash));

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. Shell Programming and Scripting

Combine files

I have n of files with ending with _ZERO.txt need to combine all file ending with _ZERO.txt into 1 file ex: A_ZERO.txt 1 2 B_ZERO.txt 3 4 Output: FINAL.txt 1 2 (3 Replies)
Discussion started by: satish1222
3 Replies

3. Shell Programming and Scripting

combine two files...

Hi, i have two files. i want to combine records from these two files in below manner :- first line from first file(1st line) 2nd line from 2nd file(1st line) 3rd line from 1st file(2nd line) 4th line from 2nd file(2nd line) so on.... (1 Reply)
Discussion started by: deepakiniimt
1 Replies

4. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

5. Shell Programming and Scripting

3 files combine into one help please

Ok here is what I have file a {{BEGIN}} {{FAX File b 5555464584 5555465292 5555465828 5555485930 5555474939 File C }} ON ORDERS LESS THAN 70 LBS AND THE PACKAGE IS A COMBINED LENGTH AND GIRTH EQUAL TO OR LESS THAN 108" PLEASE UTILIZE UPS. ... (4 Replies)
Discussion started by: sctxms
4 Replies

6. Shell Programming and Scripting

Combine new files

Hi All , Any one help me to combine two files in shell scripting .Below is my requrement i have 2 files as follows filea newyork America Texas America london Engalad Fileb abc def xyz i have to combine this file as follows newyork America abc Texas ... (1 Reply)
Discussion started by: ajmalc
1 Replies

7. HP-UX

How to combine 2 different files

Hi : I have a file containing the print queues with their IP address. I wanted to combine the 'lpstat' output with their respective IP address. For example : zebhtrmb-6078 lgonzale priority 0 Mar 17 11:50 on zebhtrmb with zebhtrmb-6078 lgonzale priority 0 ... (1 Reply)
Discussion started by: rdasari
1 Replies

8. Shell Programming and Scripting

Need To Combine 2 Files

I have 2 files that I need to combine. One file is looks like this: 71664107;1;1;05-FEB-07;12-FEB-07; The other file looks like this: U;71664107;dummy;Pirovano;M;04-SEP-75;Georgia;MI;1;1;31;S;S;;;Y;05-02-2007;0;12-FEB-07; I need to combine both files together. I need the shorter... (4 Replies)
Discussion started by: goodmis
4 Replies

9. Shell Programming and Scripting

combine two files

I have two files: file1 and file2 the content of files1 is: 13 22 333 42 56 55 ... the content of file2 is: aa dd cc ee ff gg ... (1 Reply)
Discussion started by: fredao1
1 Replies

10. Shell Programming and Scripting

How to combine 2 files

hi all i have 2 files f1 and f2 i have to combine these 2 files and make a new file f3 when i use paste f1 f2 >f3 its pasting vertically but i want to paste horizontally How to do .. pls let me know (2 Replies)
Discussion started by: ravi.sadani19
2 Replies
Login or Register to Ask a Question