Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Merge two text files by two fields and mixed output

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Merge two text files by two fields and mixed output
# 1  
Old 03-24-2016
Merge two text files by two fields and mixed output
Hello,
I'm back again looking for your precious help-

This time I need to merge two text files with matching two fields, output only common records with mixed output.

Let's look at the example:

FILE1
Code:
56153;AAA0708;3;TEST1TEST1;
89014;BBB0708;3;TEST2TEST2;
89014;BBB0708;4;TEST3TEST3;

FILE2
Code:
56153;AAA0708;7101875;
56153;AAA0708;7101361;
56153;BBB0708;7561171;
89014;BBB0708;7104873;
89014;BBB0708;7104874;
90676;AAA0708;7206967;

EXPECTED OUTPUT
Code:
7101875;3;TEST1TEST1;56153;AAA0708;
7101361;3;TEST1TEST1;56153;AAA0708;
7104873;3;TEST2TEST2;89014;BBB0708;
7104873;4;TEST3TEST3;89014;BBB0708;
7104874;3;TEST2TEST2;89014;BBB0708;
7104874;4;TEST3TEST3;89014;BBB0708;

I tried with awk, I can merge the files, but I cannot drive the output as I need.
Code:
awk  '{FS=";"} NR==FNR {a[$1]=$3";"$4;next} ($1 in a) {print $3";"a[$1]";"$1";"$2";"}'   FILE1 FILE2 > FILEOUT

Thanks a lot for any help Smilie
Last edited by emare; 03-24-2016 at 01:12 PM..
# 2  
Old 03-24-2016
Your expected output is not that easy to achieve:
- why and how did you suppress the third line starting with 56153?
- as the index of array a is just $1, the first element for 89014 will be overwritten by the second.
# 3  
Old 03-24-2016
In addition:
Code:
awk  'BEGIN{FS=";"} .....

# 4  
Old 03-24-2016
Hi,

I suppressed
56153;BBB0708;7561171;
because there is no match in FILE1 for 56153;BBB0708;

I achieved a better result with following command, but still I miss two records:
Code:
awk  '{FS=";"} NR==FNR {a[$1,$2]=$3";"$4;next} ($1,$2) in a {print $3";"a[$1,$2]";"$1";"$2";"}' FILE1 FILE2

# 5  
Old 03-24-2016
How about
Code:
awk  '
NR==FNR         {a[$1] = a[$1] $3 FS $4 SUBSEP
                 next
                }
($1 in a)       {n = split (a[$1], T, SUBSEP)
                 for (i=1; i<n; i++) print $3, T[i], $1, $2, ""
                }
' FS=";" OFS=";" file1 file2
7101875;3;TEST1TEST1;56153;AAA0708;
7101361;3;TEST1TEST1;56153;AAA0708;
7561171;3;TEST1TEST1;56153;BBB0708;
7104873;3;TEST2TEST2;89014;BBB0708;
7104873;4;TEST3TEST3;89014;BBB0708;
7104874;3;TEST2TEST2;89014;BBB0708;
7104874;4;TEST3TEST3;89014;BBB0708;

# 6  
Old 03-24-2016
A bit of Perl?

Copy to emare.pl
Run as perl emare.pl file1 file2

Code:
#!/usr/bin/perl
use strict;

my %db = ();
my $file = $ARGV[0];
while (<>) {
    my ($k, $v) = /^(\d+?;\w+?;)(.*;)$/;
    if($ARGV eq $file){
        push @{$db{$k}}, $v;
    }
    elsif($db{$k}) {
        for my $i (@{$db{$k}}) {
            print "$v$i$k\n";
        }
    }
}

Code:
perl emare.pl file{1,2}
7101875;3;TEST1TEST1;56153;AAA0708;
7101361;3;TEST1TEST1;56153;AAA0708;
7104873;3;TEST2TEST2;89014;BBB0708;
7104873;4;TEST3TEST3;89014;BBB0708;
7104874;3;TEST2TEST2;89014;BBB0708;
7104874;4;TEST3TEST3;89014;BBB0708;

---------- Post updated at 08:54 PM ---------- Previous update was at 08:45 PM ----------

Quote:
Originally Posted by RudiC
How about
Code:
awk  '
NR==FNR         {a[$1] = a[$1] $3 FS $4 SUBSEP
                 next
                }
($1 in a)       {n = split (a[$1], T, SUBSEP)
                 for (i=1; i<n; i++) print $3, T[i], $1, $2, ""
                }
' FS=";" OFS=";" file1 file2
7101875;3;TEST1TEST1;56153;AAA0708;
7101361;3;TEST1TEST1;56153;AAA0708;
7561171;3;TEST1TEST1;56153;BBB0708;
7104873;3;TEST2TEST2;89014;BBB0708;
7104873;4;TEST3TEST3;89014;BBB0708;
7104874;3;TEST2TEST2;89014;BBB0708;
7104874;4;TEST3TEST3;89014;BBB0708;

I thought that one was not supposed to be there. According to the example in post #1 and post #4 explanation.
Last edited by Aia; 03-25-2016 at 10:24 AM.. Reason: Corrects filename
# 7  
Old 03-25-2016
emare outpaced me with his/her post#4 indicating $1 and $2 should be the key. Modify like
Code:
awk  '
NR==FNR         {a[$1,$2] = a[$1,$2] $3 FS $4 ":"
                 next
                }
($1,$2) in a    {n = split (a[$1,$2], T, ":")
                 for (i=1; i<n; i++) print $3, T[i], $1, $2, ""
                }
' FS=";" OFS=";" file1 file2

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Merge two text files (oh no, not again!)

Hello, I'm new to this forum. I have always made good use of all the wise hints shown here. But this time I'm struggling with an issue that is driving me crazy. I have two text files, I have to merge them based on the first column, resulting file must contain all record from the first file... (4 Replies)
Discussion started by: emare
4 Replies

2. Shell Programming and Scripting

E-mail Output Merge Lines for Some Text..

Hi all. This is the content of the text file used for the e-mail: TM ICP-EDW BILLING REGISTER USAGE BREAKDOWN_01062014.csv TM_ICP_EDWH_FICL_13062014.TXT TM_ICP_EDWH_FICL_16062014.TXT TM_ICP_EDW_Detailed Payment Journal Report_13062014.txt TM_ICP_EDW_Detailed Payment Journal... (9 Replies)
Discussion started by: aimy
9 Replies

3. Shell Programming and Scripting

Need to merge multiple text files vertically and place comma between fields

Hello expert friends, I'm writing a script to capture stats using sar and stuck up at report generation. I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies

4. Shell Programming and Scripting

How to merge two or more fields from two different files where there is non matching column?

Hi, Please excuse for often requesting queries and making R&D, I am trying to work out a possibility where i have two files field separated by pipe and another file containing only one field where there is no matching columns, Could you please advise how to merge two files. $more... (3 Replies)
Discussion started by: karthikram
3 Replies

5. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

6. Shell Programming and Scripting

AWK: merge two files and replace some fields

Need some code tweak: awk 'END { for (i=1; i<=n; i++) if (f2]) print f2] } NR == FNR { f2 = $1] = $0 next } $1 in f2 { delete f2 }1' FS=, OFS=, 2.csv 1.csv > 3.csvfile 1.csv have: $1,$2,$3,$4,$5,$6,$7,$8,$9...... file 2.csv have: $1,$2,$3,$4,$5,$6 (2 Replies)
Discussion started by: u10
2 Replies

7. Shell Programming and Scripting

Have several text files and want to merge into a single

Hello, I have several files that begin with db. in my directory and I would like to first take it from a specific word starting from $TTL until the end of the contents then do the same all the way down the directory then merge them into one txt file. Is this possible? I am using cygwin with... (4 Replies)
Discussion started by: richsark
4 Replies

8. AIX

merge text files

Hello. Could you please help to know the command to merge multiple text files into one? I am thinking to use: cat f1.txt f2.txt f3.txt > f4.txt Is it okay to use cat command for same purpose - Or could there be any disadvantage in using it? Thank you (4 Replies)
Discussion started by: panchpan
4 Replies

9. Shell Programming and Scripting

AWK Merge Fields for Print Output

I've got a file with each record on a separate line and each record contains 34 fields separated by a colon and i'm trying to re-arrange the order of the fields and merge together certain fields separated by a slash (like field7/field28). I tried using an awk print statement like awk -F: 'BEGIN... (5 Replies)
Discussion started by: RacerX
5 Replies

10. Shell Programming and Scripting

output only numbers from mixed string

it must be late because I'm sure this is an easy task with grep sed or awk string would be anything mixing numbers letters and ) ( = output I need is just the numbers... but I just can't seem to get it to work. Any tips would be great :) (10 Replies)
Discussion started by: nortypig
10 Replies
Login or Register to Ask a Question