Merge 2 files with one reference columns


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge 2 files with one reference columns
# 1  
Old 10-22-2013
Merge 2 files with one reference columns

Hi All

Source1

Code:
servername1,patchid1
servername1,patchid2
servername1,patchid3
servername2,patchid1
servername2,patchid2
servername3,patchid4
servername3,patchid5

Source2

Code:
servername1,appname1
servername1,appname2
servername1,appname3
servername2,appname1
servername2,appname2
servername2,appname3
servername2,appname4
servername3,appname10

Basically, reference is the servername. Any given server has many patchids and applications.

When combined, example servername1 will have 9 entries

Final Data:

Code:
servername1,patchid1,appname1
servername1,patchid1,appname2
servername1,patchid1,appname3
servername1,patchid2,appname1
servername1,patchid2,appname2
servername1,patchid2,appname3
servername1,patchid3,appname1
servername1,patchid3,appname2
servername1,patchid3,appname2
servername2,patchid1,appname1
servername2,patchid1,appname2
servername2,patchid1,appname3
servername2,patchid2,appname1
servername2,patchid2,appname2
servername2,patchid2,appname3
.....
....


Hope i made sense. Thanks in advance

Cheers

MV

Last edited by Scrutinizer; 10-22-2013 at 12:07 PM.. Reason: code tags
# 2  
Old 10-22-2013
Please use code tag...


Code:
$ awk  -F, 'FNR==NR{Arr[$1 FS NR]=$0;next}($1 FS FNR in Arr){print Arr[$1 FS FNR] FS $2}'  file2 file1 -- Wrong


EDIT : modification as I was wrong in understanding user's requirement
Code:
$ awk -F, 'NR==FNR{Arr[$1 FS $2]=$1; next} {for (i in Arr) if($1==Arr[i]) print i,$2}' OFS=, file2 file1

Sorry mv_mv I was wrong...

Last edited by Akshay Hegde; 10-22-2013 at 01:03 PM..
# 3  
Old 10-22-2013
Thanks Akshay for the quick reply. Here is the output with your code

-------------------------------
Code:
servername1 patchid1,appname3
servername1 patchid2,appname3
servername1 patchid3,appname3
servername2 patchid1,appname4
servername2 patchid2,appname4
servername3 patchid4,appname10
servername3 patchid5,appname10

--------------------------------

As you can see, only the last appname from source2 is picked. Instead of 9 entries for servername1, only 3 merged lines are printed.

Thanks

Last edited by Scrutinizer; 10-22-2013 at 12:27 PM.. Reason: code tags
# 4  
Old 10-22-2013
Do appname and patchid have to be in that order? Or would this be OK too?
Code:
$ awk 'NR==FNR{A[$1 FS $2]=$1; next} {for (i in A) if($1==A[i]) print i,$2}' FS=, OFS=, file2 file1
servername1,appname1,patchid1
servername1,appname2,patchid1
servername1,appname3,patchid1
servername1,appname1,patchid2
servername1,appname2,patchid2
...

Otherwise try:
Code:
awk 'NR==FNR{A[$1 FS $2]=$1; next} {for (i in A) if($1==A[i]) {split(i,F); print $0,F[2]}}' FS=, OFS=, file2 file1


Last edited by Scrutinizer; 10-22-2013 at 12:38 PM..
# 5  
Old 10-22-2013
Great. Order doesn't matter Smilie.

Latest code seems doing the job. Let me run it on a larger data set and confirm.

Thank you very much

Cheers

---------- Post updated at 10:47 AM ---------- Previous update was at 10:31 AM ----------

Works great. Appreciate it.
# 6  
Old 10-22-2013
This shell script does not hog memory i.e. works with a zillion lines,
but becomes exponentially slower because all the file2 is read for each line in file1
Code:
while IFS=, read f1a f1b
do
 while IFS=, read f2a f2b
 do
  [ "$f1a" = "$f2a" ] && echo "$f1a,$f1b,$f2b"
 done < file2
done < file1


Last edited by MadeInGermany; 10-22-2013 at 06:12 PM..
# 7  
Old 10-22-2013
try also, another shell solution:
Code:
tdir=./$$.tmp
mkdir $tdir || { echo use writable directory. ; exit ; }
while IFS=, read d1 d2
do
   mkdir -p $tdir/$d1/$d2
done < source1
while IFS=, read d3 d4
do
   for d5 in $tdir/$d3/*
   do
      [[ -d $d5 ]] && echo $d5/$d4 | sed 's/.*[.]tmp\///; s/\//,/g'
   done
done < source2
rm -rf $$.tmp

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge columns from two files using awk

I have two csv files : say a.csv, b.csv a.csv looks like this : property1,property2,100 property3,property4,200 In a.csv, the combination of column1 and column2 will be unique b.csv looks like this property1,property2, 300, t1 property1,property2, 400,t2 property3, property4,800,t1... (2 Replies)
Discussion started by: Lakshmikumari
2 Replies

2. Shell Programming and Scripting

Merge files based on columns

011111123444 1234 1 20000 011111123444 1235 1 30000 011111123446 1234 3 40000 011111123447 1234 4 50000 011111123448 1234 3 50000 File2: 011111123444,Rsttponrfgtrgtrkrfrgtrgrer 011111123446,Rsttponrfgtrgtr 011111123447,Rsttponrfgtrguii 011111123448,Rsttponrfgtrgtjiiu I have 2 files... (4 Replies)
Discussion started by: vinus
4 Replies

3. Shell Programming and Scripting

Merge columns from multiple files

Hello and Good day I have a lot of files with same number of rows and columns.$2 and $3 are the same in all files . I need to merge $2,$3,$6 from first file and $6 from another files. File1: $1 $2 $3 $4 $5 $6... (8 Replies)
Discussion started by: ali.seifaddini
8 Replies

4. Shell Programming and Scripting

Merge columns on different files

Hello, I have two files that have this format: file 1 86.82 0.00 86.82 43.61 86.84 0.00 86.84 43.61 86.86 0.00 86.86 43.61 86.88 0.00 86.88 43.61 file 2 86.82 0.22 86.84 0.22 86.86 0.22 86.88 0.22 I would like to merge these two files such that the final file looks like... (5 Replies)
Discussion started by: kayak
5 Replies

5. UNIX for Dummies Questions & Answers

Merge columns from multiple files

Hi all, I've searched the web for a long time trying to figure out how to merge columns from multiple files. I know paste will append columns like so: paste file1 file2 file3 file4 file5 ... But this becomes inconvenient when you want to append a large number of files into a single file. ... (2 Replies)
Discussion started by: torchij
2 Replies

6. Shell Programming and Scripting

Merge columns of different files

Hi, I have tab limited file 1 and tab limited file 2 The output should contain common first column vales and corresponding 2nd column values; AND also unique first column value with corresponding 2nd column value of the file that contains it and 0 for the second file. the output should... (10 Replies)
Discussion started by: polsum
10 Replies

7. UNIX for Dummies Questions & Answers

Merge two files with two columns being similar

Hi everyone. How can I merge two files, where each file has 2 columns and the first columns in both files are similar? I want all in a file of 4 columns; join command removes the duplicate columns. 1 Dave 2 Mark 3 Paul 1 Apple 2 Orange 3 Grapes to get it like this in the 3rd file:... (9 Replies)
Discussion started by: Atrisa
9 Replies

8. Shell Programming and Scripting

merge the two files which has contain columns

Hi may i ask how to accomplish this task: I have 2 files which has multiple columns first file 1 a 2 b 3 c 4 d second file 14 a 9 .... 13 b 10.... 12 c 11... 11 d 12... I want to merge the second file to first file that will looks like this ... (2 Replies)
Discussion started by: jao_madn
2 Replies

9. UNIX for Dummies Questions & Answers

compare columns from 2 files and merge

Dear all, Being new to Unix i have a problem. I have 2 files: File 1: 118,1,0,2,3,0,5,0.3,0,0.3,0.6,1 118,2,1,2,2,0,5,0.4,0,0.4,0.4,1 118,4,2,0,3,0,5,0.7,0,0.3,0.6,1 118,6,4,1,0,0,5,0.8,0,0.2,0,1 File 2: 118,1,BFGL-NGS-109695,3610326,0,18,1,0.556,0.389,0.056,0.25,0.8183... (2 Replies)
Discussion started by: samwilkinson
2 Replies

10. Shell Programming and Scripting

Compare two files and merge columns in a third

Hi, I'm working with snmp, with a little script I'm able to obtain from a switch a list with a couple of values with this format Port Mac 1 00:0A:0B:0C:0D:0E .... (hundred of entries) Now with a simple arp on a router I am able to obtain another list 00:0A:0B:0C:0D:0E... (20 Replies)
Discussion started by: CM64
20 Replies
Login or Register to Ask a Question