Shell Programming and Scripting

I'm a newbie to perl and not much of a coder in general, but I must use it at my new task at work. I have two text files that are several hundred MBs large. I need to pull in data from a column in file 2 into file 1. Basically, file 2 is the lookup table and I can key off the unique id column that they both share. I'm not sure how to go about this though.

I figured I should do the following steps:
-add the new field in file 1 for the returned value to be sent
-delimit the files with tab and store in arrays
-loop through each record in the unique id field of file1 and then loop through the same field in file 2, returning the return_data field if there is a match

I wrote something similar in vba, but at this file size, that is not an option. A sample of the data is below. Any help is greatly appreciated.

File1:
col1 col2 col3 col4 uniq_id
blah1 blah2 blah3 blah4 id1
blah1 blah2 blah3 blah4 id2
blah1 blah2 blah3 blah4 id3

File2:
col1 col2 rtrn_data col4 uniq_id
blah1 blah2 rd77 blah4 id1
blah1 blah2 rd5 blah4 id2
blah1 blah2 rd6 blah4 id3

output:
col1 col2 col3 col4 uniq_id rtrn_data
blah1 blah2 blah3 blah4 id1 rd77
blah1 blah2 blah3 blah4 id2 rd5
blah1 blah2 blah3 blah4 id3 rd6

AWK will be a lot easier than perl:

Code:

$ cat a.sh
#! /bin/sh


awk '
NR==FNR && NR>1 {
        map[$NF]=$3
        next
}
NR!=FNR && FNR==1 {
        print $0, "rtrn-data"
}
NR!=FNR && FNR>1 {
        print $0, map[$NF]
}' file2 file1

$ ./a.sh file2 file1
col1 col2 col3 col4 uniq_id rtrn-data
blah1 blah2 blah3 blah4 id1 rd77
blah1 blah2 blah3 blah4 id2 rd5
blah1 blah2 blah3 blah4 id3 rd6

If you really want perl code, run 'a2p' (Awk to Perl translator) to convert that awk code to perl.

Code:

#!/usr/bin/perl -w

# Usage: ./this_script FILE2 FILE1

use strict;

my %id;

open my $f, '<', shift or die "$!";
<$f>; # skip the header
while (<$f>) {
    chomp;
    my @v = split;
    $id{$v[4]} = $v[2];
}
close $f;

print "col1 col2 col3 col4 uniq_id rtrn_data\n";

open $f, '<', shift or die "$!";
<$f>;
while (<$f>) {
    chomp;
    my $uniq_id = (split)[-1];
    print "$_ $id{$uniq_id}\n";
}
close $f;

Alternatively, if the files are sorted on the id field, you could try this:

Code:

join -1 5 -2 5 -o 1.1,1.2,1.3,1.4,0,2.3 File1 File2