awk for matching fields between files with repeated records
Hello all, I am having trouble with what should be an easy task, but seem to be missing something fundamental. I have two files, with File 1 consisting of a single field of many thousands of records. I also have File 2 with two fields and many thousands of records.
My goal is that when $1 of File 1 matches $1 of File 2, then print $1 and $2 of File 2, or alternatively, print $1 from File 1 with $2 of File 2 when $1 and $2 match between the files. The problem is that File 1 has repeated records in it. Thus when I apply awk 'FNR==NR{a[$1]; next} $1 in a' File 1 File 2 I can get all matches where $1 in File 1 matches $1 in File 2 and then output $1 && $2 in File 2, but without the repeated records. However, I need the order of the records in File 1 to be retained as well as all of the repeated records.
File 1
File 2
Desired Output:
NB: The records are much more varied and repeats much further spread out in the actual file than the simplified examples here.
I had a somewhat similar, albeit more involved, issue in the past that RudiC helped me with (see here), but I am having trouble applying his code to this simpler example.
I got it close with this:
While this attempt printed all of the repeated records of File 1, it only supplied $2 from File 2 along with $1 of File 1 on the first time it appears, but not every time, such as the following:
Hello all,
Would appreciate if someone can help me out on the following requirement.
INPUT FILE:
--------------------------
TPS REPORT
abc def ghi
jkl mon pqr
stu vrs lll
END OF TPS REPORT
TPS REPORT
field1 field2 field3
field4 field5 field6 (8 Replies)
Hello!
I am writing a program to run through two large lists of data (~300,000 rows), find where rows in one file match another, and combine them based on matching fields. Due to the large file sizes, I'm guessing AWK will be the most efficient way to do this. Overall, the input and output I'm... (5 Replies)
Hi every one;
I have a 31500-line text file upon which two following tasks are to be performed:
1: Rearranging the file
2: Taking the average of each column (considering number of zeros) and output the result into a new file
This is the code I've come up with:
awk '(NR%3150<3150)... (0 Replies)
Hi,
I'm very new to these forums. I was wondering if someone could help an AWK beginner with a pattern matching
an actor to his appearance in movies, which would be stored as records. Let's say we have a database of 4 movies (each movie a record with name, studio + year, and actor fields with... (2 Replies)
Hi,
input:
AA|BB|CC
DD|EE
FF
what I am trying to get:
AA|BB|CC
DD|EE|
FF||
I tried to create first an UDF for printing repeats, but I think I have an issue with my END section or my array:
function repeat(str, n, rep, i)
{
for(i=1 ;i<n;i++)
rep=rep str
return rep
}
... (6 Replies)
Hi,
I have 2 tab-delimited input files as follows.
file1.tab:
green A apple
red B apple
file2.tab:
apple - A;Z
Objective:
Return $1 of file1 if,
. $1 of file2 matches $3 of file1 and,
. any single element (separated by ";") in $3 of file2 is present in $2 of file1
In order to... (3 Replies)
Trying to use awk to match the contents of each line in file1 with $5 in file2. Both files are tab-delimited and there may be a space or special character in the name being matched in file2, for example in file1 the name is BRCA1 but in file2 the name is BRCA 1 or in file1 name is BCR but in file2... (6 Replies)
I apologize in advance, but I continue to have trouble searching for matches between two files and then printing portions of each to output in awk and would very much appreciate some help.
I have data as follows:
File1
PS012,002 PRQ 0 1 1 17 1 0 -1 3 2 1 2 -1 ... (7 Replies)
In two previous posts (here) and (here), I received help from forum members comparing multiple fields across two files and selectively printing portions of each as output based upon would-be matches using awk. I had been fairly comfortable populating awk arrays with fields and using awk's special... (3 Replies)
Hi,
My input looks like that:
A|123|qwer
A|456|tyui
A|456|wsxe
B|789|dfgh
Using awk, I am trying to get:
A|123;456|qwer;tyui;wsxe
B|789|dfgh
For records with same $1, group all the $2 in a field (without replicates), and all the $3 in a field (without replicates).
What I have tried:... (6 Replies)
Discussion started by: beca123456
6 Replies
LEARN ABOUT DEBIAN
marc::file::xml
MARC::File::XML(3pm) User Contributed Perl Documentation MARC::File::XML(3pm)NAME
MARC::File::XML - Work with MARC data encoded as XML
SYNOPSIS
## Loading with USE options
use MARC::File::XML ( BinaryEncoding => 'utf8', RecordFormat => 'UNIMARC' );
## Setting the record format without USE options
MARC::File::XML->default_record_format('USMARC');
## reading with MARC::Batch
my $batch = MARC::Batch->new( 'XML', $filename );
my $record = $batch->next();
## or reading with MARC::File::XML explicitly
my $file = MARC::File::XML->in( $filename );
my $record = $file->next();
## serialize a single MARC::Record object as XML
print $record->as_xml();
## write a bunch of records to a file
my $file = MARC::File::XML->out( 'myfile.xml' );
$file->write( $record1 );
$file->write( $record2 );
$file->write( $record3 );
$file->close();
## instead of writing to disk, get the xml directly
my $xml = join( "
",
MARC::File::XML::header(),
MARC::File::XML::record( $record1 ),
MARC::File::XML::record( $record2 ),
MARC::File::XML::footer()
);
DESCRIPTION
The MARC-XML distribution is an extension to the MARC-Record distribution for working with MARC21 data that is encoded as XML. The XML
encoding used is the MARC21slim schema supplied by the Library of Congress. More information may be obtained here:
http://www.loc.gov/standards/marcxml/
You must have MARC::Record installed to use MARC::File::XML. In fact once you install the MARC-XML distribution you will most likely not
use it directly, but will have an additional file format available to you when you use MARC::Batch.
This version of MARC-XML supersedes an the versions ending with 0.25 which were used with the MARC.pm framework. MARC-XML now uses
MARC::Record exclusively.
If you have any questions or would like to contribute to this module please sign on to the perl4lib list. More information about perl4lib
is available at <http://perl4lib.perl.org>.
METHODS
When you use MARC::File::XML your MARC::Record objects will have two new additional methods available to them:
MARC::File::XML->default_record_format([$format])
Sets or returns the default record format used by MARC::File::XML. Valid formats are MARC21, USMARC, UNIMARC and UNIMARCAUTH.
MARC::File::XML->default_record_format('UNIMARC');
as_xml()
Returns a MARC::Record object serialized in XML. You can pass an optional format parameter to tell MARC::File::XML what type of record
(USMARC, UNIMARC, UNIMARCAUTH) you are serializing.
print $record->as_xml([$format]);
as_xml_record([$format])
Returns a MARC::Record object serialized in XML without a collection wrapper. You can pass an optional format parameter to tell
MARC::File::XML what type of record (USMARC, UNIMARC, UNIMARCAUTH) you are serializing.
print $record->as_xml_record('UNIMARC');
new_from_xml([$encoding, $format])
If you have a chunk of XML and you want a record object for it you can use this method to generate a MARC::Record object. You can pass an
optional encoding parameter to specify which encoding (UTF-8 or MARC-8) you would like the resulting record to be in. You can also pass a
format parameter to specify the source record type, such as UNIMARC, UNIMARCAUTH, USMARC or MARC21.
my $record = MARC::Record->new_from_xml( $xml, $encoding, $format );
Note: only works for single record XML chunks.
If you want to write records as XML to a file you can use out() with write() to serialize more than one record as XML.
out()
A constructor for creating a MARC::File::XML object that can write XML to a file. You must pass in the name of a file to write XML to. If
the $encoding parameter or the DefaultEncoding (see above) is set to UTF-8 then the binmode of the output file will be set appropriately.
my $file = MARC::File::XML->out( $filename [, $encoding] );
write()
Used in tandem with out() to write records to a file.
my $file = MARC::File::XML->out( $filename );
$file->write( $record1 );
$file->write( $record2 );
close()
When writing records to disk the filehandle is automatically closed when you the MARC::File::XML object goes out of scope. If you want to
close it explicitly use the close() method.
If you want to generate batches of records as XML, but don't want to write to disk you'll have to use header(), record() and footer() to
generate the different portions.
$xml = join( "
",
MARC::File::XML::header(),
MARC::File::XML::record( $record1 ),
MARC::File::XML::record( $record2 ),
MARC::File::XML::record( $record3 ),
MARC::File::XML::footer()
);
header()
Returns a string of XML to use as the header to your XML file.
footer()
Returns a string of XML to use at the end of your XML file.
record()
Returns a chunk of XML suitable for placement between the header and the footer.
decode()
You probably don't ever want to call this method directly. If you do you should pass in a chunk of XML as the argument.
It is normally invoked by a call to next(), see MARC::Batch or MARC::File.
encode()
You probably want to use the as_xml() method on your MARC::Record object instead of calling this directly. But if you want to you just need
to pass in the MARC::Record object you wish to encode as XML, and you will be returned the XML as a scalar.
TODO
o Support for callback filters in decode().
SEE ALSO
<http://www.loc.gov/standards/marcxml/>
MARC::File::USMARC
MARC::Batch
MARC::Record
AUTHORS
o Ed Summers <ehs@pobox.com>
perl v5.14.2 2011-02-11 MARC::File::XML(3pm)