Sponsored Content
Top Forums Shell Programming and Scripting Deleting duplicate records from file 1 if records from file 2 match Post 302635527 by vestport on Saturday 5th of May 2012 09:20:19 AM
Old 05-05-2012
peasant thanks so much for that! It worked perfectly!

What I did was as you suggested convert the ";" delimiters in the one file first to "|" to get a common delimiter as your code uses the -F"|" option:
Code:
cat FileThatNeedsConverting | sed 's/;/|/g' > ConvertedFile


I was going to ask how to see discarded data but a simple "diff" between the 2 files (original and nonmatch.txt) accomplishes that.
Code:
diff OriginalFile nonmatch.txt

Also by doing a:
Code:
wc -l OriginalFile

and a:
Code:
wc -l nonmatch.txt


you can see that records were shaved off. I just wanted to add that in the case that this may help someone else verify or similar project.


Many thanks man!Smilie


Art

Last edited by Franklin52; 05-05-2012 at 10:45 AM.. Reason: Please use code tags for data and code samples, thank you
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove all instances of duplicate records from the file

Hi experts, I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances... (3 Replies)
Discussion started by: vukkusila
3 Replies

2. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

3. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

4. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

5. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

6. Shell Programming and Scripting

Deleting last records of a file

can you please give shell script for daleting the last 7 records of file... (7 Replies)
Discussion started by: vsairam
7 Replies

7. Shell Programming and Scripting

Deleting Duplicate Records

Hello, I'm have a file of xy data with over 1000 records. I want to delete both x and y values for any record that has the same x value as any previous record thus removing the duplicates from my file. Can anyone help? Thanks, Dan (3 Replies)
Discussion started by: DFr0st
3 Replies

8. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

9. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

10. UNIX for Dummies Questions & Answers

Deleting records from .dat file

Hi, I need to delete one row from a .dat file. error processing column PCT_USED in row 1053295 for datafile /exp/stats/ts_stats.dat ORA-01722: invalid number This is used to load records using sql loader. Please let me know the procedure to do it. Regards, VN (3 Replies)
Discussion started by: narayanv
3 Replies
MARC::File::XML(3pm)					User Contributed Perl Documentation				      MARC::File::XML(3pm)

NAME
MARC::File::XML - Work with MARC data encoded as XML SYNOPSIS
## Loading with USE options use MARC::File::XML ( BinaryEncoding => 'utf8', RecordFormat => 'UNIMARC' ); ## Setting the record format without USE options MARC::File::XML->default_record_format('USMARC'); ## reading with MARC::Batch my $batch = MARC::Batch->new( 'XML', $filename ); my $record = $batch->next(); ## or reading with MARC::File::XML explicitly my $file = MARC::File::XML->in( $filename ); my $record = $file->next(); ## serialize a single MARC::Record object as XML print $record->as_xml(); ## write a bunch of records to a file my $file = MARC::File::XML->out( 'myfile.xml' ); $file->write( $record1 ); $file->write( $record2 ); $file->write( $record3 ); $file->close(); ## instead of writing to disk, get the xml directly my $xml = join( " ", MARC::File::XML::header(), MARC::File::XML::record( $record1 ), MARC::File::XML::record( $record2 ), MARC::File::XML::footer() ); DESCRIPTION
The MARC-XML distribution is an extension to the MARC-Record distribution for working with MARC21 data that is encoded as XML. The XML encoding used is the MARC21slim schema supplied by the Library of Congress. More information may be obtained here: http://www.loc.gov/standards/marcxml/ You must have MARC::Record installed to use MARC::File::XML. In fact once you install the MARC-XML distribution you will most likely not use it directly, but will have an additional file format available to you when you use MARC::Batch. This version of MARC-XML supersedes an the versions ending with 0.25 which were used with the MARC.pm framework. MARC-XML now uses MARC::Record exclusively. If you have any questions or would like to contribute to this module please sign on to the perl4lib list. More information about perl4lib is available at <http://perl4lib.perl.org>. METHODS
When you use MARC::File::XML your MARC::Record objects will have two new additional methods available to them: MARC::File::XML->default_record_format([$format]) Sets or returns the default record format used by MARC::File::XML. Valid formats are MARC21, USMARC, UNIMARC and UNIMARCAUTH. MARC::File::XML->default_record_format('UNIMARC'); as_xml() Returns a MARC::Record object serialized in XML. You can pass an optional format parameter to tell MARC::File::XML what type of record (USMARC, UNIMARC, UNIMARCAUTH) you are serializing. print $record->as_xml([$format]); as_xml_record([$format]) Returns a MARC::Record object serialized in XML without a collection wrapper. You can pass an optional format parameter to tell MARC::File::XML what type of record (USMARC, UNIMARC, UNIMARCAUTH) you are serializing. print $record->as_xml_record('UNIMARC'); new_from_xml([$encoding, $format]) If you have a chunk of XML and you want a record object for it you can use this method to generate a MARC::Record object. You can pass an optional encoding parameter to specify which encoding (UTF-8 or MARC-8) you would like the resulting record to be in. You can also pass a format parameter to specify the source record type, such as UNIMARC, UNIMARCAUTH, USMARC or MARC21. my $record = MARC::Record->new_from_xml( $xml, $encoding, $format ); Note: only works for single record XML chunks. If you want to write records as XML to a file you can use out() with write() to serialize more than one record as XML. out() A constructor for creating a MARC::File::XML object that can write XML to a file. You must pass in the name of a file to write XML to. If the $encoding parameter or the DefaultEncoding (see above) is set to UTF-8 then the binmode of the output file will be set appropriately. my $file = MARC::File::XML->out( $filename [, $encoding] ); write() Used in tandem with out() to write records to a file. my $file = MARC::File::XML->out( $filename ); $file->write( $record1 ); $file->write( $record2 ); close() When writing records to disk the filehandle is automatically closed when you the MARC::File::XML object goes out of scope. If you want to close it explicitly use the close() method. If you want to generate batches of records as XML, but don't want to write to disk you'll have to use header(), record() and footer() to generate the different portions. $xml = join( " ", MARC::File::XML::header(), MARC::File::XML::record( $record1 ), MARC::File::XML::record( $record2 ), MARC::File::XML::record( $record3 ), MARC::File::XML::footer() ); header() Returns a string of XML to use as the header to your XML file. footer() Returns a string of XML to use at the end of your XML file. record() Returns a chunk of XML suitable for placement between the header and the footer. decode() You probably don't ever want to call this method directly. If you do you should pass in a chunk of XML as the argument. It is normally invoked by a call to next(), see MARC::Batch or MARC::File. encode() You probably want to use the as_xml() method on your MARC::Record object instead of calling this directly. But if you want to you just need to pass in the MARC::Record object you wish to encode as XML, and you will be returned the XML as a scalar. TODO
o Support for callback filters in decode(). SEE ALSO
<http://www.loc.gov/standards/marcxml/> MARC::File::USMARC MARC::Batch MARC::Record AUTHORS
o Ed Summers <ehs@pobox.com> perl v5.14.2 2011-02-11 MARC::File::XML(3pm)
All times are GMT -4. The time now is 01:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy