Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: best method to compare 2 big files in unix
Top Forums UNIX for Advanced & Expert Users best method to compare 2 big files in unix Post 302488308 by m.d.ludwig on Sunday 16th of January 2011 03:24:51 PM
Old 01-16-2011
Assuming the two files are sorted by the first field (an sql query can do that), my proposal in PERL:
Code:
use strict;
use warnings;

$\ = "\n";
$, = '|';

if (@ARGV < 2) {
    print "USAGE: $0 <file1> <file2>";
    exit 1;
}

my $inputfile1 = shift @ARGV;
open F1, '<', $inputfile1 or die $inputfile1;

my $inputfile2 = shift @ARGV;
open F2, '<', $inputfile2 or die $inputfile2;

my $h1 = <F1>; chomp $h1;
my $h2 = <F2>; chomp $h2;

if ($h1 eq $h2) {
    print $h1;
}
else {
    print STDERR "$0: different headers\n";
    exit 1;
}

my $k1 = undef; my @F1 = ();
my $k2 = undef; my @F2 = ();

while (1) {
    unless (defined $k1) { $_ = <F1>; last unless defined $_; chomp; ($k1, @F1) = split /\|/; }
    unless (defined $k2) { $_ = <F2>; last unless defined $_; chomp; ($k2, @F2) = split /\|/; }

    if ($k1 lt $k2) { print $k1, @F1; $k1 = undef; next; }
    if ($k2 lt $k1) { print $k2, @F2; $k2 = undef; next; }

    print $k1, map { $_ eq shift @F2 ? 'Y' : 'N' } @F1;

    $k1 = undef;
    $k2 = undef;
}

if (defined $k1) { print $k1, @F1; } while (<F1>) { chomp; print; } 
if (defined $k2) { print $k2, @F2; } while (<F2>) { chomp; print; }

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to compare big real numbers

Hi everyone, I need to compare 2 big Floating/Real numbers in a script. After the comparission it is showing worng values in my script. echo "Enter value1" read value1 echo "Enter value2" read value2 Result=`echo "if($value1 > $value2) 1" | bc` if ; then echo "$value1 is... (4 Replies)
Discussion started by: padarthy
4 Replies

2. UNIX for Advanced & Expert Users

UNIX; Compare two files

Hi Guys, Requirement: Want to compare two files, if the the content of both files is same then show "Good result" else Show "Bad Result" I am using the following logic if( cmp -s a b = 0 ) then echo "Good result" else echo "Bad result" exit 0 fi But this is... (1 Reply)
Discussion started by: abhishek3598
1 Replies

3. UNIX for Dummies Questions & Answers

Unix Compare Files

Hi, I need to compare 2 files based on the first field in each file and output the differences to a new file. example File 1 and File 2 both have first field as Number ie: File 1 1252652355 1859553322 1778899562 File 2 1252652355 1859553322 So I would expect File 3 to... (2 Replies)
Discussion started by: Lagre1
2 Replies

4. UNIX for Dummies Questions & Answers

how can i unix compare two files??

how can i unix compare two files?? var1 = 6499 7328 6351 7583 7573 var2 = 6499 7328 6351 7583 7777 i did: diff $var1 $var2 and i got the output: 1c1 < 6499 7328 6351 7583 7573 --- > 6499 7328 6351 7583 7777 what can i do with it? and what does it tell me?? how can i knoe that... (2 Replies)
Discussion started by: nirnir26
2 Replies

5. UNIX for Advanced & Expert Users

How to compare two files using UNIX?

I have two files which have primary key(s) for each row. I need to compare both the files and produce the output in the following format. Primary key(s),file1 value,file2 value. Both the input files will be comma separated files. I have accomplished this using perl, but it is... (6 Replies)
Discussion started by: gpsridhar
6 Replies

6. Shell Programming and Scripting

Compare two files in unix

Hi Gurus I need your kind help sorting the below query I have two text files File1.txt ID Name Address 101 Srinath BBB 102 Sidharth CCC File2.txt ID Name Address 102 Siddharth DDD 103 Suman EEE Now the requirement is if the second file has... (0 Replies)
Discussion started by: Pratik4891
0 Replies

7. Shell Programming and Scripting

Compare two files in UNIX

Hi, I have two files File1 Contents: abc dcf sdc File2 Contents: dcf sdc erg Now my program should return the contents existing in File1 but not in File2. In this case output shoud be "abc" as abc is not available in File 2. It should not return "erg" by saying it is... (4 Replies)
Discussion started by: forums123456
4 Replies

8. Shell Programming and Scripting

Compare two big files for differences using Linux

Hello everybody Looking for help in comparing two files in Linux(files are big 800MB each). Example:- File1 has below data $ cat file1 5,6,3 2.1.4 1,1,1 8,9,1 File2 has below data $ cat file2 5,6,3 8,9,8 1,2,1 2,1,4 (1 Reply)
Discussion started by: shanul karim
1 Replies

9. UNIX for Beginners Questions & Answers

Compare two big files for differences using Linux

Hello everybody Looking for help in comparing two files in Linux(files are big 800MB each). Example:- File1 has below data $ cat file1 5,6,3 2.1.4 1,1,1 8,9,1 File2 has below data $ cat file2 5,6,3 8,9,8 1,2,1 2,1,4 (8 Replies)
Discussion started by: shanul karim
8 Replies

10. Shell Programming and Scripting

Compare and merge two big CSV files

Hi all, i need help. I have two csv files with a huge amount of data. I need the first column of the first file, to be compared with the data of the second, to have at the end a file with the data not present in the second file. Example File1: (only one column) profile_id 57036226... (11 Replies)
Discussion started by: SirMannu
11 Replies

LEARN ABOUT DEBIAN

getopt::usaginator

Getopt::Usaginator(3pm) 				User Contributed Perl Documentation				   Getopt::Usaginator(3pm)

NAME

       Getopt::Usaginator - Conjure up a usage function for your applications

VERSION

       version 0.0012

SYNOPSIS

	   use Getopt::Usaginator <<_END_;

	       Usage: xyzzy <options>

	       --derp	       Derp derp derp
	       --durp	       Durp durp durp
	       -h, --help      This usage

	   _END_

	   # The 'usage' subroutine is now installed

	   ...

	   $options = parse_options( @ARGV ); # Not supplied by Usaginator

	   usage if $options{help}; # Print usage and exit with status 0

	   if ( ! $options{derp} ) {
	       # Print warning and usage and exit with status -1
	       usage "You should really derp";
	   }

	   if ( $options{durp} ) {
	       # Print warning and usage and exit with status 2
	       usage 2 => "--durp is not ready yet";
	   }

	   ...

	   usage 3 # Print usage and exit with status 3

DESCRIPTION

       Getopt::Usaginator is a tool for creating a handy usage subroutine for commandline applications

       It does not do any option parsing, but is best paired with Getopt::Long or any of the other myriad of option parsers

USAGE

   use Getopt::Usaginator <usage>
       Install a "usage" subroutine configured with the <usage> text

   $code = Getopt::Usaginator->usaginator( <usage> )
       Return a subroutine configured with the <usage> text

   ...
       More advanced usage is possible, peek under the hood for more information

	   perldoc -m Getopt::Usaginator

       An example:

	   use Getopt::Usaginator
	       # Called with the error
	       error => sub { ... },
	       # Called when usage printing is needed
	       usage => sub { ... },
	       ...
	   ;

An example with Getopt::Long parsing
	   use Getopt::Usaginator ...

	   sub run {
	       my $self = shift;
	       my @arguments = @_;

	       usage 0 unless @arguments;

	       my ( $help );
	       {
		   local @ARGV = @arguments;
		   GetOptions(
		       'help|h|?' => $help,
		   );
	       }

	       usage 0 if $help;

	       ...
	   }

AUTHOR

	 Robert Krimen <robertkrimen@gmail.com>

COPYRIGHT AND LICENSE

       This software is copyright (c) 2010 by Robert Krimen.

       This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

perl v5.10.1							    2010-06-05						   Getopt::Usaginator(3pm)
All times are GMT -4. The time now is 11:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy