02-18-2010
Huge File Comparison
Hi i need to compare two fixed length files and produce the differences if any to a seperate file. I have to capture each and every differneces line by line. Ideally my files should not have any differences but if there are any then it should be captured without any miss. Also my files sizes are very huge say more than 2 GB.
Please help me with a code in either awk or shell script which does this huge file comparison with increase in some performance.
Regards,
Naveen
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have a file {filename} which contains 65000 records
I need to split into 6 smaller files roughly 11000 records each.
Can someone advise me of the Unix command to do so ?
Many thanks (2 Replies)
Discussion started by: grinder182533
2 Replies
2. Solaris
Hi,
I have files with records of 40,00,000& 39,00,000 and i want to find out the
content
1.which is existing in file1 and not in file2.
2.Which is exisitng in file2 and not in file1.
The format of the file will be like
404ABCDEFGHIJK|CDEFGHIJK|1234567890|1
If its a smaller one i... (1 Reply)
Discussion started by: salaathi
1 Replies
3. Shell Programming and Scripting
Hi All
I am sorting a huge file
-rw-r--r-- 1 rama users 448156978 May 13 18:48 102384.temp
$ sort -k 1,40n 102384.temp > 102384.temp1
msgcnt 1468 vxfs: mesg 001: vx_nospace - /dev/vg00/var file system full (1 block extent)
sort: A write error occurred while sorting.
I thought... (3 Replies)
Discussion started by: dhanamurthy
3 Replies
4. Shell Programming and Scripting
I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract
if i use sed, i have to redirect the output to a seperate file like
sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat
the same is true for awk
and... (10 Replies)
Discussion started by: deepaktanna
10 Replies
5. Shell Programming and Scripting
Hi ,
i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character.
I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies
6. Shell Programming and Scripting
I got three different file:
Part of File 1
ARTPHDFGAA
.
.
Part of File 2
ARTGHHYESA
.
.
Part of File 3
ARTPOLYWEA
.
. (4 Replies)
Discussion started by: patrick87
4 Replies
7. Shell Programming and Scripting
Hi Experts,
I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies
8. UNIX for Dummies Questions & Answers
Hi all,
I hope you are well. I am very happy to see your contribution. I am eager to become part of it.
I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below
For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies
9. UNIX for Dummies Questions & Answers
Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now.
I did to this to find the big file but it's taking so much time. Is there any other way?
du -ah / | more
find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies
10. Solaris
Gents
I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server.
How to can I do that without changing anything in the source.
Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies
LEARN ABOUT DEBIAN
file::dircompare
DirCompare(3pm) User Contributed Perl Documentation DirCompare(3pm)
NAME
File::DirCompare - Perl module to compare two directories using callbacks.
SYNOPSIS
use File::DirCompare;
# Simple diff -r --brief replacement
use File::Basename;
File::DirCompare->compare($dir1, $dir2, sub {
my ($a, $b) = @_;
if (! $b) {
printf "Only in %s: %s
", dirname($a), basename($a);
} elsif (! $a) {
printf "Only in %s: %s
", dirname($b), basename($b);
} else {
print "Files $a and $b differ
";
}
});
# Version-control like Deleted/Added/Modified listing
my (@listing, @modified); # use closure to collect results
File::DirCompare->compare('old_tree', 'new_tree', sub {
my ($a, $b) = @_;
if (! $b) {
push @listing, "D $a";
} elsif (! $a) {
push @listing, "A $b";
} else {
if (-f $a && -f $b) {
push @listing, "M $b";
push @modified, $b;
} else {
# One file, one directory - treat as delete + add
push @listing, "D $a";
push @listing, "A $b";
}
}
});
DESCRIPTION
File::DirCompare is a perl module to compare two directories using a callback, invoked for all files that are 'different' between the two
directories, and for any files that exist only in one or other directory ('unique' files).
File::DirCompare has a single public compare() method, with the following signature:
File::DirCompare->compare($dir1, $dir2, $sub, $opts);
The first three arguments are required - $dir1 and $dir2 are paths to the two directories to be compared, and $sub is the subroutine
reference called for all unique or different files. $opts is an optional hashref of options - see OPTIONS below.
The provided subroutine is called for all unique files, and for every pair of 'different' files encountered, with the following signature:
$sub->($file1, $file2)
where $file1 and $file2 are the paths to the two files. For 'unique' files i.e. where a file exists in only one directory, the subroutine
is called with the other argument 'undef' i.e. for:
$sub->($file1, undef)
$sub->(undef, $file2)
the first indicates $file1 exists only in the first directory given ($dir1), and the second indicates $file2 exists only in the second
directory given ($dir2).
OPTIONS
The following optional arguments are supported, passed in using a hash reference after the three required arguments to compare() e.g.
File::DirCompare->compare($dir1, $dir2, $sub, {
cmp => $cmp_sub,
ignore_unique => 1,
});
cmp By default, two files are regarded as different if their contents do not match (tested with File::Compare::compare). That default
behaviour can be overridden by providing a 'cmp' subroutine to do the file comparison, returning zero if the two files are equal, and
non-zero if not.
E.g. to compare using modification times instead of file contents:
File::DirCompare->compare($dir1, $dir2, $sub, {
cmp => sub { -M $_[0] <=> -M $_[1] },
});
ignore_cmp
If you want to see all corresponding files, not just 'different' ones, set the 'ignore_cmp' flag to tell File::DirCompare to skip its
file comparison checks i.e.
File::DirCompare->compare($dir1, $dir2, $sub,
{ ignore_cmp => 1 });
ignore_unique
If you want to ignore files that only exist in one of the two directories, set the 'ignore_unique' flag i.e.
File::DirCompare->compare($dir1, $dir2, $sub,
{ ignore_unique => 1 });
SEE ALSO
File::Dircmp, which provides similar functionality (and whose directory walking code I've adapted for this module), but a simpler
reporting-only interface, something like the first example in the SYNOPSIS above.
AUTHOR AND CREDITS
Gavin Carr <gavin@openfusion.com.au>
Thanks to Robin Barker for a bug report and fix for glob problems with whitespace.
COPYRIGHT AND LICENSE
Copyright 2006-2007 by Gavin Carr.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.10.1 2010-03-02 DirCompare(3pm)