Sponsored Content
Full Discussion: sort and compare files
Top Forums UNIX for Dummies Questions & Answers sort and compare files Post 302342437 by drl on Sunday 9th of August 2009 07:50:33 PM
Old 08-09-2009
Hi.

In thinking about this, you may not need to sort the files in order to determine if they have the same content. The steps to use diff or cmp are that the files are sorted -- read once -- then compared character by character -- read again.

I wrote a short perl code that simply adds the bytes in the files together. It will not matter if the characters are re-arranged or not, the sum will always be the same. An additional advantage is that this is not restricted to text files. Here is the code:
Code:
#!/usr/bin/perl

# @(#) p1	Demonstrate linear sum of bytes in a file.
# This would show files being equal if the sums are the same.
# Adapted from the checksum fragment in "Programming perl, 3rd",
# page 821.

use warnings;
use strict;
use Carp;

my ($debug);
$debug = 0;
$debug = 1;

my ( $file, $sum );
foreach $file (@ARGV) {
  if ( !-f $file ) {
    print STDERR " Skipping $file, not a plain file.\n";
    next;
  }
  $sum = linear($file);
  print "$sum\t$file\n";
}

sub linear {
  my ($file) = $_[0];
  my ( $f, @sum );
  local $/;
  undef $/;
  open( $f, "<", $file ) || carp(" Ignoring file $file, cannot open.\n");
  $sum = unpack( "%C*", <$f> );
  close($f);
  return ($sum);
}

exit(0);

The driver code is long, so I will paste only the results of calling the perl code with 6 files, each one of a pair being a rearrangement of the other. The sum then will be the same for all files that have an identical set of characters, regardless of position in the file. Samples of the file content is shown:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
my-nl (local) 296
edges (local) 307
perl 5.10.0


==> data1 <==

  1 a
  2 b


==> data2 <==

  1 b
  2 a

data3:
     1	3 Musketeers
     2	5th Avenue
     3	100 Grand Bar
   ...
    27	Welch's Fudge
    28	York Peppermint Pattie
    29	Zagnut

data4:
     1	Oh Henry!
     2	GooGoo Supreme
     3	Forever Yours
   ...
    27	Mr. Goodbar
    28	Reggie Bar
    29	English Crunchie

data5:
     1	# Sun Nov 11 23:11:09 CST 2007
     2	# http://showcase.netins.net/web/creative/lincoln/speeches/cooper.htm
     3	# fmt -66
   ...
   826	the Government nor of dungeons to ourselves. LET US HAVE FAITH
   827	THAT RIGHT MAKES MIGHT, AND IN THAT FAITH, LET US, TO THE END,
   828	DARE TO DO OUR DUTY AS WE UNDERSTAND IT.

data6:
     1	
     2	
     3	
   ...
   826	you gain by forcing the sentiment which created it out of the
   827	you, who discarded the old policy of the fathers. We resisted,
   828	your political contests among yourselves, each faction charges

 Results:
215	data1
215	data2
27365	data3
27365	data4
17342	data5
17342	data6

Of course, if you consider 2 lines such as:
Code:
ab
ba

to be different, then this approach will not be satisfactory.

Best wishes ... cheers, drl

Last edited by drl; 08-09-2009 at 09:04 PM.. Reason: Edit 1: add disclaimer
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort and compare file

If I have 3 kinds of files in directory DATA1: FileA.20060315.dat, FileB.20060315.dat, FileC.20060315.dat FileC.20060316.dat FileA.20060317.dat, FileB.20060317.dat FileA.20060318.dat, FileB.20060318.dat, FileC.20060318.dat If 3 files have the same date then run $cat FileA.20060315.dat... (3 Replies)
Discussion started by: sabercats
3 Replies

2. Shell Programming and Scripting

Why do we use sort before compare ?

Dear All. Im trying to know how exactly the command "compare" works, does it compare line by line or field by field, and the most important thing is that why the files have to be sorted before we compare them? Thanks in advance (7 Replies)
Discussion started by: yahyaaa
7 Replies

3. Shell Programming and Scripting

Require compare command to compare 4 files

I have four files, I need to compare these files together. As such i know "sdiff and comm" commands but these commands compare 2 files together. If I use sdiff command then i have to compare each file with other which will increase the codes. Please suggest if you know some commands whcih can... (6 Replies)
Discussion started by: nehashine
6 Replies

4. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

5. Shell Programming and Scripting

comm command -- Sort and compare two files

Team, I have two files and I am trying to find the lines unique to file1. So i have executed the below command at shell prompt and got the correct results comm -23 <(sort test) <(sort test1) When i run the same command in Bash shell script, i got the correct results. But when i run... (5 Replies)
Discussion started by: forums123456
5 Replies

6. Shell Programming and Scripting

How to sort and compare files in more efficient manner?

Hello All, Iam using below method to sort and compare files. First iam doing sorting and changing the same file and then doing comparing and taking the final result to another file. sort -o temp.txt file1 mv temp.txt file1 sort -o temp.txt file2 mv temp.txt file2 sort -o temp.txt... (6 Replies)
Discussion started by: Vikram_Tanwar12
6 Replies

7. Shell Programming and Scripting

UNIX compare, sort lines and append difference

Hi, I have a file that needs to be converted: content is: a, b, 4 a ,b, 5 x, y, 1 a, b, 1 x, y, 3 how can i get: a, b, 1|4|5 x,y 1|3 (1 Reply)
Discussion started by: nike27
1 Replies

8. Shell Programming and Scripting

UNIX compare, sort lines and append difference

To make it easier, i gave following example. It is not homework or classwork. Instead, i have a huge csv file dump from tsql with 15 columns and around 300 rows. I was able to extract content that needs to be really converted. Here is the extract: ES FP,B1ES FP,70000,I,SL22,SL22 (70000) ES... (0 Replies)
Discussion started by: nike27
0 Replies

9. Shell Programming and Scripting

UNIX compare, sort lines and append difference

To make it easier, i gave following example. It is not homework or classwork. Instead, i have a huge csv file dump from tsql with 15 columns and around 300 rows. I was able to extract content that needs to be really converted. Here is the extract: ES FP,B1ES FP,70000,I,SL22,SL22 (70000) ES... (8 Replies)
Discussion started by: nike27
8 Replies

10. UNIX for Beginners Questions & Answers

Sort by record column, Compare with conditons and export the result

Hello, I am new to Unix and would like to seek a help, please. I have 2 files (file_1 and file_2), I need to perform the following actions. 1 ) Sort the both file by the column 26-36 (which is Invoice number) what is sort command with the column sort? 2) Compare the file_1.sorted and... (3 Replies)
Discussion started by: Usagi
3 Replies
All times are GMT -4. The time now is 09:03 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy