sort and compare files | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

sort and compare files

UNIX for Dummies Questions & Answers


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 08-06-2009
dnat dnat is offline
Registered User
 
Join Date: Jan 2007
Last Activity: 7 December 2012, 5:29 AM EST
Posts: 42
Thanks: 0
Thanked 0 Times in 0 Posts
sort and compare files

Hi,

I have around 14-15 files and i want to sort all the files and then compare. I dont want to sort them and store in a different file and then compare. I want to compare two files at a time. Is there a way we can do this using a single command?
Sponsored Links
    #2  
Old 08-06-2009
zaxxon's Avatar
zaxxon zaxxon is offline Forum Staff  
code tag tagger
 
Join Date: Sep 2007
Last Activity: 19 April 2014, 12:31 PM EDT
Location: St. Gallen, Switzerland
Posts: 6,160
Thanks: 108
Thanked 428 Times in 398 Posts
This is a bit abstract. To sort you use the command sort. For comparing you can use diff, comm, cmp, grep -f ..., awk, and maybe some more.
Unless you present us a concrete example of your 2 input files and the desired output I recommend to play around with those commands up there and/or use the advanced search function of this forum here since this is a question that came up relativ quite often.

When posting code, data or logs make sure you use CODE-tags, ty.

Last edited by zaxxon; 08-06-2009 at 03:18 AM..
Sponsored Links
    #3  
Old 08-07-2009
dnat dnat is offline
Registered User
 
Join Date: Jan 2007
Last Activity: 7 December 2012, 5:29 AM EST
Posts: 42
Thanks: 0
Thanked 0 Times in 0 Posts
File A contains
a
b
c
d

File b contains
b
c
a
d

Now i want to compare(cmp or diff) these two files after sorting. Ideally i should get the result that they are same.

---------- Post updated 08-07-09 at 12:08 PM ---------- Previous update was 08-06-09 at 06:16 PM ----------

Does anyone have an answer. What exactly i want to know is how to combine the diff/cmp command with the sort command
    #4  
Old 08-07-2009
zaxxon's Avatar
zaxxon zaxxon is offline Forum Staff  
code tag tagger
 
Join Date: Sep 2007
Last Activity: 19 April 2014, 12:31 PM EDT
Location: St. Gallen, Switzerland
Posts: 6,160
Thanks: 108
Thanked 428 Times in 398 Posts
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

***************************************************

I asked you to use CODE tags. Edit your post accordingly.
Also again, this is often asked in the forum.
Also again if you don't want to use the search engine of the forum try it at least yourself with some of the example tools I have given up there.
If both example files are sorted, there will be no difference left, btw. Commands are connected by pipes, the vertcial line here -> |

Take it as warning.
Sponsored Links
    #5  
Old 08-07-2009
drl's Avatar
drl drl is online now Forum Advisor  
Registered Voter
 
Join Date: Apr 2007
Last Activity: 21 April 2014, 8:09 AM EDT
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 1,632
Thanks: 24
Thanked 177 Times in 161 Posts
Hi.

Here is one approach:

Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate compare files after sort.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1) sort cmp
set -o nounset
echo

if [ $# -ne 2 ]
then
  echo " Error - must supply 2 filenames."
  exit 1
fi

FILE1="$1"
shift
FILE2="$1"

echo
echo " Data file $FILE1:"
cat $FILE1

echo
echo " Data file $FILE2:"
cat $FILE2

echo
echo " Results:"
if cmp -s <( sort $FILE1) <(sort $FILE2)
then
  echo " Files $FILE1 and $FILE2 are same."
else
  echo " Files $FILE1 and $FILE2 are different."
fi

exit 0

producing for your sample data:

Code:
% ./s1 data1 data2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
sort (GNU coreutils) 6.10
cmp (GNU diffutils) 2.8.1


 Data file data1:
a
b
c
d

 Data file data2:
b
c
a
d

 Results:
 Files data1 and data2 are same.

and for your sample data and one non-matching file:

Code:
% ./s1 data1 data3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
sort (GNU coreutils) 6.10
cmp (GNU diffutils) 2.8.1


 Data file data1:
a
b
c
d

 Data file data3:
c
b
d
e

 Results:
 Files data1 and data3 are different.

The syntax

Code:
<( command )

is called process substitution. This also worked with "ksh 93s+". See man pages for details on specific commands.

If it is possible that the files could be very long, you could add a preliminary screening for size: check the length of the two files, if the lengths are different, then you need not sort and compare, but declare immediately that they are different ... cheers, drl

Last edited by drl; 08-08-2009 at 07:41 AM.. Reason: Edit 1: spelling correction
Sponsored Links
    #6  
Old 08-09-2009
drl's Avatar
drl drl is online now Forum Advisor  
Registered Voter
 
Join Date: Apr 2007
Last Activity: 21 April 2014, 8:09 AM EDT
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 1,632
Thanks: 24
Thanked 177 Times in 161 Posts
Hi.

In thinking about this, you may not need to sort the files in order to determine if they have the same content. The steps to use diff or cmp are that the files are sorted -- read once -- then compared character by character -- read again.

I wrote a short perl code that simply adds the bytes in the files together. It will not matter if the characters are re-arranged or not, the sum will always be the same. An additional advantage is that this is not restricted to text files. Here is the code:

Code:
#!/usr/bin/perl

# @(#) p1	Demonstrate linear sum of bytes in a file.
# This would show files being equal if the sums are the same.
# Adapted from the checksum fragment in "Programming perl, 3rd",
# page 821.

use warnings;
use strict;
use Carp;

my ($debug);
$debug = 0;
$debug = 1;

my ( $file, $sum );
foreach $file (@ARGV) {
  if ( !-f $file ) {
    print STDERR " Skipping $file, not a plain file.\n";
    next;
  }
  $sum = linear($file);
  print "$sum\t$file\n";
}

sub linear {
  my ($file) = $_[0];
  my ( $f, @sum );
  local $/;
  undef $/;
  open( $f, "<", $file ) || carp(" Ignoring file $file, cannot open.\n");
  $sum = unpack( "%C*", <$f> );
  close($f);
  return ($sum);
}

exit(0);

The driver code is long, so I will paste only the results of calling the perl code with 6 files, each one of a pair being a rearrangement of the other. The sum then will be the same for all files that have an identical set of characters, regardless of position in the file. Samples of the file content is shown:

Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
my-nl (local) 296
edges (local) 307
perl 5.10.0


==> data1 <==

  1 a
  2 b


==> data2 <==

  1 b
  2 a

data3:
     1	3 Musketeers
     2	5th Avenue
     3	100 Grand Bar
   ...
    27	Welch's Fudge
    28	York Peppermint Pattie
    29	Zagnut

data4:
     1	Oh Henry!
     2	GooGoo Supreme
     3	Forever Yours
   ...
    27	Mr. Goodbar
    28	Reggie Bar
    29	English Crunchie

data5:
     1	# Sun Nov 11 23:11:09 CST 2007
     2	# http://showcase.netins.net/web/creative/lincoln/speeches/cooper.htm
     3	# fmt -66
   ...
   826	the Government nor of dungeons to ourselves. LET US HAVE FAITH
   827	THAT RIGHT MAKES MIGHT, AND IN THAT FAITH, LET US, TO THE END,
   828	DARE TO DO OUR DUTY AS WE UNDERSTAND IT.

data6:
     1	
     2	
     3	
   ...
   826	you gain by forcing the sentiment which created it out of the
   827	you, who discarded the old policy of the fathers. We resisted,
   828	your political contests among yourselves, each faction charges

 Results:
215	data1
215	data2
27365	data3
27365	data4
17342	data5
17342	data6

Of course, if you consider 2 lines such as:

Code:
ab
ba

to be different, then this approach will not be satisfactory.

Best wishes ... cheers, drl

Last edited by drl; 08-09-2009 at 08:04 PM.. Reason: Edit 1: add disclaimer
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
compare files in two directories and output changed files to third directory bkeep Shell Programming and Scripting 4 08-26-2009 06:58 AM
compare two files and to remove the matching lines on both the files shellscripter Shell Programming and Scripting 4 06-20-2009 08:48 AM
Why do we use sort before compare ? yahyaaa Shell Programming and Scripting 7 11-13-2008 09:27 AM
Sort files by date, not showing files from today carlosdivega UNIX for Dummies Questions & Answers 3 07-17-2008 10:37 AM
Sort and compare file sabercats Shell Programming and Scripting 3 03-27-2006 02:05 PM



All times are GMT -4. The time now is 08:09 AM.