sort and compare files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers sort and compare files
# 1  
Old 08-06-2009
sort and compare files

Hi,

I have around 14-15 files and i want to sort all the files and then compare. I dont want to sort them and store in a different file and then compare. I want to compare two files at a time. Is there a way we can do this using a single command?
# 2  
Old 08-06-2009
This is a bit abstract. To sort you use the command sort. For comparing you can use diff, comm, cmp, grep -f ..., awk, and maybe some more.
Unless you present us a concrete example of your 2 input files and the desired output I recommend to play around with those commands up there and/or use the advanced search function of this forum here since this is a question that came up relativ quite often.

When posting code, data or logs make sure you use CODE-tags, ty.

Last edited by zaxxon; 08-06-2009 at 04:18 AM..
# 3  
Old 08-07-2009
File A contains
a
b
c
d

File b contains
b
c
a
d

Now i want to compare(cmp or diff) these two files after sorting. Ideally i should get the result that they are same.

---------- Post updated 08-07-09 at 12:08 PM ---------- Previous update was 08-06-09 at 06:16 PM ----------

Does anyone have an answer. What exactly i want to know is how to combine the diff/cmp command with the sort command
# 4  
Old 08-07-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

***************************************************

I asked you to use CODE tags. Edit your post accordingly.
Also again, this is often asked in the forum.
Also again if you don't want to use the search engine of the forum try it at least yourself with some of the example tools I have given up there.
If both example files are sorted, there will be no difference left, btw. Commands are connected by pipes, the vertcial line here -> |

Take it as warning.
# 5  
Old 08-07-2009
Hi.

Here is one approach:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate compare files after sort.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1) sort cmp
set -o nounset
echo

if [ $# -ne 2 ]
then
  echo " Error - must supply 2 filenames."
  exit 1
fi

FILE1="$1"
shift
FILE2="$1"

echo
echo " Data file $FILE1:"
cat $FILE1

echo
echo " Data file $FILE2:"
cat $FILE2

echo
echo " Results:"
if cmp -s <( sort $FILE1) <(sort $FILE2)
then
  echo " Files $FILE1 and $FILE2 are same."
else
  echo " Files $FILE1 and $FILE2 are different."
fi

exit 0

producing for your sample data:
Code:
% ./s1 data1 data2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
sort (GNU coreutils) 6.10
cmp (GNU diffutils) 2.8.1


 Data file data1:
a
b
c
d

 Data file data2:
b
c
a
d

 Results:
 Files data1 and data2 are same.

and for your sample data and one non-matching file:
Code:
% ./s1 data1 data3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
sort (GNU coreutils) 6.10
cmp (GNU diffutils) 2.8.1


 Data file data1:
a
b
c
d

 Data file data3:
c
b
d
e

 Results:
 Files data1 and data3 are different.

The syntax
Code:
<( command )

is called process substitution. This also worked with "ksh 93s+". See man pages for details on specific commands.

If it is possible that the files could be very long, you could add a preliminary screening for size: check the length of the two files, if the lengths are different, then you need not sort and compare, but declare immediately that they are different ... cheers, drl

Last edited by drl; 08-08-2009 at 08:41 AM.. Reason: Edit 1: spelling correction
# 6  
Old 08-09-2009
Hi.

In thinking about this, you may not need to sort the files in order to determine if they have the same content. The steps to use diff or cmp are that the files are sorted -- read once -- then compared character by character -- read again.

I wrote a short perl code that simply adds the bytes in the files together. It will not matter if the characters are re-arranged or not, the sum will always be the same. An additional advantage is that this is not restricted to text files. Here is the code:
Code:
#!/usr/bin/perl

# @(#) p1	Demonstrate linear sum of bytes in a file.
# This would show files being equal if the sums are the same.
# Adapted from the checksum fragment in "Programming perl, 3rd",
# page 821.

use warnings;
use strict;
use Carp;

my ($debug);
$debug = 0;
$debug = 1;

my ( $file, $sum );
foreach $file (@ARGV) {
  if ( !-f $file ) {
    print STDERR " Skipping $file, not a plain file.\n";
    next;
  }
  $sum = linear($file);
  print "$sum\t$file\n";
}

sub linear {
  my ($file) = $_[0];
  my ( $f, @sum );
  local $/;
  undef $/;
  open( $f, "<", $file ) || carp(" Ignoring file $file, cannot open.\n");
  $sum = unpack( "%C*", <$f> );
  close($f);
  return ($sum);
}

exit(0);

The driver code is long, so I will paste only the results of calling the perl code with 6 files, each one of a pair being a rearrangement of the other. The sum then will be the same for all files that have an identical set of characters, regardless of position in the file. Samples of the file content is shown:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
my-nl (local) 296
edges (local) 307
perl 5.10.0


==> data1 <==

  1 a
  2 b


==> data2 <==

  1 b
  2 a

data3:
     1	3 Musketeers
     2	5th Avenue
     3	100 Grand Bar
   ...
    27	Welch's Fudge
    28	York Peppermint Pattie
    29	Zagnut

data4:
     1	Oh Henry!
     2	GooGoo Supreme
     3	Forever Yours
   ...
    27	Mr. Goodbar
    28	Reggie Bar
    29	English Crunchie

data5:
     1	# Sun Nov 11 23:11:09 CST 2007
     2	# http://showcase.netins.net/web/creative/lincoln/speeches/cooper.htm
     3	# fmt -66
   ...
   826	the Government nor of dungeons to ourselves. LET US HAVE FAITH
   827	THAT RIGHT MAKES MIGHT, AND IN THAT FAITH, LET US, TO THE END,
   828	DARE TO DO OUR DUTY AS WE UNDERSTAND IT.

data6:
     1	
     2	
     3	
   ...
   826	you gain by forcing the sentiment which created it out of the
   827	you, who discarded the old policy of the fathers. We resisted,
   828	your political contests among yourselves, each faction charges

 Results:
215	data1
215	data2
27365	data3
27365	data4
17342	data5
17342	data6

Of course, if you consider 2 lines such as:
Code:
ab
ba

to be different, then this approach will not be satisfactory.

Best wishes ... cheers, drl

Last edited by drl; 08-09-2009 at 09:04 PM.. Reason: Edit 1: add disclaimer
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Sort by record column, Compare with conditons and export the result

Hello, I am new to Unix and would like to seek a help, please. I have 2 files (file_1 and file_2), I need to perform the following actions. 1 ) Sort the both file by the column 26-36 (which is Invoice number) what is sort command with the column sort? 2) Compare the file_1.sorted and... (3 Replies)
Discussion started by: Usagi
3 Replies

2. Shell Programming and Scripting

UNIX compare, sort lines and append difference

To make it easier, i gave following example. It is not homework or classwork. Instead, i have a huge csv file dump from tsql with 15 columns and around 300 rows. I was able to extract content that needs to be really converted. Here is the extract: ES FP,B1ES FP,70000,I,SL22,SL22 (70000) ES... (8 Replies)
Discussion started by: nike27
8 Replies

3. Shell Programming and Scripting

UNIX compare, sort lines and append difference

To make it easier, i gave following example. It is not homework or classwork. Instead, i have a huge csv file dump from tsql with 15 columns and around 300 rows. I was able to extract content that needs to be really converted. Here is the extract: ES FP,B1ES FP,70000,I,SL22,SL22 (70000) ES... (0 Replies)
Discussion started by: nike27
0 Replies

4. Shell Programming and Scripting

UNIX compare, sort lines and append difference

Hi, I have a file that needs to be converted: content is: a, b, 4 a ,b, 5 x, y, 1 a, b, 1 x, y, 3 how can i get: a, b, 1|4|5 x,y 1|3 (1 Reply)
Discussion started by: nike27
1 Replies

5. Shell Programming and Scripting

How to sort and compare files in more efficient manner?

Hello All, Iam using below method to sort and compare files. First iam doing sorting and changing the same file and then doing comparing and taking the final result to another file. sort -o temp.txt file1 mv temp.txt file1 sort -o temp.txt file2 mv temp.txt file2 sort -o temp.txt... (6 Replies)
Discussion started by: Vikram_Tanwar12
6 Replies

6. Shell Programming and Scripting

comm command -- Sort and compare two files

Team, I have two files and I am trying to find the lines unique to file1. So i have executed the below command at shell prompt and got the correct results comm -23 <(sort test) <(sort test1) When i run the same command in Bash shell script, i got the correct results. But when i run... (5 Replies)
Discussion started by: forums123456
5 Replies

7. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

8. Shell Programming and Scripting

Require compare command to compare 4 files

I have four files, I need to compare these files together. As such i know "sdiff and comm" commands but these commands compare 2 files together. If I use sdiff command then i have to compare each file with other which will increase the codes. Please suggest if you know some commands whcih can... (6 Replies)
Discussion started by: nehashine
6 Replies

9. Shell Programming and Scripting

Why do we use sort before compare ?

Dear All. Im trying to know how exactly the command "compare" works, does it compare line by line or field by field, and the most important thing is that why the files have to be sorted before we compare them? Thanks in advance (7 Replies)
Discussion started by: yahyaaa
7 Replies

10. Shell Programming and Scripting

Sort and compare file

If I have 3 kinds of files in directory DATA1: FileA.20060315.dat, FileB.20060315.dat, FileC.20060315.dat FileC.20060316.dat FileA.20060317.dat, FileB.20060317.dat FileA.20060318.dat, FileB.20060318.dat, FileC.20060318.dat If 3 files have the same date then run $cat FileA.20060315.dat... (3 Replies)
Discussion started by: sabercats
3 Replies
Login or Register to Ask a Question