Compare the checksum of files in 2 different folders


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare the checksum of files in 2 different folders
# 1  
Old 12-18-2009
Compare the checksum of files in 2 different folders

Hi

I have 2 different folders on different machines. they are supposed to be same but some time for unknown reason they are not. then we have to generate a report for files which are not matching.

I was doing as below -

Code:
cd folder1
find . -type f | sort | cksum >1.txt

cd folder2
find . -type f | sort | cksum >2.txt

1)
Code:
diff -w 1.txt 2.txt

still it shows lot of lines in diff even when file name/path/size/checksum is same. is there any nice and clean way to report the expected result.

2) i was able to generate good results (but not good format) by using -
Code:
cat 1.txt 2.txt | sort -k 3,3 | uniq -u

it does not tell me which file is from which folder (there is no < or > sign as we have in diff)

is there any other best way to do it ?

Thanks
Rel

Last edited by Franklin52; 12-19-2009 at 08:40 AM.. Reason: Please use code tags!!
# 2  
Old 12-18-2009
Quote:
Originally Posted by reldb
...
I was doing as below -

cd folder1
find . -type f | sort | cksum >1.txt

cd folder2
find . -type f | sort | cksum >2.txt

1) diff -w 1.txt 2.txt

still it shows lot of lines in diff even when file name/path/size/checksum is same. ...
You may want to list, sort and find cksum of each file as shown in the following example:

Code:
$ # go to directory dir1
$ cd dir1
$
$ ls | sort | xargs cksum
1958180199 14 file1
1994293486 14 file2
1996893289 14 file3
$
$ # now do the same for directory dir2
$ cd ../dir2
$
$ ls | sort | xargs cksum
1958180199 14 file1
1994293486 14 file2
1996893289 14 file3
$
$

And as you are doing currently, redirect the output to two files and find the diff.

tyler_durden
# 3  
Old 12-19-2009
tyler_durden
thanks for suggestion
there are 2 issues

1) first ls wont show the files in sub directories. so we have to use find command. thats fine

2) main issue is still the same. if you do the diff of output from folder 1 and output from folder 2 it wont show properly as both the folder will have different no files so sort will be differnet and so the diff
# 4  
Old 12-19-2009
Code:
diff directory1 directory2

# 5  
Old 12-19-2009
if you can use Python, you can use its filecmp module
Code:
#!/usr/bin/env python
import filecmp
import os
directory1 = os.path.join("./","dir1")
directory2 = os.path.join("./","dir2")
cmp = filecmp.dircmp(directory1,directory2)
print cmp.report()
# print cmp.common
# print cmp.same_files # diff_files etc....

sample output
Code:
$ ls -1 dir1 dir2
dir1:
1.txt
2.txt
3.txt

dir2:
1.txt
3.txt
4.txt

$ ./python.py
diff ./dir1 ./dir2
Only in ./dir1 : ['2.txt']
Only in ./dir2 : ['4.txt']
Identical files : ['1.txt', '3.txt']
None

# 6  
Old 12-19-2009
How about using chksums -c option? On the first machine run
Code:
cd /dir1
find . -type f -exec chksum {} \; > chksums

Copy it over to the other machine and run
Code:
cd /dir2
chksum -c chksums

That should give you a listing of those files that differ.
# 7  
Old 12-19-2009
Or if you would like to program in Perl, then you may want to consider using the File:: Dircmp module -

Code:
$
$ cat -n dc.pl
     1  #!/usr/bin/perl -w
     2  use File::Dircmp;
     3  @r = dircmp("dir1","dir2");
     4  foreach $i (@r) {
     5    print $i,"\n";
     6  }
$
$ perl dc.pl
Only in dir1: dir11
Only in dir2: dir21
Files dir1/file2 and dir2/file2 are identical
Files dir1/file1 and dir2/file1 are identical
Files dir1/file3 and dir2/file3 differ
$
$

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to compare files in 2 folders and delete the large file

Hello, my first thread here. I've been searching and fiddling around for about a week and I cannot find a solution.:confused: I have been converting all of my home videos to HEVC and sometimes the files end up smaller and sometimes they don't. I am currently comparing all the video files... (5 Replies)
Discussion started by: Josh52180
5 Replies

2. Shell Programming and Scripting

How to copy files/folders and show the files/folders?

Hi, So i know we use cp -r as a basic to copy folders/files. I would like this BUT i would like to show the output of the files being copied. With the amazing knowledge i have i have gone as far as this: 1) find source/* -exec cp -r {} target/ \; 2) for ObjectToBeCopied in `find... (6 Replies)
Discussion started by: Imre
6 Replies

3. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that... (1 Reply)
Discussion started by: S.Praveen Kumar
1 Replies

4. IP Networking

Wireshark UDP checksum bad checksum

Hello I am communicating with two devices using my computer over UDP protocol. The application is running fine. When I monitored the UDP traffic using Wireshark software, I found that there were too many Checksum errors. Please find attached the png file showing this error. I am about to... (0 Replies)
Discussion started by: AustinCann
0 Replies

5. UNIX for Dummies Questions & Answers

Searching for folders/parent folders not files.

Hello again, A little while back I got help with creating a command to search all directories and sub directories for files from daystart of day x. I'm wondering if there is a command that I've overlooked that may be able to search for / write folder names to an output file which ideally... (2 Replies)
Discussion started by: Aussiemick
2 Replies

6. Shell Programming and Scripting

How to compare files in two folders using cmp?

i recently copied 400GB of data from a NTFS drive to a ext4 drive. I want to verify that the data is 100% identical to the original. I wanted to use cmp but it only does two files. The directory that was copied contains many subdirectories and all sorts of files (not just text). So I guess... (5 Replies)
Discussion started by: fuzzylogic25
5 Replies

7. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

8. Shell Programming and Scripting

Compare files in two folders and delete missing ones

I do not know much about shell scripting so I am at a loss here. If someone can help me, that would be great! I have two directories /dir1 /dir2 I need to delete all files from /dir1 and that does not have a correspondent file in /dir2. It should NOT check file suffixes in /dir2 . Why?... (20 Replies)
Discussion started by: kaah
20 Replies

9. Shell Programming and Scripting

Compare 2 folders...

Hello, I try to compare 2 folders, i explain, i have file in this 2 folder and i want to print out the difference in this folders... ex: folder1: file1 file2 file3 folder2: file1 file2 print file3 I do a ls of the 2 folders and i use the command diff (diff $var1 $var2) without... (8 Replies)
Discussion started by: protocomm
8 Replies

10. Shell Programming and Scripting

Remote compare of folders

Hi, Is there a way (either commands/tools/scripts/logic) to compare two given folders on different unix boxes. I want to compare folder a in Unix box 'A' with folder 'b' in Unix box 'B'. I can run the script in Unix box 'A'. I am looking. for following results: files/sub folders only in a... (1 Reply)
Discussion started by: sunilav
1 Replies
Login or Register to Ask a Question