Well the sorted find files differ by ~3000 lines. I take this to mean that there are ~3000 files that are missing from the one directory. The output off comm is 3091728 lines, which is the same number of lines as are in the find for the original directory. I presume this is because the col 3 output of comm are files that are in both, and output I don't need to see. I presume I want comm -3 for the output I want, meaning files that are in one director tree and not in the other?

LMHmedchem

---------- Post updated at 05:28 PM ---------- Previous update was at 02:43 PM ----------

This is the final script that I used. I have brushed it up a bit so that it is more generalized and checks a few things.

Code:

#!/usr/bin/bash

# accepts path to two directories and compares the file lists in each
# path should begin with /cygdrive/driveletter/

# assign location for output
TMPDIR='/cygdrive/c/cygwin/tmp/dir_compare'

# assign directory trees to compare
TREE1=$1
TREE2=$2

# check arguments, print help if no arguments are passed
if [ $# -eq 0 ]
then
   echo "this script expects two arguments"
   echo "each argument should be the path to a directory"
   echo "each path should start with /cygdrive/, not a relative path"
   echo "the script will compare the list of files in the directory and subdirectories"
   echo "and will report any instance where a file exists in one directory but not the other"
   echo 'output will be printed to '$TMPDIR'/comm.txt'
   exit
fi

# check if TREE1 exists
if [ ! -d $1 ];
then
   echo " "
   echo "directory " $1 "not found"
   echo "exiting"
   exit
fi
# check if TREE2 exists
if [ ! -d $2 ];
then
   echo " "
   echo "directory " $2 "not found"
   echo "exiting"
   exit
fi

# clean tmp dir if it contains files
cd $TMPDIR
FILES=(*)
FILES=${#FILES[@]}
if (( "$FILES" > 0 )) ; then
   rm *
fi

# echo some information
echo " "
echo "comparing file list of " $TREE1
echo "with file list of " $TREE2
echo " "

# cd to TREE1 and create file list for tree
cd $TREE1
find . > $TMPDIR'/check_1' &

# cd to TREE2 and create file list for tree
cd $TREE2
find . > $TMPDIR'/check_2' &

# wait for find to finish
wait

# sort output of find to keep file list from both dir trees is in registration
sort $TMPDIR'/check_1' > $TMPDIR'/check_1_sorted'
sort $TMPDIR'/check_1' > $TMPDIR'/check_2_sorted'

# print the number of lines (files) in each directory tree
wc -l $TMPDIR'/check_1_sorted'
wc -l $TMPDIR'/check_2_sorted'

# compare the two files, only print instances where a file exists in one tree but not the other
comm -3  $TMPDIR'/check_1_sorted'  $TMPDIR'/check_2_sorted' > $TMPDIR'/comm.txt'

Running this script indicates that I have 6,189,828 files in each tree and the script does not find any difference in file names. I found that I had one extra directory in one of the trees. This came from some testing I was doing to see if a copy of my files had the same issue with the time stamps as the original. When I deleted this copy directory, the comm file is empty.

The only problem is that I still have a 3GB size discrepancy between the two partitions.

$ df -h
Filesystem Size Used Avail Use% Mounted on
E: 879G 502G 378G 58% /cygdrive/e
I: 831G 499G 332G 61% /cygdrive/i

The size of the E partition didn't change when I deleted the extra directory, even though the folder was quite large. I expected that to make the sizes the same. I'm not sure what else I can do to check that my copy has all of the data from the original. The results would imply that some of the files exist on both drives, but are not the same size. Is there a reasonable way to check that? I would seem like that would be a non-trivial addition to what I am doing. Is it possible for the same exact files to be on both drives but to take up different amounts of space?

LMHmedchem

---------- Post updated at 05:44 PM ---------- Previous update was at 05:28 PM ----------

I see I had a typo in the script, so I wasn't doing the correct compare. I am running again with the corrected script.

---------- Post updated at 06:51 PM ---------- Previous update was at 05:44 PM ----------

Running the corrected script, there are a few files that are different, but the total size is not much. I keep my browser profiles here and these are different because one is the browser I am using and one is a copy made yesterday.

There is nothing here that accounts for 3GB of data.

Any suggestions on what to do next? I suppose I could use the sorted find files to do a diff between each file pair, but that wouldn't exactly be speedy. The find files don't differentiate between files and directories and I don't know what happens if you feed diff a pair of directories instead of files.

LMHmedchem