Shell Programming and Scripting

Hi,

I have few files in a directory. I have the same set of files in another directory. I need to remove the lines starting with the word 'HDR' and 'FTR' if present from all the files in both the directories. Then i need to sort the contents from all the files in both the directories and then compare the two directories.If they are not same i should report the that the two files are different. If any file is additional in either of the directories, Then i should report that the file is present in one directory and not in the other. The columns in the files are either tab separated or | (pipe) separated.

One thing is that the file size may be huge. It might be in hundrerds of MB's.

Can anyone help me in givind a script to do this job?

This would be a lot simpler if you could make a temporary copy of both directories. Is this feasible, or are they too large?

Code:

mkdir /tmp/dir1 /tmp/dir2
for file in dir1/* dir2/*; do
  egrep -v '^(HDR|FTR)' "$file" | sort >/tmp/"$file"
done
diff -rad /tmp/dir1 /tmp/dir2 | egrep '^(diff|Only in )'

... assuming dir1 and dir2 are the original directories you want to compare. If they are in different places in the file system, perhaps it would be easiest to create symlinks for the duration of this script. And of course, once you are done, you can remove the copies in /tmp/dir1 and /tmp/dir2.

The output from diff is not particularly intuitive but it contains the information you request; you can post-process it with sed, or simply load it in your editor and massage it into something your manager can understand.

I'm not sure I captured the sorting requirement correctly. This creates sorted copies of each file, after removing the HDR and FTR lines; if you want all the content in a single file, of course, that can be done, too (but running diff on a single huge file is going to be painful).

diff is really overkill for this problem, but it produces exactly the information you wanted. It might be easier on the hardware to run cmp and then separately check for files which exist in one directory but not in the other.

Hi era,

I am getting this message and the temp directories are not getting created and the files are not compared.
bash: /usr/local/bin/..: is a directory

The code i used is

mkdir /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/tmp/dir1 /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/tmp/dir2
for file in /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing/* /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing1/*; do
egrep -v '^(HDR|FTR)' "$file" | sort >/tmp/"$file"
done
diff -rad /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/tmp/dir1 /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/tmp/dir2 | egrep '^(diff|Only in )'


Why is this happening?

The variable "file" contains the whole full path, and if the same path doesn't exist under /tmp it will complain about the redirection. That's what I was alluding to with the suggestion to create symbolic links to the directories.

Code:

ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing dir1
ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing1 dir2
mkdir /tmp/tmp1 /tmp/dir2
for file in dir1/* dir2/* ...

Hi era,

I am new to unix. i am getting this error. When i run the script.

++ set -x
++ ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing dir1
++ ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing1 dir2
++ mkdir /tmp/dir1 /tmp/dir2
mkdir: Failed to make directory "/tmp/dir2"; File exists
++ egrep -v '^(HDR|FTR)' dir1/file1.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir1/file2.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir1/file3.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir1/testing
++ sort
sort: missing NEWLINE added at end of input file STDIN
++ egrep -v '^(HDR|FTR)' dir2/file1.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir2/file2.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir2/file3.txt
++ sort
++ egrep -v '^(HDR|FTR)' dir2/testing1
++ sort
sort: missing NEWLINE added at end of input file STDIN
++ diff -rad /tmp/dir1 /tmp/dir2
++ egrep '^(diff|Only in )'
diff: illegal option -- a
usage: diff [-bitw] [-c | -e | -f | -h | -n] file1 file2
diff [-bitw] [-C number] file1 file2
diff [-bitw] [-D string] file1 file2
diff [-bitw] [-c | -e | -f | -h | -n] [-l] [-r] [-s] [-S name] directory1 directory2

The directories i need to compare are testing and testing1 which are in the path /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/

The code i used is
ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing dir1
ln -s /gdw/shared/qa2/dev/scripts_regr_test/design3/extract_comparison/testing1 dir2
mkdir /tmp/dir1 /tmp/dir2
for file in dir1/* dir2/*; do
egrep -v '^(HDR|FTR)' "$file" | sort >/tmp/"$file"
done
diff -rad /tmp/dir1 /tmp/dir2 | egrep '^(diff|Only in )'

But its still failing. Can you give me the exact code?

You have a typo in the mkdir, you have "mkdir /tmp/dir11" with one 1 too many.

(Looking back, I had the wrong directory names there too, tmp1 and tmp2 instead of dir1 or dir2.)

Of course, once the symlinks and directories are in place, the code to create them won't need to be run again.

If you have the following:... then you should be ready to go with the for loop.

Looks like you will also need to drop the -a and -d options to diff, so just diff -r

(For the record, my diff has an -a option which says to always treat all files as text, and -d basically means try a little harder algorithm in order to keep the diffs small. I guess you can cope without either of those.)

If testing and testing1 are subdirectories among the files, perhaps you want to skip those from the loop.

Code:

for file in dir1/* dir2/*; do
  test -d "$file" && continue  # skip if it's a directory
  egrep -v '^(HDR|FTR)' "$file" | sort >/tmp/"$file"
done
diff -r /tmp/dir1 /tmp/dir2 | egrep '^(diff|Only in )'

Testing and testing1 are the directories containing the files which have to be compared.

This script is going to be automated and will be a generic one and will be run by using a java code.so i cant create symbolic links each and every time i need to compare two directories. can the code be modified to suit this purpose?