Shell Programming and Scripting

I have two huge files in the size of 1gb. They are produced by similar processes and the expected thing is that they should match in size and contents. I have produced both the files with the processes and they seem to be off only by few bytes.

Size file name
1634502037 Transfer_delta_reservation_file.txt
1634501922 delta_transfer_file.txt

I tried to see the differences with the diff and sdiff but both of them are showing everything as different. I checked the files manually for couple of records and they seem to match exactly. I couldn't find out why there is this small difference in bytes and if so where are this extra bytes coming from. Is there a way or a program to just show the differences in the file and their line numbers. I did the diff after sorting them. Please help.

diff(1) can be told to ignore differences with tabs (--ignore-tab-expansion), spaces (--ignore-space-change) or (--ignore-all-space) and also carriage returns (--strip-trailing-cr), i.e. any number of cpaces or tabs are treated as one space or tab respectively or any form of "whitespace" is treated as equivalent and the presence of cr+lf or just lf is treated as erquivalent, all these are worth a try. Other parameters worth trying are --text and --speed-large-files.

Otherwise you can use od(1) to look at what is really in each line in ASCII, e.g. run:

Code:

$ head -10 <file> | od -bc

against both files (looking at only the first 10 lines of each) and see if that may give you a clue as why diff thinks every line is different.

Code:

cmp -l fiel1 file2

will show where the files start to differ. Pay attention to the values differed to see if there is a shift or change of values. Add -b to cmp option if you have it:

Code:

$ cmp -bl <(echo abcdefgh) <(echo abcDefgh)
                  4 144 d    104 D
$ cmp -bl <(echo abcdefgh) <(echo abcefgh)
                  4 144 d    145 e
                  5 145 e    146 f
                  6 146 f    147 g
                  7 147 g    150 h
                  8 150 h     12 ^J