UNIX for Dummies Questions & Answers

Hi,

As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line.

As DIFF command wont work for big files, i tried to use BDIFF instead.

I am getting incorrect number of records.

I have done the following test:

I have got a dat file with a few million records in it and to generate a another file i have used sed '1,100d' oldfile > newfile

so i am using Bdiff oldfile newfile | sed -n '/^</p' > DIFF.DAT

The output(DIFF) should be having 100 records in it. But i am getting an output with several records in it.

Could anyone help me out from this situation?

Thanks

Sue

hi,

u can try this code :

printf "%s\n" $(comm -3 file1 file2)>newfile

hope it will work

Hi,

Thanks for the reply. The command is working for small files.

When i try to use with my file(6.5 GB), I am getting the following error:

comm: poel.dat: Value too large for defined data type

Please advice.

Thanks

Sue

Any Auggestions Please?

Thanks

Sue

Do you know about bdiff - meant for large files? Does what diff does, except it works on large files.

Yes, I know that Bdiff is used for large files.

Do we need to sort the files when we use bdiff?

How to get rid of linenumbers and < and > symbols in front of each record of the outputfile?

Thanks

Sue

bdiff does not work very well as you have discovered... although it should have behaved better than you reported. If you get bdiff working "right", it should report way too many differences. This is probably not what you want. If you are sure that one file is a superset of the other, a custom script to scan the file would probably be the best approach... especially with such large files. What os are you using?