UNIX for Dummies Questions & Answers

grep -f taking long time to compare for big files, any alternate for fast check

I am using grep -f file1 file2 to check - to ckeck dups/common rows prsents. But my files contains file1 contains 5gb and file 2 contains 50 mb and its taking such a long time to compare the files.
Do we have any alternate for fast/quick check comparetion ?

try 'fgrep'

Code:

User Commands                                            fgrep(1)



NAME
     fgrep - search a file for a fixed-character string

Even fgrep won't scale for larger files as the operation is o(n2). Best approach is to hash one of the files and do a o(1) lookup best case with the other file

I want to check/compare file1 rows into file2 rows, if file1 rows present in file2 then needs to display like that compare for all rows in file1 to file 2 and it varies the content also. how to do ? the current one grep -f file1 file2 is working my scenario but its looking such a long time, any alternate ?

Can you please post some sample input and output data?

Or read the man page on comm - that is designed to find rows found in two files.

file1.txt conains:
1 2 3 4 5
1 1 1 1 1
2 2 2 2 2
1 3 5 7 9
4 4 4 4 4

file2.txt contains:
1 2 3 4 5
1 3 5 7 9
7 7 7 7 7
9 9 9 99

the output of file3.txt should be
1 2 3 4 5
1 3 5 7 9

Here grep -f file1.txt file2.txt > file3.txt : its working fine for small size of files but large files size (file size conatns GB) then its not working/not at all comming back...?