Shell Programming and Scripting

If your command version is 32bit, it will use process that cannot address more than 2^32 bytes

Which is around 4GB

And if working/coded with signed number the range it can work with will be 4G wide but from -2G to 2G so the file size cannot exceed 2G.

Code:

# type fgrep
fgrep is /usr/bin/fgrep
# file /usr/bin/fgrep
/usr/bin/fgrep: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped
# bc
2^32
4294967296
# printf "%d\n" 4294967296 2>/dev/null
2147483647

so you should split your big file into piece that have a size that your fgrep will be able to handle or you need to run a 64bit version of fgrep on a 64bit plateform so it can address 2^64 so it can deal with largefile.

Also see

Code:

man largefile

i did try by splitting it.
For the first loop it worked fine. But while going to second loop it is taking a long time.
split -l 10000 file1 file1.split. for CHUNK in file1.split.* ; do fgrep -f "$CHUNK" file2 done rm file1.split.*

Can you please check if i am doing something wrong here. Or can you provide me some awk command.

Give a try with

Code:

awk 'NR==FNR{a[$1$2];next}($1$2 in a)' File1 File2

use nawk or /usr/xpg4/bin/awk instead of awk if you run SunOS or Solaris.

it worked.. thanks a lot