Shell Programming and Scripting

HI,

I have few files in two folders with the same name exceeding 2GB.I need to compare these files. These files are in the format

File1 in first folder
1|20080430|IA001|TREND DYNAMICS INC
2|20080430|IP001|AMERITAS LIFE INSURANCE CO
3|20080430|IP002|TRANSAMERICA LIFE INSURANCE CO

File1 in second folder
1|20080430|IA45|TREND DYNAMICS INC
2|20080430|IP001|AMERITAS LIFE INSURANCE CO


The files may be pipe or tab separated.

What i need to do here is to sort both the files, then compare. But the problem here is since the file exceeds 2GB sort command wont work and the diff command wont work. The comparison has to be line by line and field to field. The output should be in this format

For lines from files in first folder i need to indicate it by appending "From Test1" to the beginning of mismatching line like this

From Test1 - 1|20080430|IA001|TREND DYNAMICS INC

For lines from files in second folder i need to indicate it by appending "From Test2" to the beginning of mismatching line like this

From Test2 - 1|20080430|IA45|TREND DYNAMICS INC
And if a line found in file 1 of first folder is not found in file 1 of second folder then print that line alone to my output file


Hence my Final output should be like

From Test1 - 1|20080430|IA001|TREND DYNAMICS INC
From Test2 - 1|20080430|IA45|TREND DYNAMICS INC

From Test1 - 3|20080430|IP002|TRANSAMERICA LIFE INSURANCE CO

Is there a way to do it?

i) get yourself a better sort command (e.g. gnu sort should work and runs on nearly every unixoid system)
ii) write yourself a short sort programm in some suitable high-level language like perl

Sorry. I am new to UNIX and dont have knowledge on PERL.
Can you please help me?

So better take option i) - this is much easier. Get the package GNU coreutils or GNU fileutils (old name, quite much the same content) for your OS.