AIX

Hi,
I have migrated some processing from true64 --> AIX 5.3. my problem is to process large files in memory by diff or awk program. I need to load app. 1.3 GB of data into memory but it fails that there is not enough memory.

I need following:

diff file1 file2 [ each file has approximately 650 MB ]

orig. aix diff err message:
diff: 0653-807 The specified files are too large. Try the -h flag

gnu diff err message:
diff: memory exhausted

by awk [ Load files into array ]

err message: awk: 0602-561 There is not enough memory available now.

Max memory used by one process can be approximately 1.1 GB, then it fails.

My ulimit setting:

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000

Server has 26GB of physical memory and mem used max 50%.

Any help how to get rid of this limit ?

Thank you very much.

Petr

It is AIX 5.3.

Try running truss or tusc on the diff or awk command. Might give you a clue as to whats being exhausted. I suspect a kernel param rather than memory. Check the system hard limits

I might be totally wrong but it might be related to the memory model used by those 2 diff binaries when allocating memory, here's a link to some discussion about that.
We had once an issue in which we cant use but just very little portion of total memory available, it was solved by re-compiling the binaries using flags mentionned in the link above