You are getting two different things mixed up. The max of a soft limit is the hard limit. On SunOS 5.8 I just observed that, for file descriptors, my hard limit was 1024 and my soft limit was 256. I raised my soft limit to 512. I am using ksh which has a builtin ulimit. Switch to ksh and try it:
ulimit -n
ulimit -Hn
ulimit -Sn 512
ulimit -n
At this point I can open 512 files in a single process. But I'll admit that I've never tried to write such a program.
The next thing is how many files can stdio manage. I must admit that I was skeptical about the information in the link you provided. But "man stdio" not only backs it up, but points out that things are a tad worse...
Quote:
Note that no more than 255 files may be opened using fopen(), and only file descriptors 0 through 255 can be used in a stream.
That means that cranking the fd's up to 512 may not help much. And you may not reach 255 fopened files because a few fd's were burned for other things.
HP's docs claim that you can fopen() as many files as you have fd's. But I have never verified that.
It's hard to answer your question about "easily" reaching 255. I have never come close to that. I continue to suspect that your "3pp" is leaking fd's. Since you seem to have support, I would follow up with the vendor.
lsof works fine for me on SunOS 5.8. "I've tried lsof but it does not seem to work." is not much to go on. The best I can do with that is say "you must be doing something wrong." If you were to post a few more details, maybe someone can help.