I can't explain why your attempt to raise your softlimit failed.

Each open of a file creates a file table entry. The size of the file table is a kernel tunable parameter called nfile. You need to look at your own kernel to see what that is. You won't be able to use all of the file table entries repeatedly opening some file. You will need to open a few other files just to set up your attempt. Each file table entry for a file increments a counter in the vnode for that file. If you increment it to far, it would wrap around. This might not stop you from opening the file some more, but it would not be a good thing. The size of the integer that holds nfile and the size of the reference count should be the same size integer thus preventing this problem. It may be cheating, but you can have an opened file without opening a file. There are various ways to "dup" a file descriptor. The dup'ed fd points to the same file table entry as the original fd. Unix traditionally limits pid's to 32,000 (but your kernel may have a different limit). So I guess the limit is 32000 * max-fd-per-process. But again, minus a few for the overhead of reaching the point where you could try this stunt.