Just a wild shot in the dark: which NFS version do you use?

Historically, NFS (NFS, NFS2, whereas NFS3 optionally) used UDP as the transport protocol, because it was way faster than TCP in limited bandwidth environment (remember Ethernet being 10MBit/s). NFS3 could be configured to use either TCP or UDP and NFS4 uses TCP per default (exclusively? not sure).

The downside of UDP was lacking flow control (it is a connectionless protocol) and it is conceivable that under some adverse constellations a transmission could fail.

You could probably verify that (and several other network-related possible causes) by tracing the network extensively. Set up "tcpdump" with a customized filter to trace the NFS-related transfers from the load-balanced front-ends to the NFS-server behind.

Generally speaking, this sounds like a quite complex problem and you are probably best (even if not fastest) off by using methodical problem-solving skills. Write down all the steps necessary to transfer a file fron the user to where it should finally reside. Then write down any tool/driver/piece-of-software/... involved in getting the file there. For every such piece suppose that it could fail somehow and develop a test specifically designed to find out if it does (like i did with the "tcpdump"). Then falsify one of these "theories" after the other by applying this test. Eventually you should end with the failing piece.

You can probably shortcut the process by chosing suitable parts of the transmission and make sure these work. If you can make sure the error only happens past part X then all parts up to and including part X are working.

I hope this helps.

bakunin