A 66MHz, 64-bit PCI bus can handle 4 gbps, so unless it's a badly-engineered bus, that shouldn't be your problem.
Also, do NOT remove the autonegotiate feature from your network cards and switches if you're running gigE over copper.
1. If your links are not running at 1000 mbps, full-duplex, there's a problem. Papering over that underlying problem by forcing the link to 1000 mpbs full-duplex doesn't fix the problem.
2. Disabling autonegotiation on copper gigE places you outside the specifications of IEEE 802.3: (
http://www.sun.com/blueprints/0704/817-7526.pdf)
Quote:
The IEEE 802.3 standard states that you must support and test autonegotiation enabled to certify a product IEEE 802.3 compliant, and for multivendor interoperability (for example, testing at the UNH Interoperability Laboratory). There are no requirements in the standard to support locked down or forced configurations using autonegotiation disabled. As a result, there are no requirements for vendors to test multivendor interoperability between products with autonegotiation disabled.
The IEEE 802.3ab specification does not allow for forced mode 1000BASE-T with autonegotiation disabled running at 1000 Mbps. As a result, many switch vendors do not support forced mode.
I'd look to be sure you do all the tuning that Oracle advises for running over NFS. Especially make sure you're using jumbo frames.
And you might very well need to look into replacing your old hardware and software. There have been a lot of hardware and software advances in networking performance since Solaris 9 was current.