The Solaris bundled package can't be updated but it can be removed and an alternate third party, up to date package can be installed instead of it, or even built from source. I already suggested a...
Solaris 8 should run fine.
The U5 service manual is describing a four 256MB DIMM configuration leading to 1 GB so I guess it is possible.
The obvious risk is to run unpatched services/libraries....