Solaris 8 is quite old.
The described symptom indicates a software problem.
Most likely you need to apply a Xsun patch (X-server).
Or an X-client (Openwindows/Openlook/Xview or Motif) patch (that...
I suspect an application that traps the mouse. The application you are running (which you don't say) will hopefully have a magic-key combination to release the mouse pointer from the Active Window...
If your mouse has a round ball inside, you may want to take it out and clean it with alcohol, or at least water. Make sure that you don't have lint building up inside.