128MB of RAM is quite small by today's standards. Why don't you run the command top to see what is taking up your resources. However, if you really want to up your performance, you should get some more RAM. It's very cheap on ebay.

I have a P3-800 with 384MB of RAM running redhat fedora core 2 and KDE. I dont have problems with X being slow.