Do you have a firewall installed somewhere between you and the 'net? If you have an open machine that someone can query, you might have open, unsecured ports live and on the 'net. If so, someone could access your X Window session via one of the X ports (6000 I think is the main one). Do a netstat -a and post that (in a code block if you please), of course only if you don't have a firewall of course.

You might try the scanner over at grc.com. Try Shield's Up! and see what it scans on your system.

Carl