Why would you want VirtualBox to connect its graphical interface to the
telnet server of the
wrong machine?
Let me explain how SSH port forwarding works.
You tell SSH to forward a port on the remote host to a port on your local host.
When you connect to the remote server with SSH, the SSH server opens an extra port on the local host itself, waiting for you to connect to it.
When you connect to it, the SSH server forwards this back to the client, your
client connects to whatever you told it to earlier.
So in effect, it creates a new port on the remote host, which connects it directly to X11 on your machine.
If you don't know what port number your X11 server is, you'll have to find out and forward it, since ordinary X11 forwarding doesn't seem to be working.