The basic problem is usually that you do not have permissions to manipulate the display on another host, even if you know its IP address. This is a security feature, designed quite on purpose (how would you like me to be able to open a window on 90.136.16.128:0.0 right here and now, just to see what would happen?) but if you have legitimate access to the display, you should be able to create a connection. It's not very simple, because there are several security facilities which need to be aligned and configured correctly.

Do you have ssh access to the machine where this display is connected? Is it a local machine? Are you able to open local windows there, for a start? Because of various issues of the X client/server model, it's often simpler to use a wrapper such as VNC to get access to (windows on a) remote desktop.