There's a pretty good overview at Wikipedia.

The tl;dr version:
The X Window System can be seen as 3 components interacting.

First, there's the server component, whose job is to (basically) tell the graphics card what to draw on the monitor (screen), and reads keyboard and mouse input. Each computer can run multiple server instances, where each uses a distinctive display (defined by the server it belongs to, the size, resolution, color depth, and network port [more on that later]). The server is usually started by the xinit program, which in turn is usually started by the startx shell script, which does some preliminary setup.

The second part is the client, which is the application the user wants to interact with. It can, but doesn't have to, run on the same machine as the server. Window managers and desktops like Xfce, Gnome, KDE, or twm are all clients here, with the main job of providing easier access.

Last but not least is the network protocol, which is the way the client and server communicate with each other. Keyboard and mouse input received by the server is sent to the client, which acts on it (or not ), and as a result sends new drawing instructions back to the server, which then tells the graphics card what to draw. Rinse, repeat.

Modern Linux distributions mostly install the server and some client programs by default, and the server is automatically started at boot as a daemon, with the login manager (XDM, GDM, KDM, ...) as the first client, which then handles user logins, etc. Configuration files for this and other things reside in /etc/X11, or sometimes just in /etc, or some other subdirectories, which is distribution specific.

But in some cases there's no need to install and run a graphical user interface, for example with servers.

But some servers do have the libraries and programs installed that are required to communicate with a remote X11 server. For example, it's possible to install an X11 server on a Windows machine, connect to a remote machine using PuTTY and have a feature called "X11 forwardning" enabled. Here the network communication is tunneled through the SSH connection, and client programs like xeyes are displayed as if running on the Windows machine. With this feature the SSH daemon prepares the environment so that clients can easily connect to the server without any additional setup required by the user.