You can't connect two modems jack-to-jack and expect them to work, you're missing something important, the telephone company. You'd need something odd like long-haul modems to connect without a telco.

I read an article in a magazine long ago -- might have been Popular Electronics -- schematics for a device which would do this; a sort of telephone-company-in-a-box. It would generate dialtones and react to a number being dialed on one end by ringing on the other, but it was a big and complicated device due to the high voltages needed to drive telephone lines and the complexity of having to understand DTMF audio. They could probably miniaturize it further these days, but most hobbyists don't care about POTS anymore.