May I suggest you try using the tip command to talk out of the appropriate serial port and send:
ATI3
which the modem should respond with its version number?
The
tip command uses the file
/etc/remote to configure a connection for you, copy the hardwire line(s) and change "hardwire" in your copied line to "modem" (e.g.) the port and baud rate to what your modem expects, then do run:
# tip modem
then once you have the connected message send the ATI3 command.
this will test the wiring of the serial port and the modem without too many layers of software in the way.