First of all... you need to stay away from internal modems when using SCO Openserver Unix. Internal modems can cause spurreous (sp) interrupts on the bus. Your BEST bet is to purchase an external modem and hang off the serial port that's built in to your notebook computer.
Secondly.. there is no "Hyperterminal" application such that of Windows 95/98 etc.. for SCO Openserver. I assume you are trying to dial out and connect to a remote host? Research uucp and cu. The other option is to purchase a 3rd party communications application such as Blast, MLink, or Term for SCO Openserver unix.
Lastly, there is a product from SCO that will allow you to run DOS/Windows based applications from within Unix.... SCO MERGE.... however.. it is very resource heavy.. LOTS Of RAM..etc... and I am not sure if it's available any longer. I would simply setup a dual boot system with SCO Openserver and Windows on the same drive.. and boot the OS I needed. I guess the only problem in doing so would be that if you were using the SCO box for multi-user apps, you would lose that functioanlity while you were booted to the Windows partition.
SCO's web site is
www.sco.com Caldera bought SCO out, but support and service is still available.
Let me know what you are wanting to do with dialing out and I can give you more help in that area.