AIX

Hello guys
I am experiencing a very strange behavior on one of our AIX servers. We have an application with several processes that listen on several port numbers. Sometimes we receive complains that people cannot connect to the server on a specific port that is used by one the application processes. When we check for the process status we can see that the process is up and running and everything is ok. But when we run

Code:

netstat -an | grep 1234

where 1234 is (for example) the port number, we don't find the port in the list of the listening ports.
The obvious solution is to kill and restart the process. Well this doesn't always solve the problem as sometimes it stays that way even after restarting it and stays for a few minutes before listening.
Now the question is: what could possibly cause an application in AIX to be running but not listening on the port where it should be listening?
Does the existence of too many concurrent connections to the same port cause this behavior?
Thanks a lot in advance and I appreciate any help from your side.

Regards

Sounds more like a bug in the application rather than an OS thing. Have you tried checking with the application support team? Does the application log any error about shutting down ports?
I have seen Java applications delay in opening ports. But, shutting down a listening port is different, may be due to some restriction in the application's config, etc. In your situation, I would seek help from the apps team.

Sure i will check with the application's support team. I just want to rule out any possibility that this behavior is related to OS.
What about the number of concurrent connections? is it possible that initiating too many of these can cause the port to close ??

Does your system use the inbuilt firewall? You may want to check the max. connection settings for that port using the tcptr utility. For details, take a look at here:
IBM AIX TCP Traffic Regulation

Such a thing is impossible by design of the IP-stack. Let me elaborate:

First, what is a port: a port is an address on layer 4 of the network stack - just like the IP-address is an address on layer 3 (and the MAC address on layer 2). Other than the IP address it identifies a service rather than a host. There are 2 bytes identifying the port and therefore there are 65535 possible ports. Think of it like appartments to a house: the IP address identifies the house itself, the port number identifies the appartment. if you ring at different doors, different people will open, even if you are still in the same house. Some might not open at all, because the apartment is empty.

Communication now works in this way: behind any port(-number) a daemon can listen and offer some service - or not. If a daemon indeed listens it will pick up what comes its way and somehow react - by answering, by doing something, establishing a connection, whatever. Some ports, especially the ones up to # 1023, are for offering standard services: telnet, ftp, http, ssh, lpd, .... and so on. These are called "well known services" and only root can start a process using these ports. Usually a symbolic name is defined for these ports - see "/etc/services" - but this is not necessary.

You can even try this with a printer: if you have a network printer it sure runs a "lpd" daemon. This listens at port 515 per default. Using your telnet client (it allows to configure the port used) try:

Code:

telnet printer.yournetwork.com 515

Of course you will not get any telnet connection - the lpd speaks lpd and not telnet - but you will probably get some banner, identifying the printer. Something like "HP JetDirect Line Printer daemon v1.23 .....", maybe listing some EPROM software revision or so before it closes the connection.

If no daemon is listening, there is simply no answer. If you attempt to initiate a communication with a remote port where no daemon listens you get a "connection refused" answer usually, but this comes from your system, not the remote one - the remote one is simply not saying anything at all.

You see, there is no "closed" ports because ports can neither be closed nor opened. They are just listened at - or not. The daemon listening itself can of course terminate a certain connection and send your system the message "connection closed" - but this does not "close" any port, it just terminates the status that to a certain port someone is listening - if using some specific port was part of the session setup. Some daemons are contacted at a common port, then set up connections at different ports (above 1024) and listen again at the common port for new sessions.

I hope this helps.

bakunin

Thanks kindly for your time and effort and i really appreciate your elaboration,

Also a good tool to use (may need to install on AIX) is "lsof".
You can use this to determine what running processes are using ports (or not).
lsof

Using the -i flag will give you a better view of what's actually happening in your IP space on the server.

It will reconcile ports to services, so if you are hijacking a service port for non-standard use, be aware the output may mislead you

HTH