UNIX for Advanced & Expert Users

Hey guys,

We've been having issues on one of our CentOS 6 servers and one of the java programs that runs on it. As the software hasn't caused issues in the past, I'm wondering if its a problem with the CentOS server.

Basically, the software drops it's tcp connection and won't reconnect, timing out on each occasion. I'm wondering if the number of open tcp connections on the server has exceeded a limit ? Where would a tcp limit be held ? I've had a look at /proc/sys/net/core/somaxconn and the limit in there is 128 (which seems quite low although is the default on our other 5 CentOS boxes with no issues) - would this benefit from being increased ? Also, anyone any ideas on what else I could check ? I've had a look in the /etc/sysctl.conf and there's nothing in there that relates to any incoming connection limit.

Much thanks in advance for any help, it's driving me crazy!

Jim

Depending on the type of network connection (i suppose TCP/IP?) there are a few limits, but i doubt that you have hit them:

A normal TCP connection (say, a telnet session) is done via a "virtual channel": while it is establishing a "socket" (this is a Layer-4 addressing device, an IP-address [=layer-3] combined with a port number [=layer-4]) connection is defined via which all the communication is done. The communication runs from host-aort-x to host-bort-y and vice versa. Once the session is closed this socket connection is decomposed and the used ports are released and ready to be used again.

So, in a way, the number of available ports are a limiting factor, but as they are 16-bit numbers (1-65535) this isn't all to imposing.

Another limit is the available memory. Some OSes have tuning options how much memory i set aside for various aspects of the networking stack (for instance TCP reassemble buffers), but i don't know how this is done in Linux, just that this exists. Hopefully someone more knowledgeable regarding Linux than me will fill in this gap.

Still, modern systems on modern networks usually have enough memory to handle the network with ease. While this should be investigated (to make sure the problem doesn't sit there) it is rather unlikely that this is the problem.

What might be a problem is: if the application runs under a normal UID it might lack the necessary ulimits?

What does the application do when it drops the sessions? Does it throw a coredump or does it close normally or does it still run? What do you have to do to revive the system? Reboot? Restart of the application? Just wait?

The more you tell us about the system the easier it is to give concrete advise instead of some general chit-chat. So help us help you and explain in more detail what your system looks like (versions, how much memory/processors/swap space, which network connections) what it does (running applications, what they are doing, how many logins/how much network traffic/how much disk I/O) and some traces: "vmstat", "iostat", etc.

I hope this helps.

bakunin

Thanks for your reply, bakunin.

The application itself is run under root and it's ulimit is unlimited. When the application drops it's connection, it constantly attempts to reconnect and just logs this every 30 seconds as the service is still running. To get it to work again, we simply restart the application and on the very odd occasion, the full server. I don't think memory is a factor, all 6 servers have 16GB of RAM installed and have around 20GB of swap.

I'm probably going off on a tangent, but the number of connections was something that popped into my head

Jim

Could you post the output of vmstat 1 during the loss of connection?

You say that the application runs for some time and only then stops. This sounds like some resource being used and not being released properly, so that it exhausts over time. It might be that "vmstat" is showing some effects of this.

To be honest, there is no obvious reason one could point at. So good old problem solving skills and patient investigation is what is called for.

I hope this helps.

bakunin

Moderator's Comments:

As this is far from being a "Unix for Dummies" problem i am going to transfer the thread from here to the experts section.

I've seen this behavior when clients do not close the connection properly, resulting in lots of connections in state TIME_WAIT or CLOSE_WAIT when viewed with netstat -n on the server.

bakunin - thanks for your reply. Once the issue happens again, I'll post the output of vmstat. It's a peculiar problem in that it could happen today and then not happen again for a day, a week or a month, there's no pattern in it.

cero - thanks for your reply. When the problem arises again, I'll check the open connections.

Thanks guys, your advice and help is appreciated.