AIX

What application is that you are running? I think what you thought were recovered socket are more new sockets after a cleanup... now if you think or see an issue here it can be a question of time out ( from the other side...) where your remote hosts believed the connection lost and closed it... The issue in this case is it opens a new one again, and after some time you find yourself short of sockets as you are wating for the cleanup process to complete
So you should look at the application side if there isnt something that needs tuning...

Most likely what happened was that the connections which have been half-removed before were being completely dissolved and then new connections (which you saw as "ESTABLISHED") were made.

AIX has some timeout value between a connection going into state "CLOSE_WAIT" and it being dissolved completely (i don't know it off the top of my head, but it is some no-tunable). IIRC (sorry, its been some years i needed it last) the unit it uses are half-seconds. You can set that to some shorter value to dissolve connections not longer in use quicker if the need arises (usually it doesn't but there is always the exception to the rule).

It might also be that the application is sloppily programmed and the used sockets are not freed properly, stalling the cleanup-process. This might be tricky to track down, but if this is your problem you will have no other option to do just that (and then beat the programmer with a copy of "UNIX Network Programming").

I hope this helps.

@agent.kgb: actually in no universe 2+2 can be equal to 5, at least not when the symbols "2", "5", "+" and "=" are used in way equal to or at least similar to how we use them. For details about why this is so refer to Bertrand Russells "Introduction to Mathematical Philosophy, which is an outstanding classic in this field.

In short: a "(natural) number" is defined as a successor of its preceding number (yes, its a recursive definition) the recursion stopping at zero. So "1" is the successor of zero, "2" is the successor of 1 (or the successor of the successor of zero), and so on. In short we write "S0" for "1" and "SS0" for 2, etc.. Because the addition being defined as it is we can concatenate these lists of successors and arrive at "SS0" + "SS0" being the "successor of the successor of SS0" or, in other words, "SSSS0", which is 4. If we would define "+" as an operation where the operands "SS0" and "SS0" result in "SSSSS0" this the set of natural numbers would be no longer an abelian group regarding to addition which would have a lot more grave consequences than you can possibly envision.

bakunin