Red Hat

I have a RHEL 5.5 system that i see this behavior on:

# uname -a
Linux myhost 2.6.18-194.17.4.0.1.el5 #1 SMP Tue Oct 26 20:10:33 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

#using netcat to listen to a port tells me its in use.
$ nc -l myhostip 33371
nc: Address already in use
$

# trying to telnet to the port
$ telnet myhostip 33371
Trying myhostip...
telnet: connect to address myhostip: Connection refused
telnet: Unable to connect to remote host: Connection refused

# lsof doesn't agree. returns nothing
$ lsof | grep 33371
$

# another option to lsof thats supposed to find sockets. returns nothing.
$ lsof -i :33371
$

# netstat -an doesn't show the port listening or in time_wait or in close_wait state.
$ netstat -an | grep 33371
$

Anyone have any idea on how to figure out what is using that port? I don't want to just reboot to fix.

I don't suppose you're using selinux?

no not using selinux...

a colleague of mine used system tap to track the inet_csk_bind_conflict function and found that "nscd" process was the one hanging on to the tcp port.

this is what the system tap trace looked like:

inet_bind(nc): socket [state 1 type 1] uaddr 0.0.0.0:23426 addr_len 16
tcp_v4_get_port(nc): sock [reuse 1 state 7] snum 33371
bind_conflict(nc): sock [reuse 1 state 7] tb [port 33371 fastreuse 0 (reuse 0 state 7 owner (pid 0 uid 0 euid 0) uid 28 gid 28)]
bind_conflict: return 1 (EPERM)
tcp_v4_get_port: return 1 (EPERM)
inet_bind: return -98 (EADDRINUSE)

$ getent passwd 28
nscd:x:28:28:NSCD Daemon:/:/sbin/nologin

Restarting nscd freed up the port.

Anyway, 2 morals to the story:
1) I have some wicked smart colleagues.
2) when in doubt, restart nscd!

To find out what's using the port type:

Code:

lsof -i :(your-port-number-here-without-parens)