Port hang issue in AIX Version 5.3


 
Thread Tools Search this Thread
Operating Systems AIX Port hang issue in AIX Version 5.3
# 1  
Old 08-11-2009
Port hang issue in AIX Version 5.3

On AIX platform we are having Weblogic 8.1 as the middleware for an application.

The application is deployed in Weblogic on AIX platform and the server is having 2 dedicated listening ports for any incoming requests.

Output for netstat -an command for the port 30001 is as follows:-

tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.20501 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.47428 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.48077 ESTABLISHED
tcp 0 0 *.30001 *.* LISTEN
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.52664 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.53170 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55643 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55667 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55668 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55669 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55680 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55691 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55694 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55703 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55719 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55753 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55755 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55758 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55761 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55762 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55781 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.54436 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.54891 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.55333 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.31796 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.31979 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.32428 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59195 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59314 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.10405 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59457 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.12872 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.12943 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.39168 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.42803 ESTABLISHED

With time the no. of established connection get increased and as soon as the count reaches more than 300 the port gets hanged.

The only workaround as of now available to us is to bounce the application server to bring up the ports in listening status.

FYI, the ip's Y.Y.Y.Y and Z.Z.Z.Z can be connected through telnet.

Please suggest how to release these established ports without restarting the application.

Let me know if you require any further information.

Thank U in advance.
# 2  
Old 08-12-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

******************************************************

Are your AIX and WebSphere up-to-date? Do the application logs show something interessting when the problem shows up? Can you and users still open new connections after those 300 are reached via telnet/ssh?
# 3  
Old 08-12-2009
Port hang issue in AIX Version 5.3

Thanks zaxxon for the reply.

Will keep in mind regarding the the format while posting a thread.

--------------------------------------

1. 'AIX and WebSphere up-to-date' by any chance is this concerned with the JDK version?

2. Found the following errors in the application logs while the port is in hanged status:-

Code:
 
Exception in Constructor 
sendData:my Data Length is:194
sendData:sending length of:194
 length of the string 194
Exception in sendData64
java.net.ConnectException: Connection timed out: connect
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(Unknown Source)
 at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
 at java.net.PlainSocketImpl.connect(Unknown Source)
 at java.net.SocksSocketImpl.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.<init>(Unknown Source)
 at java.net.Socket.<init>(Unknown Source)

At this point of time when we are trying to connect to the same port it is throwing an error ' Not able to connect to the port'.

3. Nope, we are not able to create any further connection after the port gets hanged. The output of netstat -an | grep <port no.> shows all the connections in ESTABLISHED status.
# 4  
Old 08-13-2009
Port limit of 300 - Are you sure this isn't a setting in Weblogic, which is perhaps "per applicaiton"?

How to release ports - if they are established, then neither end has closed them. Sounds like the application (client or server end) needs to be modified to close ports when they are no longer needed.
# 5  
Old 08-13-2009
Question

We have confirmed the same from the development team that there is no such setting for the max. no of connections allowed to the ports. Moreover from our analysis point of view the no. 300 may not be hard coded because everytime the no. of established connection varies before the ports get into hanged status, but everytime it is > 300.

Few quick questions for better understanding of the issue:-

1.Is there any other workaround available to analyze from OS point of view before throwing the ball to the application developers court?

2.Please let me know what may be the possible reason or the scenarios where ports remain in ESTABLISHED status.

3. Is there any system timeout variable/parameter from AIX point of view where we can mention the time after which the established connection will get automatically closed.

4. Does established connection for a long time mean an IDLE conection?

5. Is there any possibility that due to some never ending processes these connections are getting created with time?

Being very much new to the platform, asked you all the above questions for better understanding of the issue. Thanks in advance.
# 6  
Old 08-14-2009
From OS view I know no limit how many connections may be opened up to a port. The will be one I think, but we have no problem with applications where more than 300 people connect to a port and there was nothing to be done in the OS to support this.

I did a simple
Code:
no -L| grep 300

to see if there is any network option that might have a value of 300 as default and could be taken into relationship with your problem, but I did not find that value.
So I either still think it is an application set parameter or you might have different values on your OS regarding number of connections. Have no better idea, sorry.
# 7  
Old 08-14-2009
Below is the output of the 'no -a' command. Please suggest if any of the parameters in the list may be the root cause of the issue:-

# no -a
arpqsize = 12
arpt_killc = 20
arptab_bsiz = 7
arptab_nb = 149
bcastping = 0
clean_partial_conns = 0
delayack = 0
delayackports = {}
dgd_packets_lost = 3
dgd_ping_time = 5
dgd_retry_time = 5
directed_broadcast = 0
extendednetstats = 0
fasttimo = 200
icmp6_errmsg_rate = 10
icmpaddressmask = 0
ie5_old_multicast_mapping = 0
ifsize = 256
inet_stack_size = 16
ip6_defttl = 64
ip6_prune = 1
ip6forwarding = 0
ip6srcrouteforward = 1
ip_ifdelete_notify = 1
ip_nfrag = 200
ipforwarding = 0
ipfragttl = 2
ipignoreredirects = 0
ipqmaxlen = 100
ipsendredirects = 1
ipsrcrouteforward = 1
ipsrcrouterecv = 0
ipsrcroutesend = 1
llsleep_timeout = 3
lo_perf = 1
lowthresh = 90
main_if6 = 0
main_site6 = 0
maxnip6q = 20
maxttl = 255
medthresh = 95
mpr_policy = 1
multi_homed = 1
nbc_limit = 6935347
nbc_max_cache = 131072
nbc_min_cache = 1
nbc_ofile_hashsz = 12841
nbc_pseg = 0
nbc_pseg_limit = 8159232
ndd_event_name = {all}
ndd_event_tracing = 0
ndp_mmaxtries = 3
ndp_umaxtries = 3
ndpqsize = 50
ndpt_down = 3
ndpt_keep = 120
ndpt_probe = 5
ndpt_reachable = 30
ndpt_retrans = 1
net_buf_size = {all}
net_buf_type = {all}
net_malloc_police = 0
nonlocsrcroute = 0
nstrpush = 8
passive_dgd = 0
pmtu_default_age = 10
pmtu_expire = 10
pmtu_rediscover_interval = 30
psebufcalls = 20
psecache = 1
pseintrstack = 24576
psetimers = 20
rfc1122addrchk = 0
rfc1323 = 0
rfc2414 = 1
route_expire = 1
routerevalidate = 0
rto_high = 64
rto_length = 13
rto_limit = 7
rto_low = 1
sack = 0
sb_max = 1048576
send_file_duration = 300
site6_index = 0
sockthresh = 85
sodebug = 0
sodebug_env = 0
somaxconn = 1024
strctlsz = 1024
strmsgsz = 0
strthresh = 85
strturncnt = 15
subnetsarelocal = 1
tcp_bad_port_limit = 0
tcp_ecn = 0
tcp_ephemeral_high = 65535
tcp_ephemeral_low = 32768
tcp_finwait2 = 1200
tcp_icmpsecure = 0
tcp_init_window = 0
tcp_inpcb_hashtab_siz = 24499
tcp_keepcnt = 8
tcp_keepidle = 14400
tcp_keepinit = 150
tcp_keepintvl = 150
tcp_limited_transmit = 1
tcp_low_rto = 0
tcp_maxburst = 0
tcp_mssdflt = 1460
tcp_nagle_limit = 65535
tcp_nagleoverride = 0
tcp_ndebug = 100
tcp_newreno = 1
tcp_nodelayack = 0
tcp_pmtu_discover = 1
tcp_recvspace = 16384
tcp_sendspace = 16384
tcp_tcpsecure = 0
tcp_timewait = 1
tcp_ttl = 60
tcprexmtthresh = 3
thewall = 8159232
timer_wheel_tick = 0
udp_bad_port_limit = 0
udp_ephemeral_high = 65535
udp_ephemeral_low = 32768
udp_inpcb_hashtab_siz = 24499
udp_pmtu_discover = 1
udp_recvspace = 42080
udp_sendspace = 9216
udp_ttl = 30
udpcksum = 1
use_isno = 1
use_sndbufpool = 1
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

AIX 5.3 NFS export not mounting, rpcinfo hang/timeout on server

Been trying to get a directory NFS-mounted with no success. I've tried both NFS v3 and v4, but currently trying v4. I can't figure out what's going on here. server: sbkovwadmd01 sbkovwadmd01# chnfsdom Current local domain: edw.dev sbkovwadmd01# lssrc -a | grep nfs | grep active nfsd ... (3 Replies)
Discussion started by: eckertd
3 Replies

2. Shell Programming and Scripting

How to solve hang issue in script?

i have one function block in the beginning of my script and there are some commands inside that function which will perform some operations. And i am invoking that function from my main script by passing some values. Sometimes it is hanging in the middle for some value. For example: For 1st... (3 Replies)
Discussion started by: thomasraj87
3 Replies

3. Programming

A system hang issue in porting an old fs to kernel 2.6.32.*

Hi, I ported a file system from Kernel version 2.6.18.* to 2.6.32.46 as VFS implementation has changed and prepare_write/commit_write are replaced by write_begin/write_end pairs. Implemented the feature by writing a wrapper for write_begin and write_end and calling original functions for... (0 Replies)
Discussion started by: Praveen_218
0 Replies

4. AIX

Need Help Urgent AIX Hang

hello all, I'm newbie on AIX. Can any one tell why this happen ? I have expectation this error came because paging memory. This is my error : ( I cann;t read this log and need your help) --------------------------------------------------------------------------- LABEL: ... (7 Replies)
Discussion started by: Fandikurnia
7 Replies

5. AIX

DNS is hang in AIX 6.1

Guy's I have configured DNS like the below but I faced another issue ! nameserver 172.18.14.7 nameserver 172.18.14.8 domain domain.com When I ping any Host from the server itself its replaying me the IP of that hosts that's mean DNS is working fain but when I try to ftp or telnet... (8 Replies)
Discussion started by: Mr.AIX
8 Replies

6. Solaris

Zone hang issue

I have a solaris10 zone running an oracle db in it. The zone hangs after some time the db is started and i 'm unable to log into the zone after that so i have to halt and restart the zone along with db .. but after few hrs again it hangs. What would be the reason for it. I did svcs -xv and got... (13 Replies)
Discussion started by: fugitive
13 Replies

7. Solaris

Solaris 10 CDE hang issue

Hi, I am facing some issues on Solaris 10 (SPARC 5.10 Generic_118833-36). The issue is when I try to connect/login through CDE it hangs (hangs at login screen). I tried connecting thourgh VNC client also but that also not helped. I had done no configuration change or modified any kernel... (2 Replies)
Discussion started by: kgupta
2 Replies

8. UNIX for Advanced & Expert Users

removing hang processes (AIX)

Hi Guys, Just wondering if I have a child process which is basically hanging and I can see that is on sleep or wait mode if I want to remove/terminate this process , the signal has to come from its parent I assume. The reason I'm asking this is because I'm facing a db2 hang situation and we... (3 Replies)
Discussion started by: hariza
3 Replies

9. AIX

how to up the aix machine when the desktop was hang

Hi all, My AIX machine desktop was hanged. What to do. please help me. It's very urgent. Thank's & regards, Balaraju. (7 Replies)
Discussion started by: balarajum
7 Replies

10. AIX

AIX 4.1.5/RS6000 boot hang, help requested

Hello there! I have a RS-6000 7043-140 machine with AIX version 4.1.5, that is working for almost 8 years now. It has a tty monitor. My problem started when I upgraded my machine to install a gxt250 graphics adapter card together with a 15" AOC VGA Monitor, logitech keyboard and mouse, ... (2 Replies)
Discussion started by: bright_genius
2 Replies
Login or Register to Ask a Question