Programming

Hi All,
My application gets hang at the popen statement in Soalris 5.8. While debugging I found that it halts at _lwp_sema_wait. After googling I have added /usr/lib/lwp in the LD_LIBRARY_PATH and now it halts at lwp_park. Please help me what I need to do to solve this issue. The application runs fine in AIX and other solaris box.
The dbx, pstack and truss output are as below.
Dbx
====================================================================================
81 fp=popen(command,”r”);
(dbx 1) where
current thread: t@1
=>[1] _lwp_sema_wait(0xb51b8, 0x0, 0x0, 0x0, 0x0, 0x2), at 0xfeb9f8bc
[2] _park(0xb51b8, 0xff0bc000, 0x0, 0xb50f0, 0x23154, 0x0), at 0xff0996f4
[3] _swtch(0xb50f0, 0xb50f0, 0xff0bc000, 0x5, 0x1000, 0x0), at 0xff0993bc
[4] _mutex_adaptive_lock(0xff0c7864, 0x4c00, 0x1000, 0xfffeffff, 0x1, 0x4d58), at 0xff09adc8
[5] _cmutex_lock(0xfebc0608, 0xff0bc000, 0xff0bc000, 0xfeb41da0, 0xfebc0438, 0x0), at 0xff09aadc
[6] malloc(0xc, 0x27cac, 0xb4040, 0xfebc0430, 0xb4050, 0x10), at 0xfeb41da0
[7] _insert_nolock(0x0, 0x6, 0xfebbc008, 0x2711, 0x40, 0x40), at 0xfeb94394
[8] popen(0xb4040, 0x6, 0x2711, 0x5, 0xffbe8744, 0x38), at 0xfeb9423c
[9] zcxeval(0xffbe8844, 0xffbe8614, 0xffbe872c, 0xfefeec9c, 0x0, 0x2), at 0xfee503f8
[10] openfile(0xffbe89d8, 0x1, 0xffbe8ac4, 0x0, 0xb, 0x0), at 0xfedb1508
[11] ypmbacktrace(0xb, 0xfecaa358, 0x0, 0x0, 0x0, 0x0), at 0xfedb04a4
[12] error_handler(0xb, 0xffbe9020, 0xffbe8d68, 0x0, 0x0, 0x0), at 0xfecaa570
[13] __sighndlr(0xb, 0xffbe9020, 0xffbe8d68, 0xfecaa358, 0xb5198, 0xb5188), at 0xff0ab218
---- called from signal handler with signal 11 (SIGSEGV) ------
[14] _free_unlocked(), at 0xfeb42c8c
[15] free(0xc300000, 0x0, 0x0, 0xfeb91fe4, 0x36, 0x0), at 0xfeb42c3c
[16] SPIBase::~SPIBase(0xffbea79c, 0x0, 0xfebbfc78, 0xfebbfc78, 0x0, 0x0), at 0xff2d5308
[17] SPIFiles::~SPIFiles(0xffbea79c, 0x2, 0x1, 0x1, 0x7c53c, 0x40000000), at 0xff2da8ec
[18] Load_Band_Struct::~Load_Band_Struct(0xffbea79c, 0x2, 0x7d074, 0x0, 0x0, 0xffbe9f9a), at 0x362a0
[19] main_code(0x0, 0x0, 0xffbe9fde, 0x7c6b8, 0xffbed8c1, 0x7c6bc), at 0x346e8
[20] ycx_init_term(0x1, 0x0, 0x2ddd0, 0x3, 0xffbeec2c, 0x0), at 0xfeca9b58
[21] ycxonxp(0x2ddd0, 0x3, 0xffbeec2c, 0x67adc, 0x0, 0x0), at 0x36a10
[22] main(0x3, 0xffbeec2c, 0xffbeec3c, 0x7c400, 0x0, 0x0), at 0x2dd8c

Pstack output
====================================================================================
----------------- lwp# 1 / thread# 1 --------------------
feb9f8bc lwp_sema_wait (b51b8)
ff0996f4 _park (b51b8, ff0bc000, 0, b50f0, 23154, 0) + 114
ff0993bc _swtch (b50f0, b50f0, ff0bc000, 5, 1000, 0) + 424
ff09adc8 _mutex_adaptive_lock (ff0c7864, 4c00, 1000, fffeffff, 1, 4d58) + 160
ff09aadc _cmutex_lock (febc0608, ff0bc000, ff0bc000, feb41da0, febc0438, 0) + 84
feb41da0 malloc (c, 27cac, b4040, febc0430, b4050, 10) + 18
feb94394 _insert_nolock (0, 6, febbc008, 2711, 40, 40) + 3c
feb9423c popen (b4040, 6, 2711, 5, ffbe8744, 38) + 264
fee503f8 __0FHzcxevalR6HYsgstrdRC6HYsgstrd (ffbe8844, ffbe8614, ffbe872c, fefeec9c, 0, 2) + 90
fedb1508 __0FIopenfileRP6EFILEUc (ffbe89d8, 1, ffbe8ac4, 0, b, 0) + 258
fedb04a4 __0FMypmbacktracev (b, fecaa358, 0, 0, 0, 0) + 3c
fecaa570 __0FNerror_handleri (b, ffbe9020, ffbe8d68, 0, 0, 0) + 218
ff0ab218 __sighndlr (b, ffbe9020, ffbe8d68, fecaa358, b5198, b5188) + c
ff0a821c sigacthandler (b, b50f0, 0, 0, 0, ff0bc000) + 708
--- called from signal handler with signal 11 (SIGSEGV) ---
feb42c8c _free_unlocked (c300000, febbc008, c300000, febbc008, 148ea0, 0) + 40
feb42c3c free (c300000, 0, 0, feb91fe4, 36, 0) + 20
ff2d5308 __0oHSPIBasedtv (ffbea79c, 0, febbfc78, febbfc78, 0, 0) + b0
ff2da8ec __0oISPIFilesdtv (ffbea79c, 2, 1, 1, 7c53c, 40000000) + 84
000362a0 __0oQLoad_Band_Structdtv (ffbea79c, 2, 7d074, 0, 0, ffbe9f9a) + 50
000346e8 __0FJmain_codeRC6HYsgstrd (0, 0, ffbe9fde, 7c6b8, ffbed8c1, 7c6bc) + 6918
feca9b58 __0FNycx_init_termUcPFv_iPFRC6HYsgstrd_iiPPc (1, 0, 2ddd0, 3, ffbeec2c, 0) + 4d8
00036a10 __0FHycxonxpPFRC6HYsgstrd_iiPPc (2ddd0, 3, ffbeec2c, 67adc, 0, 0) + 88
0002dd8c main (3, ffbeec2c, ffbeec3c, 7c400, 0, 0) + 7c
0002dbc4 _start (0, 0, 0, 0, 0, 0) + dc
----------------- lwp# 2 / thread# 2 --------------------
feb9f20c signotifywait ()
ff09e980 _dynamiclwps (ff0bc000, ff3de0c4, ff3813f8, ff381b78, 2f, 7efefeff) + 24
ff0a1c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 3 --------------------------------
feb9cdb0 door (0, 0, 0, 0, ff045d38, 4)
ff0a687c _sc_door_func (4, ff0bc000, ff0bd678, 3, ff0bc000, 1) + 6c
ff09a380 _lwp_start (ff045d98, 0, 6000, ffbee534, 0, 0) + 18
ff0a1c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 4 --------------------------------
ff0a92fc lwp_cond_wait (ff0c34f0, ff0c3500, ff0bcd88)
ff098cdc _age (3e, ff0bcd74, ff0bc000, 3, ff0bc000, 1) + 74
ff09a380 _lwp_start (fe60bd98, 0, 0, 0, 0, 0) + 18
ff0a1c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 5 / thread# 4 --------------------
feb9f8bc lwp_sema_wait (ff0c30a8)
ff0a6b74 _tdb_agent (ff0c34d0, ff0c2e48, 0, 1, 0, fe60bd98) + 50
ff0a1c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
-------------------------- thread# 3 --------------------
ff09d9e0 _reap_wait (ff0c0988, 1e8fc, 0, ff0bc000, 0, 0) + 38
ff09d738 _reaper (ff0bce08, ff0c2710, ff0c0988, ff0bcde0, 1, fe400000) + 38
ff0ab11c _thread_start (0, 0, 0, 0, 0, 0) + 40
truss -p 10000
====================================================================================
lwp_sema_wait(0x000B51B8) (sleeping...)
signotifywait() (sleeping...)
door_return(0x00000000, 0, 0x00000000, 0) (sleeping...)
lwp_cond_wait(0xFF0C34F0, 0xFF0C3500, 0xFF0BCD88) (sleeping...)
lwp_sema_wait(0xFF0C30A8) (sleeping...)

Thanks in advance for help and suggestions.
Reagrds
Amy

Your signal handler is not async-signal safe, so you've deadlocked.

You can't do things like malloc() or free() calls in a signal handler. Your application deadlocks because it get a SEGV when calling free(). The free() call will lock a mutex. Your SEGV happens after the mutex is locked. Your signal handler then gets called, and it looks like your handler makes a popen() call. The popen() call calls malloc(), which blocks waiting for the mutex that the free() call locked.

If you look at the man page for popen(), you'll see that it's MT-safe but not async-signal safe.

Hi achenle,

Thanks for the reply.

Can you please let me know in details what I need to do to overcome this deadlock? Do you mean to say there is some issue with the application code. The application is running fine with same binaries in other sun boxes.

Regards,
Amy

From "man attributes":

Your code is doing a popen() in your SEGV signal handler. From "man popen":



So, popen() is not async-signal safe. Hence your deadlock.

The fact that it works on other servers does not make it correct.

Of cource, fixing this deadlock isn't going to fix your code. You still have the SEGV in free() to deal with. That's almost certainly caused by application code corrupting the memory heap by overwriting dynamically-allocated memory or passing an improper pointer to a dynamic memory routine such as free().

I found the issue is not with popen. I commented the line and still it halts.

Regard
Amy

Where is it now halting?

Can you show us your source code?

...yet. It is not async-signal safe so could cause unpredictable problems in the future, especially for portability.