Solaris

Hello Gurus,

I got a system reboot due to "unix: [ID 799565 kern.notice] BAD TRAP: type=30 rp=2a10001b840 addr=2a000012040 mmu_fsr=0" on SUNW,Netra-CP2300. I guess it is caused by HW fault but not very sure, and don't know which part should be relaced incase it is HW reason really. Sincerely appreciate for your advice, or tell me I'm completely wrong.

SolarisCAT(vmcore.2)> coreinfo

core file: /gsn/bmcxuxi/vmcore.2
user: Super-User (root:0)
release: 5.8 (64-bit)
version: Generic_117350-02
machine: sun4u
node name: eqm02s13p2
hw_provider: Sun_Microsystems
system type: SUNW,Netra-CP2300
hostid: 83cf17e0
time of crash: Mon Dec 26 23:22:14 CST 2005 (core is 13 days old)
age of system: 12 days 9 hours 41 minutes 47.49 seconds
panic cpu: 0 (ncpus: 1)
panic string: BAD TRAP: type=30 rp=2a10001b840 addr=2a000296fc0 mmu_fsr=0

running sanity checks.../etc/system...
WARNING: sq_max_size set to 0
ndd...sysent...misc...done
SolarisCAT(vmcore.2)> analyze
PANIC: BAD TRAP: type=%x rp=%p addr=%p mmu_fsr=%x


core file: /gsn/bmcxuxi/vmcore.2
user: Super-User (root:0)
release: 5.8 (64-bit)
version: Generic_117350-02
machine: sun4u
node name: eqm02s13p2
hw_provider: Sun_Microsystems
system type: SUNW,Netra-CP2300
hostid: 83cf17e0
time of crash: Mon Dec 26 23:22:14 CST 2005 (core is 13 days old)
age of system: 12 days 9 hours 41 minutes 47.49 seconds
panic cpu: 0 (ncpus: 1)
panic string: BAD TRAP: type=30 rp=2a10001b840 addr=2a000296fc0 mmu_fsr=0


1 cpu

==== checking for trap information ====

cpu 0 had the panic


==== panic thread: 0x2a10001bd20 ==== cpu: 0 ====
==== panic kernel thread: 0x2a10001bd20 pid: 0 on cpu: 0 ====
cmd: sched

t_stk: 0x2a10001bb10 sp: 0x10423f91 t_stkbase: 0x2a10001a000
t_pri: 60(SYS) pctcpu: 0.000000 t_lwp: 0x0
t_procp: 0x10424d30(proc_sched) p_as: 0x10424c40(kas)
last cpuid: 0
idle: 33701 ticks (5 minutes 37.01 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg: T_TALLOCSTK - thread structure allocated from stk
T_PANIC - thread initiated a system panic
tpflg: none set
tsched: TS_LOAD - thread is in memory
TS_DONT_SWAP - thread/LWP should not be swapped
pflag: SSYS - system resident process
SLOAD - in core
SLOCK - process cannot be swapped
SULOAD - u-block in core

pc: 0x10048738 unix:panicsys+0x44: call unix:setjmp
startpc: 0x10025258 unix:memscrubber+0x0: save %sp, -0xf0, %sp

unix:panicsys+0x44 (0x10424960, 0x2a10001b5f8, 0x10055dc8, 0x78002000, 0x0, 0x0)
unix:vpanic+0xcc (0x10055dc8, 0x2a10001b5f8, 0x23, 0x0, 0x8, 0x8)
unix:panic+0x1c (0x10055dc8, 0x30, 0x2a10001b840, 0x2a000296fc0, 0x0, 0x0)
unix:die+0xa4 (0x30, 0x2a10001b840, 0x2a000296fc0, 0x0, 0x2a10001b840, 0xe39a1e00)
unix:trap+0x8b8 (0x2a000296000, 0x1, 0x5, 0x0, 0x2a10001b840, 0x0)
unix:prom_rtt+0x0 (0x2a000296fc0, 0x1691, 0x2, 0x21, 0x80000, 0x1)
-- trap data type: 0x30 (data access exception) rp: 0x2a10001b840 LEAF --
addr: 0x2a000296fc0
pc: 0x1002604c unix:memscrub_read+0x24: ldda [%o0] ASI_BLK_P, %f48
npc: 0x10026050 unix:memscrub_read+0x28: sub %o1, 0x1, %o1 ( dec %o1 )
global: %g1 0x1041b800
%g2 0x1041c000 %g3 0x1041ba38
%g4 0x3 %g5 0x2
%g6 0 %g7 0x2a10001bd20
out: %o0 0x2a000296fc0 %o1 0x1691
%o2 0x2 %o3 0x21
%o4 0x80000 %o5 0x1
%sp 0x2a10001b0e1 %o7 0x10025cc4
loc: %l0 0x2a000000000 %l1 0x16
%l2 0x6 %l3 0x16
%l4 0x3000078f800 %l5 0
%l6 0 %l7 0x2a10001b980
in: %i0 0x8000 %i1 0x93800000
%i2 0x10417408 %i3 0x10414a30
%i4 0x1000c364 %i5 0
%fp 0x2a10001b221 %i7 0x10025588
<leaf trap>unix:memscrub_read+0x24 (%o:0x2a000296fc0, 0x1691, 0x2, 0x21, 0x80000, 0x1)
unix:memscrub_scan+0x158 (0x8000, 0x93800000, 0x10417408, 0x10414a30, 0x1000c364, 0x0)
unix:memscrubber+0x330 (0x0, 0x1040d994, 0x1041740c, 0x151, 0x10417400, 0x1040da98)
unix:thread_start+0x4 (0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
-- end of kernel thread's stack --


==== analyzing panic thread stack for trap frames ====


==== using trap() frame 1 @ 0x2a10001b650, rp(%i4): 0x2a10001b840 ====

type(%l4): 0x30 (data access exception)

pc: 0x1002604c unix:memscrub_read+0x24: ldda [%o0] ASI_BLK_P, %f48
npc: 0x10026050 unix:memscrub_read+0x28: sub %o1, 0x1, %o1 ( dec %o1 )

global: %g1 unix(data):pcf+0x320
%g2 unix(data):pio_mutex+0x138 %g3 unix(data):cpu+0x0
%g4 0x3 %g5 0x2
%g6 0 %g7 0x2a10001bd20
out: %o0 0x2a000296fc0 %o1 0x1691
%o2 0x2 %o3 0x21
%o4 0x80000 %o5 0x1
%sp 0x2a10001b0e1 %o7 unix:memscrub_scan+0x158
loc: %l0 0x2a000000000 %l1 0x16
%l2 0x6 %l3 0x16
%l4 0x3000078f800 %l5 0
%l6 0 %l7 0x2a10001b980
in: %i0 0x8000 %i1 0x93800000
%i2 unix(data):memscrub_cv+0x0 %i3 unix(data):_DYNAMIC+0x9c0
%i4 unix:bzero+0x298 %i5 0
%fp 0x2a10001b221 %i7 unix:memscrubber+0x330

unix:memscrub_uninit_mem_config+0x14: or %g0, %g1, %o7 ( mov %g1, %o7 )
unix:memscrub_read+0x0: srl %o1, 0x0, %o1
unix:memscrub_read+0x4: rd %fprs, %o2 ( mov %fprs, %o2 )
unix:memscrub_read+0x8: wr 0x4, %fprs ( mov 0x4, %fprs )

unix:memscrub_read+0xc: 1: ldda [%o0] ASI_BLK_P, %f0
unix:memscrub_read+0x10: add %o0, 0x40, %o0 ( inc 0x40, %o0 )
unix:memscrub_read+0x14: ldda [%o0] ASI_BLK_P, %f16
unix:memscrub_read+0x18: add %o0, 0x40, %o0 ( inc 0x40, %o0 )
unix:memscrub_read+0x1c: ldda [%o0] ASI_BLK_P, %f32
unix:memscrub_read+0x20: add %o0, 0x40, %o0 ( inc 0x40, %o0 )
unix:memscrub_read+0x24: ldda [%o0] ASI_BLK_P, %f48
unix:memscrub_read+0x28: sub %o1, 0x1, %o1 ( dec %o1 )
unix:memscrub_read+0x2c: brnz,pt %o1, unix:memscrub_read+0xc (1b)
unix:memscrub_read+0x30: add %o0, 0x40, %o0 ( inc 0x40, %o0 )

unix:memscrub_read+0x34: jmpl [%o7 + 0x8], %g0 ( retl )
unix:memscrub_read+0x38: wr %o2, %fprs ( mov %o2, %fprs )
unix:mlsetup+0x0: save %sp, -0xc0, %sp
unix:mlsetup+0x4: sethi %hi(0x10408000) %g3 ( set 0x10408000, %g3 )
unix:mlsetup+0x8: sethi %hi(0x1041c000) %g5 ( set 0x1041c000, %g5 )
unix:mlsetup+0xc: add %g3, 0x30, %i3
unix:mlsetup+0x10: add %g5, 0x380, %g5 ( inc 0x380, %g5 )
SolarisCAT(vmcore.2)>

Thanks in advance
Fangfang

"Bad trap" usually means that kernel tried something impossible. Usually it is dereferencing a bad pointer. It could be hardware, but my first guess would be a kernel bug, most likely in some driver. However, that "cmd: sched" makes me suspect the scheduler in this case. Really, I would call place a service call with Sun for something like this. If that is not possible, I would make sure that I have the last patch cluster for the os. Did you recently add some hardware to your box? If so, check for patches to the driver it uses. Also there is prtdiag -v which might yield some clues.

I agree with Perderabo that this probably isn't hardware. I've seen many hardware panics caused by bad CPU or memory in a Sun box, and the messages you got are quite different. I agree talking to Sun support is the best choice here. You could also go on their website and search the knowledge base for some of the error strings you got and see if that turns up anything.

Thanks alot for your comments. I will come to Sun for further analysis.

The reason that I suspect HW is, there are 18 Solaris nodes in my system, they are exactly same in plateform/HW/OS/App(auto installed by DHCP script), only this node corrupted 4 times in 20 days with same reason(a little differ of addr). I thought high work load might crash the scheduler but it seems not true because the 18 nodes are configured in load balance...

Anyway, see what answer from Sun. Thanks again.

Hola,

Can you please update your reply received from SUN about the panic error in SUN...since I also getting the same error

Thanks in advance!!!

Regards,
Jagdish Machhi

Hi Jagdish,

No exact explanation for this case from SUN, however we upgraded system to 5.9 OS and then no more this kind of error. Hope helpful for u.

good luck
Fangfang

There should be coredump analysis and RCA provided from SUN right?