CPU Panic.
Hi Guru's,
My Sun Fire v490 PRODUCTION server(Sol 5.9) has rebooted twice - on Apr'30 & May'08 2009.
And i found the following Error messages in /var/adm/messages file:
Apr 30 11:36:42 mumux201 unix: [ID 836849 kern.notice]
Apr 30 11:36:42 mumux201 ^Mpanic[cpu16]/thread=2a1000c5d20:
Apr 30 11:36:42 mumux201 unix: [ID 340138 kern.notice] BAD TRAP: type=31 rp=2a1000c4fd0 addr=48 mmu_fsr=0 occurred in module "emlxs" due to a NULL pointer dereference
Apr 30 11:36:42 mumux201 unix: [ID 100000 kern.notice]
Apr 30 11:36:42 mumux201 unix: [ID 839527 kern.notice] sched:
Apr 30 11:36:42 mumux201 unix: [ID 520581 kern.notice] trap type = 0x31
Apr 30 11:36:42 mumux201 unix: [ID 381800 kern.notice] addr=0x48
Apr 30 11:36:42 mumux201 unix: [ID 101969 kern.notice] pid=0, pc=0x13c9ed8, sp=0x2a1000c4871, tstate=0x4400001601, context=0x0
Apr 30 11:36:42 mumux201 unix: [ID 743441 kern.notice] g1-g7: 1, c9687f, 3ffffffffdfa98ef, 300bcb3cf88, 410, edffacb92b2dcba, 2a1000c5d20
Apr 30 11:36:42 mumux201 unix: [ID 100000 kern.notice]
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c4cf0 unix:die+80 (31, 2a1000c4fd0, 48, 0, 2a1000d1d20, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000001413878 000002a1000c4fd0 000002a1000c4ec0
Apr 30 11:36:42 mumux201 %l4-7: 0000000000000031 00000300003d777a 00000300003d7778 00000300003d7770
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c4dd0 unix:trap+8e4 (2a1000c4fd0, 0, 10000, 10200, 0, 80)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000001 0000000000000000 00000000014527d8 0000000000000031
Apr 30 11:36:42 mumux201 %l4-7: 0000000000000005 0000000000000001 0000000000000000 0000000000000000
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c4f20 unix:ktl0+48 (1157, 9, ffbfa371, 0, 0, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000002 0000000000001400 0000004400001601 000000000102db30
Apr 30 11:36:42 mumux201 %l4-7: 0000000000000000 00000000ffbf9d53 0000000000000000 000002a1000c4fd0
Apr 30 11:36:42 mumux201 genunix: [ID 562518 kern.notice] 000002a1000c5070 7f52b268 (300007be028, 0, 20, 2, 1, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000030004bdafe0 0000000000000000 0000030004bdaf20
Apr 30 11:36:42 mumux201 %l4-7: 000000000000ff00 000000007b9afd8e 000000007ee3f82d 000000007f5eba54
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c5170 emlxs:emlxs_issue_fcp_iocb_cmd+2e4 (300007be028, 0, 20, 3000517a0d0, 0, ff00)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 00000300007be028 0000000000000038 0000000000000001
Apr 30 11:36:42 mumux201 %l4-7: 0000000000000001 0000000081010000 0000000000000000 000000007f5eba54
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c5290 emlxs:emlxs_issue_iocb_cmd+30 (300007be028, 300007be5a0, 0, 1, 8, 8)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 00000300007be028 0000000000000000 ffffffffffffffff
Apr 30 11:36:42 mumux201 %l4-7: 000003000517a090 000000000143c6f8 0000000000000000 000000000128c65c
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c53b0 emlxs:emlxs_timer+15ac (300007be000, 2a1000c5d20, 20, 1, 16, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 00000300007be5a0 00000300007be028 0000000000000000 00000300007be5a0
Apr 30 11:36:42 mumux201 %l4-7: 0000000000000004 00000000001f0000 0000000000000002 00000300003e9670
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c5990 genunix:callout_execute+90 (30000334038, 3f8, bffffffffdfa98ef, 1, 0, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 000000000138ea58 0000030033f9e2f8 0000000000c9687f 0000000000c9687f
Apr 30 11:36:42 mumux201 %l4-7: 0000030000334000 8000000000000000 0000000000000000 0000030000335038
Apr 30 11:36:42 mumux201 genunix: [ID 723222 kern.notice] 000002a1000c5a40 genunix:taskq_thread+188 (300003d77a0, 0, 14527d8, 14527d8, 2a1000d1d20, 0)
Apr 30 11:36:42 mumux201 genunix: [ID 179002 kern.notice] %l0-3: 00000000010805b8 00000300003d7768 00000300003d3888 0000000000010000
Apr 30 11:36:42 mumux201 %l4-7: 00000300003d7748 00000300003d777a 00000300003d7778 00000300003d7770
Apr 30 11:36:42 mumux201 unix: [ID 100000 kern.notice]
Apr 30 11:36:42 mumux201 genunix: [ID 672855 kern.notice] syncing file systems...
Apr 30 11:36:45 mumux201 genunix: [ID 733762 kern.notice] 140
Apr 30 11:36:48 mumux201 genunix: [ID 733762 kern.notice] 90
Apr 30 11:36:49 mumux201 genunix: [ID 733762 kern.notice] 74
Apr 30 11:37:23 mumux201 last message repeated 20 times
Apr 30 11:37:24 mumux201 genunix: [ID 622722 kern.notice] done (not all i/o completed)
Apr 30 11:37:25 mumux201 genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c1t0d0s1, offset 65536, content: kernel
Apr 30 11:38:16 mumux201 genunix: [ID 409368 kern.notice] ^M100% done: 202861 pages dumped, compression ratio 2.74,
Apr 30 11:38:16 mumux201 genunix: [ID 851671 kern.notice] dump succeeded
Apr 30 11:39:06 mumux201 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.9 Version Generic_122300-29 64-bit
Apr 30 11:39:06 mumux201 genunix: [ID 943905 kern.notice] Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Apr 30 11:39:06 mumux201 Use is subject to license terms.
These error messages were found during both the reboots.
We have a crash dump on Apr'30 in /var/crash/<hostname>.
What could be the cause for this? Awaiting your valuable inputs.
Note - The vendor has proposed a OBP upgrade and a SAN patch upgrade.
HG
