Solaris

hi experts,

i'm having more error log on /var/adm/messages could anyone help to description what is this error logs means???? and how to fix it? below is the logs :

Nov 22 20:03:40 USSDGtbs-1 unix: [ID 836849 kern.notice]
Nov 22 20:03:40 USSDGtbs-1 ^Mpanic[cpu6]/thread=3001345edc0:
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 340138 kern.notice] BAD TRAP: type=31 rp=2a100e84e80 addr=28 mmu_fsr=0 occurred in module "tcap" due to a
NULL pointer dereference
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 100000 kern.notice]
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 839527 kern.notice] dsmd:
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 520581 kern.notice] trap type = 0x31
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 381800 kern.notice] addr=0x28
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 101969 kern.notice] pid=11540, pc=0x78548850, sp=0x2a100e84721, tstate=0x4480001606, context=0x37e
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 743441 kern.notice] g1-g7: 0, 100, ff00, 78552008, 0, 0, 3001345edc0
Nov 22 20:03:40 USSDGtbs-1 unix: [ID 100000 kern.notice]
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e84a90 unix:die+80 (31, 28, 10415270, 0, 2a100e84e80, c4022028)
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000004 0000000000000001 0000000078334c26 0000000000000002
Nov 22 20:03:40 USSDGtbs-1 %l4-7: 0000000000000000 0000030012def668 0000000078334c38 0000030013c76cc8
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e84b70 unix:trap+8b8 (0, 1, 5, 0, 2a100e84e80, 0)
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000001 0000000000000016 000003000f81b520 0000000000000000
Nov 22 20:03:40 USSDGtbs-1 %l4-7: 0000000000000031 00000300137ed8a8 0000000000010000 000002a100e85320
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e84cb0 unix:sfmmu_tsb_miss+66c (104293b0, 0, 30000221f88, 0, 30000221f88,
19)
Nov 22 20:03:40 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000004 0000000000010000 000003100043f100
Nov 22 20:03:40 USSDGtbs-1 %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000003
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e84dd0 unixrom_rtt+0 (0, a200d1f, 0, 2a100e84ffc, 2a100e85000, 20)
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000007 0000000000001400 0000004480001606 000000001001a8bc
Nov 22 20:03:41 USSDGtbs-1 %l4-7: 0000000000000000 000003001663b980 0000000000000000 000002a100e84e80
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e84f20 tcap:tcap_c_process_msg_rcvd+1798 (78553750, 78552008, 78554670, f
, 30014828900, 30013937cd0)
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000020 0000000000000065 ffffffffffffffff 0000000000000004
Nov 22 20:03:41 USSDGtbs-1 %l4-7: 00000000785546a4 000003001663d0f8 0000000078554550 0000000000000003
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85010 tcap:tcaplrput+520 (3001663d0f8, 30014828900, 30014828900, 3001393
7cd0, 78554550, d7)
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000078546b80 00000300135fc438 00000300135fc0f8 0000000000ff0000
Nov 22 20:03:41 USSDGtbs-1 %l4-7: 0000000010288968 0000000000000000 0000000000000000 0000000000000000
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e850d0 unixutnext+218 (0, 30012a92ba8, 20, 30014828900, 0, 0)
Nov 22 20:03:41 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000030013937cd0 0000030013937e90 0000030012def668 0000000000000016
Nov 22 20:03:41 USSDGtbs-1 %l4-7: 00000000785311e0 0000000000000000 0000000000000000 000002a100e85110
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85180 sccp:kdb_putnext+74 (78520da8, 30014828900, 78520da8, 2f1, 30012de
f668, 0)
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000004480001602 000003000f5e5218 000000000000000e 0000000000000016
Nov 22 20:03:42 USSDGtbs-1 %l4-7: 000000007810e470 0000000000000000 0000000000000000 000002a100e851c0
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85230 sccp:sccpursrv+350 (30014828900, 3001345edc0, 78520b78, f000, 7833
4bbc, 7835eba0)
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000001 0000000078334c26 0000000000000002
Nov 22 20:03:42 USSDGtbs-1 %l4-7: 0000000000000000 0000030012def668 0000000078334c38 0000030013c76cc8
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85310 genunix:runservice+3c (2200, 2000, 20000, 0, 30012def6d0, 30012def
668)
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 00000000784c1560 0000000000000016 000000000000000e 0000000000000016
Nov 22 20:03:42 USSDGtbs-1 %l4-7: 0000000000000000 0000000000000000 0000000000000000 000002a100e85320
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e853c0 genunix:queuerun+160 (10431fc4, 3001345edc0, 30012def668, 10477898
, 104320d8, 10475390)
Nov 22 20:03:42 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000030012def6d0 000003000f5fc648 0000000000010000 0000000000000000
Nov 22 20:03:42 USSDGtbs-1 %l4-7: 00000000782403e0 0000000000000000 0000000000000000 0000000000000000
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85470 genunix:strdoioctl+3a4 (1, 3001274c9f2, 1, 30012eff570, 3001274c9f
0, 0)
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000c50 000003001274c968 000002a100e858b8 0000000000000c50
Nov 22 20:03:43 USSDGtbs-1 %l4-7: 000003001274c9e8 000003001274c968 000002a100e85aec 00000300137f9580
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e85570 genunix:strioctl+1980 (3001274c9e8, 0, 2a100e85aec, 30012eff570, 1
, 300137f9660)
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 00000000ffbea598 0000030013457778 0000000000005308 0000000000100083
Nov 22 20:03:43 USSDGtbs-1 %l4-7: 000003001274c9e8 000003001274c968 000002a100e85aec 00000300137f9580
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e858e0 specfs:spec_ioctl+8c (30013457778, 5308, ffbea598, 100083, 30012ef
f570, 2a100e85aec)
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 000002a100e85ba0 00000300137ed8a8 0000000000000000
Nov 22 20:03:43 USSDGtbs-1 %l4-7: 0000000000000000 000003000f81b520 0000000000000000 0000000000000000
Nov 22 20:03:43 USSDGtbs-1 genunix: [ID 723222 kern.notice] 000002a100e859a0 genunix:ioctl+1e4 (30012fbe628, 4, ffbea598, 0, 5308, 3ce68)
Nov 22 20:03:44 USSDGtbs-1 genunix: [ID 179002 kern.notice] %l0-3: 000000001015e600 0000000000004000 0000000000000000 0000000000000003
Nov 22 20:03:44 USSDGtbs-1 %l4-7: 0000000000000000 0000000000000000 000003001345edc0 000002a100e85ba0
Nov 22 20:03:44 USSDGtbs-1 unix: [ID 100000 kern.notice]
Nov 22 20:03:44 USSDGtbs-1 genunix: [ID 672855 kern.notice] syncing file systems...
Nov 22 20:03:45 USSDGtbs-1 md_stripe: [ID 641072 kern.warning] WARNING: md: d61: write error on /dev/dsk/c1t0d0s7
Nov 22 20:03:46 USSDGtbs-1 md_stripe: [ID 641072 kern.warning] WARNING: md: d62: write error on /dev/dsk/c1t1d0s7
Nov 22 20:03:49 USSDGtbs-1 genunix: [ID 733762 kern.notice] 19
Nov 22 20:03:53 USSDGtbs-1 genunix: [ID 733762 kern.notice] 16
Nov 22 20:03:57 USSDGtbs-1 genunix: [ID 733762 kern.notice] 9
Nov 22 20:05:13 USSDGtbs-1 last message repeated 20 times
Nov 22 20:05:14 USSDGtbs-1 genunix: [ID 622722 kern.notice] done (not all i/o completed)
Nov 22 20:05:16 USSDGtbs-1 genunix: [ID 353387 kern.notice] dumping to /dev/md/dsk/d20, offset 1719336960
Nov 22 20:08:26 USSDGtbs-1 genunix: [ID 409368 kern.notice] ^M100% done: 331504 pages dumped, compression ratio 3.90,
Nov 22 20:08:26 USSDGtbs-1 genunix: [ID 851671 kern.notice] dump succeeded
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.8 Version Generic_117350-25 64-bit
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 913632 kern.notice] Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 678236 kern.info] Ethernet address = 0:3:ba:b0:28:37
Nov 22 20:10:08 USSDGtbs-1 unix: [ID 389951 kern.info] mem = 67108864K (0x1000000000)
Nov 22 20:10:08 USSDGtbs-1 unix: [ID 930857 kern.info] avail mem = 66384666624
Nov 22 20:10:08 USSDGtbs-1 rootnex: [ID 466748 kern.info] root nexus = Sun Fire V890
Nov 22 20:10:08 USSDGtbs-1 rootnex: [ID 349649 kern.info] pcisch1 at root: SAFARI 0x8 0x600000
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 936769 kern.info] pcisch1 is /pci@8,600000
Nov 22 20:10:08 USSDGtbs-1 qlc: [ID 486054 kern.info] Qlogic FCA Driver v0.40.5 (0): F/W version 2.01.112
Nov 22 20:10:08 USSDGtbs-1 rootnex: [ID 349649 kern.info] pcisch0 at root: SAFARI 0x8 0x700000
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 936769 kern.info] pcisch0 is /pci@8,700000
Nov 22 20:10:08 USSDGtbs-1 rootnex: [ID 349649 kern.info] pcisch2 at root: SAFARI 0x9 0x700000
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 936769 kern.info] pcisch2 is /pci@9,700000
Nov 22 20:10:08 USSDGtbs-1 rootnex: [ID 349649 kern.info] pcisch3 at root: SAFARI 0x9 0x600000
Nov 22 20:10:08 USSDGtbs-1 genunix: [ID 936769 kern.info] pcisch3 is /pci@9,600000
Nov 22 20:10:09 USSDGtbs-1 qlc: [ID 686697 kern.info] NOTICE: Qlogic qlc(0): Loop ONLINE
Nov 22 20:10:09 USSDGtbs-1 pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@2, qlc0
Nov 22 20:10:09 USSDGtbs-1 genunix: [ID 936769 kern.info] qlc0 is /pci@8,600000/SUNW,qlc@2
Nov 22 20:10:09 USSDGtbs-1 genunix: [ID 936769 kern.info] fp0 is /pci@8,600000/SUNW,qlc@2/fp@0,0
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd0 at fp0: name w500000e01114c581,0, bus address e4
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd0 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114c581,0
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114c581,0 (ssd0) online
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd1 at fp0: name w500000e01113edf1,0, bus address ef
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd1 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113edf1,0
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113edf1,0 (ssd1) online
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd2 at fp0: name w500000e01113f001,0, bus address e0
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd2 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113f001,0
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113f001,0 (ssd2) online
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd3 at fp0: name w500000e01114be41,0, bus address e1
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd3 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114be41,0
Nov 22 20:10:12 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:12 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114be41,0 (ssd3) online
Nov 22 20:10:13 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd4 at fp0: name w500000e01114c6c1,0, bus address e2
Nov 22 20:10:13 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd4 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114c6c1,0
Nov 22 20:10:13 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:13 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01114c6c1,0 (ssd4) online
Nov 22 20:10:13 USSDGtbs-1 scsi: [ID 799468 kern.info] ssd5 at fp0: name w500000e01113e3c1,0, bus address e8
Nov 22 20:10:13 USSDGtbs-1 genunix: [ID 936769 kern.info] ssd5 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113e3c1,0
Nov 22 20:10:13 USSDGtbs-1 scsi: [ID 365881 kern.info] <SUN146G cyl 14087 alt 2 hd 24 sec 848>
Nov 22 20:10:13 USSDGtbs-1 genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01113e3c1,0 (ssd5) online
Nov 22 20:10:13 USSDGtbs-1 scsi: [ID 243001 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0 (fcp0):
Nov 22 20:10:13 USSDGtbs-1 ndi_devi_online failed for ses: target=dc lun=0
Nov 22 20:10:14 USSDGtbs-1 swapgeneric: [ID 308332 kern.info] root on /pseudo/md@0:0,10,blk fstype ufs
Nov 22 20:10:14 USSDGtbs-1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_trans failed
Nov 22 20:10:14 USSDGtbs-1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_raid failed
Nov 22 20:10:14 USSDGtbs-1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_hotspares failed
Nov 22 20:10:14 USSDGtbs-1 genunix: [ID 370176 kern.warning] WARNING: forceload of misc/md_sp failed
Nov 22 20:10:15 USSDGtbs-1 pcisch: [ID 370704 kern.info] PCI-device: ebus@1, ebus0
Nov 22 20:10:24 USSDGtbs-1 ebus: [ID 521012 kern.info] todds12870 at ebus0: offset 1,300070


please advice

thank you


wu

Maybe driver error/bug on one of your pci cards,..A multi-threaded daemon processes for both the SCCP and TCAP layers that provide an interface to the respective drivers.

Log a call with Sun and send them the core dump to get analised..

on the surface, it looks like transport errors on your fibre disks

run an iostat -En command to see if there are any transport errors.

But from your logs it shows.

Nov 22 20:03:45 USSDGtbs-1 md_stripe: [ID 641072 kern.warning] WARNING: md: d61: write error on /dev/dsk/c1t0d0s7
Nov 22 20:03:46 USSDGtbs-1 md_stripe: [ID 641072 kern.warning] WARNING: md: d62: write error on /dev/dsk/c1t1d0s7

are u using slice 7 as your metadb? run a metadb -i make sure your metadb are still there.

Then run a metastat -p to find out if anything has happened to your mirrors.