Solaris

I have a server on Solaris 8 who had a sudden reboot once and now the messages log is filled with these errors:

Mar 23 02:14:42 srt-hlr-b unix: [ID 274950 kern.notice] NOTICE: kstat_create('unix', 0, 'icmp_minor_1'): cannot create kstat before kstat_init()
Mar 23 02:14:42 srt-hlr-b unix: [ID 274950 kern.notice] NOTICE: kstat_create('vmem', 1073741785, 'icmp_minor'): cannot create kstat before kstat_init()
Mar 23 02:14:42 srt-hlr-b unix: [ID 274950 kern.notice] NOTICE: kstat_create('unix', 0, 'icmp_minor_1'): cannot create kstat before kstat_init()
Mar 23 02:14:42 srt-hlr-b unix: [ID 274950 kern.notice] NOTICE: kstat_create('vmem', 1073741786, 'icmp_minor'): cannot create kstat before kstat_init()
Mar 23 02:14:42 srt-hlr-b unix: [ID 274950 kern.notice] NOTICE: kstat_create('unix', 0, 'icmp_minor_1'): cannot create kstat before kstat_init()


The messages file size is almost 0.8 GB. I noticed kstat is not working and netstat is returning an error:

root.srt-hlr-b.147> kstat
root.srt-hlr-b.148> netstat
netstat: kstat_open(): can't open /dev/kstat: No such file or directory
root.srt-hlr-b.149>
root.srt-hlr-b.142> l /dev/kstat
lrwxrwxrwx 1 root 31 Oct 27 18:04 /dev/kstat -> ../devices/pseudo/kstat@0:kstat
root.srt-hlr-b.143> l /devices/pseudo/kstat@0:kstat
crw-rw-rw- 1 sys 90, 0 Oct 27 18:04 /devices/pseudo/kstat@0:kstat
root.srt-hlr-b.144> file /devices/pseudo/kstat@0:kstat
/devices/pseudo/kstat@0:kstat: character special (90/0)
root.srt-hlr-b.149> top
kstat_open : Error 0

Can anyone help understanding this and the fix?

Is it repeatedly happening? Are your interfaces ok? ifconfig .. netstat -rn.. physical check on the LED on NIC

Yes, this error is reported almost more than one time every second. ifconfig shows all the ports configured but netstat returns an error:

netstat -rn
netstat: kstat_open(): can't open /dev/kstat: No such file or directory

System reboot seems to resolve the issue for some time... but it does come back after few days. I would really appreciate if some one can help or give a hint... Thanks!

What caused the original reboot? Has the system been patched recently? Have there been any new devices or applications installed?

The reboot was caused by Oracle which evicted the node and hence rebooted the server (3320 Disk Array Configuration for Oracle 10). I do not like the way Oracle force the reboot (as if it is power outage); but how can I tell what caused this problem, is it corruption somewhere, etc?

First I'd try a reboot into single-user mode and see how kstat behaves. If you have system support you should open a Sun case, it appears the OS is corrupted, it may be minor or it may be major, I'm not knowledgeable enough to make that call.

If you have configured coreadm to generate core files you should see if one was generated for the Oracle reboot. Same goes for dumpadm and a kernel dump.