Solaris

Hello everyone,

I'm here today looking for help...

I have a SunFire v440 running Solaris 9 at work that manages the SCSI tape drives(LTO3) in our L700 tape library (runs SAMFS).

I went into work today and found the machine was ping-able but I could not connect to it via SSH and could not get video on the KVM. I also noticed our other SAMFS server could not connect back to it anymore.

I attached my laptop to the serial ALOM and attached to the console.... NOTHING!

So, I exited the console (.#) and issued a 'poweroff' command; sure enough the server powered off.

However, when I issued 'poweron' and attached to the console again I noticed the machine is stuck in a reboot loop. It POSTS but when it attempts to boot then it just resets. I can see video on the KVM for a short period of time between the end of the post and the begining of the boot process.

I then turned the servers key to diagnostic mode and issued the poweron command again. When I attach to the console I can see that all the POST tests pass but once again when the system attempts to boot it hangs for a while then quickly stack dumps and resets. I was able to Stop-A and and get into the OpenBoot menu. The system has 4 internal drives. disk0 and disk1 are SVM RAID (mirror). I noticed the LED on disk0 is not illuminating but still no fault lights. When I check the boot device it is configured as "disk0 disk1". I changed the boot device to "disk1", committed the NVRAM changes, and issued a 'boot' command--SAME thing.... stacks dumps and resets.

Mind you this is all very preliminary diagnosis (from 3pm to 4:30pm today). Although this is a production box it can go a day or two being down without any serious problems.

At this point I'm thinking data corruption. I plan on booting off the Solaris 9 CD and fsck the volume.

Does this seem appropriate? Should I FLAR the drive before just to be safe (if it is readable)?

Any thing else I may not be thinking of?

Any help is greatly appreciated!

Thanks!

What is the output:

Code:

boot -v

The last time my Sun Fire v490 went to a loop without booting(post POST) ,i had to replace the System Board though there were no hardware issues posted during diagnostics mode...I hope the server is on support.

HG

Plug out your disks, THEN power on, the system will drop to ok> prompt. setenv auto-boot? false then do a reset and do MAX POST test/obdiag to see what errors you get.

@honglus: Ok, boot -v provided a more verbose boot up but still crash dumps and reboots. I'll try this again now that auto-boot? is false.

@Hari_Ganesh: As for coverage.... no! My company, against my suggestions, let the contract drop. And, I was just told that Oracle (according to my Sun FSE) will not do T&M anymore.

@incredible: I unplugged all drives.... got to the ok> prompt, tehn set auto-boot? to false, saved the nvram, reset the machine, and this time ran obdiag. Within obdiag I setenv diag-level max and then did a test-all. Everything passes!

---------- Post updated at 09:14 AM ---------- Previous update was at 08:55 AM ----------

I was able to see the error message before the stack dump and reset.

The error appears to be: Illegal major device number

I think the system is having a problem initializing the SCSI tape drive or the SCSI card itself. Although it doesn't list any errors it appears to be very slow during boot while initializing PCI/SCSI2. PCI/SCSI1 does not appear to have the same problem.

---------- Post updated at 09:25 AM ---------- Previous update was at 09:14 AM ----------

Ok, illegal major device number appears to be a SVM mirror issue so I'm back to square one.

Re: SVM mirror not working , illegal major device number...

I'll let you all know how this pans out. Thanks for the suggestions. As always, more suggestions are welcome--I am by no means a SVM expert

Remove the suspected card, or with minimal configuration, reboot the system.
If possible,plug in the boot disk and reboot to single user (as a raw disk) after doing so. If it goes to single user without ANY errors, you may Ctrl+D to proceed to up it in multi user mode. If got errors, let us know

@incredible: I'm sure it is a disk problem at this point. I booted to the Solaris 9 CD and ran format. Format cannot even stat disk0 (c0t0d0). I tried an fsck against /dev/dsk/c0t0d0s0 and it failed (no surprise there). So, I ran a fsck -m against /dev/dsk/c1t1d0s0 (SVM mirror) and it stated the drive needed to be fscked. So I've run fsck twice so far against /dev/dsk/c1t1d0s0 and it found a bunch of errors. Once fsck runs with no errors (probably after this run) I'll try mounting the volume to see what kind of carnage I'm dealing with.

With a little luck, I should be able to get it booting off disk1 and rebuild disk0 with a new disk.

I'll keep you guys updated