Solaris

i doubt that /var was ever mounted ro as that would mean that the system would have complained way lot earlier as the system does write to /var/adm/messages fairly regularly even if it doesn't need to write to /var/adm/wtmpx ... i'm convinced that the filesystem had some corruption that gave it a problem booting up ... btw, what was the last thing that happened to the server prior to it having the boot problem? was it patched and then rebooted? if patched, was the server in single-user mode or without regular user logins/processes at all during the patching?

The read only response was after the system came up in maintenance mode.

Absolutely. I didn't start the troubleshooting on the system. It was handed over to me after the first guy had to go home. It was at the maintenance prompt when I got it. They were unable to Ctrl-Brk to the open boot prom since Chris (the H&E guy) couldn't find the ctrl or brk keys. I picked it up there.

The error I had was along the lines of (it's been two weeks) "can't create utmp".

Attempting to umount /var returned the mount point busy message. Chris hit enter on fsck before I could tell him to type in the specific file system I wanted to check. When it got to /var, it responded that /var was mounted read only.

According to the team lead, an application (a cisco management package) was being upgraded which required a reboot after it was done.

Carl

While I haven't seen this on Solaris, I have seen /var remounted as read-only on linux boxes. Happened to two of our systems in the past couple of months and I have one in the state right now. Apparently it's a symptom of a disk getting ready to go south.

Code:

Linux$ sudo ls -la
Password:
sudo: Can't open /var/run/sudo/carls/0: Read-only file system
collect: Cannot write ./dfjBSI0WES028034 (bfcommit, uid=51, gid=51): Read-only file system
queueup: cannot create queue file ./qfjBSI0WES028034, euid=51: Read-only file system

But that's linux.

Carl

Actually, that's not quite right. A system where var (or even root if var is on root disk) is ro then the first complaint will be that message indicated by BOHF, if you have a spare test box try it and see. This somtimes does happen even after an fsck has completed and the command

remount -o rw,remount /var

is required to remount it read-write, i think BOHF has already said that he tried this, it might appear that a reboot should fix this but it is not always the case. I have seen this message many times but it has never that I can recall been the precursor to a disk failure nor even left the system unrecoverable.

Whats the problem with running an fuser to see what process is holding onto the /var filesystem?

i didn't say anything to the contrary ... all i said was it is highly unlikely that the /var filesystem was ever intentionally mounted ro --- as in somebody edited /etc/vfstab then rebooted the box ...

i agree that the ro filesystem error in itself does not necessarily mean that the drive is going to undergo disk failure or that the system is unrecoverable --- however --- experience tells me that a persistent ro filesystem error even after several successful fsck runs may be a symptom of a corrupted filesystem ... and since disk errors can also contribute to filesystem errors, i would not disqualify a failing disk from the picture that quickly when i start troubleshooting ...

i just googled this and found it interesting how a cisco software application that just sits on the server gets to corrupt the filesystem ...

Agreed, however since the filesystem was mounted ro, I didn't understand how was a successful fsck ever run. That was really the point I was getting at: to run the fsck.