Login or Register to Ask a Question and Join Our Community


Emergency UNIX and Linux Support

Rootvg in read only state

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Homework and Emergencies Emergency UNIX and Linux Support Rootvg in read only state
# 1  
Old 01-25-2013
Rootvg in read only state
I have Oracle Linux 5.7. It was hanged, so I rebooted it and now it came into maintenance mode and asking for root password. After password, it is at
Code:
(Repair filesystem) 10 #

I am not sure, where to go from here. All file-systems are in vg. Non-root is coming from EMC storage and root is from local disks.
Root filesystem is as below -
Code:
/dev/cciss/c0d0p2 62G 29G 31G 49% /
/dev/cciss/c0d0p1 62G 29G 31G 49% /boot

I also ran fsck as below and rebooted, but no luck.
Code:
fsck /dev/cciss/c0d0p2
fsck /dev/cciss/c0d0p1

Please suggest.
Last edited by vbe; 01-25-2013 at 12:58 PM..
# 2  
Old 01-25-2013
You probably need to reactivate your volume group. See the vgchange man page.
# 3  
Old 01-26-2013
Rootvg in read only state
It was ASR, which rebooted server. Now I am able to bring server up.
- Remounted root fs as rw
- commented all file-systems coming from SAN from /etc/fstab
- Rebooted server
- Uncommented SAN fs and mounted all file-system manually.
But still need to check an issue. If I give a clean reboot, it again stops for maintenance and I have to follow above process. I have attached screenshot for referance. Console is in GUI, so can not copy output, so attaching screenshot.
---------------------------------------------------------------------
I am able to get some more information, which it went into file not found for lv, while booting. I think, there is something not standard in /etc/fstab. Here is the copy
Code:
[root@tlprd_spc05 ]# cat /etc/fstab
LABEL=/                 /                       ext3    noatime,defaults        1 1
LABEL=/boot             /boot                   ext2    noatime,defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SW-cciss/c0d0p3   swap                    swap    defaults        0 0
/dev/vg_local/lv_oel5u7_dump                     /oel5u7                           ext3 noatime,defaults 1 2
/dev/vg_local/lv_opt_simpana                     /opt/simpana                      ext3 noatime,defaults 1 2
/dev/vg_tssd/lv_tss                              /tss                              ext3 noatime,defaults 1 2
/dev/vg_tssd/lv_tss_apps                         /tss/apps                         ext3 noatime,defaults 1 2
/dev/vg_tssd/lv_tss_oracle_oradata_TSSD          /tss/oracle/oradata/TSSD          ext3 noatime,defaults 1 2
/dev/vg_tssd/lv_tss_oracle_product_11.2.0_TSSD   /tss/oracle/product/11.2.0/TSSD   ext3 noatime,defaults 1 2
/dev/vg_tssq/lv_tss_oracle_oradata_TSSQ          /tss/oracle/oradata/TSSQ          ext3 noatime,defaults 1 2
/dev/vg_tssq/lv_tss_oracle_product_11.2.0_TSSQ   /tss/oracle/product/11.2.0/TSSQ   ext3 noatime,defaults 1 2
/dev/vg_u01_oracle/lv_u01_oracle                 /u01/oracle                       ext3 noatime,defaults 1 2
/dev/vg_u01_oracle/lv_u01_oracle_export          /u01/oracle/export                ext3 noatime,defaults 1 2
/dev/vg_u01_oracle/lv_u01_oracle_housekeeping    /u01/oracle/housekeeping          ext3 noatime,defaults 1 2
/dev/vg_u01_oracle/lv_u01_oracle_product_11.2.0  /u01/oracle/product/11.1.0        ext3 noatime,defaults 1 2
/dev/vg_oe9d/lv_tss_oracle_oradata_OE9D          /tss/oracle/oradata/OE9D          ext3 noatime,defaults 1 2
/dev/vg_oe9d/lv_tss_oracle_product_11.2.0_OE9D   /tss/oracle/product/11.2.0/OE9D   ext3 noatime,defaults 1 2

I think, while server is coming up, OS started finding logical volumes from SAN. Since HBA may be completely intialised till that time and it resulted in throwing error below
Code:
fsck.ext3: No such file or directory while trying to open /dev/vg_vg_tssd/lv_tss

What can be best to avoid this issue, if I reboot the box again. Please suggest.
Rootvg in read only state-server_mount_issuejpg
Last edited by solaris_1977; 01-28-2013 at 06:37 AM.. Reason: Found more infomration
# 4  
Old 01-28-2013
I've seen this happen with old IBMsdd drivers/software where it attempts to mount filesystems before the driver is actually loaded. We ended up commenting out the entries in /etc/fstab and hacking /etc/rc.local - you should not do that Smilie

What HBAs do you have in the server, is the storage managed by multipathd, etc.?

When the server hung for the first time, was that the first time it was rebooted since these logical volumes were added, or has the server been rebooted successfully before?
# 5  
Old 01-28-2013
You are right about drivers. It seems, same thing is happening here.
EMC Storage is connected to this server via Brocade HBA card (2 nos.) and multipath is managed by EMC powerpath.
Server hung is not a problem, we got the cause for that, but worried about unsuccessful reboot. I am not sure about past reboots, but whenever I am rebooting this Linux server, it always stops at maintenance level. I am suspecting /etc/fstab is playing a role here.
I have already added noatime in /etc/fstab, should it not delay mounting of SAN file-systems ?
# 6  
Old 01-28-2013
No - something lower-level is happening if the device entires do not exist - no manner of changes within /etc/fstab will help that. noatime stops access time being updated within inodes and is a performance boon (sometimes) - it will not delay the mounting of anything.

Does the PowerPath software log anything?
# 7  
Old 01-28-2013
EMC powerpath does not log anything. However, device paths are correct. Once I am able to bring server up without SAN file-systems, I can run mount -a and it mount all devices without any problem.
In /etc/fstab, we have enabled fsck to run on EMC SAN devices, as marked 2 in last field
PHP Code:
/dev/vg_oe9d/lv_tss_oracle_product_11.2.0_OE9D   /tss/oracle/product/11.2.0/OE9D   ext3 noatime,defaults 1 2 
I am suspecting this is forcing SAN file-system to fsck, while system is not yet completely initialised. Is it really required ?
if not, we can change this 2 to something else. Your thoughts please.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

Reducing / on rootvg

The root filesystem was mirrored, someone/something stopped mirroring, and increased / and /home to ridiculous values (/ got increased to 102gb and its only using 4.3gb, so 98gb is free). Can I reduce the / (/dev/hd4) filesytem down WITHOUT corrupting the the OS? I would do a: chfs -a size=10g... (6 Replies)
Discussion started by: mrmurdock
6 Replies

2. AIX

Open firmware state to running state

Hi Admins, I am having a whole system lpar in open firmware state on HMC. How can I bring it to running state ? Let me know. Thanks. (2 Replies)
Discussion started by: snchaudhari2
2 Replies

3. AIX

rootvg mirrored

I want to increase the size of /tmp by 1GB I know that the command is chfs -a size=+1G /tmp But the rootvg is mirrored and when I do a lsvg -p rootvg, I could see 2 disks. Will there be any impact if I increase the size of /tmp when the rootvg is mirrored ? Please advise. (1 Reply)
Discussion started by: newtoaixos
1 Replies

4. Solaris

Solaris rootvg backup

Guy's how can we take rootvg bakup in soraris .. Please let me know the command Pls advice . (5 Replies)
Discussion started by: Mr.AIX
5 Replies

5. AIX

mirroring rootvg

I would like to konw wheather below steps are sufficient for mirroring rootvg. extendvg rootvg hdisk1 mirrorvg rootvg bootlist -m normal hdisk0 hdisk1 bosboot -ad hdisk0 bosboot -ad hdisk1 or anything needs to be added. (7 Replies)
Discussion started by: manoj.solaris
7 Replies

6. AIX

extendvg and rootvg

When attempting to extend rootvg to a new physical disk I received the following message: 0516-1162 extendvg: Warning, The Physical Partition Size of 64 requires the creation of 1093 partitions for hdisk5. The limitation for volume group rootvg is 1016 physical partitions... (4 Replies)
Discussion started by: jyoung
4 Replies

7. AIX

Anyone know how to access rootvg?

I'm trying to troubleshoot an issue with an AIX system here. I have no experience with system administration of IBM stuff. Our local admin is ... um ... not available at this time. I found something online that said enter service mode - How the heck do I do that??? Why is IBM stuff lack proper... (2 Replies)
Discussion started by: bbbngowc
2 Replies

8. AIX

mirror rootvg

Hello I have a question I have a box with Aix 5.3 with rootvg on mirror. I deleted a filesystem and I create a new one to install some software but when I type lsvg -l rootvg all my filesystems has mirror unless the new one. My question is I can mirror this fs only ??? or I have to... (9 Replies)
Discussion started by: lo-lp-kl
9 Replies

9. AIX

Cloned Rootvg

New Question: The purpose of an alternate disk install is in my opinion to have a ready-to-use-backup and i've read that it is possible to install filesets or software on the cloned disk for testing purposal while the normal system is still running. The question is: how do you tell AIX (5.2... (5 Replies)
Discussion started by: kvanelshocht
5 Replies
Login or Register to Ask a Question