Hi all,
Today I´m faced a situation where two of my ext3 filesystems on a san become read only. This happens 3 times today.
The usual way to get this filesystems read and write was to reboot the server.
But now the continuous go to ro mode instead of rw mode, crashes my database.
I have readed the messages from the server and appears a warning about a EX3-fs error. My firs guest was a damaged hd, but after reviewing my san, there all ok.
Hi,
Thanks for your reply.
Yes here is a part of my messages log.
Code:
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 350662791
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 527806119
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 527806127
Aug 11 10:02:16 FriscoDB kernel: Aborting journal on device dm-9.
Aug 11 10:02:16 FriscoDB kernel: EXT3-fs error (device dm-9) in ext3_ordered_writepage: IO failure
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 383382207
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 383382215
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 383382223
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 383382231
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 474560831
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Aug 11 10:02:16 FriscoDB kernel: end_request: I/O error, dev sdc, sector 383382239
Aug 11 10:02:16 FriscoDB kernel: ext3_abort called.
Aug 11 10:02:16 FriscoDB kernel: EXT3-fs error (device dm-9): ext3_journal_start_sb: Detected aborted journal
Aug 11 10:02:16 FriscoDB kernel: Remounting filesystem read-only
Aug 11 10:02:16 FriscoDB kernel: SCSI error : <2 0 1 0> return code = 0x10000
Apparently there is a physical hd defective. But over my san I can´t see nothing wrong.
After that, the filesystems go to read only and the database crashes.
What is your adivce?
Thank you in advance.
Moderator's Comments:
Please use code tags next time for your code and data. Thanks
A SAN shows LUNs which are virtual disks, so I doubt disk issues, or the SAN disk subsystem should be complaining... but you have controllers, fiber cables etc...
And there is some configuration also... but the command are OS /equipment dependant, Im thinking of queue depth... or jounal log too small
There are plenty reasons why writes fail apart disk failure...
The fact is, due to this is my productive database and this server is off-site, I can´t has Access to the hardware surrounding the SAN, fiber, etc.
When the system is starting, we see a double path for the SAN LUN.
The weird thing is: when I started to use the "default" path, the disks begin to fail after 15 minutes and the failure message start to appear. And then, the filesystems switchs to read only in order to protect the information.
Under this conditions, on the mounting points file, we comment the secondary path, in order to see just one. After doing this, the error message go away.
So, when I run the requested command multipath -l, appears like the command are unavailable.
This is possible?
I mean, work over a SAN without multipath enabled?
What is this two paths that I see in the boot process?
As far as I have knowledge on the hardware configuration, the server has 1 HBA card with two fiber distributed over 2 switches connected to the SAN.
Can be possible to have a one fibre with failure?
If this is true, an alert must be trown to the console, right?
hi! i got a rhel 6.3 host that already have an xfs filesystem mounted from a SAN (let's call it SAN-1) whose size is 9TB.
i will be receiving another SAN (let's call it SAN-2) storage of 15TB size. this new addition is physically on another SAN storage. SAN-1 is on a Pillar storage while the new... (6 Replies)
I have two test machines having solaris 10. I have shared a location which have a package on machine1 and mounted that location onto machine2 as below.
machine1: share -F nfs -o rw /home1/pkg/test
machine2: mount -F nfs -o rw machine2:/home1/pkg/test /tmp/test
Now, when i am trying to... (8 Replies)
Has anyone tried SAN to SAN mirroring on IBM DS SAN Storage.
DS5020 mentions Enhanced Remote Mirror to multi-LUN applications
I wonder if Oracle High availibility can be setup using Remote Mirror option of SAN ? (1 Reply)
I am researching the cause of an issue. The SAN file system /export/pools/zd-xxxxxxxxxxx is having a high amount of read traffic even though it is empty. It is ZFS with MPXIO. Any ideas? It's really strange considering the file system is empty, and I don't see any errors.
cpu
us sy... (2 Replies)
hi guys
suse 10.2
today some guys are adding some luns from a SAN so in order to add them I followed this command
echo 1 > /sys/class/fc_host/host1/issue_lip
echo "- - -" > /sys/class/scsi_host/host<number>/scan
nice now
after that I tried to create a folder in /
but it won't let... (2 Replies)
I'm trying to find a correct command that will mount a filesystem ext3 the device /dev/sda1 to mount point /mnt/usb but not allow any programs to run from it.
I want to be honest as I've read the forum rules... This is a homework question BUT, i'm generally interested in the correct process and... (3 Replies)
Hi everyone,
I wonder if I can canvas any opinions or thoughts (good or bad) on SAN attaching a SUN V880/490 to an EMC Clarion SAN?
At the moment the 880 is using 12 internal FC-AL disks as a db server and seems to be doing a pretty good job. It is not I/O, CPU or Memory constrained and the... (2 Replies)
Hi all,
I was installing linux, so i want to know the basic difference between /ext2 and /ext3 filetype..
what is the significance of using /ext2 and /ext3
pls clarify
Thanks
Vasikaran (2 Replies)
As i'm sure you all know, ext2 (and then ext3) is the most widely used linux fs... but some distros (SUSE & Debian) are now chosing to either use by default, or at least give the option of using ReiserFS.
What is it about ReiserFS that's so great exactly? I understand that it has a limitless... (6 Replies)
08-12-2014
