Linux

We have a Dell 2950 attached to a MD3000 Disk Array and it is running Red Hat ES4. The system was running flawlessly for a year and then this month we started getting filesystem errors and the filesystem is setting itself to read-only, even after running fsck NUMEROUS times and having it tell us that it is ok.

We have contacted our Dell support and they suggest doing all sorts of Software and Firmware Updates. Red Hat says that it is a Hardware Issue. We are a Community College located in Central California and we are trying to find a individual or Company that could has VAST knowledge of Red Hat running on a Dell system that would like to take a look at our system.

You can contact me at:
Jae Markham
Oracle DBA
College of the Sequoias
jaem@cos.edu

What is the output of the "dmesg" command, and what is the filesystem type ("cat /proc/mounts" the type is the third column)?

sdb1 is the filesystem with the errors....it is located on the DELL MD3000 disk array.

Buffer I/O error on device sdd, logical block 15
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
Buffer I/O error on device sdd, logical block 1
Buffer I/O error on device sdd, logical block 2
Buffer I/O error on device sdd, logical block 3
Buffer I/O error on device sdd, logical block 4
Buffer I/O error on device sdd, logical block 5
Buffer I/O error on device sdd, logical block 6
Buffer I/O error on device sdd, logical block 7
Buffer I/O error on device sdd, logical block 8
Buffer I/O error on device sdd, logical block 9
Buffer I/O error on device sdd, logical block 10
Buffer I/O error on device sdd, logical block 11
Buffer I/O error on device sdd, logical block 12
Buffer I/O error on device sdd, logical block 13
Buffer I/O error on device sdd, logical block 14
Buffer I/O error on device sdd, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
Buffer I/O error on device sde, logical block 1
Buffer I/O error on device sde, logical block 2
Buffer I/O error on device sde, logical block 3
Buffer I/O error on device sde, logical block 4
Buffer I/O error on device sde, logical block 5
Buffer I/O error on device sde, logical block 6
Buffer I/O error on device sde, logical block 7
Buffer I/O error on device sde, logical block 8
Buffer I/O error on device sde, logical block 9
Buffer I/O error on device sde, logical block 10
Buffer I/O error on device sde, logical block 11
Buffer I/O error on device sde, logical block 12
Buffer I/O error on device sde, logical block 13
Buffer I/O error on device sde, logical block 14
Buffer I/O error on device sde, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
end_request: I/O error, dev sde, sector 1754664952
Buffer I/O error on device sde, logical block 219333119
end_request: I/O error, dev sde, sector 1754664952
Buffer I/O error on device sde, logical block 219333119
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
Buffer I/O error on device sde, logical block 1
Buffer I/O error on device sde, logical block 2
Buffer I/O error on device sde, logical block 3
Buffer I/O error on device sde, logical block 4
Buffer I/O error on device sde, logical block 5
Buffer I/O error on device sde, logical block 6
Buffer I/O error on device sde, logical block 7
Buffer I/O error on device sde, logical block 8
Buffer I/O error on device sde, logical block 9
Buffer I/O error on device sde, logical block 10
Buffer I/O error on device sde, logical block 11
Buffer I/O error on device sde, logical block 12
Buffer I/O error on device sde, logical block 13
Buffer I/O error on device sde, logical block 14
Buffer I/O error on device sde, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sde, sector 0
kjournald starting. Commit interval 5 seconds
EXT3-fs warning (device sdb1): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
EXT3-fs warning (device sdb1): ext3_clear_journal_err: Marking fs in need of filesystem check.
EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
EXT3 FS on sdb1, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sdb1, type ext3), uses xattr
end_request: I/O error, dev sda, sector 0
Buffer I/O error on device sda, logical block 0
Buffer I/O error on device sda, logical block 1
Buffer I/O error on device sda, logical block 2
Buffer I/O error on device sda, logical block 3
Buffer I/O error on device sda, logical block 4
Buffer I/O error on device sda, logical block 5
Buffer I/O error on device sda, logical block 6
Buffer I/O error on device sda, logical block 7
Buffer I/O error on device sda, logical block 8
Buffer I/O error on device sda, logical block 9
Buffer I/O error on device sda, logical block 10
Buffer I/O error on device sda, logical block 11
Buffer I/O error on device sda, logical block 12
Buffer I/O error on device sda, logical block 13
Buffer I/O error on device sda, logical block 14
Buffer I/O error on device sda, logical block 15
end_request: I/O error, dev sda, sector 0
Buffer I/O error on device sda, logical block 0
Buffer I/O error on device sda, logical block 1
Buffer I/O error on device sda, logical block 2
Buffer I/O error on device sda, logical block 3
Buffer I/O error on device sda, logical block 4
Buffer I/O error on device sda, logical block 5
Buffer I/O error on device sda, logical block 6
Buffer I/O error on device sda, logical block 7
Buffer I/O error on device sda, logical block 8
Buffer I/O error on device sda, logical block 9
Buffer I/O error on device sda, logical block 10
Buffer I/O error on device sda, logical block 11
Buffer I/O error on device sda, logical block 12
Buffer I/O error on device sda, logical block 13
Buffer I/O error on device sda, logical block 14
Buffer I/O error on device sda, logical block 15
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
Buffer I/O error on device sdc, logical block 1
Buffer I/O error on device sdc, logical block 2
Buffer I/O error on device sdc, logical block 3
Buffer I/O error on device sdc, logical block 4
Buffer I/O error on device sdc, logical block 5
Buffer I/O error on device sdc, logical block 6
Buffer I/O error on device sdc, logical block 7
Buffer I/O error on device sdc, logical block 8
Buffer I/O error on device sdc, logical block 9
Buffer I/O error on device sdc, logical block 10
Buffer I/O error on device sdc, logical block 11
Buffer I/O error on device sdc, logical block 12
Buffer I/O error on device sdc, logical block 13
Buffer I/O error on device sdc, logical block 14
Buffer I/O error on device sdc, logical block 15
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 1754664952
Buffer I/O error on device sdc, logical block 219333119
end_request: I/O error, dev sdc, sector 1754664952
Buffer I/O error on device sdc, logical block 219333119
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
Buffer I/O error on device sdc, logical block 1
Buffer I/O error on device sdc, logical block 2
Buffer I/O error on device sdc, logical block 3
Buffer I/O error on device sdc, logical block 4
Buffer I/O error on device sdc, logical block 5
Buffer I/O error on device sdc, logical block 6
Buffer I/O error on device sdc, logical block 7
Buffer I/O error on device sdc, logical block 8
Buffer I/O error on device sdc, logical block 9
Buffer I/O error on device sdc, logical block 10
Buffer I/O error on device sdc, logical block 11
Buffer I/O error on device sdc, logical block 12
Buffer I/O error on device sdc, logical block 13
Buffer I/O error on device sdc, logical block 14
Buffer I/O error on device sdc, logical block 15
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
Buffer I/O error on device sdd, logical block 1
Buffer I/O error on device sdd, logical block 2
Buffer I/O error on device sdd, logical block 3
Buffer I/O error on device sdd, logical block 4
Buffer I/O error on device sdd, logical block 5
Buffer I/O error on device sdd, logical block 6
Buffer I/O error on device sdd, logical block 7
Buffer I/O error on device sdd, logical block 8
Buffer I/O error on device sdd, logical block 9
Buffer I/O error on device sdd, logical block 10
Buffer I/O error on device sdd, logical block 11
Buffer I/O error on device sdd, logical block 12
Buffer I/O error on device sdd, logical block 13
Buffer I/O error on device sdd, logical block 14
Buffer I/O error on device sdd, logical block 15
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
Buffer I/O error on device sdd, logical block 1
Buffer I/O error on device sdd, logical block 2
Buffer I/O error on device sdd, logical block 3
Buffer I/O error on device sdd, logical block 4
Buffer I/O error on device sdd, logical block 5
Buffer I/O error on device sdd, logical block 6
Buffer I/O error on device sdd, logical block 7
Buffer I/O error on device sdd, logical block 8
Buffer I/O error on device sdd, logical block 9
Buffer I/O error on device sdd, logical block 10
Buffer I/O error on device sdd, logical block 11
Buffer I/O error on device sdd, logical block 12
Buffer I/O error on device sdd, logical block 13
Buffer I/O error on device sdd, logical block 14
Buffer I/O error on device sdd, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
Buffer I/O error on device sde, logical block 1
Buffer I/O error on device sde, logical block 2
Buffer I/O error on device sde, logical block 3
Buffer I/O error on device sde, logical block 4
Buffer I/O error on device sde, logical block 5
Buffer I/O error on device sde, logical block 6
Buffer I/O error on device sde, logical block 7
Buffer I/O error on device sde, logical block 8
Buffer I/O error on device sde, logical block 9
Buffer I/O error on device sde, logical block 10
Buffer I/O error on device sde, logical block 11
Buffer I/O error on device sde, logical block 12
Buffer I/O error on device sde, logical block 13
Buffer I/O error on device sde, logical block 14
Buffer I/O error on device sde, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
end_request: I/O error, dev sde, sector 1754664952
Buffer I/O error on device sde, logical block 219333119
end_request: I/O error, dev sde, sector 1754664952
Buffer I/O error on device sde, logical block 219333119
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
Buffer I/O error on device sde, logical block 1
Buffer I/O error on device sde, logical block 2
Buffer I/O error on device sde, logical block 3
Buffer I/O error on device sde, logical block 4
Buffer I/O error on device sde, logical block 5
Buffer I/O error on device sde, logical block 6
Buffer I/O error on device sde, logical block 7
Buffer I/O error on device sde, logical block 8
Buffer I/O error on device sde, logical block 9
Buffer I/O error on device sde, logical block 10
Buffer I/O error on device sde, logical block 11
Buffer I/O error on device sde, logical block 12
Buffer I/O error on device sde, logical block 13
Buffer I/O error on device sde, logical block 14
Buffer I/O error on device sde, logical block 15
end_request: I/O error, dev sde, sector 0
Buffer I/O error on device sde, logical block 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sde, sector 0
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdf1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sdf1, type ext3), uses xattr
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 56173
CIFS VFS: No response to cmd 47 mid 56172
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 21333
CIFS VFS: No response to cmd 47 mid 21334
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 28984
CIFS VFS: No response to cmd 47 mid 28985
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 7408
inode_doinit_with_dentry: getxattr returned 5 for dev=sdb1 ino=26083332
inode_doinit_with_dentry: getxattr returned 5 for dev=sdb1 ino=26083335
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 33912
CIFS VFS: No response to cmd 47 mid 33913
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 36441
CIFS VFS: No response to cmd 47 mid 36442
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 48932
CIFS VFS: No response to cmd 47 mid 48933
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 43751
CIFS VFS: No response to cmd 47 mid 43750
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 62470
CIFS VFS: No response to cmd 47 mid 62471
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 5485
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 10546
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 38223
CIFS VFS: No response to cmd 47 mid 38224
CIFS VFS: No response to cmd 47 mid 38225
CIFS VFS: No response to cmd 47 mid 38227
CIFS VFS: No response to cmd 47 mid 38226
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 22232
CIFS VFS: No response to cmd 47 mid 22233
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 44658
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 61974
CIFS VFS: No response to cmd 47 mid 61975
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 23684
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 31655
CIFS VFS: No response to cmd 47 mid 31657
CIFS VFS: No response to cmd 47 mid 31656
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 53579
CIFS VFS: No response to cmd 47 mid 53580
CIFS VFS: No response to cmd 47 mid 53582
CIFS VFS: No response to cmd 47 mid 53581
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 9676
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 49986
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 5669
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 21107
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 46389
CIFS VFS: No response to cmd 47 mid 46390
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 12051
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 29197
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 34412
CIFS VFS: No response to cmd 47 mid 34413
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 41794
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 46182
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 16270
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 19862
CIFS VFS: No response to cmd 47 mid 19863
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 15817
CIFS VFS: No response to cmd 47 mid 15818
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 31143
CIFS VFS: No response to cmd 47 mid 31144
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 40825
CIFS VFS: No response to cmd 47 mid 40826
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 48864
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 49842
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 63449
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 4484
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 36665
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 57860
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 58353
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 3254
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 11593
CIFS VFS: No response to cmd 47 mid 11594
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 13563
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 16132
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 53666
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 1337
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 47546
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 52942
CIFS VFS: No response to cmd 47 mid 52943
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 7400
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 9912
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 16184
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 33141
CIFS VFS: No response to cmd 47 mid 33142
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 40028
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 41550
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 54665
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 61039
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 14333
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 59417
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 33978
CIFS VFS: No response to cmd 47 mid 33979
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 41277
CIFS VFS: No response to cmd 47 mid 41278
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 11487
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 24437
CIFS VFS: No response to cmd 47 mid 24438
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 30806
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 14269
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 48638
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 54119
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 91
CIFS VFS: No response to cmd 47 mid 94
CIFS VFS: No response to cmd 47 mid 93
CIFS VFS: No response to cmd 47 mid 92
CIFS VFS: No response to cmd 47 mid 95
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 18137
CIFS VFS: No response to cmd 47 mid 18139
CIFS VFS: No response to cmd 47 mid 18138
CIFS VFS: No response to cmd 47 mid 18140
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 51058
CIFS VFS: No response to cmd 47 mid 51059
CIFS VFS: No response to cmd 47 mid 51060
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 18918
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 46632
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 51455
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 56482
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 31902
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 6236
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 11291
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 19711
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 56731
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 63011
attempt to access beyond end of device
sdb1: rw=0, want=31664112360, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=31664112360, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=22401516072, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=22401516072, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=33403700936, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=33403700936, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=25601061832, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=29836160816, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=25851426896, limit=1754651367
attempt to access beyond end of device
sdb1: rw=0, want=25601061832, limit=1754651367
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 9486
CIFS VFS: No response to cmd 47 mid 9487
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 17165
CIFS VFS: No response to cmd 47 mid 17166
CIFS VFS: server not responding
CIFS VFS: No response to cmd 47 mid 20619
CIFS VFS: No response to cmd 47 mid 20620



/dev/sdb1 /u01 ext3 rw 0 0

hi jd,

wow ... that is bad.
AFAIK - if i'm not mistaken - that is a hardware (both the disks and disk controller) failure - that can render your disk to have bad sectors/unusable (or maybe it already did). or perhaps - you previously have a bad power supply/outage?