iSCSI issue on RHEL 5 Post: 302667433

Sponsored Content

Operating Systems Linux Red Hat iSCSI issue on RHEL 5 Post 302667433 by zer0sig on Friday 6th of July 2012 09:24:09 AM

07-06-2012

Registered User

I've found that RHEL gets bound up pretty easily when the iSCSI/FCOE storage requests don't come back as expected.
I was building a small CAD cluster last year, and the build was fairly heavily automated (PXE boot to a kickstart minimal image to HP-SAS for provisioning (we were also automating loading/patching/etc through it, which was dandy..when it worked).. these servers were each using a volume group of ~20 SAN-allocated disks, and there were scripts to initialize, test and lay down a raw Oracle filesystem.

Well, these machines were running dual 10Gb NICS to eachother, and half of the volume group was allocated to each NIC. So the series of scripts, in between the base OS and Oracle installs runs the init disk routine. These machines have 16 cores, and the CPU utilization goes from almost nothing to 100% and the build is taking longer than expected. Noticing that the machine was not responding right, I start going through the logs. I notice one core is unresponsive and the kernel is throwing panic messages.

Turns out, one of the 10Gb cards was actually faulty, but with the 6 or so different links and the other 15 cores running normally, this one was stuck trying to execute a temporary script, and all it was doing was sending requests via iSCSI and not getting them back. The kernel was so caught up with this that it brought the entire machine to a crawl, as though it were in a race condition. kill the script and all is hunky-dory except for the build, which of course failed. Once the NOC team got to check on the links, they found the problem, replaced the card, and a little manual scrubbing of the now-complete volume group and the rest went smoothly.

I haven't seen how other PC-based Unix-style OSes fare under these conditions, but I can't help but think that there should be some failsafes to keep one core running one simple script from turning the machine into a $40K paperweight. I assume that the priority given to the high-end storage subsystems is likely to improve overall performance - as long as everything is working.

zer0sig

View Public Profile for zer0sig

Find all posts by zer0sig

10 More Discussions You Might Find Interesting

1. Red Hat

RHEL Installation issue.

Hi Gurus I am trying to install RHEL linux enterprise Edition v5 on a Dell Optiplex 320 desktop. After the Grub menu,i get the error,Kernel could not be loaded. I am a newbie in linux and would appreciate any help in moving forward. Thanks HG

2. Red Hat

Time Sync issue on RHEL 5.3

Hi, My Linux VMs are running on citrix XEN.Somehow the time drift is too high (going very fast). Even if the correct time configured using date -s command, for every 1 hr it is differing almost 5 - 10 mins. Tried configuring ntpd (local NTP server) and still it is not synchronized. Machine...

3. UNIX for Dummies Questions & Answers

How to change ethernet cable of iscsi initiator (iscsi client)?

how to change ethernet cable of iscsi initiator (iscsi client)

4. Red Hat

Does RHEL 5 provide a command to collect RHEL system log in single compress file?

Hi, I heard a command that can collect all RHEL 5 log in a single compress file before I forget. Does any body know...What the command is ? Thanks.

5. Red Hat

Error throwing while installing vsftpd package in rhel 6. using rhel 6 dvd.

Hi all, Im studying rhcsa as of now, so yum installation and dependencies are messing me to not workit out. i have dual os, win 7 & rhel 6. i have tried this installation of vsftpd package with rhel 6 dvd in VM rhel 6 in win 7 as well as host rhel 6.still the same issue. below error...

6. Red Hat

Logrotate issue in RHEL

Hi, I have a requirement to compress logs that generate daily. Below is the logrotate configuration for this But for some kind of reason the logs are not getting compressed daily. When tried to execute the logrotate manually with -d it says "log does not need rotating." The path...

7. Linux

Installation issue RHEL 6.5 on SAN

HI All I am facing issue in installation of Red Hat 6.5 on SAN.. i am having a partion of 2 TB in my SAN and i am installing the operating system after completed the installation it not booting and giving the error.

8. Red Hat

RHEL Memory issue

Friends, I have 8CPU/24gb RAM in my Linux servers around 30+. All the servers are running with Java based application with NFS mounted. The problem what I have is, When I reboot servers the server memory looks decent. But after some time the "cached" memory increasing and "free" memory...

9. Shell Programming and Scripting

AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue

Greetings Experts, We are migrating from AIX to RHEL Linux. I have created a script to verify and report the NULLs and SPACEs in the key columns and duplicates on key combination of "|" delimited set of big files. Following is the code that was successfully running in AIX. awk -F "|" 'BEGIN {...

10. UNIX for Advanced & Expert Users

RHEL 5.11 - gid issue

Hello folks, I am facing a weird issue on one of the red hat linux 5 box. There is an oracle id which i want to change the primary group. Currently the primary group id is 16600, I did changed it to 16601 using below command: usermod -g 16601 oracle But it did not changed. I have tried...

LEARN ABOUT DEBIAN

xen-update-image

XEN-UPDATE-IMAGE(8)					 Perl Programmers Reference Guide				       XEN-UPDATE-IMAGE(8)

NAME

       xen-update-image - Update the software installed upon offline Xen images.

SYNOPSIS

	 xen-update-image [options] imageName1 imageName2 .. imageNameN

	 Help Options:
	  --help      Show this scripts help information.
	  --manual    Read this scripts manual.
	  --version   Show the version number and exit.

	 General Options:
	  --dir       Specify the directory which contains the image(s).
	  --lvm       Specify the LVM volume group which contains the image(s).
	  --evms      Specify the EVMS container which contains the image(s).

OPTIONS

       --dir Specify the directory which contains the image(s).
       --evms Specify the EVMS container which contains the image(s).
       --help Show the script help.
       --lvm Specify the LVM volume group which contains the image(s).
       --manual Read the manual.
       --version Show the version number and exit.

DESCRIPTION

	 xen-update-image is a simple script which allows you to update
	a Xen image of Debian which has been created with xen-create-image.

	 It does this by mounting the image inside a temporary directory
	then running:

	     apt-get update

	     apt-get upgrade

	 NOTE If the image is already running within Xen this will cause
	corruption otherwise it will allow you to update your image without
	booting it.

EXAMPLES

	 The following assumes there are two images which are not currently
	running.  The images are called 'test.my.flat', and 'x11.my.flat'.

	 Updating both images can be accomplished by executing:

	    xen-update-images --dir=/home/xen test.my.flat x11.my.flat

AUTHORS

	Steve Kemp, http://www.steve.org.uk/
	Axel Beckert, http://noone.org/abe/
	StA~Xphane Jourdois

LICENSE

       Copyright (c) 2005-2009 by Steve Kemp, (c) 2010 by The Xen-Tools Development Team. All rights reserved.

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.  The LICENSE file contains the
       full text of the license.

4.3.1								    2012-06-30						       XEN-UPDATE-IMAGE(8)

10 More Discussions You Might Find Interesting

1. Red Hat

RHEL Installation issue.

Discussion started by: Hari_Ganesh

2. Red Hat

Time Sync issue on RHEL 5.3

Discussion started by: uday123

3. UNIX for Dummies Questions & Answers

How to change ethernet cable of iscsi initiator (iscsi client)?

Discussion started by: pankajd

4. Red Hat

Does RHEL 5 provide a command to collect RHEL system log in single compress file?

Discussion started by: nnnnnnine