Login or Register to Ask a Question and Join Our Community


AIX

AIX hangs through the boot process

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX AIX hangs through the boot process
# 1  
Old 10-27-2007
Question AIX hangs through the boot process
To All:

I have this AIX machine (see boot process below). The problem was it hangs at:

The ctrmc Subsystem has been started. Subsystem PID is 11906..

I checked with other forums but their advise were just "strong punches in the air" with no specific explanation.

I really appreciate if you can help me boot this up directly to the user prompt.

Looking forward for your reply.
AIXXIA


-------------------------------------------------------------------------------
Welcome to AIX.
boot image timestamp: 14:44 10/18
The current time and date: 16:33:53 10/27/2007
number of processors: 2 size of memory: 2048MB
boot device: /pci@400000000110/pci@2,4/scsi@1/sd@8:2
kernel size: 12101874; 32 bit kernel
-------------------------------------------------------------------------------

Saving Base Customize Data to boot disk
Starting the sync daemon
Starting the error daemon
System initialization completed.
Setting tunable parameters...
Sat Oct 27 11:35:05 CDT 2007
Automatic Error Log Analysis for sysplanar0 has detected a problem.
The Service Request Number is
651-880: The CEC or SPCN reported an error. Report the SRN and the following
reference and physical location codes to your service provider..

complete
Starting Multi-user Initialization
Performing auto-varyon of Volume Groups
Activating all paging spaces
0517-075 swapon: Paging device /dev/hd6 is already active.
/dev/rhd1 (/home): ** Unmounted cleanly - Check suppressed
/dev/rhd10opt (/opt): ** Unmounted cleanly - Check suppressed
Performing all automatic mounts
Multi-user initialization completed
Checking for srcmstr active...complete
Starting tcpip daemons:
0513-059 The syslogd Subsystem has been started. Subsystem PID is 5996.
0513-059 The sendmail Subsystem has been started. Subsystem PID is 6496.
0513-059 The portmap Subsystem has been started. Subsystem PID is 8596.
0513-059 The inetd Subsystem has been started. Subsystem PID is 8280.
0513-059 The hostmibd Subsystem has been started. Subsystem PID is 7752.
0513-059 The snmpmibd Subsystem has been started. Subsystem PID is 10332.
0513-059 The snmpd Subsystem has been started. Subsystem PID is 10858.
0513-059 The aixmibd Subsystem has been started. Subsystem PID is 6274.
Finished starting tcpip daemons.
0513-059 The muxatmd Subsystem has been started. Subsystem PID is 13420.
0513-059
The ctrmc Subsystem has been started. Subsystem PID is 11906.
# 2  
Old 10-27-2007
What do you mean by "hang"? Does the machine crash? freeze completely? freeze only for a longer time? become unresponsive? slows down?

What does the errpt say?

The ctrmc subsystem is one of the leftovers PSSP, the middleware of the (*sigh*, those were the times) SP/2. It will be necessary for CSM, HACMP or if the machine is connected to a HMC. If this is not the case you could remove it by "rmcctrl" but consult the manpages of rmcctrl first!

If you want to configure the RMC it is done by the command csmconfig, but read this link (Understanding RMC and resource managers) first. There is a configuration file (/var/ct/cfg/ctrmc.acls) you could have look at and a command (chrmcacl) to modify that file.

I hope this helps.

bakunin
# 3  
Old 10-29-2007
Hang :
> the logic of the boot process stops at "The ctrmc Subsystem has been started. Subsystem PID is xxxxxx.." The behaviour is UNRESPONSIVE even i wait for 30mins.

I dont know how to gather this.
Quote:
What does the errpt say?
It is possible to access the ">" prompt once the system becomes non-responsive such like in solaris. if it is: what it the kestroke to it (solaris - stop + A )

Please be informed that I am new to AIX, kindly consider my unfamiliarity with the system.

Thanks,
aixxia
# 4  
Old 10-29-2007
Maintenance mode
I read some AIX related online docs and advise to proceed to single user mode and finally i got the errpt.

# errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BE0A03E5 0811092871 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0811092671 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0811092371 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0811092471 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0809084471 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0809084271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0809084171 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0809084171 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0809084171 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0809084371 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0809083271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0809083171 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0809083071 I O RMCdaemon The daemon is started.
BE0A03E5 0809082171 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0809082171 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0809082271 T O errdemon ERROR LOGGING TURNED ON
192AC071 0808161371 T O errdemon ERROR LOGGING TURNED OFF
BE0A03E5 0808140271 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0808135971 P H sysplanar0 UNDETERMINED ERROR
BA431EB7 0808135971 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0808140171 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0807162371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154471 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807154471 I O RMCdaemon The daemon is started.
BE0A03E5 0807153571 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807153571 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807153671 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807153071 T O errdemon ERROR LOGGING TURNED OFF
BFE4C025 0807152671 P H sysplanar0 UNDETERMINED ERROR
BE0A03E5 0807152471 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0807152171 P H sysplanar0 ENVIRONMENTAL PROBLEM
E18E984F 0807152171 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0807152371 T O errdemon ERROR LOGGING TURNED ON
E18E984F 0807151371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807151371 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807151371 I O RMCdaemon The daemon is started.
BFE4C025 0807151171 P H sysplanar0 UNDETERMINED ERROR
1BA7DF4E 0807151171 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0807151271 T O errdemon ERROR LOGGING TURNED ON
1BA7DF4E 0807135771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807135771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807135771 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807135771 I O RMCdaemon The daemon is started.
BE0A03E5 0807135571 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807135571 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807135671 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807135071 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807125271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807125071 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0807124871 P H sysplanar0 UNDETERMINED ERROR
BFE4C025 0807124671 P H sysplanar0 UNDETERMINED ERROR
BE0A03E5 0807124471 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6D1BD62 0807124171 I H Firmware Event
F48137AC 0807124171 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807124171 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807125271 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0807121771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807121671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807121571 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807121471 I O RMCdaemon The daemon is started.
BE0A03E5 0807120671 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807120671 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807120771 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807120071 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807115171 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807114871 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807114871 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807114871 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807115171 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807113271 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0807112971 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807112971 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807112971 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807112971 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807113071 T O errdemon ERROR LOGGING TURNED ON
BFE4C025 0807112171 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0807112271 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807111271 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807111271 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807111371 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807110771 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807100071 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807095971 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807095971 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807095971 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807100071 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807094271 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0807093971 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0807094071 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806162871 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0806162871 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0806162971 T O errdemon ERROR LOGGING TURNED ON
192AC071 0806162371 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0806155171 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0806155071 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0806155071 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806155071 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806155271 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806153071 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806152871 I O RMCdaemon The daemon is started.
67145A39 0806152771 U S SYSDUMP SYSTEM DUMP
BFE4C025 0806152771 P H sysplanar0 UNDETERMINED ERROR
F48137AC 0806152771 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806152771 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806152871 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806151971 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806151771 I O RMCdaemon The daemon is started.
BFE4C025 0806151671 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0806151671 I O RMCdaemon The daemon is started.
9DBCFDEE 0806151771 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0806141571 I O RMCdaemon The daemon is started.
67145A39 0806141471 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0806141471 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0806141471 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806141471 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806141571 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806135471 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806135271 I O RMCdaemon The daemon is started.
BFE4C025 0806135171 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0806135171 I O RMCdaemon The daemon is started.
9DBCFDEE 0806135271 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0806122571 I O RMCdaemon The daemon is started.
BE0A03E5 0806122371 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0806122371 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0806122471 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0731175571 I O RMCdaemon The daemon is started.
CF8CADB6 0731175371 P H sysplanar0 SYSTEM FAILURE WITH SCAN DATA
BE0A03E5 0731175271 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0731175271 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0731175371 T O errdemon ERROR LOGGING TURNED ON
#
# 5  
Old 10-30-2007
I don't know for sure but this seems like it has nothing to do with the ctrmc subsystem. This seems to be the culprit:

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BE0A03E5 0811092871 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0811092671 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0811092371 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0811092471 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0809084471 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0809084271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0809084171 P H sysplanar0 ENVIRONMENTAL PROBLEM

Boot again to single user mode, and issue "errpt -a | more" to see the details. Each of these entries is only the headline to a problem, "errpt -a" will give you the full text.

Without wanting to rush to conclusions this looks more lika problem suited for the technician than one suited for system administrators. "sysplanar0" is the system planar, something like the "motherboard" in PCs. This thing failing is usually pretty bad news from the hardware.

bakunin
# 6  
Old 10-30-2007
We got that too and it was due to either power or an overheating room Smilie
# 7  
Old 10-21-2008
CF8CADB6 sysplanar0 SYSTEM FAILURE WITH SCAN DATA
We are having the same problem-we have replaced the sysplaner board(s) and the issue is still happening..
os level is 4.3
are you using SAN attached storage?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

Solaris 10 - hangs during boot

Power server up runs through diagnostics. Goes to the boot section and then the bit where the cursor is spinning and thats it. Hangs completely. Any ideas? (10 Replies)
Discussion started by: psychocandy
10 Replies

2. Solaris

Server Hangs at boot

Hi Guys, I have a SunBlade 1500 which I use to practice at home. Suddenly when I power on the server it just displays the solaris banner and Boot device: disk File and args: Then it stops and says " rebooting with command: boot" It doesn't boot to the kernel. Is there anyway I can possibly... (5 Replies)
Discussion started by: cjashu
5 Replies

3. AIX

AIX 5.3 Boot Hangs On Power4 Hardware

Hi all, We have a old Power4 server with AIX version 5.3 installed. Upon boot up, it hangs right after the AIX welcome screen. On the diagnostic load progress LED indicator, I saw that it stopped at '0c32' then after awhile the LED would be blank. The screen on the monitor will hang at the black... (15 Replies)
Discussion started by: a_sim
15 Replies

4. AIX

AIX 6.1 rogue process starts at boot up

We are having a bit of trouble finding where the following process is being started from at bootup. /opt/IBM/tdsV6.2db2/itma/aix526/ud/bin/kuddb2 db2tdsWe have looked in the typical areas /etc/inittab & /etc/rc.d but have had no luck finding it, any ideas? (10 Replies)
Discussion started by: j_aix
10 Replies

5. Ubuntu

Boot hangs while adding swap

Boot process hangs after reaching the following line: Adding 5853176k swap on /dev/mapper/mch-swap_1. Priority:-1 extents:1 across:5853176kUsing knoppix 6.2 as LiveCD, and mounting the partitions, I got the following: # /etc/fstab: static file system information. # # Use 'blkid -o value -s... (0 Replies)
Discussion started by: fzaker
0 Replies

6. AIX

boot process stuck on "Welcome to AIX" banner

Good afternoon, I'm currently working on an IBM 9110-510, with an AIX 6.1 on it. Currently, I've put an HyperTerminal on the machine vty0. Folowing the procedure given by a support guy(§details), I'm able to get to the AIX banner ... (6 Replies)
Discussion started by: nilleb
6 Replies

7. AIX

AIX hangs on boot

Hi guys, Got a tiny bit of a problem on one of our servers here at work. I installed AIX 5.3 on the server and that went fine. Then I upgraded to ML 5300-05-01, but now the system just hangs on boot. The system starts up and the monitor comes on in few minutes, but a few seconds later when... (6 Replies)
Discussion started by: drchris
6 Replies

8. Solaris

Boot hangs up on solaris

Hi all, I have a SUN server Sun-Fire-V890 running solaris 9, with a remote system console . Both the IP adresses of the server and the console are in a private network (address 10.67.xxx). For some reasons I need to give the access to the server from outside address 194.xxx, but I prefer that... (2 Replies)
Discussion started by: aribault
2 Replies

9. AIX

RS6K B50 Hangs at E140 on Boot

I've got an RS6K B50, and i've installed AIX 5.3, TL5. It installs fine, the only customisations i make to the BOS are to change the locale / keyboard layout to UK, and to choose not to install graphical software. I have no video card in this box, and i connect via a serial console (null modem... (4 Replies)
Discussion started by: craigp84
4 Replies

10. Shell Programming and Scripting

nohup process hangs

Hi All, I tried searching for this, but I have yet to find anything useful. So here goes, if a script executed from another script with nohup & hangs, does it affect the parent script? Reason I ask, we have a windows box with NFS, and we use it to store some of our files. Currently, I mount the... (2 Replies)
Discussion started by: Sully
2 Replies
Login or Register to Ask a Question