AIX

Hello,

I tried searching for the solution on the google...nothing so far...

anyone know anything about this.

Broadcast message from root@pr2serv (tty) at 00:01:04 ...
Cluster Verification detected cluster configuration errors on node pr1serv. Detailed clverify output is available in file /var/hacmp/log/autoclstrcfgmonitor.out on node pr1serv.


more autoclstrcfgmonitor.out
autoclstrcfgmonitor.out: A file or directory in the path name does not exist.
# pwd
/var/hacmp/log

ls -ltr hacmp.out
-rw-r--r-- 1 root system 0 Feb 11 00:00 hacmp.out

At pr2serv
errpt

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
4BE53A52 0227150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0226150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0225150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0224150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0223150009 P O dumpcheck Livedump filesystem almost full
F7FA22C9 0223000409 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
4BE53A52 0222150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0221150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0220150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0219150009 P O dumpcheck Livedump filesystem almost full
F7FA22C9 0219000509 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
4BE53A52 0218150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0217150009 P O dumpcheck Livedump filesystem almost full
4BE53A52 0216150009 P O dumpcheck Livedump filesystem almost full
A924A5FC 0216123609 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
A924A5FC 0216122809 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
A924A5FC 0216122309 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
A924A5FC 0216122209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED

Anything relevant in clstmgr.out - usually located in /tmp

No clstmgr.out exists in /tmp

any other solutions ?

thanks

You use /var as filesystem(*) for cluster logs which filled this filesystem to 100% subsequentally leading to no space being available for autoclstrcfgmonitor.out. You are lucky that that node did not crash completely as AIX is not amused when either of /, /var and /tmp is filled a 100%. It will prevent your node from booting or might crash it during operation. Hence create a separate filesystem, size it adequately (start with 512M) mount it on /var/hacmp/log and make ALL cluster logs use that (with cllog command or from CSPOC).

(*) Using /var for logs is really stupid however that is not your fault but IBM's. They should know better but are using /tmp and /var as a default filesystem for clusterlogs for ages now.

Hello
On All the nodes /var is less then 80% utilized , anyway I will increase the size and see if the problem occurs again.

Thanks for the help

Sure. Some logs will be rotated, some open files will dissappear when a process exits. So you might end up with a file usage of less than 100% eventually. However that usually happens after you discoverd a problem.
Secondly: Not sure whether you got me right - increasing the size of /var will just lead to the problem turning up later. If you use /var for logs after you have been warned ....

I understood that I should shift the logs to another filesystem. I just want to be monitor how the log file is increasing and at what point ? During startup or shutdown ... etc..

Thanks for the advice.

I will post here the results..


Thanks once again.