Most *NIX systems (AIX, Linux, Solaris, BSD) have some kind of system and accounting records. You can run

Code:

sar

to see if it is properly deployed on your system. If you run it and get loads of output, you may be in luck. To use it, refer to the man pages. Typically you want to check options for memory and swap usage, CPU usage, and I/O activity.

If it's not installed, consider deploying this first before installing some complex monitoring software; it's a very standard unix utility that has been around for ages, but the implementation and features vary from platform to platform. For Linux install the sysstat package.

On most systems, sar's data is collected through another program which is run as a cronjob. On a typical RedHat/CentOS Linux system, you will find /etc/cron.d/sysstat to contain:

Code:

* * * * * root /usr/lib64/sa/sa1 -S XALL 1 1

which I immediately change to

Code:

*/5 * * * * root /usr/lib64/sa/sa1 -L -S XALL 10 30

The original form collects data once per minute, which is often simply not enough granularity to get a feel for rapid changes to the system, the kind that cause instability and crashes. Also, if memory becomes extremely sparse, cron might not be able to spawn the job every minute.

My form, however, spawns a new job every 5 minutes. It writes 30 records, one every 10 seconds. The corresponding reports contain enough detail to know very precisely when the problem started. You will need an additional 1.5 GB of disk space on /var/log if you do this.

If you want graphs and pretty output, you may be able to export the data into graphing engines or spreadsheets. Linux's sar has such a program (sadf), and other related projects can slurp of the data and present graphs.