so i want to monitor a variety of things on hundreds of servers. the old process was to have an agent running on each one of these servers.

but now i'm looking to see if its possible to have agentless monitoring. the only other straight forward option other than having an agent on the hosts, is to use ssh. but i dont know how to go about this.

I have 2 hosts that will serve as the monitoring servers. One will be Primary, the second will be Backup. the secondary will have the same exact setup as the primary.

what do i need to do on these two hosts to be able to log into all my servers and run the commands i need to run? vmstat, uptime, mpstat etc.

The number of servers i need to monitor is 600.
The OS running on these servers is a mixture of Sun Solaris and Linux Red Hat.

Since you did not specify what you are trying to monitor, I wanted to think of this differently...
There are 600 computers you want to analyze. So, if it take 60 seconds to connect, execute remote commands, transmit results file back, do something (on master computer), and then go to next remote computer.
That would mean ten hours, start to finish, if done sequentially. I suppose you could write something to do some in parallel?

Which brings me back to my initial premise...that this may not be possible based on the number of remote computers to analyze.

In human terms...
(a) you call 600 people to find out if they are coming for dinner
(b) receive emails of those who cannot attend

Probably easier to process the (b) list. Likewise, easier to process error files from remote computers.

It all works the same. You have to have the SAME username on all 602 boxes, two monitoring boxes, 600 monitor-ees. Likewise everybody needs 602 home directories with an .ssh subdirectory, that has protections 600. The home directory for each has to be 755 or 751 or even 700. The perms cannot be 77[n].

You can also have a mishmash of usernames if you want, but keeping track and matching everything up is a nightmare.

The account on each remote box has to have perms to run your commands.

1. create ssh-keys on both servers, place the keys from each server on the remotes.
This is 1200 cp operations, assuming you have accounts and home directories and .ssh directories set up.

You likely already have infrastructure existing to do this - like an ftp server for everybody.

2. Run your monitoring script using ssh from one of the two monitor boxes.

ssh remotenodename 'mpstat; iostat -n'  >> /logs/remoteserver.log