SOX, uggh! best describes it.

We have the following for jobs that are repeated periodically:
a job in root cron on each remote node that runs once per (U pick it) minute interval. It reads a command file. The file format is something like this:
The file lives in a protected sftp only folder. The 1 refers to #1 in a list of commands known to the cron script.

The cron script decodes the one, adds the arguments. It then goes thru submitting (at now) each job request in the file. The command file is then zeroed. The cron job keeps a log of when & what was submitted.

The 'control panel' is on a single box. It's job is to build the command request files and scp them to the remote box(es) using the protected account. It keeps track of pending requests so that an unprocessed request file on a remote node does not get overwritten.
There is also an 'adhoc' request screen.

In practice the command file sent to remote boxes is encrypted. On a per remote box basis. And there are validation fields as well. This is to prevent spoofing.
We have 400+ Linux servers, 20+ HP-UX boxes and some oddballs.