HP-UX

We have a site which uses "at" extensively to submit background tasks. From time to time, we noticed that some jobs were not executing. After some research, we noticed that "at now" would assign the same jobid. Since the jobid is apparently the epoch time, this would happen if 2 (or more) "at now" commands were entered at precisely the same second. So, one job would override (overwrite) the other.

It seems like adding a time offset e.g. "at now +1 minutes" will (or is less likely to) generate unique ids but this would introduce 1-minute delay for almost every job submitted. Since we use this heavily, these delays could quickly add up.

We could also switch from "at now" to "batch" but this would also change the cron queue from "a" to "b". This may have an impact on the current application.

I have seen the MULTI_JOB_SUPPORT option in some versions of HP-UX (PA-RISC 11.11) but it doesn't seem to be available on this particular system (Itanium 11.23). But, even if it were available, the jobid would have a slightly different format.

So, my questions are:
1) Is "at now" supposed to generate unique ids? In other words, is this by design or is it a bug?
2) Is adding an offset really generate unique ids or does it simply reduce the likelihood? In other words, the problem may still occur but much less frequently.

Of course, I'll get in touch with HP Support and ask the question.

Any ideas/suggestions?

On our HPUX box job-id == jobnumber.queuename, job number is epoch time.
You can create any number of queues, one for each system user if you really need it.

Code:

at now -q [queuename] [command]

Does this not work?

This behavior is a bug. The job number is part of a file name which gets created in the AT directory. When the number is allocated, the file should be opened with O_CREAT. If this succeeds, cool. If it fails the job number should be incremented and the open retried. After a few retries, it should give up and tell the user that the queue is full. Creating a unique file name is atomic and has been used to syncronize unrelated unix processes for decades.

See
man 4 queuedefs

The default "at" queue is limited to 4 running jobs.
The man pages suggest that "at" jobs will queue until there is a free slot. In my experience there is a limit to the size of this queue of about 10 jobs before "at" ignores new jobs.

The "at" command must be seriously broken on recent versions of HP-UX if that is true... but I have no access to an HP-UX system to verify that. It should never "ignore" jobs. It should either schedule them or deliver an error message to the user. The size of the queue should really be limited by disk space and ten would be an incredibly small limit. Only 4 jobs can run simultaneously provided that you accept the defaults. queuedefs can be edited though.

Jim, we already use different queues for different job types.

methyl, we use multiple job queues and customized queuedefs for our needs. It works...most of the times.

Perderabo, cron doesn't ignore/bypass jobs. It executes all the scheduled jobs. Our problem is we have 2 jobs "sharing" the same jobid. Only one effectively gets executed.

Here's my guess: cron has an internal schedule. When time comes, it goes out and executes the atjobs file, whatever happens to be in there at that time. The atjobs file is then replaced with something else. cron does not detect that, moves on to the next and never looks back. I don't have any documentation to back this up. This is based on what I observed so I may be totally off the track.

In my case, this translates into:

JobA is scheduled and is assigned atjobs/12345678.a (JobA actually gets copied into 12345678.a)
cron sees the job and launches the process. It's removed from the internal list.
JobB is scheduled and is also assigned atjobs/12345678.a. It gets copied to 12345678.a but cron is gone already.