Programming

This is all a bit new to me, but I'm coming up to speed. You are almost certainly correct that you have a gettimeofday() problem and the multi-core environment is to blame.

I was congratulating myself for noticing that you are alternating between 2 different values. Too bad I didn't also notice that they were over 1 hour apart. Two cpu's are fighting over which second it is. Neither tried to increment the second because all of the calls you show occurred during the same second.

What OS are you using? If linux, please tell us the distro and the kernel rev level (use uname -a to find out). Also what cpu do you have? Workstation or laptop?

Some background: Intel introduced a feature called TSM (time stamp counter). It gets incremented with each CPU cycle. AMD copied the idea. Linux (and maybe other unix versions too?) is using this to improve the accuracy of the clock. It worked great for a while but stuff like hibernation and speed-step screwed it up for laptops. Speed-step is now used for workstations (reduce power consumption). And multi-cores really hurt it.

OS: red hat 4
kernel: 2.6.18-92.el5PAE
CPU: Intel(R) Xeon(R) CPU X7350 @ 2.93GHz
16 cores.

The magic number 4389 is my concern. It occurs every time when this problem arises.
Therefore, I try to do some search in the internet and found that website.

Since it may be the kernel issue, I have no idea how to fix. As the website mentioned the cause may be the clock source if it is TSC. However, the clock source I use is HPET.

How do you know that you're using HPET? Regardless, you have a kernel bug that has been discussed quite a bit recently and yet you are using an older release of RedHat. Upgrading would be the thing I would try. Once you're on the current (or at least recent) release, if you still have a problem, try opening a call to RedHat support.

my colleague told me about it. I found redhat provides a patch. I hope it can solve it.