The Enterprise Manager log for the MOM is reporting clock skew errors for a collector like the following:
5/13/13 11:13:17.402 AM CEST [WARN] [Collector [email protected]]
[Manager.Cluster] Collector clock is too far skewed from MOM. Collector clock
is skewed from MOM clock by 7,735 ms. The maximum allowed skew is 3,000 ms.
Please change the system clock on the collector EM.
But when the system clock on the offending EM is checked, it shows the same time as the MOM clock.
All APM releases
If the last update time of the IntroscopeEnterpriseManager.log file on the collector does not match the timestamp of the last entry, check if there is a genuine discrepancy in the time the JVM reports in the log compared to the system time.
A simple script can be run to put a timestamp into the log file approximately once a second in parallel with the normal EM logging, as follows:
$ while true
> date >> IntroscopeEnterpriseManager.log
> sleep 1
The output produced may show that the time reported by the JVM is not in sync with the time reported by the date command (which uses the system clock).
Wed May 29 13:06:45 METDST 2013
Wed May 29 13:06:46 METDST 2013
Wed May 29 13:06:47 METDST 2013
5/29/13 01:06:30.192 PM CEST [INFO] [PO Route Down Executor]
[Manager.Cluster] MOM Introscope Enterprise Manager disconnected
Wed May 29 13:06:48 METDST 2013
Wed May 29 13:06:49 METDST 2013
In the instance reported, the Java VM version in use was "Java HotSpot(TM) Server VM 1.6.0.06" from Hewlett-Packard Co.
Research showed that time sync problems have been reported on this Java platform in HP IA64 environments.
To correct this, use the JVM option -XX:+UseGetTimeOfDay
This option can be defined by including it in the value defined for the property lax.nl.java.option.additional in the IntroscopeEnetrpriseManager.lax file.