The MOM reports clock skew errors for a collector, but the system clocks are in sync.

book

Article ID: 4029

calendar_today

Updated On:

Products

APP PERF MANAGEMENT CA Application Performance Management Agent (APM / Wily / Introscope) CUSTOMER EXPERIENCE MANAGER INTROSCOPE

Issue/Introduction

 The Enterprise Manager log for the MOM is reporting clock skew errors for a collector like the following:

5/13/13 11:13:17.402 AM CEST [WARN] [Collector [email protected]
[Manager.Cluster] Collector clock is too far skewed from MOM. Collector clock 
is skewed from MOM clock by 7,735 ms. The maximum allowed skew is 3,000 ms. 
Please change the system clock on the collector EM.

 But when the system clock on the offending EM is checked, it shows the same time as the MOM clock.

Environment

All APM releases

Resolution

 If the last update time of the IntroscopeEnterpriseManager.log file on the collector does not match the timestamp of the last entry, check if there is a genuine discrepancy in the time the JVM reports in the log compared to the system time.

A simple script can be run to put a timestamp into the log file approximately once a second in parallel with the normal EM logging, as follows:

$ while true 
> do 
> date >> IntroscopeEnterpriseManager.log 
> sleep 1 
> done

The output produced may show that the time reported by the JVM is not in sync with the time reported by the date command (which uses the system clock).

Wed May 29 13:06:45 METDST 2013 
Wed May 29 13:06:46 METDST 2013 
Wed May 29 13:06:47 METDST 2013 
5/29/13 01:06:30.192 PM CEST [INFO] [PO Route Down Executor] 
[Manager.Cluster] MOM Introscope Enterprise Manager disconnected 
Wed May 29 13:06:48 METDST 2013 
Wed May 29 13:06:49 METDST 2013

In the instance reported, the Java VM version in use was "Java HotSpot(TM) Server VM 1.6.0.06" from Hewlett-Packard Co.

Research showed that time sync problems have been reported on this Java platform in HP IA64 environments.

To correct this, use the JVM option -XX:+UseGetTimeOfDay

This option can be defined by including it in the value defined for the property lax.nl.java.option.additional in the IntroscopeEnetrpriseManager.lax file.