GemFire Error: "New Cache Time Offset Calculated is off more than "XXX" ms from Earlier Offset

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:

The following warning entry is observed in the server logs:

[warning 2016/10/31 11:57:17.496 JST tid=0x1e] New cache time offset calculated is off more than 212 ms from earlier offset.

Environment

Cause

GemFire internally has a distributed clock synchronization implementation. When membership views change, this distributed clock synchronization sends a request for the current MilliSecond clock to all members at a configurable interval. It computes the average round-trip time and the average clock value, throwing out samples outside the standard deviation. The result is used to compute and send clock offsets to each member.

Typically, this warning message is harmless and it can be ignored. But it might indicate a CPU overload or high network latency between the members if the offset is higher than 1000 MilliSeconds.

The current GemFire distributed clock synchronization, in Gemfire6.x to 8.x, is implemented based on a Berkeley Algorithm for clock synchronization. GemFire 9.x will replace this implementation with a new clock synchronization service and we can expect to see this message less often. 

Resolution

You can dismiss the error message if it is only in MilliSecond unit. If the offset is in seconds, then check the GemFire statistics for network latency or high CPU utilization.

GemFire 9.x will implement a new clock synchronization service replacing the current GemFire distributed clock synchronization, which is expected to decrease the occurrence of the above message.