Time related mismatch errors in Site Recovery nodes can cause multiple operational failures.

Products

VMware Live Recovery

Issue/Introduction

VMware Site Recovery documentation recommends synchronizing the time across all nodes in a site pair (i.e., vCenter Server, Platform Services Controller, vSphere Replication, and Site Recovery Manager appliances). You can achieve this by either using the time settings of the ESXi host where the appliance resides, or by configuring time synchronization with an NTP server.

Configure the Time Zone and Time Synchronization Settings for the Site Recovery Manager Appliance

Configure the Time Zone and Time Synchronization Settings for the vSphere Replication Appliance

A time discrepancy of even 5 to 30 seconds between nodes can lead to communication issues, misconfigurations, or basic operational failures.

Specifically, timestamp skew or mismatches can result in the following issues:

Failure to configure the SRM or vSphere Replication appliance in the VAMI wizard:
- "Failed to register VRMS..."
- "Failed to connect to Lookup Service..."
SRM or vSphere Replication service startup failures:
- "Failed trying to retrieve token: ns0:RequestFailed:"
Operational recovery task failures:
- "Unable to retrieve pairs from extension server..."
- "Cannot complete customization..."
Failures when collecting logs from SRM or VR appliances:
- "Invalid message timestamp, created time is in the future"

Environment

Site Recovery Manager 8.x

vSphere Replication 8.x

VMware Live Site Recovery 9.0.2

Cause

Communication between multiple nodes in a Site Recovery system requires certificate/token exchange. Verification of the certificate/token relies on a 'time to live' value associated with the creation timestamp. If the timestamp is mismatched or skewed, the task receiving the certificate/token can fail, citing certificate/token as expired/invalid due to the time difference.

Resolution

Ensure all nodes (vCenter Server, Platform Services Controller, Site Recovery Manager, and vSphere Replication) are synchronized within a maximum of 5 seconds of each other. Verify that the time and NTP configurations are correct across all appliances.

To verify and correct the timestamps across the environment, perform the following steps:

Open an SSH session (e.g., using PuTTY) to all vCenter Server, SRM, and vSphere Replication nodes simultaneously.
Run the following command on all nodes to monitor the current UTC time:
watch -d date -u
Compare the outputs. If any node is out of sync by more than 5 seconds, investigate and correct the time source for that specific node. Standard troubleshooting steps include:
- Checking and correcting the NTP server IP address or FQDN parameters.
- Restarting the NTP daemon on the affected node.
- Verifying the time configuration of the underlying ESXi host where the virtual machine resides.
Continue to monitor the output of the watch command simultaneously across all sessions to confirm the timestamps have synchronized and remain stable.

Additional Information

If certificates have been changed, we would recommend checking that NTP is matching across all nodes and also to perfom a reconfigure of replication based appliances

The time skew issue between the vCenter and SRM was causing the plugin issue to throw error "Unable to retrieve Site Recovery Manager summary data." https://knowledge.broadcom.com/external/article?articleNumber=376950

Configuring vCenter Server to use a Network Time Protocol (NTP) server - https://knowledge.broadcom.com/external/article?articleNumber=313945

SRM login to remote site fails with error: Failed trying to retrieve token: ns0:RequestFailed - https://knowledge.broadcom.com/external/article?articleNumber=319353

Site Recovery Manager or vSphere Replication cannot complete a site pair operation. The received single sign-on token is valid from XX to YY - https://knowledge.broadcom.com/external/article?articleNumber=312750

How to address error: Clock skew too great - https://knowledge.broadcom.com/external/article?articleNumber=160526