vSphere Replication fails with "Broker or host clocks may be out of sync" errors

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• vSphere Replication fails for multiple virtual machines, resulting in a Not Active (RPO Violation) status.The replication task details show the following error
  
  "No connection to VR Server for virtual machine on host."
  

• Attempting to configure replication or run a mapping test fails with below error 

  Fault occurred while performing health check. Details: "Login Denied. Broker or host clocks may be put of sync".

  

• While running test for VMware Live Site Recovery 9.0.3 from Site Recover page > Configure > vSphere Replication > Replication Mappings, fails with below error:

  "The source host (id: 'host-###', name: '###.###.#.###') successfully connected to both the target broker '###.###.#.###' and the target host (id: 'host-##', name: '###.###.#.###'), but the login to the target host '###.###.#.###' failed. The server mappings might not have been updated for the source host '###.###.#.###' on the target broker '1###.###.#.###', or the target host's '###.###.#.###' certificate might have expired. The target host '###.###.#.###' might not have yet received the certificate from the target broker '###.###.#.###' or the broker's certificate on the target host '###.###.#.###' might have expired."

  

Environment

vSphere Replication 9.x

vSphere Replication 8.x

Cause

• This issue occurs due to time synchronization discrepancies between the vSphere replication appliances, vCenter Server, and ESXi hosts. This time drift prevents successful communication with the vSphere Replication service, leading to replication configuration task failures
  
  Following events confirms host clocks may be out of sync

  Validate the vmkernel.log file (less /var/run/log/vmkernel.log) on the source ESXi host to confirm the  Failed to establish connection to VR server 
  
  YYYY-MM-DDT09:42:23.322Z Wa(180) vmkwarning: cpu22:29907991)WARNING: Hbr: 788: Failed to receive from ###.#.#.# (groupID=GID-#####): Broken pipe
YYYY-MM-DDT09:42:23.322Z Wa(180) vmkwarning: cpu22:29907991)WARNING: Hbr: 2542: Failed to receive extended handshake response
YYYY-MM-DDT09:42:23.322Z Wa(180) vmkwarning: cpu22:29907991)WARNING: Hbr: 5362: Failed to establish connection to [###.#.#.#]:32032 (groupID=GID-#####): Broken pipe

  Validate the hbr-agent.log file (less /var/run/log/hbr-agent.log) on the source ESXi host to confirm the unsuccessful login to the brokered server.

  
  YYYY-MM-DDT09:54:23.341Z In(166) hbr-agent-bin[2100560]: [0x0000003ace2c5700] error: [Proxy [Group: GID-#####] -> [#####.63:32032]] Failed to login to brokered server additional error info: Broker or host clocks may be out of sync.
YYYY-MM-DDT09:54:23.341Z In(166) hbr-agent-bin[2100560]: [0x0000003ace2c5700] error: [Proxy [Group: GID-#####] -> [#####.63:32032]] Exhausted all server endpoints reported by broker.

  
Validate the  hbrsrv.log (less /var/run/log/hbrsrv.log) on the  Destination Host to confirm the Broker or host clocks may be out of sync
  
  YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0] HbrError stack:
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]    [0] Broker or host clocks may be out of sync.
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]    [1] Unverified token with ID '#####'is not valid: TOO_NEW (8)
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]    [2] ClientConnection (client=[#####.11]:50248) failed login attempt
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]    [3] Hiding full error from unauthenticated client.
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]  HbrError stack:
YYYY-MM-DDT09:54:22.435Z Er(163) hbrsrv[2100365]: [Originator@6876 sub=Main opID=hsl-0]    [0] Broker or host clocks may be out of sync.

Resolution

For vSphere Replication to function correctly, time must be synchronized across all ESXi hosts, VMware Live Site Recovery appliances, vSphere Replication appliances, and vCenter Server appliances.

To resolve the issue, manually set the time on the VR, ESXi, and SRM appliances to match the vCenter Server.

Steps to synchronize the Date and Time manually:

1. Verify the current time: Log in to the vCenter Server appliance via SSH and check the current time to use as your baseline. Repeat this command on the other appliances to identify the drift.

     watch date

2. Set the time manually: Log in to the affected SRM/VR appliances via SSH and run the following command, replacing the string with the correct vCenter time.

   date --set="YY-MM-DD HH:MM:SS"

3.Restart the NTP Services: After the time is manually corrected, restart the NTP service to ensure it takes over and maintains synchronization. Run the following commands in sequence

• service ntpd stop
• service ntpd start
• service ntpd status

Additional Information

For the following error, you can try the following steps to fix :

Symptoms:

• Suddenly, vSphere Replication fails for multiple VMs with status Not Active (RPO Violation)
• Enhanced replication mapping error:
  "The source host (id: 'host-###', name: '###.###.#.###') successfully connected to both the target broker '###.###.#.###' and the target host (id: 'host-##', name: '###.###.#.###'), but the login to the target host '###.###.#.###' failed. The server mappings might not have been updated for the source host '###.###.#.###' on the target broker '1###.###.#.###', or the target host's '###.###.#.###' certificate might have expired. The target host '###.###.#.###' might not have yet received the certificate from the target broker '###.###.#.###' or the broker's certificate on the target host '###.###.#.###' might have expired."
• Last change vCenter/ESXi upgrade

Resolution :

• Reconfigure the appliances

• Reconnect the site pairing