Initial or incremental sync fails after adding new (dedicated) vmkernel adapter for Replication traffic

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Initial or incremental sync fails for certain VMs on configuring dedicated vmkernel adapter for Replication & Replication NFC traffic after initial use of Management VMkernel adapter (vmk0).
  
  
• Following error observed for VM replication in Site Recovery UI:
  
  "A replication error occurred at the vSphere Replication Server for replication 'VM_Name'. Details: 'No connection to VR Server for virtual machine VM_Name on host Host_FQDN in cluster Source_Cluster in Source_DC Resources: Unknown'.

Environment

vSphere Replication 9.0.2

Cause

• Issue occurs as Source ESXi that hosts VMs with errored replications tries to connect to Management vmkernel adapter of target ESXi hosts used earlier for Replication and Replication NFC traffic.
  
  

• Connection from Source ESXi to target ESXi host's fails and reports following errors as it is trying to connect to Management vmkernel adapter (vmk0):
  
  (from /var/run/log/hbr-agent.log)

  YYYY-MM-DDTHH:MM:SS.553Z In(166) hbr-agent-bin[#######] : [0x000000423a7fa700]  info: [Proxy [Group: GID-########-####-####-####-##############]  -> [#.#.#.#:32032] ] Setting up secure tunnel to brokered server #.#.#.#:32032 (1 of 1)

  YYYY-MM-DDTHH:MM:SS.565Z In(166) hbr-agent-bin[#######] : [0x000000423a6f8700] error: [Proxy [Group: GID-########-####-####-####-##############] -> > [#.#.#.#:32032]] Failed to connect to #.#.#.#:32032. Using nic 'vmk0'.

  YYYY-MM-DDTHH:MM:SS.273Z> [1#.#.#.#:32032]] Failed to connect to #.#.#.#:32032. Using nic 'vmk0'. error: [Proxy [Group: GID-########-####-####-####-##############] -> [#.#.#.#:32032]] Exhausted all server endpoints reported by broker.



• This in-turn is because VRMS database of Source VR appliance does not have replication traffic address listed in hbrserverentity table and thus default vmk0 is taken.
  
  
• Here is sample output of hbrserverentity table in VRMS database on Source VR appliance that shows Target ESXi hosts with no Replication traffic address populated after change of vmkernel adapter.
  
  vrmsdb=# select vsrv address, hbrservername, replicationtrafficaddress from hbrserverentity;

vsrv_address    hbrservername    replicationtrafficaddress
10.#.100.#1       10.#.100.#1         10.#.200.#1
10.#.100.#2       10.#.100.#2         10.#.200.#2
10.#.100.#3       10.#.100.#3                                  -----> No Replication traffic address
10.#.100.#4       10.#.100.#4         10.#.200.#4
10.#.100.#5       10.#.100.#5                                  -----> No Replication traffic address
  
  
• Due to this VMs at Source site report issue connecting to VR server.

Resolution

1. Restart hbr-agent and hbrsrv service on all hosts with missing entries in VRMS database.
   
   (In vSphere Client > Select the ESXi host in question > Configure > Services > Select above stated two services > Click on "Restart")
   
   
2. Remove and re-add Not-active replications in Site Recovery UI.
   
   (Open Site Recovery UI > Under 'Replications' > Select 'Not-Active' VM > 'Remove') 
   
   (Note : Please do not retain seeds)
   
   
3. Re-add the VM replication