Error : "The object has already been deleted or has not been completely created" observed on reconnecting VR server

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

1. Replication server is disconnected in Site Recovery UI.

2. VM replications show the error - No host can be used to access datastore path.

3. Virtual machines in the replication tab shows Error(RPO) at one site and OK on the other site. 

4. Recent tasks in Site Recovery UI continuously logs following error:

"The object 'vim.VirtualMachine:vm-###' has already been deleted or has not been completely created"

5. VRMS or VR add-on servers has been upgraded.

Environment

VMware vSphere Replication 8.X

VMware vSphere Replication 9.X

Cause

Issue occurs due to one of the following reasons:

1. This happens when a certificate is changed in vSphere Replication appliance VAMI. The appliance is normally supposed to update the certificate in the database but this process fails at times leading to this problem. You can also hit this problem after performing appliance upgrades. 
   
   
2. IP address of management (eth0) or replication (eth1) virtual NICs on add-on VR server is replaced. SSL certificate consists of the 'Subject Alternative Name' old IP which renders the VR stay disconnected.

Resolution

1. If IP has been changed for the add-on VR server, regenerate the Certificate of the add-on VR and replace the old IP by the new one:
   
   Refer : Change the SSL Certificate of the vSphere Replication Appliance or Server
   
   
2. Take snapshot of the disconnected VR server (embedded or add-on).
   
   
3. The hms & hbrsrv thumbprint must be checked and updated. Run following command to check if these thumbprints are updated and matching:
   # /usr/bin/hbrsrv-replicainfo.sh | grep -i thumb
   
   (Note : If the above command is run on add-on VR server, the hms thumbprint is that of embedded VR appliance while hbrsrv thumbprint is that of add-on VR server)
   
   For example:
   
   guestinfo-cache/guestinfo.hbr.hms-thumbprint 41:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
guestinfo-cache/guestinfo.hbr.hbrsrv-thumbpr 6D:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
   
   
4. The hbrserverentity table in VRMS database on embedded VR appliance is to be updated with the new certificate and thumbprint (and IP address if changed) for the embedded hbrsrv or the add-ons. 
   
   If replication is at same site, hbrserverinfoentity table needs to be modified as well.
   
   (Note : Ensure hms service is stopped before modifying these table/s. Command : systemctl stop hms) 
   
   
   

Please raise a case with Broadcom for any assistance needed in implementing the above stated action plan.