Attempting to remove a disconnected ESXi host from the vCenter inventory fails due to DVS Port information leftovers for a template VM

Products

• VMware vCenter Server

Issue/Introduction

• The ESXi host remains in the inventory in the vSphere Client with a disconnected status.
• The ESXi host has been physically decommissioned and cannot be reconnected.
• Removing the disconnected ESXi host from the inventory results in an error as follows:
  Cannot synchronize host #####. An error occurred while communicating with the remote host.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

• The DVS information of the ESXi host is deleted when it is disconnected.
  [ vpxd.log ]
<time stamp> info vpxd[32695] [Originator@6876 sub=corehostCapsule opID=47e53a20] Host [[vim.HostSystem:host-####,#####]] left the DVS [## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##]
<time stamp> info vpxd[32695] [Originator@6876 sub=corehostSync opID=47e53a20] Delete port [#####:########-####-####-####-##########01] for host [#####]
<time stamp> info vpxd[32695] [Originator@6876 sub=corehostSync opID=47e53a20] Delete port [#####:########-####-####-####-##########02] for host [#####]
  
  
• A template VM that was registered to a disconnected ESXi host has a NIC connected to a DVS port.
  vCenter stores information about the target template VM in its database.
  
  # select count(1) from vc.vpx_vm_virtual_device where device_text like '%########-####-####-####-##########01%';
 count 
-------
     1
(1 row)
  
  
• As a result, when you try to remove the ESXi host from the inventory, the DVS Port information held by the template VM cannot be found, resulting in an error.
  [ vpxd.log ]
<timestamp> info vpxd[06899] [Originator@6876 sub=Default opID=m6x0xap1-67050-auto-1fqj-h5:70016454-f1] [VpxLRO] -- ERROR task-11757 -- domain-##### -- vim.ManagedEntity.destroy: vim.fault.NotFound:
--> Result:
--> (vim.fault.NotFound) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vpxd.dvs.portNotFound.label", 
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "port", 
-->                value = "########-####-####-####-##########01"
-->             }
-->          ], 
-->          message = <unset>
-->       }
-->    ]
-->    msg = ""
--> }
--> Args:
--> 

Resolution

Preparation

Before applying the steps outlined below to fix the issue, please take an offline snapshot of the vCenter Server Appliance:

1. • Shut down the vCenter Server VM and take a snapshot as a recovery point in case of an emergency.
   • If the vCenter Server Appliance is part of an Enhanced Linked Mode replication group, please ensure to take offline snapshots for all ELM nodes
   • After taking the snapshot, power the VCSA(s) back on.

Delete the relevant information from the vCenter DB

1. Once the vCenter is running, connect to the shell via SSH.
2. Stop the vpxd service with the following command. Be aware that after this step, you will be unable to use the vSphere Client or any other vCenter functions. Running virtual machines will not be affected.

   # service-control --stop vmware-vpxd

3. Connect to the vCenter database using the following command:

   # /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

4. Check the deleted records. Run the following command and confirm that two records exist (Please change ########-####-####-####-##########01 to your desired DVS Port ID):

   # \x
   # select * from vc.vpx_vm_virtual_device where device_text like '%########-####-####-####-##########01%';

5. Run the following command to delete the target record (Please change the ID and device_info_label to those confirmed in step 4):

   # delete from vc.vpx_vm_virtual_device where ID = '####' and device_info_label = 'Network adapter #';

6. Confirm record deletion. The following command should return 0 results:

   # select * from vc.vpx_vm_virtual_device where device_text like '%########-####-####-####-##########01%';

7.  Use to following command to disconnect from the vCenter database:

   # \q

8. Start the vpxd service:

   # service-control --start vmware-vpxd

The removal of the ESXi host from the vSphere Client inventory should now no longer be failing due to the above error.