Manually removing an ESXi host from the vCenter Server database

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0
• VMware vCenter Server 6.0

Issue/Introduction

This article provides steps to manually remove an ESXi host from the vCenter Server database.

 

Warnings:

• This procedure should only be used as a last resort where all other attempts to remove an ESXi host from the vCenter Server Inventory fail.
• Before applying the steps below, please take a backup or an offline-snapshot (in powered-off state) of the vCenter Server Appliance . If the vCenter is part of a Linked Mode replication setup, please backup/snapshot all replicating nodes as well.
• Please also take a full backup of the database. VMware recommends that you consult a Database Administrator (DBA) before implementing any changes to the database configuration.
• This procedure does not work for Oracle database.

Symptoms:

• An ESXi host is decommissioned without first removing it from the Inventory in vCenter Server.
• The ESXi host no longer exists, physically or in DNS, but is shown as Disconnected in vCenter Server.
• The ESXi host cannot be removed from the vCenter Inventory.
• When selecting either Remove or Connect in the vCenter GUI, there is an error similar to:
  
  Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem.
   
• When there is a ghosted host on the vDS (vNetwork Distributed Switch), there are entries similar to these in the vpxd.log file:
  

  [YYYY-MM-DD hh:mm:ss.283 01624 warning 'App' opID=#######-#######] [MoDVSwitch::UpdateHostNetworkSummaryOnDvsDestroy] Invalid host member [host-###] found in the vDS. [YYYY-MM-DD hh:mm:ss.295 01624 error 'App' opID=#######-########] [MoDVSwitch::HostCallPreFlightCheckInt] DVS.ConfigSpec.host[host-###].host not registered in vmom

   
• The host is already connected to the vCenter Server.

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.x

VMware vCenter Server 8.x

Resolution

Warning: These steps should not be used in environments using vCloud Director.

Caution: These steps are for vCenter Server using a Microsoft SQL database or vPostgres.
Please consult a Broadcom Support Engineer before performing these steps. For more information, please see Creating and managing Broadcom support cases 

Important: Ensure a non-memory vCenter snapshot is taken for a standalone vCenter. If the vCenter is in ELM, ensure to take offline snapshots for all the vCenters that are in linked mode. 

ESXi 7.x and 8.x

To manually remove an ESXi 7.x and 8.x host from the vCenter Server database:

• Stop the vpxd service on the vCenter Server. For more information, please see Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services for vCenter Appliance.
  

  service-control --stop vmware-vpxd

• Connect to the vCenter Server vPostgres database:
  
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

• Identify the ESXi host ID in the vCenter database :
  
  select ID from VPX_ENTITY where name ='<esxi host name>';
  
  Note: For example, the ESXi host being removed is esx01.example.com. This host returns a host ID of ##### when the preceding select statement is executed.
   
• Execute these SQL queries to remove the ESXi host from the vCenter database.
  Note: The SQL queries must be executed in the order specified to ensure that any constraint definitions in the database are not violated. All the following commands should be run with the same ID noted from the output of the above command.
  
   
  1. Remove all the datastore assignments for virtual machines that exist on this ESXi host:
     
delete from vpx_ds_assignment where entity_id in (select id from vpx_vm where host_id = #####);
      
  2. Remove all the network assignments of virtual machines that exist on this ESXi host:
     
delete from vpx_nw_assignment where entity_id in (select id from vpx_vm where host_id = #####);
      
  3. List the virtual machines in the virtual machine table that exist on this ESXi host:
     
select id from vpx_vm where host_id = #####;
     Note: Make a note of the virtual machine IDs returned in the Output.
      
  4. Remove the virtual machines from the previous step using the article: Manually removing a stale VM from the vCenter Server vpostgres database.
     
     
  5. Delete the DVS entries associated with the ESXi host:
     
delete from vpx_dvhost where host_id = #####;
delete from vpx_dvport_membership where host_id = #####;
delete from vpx_dvs_blob where host_id = #####;
      
  6. Delete the ESXi host from the host table:
     
delete from vpx_host where id = #####;

     Notes:
     1. A cascade delete constraint definition (constraint FK_VPX_HOST_REFERENCE_VPX_HOST foreign key (HOST_ID) references VPX_HOST (ID) on delete cascade) is created on these tables. Corresponding entries with matching HOST_ID=##### automatically get removed from these tables:
        vpx_host_x
vpx_host_vm_config_option
vpx_compute_resource_dpm_host
vpx_host_cpu
vpx_host_node
vpx_host_node_cpu
vpx_host_pci_device
         
     2. A cascade delete constraint definition (constraint FK_VPX_CPU_THREAD_REF_HOST_CPU foreign key (HOST_ID, CPU_INDEX) references VPX_HOST_CPU (HOST_ID, CPU_INDEX) on delete cascade) is created on these tables. Corresponding entries with matching HOST_ID=#### automatically get removed from these tables:
        vpx_host_cpu_thread
vpx_host_cpu_cpuid_feature
         
  7. Delete the ESXi host from the entity table:
     
delete from vpx_entity where id = #####;
      
  8. Remove all relations to this ESXi host from the entity table:
     
delete from vpx_entity where parent_id = #####;
delete from vpx_entity where id = #####;


  9. Exit the database:
     
     \q

• Start the vpxd service on the vCenter Server to pick up the database changes.
  
  service-control --start vmware-vpxd

 

In case the ESXi Host is a standalone host (not part of a cluster), additional steps need to be taken to avoid a situation where a standalone compute resource is left without a host.

To remove a standalone ESXi Host from the vCenter Database:

• Stop the vpxd service on the vCenter Server.
  

  service-control --stop vmware-vpxd

• Connect to the vCenter Server vPostgres database:
  
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

• In the Execute SQL query steps (Step 1 to 8)  to delete the entries from different tables, along with implementing the above steps, please implement the additional two steps:
  • Check for orphaned compute resources with this statement:

SELECT id, name FROM vpx_entity WHERE type_id = 2 and id not in (SELECT a.id FROM vpx_entity as A LEFT JOIN vpx_entity as B ON A.id = B.parent_id where a.type_id=2 and b.type_id=1 GROUP BY A.id);

Example Output: 

id | name |

--------+--------------------------+---------+-----------

##### |<Resource name> |

##### |<Resource name> |

(2 rows)

• • If some records are displayed, delete all zombie entries (##### is the ID of each compute resource from step 1):

delete from vpx_entity where parent_id = #####;
delete from vpx_entity where id = #####;

• • Start the vpxd service:
    
    service-control --start vmware-vpxd