vCenter Server vpxd Service Repeatedly Crashes Due to “NoCompatibleHost” After Snapshot Revert from vCenter version 8.0 U3 to 8.0 U2

Products

VMware vSphere ESXi 8.0 VMware vCenter Server 8.0

Issue/Introduction

The vmware-vpxd service stops within minutes of a successful start.
vCenter objects unavailable in the vSphere Client.
Restarting the vpxd service via service-control --start vmware-vpxd works temporarily, but the service crashes again.
The issue occurs immediately following a snapshot revert or a failed upgrade/downgrade of the vCenter Server.

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DD info vpxd[1234] [Originator@6876 sub=MoCluster opID=27XXXXXX-01] Recommended placement for HDCS VM; [vim.VirtualMachine:vm-X17,vCLS-3XXXXXX-XXX-XXXX-8XXXXXXXXXXXX7], cluster: [vim.ClusterComputeResource:domain-cX,Gold Cluster], host: (null), fault: (vim.fault.NoCompatibleHost) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> host = (vim.HostSystem) [
--> 'vim.HostSystem:5XXXXcXX-XXXX-XXXX-XXXX-fXXXXXXXXf:host-XXXX'
--> ],
--> error = (vmodl.MethodFault) [
--> (vim.fault.HAErrorsAtDest) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""
--> }
--> ]
--> msg = ""
-->

YYYY-MM-DD info vpxd[4567] [[Originator@6876 sub=MoCluster opID=27XXXXXX-01] Recommended placement for HDCS VM; [vim.VirtualMachine:vm-X17,vCLS-3XXXXXX-XXX-XXXX-8XXXXXXXXXXXX7], cluster: [vim.ClusterComputeResource:domain-cX,Gold Cluster], host: (null), fault: (vim.fault.NoCompatibleHost) {

Environment

vCenter server 8.0.U2 and 8.0 U3

ESXI 8.0.U2 and 8.0 U3

Cause

The version mismatch between the vCenter Server and the Embedded vCLS VMs on certain hosts. This mismatch occurs due to the snapshot revert process, which doesn't account for the Embedded vCLS feature introduced in later versions.
This presents as a database incompatibility which appears to the vCenter Server under 8.0 U2 to be a corrupt object in the vcdb.

Resolution

Option 1.

Before following the below steps have a vCenter backup in place, Refer to the KB article for steps to manually make the vCenter backup from VAMI : KB 375278
Also take vCenter snapshot before proceeding with the below steps, if the vCenter are in ELM please take the offline snapshot, Refer to the KB article for VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice : KB 313886

Identify the hosts with Embedded vCLS VMs:
1. Connect to the vCenter Server Appliance via SSH.
2. Connect to the vCenter PostgreSQL database by running the following command:
  - ```
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
```
3. Once connected to the database, run the following SQL query to identify hosts with a product version mismatch:
  - ```
  select id, dns_name, product_version from vpx_host;
```
4. Look for hosts with product version 8.0.3, which is higher than the current vCenter Server version (8.0.2).
5. After you've finished, exit the PostgreSQL prompt by typing:
  - ```
  \q
```
For each identified host:
1. Connect to the ESXi host via SSH.
2. Identify the Embedded vCLS VMs:
  - ```
  vim-cmd vmsvc/getallvms | grep vCLS
```
3. Power off each vCLS VM:
  - ```
  vim-cmd vmsvc/power.off <VM_ID>
```
4. Unregister each vCLS VM:
  - ```
  vim-cmd vmsvc/unregister <VM_ID>
```

Restart the vpxd service on the vCenter Server:

```
service-control --stop vmware-vpxd
```
```
service-control --start vmware-vpxd
```

Monitor the vpxd service to ensure it remains stable.
If the issue persists, check to see if starting vpxd is regenerating the vCLS vms using the command in step b. If so, follow THIS KB to manually disable embedded vCLS on the affected hosts
If it persists beyond this point, consider disconnecting and reconnecting the problematic hosts one at a time to identify any remaining conflicts.

Option 2.

Before following the below steps have a vCenter backup in place, Refer to the KB article for steps to manually make the vCenter backup from VAMI : KB 375278
Also take vCenter snapshot before proceeding with the below steps, if the vCenter are in ELM please take the offline snapshot, Refer to the KB article for VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice : KB 313886

Connect to the vCenter Server Appliance via SSH.

1. Disables ALL ESXi Hosts from the vCenter using the below command.
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled = 0"
2. Update the vCenter appliance to the latest vCenter version. The updated vCenter must be a higher version than the ESXi hosts connected to it.
3. Enables ALL ESXi Hosts back:
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled = 1"
4. Connect all the ESXi if disconnected in the vCenter UI.
5. Monitor the vpxd service to ensure it remains stable.

Script to connect all hosts including 30 second wait between connects.
- - Open Powershell
  - To connect the vCenter : Connect-VIServer -Server EXAMPLE_FQDN/IP_OF_VCENTER
  - Create a file with the below contents and run the script from the same location as powercli (desktop)

./<filename>.ps1

-----------------------------------

$password = "EXAMPLE_PASSWORD"

get-vmhost | % {
$view = get-view $_.id
$arg = new-object VMware.Vim.HostConnectSpec
$arg.userName = "root"
$arg.password = $password
$arg.force = $true

$view.ReconnectHost_Task($arg,$null)
}

Disconnect-VIServer -Confirm:$false

--------------------------------------

Note that : EXAMPLE_PASSWORD = your custom ESXi password