vCenter Server vpxd Service Repeatedly Stops After Reverting from 8.0 U3 to 8.0 U2

Products

• VMware vCenter Server

Issue/Introduction

The vCenter Server vpxd service stops within minutes of starting, causing the vCenter to show objects and items being unavailable. Only the vpxd service stops, and is easily restarted. However, it soon stops again. This issue occurs after reverting a vCenter Server to a previous version snapshot.

Environment

- VMware vSphere 8.0 and later
- vCenter Server Appliance (VCSA) reverted from 8.0 U3 to 8.0 U2
- At least some ESXi hosts managed by the vCenter running ESXi version 8.0 U3

Cause

This issue occurs when a vCenter Server running version 8.0 Update 3 is reverted to version 8.0 Update 2, and some ESXi hosts in the environment have already been upgraded to 8.0 Update 3. The older vCenter version (8.0 U2) is not aware of Embedded vSphere Cluster Services (vCLS) VMs introduced in Update 3, leading to conflicts on the upgraded hosts that cause the vpxd service to stop repeatedly.

Resolution

1. Identify the hosts with Embedded vCLS VMs:
   1. Connect to the vCenter Server Appliance via SSH.
   2. Connect to the vCenter PostgreSQL database by running the following command:

      • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

   3. Once connected to the database, run the following SQL query to identify hosts with a product version mismatch:

      • select id, dns_name, product_version from vpx_host;

   4. Look for hosts with product version 8.0.3, which is higher than the current vCenter Server version (8.0.2).
   5. After you've finished, exit the PostgreSQL prompt by typing:

      • \q

2. For each identified host:
   1. Connect to the ESXi host via SSH.
   2. Identify the Embedded vCLS VMs:

      • vim-cmd vmsvc/getallvms | grep vCLS

   3. Power off each vCLS VM:

      • vim-cmd vmsvc/power.off <VM_ID>

   4. Unregister each vCLS VM:

      • vim-cmd vmsvc/unregister <VM_ID>

3. Restart the vpxd service on the vCenter Server:

   • • service-control --stop vmware-vpxd

     • service-control --start vmware-vpxd

4. Monitor the vpxd service to ensure it remains stable.
5. If the issue persists, consider disconnecting and reconnecting the problematic hosts one at a time to identify any remaining conflicts.

Additional Information

vSphere Cluster Services (vCLS) Known Issues/Corner Cases