Error: "The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate"

Products

• VMware vCenter Server

Issue/Introduction

• After upgrading vCenter Server to version 9.0, ESXi host SSL certificates may fall out of synchronization with the vCenter Server database (VCDB). This mismatch causes the vpxd service to crash or prevents hosts from reconnecting, resulting in stalled upgrade tasks in SDDC Manager.

• The /var/core/ directory in vCenter contains "core.vpxd-worker" files whenever the service crashes.

• In /var/log/vmware/vpxd/vpxd.log, the following error is present: 

  error vpxd[59759] [Originator@6876 sub=HostAccess] The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate

• Reverting to the previous vCenter version consistently resolves SSL certificate conflicts, ensuring seamless operation with existing ESXi hosts without 'sslThumbprint' or 'sslCertificate' parameter.

Environment

• VMware vCenter Server 9.0
• VMware Cloud Foundation 9.x

Cause

Upon upgrading vCenter server to version 9.0, ESXi host SSL certificates are observed to fall out of synchronization, leading to subsequent vpxd service panics.

Resolution

Broadcom Engineering is currently engaged in the development of a code-level solution to identify and rectify the root cause of ESXi host SSL certificate synchronization issue. Subscribe to this KB for latest updates on the fix.

Workaround:

1. Log in to vCenter Server (VCSA) using SSH.

2. Stop vpxd service:

   service-control --stop vpxd

3. Backup the vpxd.cfg file:

   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.original

4. Edit vpxd.cfg and disable mobility health monitoring.

   vi vpxd.cfg

5. Add the below lines in vpxd.cfg file (Can be pasted anywhere within the <config> section):

   <mobility>

    <healthTimerInterval>-1</healthTimerInterval>

   </mobility>

6. Save and exit the vi editor (!wq).

7. Start the vpxd service:

   service-control --start vpxd

8. Disconnect and Reconnect all the hosts in the inventory so that SSL certs are updated. Refer to Disconnect hosts from vCenter via database.

9. If a specific host remains in a Not Responding or Disconnected state, update that host individually using its Managed Object Reference (MoRef) ID. Replace #### with your specific ID:

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled=1 WHERE id = '####';"

10. Proceed to revert the change so that mobility health monitoring feature will be effective again.

    cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.modified && cp /etc/vmware-vpx/vpxd.cfg.original /etc/vmware-vpx/vpxd.cfg

11. Restart vpxd service:

    service-control --restart vpxd

Additional Information

Note: The below step requires the vpxd service to be in stopped state as this involves VCDB update.

Command to disconnect all hosts from VCDB:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled=0"

Command to reconnect all hosts from VCDB:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled=1"