VPXD service crashes after 8.x to 9.0 upgrade, reporting with the error "The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate"

Products

• VMware vCenter Server

Issue/Introduction

• /var/core/ folder in vCenter will be seen with "core.vpxd-worker" files whenever the service crashes
• vpxd logs report below error,
  Path:/var/log/vmware/vpxd/vpxd.log
YYYY-MM-DDTxx:xx:xx error vpxd[59759] [Originator@6876 sub=HostAccess] The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate
  
  
• Reverting to the previous vCenter version consistently resolves SSL certificate conflicts, ensuring seamless operation with existing ESXi hosts without 'sslThumbprint' or 'sslCertificate' parameter

Environment

VMware vCenter Server 9.0

Cause

Upon upgrading vCenter to version 9.0, ESXi host SSL certificates are observed to fall out of synchronization, leading to subsequent vpxd service panics

Resolution

Broadcom Engineering is currently engaged in the development of a code-level solution to identify and rectify the root cause of ESXi host SSL certificate synchronization issue.

Workaround:

• Log in to vCenter Server (VCSA) using ssh
• Disable mobility health monitoring feature in vpxd.cfg file
   
  • Stop vpxd service

# service-control --stop vpxd

• • Backup vpxd.cfg file
  • 
    cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.original
    
    
  • Edit vpxd.cfg using vi editor
  • 
    # vi vpxd.cfg
     
• Add the below lines in vpxd.cfg file (Can be pasted anywhere within the <config> section)
  
  <mobility>
 <healthTimerInterval>-1</healthTimerInterval>
</mobility>
  
  
• Save and exit the vi editor (!wq)
• Start the vpxd service

# service-control --start vpxd

• Disconnect and Reconnect all the hosts in the inventory so that SSL certs are updated. Refer to Disconnect hosts from vCenter via database
• Proceed to revert the change so that mobility health monitoring feature will be effective again.
  

cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.modified && cp /etc/vmware-vpx/vpxd.cfg.original /etc/vmware-vpx/vpxd.cfg

• Restart vpxd service

# service-control --restart vpxd

Additional Information

Command to disconnect all hosts from VCDB: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled=0"

Command to reconnect all hosts from VCDB: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled=1"

Note: The above step requires the vpxd service to be in stopped state as this involves VCDB update