HA clusters fails to configure in vCenter Server 8.0 U3 when the VPXD certificate mode is set to "thumbprint".

Products

• VMware vCenter Server 8.0

Issue/Introduction

• After upgrading to vCenter Server 8.0.3, HA enabled clusters fail to configure, and only the primary HA host remains active. 
• Messages in fdm.log mention "tlsv1 alert unknown ca" when contacting other hosts
  
  Failed to SSL handshake; SSL(<io_obj p:0x0000002e86fda5f0, h:30, <TCP '10.10.10.10 : 8111'>, <TCP '10.10.10.11 : 11911'>>), e: 167773208(tlsv1 alert unknown ca (SSL routines)), duration: 5msec
  
  
• fdm.log also contains messages similar to the following when attempting to contact the master FDM host
  
  Er(163) Fdm[1151234]: --> The remote host certificate has these problems:
Er(163) Fdm[1151234]: -->
Er(163) Fdm[1151234]: --> * unable to get local issuer certificate)


• vCenter Server's certificate mode is set to thumbprint mode.

Environment

vCenter Server 8.0.3

Cause

This is caused by a change to vSphere HA in version 8.0U3 which now validates the certificate during fdm operations.

Resolution

To work around this issue, change the vpxd certificate mode to "vmca" or "custom" depending on your environment. See the links below for further instructions.

NOTE: Ensure to restart services to reflect the change:
service-control --stop --all && service-control --start --all

Important: If the ESXi hosts use custom CA signed certificates, enabling vmca mode and renewing will overwrite the custom certificates. Please use caution before choosing the proper path.

 

Step 1: Refer to Managing Certificates for ESXi Hosts for details on choosing the correct mode for your environment. 

Step 2: Refer to Change the ESXi Certificate Mode for steps to update the certificate mode.

Step 3: Refer to Renew or Refresh ESXi Certificates for steps to renew the certificates on the ESXi hosts.

 

Important Note:

• There are chances of communication issues between vCenter Server and ESXi hosts after changing certificate mode on vCenter Server. To avoid the host communication issues, please perform "Refresh CA," and "Renew certificate" for each host using vSphere Client or using PowerCLI after changing the Certificate Mode.
• Sample script with PowerCLI method available in Additional Information section of this KB.
• Please refer Renew or Refresh ESXi Certificates for more details on Refresh and Renew certificate operations.

Additional Information

Some environments may be using thumbprint mode to work around an existing certificate issue causing host connectivity problems. These issues will have to be properly diagnosed and corrected when using vmca and custom modes.

Note: Thumbprint mode is not recommended as a permanent solution for certificate related issues, and should only be used as a temporary fallback while troubleshooting.

Sample PowerCLI script to refresh ESXi host certificates.

#To use this script, pipe hostnames to it.
#first step is to import it:
#> . c:\path\to\script.ps1
#> get-cluster | get-vmhost | foreach {$_.Name} | refreshcerts
function refreshcerts
{
 Begin{

 
 }
 Process{
 Write-host "Processing $_"
 $hostid = Get-VMHost $_ | Get-View
 $hostParam = New-Object VMware.Vim.ManagedObjectReference[] (1)
 $hostParam[0] = New-Object VMware.Vim.ManagedObjectReference
 $hostParam[0].value = $hostid.moref.value
 $hostParam[0].type = 'HostSystem'
 $_this = Get-View -Id 'CertificateManager-certificateManager'
 $task = $_this.CertMgrRefreshCACertificatesAndCRLs_Task($hostParam)
 Wait-Task -Task $task
 $task = $_this.CertMgrRefreshCertificates_Task($hostParam)
 Wait-Task -Task $task
 }
 End{
 Start-Sleep -Seconds 10

 Write-host "Done"

 }
}