BoringSSL IP SAN validation error causing vsan-health service failure - vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

The vmware-vsan-health service fails to initialize following a Machine SSL certificate replacement in vCenter Server 8.x. Logs show repeated "526 (Invalid SSL Certificate)" errors during the loginExtensionByCertificate handshake and sslv3 alert certificate unknown in vpxd.log. This occurs when the vCenter Server is using custom certificates and a BoringSSL validation conflict is triggered.

This issue may also manifest as failure to add ESXi hosts to vCenter server with the error:

'Authenticity of the host's SSL certificate is not verified'.

Environment

• VMware vCenter Server 8.x
• Custom or CA-signed certificates
• Environment contains CA chains with IP-typed nameConstraints

Cause

 BoringSSL (used by the Envoy sidecar in vCenter 8.x) contains a validation limitation where it unconditionally rejects leaf certificates containing an IP Subject Alternative Name (SAN) if any CA in the chain (including intermediates) possesses an IP-typed nameConstraints extension. This failure occurs regardless of whether the IP address in the SAN is actually valid within the defined constraints.

Resolution

 To resolve this issue, the Machine SSL certificate must be re-issued without IP-based Subject Alternative Names (SANs).

1. Verify Certificate Structure: Identify if the current Machine SSL certificate contains an IP SAN:

   openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -text -noout | grep "IP Address"

2. Generate New Certificate: Re-generate the CSR and request a new Machine SSL certificate from your Certificate Authority (CA). Ensure the new certificate configuration excludes any IP addresses from the Subject Alternative Name (SAN) fields.

   • Use FQDNs for all SAN entries instead.

3. Install and Restart Services:

   • Install the new certificate via the certificate-manager utility.
   • Perform a full service restart to clear stale certificate caches:

     service-control --stop --all
     service-control --start --all

Additional Information

• If the issue persists, review the full CA chain for other inconsistencies. Refer to KB 395218 for general vsan-health troubleshooting.
• For help with certificate replacement procedures, see Replace vCenter Machine SSL certificate.
• If you need further assistance, please Contact Broadcom Support.