ESXi Host Fails to Join vCenter Server Resulting in a "526 Invalid SSL Certificate" Error

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

An ESXi host fails to join a vCenter Server instance when utilizing the Fully Qualified Domain Name (FQDN). The operation results in a 526 Invalid SSL Certificate error, whereas it succeeds if the IP address is provided.

The failure can be validated via the vCenter service logs (/var/log/vmware/vpxd/vpxd.log), where an SSLVerifyException is recorded. The logs indicate that the expected peer name does not match the subject name provided in the certificate presented by the host.

vapi.invalid.result.code<Recv of frame failed with code: 526 Invalid SSL Certificate>

 

Environment

VMware Cloud Foundation 9.1

Cause

The underlying cause is a name resolution discrepancy. The internal Envoy proxy on the vCenter Server resolves the host FQDN to an incorrect IP address belonging to a different cluster. This results in the proxy receiving a Transport Layer Security (TLS) certificate that does not match the expected hostname, triggering the validation failure.

Additionally, the host certificate Subject Alternative Name (SAN) field contains the IP address but lacks the full FQDN required for proper VAPI/Envoy validation.

Resolution

To resolve this issue, the host certificates and name resolution records must be corrected:

1. Verify the DNS A records in the specific subdomain to ensure the <ESXi Host Name> FQDN points to the correct management IP (##.###.##.##.###).

2. Regenerate the ESXi host certificates using a Certificate Signing Request (CSR) that explicitly includes the full FQDN in the SAN field.

3. Verify the updated DNS state by running the following command from the vCenter Server shell: nslookup <ESXi Host Name FQDN> ##.###.##.##.### (Note: Replace the trailing IP with the environment's DNS server IP)

4. Temporarily toggle the vCenter Server advanced setting vpxd.certmgmt.mode to thumbprint to verify if the join succeeds by bypassing hostname validation.

5. Restart the vpxd service on the <vCenter FQDN> Appliance to clear any cached Envoy proxy resolution data.

Additional Information

Standardizing the certificate SAN to include the FQDN ensures compliance with the strict identity validation enforced by the VMware Cloud Foundation 9.x VAPI layer.

Correcting the DNS mapping aligns the environment with the expected architectural state required for secure communication between vCenter Server and ESXi hosts.