vCenter Server, ESXi and/or NSX are disconnected after a Cloud Director 10.1, 10.3 or 10.4 upgrade
Article ID: 320513
Updated On:
Products
Issue/Introduction
Symptoms:
- After upgrading to VMware Cloud Director (VCD) 10.1, 10.3, 10.4, vCenter Server and/or NSX-V / NSX-T servers are disconnected.
- After upgrading to VMware Cloud Director (VCD) 10.4, you are presented with the following banner
- After upgrading to VMware Cloud Director (VCD) 10.4, tasks which require direct ESXi connections such as Consoles, Guest Customization or OVF/Media Uploads will fail.
- After a fresh install of VMware Cloud Director (VCD) 10.4 or above tasks which require direct ESXi connections such as Consoles, Guest Customization or OVF/Media Uploads will fail.
- Within log reviews, the provider might see the following errors :
- Error: Unable to find valid certification path to requested target
- Error: Certificate for <X.Y.Z> doesn't match any of the subject alternative names: [A.B.C]
- Error: IOException while reading data. Closing transfer ... Received fatal alert: bad_certificate
Environment
VMware Cloud Director 10.x
Cause
- VMware Cloud Director introduces enhanced SSL validation which prohibits the option to trust all VC/NSX certificates without validation. If this configuration was previously used with self signed certificates that were not uploaded to VMware Cloud Director, this would result in the connection not being trusted by VCD upon upgrade.
- VMware Cloud Director 10.4 extends the above requirements to ESXi and requires that the VMCA certificate is imported into the Cloud Director Truststore.
- As of VMware Cloud Director 10.4, Hostname Verification is enabled by default.
- VCD has removed a security hole that allowed cells to run without certificate verification.
Resolution
Note:
- In VMware Cloud Director 10.4, Hostname Verification is enabled by default and must remain enabled.
- The option to disable Hostname Verification is deprecated and will be removed in a future release of Cloud Director.
- Hostname Verification can be validated via the UI or API
For the UI
- Login to the Provider portal
- Click Administration
- Click General (within the Settings subsection)
- Navigate to the Certificates section
- Ensure Use hostname verification for vCenter Server and vSphere Certificates is enabled
For the API
- Using a REST Client, update the General Settings of Cloud Director.
- Ensure the VerifyVcCertificates element is set to true.
For SSL related errors, below are the options to resolve those issues:
Note:
- For vCenter Server 7.x, all Options below are valid.
- For vCenter Server 6.7.x, only Options 2 and 3 below are valid.
Option 1: Individually
Using the UI, edit each individual connection and click Save.
The UI will allow you to interactively review and trust individual certificates presented by each respective server (Trust-on-first-use principle).
In 10.4, post trusting the vCenter Server certificate, another prompt will appear for the VMCA certificate which is required for Cloud Director to trust the connections to ESXi Hosts.
Option 2: Automated
Run this cell-management-tool command to retrieve and trust certificates from all configured vCenter Server and NSX servers as well as the VMCA certificate (10.4):
/opt/vmware/vcloud-director/bin/cell-management-tool trust-infra-certs --vsphere --unattended
This command helps by providing additional options on how to review the certificates that will be trusted before trusting them.
Option 3: Manual
Should the above options not be possible, the VMCA certificate can be downloaded manually and uploaded to the Cloud Director Truststore.
- To manually download the vCenter Server Root Certificates please refer to the following KB .
- To import the VMCA certificate follow the Cloud Director Import Trusted Certificates documentation.
For SAN related errors, below are the options to resolve the issue:
Option 1: For 10.3
The Provider must replace any certificate for VC, NSX-V or NSX-T managers to have valid SANs that match the hostname.
Additional Information
Feedback
