vCenter Server, ESXi and/or NSX are disconnected after a Cloud Director upgrade
search cancel

vCenter Server, ESXi and/or NSX are disconnected after a Cloud Director upgrade

book

Article ID: 320513

calendar_today

Updated On:

Products

VMware Cloud Director VMware NSX VMware vSphere ESXi

Issue/Introduction

  • After upgrading VMware Cloud Director (VCD) , vCenter Server and/or NSX servers are disconnected. 
  • After upgrading  you are presented with a banner similar to the following:
VMware Cloud Director and later versions use the latest SSL practices for integration with the vSphere infrastructure. You must verify that VMware Cloud Director trusts all necessary vSphere certificates. 
  • After upgrading VMware Cloud Director (VCD) cannot deploy Virtual Machine  and getting error:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 

  • After upgrading VMware Cloud Director (VCD) , tasks which require direct ESXi connections such as Consoles, Guest Customization or OVF/Media Uploads will fail.
  • After a fresh install of VMware Cloud Director (VCD) 10.4 or above tasks which require direct ESXi connections such as Consoles, Guest Customization or OVF/Media Uploads will fail.
  • After a fresh install of VMware Cloud Director (VCD) 10.4 or newer endpoint connections will fail with an error for No subject alternative names present.
  • Within log reviews, the provider might see the following errors : 
    • Error: Unable to find valid certification path to requested target 
    • Error: Certificate for <X.Y.Z> doesn't match any of the subject alternative names: [A.B.C] 
    • Error: IOException while reading data. Closing transfer ... Received fatal alert: bad_certificate 



Environment

VMware NSX-T Data Center
VMware Cloud Director 10.x

Cause

  1. VMware Cloud Director introduces enhanced SSL validation which prohibits the option to trust all VC/NSX certificates without validation. If this configuration was previously used with self signed certificates that were not uploaded to VMware Cloud Director, this would result in the connection not being trusted by VCD upon upgrade. 
  2. Since VMware Cloud Director 10.4 extends the above requirements to ESXi and requires that the VMCA certificate is imported into the Cloud Director Truststore. 
  3. As of VMware Cloud Director 10.4, Hostname Verification is enabled by default.
  4. VCD has removed a security hole that allowed cells to run without certificate verification. 
  5. This issue can also occurs if the certificates are expired for the underlying Host or the certificates are NOT listed under trusted certificates for Cloud Director.

Resolution

Note: 

  • Since VMware Cloud Director 10.4, Hostname Verification is enabled by default and must remain enabled. 
  • The option to disable Hostname Verification is deprecated and will be removed in a future release of Cloud Director.
  • Hostname Verification can be validated via the UI or API 
  • Ensure that ESXi, vCenter, NSX are up to date


For the UI 

  1. Login to the Provider portal 
  2. Click Administration  
  3. Click General (within the Settings subsection) 
  4. Navigate to the Certificates section 
  5. Ensure Use hostname verification for vCenter Server and vSphere Certificates is enabled 


For the API 

  1. Using a REST Client, update the General Settings of Cloud Director. 
  2. Ensure the VerifyVcCertificates element is set to true. 


For SSL related errors, below are the options to resolve those issues: 

Note:

  • For vCenter Server 8.x and 7.x, all Options below are valid.
  • For vCenter Server 6.7.x, only Options 2 and 3 below are valid.


Option 1: Individually 
Using the UI, edit each individual connection and click Save.
The UI will allow you to interactively review and trust individual certificates presented by each respective server (Trust-on-first-use principle). 
In 10.4, post trusting the vCenter Server certificate, another prompt will appear for the VMCA certificate which is required for Cloud Director to trust the connections to ESXi Hosts. 

Option 2: Automated 
Run this cell-management-tool command to retrieve and trust certificates from all configured vCenter Server and NSX servers as well as the VMCA certificate (10.4): 
/opt/vmware/vcloud-director/bin/cell-management-tool trust-infra-certs --vsphere --unattended 
This command helps by providing additional options on how to review the certificates that will be trusted before trusting them. 

Option 3: Manual 
Should the above options not be possible, the VMCA certificate can be downloaded manually and uploaded to the Cloud Director Truststore. 

  1. To manually download the vCenter Server Root Certificates please refer to the following KB . 
  2. To import the VMCA certificate follow the Cloud Director Import Trusted Certificates documentation. 



For SAN related errors, below are the options to resolve the issue: 

Option 1: For 10.3 
The Provider must replace any certificate for VC, NSX-V, NSX-T managers, or other endpoints like Aria Operations For Logs to have valid SANs that match the hostname. 

Additional Information

07/19/2021 - Published externally

Powered by Wolken Software