Console Proxy Failures After Upgrading to 10.4 or Later

Products

• VMware Cloud Director

Issue/Introduction

• This KB is designed to address console proxy failures after upgrading to 10.4 or later

Symptoms:

• After upgrading to version 10.4 or later, the Console Proxy feature is no longer functional in the environment
• VMRC and web consoles fail to instantiate from the VCD UI, despite this feature working as expected in the underlying vCenter Server
• When a web console is launched, the window never returns the VM terminal, instead it says "wmks.connecting" before returning with "Disconnected."

Environment

VMware Cloud Director for Service Provider 10.x
VMware Cloud Director 10.x

Cause

• The console proxy feature was re-factored for version 10.4 and later. Anything prior to version 10.4 utilizes the "legacy" console proxy implementation. The new console proxy no longer uses port 8443, but instead integrates with the other services running on port 443 directly. Additionally, the new console proxy code base requires that the underlying ESXi hosts have a trusted certificate in the "Trusted Certificates" store in VCD. This is a security measure that furthers the enhanced SSL validation standards rolled out in version 10.3.3

Resolution

There are a number of possible misconfigurations; the troubleshooting steps are listed below...
 

• Validate that you can open a remote console for a given VM in the underlying vCenter. If you cannot, VCD will not be able to either
• Verify that the SSL certificate imported in each cell is also imported into the "Public Addresses" tab in the provider portal. The easiest way to do this is to grab the .pem file from one of the appliances and use the built-in UI feature to import that file directly on the "Public Addresses" page
• Verify that the vCenter VMCA certificate is imported and trusted in the "Trusted Certificates" tab in the provider portal.
• Trust infrastructure certificates: 

./cell-management-tool trust-infra-certs --vsphere --unattended

• Set the httptransfer.useVcenterUrl configuration to false: 

./cell-management-tool manage-config -n vcloud.val.httptransfer.useVcenterUrl -v false

• If the remote console still cannot connect, verify that the LB configuration is not breaking the connection

Workaround:

• For version 10.4, you can workaround this issue by enabling the Legacy Console Proxy Feature Flag, however, this is not available in later versions
• All ESXi Certificates should be automatically trusted under the umbrella of our trusted VMCA certificate. Circumstances may exist where there is a trust issue with the ESXi certificate in which we need to take additional actions to remedy. 
  • The best option is to try and repair the certificate relationship by performing a Renew from the vSphere Client Inventory. This will usually automatically fix any VCD related trust issues with the ESXi hosts. 
    • Renew or Refresh ESXi Certificates
  • Alternatively, if that does not resolve the issue, we can:
    • Create a test VM
    • Identify the host on which that VM is running.
    • Grab the FQDN of that host and browse to it in a browser.
    • Using built-in browser functionality, download the certificate for that host
    • Import it into the "Trusted Certificates" tab.
  • This will manually trust the ESXi certificate and usually allow a successful console connection when there is an certificate trust issue related to the ESXi host. 

Additional Information

If you need to manually refresh your ESXi host certificates as outlined in Renew or Refresh ESXi Certificates but have a large number of hosts, you may reference this documentation to accomplish this task via CLI commands:

Managing Services and Certificates with CLI Commands

Impact/Risks:

• The VMRC and web consoles will not instantiate after upgrading to 10.4
• End users will not be able to use console proxy feature