VMRC connection fails with Could not negotiate SSL via external Load Balancer in VCFA 9.x

Products

• VCF Automation

Issue/Introduction

• When launching a remote console of a virtual machine through the VCF Automation (VCFA) Portal using an external Load Balancer (LB), the connection fails with the error message "Connection error could not negotiate ssl". WebMKS console connections work successfully.
  
  
• VMRC logs display an SSL negotiation failure and a certificate thumbprint mismatch:
  

  In(05) mks SSL Error: error:0A000086:SSL routines::certificate verify failed
  Wa(03) mks SOCKET 2 (1140) Could not negotiate SSL
  Wa(03)+ mks The remote host certificate has these problems:
  Wa(03)+ mks * self-signed certificate in certificate chain
  Wa(03) mks SOCKET 2 (1140) Expected thumbprint doesn't match actual thumbprint.

Environment

• VCF Automation 9.x

Cause

The most common cause is a certificate thumbprint mismatch. VMRC validates the certificate thumbprint of the connection against the portal's public address. If the certificate presented by the Load Balancer (typically a Public CA) does not match the certificate applied to the API/Portal (which may be an Internal CA), the SSL negotiation fails. Additionally, some security appliances block the specific TCP traffic required for VMRC (MKS/VNC protocol).The external load balancer is terminating SSL and presenting a different certificate rather than the VCFA certificate. Because of this, the actual SSL thumbprint presented to the VMRC client does not match the expected thumbprint provided by the VCFA portal, causing the connection to drop.

Resolution

To resolve the certificate mismatch, apply one of the following two options:

Option 1: Configure TLS Pass-through

1. Reconfigure the external Load Balancer for TCP/TLS pass-through instead of SSL termination. This ensures the client communicates directly with the VCFA certificate.

Option 2: Synchronize Certificates

1. Create a Certificate Signing Request (CSR) in VCF Operations.

2. Sign the CSR using a public Certificate Authority (CA).

3. Import the signed certificate into VCF Operations/VCFA.

4. Extract the private key using the following command: kubectl get secret ingress-vcfa-tls-external -n prelude -o jsonpath='{.data.tls\.key}' | base64 -d

5. Import the exact same certificate and extracted private key into the external Load Balancer.

Additional Information

• Log Retrieval: To retrieve VMRC logs, locate the vmware-vmrc-*.log file in the user's %APPDATA%\VMware (Windows) or ~/Library/Logs/VMware (macOS) directory.
• Contact Support: To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.

VCFA BLOG