Cannot establish a remote console connection with certificate issues when trying to use Connect to Remote Console after upgrading to VMware Aria Automation 8.18

Products

VMware Aria Suite

Issue/Introduction

Cannot open a Remote Console in Aria Automation 8.18.x when using vCenter 8.0.3
Launching the VM remote console from vRA, reports one of these errors:
- 408 OK
- Cannot establish a remote console connection. Verify that the machine is powered on. If the server has a self-signed certificate, you might need to accept the certificate, then close and retry the connection
- Error recieved from server: Failed to verify SSL context: [vRA host: automation-FQDN.example.com communicating with vCenter Host: wss://vCenter-FQDN.example.com]
Remote Console Day 2 Action does not work for cloud accounts with missing 'certificate' property.
- The certificate property (normally found in customProperties and endpointProperties) may be missing if the cloud account is using 'acceptSelfSignedCertificate=true', cloud account is added across multiple tenants or was deleted and re-added.
The /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log file may have a NullPointerException error similar to:

2024-09-03T05:19:27.504Z ERROR provisioning [host='provisioning-service-app-<UID>' thread='reactor-http-epoll-5' user='' org='' trace='' parent='' span=''] c.v.w.p.server.WebsocketHandlerFromUI.handleSSLContext:160 - [vRA host: <AriaAutomationFQDN> communicating with vCenter Host: wss://<vCenterServerFQDN>:443/ticket/<ID>]: Trusted connection cannot be established with host wss://<vCenterServerFQDN>:443/ticket/<ID>. Error:
java.lang.NullPointerException: Cannot invoke "com.vmware.webmks.proxy.ticket.api.ssl.WebMksCertificateThumbprint.toString()" because the return value of "com.vmware.webmks.proxy.ticket.api.WebMksTicketPayload.getEndpointThumbprint()" is null

Environment

VMware Aria Automation 8.18.x

Cause

Beginning with VMware Aria Automation 8.18, a new remote console proxy was introduced to add support for webMKS based remote web console sessions which is mandatory in vSphere 8.0.

For Cloud Accounts in Aria Automation with acceptSelfSignedCertificate set to true (found within the Cloud Account customProperties), the remote web console day 2 action will fail.

This is due to the new proxy validating the certificate stored in the Cloud Account state (found in endpointProperties and customProperties), for any Cloud Account created with acceptSelfSignedCertificate=true, this certificate property will not be added automatically.

By default, vSphere Cloud Accounts in Aria Automation are not created with this property set, only when created (or updated) via the API.

Impact

Patching the Cloud Account with the correct certificate as described in the resolution below will add the certificate to the cloud account state within both the endpointProperties and customProperties.

Resolution

In many cases, it is necessary to patch the Cloud Account with the valid vCenter certificate as outlined in:
- KB 318756 - vSphere Cloud Account certificate is changed causing errors such as Unavailable for Deployment and Failed to validate.
If there are multiple tenants using the remote console which should be proxied through Automation, please run the resolution workflow on each tenant.
This may be needed even when the correct certificate can be seen in the cloud account, as it will correct any missing information here.
The JAVA_OPTS for the provisioning-service-app must also contain -Denable.remote-console-proxy=true
- See this KB article for details around this option and how to set it:
- KB 314900 - Cannot establish a remote console connection in Aria Automation on-prem and Cloud

Workaround

It is also possible to work around this by disabling the remote console proxy on 8.18.x:

Disable the new remote proxy by toggling the feature remote.console.proxy.webmks.enabled to false.
This will fall back to the remote console behavior that exists in previous versions of Aria Automation, meaning connections are made directly to ESXi (or vCenter proxy).
```
https://AriaAutomationFQDN/automation/#/service/automation-ui/provisioning-ui;ash=%2FconfigurationProperties
```

Additional Information

If the issue persists after performing steps in vSphere Cloud Account certificate is changed causing errors such as Unavailable for Deployment and Failed to validate

Check again the provisioning service logs /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log

If you now see an entry similar to below its likely that the certificate has been patched but in an incorrect format including line break characters '/n'.

2024-09-05T15:16:40.312Z ERROR provisioning [host='provisioning-service-app-<UID>' thread='reactor-http-epoll-6' user='<User>' org='<OrgId>' trace='<TraceId>' parent='<ParentId>' span='<SpanId>'] c.v.a.r.c.ComputeRemoteConsoleController.lambda$authenticateAndGetWebMksTicket$3:257 - Failed processing certificate from endpoint for remote console operation. Failure:
java.security.cert.CertificateException: Unable to initialize, java.io.EOFException: not enough contentFollow the steps below to capture the correct certificate format and submit the api calls again from vSphere Cloud Account certificate is changed causing errors such as Unavailable for Deployment and Failed to validate:

Step 1. Export/Download the Certificate

Step 2. Right Click the certificate>>Edit with Notepad++

Step 3. This will show the right certificate without unwanted "/n"

Step 4. You can also get the same by command: openssl s_client -showcerts -connect IP/FQDN:443