Troubleshooting VMRC connectivity in vRealize Automation
search cancel

Troubleshooting VMRC connectivity in vRealize Automation

book

Article ID: 326014

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

This article will provide details on how the VMware Remote Console brokering works through vRealize Automation including required prerequisites and troubleshooting tips for identifying and resolving problems.

Symptoms:
When connecting to a resource using the Remote Console (VMRC) option in vRealize Automation, you may experience these symptoms:

  • Cannot connect to a resource using the Remove Console (VMRC) in VMware vRealize Automation.
  • Connecting to a resource using VMRC in VMware vRealize Automation fails
  • Users see a blank screen when attempting the VMRC connection.
  • Users are unable to connect using VMRC but only intermittently.
  • The following error(s) are seen:

    Cannot establish a remote console connection, verify that the machine is powered on if the server has self-signed certificate, you might need to accept certificate, then close and retry the connection.
[UTC:2018-06-05 10:15:15,547 Local:2018-06-05 15:45:15,547] vcac: [component="cafe:console-proxy" priority="DEBUG" thread="consoleproxy" tenant="" context="" parent="" token=""] com.vmware.consoleproxy.async.impl.ssl.operations.SSLEstablishConnectionOperation.performSelectionOperation:131 - Security Exception while establishing SSL channel

Environment

VMware vRealize Automation 7.x

Resolution

Prerequisites for successful VMRC connections:

  • Network traffic must be allowed between vRealize Automation appliance(s) and the ESXi server over port 902.
  • Network traffic must be allowed between vRealize Automation appliance(s) and the client browser over port 8444.
  • Network traffic must be allowed between vRealize IaaS Web component Windows server(s) and associated vSphere Endpoint(s) over port 443.
  • Certificate trust between the client browser and the vRealize Automation appliance(s).
  • Certificate trust between vRealize Automation appliance(s) and the ESXi server must be established, or you will see SSL handshake failures in the catalina.out (after increasing the console-proxy logging level).
  • Client browser must have WebSockets support enabled (this is usually on by default)
  • Servers must be resolvable by name between vRealize Automation and ESXi.
  • If using vCD, the VMRC plugin must be installed and allowed in the client browser.
  • Time Sync (not Time Zone) must be up to date and accurate for the local client, vRA Appliance, and ESXi host.
 

How VMRC communication works (in relation to vRealize Automation):

  • When you choose Connect to Remote Console from the browser, a new window or tab will come up with a session initiated between your browser and port 8444 on the vRealize Automation appliance your session is running on. If you choose Connect using VMRC, vRealize Automation will attempt to use the installable VMRC client instead of a proxied window.
  • vRealize Automation then initiates a TCP secure (SSL trusted) handshake to the ESXi host that the VM resides on using port 902.
  • Once the handshake is accepted, a session is established between a random port on vRealize Automation and port 902 on the ESXi host.
  • The session is then presented to the browser through the new window or browser over port 8444.

How it works from the vCenter thick client (briefly):

  • When you open a console, the 902 session is established directly between your client and ESXi.

How it works from the vSphere web client (briefly):

  • When you open a console, the 902 session is established between the vCenter Web Client server and ESXi and the session is brokered to your browser similiarly to vRealize Automation.

Troubleshooting Methodology:

  • You will first need to identify if the problem is occurring in vRealize Automation, vCenter, or a specific ESXi host by testing VMRC connections to various vCenters and hosts.
  • Then you will want to confirm prerequisites using ping, netstat, curl, and checking browser settings, etc.

Troubleshooting VMRC issues in vRealize Automation:

Increase logging for the console proxy service temporarily for testing:

  1. SSH into the vRealize Automation appliance(s) and edit the file: /usr/lib/vcac/server/webapps/console-proxy-service/WEB-INF/classes/log4j2.xml
  2. Add the following line to the top of the <Loggers> section: <Logger name="com.vmware.consoleproxy" level="all" includeLocation="true"/>
  3. Be sure to remove this line after testing (or change the level to "info" to avoid filling up your vRealize Automation logs)
  4. Monitor the console proxy logs by then running the command: tail -f /var/log/vcac/catalina.out | grep "console-proxy"

Network troubleshooting:

  1. SSH into the vRealize Automation appliance(s) and confirm that you can:
    • Ping the ESXi hosts by name.
    • Connect to port 902 on the ESXi host by using the command 'curl -vvv telnet://IP of ESXi:902'. You can also use 'nc6 IP of ESXi 902'
    • Look for open 902 sessions on vRealize Automation by running the command 'netstat -anp | grep 902'
  2. SSH into the ESXi server and confirm that you can:
    • Ping the vRealize Automation hosts by name.
    • Look for open 902 sessions on the ESXi server by running the command 'esxcli network ip connection list'

Certificate troubleshooting:

  1. Navigate to the ESXi host certificate by navigating to https://FQDN-orIP-ofHOST.
  2. Review certificate details to confirm validity and that the certificate would be trusted by the vRealize Automation appliance(s). If the certificate was issued by vCenter (default behavior), copy the SSL thumbprint.
  3. Compare the host certificate thumbprint to the VPX_HOST table EXPECTED_SSL_THUMBPRINT and HOST_SSL_THUMBPRINT to verify that the certificate matches. If it does not, then the problem may be that the host certificate has been updated and is not secure when vRA attempts to make the handshake.

Note: If multiple DNS entries exists for ESXi in DNS settings, add all host FQDN in /etc/hosts file.

Increase VMRC default timeout value:


Before proceeding best practice dictates you first snapshot any vRealize Automation appliances and take manual backup of the /etc/vcac/security.properties file.

Steps to increase the timeout value:

  1. Open an SSH connection to each vRealize Automation appliance node
  2. Edit security.properties
vi /etc/vcac/security.properties
  1. Append the following line to the end of the file (Increasing from default 10 secs to 20 secs)
consoleproxy.timeout.connectionInitMs=20000
  1. Save and exit
  2. Stop and start the vcac-server service on each node.
  3. Stop services on all replica nodes:
service vcac-server stop
  1. Stop services on the primary node
service vcac-server stop
  1. Start services on primary node
service vcac-server start
  1. Verify all services are correctly registered on the primary node before starting the replica node services.
  2. Start services on all replica nodes:
service vcac-server start

Remove invalid TLS / SSL protocols:

Invalid protocols may reside within/etc/vcac/security.properties causing connection failures:

  1. Open an SSH connection to each vRealize Automation appliance node
  2. Edit security.properties
vi /etc/vcac/security.properties
  1. Remove the following line:
consoleproxy.ssl.server.protocols=SSLv2Hello,TLSv1.2,TLSv1.1
  1. Save and exit
  2. Stop and start the vcac-server service on each node.
  3. Stop services on all replica nodes:
service vcac-server stop
  1. Stop services on the primary node
service vcac-server stop
  1. Start services on primary node
service vcac-server start
  1. Verify all services are correctly registered on the primary node before starting the replica node services.
  2. Start services on all replica nodes:
service vcac-server start

 

 

Additional Information

To be alerted when this article is update, click Subscribe to Article.
 
To enable support for SSLv3 in ESXi 6.x, refer to the KB 2121021
Powered by Wolken Software