Prerequisites for successful VMRC connections:
- Network traffic must be allowed between vRealize Automation appliance(s) and the ESXi server over port 902.
- Network traffic must be allowed between vRealize Automation appliance(s) and the client browser over port 8444.
- Network traffic must be allowed between vRealize IaaS Web component Windows server(s) and associated vSphere Endpoint(s) over port 443.
- Certificate trust between the client browser and the vRealize Automation appliance(s).
- Certificate trust between vRealize Automation appliance(s) and the ESXi server must be established, or you will see SSL handshake failures in the catalina.out (after increasing the console-proxy logging level).
- Client browser must have WebSockets support enabled (this is usually on by default)
- Servers must be resolvable by name between vRealize Automation and ESXi.
- If using vCD, the VMRC plugin must be installed and allowed in the client browser.
- Time Sync (not Time Zone) must be up to date and accurate for the local client, vRA Appliance, and ESXi host.
How VMRC communication works (in relation to vRealize Automation):
- When you choose Connect to Remote Console from the browser, a new window or tab will come up with a session initiated between your browser and port 8444 on the vRealize Automation appliance your session is running on. If you choose Connect using VMRC, vRealize Automation will attempt to use the installable VMRC client instead of a proxied window.
- vRealize Automation then initiates a TCP secure (SSL trusted) handshake to the ESXi host that the VM resides on using port 902.
- Once the handshake is accepted, a session is established between a random port on vRealize Automation and port 902 on the ESXi host.
- The session is then presented to the browser through the new window or browser over port 8444.
How it works from the vCenter thick client (briefly):
- When you open a console, the 902 session is established directly between your client and ESXi.
How it works from the vSphere web client (briefly):
- When you open a console, the 902 session is established between the vCenter Web Client server and ESXi and the session is brokered to your browser similiarly to vRealize Automation.
Troubleshooting Methodology:
- You will first need to identify if the problem is occurring in vRealize Automation, vCenter, or a specific ESXi host by testing VMRC connections to various vCenters and hosts.
- Then you will want to confirm prerequisites using ping, netstat, curl, and checking browser settings, etc.
Troubleshooting VMRC issues in vRealize Automation:
Increase logging for the console proxy service temporarily for testing:
- SSH into the vRealize Automation appliance(s) and edit the file: /usr/lib/vcac/server/webapps/console-proxy-service/WEB-INF/classes/log4j2.xml
- Add the following line to the top of the <Loggers> section: <Logger name="com.vmware.consoleproxy" level="all" includeLocation="true"/>
- Be sure to remove this line after testing (or change the level to "info" to avoid filling up your vRealize Automation logs)
- Monitor the console proxy logs by then running the command: tail -f /var/log/vcac/catalina.out | grep "console-proxy"
Network troubleshooting:
- SSH into the vRealize Automation appliance(s) and confirm that you can:
- Ping the ESXi hosts by name.
- Connect to port 902 on the ESXi host by using the command 'curl -vvv telnet://IP of ESXi:902'. You can also use 'nc6 IP of ESXi 902'
- Look for open 902 sessions on vRealize Automation by running the command 'netstat -anp | grep 902'
- SSH into the ESXi server and confirm that you can:
- Ping the vRealize Automation hosts by name.
- Look for open 902 sessions on the ESXi server by running the command 'esxcli network ip connection list'
Certificate troubleshooting:
- Navigate to the ESXi host certificate by navigating to https://FQDN-orIP-ofHOST.
- Review certificate details to confirm validity and that the certificate would be trusted by the vRealize Automation appliance(s). If the certificate was issued by vCenter (default behavior), copy the SSL thumbprint.
- Compare the host certificate thumbprint to the VPX_HOST table EXPECTED_SSL_THUMBPRINT and HOST_SSL_THUMBPRINT to verify that the certificate matches. If it does not, then the problem may be that the host certificate has been updated and is not secure when vRA attempts to make the handshake.
Note: If multiple DNS entries exists for ESXi in DNS settings, add all host FQDN in /etc/hosts file.
Increase VMRC default timeout value:
Before proceeding best practice dictates you first snapshot any vRealize Automation appliances and take manual backup of the /etc/vcac/security.properties file.
Steps to increase the timeout value:
- Open an SSH connection to each vRealize Automation appliance node
- Edit security.properties
vi /etc/vcac/security.properties
- Append the following line to the end of the file (Increasing from default 10 secs to 20 secs)
consoleproxy.timeout.connectionInitMs=20000
- Save and exit
- Stop and start the vcac-server service on each node.
- Stop services on all replica nodes:
service vcac-server stop
- Stop services on the primary node
service vcac-server stop
- Start services on primary node
service vcac-server start
- Verify all services are correctly registered on the primary node before starting the replica node services.
- Start services on all replica nodes:
service vcac-server start
Remove invalid TLS / SSL protocols:
Invalid protocols may reside within/etc/vcac/security.properties causing connection failures:
- Open an SSH connection to each vRealize Automation appliance node
- Edit security.properties
vi /etc/vcac/security.properties
- Remove the following line:
consoleproxy.ssl.server.protocols=SSLv2Hello,TLSv1.2,TLSv1.1
- Save and exit
- Stop and start the vcac-server service on each node.
- Stop services on all replica nodes:
service vcac-server stop
- Stop services on the primary node
service vcac-server stop
- Start services on primary node
service vcac-server start
- Verify all services are correctly registered on the primary node before starting the replica node services.
- Start services on all replica nodes:
service vcac-server start