Error: Unable to Connect to the MKS while Connecting to Remote Console

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to connect to the remote console you see this error:

Unable to connect to the MKS: Could not connect to pipe \\. \pipe\vmware-authdpipe: The system cannot find the file specified.

The VMRC logs located at %TEMP%\vmware-USERNAME\vmware-vmrc-####.log contain messages similar to:

YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxOpenTCPSocket: Cannot connect to server SERVERNAME:902: No connection could be made because the target machine actively refused it
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxConnectAuthd: Returning false because CnxAuthdConnect failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: Cnx_Connect: Returning false because CnxConnectAuthd failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: Cnx_Connect: Error message: Failed to connect to server SERVERNAME:902
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: VMClient_ConnectMksClientEx - Remote socket connection failed.
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: VMClient_ConnectMksClientEx - trying local socket connection to /vmfs/volumes/DATASTORE/VM/VM.vmx
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: VMClientConnectSocketEx
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxAuthdConnectPipe: Returning false because we ran out of retries when CreateFileW
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxAuthdConnect: Returning false because CnxAuthdConnectPipe failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: CnxConnectAuthd: Returning false because CnxAuthdConnect failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: Cnx_Connect: Returning false because CnxConnectAuthd failed
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: Cnx_Connect: Error message: Could not connect to pipe \\.\pipe\vmware-authdpipe within retry period
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: VMClient_ConnectMksClientEx - Local socket connection failed.
...
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: DlgUI: Unable to connect to the MKS: Could not connect to pipe \\.\pipe\vmware-authdpipe within retry period.
YYYY-MM-DDTHH:MM:SS-00:00| vmrc| I125: VMMgr: Finished opening VM at /vm/#_#######_vm-###/ from SERVERNAME:vm-###

Cause

The error "Unable to Connect to the MKS" occurs when:

• Attempting to connect to the remote console from vCenter Server to a virtual machine.
• connecting to the remote console from different subnets through a firewall.
• When name resolution for the ESXi or vCenter FQDN does not return the correct or valid IP address. 

Resolution

To resolve the issue allow the VM Remote Console port-902 to vCenter and the ESXi hosts.

1. If the user is in a different layer 3 subnet, check firewall settings:

   In other words, validate the Port connectivity from the desktop machine that you have logged in to the vCenter Server(from where the Host Client or Web Client is launched), it could be that the Firewall in the environment is blocked for port 902
   The connection is as follows: User --> TCP/UDP 902 --> vCenter

2. Remote Console access from the ESXi Host directly as below 

   The connection is as follows: User redirect --> TCP/UDP 902 --> ESXi Host

Note: If the host is added by DNS, it must have DNS resolution for both vCenter Server and the ESXi hosts. Use the NSLOOKUP command from both the client and the vCenter Server to confirm a valid forward and reverse DNS entry exists.