vCenter and host communication issue for HA agent - Error "Select vSphere HA agent for this host has an error: vSphere HA agent cannot be installed or configured"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Symptom:

• vSphere HA agent error is seen in Web UI - "Select vSphere HA agent for this host has an error: vSphere HA agent cannot be installed or configured"
  • During troubleshooting, vCenter is found to be unable to connect to a host reliably for transferring files or when checking the host's SSL certificate to verify connectivity:  
    • Examples: 
      • When trying to scp a file from vCenter to a host, the connection is established, but the file fails to transfer to the host. 
        Basic networking connectivity tests are successful: - ping, curl from vCenter to host, and nc from the host to vCenter.
        
        
      • The output from an OpenSSL connectivity command to show a remote host's certificate information is incomplete.
        • OpenSSL Command Syntax: 
          openssl s_client -connect <hostNameFQDN>:443

Environment

vSphere vCenter and ESXi environments

 

Cause

vCenter's operating system will evaluate network connections based on the connected system's MAC addresses to determine the shortest path for sending traffic back to to a connection. 

• Where a host is configured with multiple vmkernel adapters which use the same physical network adapters, vCenter will transmit packets over the shortest networking path back to that host for communication. 
  
  
• In an environment where the host's vmkernel port used for the management service traffic (typically vmk0) is configured on a remote network subnet and another vmkernel adapter has its IP configured on the same network as the vCenter, the vCenter will send traffic packets to the local vmkernel IP address resulting in connectivity issues between the vCenter and host.  
  
  
  • Example Impacted Configurations:
    • Host:
      • vmk0: 
        • IP address on Network A
        • Management service enabled. 
      • vmk1: 
        • IP address configured for network B
        • Management service is not enabled.  The vmkernel is used for other host service traffic.
      • Both vmk0/1 use same physical network adapters (share the same MAC addresses)
        
        
    • vCenter: 
      • VCSA IP address configured on network B

Resolution

Review hosts' vmkernel adapters IP addresses configurations and identify if any of the IPs are configured on the same network as the vCenter's IP address. 
Verify the vmkernel on the same network is not enabled for management service traffic.

Resolve the networking configuration issue - 

1. On the host(s) - Reconfigure/move the non-management service vmkernel adapter that is on the same network as the vCenter's IP to a different network subnet
   or
2. Enable/move management service for a vmkernel on the same network the vCenter's IP is on.
   NOTE:  Enable management service for only one vmkernel adapter on a host. Enabling management service on multiple VMkernel adapters creates a multihomed configuration for the management service which is not a recommended configuration by VMware.