Error: "Compatibility check took too long to complete" when selecting target Host during cross vCenter vMotion

Products

• VMware vCenter Server

Issue/Introduction

• Cross vCenter vMotion or Virtual Machine import operations may fail at the validation stage with the following error: "Compatibility check took too long to complete. Please check vCenter Server logs for more details."
  
  
• Log entries in the destination host's /var/run/log/hostd.log indicate connectivity issues on port 902:
  [NFC ERROR]NfcNewAuthdConnectionEx: Failed to connect to server <IP_address>:902
[NFC ERROR]NfcEstablishAuthCnxToServer: Failed to create authentication connection
[NFC ERROR]Nfc_BindAndEstablishAuthdCnx3: Failed to connect to server <IP_address>:902


• These errors prevent successful Virtual Machine transfers, as port 902 is critical for vMotion operations.

Environment

• VMware vCenter Server 7.x & 8.x
• ESXi hosts 7.x & 8.x

Cause

The primary cause of this issue is network connectivity problems between the ESXi hosts involved in the cross vCenter vMotion. Specifically:

1. Port 902 is blocked or unreachable between the source and destination ESXi hosts.
2. This port is required for the Network File Copy (NFC) service, which handles Virtual Machine data transfer during vMotion.
3. Firewall rules, NAT configurations, or routing issues may be preventing the direct host-to-host communication required for vMotion.

Additional factors that can contribute to this issue include:

• Time synchronization problems between vCenter Servers and ESXi hosts.
• Firmware and driver incompatibilities.
• Storage connectivity issues between environments.

Resolution

To resolve the "Compatibility check took too long to complete" error during cross vCenter vMotion:

1. Verify direct network connectivity between source and destination ESXi hosts:

   1. Test connectivity to port 902 using netcat from the source ESXi host:
      
      nc -z -v <destination_host_IP> 902

   2. If the connection failure persists, verify that port 902 is open between the hosts by contacting the internal network team.

2. Ensure all cross vCenter vMotion prerequisites are met:

   1. Review and implement the requirements in KB article (Cross vCenter Migration and Clone requirements).

   2. Particular attention must be given to the networking requirements section.

3. Check and correct time synchronization:

   1. Configure all vCenter servers and ESXi hosts to use the same NTP servers.

   2. Verify NTP synchronization is working.

4. Check for NAT configurations:

   1. Note that using NAT between vCenter Server and ESXi hosts is an unsupported configuration with no workaround per KB "Using NAT between the vCenter Server system and ESXi hosts will disconnect the host from the vCenter".

   2. Ensure direct routing between all components involved in the migration.

5. Verify storage connectivity:

   1. Ensure the destination host has proper access to the target datastore.
      
      
   2. For NFS datastores, check that the NFS server is accessible from the destination host.
      
      
6. Review and address any firmware or driver update requirements:
   1. Update host firmware and drivers to the latest supported versions.
      
      
   2. Ensure compatibility between source and destination hosts.

Additional Information

• For additional troubleshooting of connectivity issues, capture and analyze network traffic during a migration attempt using the pktcap-uw utility. Replace "vmk#" with the actual management vmk number for the affected ESXi host.
  pktcap-uw --vmk vmk# --dir 2 --outfile /tmp/vmk#_capture.pcap

• MTU mismatches can also cause connectivity issues. Ensure consistent MTU settings across all virtual and physical network infrastructure.
  
  
• As a possible work around, attempt to import the Virtual Machine from the destination vCenter - Import or Clone a Virtual Machine with Advanced Cross vCenter vMotion