VCF deployment fails with error “Hosts cannot be added to the cluster”

Products

• VMware Cloud Foundation

Issue/Introduction

• VCF deployment fails with error:
  
  Hosts cannot be added to the cluster domain-XX


• At this stage, the vCenter has already been deployed.
• Reconnecting the ESXi host from vCenter fails and returns the error:
  
  vapi.invalid.code <recv of frame failed with code 503 service unavailable>
  
  
• ESXi logs /var/run/log/envoy.log display the following error:
  
  remote address:<vCenter IP>:XXXX,TLS_error:|:SSL routines:OPENSSL_internal:BAD_PACKET_LENGTH|:SSL routines:OPENSSL_internal:DECRYPTION_FAILED_OR_BAD_RECORD_MAC:TLS_error_end"

Environment

VMware Cloud Foundation 9.X

Cause

The BAD_PACKET_LENGTH error indicates a network issue between vCenter and ESXi, such as MTU or DNS misconfiguration.

Resolution

1. Verify the MTU settings on the vSphere Standard or Distributed Switch, as well as on the physical switch. TCPDUMP may help perform deeper troubleshooting. Refer Packet capture on ESXi using the pktcap-uw tool.
2. Check the DNS configuration on vCenter and the DNS server; nslookup can help validate name resolution.