VCF Cluster Expansion Fails with Error 'No free hosts available, Add Host operation requires host with at least two physical NICs to proceed'

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When attempting to add a host to an existing cluster in SDDC Manager, the operation fails with one of the following errors:

  When attempting to add a host to an existing cluster in SDDC Manager, the operation fails with one of the following errors:

•  /var/log/vmware/vcf/domainmanager/domainmanager.log in the SDDC manager reports the the following entries:

  YYYY-MM-DDTHH:MM:SS.712+0000 INFO  [vcf_om,#######################,####] [c.v.e.s.c.s.c.CommonsServicesSdkClient,http-nio-127.0.0.1-7300-exec-10] Common svcs connection timeout is 30000
  YYYY-MM-DDTHH:MM:SS.712+0000 ERROR [vcf_om,#######################,####] [c.v.v.q.vsphere.VsphereQueryService,http-nio-127.0.0.1-7300-exec-10] Failed to acquire data for the following hosts [########-####-####-####-###########, ########-####-####-####-###########, ########-####-####-####-###########, ########-####-####-####-###########, ########-####-####-####-###########, ########-####-####-####-###########]
  java.lang.IllegalArgumentException: VMwareProductVersion can not be null or empty
          at com.vmware.evo.sddc.common.version.VMwareProductVersion.<init>(VMwareProductVersion.java:98)
          at com.vmware.vcf.query.vsphere.util.CssClientUtil.lambda$getHostCredentials$0(CssClientUtil.java:86)
          at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197

Environment

• VMware Cloud Foundation 9.x 

Cause

• This failure occurs due to missing metadata in the SDDC Manager platform database. Specifically, the host version field within the database host table is blank. This prevents the "Add Host" workflow from recognizing the host as a compatible resource.

Resolution

• To resolve this issue, the missing metadata must be updated manually in the SDDC Manager database:
  1. Take an offline snapshot of the SDDC Manager VM via the vCenter UI.
  
  2. Update the status and version information for the affected host within the SDDC Manager database.
     a. SSH to the SDDC Manager appliance using the vcf user credentials, then switch to the root user:
     b. Access the Platform Database. 
  
  

  psql -h localhost -U postgres -d platform

   c. psql -h localhost -U postgres -d platform. 

  SELECT id, hostname, status, version FROM host;

   d. Update the Host version.

  update host set version='ESXI version and build' where id='ESXi ID';

   e. Verify and Exit.

  SELECT id, hostname, status, version FROM host WHERE id = 'ESXi ID';

  f. Once verified, exit the PostgreSQL interface:

  \q

  3. Re-attempt the Add Host operation from the SDDC Manager UI.