Cluster expansion fails with "HOST_NETWORK_VALIDATION_FAILED" due to missing faultLevel metadata in VCF Stretched Clusters

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting to expand a vSAN Stretched Cluster in VMware Cloud Foundation (VCF) by adding new ESXi hosts, the workflow fails at the Validate Expand Stretch Cluster ESXi Host(s) Network task.

• Error in SDDC Manager UI:
  • Failed to validate host's network.
  • Please make sure host management VLAN id and networkPoolId matches the specified fault domain.
• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log:
  
  

  ERROR [vcf_dm,uuid,tid] [c.v.e.s.s.t.ValidateExpandStretchHostNetworkAction,dm-exec-X] Host host_name VlanId XXXX is not same as availability zones VlanId YYYY
ERROR [vcf_dm,uuid,tid] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-X] [3KERS8] HOST_NETWORK_VALIDATION_FAILED Failed to validate hosts network.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate hosts network.

Environment

VCF 9.x

Cause

This stems from a known issue in VMware Cloud Foundation 9.0.0 inventory synchronization process. During sync operations, the faultLevel metadata tag is incorrectly stripped from the  the port_groups JSON array in vds table. If a synchronization occurred while the environment was on version 9.0.0, the field remains blank even after upgrading to version 9.0.1 or 9.0.2.

In a stretched cluster configuration, SDDC Manager uses these tags (PRIMARY for AZ1 and SECONDARY for AZ2) to distinguish between site-specific network settings. If these tags are stripped (which can happen following certain out-of-band changes or sync operations in VCF 9.x), the orchestrator cannot correctly identify the Availability Zone VLAN, leading to a validation mismatch.

Resolution

The issue where inventory synchronization stripsfaultLevel metadata field is resolved in VCF 9.0.1.0.

To resolve this issue, the missing faultLevel metadata must be manually restored to the VDS entry in the SDDC Manager database.

Note: It is recommended to take an offline snapshot of the SDDC Manager VM before performing manual database modifications.

Step 1: Identify the VDS ID

1. SSH to the SDDC Manager as vcf and switch to root.
2. Log in to the platform database:

   psql -U postgres -h localhost -d platform

3. Identify the VDS associated with the failing cluster:

   SELECT id, name FROM vds;

   Note the ID for the VDS used by the stretched cluster (e.g., b7df4fea-5cc2-4ff5-a344-a4b241jka7c00).

Step 2: Verify the missing metadata

Check the current port_groups configuration:

SELECT port_groups FROM vds WHERE id='YOUR_VDS_ID';

Look for the absence of "faultLevel": "PRIMARY" or "faultLevel": "SECONDARY" within the port group objects.

Step 3: Update the Database

Run an UPDATE statement to re-inject the correct JSON structure including the faultLevel tags for your specific environment. Note: Ensure you include all existing port groups in the string to avoid data loss.

Example Update (Adjust names/IDs/VLANs to match your specific output from Step 2):

UPDATE vds SET port_groups='[  {"name":"pg-mgmt-az1","transportType":"MANAGEMENT","vlanId":2300,"faultLevel":"PRIMARY",...},  {"name":"pg-mgmt-az2","transportType":"MANAGEMENT","vlanId":2307,"faultLevel":"SECONDARY",...}]' WHERE id='YOUR_VDS_ID';

Step 4: Restart Services

Exit the database and restart the core SDDC Manager services:

systemctl restart commonsvcs
systemctl restart domainmanager

Step 5: Retry the Workflow