Expansion of Management vSphere Cluster is failing on VLAN based AVN

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
Any expansion operations (Add Host/vSAN Stretch Cluster/Expand vSAN Stretch Cluster) executed on the Management vSphere Cluster in the Management Workload Domain where the environment's network topology is VLAN-based AVN configured during VCF 3x4x Migration with MR1 and MR2 releases and/or ingested AVN following this KB: https://kb.vmware.com/s/article/80864, is going to fail at "Attach VLAN Transport Zone to Host Transport Nodes" action with the following error in domainmanager.log:

2022-08-02T16:59:17.024+0000 ERROR [vcf_dm,9d0a4a46cfca44fa,0d6f] [c.v.v.c.f.p.n.a.NsxtAddHostVlanHeader,dm-exec-10] Error occurred while generating input for add hosts in NSX-T environment
java.lang.NullPointerException: null

2022-08-02T16:59:17.024+0000 ERROR [vcf_dm,9d0a4a46cfca44fa,0d6f] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10] [GL0M7C] ERROR_OCCURRED_WHILE_GENERATING_INPUT Error occurred while generating input for add hosts in nsxt environment

com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Error occurred while generating input for add hosts in the NSX-T environment

Caused by: java.lang.NullPointerException: null at com.vmware.vcf.common.fsm.plugins.nsxt.action.NsxtAddHostVlanHeader.execute(NsxtAddHostVlanHeader.java:218)
... 18 common frames omitted

2022-08-02T16:59:17.471+0000 DEBUG [vcf_dm,9d0a4a46cfca44fa,e5d3] [c.v.e.s.c.s.a.l.LockingServiceAdapterImpl,dm-exec-12] Execution name ADD_HOST_WORKFLOW, execution ID 99ad7352-####-####-####-########06e, resource type DEPLOYMENT, resource ID null.

Environment

VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 3.x
VMware Cloud Foundation 3.0.x
VMware Cloud Foundation 4.x

Cause

The association between the Management vSphere Cluster and the VLAN Transport Zone in the SDDC Manager Database is not populated.

Resolution

This issue will be resolved in future releases.

In order to have expansion operations on the Management vSphere Cluster in the Management Workload domain, the following resolution must be executed.

Workaround:

1. Login to the NSX-T instance for the Management workload domain with user admin.
2. Navigate to System > Fabric > Transport Zones
3. Select the VLAN Transport Zone created by VCF, which the AVN Network Segment is connected to, copy the ID to notepad.
4. SSH to the SDDC Manager VM as user vcf.
5. Run the following commands to access the SDDC manager database.
   su -
   psql -h localhost -U postgres
   \c platfom;
6. Run the following command to verify the current entries in the entity_and_transport_zone table.
   select * from entity_and_transport_zone;
7. The VLAN Transport ID should be missing from the entity_id which should be the correspinding vSphere cluster. Copy the entity_id to a notepad:
   platform=# select * from entity_and_transport_zone;
                     id | creation_time | modification_time | entity_id | vlan_transport_zone_ids | overlay_transport_zone_ids--------------------------------------+---------------+-------------------+--------------------------------------+------------------------------------------+------------------------------------------ 40e22d0d-####-####-####-########f21 | 1660897720387 | 1660897720387 | 3636b7cb-####-####-####-########dd1 | <Null> | ["d1b4de1d-####-####-####-########381"]
8. Verify that the entity_id missing the VLAN transport id is that of a vSphere cluster. Run the following command to verify the Management cluster is the entity_id
   
   select id,name from cluster where id='<entity_id>';
9. Update the database with the missing entry for VLAN transport zone id.
   update entity_and_transport_zone set vlan_transport_zone_ids ='["<Transport_Zone_ID from step 3"]' where entity_id='<entity_id from st