[HCX] Network Extension task fails with the error : "Failed to resolve vc backing network moref for network <logical-switch path>"

Products

• VMware HCX
• VMware NSX

Issue/Introduction

Attempting to extend VDS network using HCX fails with the below error:

Source HCX-MGR UI:

 "Failed to resolve vc backing network moref for network <logical-switch path>"

 

Source HCX-MGR app.log (common/logs/admin/app.log):

2024-10-01 21:27:16.173 UTC [NetworkStretchService_SvcThread-1092, j: 4b4f36d1, s: 807d07ef, , TxId: ########-####-####-####-d4a11210bcbe] INFO  c.v.v.h.n.i.NetworkStretchJobInt- Preparing destination network.
2024-10-01 21:35:01.963 UTC [NetworkStretchService_SvcThread-1097, j: 4b4f36d1, s: 807d07ef, , TxId: ########-####-####-####-d4a11210bcbe] INFO  c.v.v.h.n.i.NetworkStretchJobInt- Running in state PROCESS_PREPARE_DESTINATION_NETWORK_RESULT
2024-10-01 21:35:01.965 UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- Received a Failed Job, jobType: NetworkStretchJobs workflow type: prepareCloudNetwork
2024-10-01 21:35:01.970 UTC [NetworkStretchService_SvcThread-1097, j: 4b4f36d1, s: 807d07ef, , TxId: ########-####-####-####-d4a11210bcbe] ERROR c.v.v.h.n.i.AbstractJobInt- NetworkStretchJobs workflow prepareCloudNetwork failed. Error: Operation timedout in state CREATE_NETWORK

 

Cloud HCX-MGR app.log (common/logs/admin/app.log):

2024-10-01 21:27:23.341 UTC [NetworkStretchService_SvcThread-59, j: 6795c634, s: 807d07ef, , TxId: TxId: ########-####-####-####-8ffd717a64a2] WARN  c.v.v.h.networkstretch.AbstractJob- Failed to resolve vc opaque network moref for logical switch ########-####-####-####-68fcd8e3442f: Failed to get opaque network moref with logical switch id ########-####-####-####-68fcd8e3442f

Environment

HCX

NSX

Cause

This message indicates that HCX is not able to retrieve the NSX Segment information via vCenter for the cluster specified in the cloud side Compute Profile. 

The steps taken during HCX network extension as it relates to this error are as follows:

1. HCX sends API call to NSX to create X segment in the transport zone specified in cloud compute profile.
2. NSX chooses first host/cluster in its list to create the segment.
3. NSX creates the segment and returns the name to HCX.
4. HCX then requests VC to check if this network exists in resource cluster chosen in cloud compute profile.

At this time there is no way for HCX to specify a particular cluster/TZ to create a network segment in. 

Verify that the HCX L2E segment is being created on the appropriate VDS. In this example, there were about 5 Clusters attached to 1 Transport zone (1 VDS per cluster). It was discovered that the HCX segment was not being created on all VDS's in the transport zone due to an NSX inventory sync issue.

Resolution

Perform the below steps:

1. Verify there are no decommissioned hosts still present in NSX inventory. If so, perform a force delete from NSX UI.
2. Run a "start search resync all" & "start search resync inventory" on all 3 NSX Managers from admin CLI.
3. Create a test segment in the same T1 which is being selected for HCX L2E deployment.
4. Verify the segment is created on all VDS's attached to the Transport Zone.