Workaround for HCX Service Mesh Deployment Uplink Profile Failure in Multi-Org VMware Cloud Director Environments

Products

VMware HCX VMware Cloud Director

Issue/Introduction

When deploying or updating a VMware HCX Service Mesh in environments where multiple VMware Cloud Director (VCD) organizations are managed by a centralized HCX Cloud Manager, the deployment fails. The destination Interconnect appliance installs as a 2-vNIC node (Management and vMotion) instead of a 3-vNIC node, completely omitting the text interface mapping for the Uplink.

When deploying Service Mesh from HCX Manager connector appliance. In the Tasks view of the Service Mesh, the following error is presented

Service Mesh Creation Failed. Process Service Mesh failed. Interconnect Service Workflow GenerateAndPostConfig failed. Error: Operation timedout in state POST_CONFIG_VIX, Operation timedout in state POST_CONFIG_VIX

During runtime datapath compilation, the appliance configuration fails with explicit kernel execution errors visible in the virtual machine logs.

Log signatures from /var/log/messages inside the destination appliance show:

<131>1 <Date/Time> <REDACTED_HOSTNAME> cgw 1671 - - [Err-configer] : genIfIrqSmpAffsSameMaskCmds: num irqs invalid: 0 for if serv
<131>1 <Date/Time> <REDACTED_HOSTNAME> cgw 1671 - - [Err-rpcServer] : Config FAILED, (genIfIrqSmpAffsSameMaskCmds: num irqs invalid)(<nil>)

Companion alerts within the HCX Cloud Manager job engine log include:

ERROR c.v.v.h.s.v.OrgVdcInventoryRefreshJob- Error in VDC stats sync.JSONObject["enterprise"] not found.
java.lang.RuntimeException: Failed applying appliance configuration. Please check Service Mesh configuration. Please retry operation after ServiceMesh resync.

Environment

VMware Cloud Director (VCD) Multi-Tenant Infrastructure
VMware HCX 4.11.3 and 4.11.4

Cause

A software integration defect exists in the multi-org VCD environments when multiple VCD organizations leverage the same cloud migration gateway endpoints, target Uplink network profile override selection within the source Service Mesh wizard are silently ignored on the destination side for all but one primary organization.

If the destination Compute Profile contains no explicit Uplink Network Profile definition, the automated deployment engine assumes a 2-vNIC deployment. The virtual appliance virtual machine is instantiated at the hypervisor layer without a physical placeholder interface card for the Uplink (vNIC2).

Resolution

To bypass this multi-org processing defect, a structural workaround must be configured within the destination Compute Profile to force physical network device allocation. Follow these steps:

Log into the source HCX Connector UI on-premises. Go to Infrastructure > Service Mesh, select the failing Service Mesh instance, and select Delete to clear out unconfigured, incomplete virtual machine appliances from the inventory.
Log into the destination HCX Cloud Manager UI with cloud provider system administrator credentials.
Navigate to Infrastructure > Compute Profiles and select Edit on the Compute Profile instance mapped to the affected tenant organization VDC.
Advance to the Network Profiles allocation section of the configuration wizard.
Under the External / Uplink network traffic type assignment, configure a baseline default mapping by selecting an HCX Network Profile mapped directly to the appropriate backing NSX segment allocated to that VCD Org Network.
Progress through the remaining steps of the wizard and select Finish to commit and save the updated profile.
Return to the source on-premises HCX Connector UI. Launch the Create Service Mesh workflow. On the override uplink panel, re-select the target local and destination networks matching the architectural blueprint.
Complete the wizard to instantiate the mesh. The destination appliance will now deploy successfully with three distinct physical interfaces (vNIC0, vNIC1, vNIC2), allowing subsequent reconfiguration attempts to execute without a kernel IRQ fault.

Additional Information

VMware HCX for VMware Cloud Director