HCX L2E VMs lose connectivity to source gateway after NSX Segment updates

Products

• VMware Cloud on AWS
• VMware HCX
• VMware NSX

Issue/Introduction

• HCX L2E segments were extended when NSX on cloud/destination side is/was on version lower than 4.1.1
• After NSX Segment updates (either manual update or during NSX upgrade workflow), VMs on the destination/cloud side with MON (Mobility Optimized Networking) disabled may lose connectivity to the source side gateway.
• VMs hosted on the source/connector side would be able to reach the gateway.
• HCX L2E VMs on the destination/cloud side would be able to communicate to VMs on same segment at the source/connector side.

Environment

VMware HCX

Cause

• The affected L2E segments on the HCX destination/cloud side SDDC will have the property "com.vmware.nsx.port.extraConfig.remoteRtr" missing without which HCX will not be able to translate the MAC address. This results the packets to be forwarded to the VDR port instead of the NE (Network Extension) appliance port.
• Underlying cause for the property "com.vmware.nsx.port.extraConfig.remoteRtr" missing:-
  • For all NSX versions lower than 4.1.1, HCX sets the property "com.vmware.nsx.port.extraConfig.remoteRtr" directly on the NSX Logical Switch (NSX Manager Mode entity) while not on the NSX Segment (NSX Policy Mode entity).
  • Any changes set on NSX Logical Switch would not be reflected in NSX Segment.
  • With this configuration where the property "com.vmware.nsx.port.extraConfig.remoteRtr" is set directly on the NSX Logical Switch while not on the NSX Segment, any changes made to the NSX Segment (Eg. Rename, configuration change etc.) will result in the property "com.vmware.nsx.port.extraConfig.remoteRtr" to be lost on the NSX Logical Switch (because NSX Segment properties gets pushed to it).

Resolution

• If this issue arises, the fix would be to unextend and re-extend the affected HCX L2E segments from the HCX source/connector side (Note: Do not perform this operation from HCX destination/cloud side as that is not supported and may result in issues). This will result in the reinjection of the property "com.vmware.nsx.port.extraConfig.remoteRtr".
• If fix is not possible or delayed, a potential workaround is to migrate the VMs temporarily back to the source/connector side.
• This issue will not arise on HCX L2E segments which were extended while NSX on cloud/destination side is on version 4.1.1 or higher as HCX will set the property "com.vmware.nsx.port.extraConfig.remoteRtr" on the NSX Segment (which will in turn get pushed to NSX Logical Switch).
• Once NSX on cloud/destination side is upgraded to version 4.1.1 or higher, for all HCX L2E segments extended before the upgrade, the proactive step of unextend and re-extend of the segment can be done so that HCX will set the property "com.vmware.nsx.port.extraConfig.remoteRtr" on the NSX Segment (which will in turn get pushed to NSX Logical Switch). This is a permanent fix and will result in this issue to not arise again.

Additional Information

Related KB: [VMC] HCX L2E VMs lose connectivity to source gateway post NSX Manager upgrade