VMs on NSX segment cannot communicate unless moved to another host

Products

VMware NSX

Issue/Introduction

VMs on an NSX segment cannot ping their gateway
This is affecting all VMs on an ESXi Transport Node.
VMs have full connectivity when vMotioned to another ESXi Transport Node.
Additional identification:

SSH to the affected ESXi Transport Node as the root user.
Execute:
```
# nsxcli
```

Find the UUID for the affected segment:

> get logical-switches | find "<segment name>"
<timestamp>
66561     ########-####-####-####-#########dde   <segment name>

Find the logical router UUID for this segment:

> get logical-switch ########-####-####-####-#########dde | find "Routing Domain"
<timestamp>
Routing Domain           : ########-####-####-####-#########3f4 <--- Note this UUID for the next command
Multicast Routing Domain : ########-####-####-####-############

Review the forwarding table for the identified logical router:

> get logical-router ########-####-####-####-#########3f4 forwarding
<timestamp>
                                       Logical Routers Forwarding Table
--------------------------------------------------------------------------------------------------------------
Flags Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]
[H: Host], [R: Reject], [B: Blackhole], [F: Soft Flush], [E: ECMP]

                   Network                               Gateway                Type               Interface UUID
==============================================================================================================
0.0.0.0/0                                              169.254.0.#              UGE     ########-####-####-####-############   <---  Entry of Type 'UG' will be missing
##.##.##.##/##                                           0.0.0.0                UCI     ########-####-####-####-############
##.##.##.##/##                                           0.0.0.0                UCI     ########-####-####-####-############
...

The above example is a working scenario. A default route (0.0.0.0/0) is present. In a scenario matching this KB, no default route of Type UG exists.

Environment

VMware NSX

Resolution

The routes were not fully created for this segment. Forcing a change to the segment may cause the routes to be created on the affected hosts.

Steps to force a segment update:

In the NSX UI, Navigate to Network>Segments and edit the segment that is not working.
Disable Gateway Connectivity for the segment and save the change (this will impact connectivity for all VMs on the segment).
Re-enable Gateway Connectivity for the segment and save the change.
Confirm the segment is now working on all ESXi Transport Nodes.

If the workaround did not address the problem, or if you would like to diagnose the cause of the issue, please open an issue with Broadcom support and provide the following:

NSX Manager support bundles.
ESXi host support bundles for hosts that are failing to create the segment routes.
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support: